e2e-kops-aws-upgrade-kstable-kolatest-to-klatest-kolatest #1639492248842276864
Recent runs || View in Spyglass
Recent runs || View in Spyglass
| Result | FAILURE |
| Tests | 269 failed / 647 succeeded |
| Started | |
| Elapsed | 1h2m |
| Revision | master |
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[It\]\sExternal\sStorage\s\[Driver\:\sebs\.csi\.aws\.com\]\s\[Testpattern\:\sDynamic\sPV\s\(default\sfs\)\]\ssubPath\sshould\ssupport\sexisting\sdirectories\swhen\sreadOnly\sspecified\sin\sthe\svolumeSource$'
[FAILED] failed to get logs from pod-subpath-test-dynamicpv-ptqg for test-container-subpath-dynamicpv-ptqg: Get "https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/provisioning-3562/pods/pod-subpath-test-dynamicpv-ptqg/log?container=test-container-subpath-dynamicpv-ptqg&previous=false": stream error: stream ID 663; INTERNAL_ERROR; received from peer In [It] at: test/e2e/framework/pod/output/output.go:247 @ 03/25/23 05:40:43.421from junit_01.xml
> Enter [BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/storage/framework/testsuite.go:51 @ 03/25/23 05:40:26.176 < Exit [BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/storage/framework/testsuite.go:51 @ 03/25/23 05:40:26.176 (0s) > Enter [BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath - set up framework | framework.go:191 @ 03/25/23 05:40:26.176 STEP: Creating a kubernetes client - test/e2e/framework/framework.go:211 @ 03/25/23 05:40:26.176 Mar 25 05:40:26.176: INFO: >>> kubeConfig: /root/.kube/config STEP: Building a namespace api object, basename provisioning - test/e2e/framework/framework.go:250 @ 03/25/23 05:40:26.178 STEP: Waiting for a default service account to be provisioned in namespace - test/e2e/framework/framework.go:259 @ 03/25/23 05:40:26.33 STEP: Waiting for kube-root-ca.crt to be provisioned in namespace - test/e2e/framework/framework.go:262 @ 03/25/23 05:40:26.426 < Exit [BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath - set up framework | framework.go:191 @ 03/25/23 05:40:26.522 (346ms) > Enter [BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/framework/metrics/init/init.go:33 @ 03/25/23 05:40:26.522 < Exit [BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/framework/metrics/init/init.go:33 @ 03/25/23 05:40:26.522 (0s) > Enter [It] should support existing directories when readOnly specified in the volumeSource - test/e2e/storage/testsuites/subpath.go:397 @ 03/25/23 05:40:26.522 Mar 25 05:40:26.522: INFO: Creating resource for dynamic PV Mar 25 05:40:26.522: INFO: Using claimSize:1Gi, test suite supported size:{ 1Mi}, driver(ebs.csi.aws.com) supported size:{ 1Mi} STEP: creating a StorageClass provisioning-3562-e2e-sc6w8rh - test/e2e/storage/framework/volume_resource.go:102 @ 03/25/23 05:40:26.522 STEP: creating a claim - test/e2e/storage/framework/volume_resource.go:294 @ 03/25/23 05:40:26.573 Mar 25 05:40:26.573: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil STEP: Creating pod pod-subpath-test-dynamicpv-ptqg - test/e2e/storage/testsuites/subpath.go:691 @ 03/25/23 05:40:26.673 STEP: Creating a pod to test subpath - test/e2e/framework/pod/output/output.go:243 @ 03/25/23 05:40:26.673 STEP: Saw pod success - test/e2e/framework/pod/wait.go:409 @ 03/25/23 05:40:43.191 Mar 25 05:40:43.243: INFO: Trying to get logs from node i-072608201d2c0f1ad pod pod-subpath-test-dynamicpv-ptqg container test-container-subpath-dynamicpv-ptqg: <nil> Mar 25 05:40:43.293: INFO: Failed to get logs from node "i-072608201d2c0f1ad" pod "pod-subpath-test-dynamicpv-ptqg" container "test-container-subpath-dynamicpv-ptqg". Get "https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/provisioning-3562/pods/pod-subpath-test-dynamicpv-ptqg/log?container=test-container-subpath-dynamicpv-ptqg&previous=false": stream error: stream ID 663; INTERNAL_ERROR; received from peer STEP: delete the pod - test/e2e/framework/pod/output/output.go:168 @ 03/25/23 05:40:43.293 Mar 25 05:40:43.420: INFO: Unexpected error: <*fmt.wrapError | 0xc000abc460>: failed to get logs from pod-subpath-test-dynamicpv-ptqg for test-container-subpath-dynamicpv-ptqg: Get "https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/provisioning-3562/pods/pod-subpath-test-dynamicpv-ptqg/log?container=test-container-subpath-dynamicpv-ptqg&previous=false": stream error: stream ID 663; INTERNAL_ERROR; received from peer { msg: "failed to get logs from pod-subpath-test-dynamicpv-ptqg for test-container-subpath-dynamicpv-ptqg: Get \"https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/provisioning-3562/pods/pod-subpath-test-dynamicpv-ptqg/log?container=test-container-subpath-dynamicpv-ptqg&previous=false\": stream error: stream ID 663; INTERNAL_ERROR; received from peer", err: <*url.Error | 0xc002d1b9e0>{ Op: "Get", URL: "https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/provisioning-3562/pods/pod-subpath-test-dynamicpv-ptqg/log?container=test-container-subpath-dynamicpv-ptqg&previous=false", Err: <http2.StreamError>{ StreamID: 663, Code: 2, Cause: <*errors.errorString | 0xc00022b650>{ s: "received from peer", }, }, }, } [FAILED] failed to get logs from pod-subpath-test-dynamicpv-ptqg for test-container-subpath-dynamicpv-ptqg: Get "https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/provisioning-3562/pods/pod-subpath-test-dynamicpv-ptqg/log?container=test-container-subpath-dynamicpv-ptqg&previous=false": stream error: stream ID 663; INTERNAL_ERROR; received from peer In [It] at: test/e2e/framework/pod/output/output.go:247 @ 03/25/23 05:40:43.421 < Exit [It] should support existing directories when readOnly specified in the volumeSource - test/e2e/storage/testsuites/subpath.go:397 @ 03/25/23 05:40:43.421 (16.899s) > Enter [AfterEach] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/framework/node/init/init.go:33 @ 03/25/23 05:40:43.421 Mar 25 05:40:43.421: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready < Exit [AfterEach] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/framework/node/init/init.go:33 @ 03/25/23 05:40:43.471 (51ms) > Enter [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/storage/testsuites/subpath.go:399 @ 03/25/23 05:40:43.472 STEP: Deleting pod - test/e2e/storage/testsuites/subpath.go:172 @ 03/25/23 05:40:43.472 Mar 25 05:40:43.472: INFO: Deleting pod "pod-subpath-test-dynamicpv-ptqg" in namespace "provisioning-3562" STEP: Deleting pvc - test/e2e/storage/framework/volume_resource.go:181 @ 03/25/23 05:40:43.524 Mar 25 05:40:43.622: INFO: Deleting PersistentVolumeClaim "ebs.csi.aws.comt2mk7" Mar 25 05:40:43.673: INFO: Waiting up to 5m0s for PersistentVolume pvc-653469a4-484b-44a7-bc58-926cdcdff4ee to get deleted Mar 25 05:40:43.721: INFO: PersistentVolume pvc-653469a4-484b-44a7-bc58-926cdcdff4ee found and phase=Released (48.230964ms) Mar 25 05:40:48.772: INFO: PersistentVolume pvc-653469a4-484b-44a7-bc58-926cdcdff4ee found and phase=Released (5.099086451s) Mar 25 05:40:53.825: INFO: PersistentVolume pvc-653469a4-484b-44a7-bc58-926cdcdff4ee found and phase=Released (10.151483573s) Mar 25 05:40:58.885: INFO: PersistentVolume pvc-653469a4-484b-44a7-bc58-926cdcdff4ee found and phase=Released (15.212309105s) Mar 25 05:41:03.937: INFO: PersistentVolume pvc-653469a4-484b-44a7-bc58-926cdcdff4ee was removed STEP: Deleting sc - test/e2e/storage/framework/volume_resource.go:228 @ 03/25/23 05:41:03.937 < Exit [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/storage/testsuites/subpath.go:399 @ 03/25/23 05:41:03.989 (20.517s) > Enter [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/framework/metrics/init/init.go:35 @ 03/25/23 05:41:03.989 < Exit [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/framework/metrics/init/init.go:35 @ 03/25/23 05:41:03.989 (0s) > Enter [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] subPath - dump namespaces | framework.go:209 @ 03/25/23 05:41:03.989 STEP: dump namespace information after failure - test/e2e/framework/framework.go:288 @ 03/25/23 05:41:03.989 STEP: Collecting events from namespace "provisioning-3562". - test/e2e/framework/debug/dump.go:42 @ 03/25/23 05:41:03.989 STEP: Found 15 events. - test/e2e/framework/debug/dump.go:46 @ 03/25/23 05:41:04.039 Mar 25 05:41:04.039: INFO: At 2023-03-25 05:40:26 +0000 UTC - event for ebs.csi.aws.comt2mk7: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding Mar 25 05:41:04.039: INFO: At 2023-03-25 05:40:26 +0000 UTC - event for ebs.csi.aws.comt2mk7: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator Mar 25 05:41:04.039: INFO: At 2023-03-25 05:40:26 +0000 UTC - event for ebs.csi.aws.comt2mk7: {ebs.csi.aws.com_i-0168f82458996cc53_0b589c46-3396-4499-94ba-eb51a48f7a58 } Provisioning: External provisioner is provisioning volume for claim "provisioning-3562/ebs.csi.aws.comt2mk7" Mar 25 05:41:04.039: INFO: At 2023-03-25 05:40:30 +0000 UTC - event for ebs.csi.aws.comt2mk7: {ebs.csi.aws.com_i-0168f82458996cc53_0b589c46-3396-4499-94ba-eb51a48f7a58 } ProvisioningSucceeded: Successfully provisioned volume pvc-653469a4-484b-44a7-bc58-926cdcdff4ee Mar 25 05:41:04.039: INFO: At 2023-03-25 05:40:30 +0000 UTC - event for pod-subpath-test-dynamicpv-ptqg: {default-scheduler } Scheduled: Successfully assigned provisioning-3562/pod-subpath-test-dynamicpv-ptqg to i-072608201d2c0f1ad Mar 25 05:41:04.039: INFO: At 2023-03-25 05:40:33 +0000 UTC - event for pod-subpath-test-dynamicpv-ptqg: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-653469a4-484b-44a7-bc58-926cdcdff4ee" Mar 25 05:41:04.039: INFO: At 2023-03-25 05:40:37 +0000 UTC - event for pod-subpath-test-dynamicpv-ptqg: {kubelet i-072608201d2c0f1ad} Pulled: Container image "registry.k8s.io/e2e-test-images/busybox:1.29-4" already present on machine Mar 25 05:41:04.039: INFO: At 2023-03-25 05:40:37 +0000 UTC - event for pod-subpath-test-dynamicpv-ptqg: {kubelet i-072608201d2c0f1ad} Created: Created container init-volume-dynamicpv-ptqg Mar 25 05:41:04.039: INFO: At 2023-03-25 05:40:37 +0000 UTC - event for pod-subpath-test-dynamicpv-ptqg: {kubelet i-072608201d2c0f1ad} Started: Started container init-volume-dynamicpv-ptqg Mar 25 05:41:04.039: INFO: At 2023-03-25 05:40:37 +0000 UTC - event for pod-subpath-test-dynamicpv-ptqg: {kubelet i-072608201d2c0f1ad} Pulled: Container image "registry.k8s.io/e2e-test-images/agnhost:2.43" already present on machine Mar 25 05:41:04.039: INFO: At 2023-03-25 05:40:38 +0000 UTC - event for pod-subpath-test-dynamicpv-ptqg: {kubelet i-072608201d2c0f1ad} Created: Created container test-init-subpath-dynamicpv-ptqg Mar 25 05:41:04.039: INFO: At 2023-03-25 05:40:38 +0000 UTC - event for pod-subpath-test-dynamicpv-ptqg: {kubelet i-072608201d2c0f1ad} Started: Started container test-init-subpath-dynamicpv-ptqg Mar 25 05:41:04.039: INFO: At 2023-03-25 05:40:39 +0000 UTC - event for pod-subpath-test-dynamicpv-ptqg: {kubelet i-072608201d2c0f1ad} Pulled: Container image "registry.k8s.io/e2e-test-images/agnhost:2.43" already present on machine Mar 25 05:41:04.039: INFO: At 2023-03-25 05:40:39 +0000 UTC - event for pod-subpath-test-dynamicpv-ptqg: {kubelet i-072608201d2c0f1ad} Created: Created container test-container-subpath-dynamicpv-ptqg Mar 25 05:41:04.039: INFO: At 2023-03-25 05:40:39 +0000 UTC - event for pod-subpath-test-dynamicpv-ptqg: {kubelet i-072608201d2c0f1ad} Started: Started container test-container-subpath-dynamicpv-ptqg Mar 25 05:41:04.088: INFO: POD NODE PHASE GRACE CONDITIONS Mar 25 05:41:04.088: INFO: Mar 25 05:41:04.137: INFO: Logging node info for node i-0168f82458996cc53 Mar 25 05:41:04.186: INFO: Node Info: &Node{ObjectMeta:{i-0168f82458996cc53 f4eb302c-c477-45ac-bcac-ffed65f6580f 5776 0 2023-03-25 05:18:57 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-1 failure-domain.beta.kubernetes.io/zone:us-west-1b kops.k8s.io/kops-controller-pki: kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0168f82458996cc53 kubernetes.io/os:linux node-role.kubernetes.io/control-plane: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:us-west-1b topology.kubernetes.io/region:us-west-1 topology.kubernetes.io/zone:us-west-1b] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0168f82458996cc53"} node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:172.20.48.81/19 projectcalico.org/IPv4IPIPTunnelAddr:100.100.215.192 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-03-25 05:18:57 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {protokube Update v1 2023-03-25 05:18:58 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {calico-node Update v1 2023-03-25 05:20:22 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4IPIPTunnelAddr":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {aws-cloud-controller-manager Update v1 2023-03-25 05:20:26 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-25 05:20:26 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2023-03-25 05:20:31 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.5.0/24\"":{}},"f:taints":{}}} } {kubelet Update v1 2023-03-25 05:36:59 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.5.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-1b/i-0168f82458996cc53,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/control-plane,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.5.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3892310016 0} {<nil>} 3801084Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3787452416 0} {<nil>} 3698684Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-03-25 05:20:22 +0000 UTC,LastTransitionTime:2023-03-25 05:20:22 +0000 UTC,Reason:CalicoIsUp,Message:Calico is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-25 05:36:59 +0000 UTC,LastTransitionTime:2023-03-25 05:18:50 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-25 05:36:59 +0000 UTC,LastTransitionTime:2023-03-25 05:18:50 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-25 05:36:59 +0000 UTC,LastTransitionTime:2023-03-25 05:18:50 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-25 05:36:59 +0000 UTC,LastTransitionTime:2023-03-25 05:20:29 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.48.81,},NodeAddress{Type:ExternalIP,Address:3.101.135.77,},NodeAddress{Type:InternalDNS,Address:i-0168f82458996cc53.us-west-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-0168f82458996cc53.us-west-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-3-101-135-77.us-west-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2d1ccc7ebf313420bd1f530a94fbfd,SystemUUID:ec2d1ccc-7ebf-3134-20bd-1f530a94fbfd,BootID:15cee321-56e7-4290-ab48-47b60b0bc657,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.28.0-alpha.0,KubeProxyVersion:v1.28.0-alpha.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcd@sha256:dd75ec974b0a2a6f6bb47001ba09207976e625db898d1b16735528c009cb171c registry.k8s.io/etcd:3.5.6-0],SizeBytes:102542580,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:6f72b851544986cb0921b53ea655ec04c36131248f16d4ad110cb3ca0c369dc1 registry.k8s.io/etcd:3.5.4-0],SizeBytes:102157811,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:13f53ed1d91e2e11aac476ee9a0269fdda6cc4874eba903efd40daf50c55eee5 registry.k8s.io/etcd:3.5.3-0],SizeBytes:102143581,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:51eae8381dcb1078289fa7b4f3df2630cdc18d09fb56f8e56b41c40e191d6c83 registry.k8s.io/etcd:3.5.7-0],SizeBytes:101639218,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:4afb99b4690b418ffc2ceb67e1a17376457e441c1f09ab55447f0aaf992fa646 registry.k8s.io/etcd:3.4.3-0],SizeBytes:100947667,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:9ce33ba33d8e738a5b85ed50b5080ac746deceed4a7496c550927a7a19ca3b6d registry.k8s.io/etcd:3.5.0-0],SizeBytes:99868722,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:64b9ea357325d5db9f8a723dcf503b5a449177b17ac87d69481e126bb724c263 registry.k8s.io/etcd:3.5.1-0],SizeBytes:98888614,},ContainerImage{Names:[docker.io/calico/cni@sha256:e282ea2914c806b5de2976330a17cfb5e6dcef47147bceb1432ca5c75fd46f50 docker.io/calico/cni:v3.24.5],SizeBytes:87458443,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:4ad90a11b55313b182afc186b9876c8e891531b8db4c9bf1541953021618d0e2 registry.k8s.io/etcd:3.4.13-0],SizeBytes:86742272,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:0d2ab6b15b4fe0b903b9e81192d942fb39c29be61fe341cd16db5d0165b0435c registry.k8s.io/etcd:3.3.17-0],SizeBytes:85524408,},ContainerImage{Names:[docker.io/calico/node@sha256:5972ad2bcbdc668564d3e26960c9c513b2d7b05581c704747cf7c62ef3a405a6 docker.io/calico/node:v3.24.5],SizeBytes:81584866,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:17da501f5d2a675be46040422a27b7cc21b8a43895ac998b171db1c346f361f7 registry.k8s.io/etcd:3.3.10-0],SizeBytes:76162358,},ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager-slim@sha256:8727adc059334e0e3329dbccce116a734da4a090da5e86c8cddccaf333caa666 registry.k8s.io/etcdadm/etcd-manager-slim:v3.0.20230201],SizeBytes:63926165,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:905d7ca17fd02bc24c0eba9a062753aba15db3e31422390bc3238eb762339b20 registry.k8s.io/etcd:3.2.24-1],SizeBytes:63423287,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.27.0-alpha.1],SizeBytes:45219545,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.27.0-alpha.1],SizeBytes:43628585,},ContainerImage{Names:[registry.k8s.io/kube-apiserver@sha256:49934a29c559e79ed19fc38b4697b18d8fe174b17bf56192588575f980c93956 registry.k8s.io/kube-apiserver:v1.28.0-alpha.0],SizeBytes:33212506,},ContainerImage{Names:[docker.io/calico/kube-controllers@sha256:2b6acd7f677f76ffe12ecf3ea7df92eb9b1bdb07336d1ac2a54c7631fb753f7e docker.io/calico/kube-controllers:v3.24.5],SizeBytes:31137169,},ContainerImage{Names:[registry.k8s.io/kube-controller-manager@sha256:974748384634982c73527fdb3c9ec0ec476bc0d52b3dd1d611a380b7f99af1f3 registry.k8s.io/kube-controller-manager:v1.28.0-alpha.0],SizeBytes:30820280,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/kube-proxy@sha256:8a03b3147cd7be8e8235ae1cd70cf69aca9ce03ba1cfaffea320540bb5afa496 registry.k8s.io/kube-proxy:v1.28.0-alpha.0],SizeBytes:23896850,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:23345856,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:22381475,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:22085298,},ContainerImage{Names:[registry.k8s.io/provider-aws/cloud-controller-manager@sha256:fdeb61e3e42ecd9cca868d550ebdb88dd6341d9e91fcfa9a37e227dab2ad22cb registry.k8s.io/provider-aws/cloud-controller-manager:v1.26.0],SizeBytes:20154862,},ContainerImage{Names:[registry.k8s.io/kube-scheduler@sha256:cbf4007abb076188f16b975bc2b74452603cc7b34c60a29d92f14c38ad02dc11 registry.k8s.io/kube-scheduler:v1.28.0-alpha.0],SizeBytes:18078674,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.27.0-alpha.1],SizeBytes:5074298,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Mar 25 05:41:04.186: INFO: Logging kubelet events for node i-0168f82458996cc53 Mar 25 05:41:04.241: INFO: Logging pods the kubelet thinks is on node i-0168f82458996cc53 Mar 25 05:41:04.297: INFO: kube-controller-manager-i-0168f82458996cc53 started at 2023-03-25 05:17:36 +0000 UTC (0+1 container statuses recorded) Mar 25 05:41:04.297: INFO: Container kube-controller-manager ready: true, restart count 4 Mar 25 05:41:04.297: INFO: etcd-manager-events-i-0168f82458996cc53 started at 2023-03-25 05:17:36 +0000 UTC (11+1 container statuses recorded) Mar 25 05:41:04.297: INFO: Init container init-etcd-3-2-24 ready: true, restart count 0 Mar 25 05:41:04.297: INFO: Init container init-etcd-3-3-10 ready: true, restart count 0 Mar 25 05:41:04.297: INFO: Init container init-etcd-3-3-17 ready: true, restart count 0 Mar 25 05:41:04.297: INFO: Init container init-etcd-3-4-13 ready: true, restart count 0 Mar 25 05:41:04.297: INFO: Init container init-etcd-3-4-3 ready: true, restart count 0 Mar 25 05:41:04.297: INFO: Init container init-etcd-3-5-0 ready: true, restart count 0 Mar 25 05:41:04.297: INFO: Init container init-etcd-3-5-1 ready: true, restart count 0 Mar 25 05:41:04.297: INFO: Init container init-etcd-3-5-3 ready: true, restart count 0 Mar 25 05:41:04.297: INFO: Init container init-etcd-3-5-4 ready: true, restart count 0 Mar 25 05:41:04.297: INFO: Init container init-etcd-3-5-6 ready: true, restart count 0 Mar 25 05:41:04.297: INFO: Init container init-etcd-3-5-7 ready: true, restart count 0 Mar 25 05:41:04.297: INFO: Container etcd-manager ready: true, restart count 0 Mar 25 05:41:04.297: INFO: ebs-csi-node-zjtj6 started at 2023-03-25 05:20:06 +0000 UTC (0+3 container statuses recorded) Mar 25 05:41:04.297: INFO: Container ebs-plugin ready: true, restart count 0 Mar 25 05:41:04.297: INFO: Container liveness-probe ready: true, restart count 0 Mar 25 05:41:04.297: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 25 05:41:04.297: INFO: kube-apiserver-i-0168f82458996cc53 started at 2023-03-25 05:17:36 +0000 UTC (0+2 container statuses recorded) Mar 25 05:41:04.297: INFO: Container healthcheck ready: true, restart count 0 Mar 25 05:41:04.297: INFO: Container kube-apiserver ready: true, restart count 2 Mar 25 05:41:04.297: INFO: kube-proxy-i-0168f82458996cc53 started at 2023-03-25 05:17:36 +0000 UTC (0+1 container statuses recorded) Mar 25 05:41:04.297: INFO: Container kube-proxy ready: true, restart count 0 Mar 25 05:41:04.297: INFO: kube-scheduler-i-0168f82458996cc53 started at 2023-03-25 05:17:36 +0000 UTC (0+1 container statuses recorded) Mar 25 05:41:04.297: INFO: Container kube-scheduler ready: true, restart count 0 Mar 25 05:41:04.297: INFO: etcd-manager-main-i-0168f82458996cc53 started at 2023-03-25 05:17:36 +0000 UTC (11+1 container statuses recorded) Mar 25 05:41:04.297: INFO: Init container init-etcd-3-2-24 ready: true, restart count 0 Mar 25 05:41:04.297: INFO: Init container init-etcd-3-3-10 ready: true, restart count 0 Mar 25 05:41:04.297: INFO: Init container init-etcd-3-3-17 ready: true, restart count 0 Mar 25 05:41:04.297: INFO: Init container init-etcd-3-4-13 ready: true, restart count 0 Mar 25 05:41:04.297: INFO: Init container init-etcd-3-4-3 ready: true, restart count 0 Mar 25 05:41:04.297: INFO: Init container init-etcd-3-5-0 ready: true, restart count 0 Mar 25 05:41:04.297: INFO: Init container init-etcd-3-5-1 ready: true, restart count 0 Mar 25 05:41:04.297: INFO: Init container init-etcd-3-5-3 ready: true, restart count 0 Mar 25 05:41:04.297: INFO: Init container init-etcd-3-5-4 ready: true, restart count 0 Mar 25 05:41:04.297: INFO: Init container init-etcd-3-5-6 ready: true, restart count 0 Mar 25 05:41:04.297: INFO: Init container init-etcd-3-5-7 ready: true, restart count 0 Mar 25 05:41:04.297: INFO: Container etcd-manager ready: true, restart count 0 Mar 25 05:41:04.297: INFO: dns-controller-69987775c6-hbn69 started at 2023-03-25 05:19:47 +0000 UTC (0+1 container statuses recorded) Mar 25 05:41:04.297: INFO: Container dns-controller ready: true, restart count 0 Mar 25 05:41:04.297: INFO: aws-cloud-controller-manager-26vbv started at 2023-03-25 05:20:06 +0000 UTC (0+1 container statuses recorded) Mar 25 05:41:04.297: INFO: Container aws-cloud-controller-manager ready: true, restart count 0 Mar 25 05:41:04.297: INFO: calico-node-rdjq2 started at 2023-03-25 05:20:06 +0000 UTC (3+1 container statuses recorded) Mar 25 05:41:04.297: INFO: Init container upgrade-ipam ready: true, restart count 0 Mar 25 05:41:04.297: INFO: Init container install-cni ready: true, restart count 0 Mar 25 05:41:04.297: INFO: Init container mount-bpffs ready: true, restart count 0 Mar 25 05:41:04.297: INFO: Container calico-node ready: true, restart count 0 Mar 25 05:41:04.297: INFO: kops-controller-cgw95 started at 2023-03-25 05:20:06 +0000 UTC (0+1 container statuses recorded) Mar 25 05:41:04.297: INFO: Container kops-controller ready: true, restart count 0 Mar 25 05:41:04.297: INFO: calico-kube-controllers-55f488656-rr8rz started at 2023-03-25 05:20:29 +0000 UTC (0+1 container statuses recorded) Mar 25 05:41:04.297: INFO: Container calico-kube-controllers ready: true, restart count 0 Mar 25 05:41:04.297: INFO: ebs-csi-controller-79f984545f-9q5ks started at 2023-03-25 05:21:26 +0000 UTC (0+5 container statuses recorded) Mar 25 05:41:04.297: INFO: Container csi-attacher ready: true, restart count 0 Mar 25 05:41:04.297: INFO: Container csi-provisioner ready: true, restart count 0 Mar 25 05:41:04.297: INFO: Container csi-resizer ready: true, restart count 0 Mar 25 05:41:04.297: INFO: Container ebs-plugin ready: true, restart count 0 Mar 25 05:41:04.297: INFO: Container liveness-probe ready: true, restart count 0 Mar 25 05:41:04.570: INFO: Latency metrics for node i-0168f82458996cc53 Mar 25 05:41:04.570: INFO: Logging node info for node i-03d301e6f030748f1 Mar 25 05:41:04.632: INFO: Node Info: &Node{ObjectMeta:{i-03d301e6f030748f1 897f52bd-65e2-4107-843a-37e36aa74032 17952 0 2023-03-25 05:31:35 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-1 failure-domain.beta.kubernetes.io/zone:us-west-1b kubernetes.io/arch:amd64 kubernetes.io/hostname:i-03d301e6f030748f1 kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:us-west-1b topology.hostpath.csi/node:i-03d301e6f030748f1 topology.kubernetes.io/region:us-west-1 topology.kubernetes.io/zone:us-west-1b] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-volumemode-8103":"i-03d301e6f030748f1","ebs.csi.aws.com":"i-03d301e6f030748f1"} node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:172.20.50.199/19 projectcalico.org/IPv4IPIPTunnelAddr:100.110.167.64 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-03-25 05:31:35 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-25 05:31:35 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-03-25 05:31:35 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2023-03-25 05:31:35 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2023-03-25 05:31:48 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.8.0/24\"":{}}}} } {calico-node Update v1 2023-03-25 05:31:56 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4IPIPTunnelAddr":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2023-03-25 05:40:53 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status} {kube-controller-manager Update v1 2023-03-25 05:40:57 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.8.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-1b/i-03d301e6f030748f1,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.8.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-03-25 05:31:56 +0000 UTC,LastTransitionTime:2023-03-25 05:31:56 +0000 UTC,Reason:CalicoIsUp,Message:Calico is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-25 05:40:46 +0000 UTC,LastTransitionTime:2023-03-25 05:31:35 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-25 05:40:46 +0000 UTC,LastTransitionTime:2023-03-25 05:31:35 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-25 05:40:46 +0000 UTC,LastTransitionTime:2023-03-25 05:31:35 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-25 05:40:46 +0000 UTC,LastTransitionTime:2023-03-25 05:31:48 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.50.199,},NodeAddress{Type:ExternalIP,Address:52.53.245.63,},NodeAddress{Type:InternalDNS,Address:i-03d301e6f030748f1.us-west-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-03d301e6f030748f1.us-west-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-52-53-245-63.us-west-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec27152f7af9b9bfd1355ee19c256086,SystemUUID:ec27152f-7af9-b9bf-d135-5ee19c256086,BootID:1c303852-f2ce-45f9-ab14-6e8de97a0a3e,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.28.0-alpha.0,KubeProxyVersion:v1.28.0-alpha.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[docker.io/calico/cni@sha256:e282ea2914c806b5de2976330a17cfb5e6dcef47147bceb1432ca5c75fd46f50 docker.io/calico/cni:v3.24.5],SizeBytes:87458443,},ContainerImage{Names:[docker.io/calico/node@sha256:5972ad2bcbdc668564d3e26960c9c513b2d7b05581c704747cf7c62ef3a405a6 docker.io/calico/node:v3.24.5],SizeBytes:81584866,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nautilus@sha256:80ba6c8c44f9623f06e868a1aa66026c8ec438ad814f9ec95e9333b415fe3550 registry.k8s.io/e2e-test-images/nautilus:1.7],SizeBytes:49641698,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:e468dddcd275163a042ab297b2d8c2aca50d5e148d2d22f3b6ba119e2f31fa79 registry.k8s.io/sig-storage/csi-provisioner:v3.4.0],SizeBytes:27427836,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/kube-proxy@sha256:8a03b3147cd7be8e8235ae1cd70cf69aca9ce03ba1cfaffea320540bb5afa496 registry.k8s.io/kube-proxy:v1.28.0-alpha.0],SizeBytes:23896850,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:23881995,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18758628,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18233005,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonroot@sha256:ee9f50b3c64b174d296d91ca9f69a914ac30e59095dfb462b2b518ad28a63655 registry.k8s.io/e2e-test-images/nonroot:1.4],SizeBytes:17747885,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8688564,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:7000082,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6978614,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac registry.k8s.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-volumemode-8103^9c6b2c15-cacf-11ed-8e2c-a6dc5d7f0a3d,DevicePath:,},},Config:nil,},} Mar 25 05:41:04.632: INFO: Logging kubelet events for node i-03d301e6f030748f1 Mar 25 05:41:04.709: INFO: Logging pods the kubelet thinks is on node i-03d301e6f030748f1 Mar 25 05:41:04.777: INFO: kube-proxy-i-03d301e6f030748f1 started at 2023-03-25 05:31:35 +0000 UTC (0+1 container statuses recorded) Mar 25 05:41:04.777: INFO: Container kube-proxy ready: true, restart count 0 Mar 25 05:41:04.777: INFO: ebs-csi-node-lljwl started at 2023-03-25 05:31:35 +0000 UTC (0+3 container statuses recorded) Mar 25 05:41:04.777: INFO: Container ebs-plugin ready: true, restart count 0 Mar 25 05:41:04.777: INFO: Container liveness-probe ready: true, restart count 0 Mar 25 05:41:04.777: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 25 05:41:04.777: INFO: test-rolling-update-with-lb-55d4c6cb8c-xdcdd started at 2023-03-25 05:40:55 +0000 UTC (0+1 container statuses recorded) Mar 25 05:41:04.777: INFO: Container agnhost ready: true, restart count 0 Mar 25 05:41:04.777: INFO: hostexec-i-03d301e6f030748f1-qn44v started at 2023-03-25 05:40:50 +0000 UTC (0+1 container statuses recorded) Mar 25 05:41:04.777: INFO: Container agnhost-container ready: true, restart count 0 Mar 25 05:41:04.777: INFO: csi-hostpathplugin-0 started at 2023-03-25 05:40:52 +0000 UTC (0+7 container statuses recorded) Mar 25 05:41:04.777: INFO: Container csi-attacher ready: true, restart count 0 Mar 25 05:41:04.777: INFO: Container csi-provisioner ready: true, restart count 0 Mar 25 05:41:04.777: INFO: Container csi-resizer ready: true, restart count 0 Mar 25 05:41:04.777: INFO: Container csi-snapshotter ready: true, restart count 0 Mar 25 05:41:04.777: INFO: Container hostpath ready: true, restart count 0 Mar 25 05:41:04.777: INFO: Container liveness-probe ready: true, restart count 0 Mar 25 05:41:04.777: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 25 05:41:04.777: INFO: pod-ephm-test-configmap-f48v started at 2023-03-25 05:40:32 +0000 UTC (0+1 container statuses recorded) Mar 25 05:41:04.777: INFO: Container test-container-subpath-configmap-f48v ready: false, restart count 0 Mar 25 05:41:04.777: INFO: pod-d85ac3d5-f13b-481e-9377-5d1c22b0f1f0 started at 2023-03-25 05:40:56 +0000 UTC (0+1 container statuses recorded) Mar 25 05:41:04.777: INFO: Container write-pod ready: true, restart count 0 Mar 25 05:41:04.777: INFO: calico-node-h96vq started at 2023-03-25 05:31:35 +0000 UTC (3+1 container statuses recorded) Mar 25 05:41:04.777: INFO: Init container upgrade-ipam ready: true, restart count 0 Mar 25 05:41:04.777: INFO: Init container install-cni ready: true, restart count 0 Mar 25 05:41:04.777: INFO: Init container mount-bpffs ready: true, restart count 0 Mar 25 05:41:04.777: INFO: Container calico-node ready: true, restart count 0 Mar 25 05:41:04.777: INFO: terminate-cmd-rpad8823b7d-3cde-4c69-8b13-3a7369011043 started at 2023-03-25 05:40:46 +0000 UTC (0+1 container statuses recorded) Mar 25 05:41:04.777: INFO: Container terminate-cmd-rpa ready: false, restart count 1 Mar 25 05:41:04.777: INFO: hostexec-i-03d301e6f030748f1-swmwz started at 2023-03-25 05:40:58 +0000 UTC (0+1 container statuses recorded) Mar 25 05:41:04.777: INFO: Container agnhost-container ready: true, restart count 0 Mar 25 05:41:04.777: INFO: adopt-release-bj998 started at 2023-03-25 05:40:41 +0000 UTC (0+1 container statuses recorded) Mar 25 05:41:04.777: INFO: Container c ready: true, restart count 0 Mar 25 05:41:04.777: INFO: test-rolling-update-with-lb-745c47894-rmk6p started at 2023-03-25 05:41:02 +0000 UTC (0+1 container statuses recorded) Mar 25 05:41:04.777: INFO: Container agnhost ready: false, restart count 0 Mar 25 05:41:04.777: INFO: netserver-0 started at 2023-03-25 05:40:40 +0000 UTC (0+1 container statuses recorded) Mar 25 05:41:04.777: INFO: Container webserver ready: true, restart count 0 Mar 25 05:41:05.161: INFO: Latency metrics for node i-03d301e6f030748f1 Mar 25 05:41:05.161: INFO: Logging node info for node i-049fdb2c1cde894df Mar 25 05:41:05.213: INFO: Node Info: &Node{ObjectMeta:{i-049fdb2c1cde894df b43a51e6-dda7-4047-b368-72d515695b4e 18314 0 2023-03-25 05:23:34 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-1 failure-domain.beta.kubernetes.io/zone:us-west-1b io.kubernetes.storage.mock/node:some-mock-node kubernetes.io/arch:amd64 kubernetes.io/hostname:i-049fdb2c1cde894df kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:us-west-1b topology.hostpath.csi/node:i-049fdb2c1cde894df topology.kubernetes.io/region:us-west-1 topology.kubernetes.io/zone:us-west-1b] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-9675":"i-049fdb2c1cde894df","csi-hostpath-provisioning-1698":"i-049fdb2c1cde894df","csi-hostpath-read-write-once-pod-7024":"i-049fdb2c1cde894df","ebs.csi.aws.com":"i-049fdb2c1cde894df"} node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:172.20.43.61/19 projectcalico.org/IPv4IPIPTunnelAddr:100.97.18.64 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-03-25 05:23:34 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-25 05:23:34 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-03-25 05:23:34 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2023-03-25 05:23:34 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2023-03-25 05:23:45 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.6.0/24\"":{}}}} } {calico-node Update v1 2023-03-25 05:23:52 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4IPIPTunnelAddr":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2023-03-25 05:40:55 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-03-25 05:41:03 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:io.kubernetes.storage.mock/node":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.6.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-1b/i-049fdb2c1cde894df,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.6.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-03-25 05:23:52 +0000 UTC,LastTransitionTime:2023-03-25 05:23:52 +0000 UTC,Reason:CalicoIsUp,Message:Calico is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-25 05:40:55 +0000 UTC,LastTransitionTime:2023-03-25 05:23:34 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-25 05:40:55 +0000 UTC,LastTransitionTime:2023-03-25 05:23:34 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-25 05:40:55 +0000 UTC,LastTransitionTime:2023-03-25 05:23:34 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-25 05:40:55 +0000 UTC,LastTransitionTime:2023-03-25 05:23:45 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.43.61,},NodeAddress{Type:ExternalIP,Address:18.144.75.67,},NodeAddress{Type:InternalDNS,Address:i-049fdb2c1cde894df.us-west-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-049fdb2c1cde894df.us-west-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-18-144-75-67.us-west-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec252e592d19e1e6c583e87505431016,SystemUUID:ec252e59-2d19-e1e6-c583-e87505431016,BootID:45436b5d-1eb0-4b6f-a1d1-a1492b31b5d7,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.28.0-alpha.0,KubeProxyVersion:v1.28.0-alpha.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[docker.io/calico/cni@sha256:e282ea2914c806b5de2976330a17cfb5e6dcef47147bceb1432ca5c75fd46f50 docker.io/calico/cni:v3.24.5],SizeBytes:87458443,},ContainerImage{Names:[docker.io/calico/node@sha256:5972ad2bcbdc668564d3e26960c9c513b2d7b05581c704747cf7c62ef3a405a6 docker.io/calico/node:v3.24.5],SizeBytes:81584866,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:e468dddcd275163a042ab297b2d8c2aca50d5e148d2d22f3b6ba119e2f31fa79 registry.k8s.io/sig-storage/csi-provisioner:v3.4.0],SizeBytes:27427836,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/kube-proxy@sha256:8a03b3147cd7be8e8235ae1cd70cf69aca9ce03ba1cfaffea320540bb5afa496 registry.k8s.io/kube-proxy:v1.28.0-alpha.0],SizeBytes:23896850,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:23881995,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:68d396900aeaa072c1f27289485fdac29834045a6f3ffe369bf389d830ef572d registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.6],SizeBytes:20293261,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18758628,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18233005,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:15273057,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8688564,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:7000082,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6978614,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[kubernetes.io/csi/csi-hostpath-ephemeral-9675^826685bb-cacf-11ed-a589-2ee150a4b4f1 kubernetes.io/csi/csi-hostpath-ephemeral-9675^874aa77d-cacf-11ed-a589-2ee150a4b4f1],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-9675^874aa77d-cacf-11ed-a589-2ee150a4b4f1,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-9675^826685bb-cacf-11ed-a589-2ee150a4b4f1,DevicePath:,},},Config:nil,},} Mar 25 05:41:05.213: INFO: Logging kubelet events for node i-049fdb2c1cde894df Mar 25 05:41:05.274: INFO: Logging pods the kubelet thinks is on node i-049fdb2c1cde894df Mar 25 05:41:05.331: INFO: csi-hostpathplugin-0 started at 2023-03-25 05:40:07 +0000 UTC (0+7 container statuses recorded) Mar 25 05:41:05.331: INFO: Container csi-attacher ready: true, restart count 0 Mar 25 05:41:05.331: INFO: Container csi-provisioner ready: true, restart count 0 Mar 25 05:41:05.331: INFO: Container csi-resizer ready: true, restart count 0 Mar 25 05:41:05.331: INFO: Container csi-snapshotter ready: true, restart count 0 Mar 25 05:41:05.331: INFO: Container hostpath ready: true, restart count 0 Mar 25 05:41:05.331: INFO: Container liveness-probe ready: true, restart count 0 Mar 25 05:41:05.331: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 25 05:41:05.331: INFO: inline-volume-tester-97vjb started at 2023-03-25 05:40:11 +0000 UTC (0+1 container statuses recorded) Mar 25 05:41:05.331: INFO: Container csi-volume-tester ready: true, restart count 0 Mar 25 05:41:05.331: INFO: coredns-autoscaler-7cb5c5b969-h485x started at 2023-03-25 05:24:36 +0000 UTC (0+1 container statuses recorded) Mar 25 05:41:05.331: INFO: Container autoscaler ready: true, restart count 0 Mar 25 05:41:05.331: INFO: netserver-1 started at 2023-03-25 05:40:40 +0000 UTC (0+1 container statuses recorded) Mar 25 05:41:05.331: INFO: Container webserver ready: true, restart count 0 Mar 25 05:41:05.331: INFO: csi-mockplugin-0 started at 2023-03-25 05:40:16 +0000 UTC (0+4 container statuses recorded) Mar 25 05:41:05.331: INFO: Container busybox ready: true, restart count 0 Mar 25 05:41:05.331: INFO: Container csi-provisioner ready: true, restart count 0 Mar 25 05:41:05.331: INFO: Container driver-registrar ready: true, restart count 0 Mar 25 05:41:05.331: INFO: Container mock ready: true, restart count 0 Mar 25 05:41:05.331: INFO: csi-hostpathplugin-0 started at 2023-03-25 05:41:02 +0000 UTC (0+7 container statuses recorded) Mar 25 05:41:05.331: INFO: Container csi-attacher ready: true, restart count 0 Mar 25 05:41:05.331: INFO: Container csi-provisioner ready: true, restart count 0 Mar 25 05:41:05.331: INFO: Container csi-resizer ready: true, restart count 0 Mar 25 05:41:05.331: INFO: Container csi-snapshotter ready: true, restart count 0 Mar 25 05:41:05.331: INFO: Container hostpath ready: true, restart count 0 Mar 25 05:41:05.331: INFO: Container liveness-probe ready: true, restart count 0 Mar 25 05:41:05.331: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 25 05:41:05.331: INFO: csi-hostpathplugin-0 started at 2023-03-25 05:40:41 +0000 UTC (0+7 container statuses recorded) Mar 25 05:41:05.331: INFO: Container csi-attacher ready: true, restart count 0 Mar 25 05:41:05.331: INFO: Container csi-provisioner ready: true, restart count 0 Mar 25 05:41:05.331: INFO: Container csi-resizer ready: true, restart count 0 Mar 25 05:41:05.331: INFO: Container csi-snapshotter ready: true, restart count 0 Mar 25 05:41:05.331: INFO: Container hostpath ready: true, restart count 0 Mar 25 05:41:05.331: INFO: Container liveness-probe ready: true, restart count 0 Mar 25 05:41:05.331: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 25 05:41:05.331: INFO: kube-proxy-i-049fdb2c1cde894df started at 2023-03-25 05:23:35 +0000 UTC (0+1 container statuses recorded) Mar 25 05:41:05.331: INFO: Container kube-proxy ready: true, restart count 0 Mar 25 05:41:05.331: INFO: ebs-csi-node-m9zxz started at 2023-03-25 05:23:35 +0000 UTC (0+3 container statuses recorded) Mar 25 05:41:05.331: INFO: Container ebs-plugin ready: true, restart count 0 Mar 25 05:41:05.331: INFO: Container liveness-probe ready: true, restart count 0 Mar 25 05:41:05.331: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 25 05:41:05.331: INFO: calico-node-wt5p7 started at 2023-03-25 05:23:35 +0000 UTC (3+1 container statuses recorded) Mar 25 05:41:05.331: INFO: Init container upgrade-ipam ready: true, restart count 0 Mar 25 05:41:05.331: INFO: Init container install-cni ready: true, restart count 0 Mar 25 05:41:05.331: INFO: Init container mount-bpffs ready: true, restart count 0 Mar 25 05:41:05.331: INFO: Container calico-node ready: true, restart count 0 Mar 25 05:41:05.331: INFO: coredns-559769c974-pgqhl started at 2023-03-25 05:24:36 +0000 UTC (0+1 container statuses recorded) Mar 25 05:41:05.331: INFO: Container coredns ready: true, restart count 0 Mar 25 05:41:05.636: INFO: Latency metrics for node i-049fdb2c1cde894df Mar 25 05:41:05.636: INFO: Logging node info for node i-062679bb8c0ba4bce Mar 25 05:41:05.686: INFO: Node Info: &Node{ObjectMeta:{i-062679bb8c0ba4bce ee2da569-bb88-4297-9e6f-d2b15ba7ce86 17600 0 2023-03-25 05:27:31 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-1 failure-domain.beta.kubernetes.io/zone:us-west-1b kubernetes.io/arch:amd64 kubernetes.io/hostname:i-062679bb8c0ba4bce kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:us-west-1b topology.hostpath.csi/node:i-062679bb8c0ba4bce topology.kubernetes.io/region:us-west-1 topology.kubernetes.io/zone:us-west-1b] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-062679bb8c0ba4bce"} node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:172.20.45.43/19 projectcalico.org/IPv4IPIPTunnelAddr:100.106.79.0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-03-25 05:27:31 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-25 05:27:31 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-03-25 05:27:31 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2023-03-25 05:27:31 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2023-03-25 05:27:44 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.7.0/24\"":{}}}} } {calico-node Update v1 2023-03-25 05:27:50 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4IPIPTunnelAddr":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2023-03-25 05:40:50 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.7.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-1b/i-062679bb8c0ba4bce,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.7.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051689472 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946831872 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-03-25 05:27:50 +0000 UTC,LastTransitionTime:2023-03-25 05:27:50 +0000 UTC,Reason:CalicoIsUp,Message:Calico is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-25 05:40:36 +0000 UTC,LastTransitionTime:2023-03-25 05:27:31 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-25 05:40:36 +0000 UTC,LastTransitionTime:2023-03-25 05:27:31 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-25 05:40:36 +0000 UTC,LastTransitionTime:2023-03-25 05:27:31 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-25 05:40:36 +0000 UTC,LastTransitionTime:2023-03-25 05:27:44 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.45.43,},NodeAddress{Type:ExternalIP,Address:3.101.133.108,},NodeAddress{Type:InternalDNS,Address:i-062679bb8c0ba4bce.us-west-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-062679bb8c0ba4bce.us-west-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-3-101-133-108.us-west-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2528f599e9b1e199793fb5cd06d205,SystemUUID:ec2528f5-99e9-b1e1-9979-3fb5cd06d205,BootID:ff56b70b-c8c5-46db-8cc5-4353c8f535c6,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.28.0-alpha.0,KubeProxyVersion:v1.28.0-alpha.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:112030336,},ContainerImage{Names:[docker.io/calico/cni@sha256:e282ea2914c806b5de2976330a17cfb5e6dcef47147bceb1432ca5c75fd46f50 docker.io/calico/cni:v3.24.5],SizeBytes:87458443,},ContainerImage{Names:[docker.io/calico/node@sha256:5972ad2bcbdc668564d3e26960c9c513b2d7b05581c704747cf7c62ef3a405a6 docker.io/calico/node:v3.24.5],SizeBytes:81584866,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nautilus@sha256:80ba6c8c44f9623f06e868a1aa66026c8ec438ad814f9ec95e9333b415fe3550 registry.k8s.io/e2e-test-images/nautilus:1.7],SizeBytes:49641698,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:e468dddcd275163a042ab297b2d8c2aca50d5e148d2d22f3b6ba119e2f31fa79 registry.k8s.io/sig-storage/csi-provisioner:v3.4.0],SizeBytes:27427836,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/kube-proxy@sha256:8a03b3147cd7be8e8235ae1cd70cf69aca9ce03ba1cfaffea320540bb5afa496 registry.k8s.io/kube-proxy:v1.28.0-alpha.0],SizeBytes:23896850,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:23881995,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18233005,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:15273057,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8688564,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:7000082,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6978614,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Mar 25 05:41:05.686: INFO: Logging kubelet events for node i-062679bb8c0ba4bce Mar 25 05:41:05.805: INFO: Logging pods the kubelet thinks is on node i-062679bb8c0ba4bce Mar 25 05:41:05.894: INFO: netserver-2 started at 2023-03-25 05:40:40 +0000 UTC (0+1 container statuses recorded) Mar 25 05:41:05.894: INFO: Container webserver ready: true, restart count 0 Mar 25 05:41:05.894: INFO: adopt-release-7576w started at 2023-03-25 05:40:50 +0000 UTC (0+1 container statuses recorded) Mar 25 05:41:05.894: INFO: Container c ready: true, restart count 0 Mar 25 05:41:05.894: INFO: test-container-pod started at 2023-03-25 05:41:01 +0000 UTC (0+1 container statuses recorded) Mar 25 05:41:05.894: INFO: Container webserver ready: true, restart count 0 Mar 25 05:41:05.894: INFO: replace-27995381-f44vd started at 2023-03-25 05:41:00 +0000 UTC (0+1 container statuses recorded) Mar 25 05:41:05.894: INFO: Container c ready: true, restart count 0 Mar 25 05:41:05.894: INFO: pod-0 started at 2023-03-25 05:41:04 +0000 UTC (0+1 container statuses recorded) Mar 25 05:41:05.894: INFO: Container donothing ready: false, restart count 0 Mar 25 05:41:05.894: INFO: kube-proxy-i-062679bb8c0ba4bce started at 2023-03-25 05:27:32 +0000 UTC (0+1 container statuses recorded) Mar 25 05:41:05.894: INFO: Container kube-proxy ready: true, restart count 0 Mar 25 05:41:05.894: INFO: ebs-csi-node-7twrq started at 2023-03-25 05:27:32 +0000 UTC (0+3 container statuses recorded) Mar 25 05:41:05.894: INFO: Container ebs-plugin ready: true, restart count 0 Mar 25 05:41:05.894: INFO: Container liveness-probe ready: true, restart count 0 Mar 25 05:41:05.894: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 25 05:41:05.894: INFO: client-containers-777cf008-9df3-4637-9b03-e4e78a78533b started at 2023-03-25 05:40:52 +0000 UTC (0+1 container statuses recorded) Mar 25 05:41:05.894: INFO: Container agnhost-container ready: true, restart count 0 Mar 25 05:41:05.894: INFO: pod-2 started at 2023-03-25 05:41:04 +0000 UTC (0+1 container statuses recorded) Mar 25 05:41:05.894: INFO: Container donothing ready: false, restart count 0 Mar 25 05:41:05.894: INFO: downwardapi-volume-cb2ff7f5-8691-46f8-aa97-967744c122ce started at 2023-03-25 05:41:03 +0000 UTC (0+1 container statuses recorded) Mar 25 05:41:05.894: INFO: Container client-container ready: false, restart count 0 Mar 25 05:41:05.894: INFO: test-rolling-update-with-lb-745c47894-gmvg4 started at 2023-03-25 05:41:01 +0000 UTC (0+1 container statuses recorded) Mar 25 05:41:05.894: INFO: Container agnhost ready: true, restart count 0 Mar 25 05:41:05.894: INFO: hostexec-i-062679bb8c0ba4bce-gg2x9 started at 2023-03-25 05:41:01 +0000 UTC (0+1 container statuses recorded) Mar 25 05:41:05.894: INFO: Container agnhost-container ready: true, restart count 0 Mar 25 05:41:05.894: INFO: calico-node-zwm5j started at 2023-03-25 05:27:32 +0000 UTC (3+1 container statuses recorded) Mar 25 05:41:05.894: INFO: Init container upgrade-ipam ready: true, restart count 0 Mar 25 05:41:05.894: INFO: Init container install-cni ready: true, restart count 0 Mar 25 05:41:05.894: INFO: Init container mount-bpffs ready: true, restart count 0 Mar 25 05:41:05.894: INFO: Container calico-node ready: true, restart count 0 Mar 25 05:41:05.894: INFO: coredns-559769c974-p9vg6 started at 2023-03-25 05:36:16 +0000 UTC (0+1 container statuses recorded) Mar 25 05:41:05.894: INFO: Container coredns ready: true, restart count 0 Mar 25 05:41:05.894: INFO: pod-1 started at 2023-03-25 05:41:04 +0000 UTC (0+1 container statuses recorded) Mar 25 05:41:05.894: INFO: Container donothing ready: false, restart count 0 Mar 25 05:41:06.354: INFO: Latency metrics for node i-062679bb8c0ba4bce Mar 25 05:41:06.354: INFO: Logging node info for node i-072608201d2c0f1ad Mar 25 05:41:06.403: INFO: Node Info: &Node{ObjectMeta:{i-072608201d2c0f1ad 264dfba1-3cf0-4c99-8e8c-098f0f792077 17410 0 2023-03-25 05:35:29 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-1 failure-domain.beta.kubernetes.io/zone:us-west-1b kubernetes.io/arch:amd64 kubernetes.io/hostname:i-072608201d2c0f1ad kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:us-west-1b topology.hostpath.csi/node:i-072608201d2c0f1ad topology.kubernetes.io/region:us-west-1 topology.kubernetes.io/zone:us-west-1b] map[csi.volume.kubernetes.io/nodeid:{"csi-mock-csi-mock-volumes-expansion-2568":"csi-mock-csi-mock-volumes-expansion-2568","ebs.csi.aws.com":"i-072608201d2c0f1ad"} node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:172.20.38.125/19 projectcalico.org/IPv4IPIPTunnelAddr:100.107.220.192 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-03-25 05:35:29 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-25 05:35:29 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-03-25 05:35:29 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2023-03-25 05:35:29 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2023-03-25 05:35:41 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.9.0/24\"":{}}}} } {calico-node Update v1 2023-03-25 05:35:48 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4IPIPTunnelAddr":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2023-03-25 05:40:46 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-03-25 05:40:46 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.9.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-1b/i-072608201d2c0f1ad,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.9.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051689472 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946831872 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-03-25 05:35:47 +0000 UTC,LastTransitionTime:2023-03-25 05:35:47 +0000 UTC,Reason:CalicoIsUp,Message:Calico is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-25 05:40:46 +0000 UTC,LastTransitionTime:2023-03-25 05:35:29 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-25 05:40:46 +0000 UTC,LastTransitionTime:2023-03-25 05:35:29 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-25 05:40:46 +0000 UTC,LastTransitionTime:2023-03-25 05:35:29 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-25 05:40:46 +0000 UTC,LastTransitionTime:2023-03-25 05:35:41 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.38.125,},NodeAddress{Type:ExternalIP,Address:3.101.88.10,},NodeAddress{Type:InternalDNS,Address:i-072608201d2c0f1ad.us-west-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-072608201d2c0f1ad.us-west-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-3-101-88-10.us-west-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2d3818c4f80a253dcc1a60e148dbc8,SystemUUID:ec2d3818-c4f8-0a25-3dcc-1a60e148dbc8,BootID:77d046cf-75ea-4a8c-970a-3190d64e019b,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.28.0-alpha.0,KubeProxyVersion:v1.28.0-alpha.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[docker.io/calico/cni@sha256:e282ea2914c806b5de2976330a17cfb5e6dcef47147bceb1432ca5c75fd46f50 docker.io/calico/cni:v3.24.5],SizeBytes:87458443,},ContainerImage{Names:[docker.io/calico/node@sha256:5972ad2bcbdc668564d3e26960c9c513b2d7b05581c704747cf7c62ef3a405a6 docker.io/calico/node:v3.24.5],SizeBytes:81584866,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:e468dddcd275163a042ab297b2d8c2aca50d5e148d2d22f3b6ba119e2f31fa79 registry.k8s.io/sig-storage/csi-provisioner:v3.4.0],SizeBytes:27427836,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/kube-proxy@sha256:8a03b3147cd7be8e8235ae1cd70cf69aca9ce03ba1cfaffea320540bb5afa496 registry.k8s.io/kube-proxy:v1.28.0-alpha.0],SizeBytes:23896850,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:23881995,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18758628,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18233005,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8688564,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:7000082,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6978614,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[kubernetes.io/csi/csi-mock-csi-mock-volumes-expansion-2568^4 kubernetes.io/csi/ebs.csi.aws.com^vol-010a5e56be2318f09],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-mock-csi-mock-volumes-expansion-2568^4,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-010a5e56be2318f09,DevicePath:,},},Config:nil,},} Mar 25 05:41:06.403: INFO: Logging kubelet events for node i-072608201d2c0f1ad Mar 25 05:41:06.465: INFO: Logging pods the kubelet thinks is on node i-072608201d2c0f1ad Mar 25 05:41:06.521: INFO: webserver-pod started at 2023-03-25 05:39:46 +0000 UTC (0+1 container statuses recorded) Mar 25 05:41:06.521: INFO: Container agnhost ready: false, restart count 0 Mar 25 05:41:06.521: INFO: liveness-78b08cf2-0e88-4ba6-aa64-2fa547995658 started at 2023-03-25 05:40:42 +0000 UTC (0+1 container statuses recorded) Mar 25 05:41:06.521: INFO: Container agnhost-container ready: true, restart count 0 Mar 25 05:41:06.521: INFO: ebs-csi-node-r7mcm started at 2023-03-25 05:35:30 +0000 UTC (0+3 container statuses recorded) Mar 25 05:41:06.521: INFO: Container ebs-plugin ready: true, restart count 0 Mar 25 05:41:06.521: INFO: Container liveness-probe ready: true, restart count 0 Mar 25 05:41:06.521: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 25 05:41:06.522: INFO: csi-mockplugin-0 started at 2023-03-25 05:40:27 +0000 UTC (0+4 container statuses recorded) Mar 25 05:41:06.522: INFO: Container busybox ready: true, restart count 0 Mar 25 05:41:06.522: INFO: Container csi-provisioner ready: true, restart count 0 Mar 25 05:41:06.522: INFO: Container driver-registrar ready: true, restart count 0 Mar 25 05:41:06.522: INFO: Container mock ready: true, restart count 0 Mar 25 05:41:06.522: INFO: test-rolling-update-with-lb-55d4c6cb8c-d4kmk started at 2023-03-25 05:40:57 +0000 UTC (0+1 container statuses recorded) Mar 25 05:41:06.522: INFO: Container agnhost ready: true, restart count 0 Mar 25 05:41:06.522: INFO: pod-with-poststart-https-hook started at 2023-03-25 05:40:57 +0000 UTC (0+1 container statuses recorded) Mar 25 05:41:06.522: INFO: Container pod-with-poststart-https-hook ready: true, restart count 0 Mar 25 05:41:06.522: INFO: rs-jcfcz started at 2023-03-25 05:40:39 +0000 UTC (0+1 container statuses recorded) Mar 25 05:41:06.522: INFO: Container donothing ready: false, restart count 0 Mar 25 05:41:06.522: INFO: csi-mockplugin-attacher-0 started at 2023-03-25 05:40:27 +0000 UTC (0+1 container statuses recorded) Mar 25 05:41:06.522: INFO: Container csi-attacher ready: true, restart count 0 Mar 25 05:41:06.522: INFO: csi-mockplugin-resizer-0 started at 2023-03-25 05:40:27 +0000 UTC (0+1 container statuses recorded) Mar 25 05:41:06.522: INFO: Container csi-resizer ready: true, restart count 0 Mar 25 05:41:06.522: INFO: calico-node-lc9zl started at 2023-03-25 05:35:30 +0000 UTC (3+1 container statuses recorded) Mar 25 05:41:06.522: INFO: Init container upgrade-ipam ready: true, restart count 0 Mar 25 05:41:06.522: INFO: Init container install-cni ready: true, restart count 0 Mar 25 05:41:06.522: INFO: Init container mount-bpffs ready: true, restart count 0 Mar 25 05:41:06.522: INFO: Container calico-node ready: true, restart count 0 Mar 25 05:41:06.522: INFO: pod-projected-configmaps-61878de4-bfbc-4c4c-bb90-4daf9441ed77 started at 2023-03-25 05:39:21 +0000 UTC (0+3 container statuses recorded) Mar 25 05:41:06.522: INFO: Container createcm-volume-test ready: true, restart count 0 Mar 25 05:41:06.522: INFO: Container delcm-volume-test ready: true, restart count 0 Mar 25 05:41:06.522: INFO: Container updcm-volume-test ready: true, restart count 0 Mar 25 05:41:06.522: INFO: adopt-release-68st7 started at 2023-03-25 05:40:41 +0000 UTC (0+1 container statuses recorded) Mar 25 05:41:06.522: INFO: Container c ready: true, restart count 0 Mar 25 05:41:06.522: INFO: pvc-volume-tester-4sk4l started at 2023-03-25 05:40:58 +0000 UTC (0+1 container statuses recorded) Mar 25 05:41:06.522: INFO: Container volume-tester ready: false, restart count 0 Mar 25 05:41:06.522: INFO: kube-proxy-i-072608201d2c0f1ad started at 2023-03-25 05:35:30 +0000 UTC (0+1 container statuses recorded) Mar 25 05:41:06.522: INFO: Container kube-proxy ready: true, restart count 0 Mar 25 05:41:06.522: INFO: test-rolling-update-with-lb-745c47894-npn8g started at 2023-03-25 05:41:04 +0000 UTC (0+1 container statuses recorded) Mar 25 05:41:06.522: INFO: Container agnhost ready: false, restart count 0 Mar 25 05:41:06.522: INFO: netserver-3 started at 2023-03-25 05:40:40 +0000 UTC (0+1 container statuses recorded) Mar 25 05:41:06.522: INFO: Container webserver ready: true, restart count 0 Mar 25 05:41:06.522: INFO: sample-apiserver-deployment-6dfd6dfd5b-jv27c started at 2023-03-25 05:40:42 +0000 UTC (0+2 container statuses recorded) Mar 25 05:41:06.522: INFO: Container etcd ready: true, restart count 0 Mar 25 05:41:06.522: INFO: Container sample-apiserver ready: false, restart count 0 Mar 25 05:41:06.522: INFO: pod-handle-http-request started at 2023-03-25 05:40:55 +0000 UTC (0+2 container statuses recorded) Mar 25 05:41:06.522: INFO: Container container-handle-http-request ready: true, restart count 0 Mar 25 05:41:06.522: INFO: Container container-handle-https-request ready: true, restart count 0 Mar 25 05:41:06.784: INFO: Latency metrics for node i-072608201d2c0f1ad END STEP: dump namespace information after failure - test/e2e/framework/framework.go:288 @ 03/25/23 05:41:06.784 (2.795s) < Exit [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] subPath - dump namespaces | framework.go:209 @ 03/25/23 05:41:06.784 (2.795s) > Enter [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] subPath - tear down framework | framework.go:206 @ 03/25/23 05:41:06.784 STEP: Destroying namespace "provisioning-3562" for this suite. - test/e2e/framework/framework.go:351 @ 03/25/23 05:41:06.784 < Exit [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] subPath - tear down framework | framework.go:206 @ 03/25/23 05:41:06.837 (53ms) > Enter [ReportAfterEach] TOP-LEVEL - test/e2e/e2e_test.go:144 @ 03/25/23 05:41:06.838 < Exit [ReportAfterEach] TOP-LEVEL - test/e2e/e2e_test.go:144 @ 03/25/23 05:41:06.838 (0s)
Filter through log files | View test history on testgrid
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[It\]\sExternal\sStorage\s\[Driver\:\sebs\.csi\.aws\.com\]\s\[Testpattern\:\sDynamic\sPV\s\(default\sfs\)\]\ssubPath\sshould\ssupport\sexisting\sdirectory$'
[FAILED] failed to get logs from pod-subpath-test-dynamicpv-72sq for test-container-volume-dynamicpv-72sq: Get "https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/provisioning-4160/pods/pod-subpath-test-dynamicpv-72sq/log?container=test-container-volume-dynamicpv-72sq&previous=false": stream error: stream ID 561; INTERNAL_ERROR; received from peer In [It] at: test/e2e/framework/pod/output/output.go:247 @ 03/25/23 05:54:51.735from junit_01.xml
> Enter [BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/storage/framework/testsuite.go:51 @ 03/25/23 05:54:36.547 < Exit [BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/storage/framework/testsuite.go:51 @ 03/25/23 05:54:36.547 (0s) > Enter [BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath - set up framework | framework.go:191 @ 03/25/23 05:54:36.547 STEP: Creating a kubernetes client - test/e2e/framework/framework.go:211 @ 03/25/23 05:54:36.547 Mar 25 05:54:36.547: INFO: >>> kubeConfig: /root/.kube/config STEP: Building a namespace api object, basename provisioning - test/e2e/framework/framework.go:250 @ 03/25/23 05:54:36.548 STEP: Waiting for a default service account to be provisioned in namespace - test/e2e/framework/framework.go:259 @ 03/25/23 05:54:36.703 STEP: Waiting for kube-root-ca.crt to be provisioned in namespace - test/e2e/framework/framework.go:262 @ 03/25/23 05:54:36.8 < Exit [BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath - set up framework | framework.go:191 @ 03/25/23 05:54:36.895 (348ms) > Enter [BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/framework/metrics/init/init.go:33 @ 03/25/23 05:54:36.895 < Exit [BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/framework/metrics/init/init.go:33 @ 03/25/23 05:54:36.895 (0s) > Enter [It] should support existing directory - test/e2e/storage/testsuites/subpath.go:205 @ 03/25/23 05:54:36.895 Mar 25 05:54:36.896: INFO: Creating resource for dynamic PV Mar 25 05:54:36.896: INFO: Using claimSize:1Gi, test suite supported size:{ 1Mi}, driver(ebs.csi.aws.com) supported size:{ 1Mi} STEP: creating a StorageClass provisioning-4160-e2e-scb8nnh - test/e2e/storage/framework/volume_resource.go:102 @ 03/25/23 05:54:36.896 STEP: creating a claim - test/e2e/storage/framework/volume_resource.go:294 @ 03/25/23 05:54:36.955 Mar 25 05:54:36.955: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil E0325 05:54:37.013026 7722 retrywatcher.go:130] "Watch failed" err="context canceled" STEP: Creating pod pod-subpath-test-dynamicpv-72sq - test/e2e/storage/testsuites/subpath.go:691 @ 03/25/23 05:54:37.06 STEP: Creating a pod to test subpath - test/e2e/framework/pod/output/output.go:243 @ 03/25/23 05:54:37.06 E0325 05:54:38.013950 7722 retrywatcher.go:130] "Watch failed" err="context canceled" E0325 05:54:39.014265 7722 retrywatcher.go:130] "Watch failed" err="context canceled" E0325 05:54:40.014840 7722 retrywatcher.go:130] "Watch failed" err="context canceled" E0325 05:54:41.015546 7722 retrywatcher.go:130] "Watch failed" err="context canceled" E0325 05:54:42.015657 7722 retrywatcher.go:130] "Watch failed" err="context canceled" E0325 05:54:43.015814 7722 retrywatcher.go:130] "Watch failed" err="context canceled" E0325 05:54:44.015959 7722 retrywatcher.go:130] "Watch failed" err="context canceled" E0325 05:54:45.016467 7722 retrywatcher.go:130] "Watch failed" err="context canceled" E0325 05:54:46.017334 7722 retrywatcher.go:130] "Watch failed" err="context canceled" E0325 05:54:47.017935 7722 retrywatcher.go:130] "Watch failed" err="context canceled" E0325 05:54:48.018930 7722 retrywatcher.go:130] "Watch failed" err="context canceled" E0325 05:54:49.018991 7722 retrywatcher.go:130] "Watch failed" err="context canceled" E0325 05:54:50.019874 7722 retrywatcher.go:130] "Watch failed" err="context canceled" E0325 05:54:51.020238 7722 retrywatcher.go:130] "Watch failed" err="context canceled" STEP: Saw pod success - test/e2e/framework/pod/wait.go:409 @ 03/25/23 05:54:51.524 Mar 25 05:54:51.574: INFO: Trying to get logs from node i-062679bb8c0ba4bce pod pod-subpath-test-dynamicpv-72sq container test-container-volume-dynamicpv-72sq: <nil> Mar 25 05:54:51.623: INFO: Failed to get logs from node "i-062679bb8c0ba4bce" pod "pod-subpath-test-dynamicpv-72sq" container "test-container-volume-dynamicpv-72sq". Get "https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/provisioning-4160/pods/pod-subpath-test-dynamicpv-72sq/log?container=test-container-volume-dynamicpv-72sq&previous=false": stream error: stream ID 561; INTERNAL_ERROR; received from peer STEP: delete the pod - test/e2e/framework/pod/output/output.go:168 @ 03/25/23 05:54:51.623 Mar 25 05:54:51.735: INFO: Unexpected error: <*fmt.wrapError | 0xc0005b2220>: failed to get logs from pod-subpath-test-dynamicpv-72sq for test-container-volume-dynamicpv-72sq: Get "https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/provisioning-4160/pods/pod-subpath-test-dynamicpv-72sq/log?container=test-container-volume-dynamicpv-72sq&previous=false": stream error: stream ID 561; INTERNAL_ERROR; received from peer { msg: "failed to get logs from pod-subpath-test-dynamicpv-72sq for test-container-volume-dynamicpv-72sq: Get \"https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/provisioning-4160/pods/pod-subpath-test-dynamicpv-72sq/log?container=test-container-volume-dynamicpv-72sq&previous=false\": stream error: stream ID 561; INTERNAL_ERROR; received from peer", err: <*url.Error | 0xc00413bbc0>{ Op: "Get", URL: "https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/provisioning-4160/pods/pod-subpath-test-dynamicpv-72sq/log?container=test-container-volume-dynamicpv-72sq&previous=false", Err: <http2.StreamError>{ StreamID: 561, Code: 2, Cause: <*errors.errorString | 0xc000233690>{ s: "received from peer", }, }, }, } [FAILED] failed to get logs from pod-subpath-test-dynamicpv-72sq for test-container-volume-dynamicpv-72sq: Get "https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/provisioning-4160/pods/pod-subpath-test-dynamicpv-72sq/log?container=test-container-volume-dynamicpv-72sq&previous=false": stream error: stream ID 561; INTERNAL_ERROR; received from peer In [It] at: test/e2e/framework/pod/output/output.go:247 @ 03/25/23 05:54:51.735 < Exit [It] should support existing directory - test/e2e/storage/testsuites/subpath.go:205 @ 03/25/23 05:54:51.735 (14.84s) > Enter [AfterEach] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/framework/node/init/init.go:33 @ 03/25/23 05:54:51.735 Mar 25 05:54:51.735: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready < Exit [AfterEach] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/framework/node/init/init.go:33 @ 03/25/23 05:54:51.788 (53ms) > Enter [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/storage/testsuites/subpath.go:207 @ 03/25/23 05:54:51.788 STEP: Deleting pod - test/e2e/storage/testsuites/subpath.go:172 @ 03/25/23 05:54:51.788 Mar 25 05:54:51.788: INFO: Deleting pod "pod-subpath-test-dynamicpv-72sq" in namespace "provisioning-4160" STEP: Deleting pvc - test/e2e/storage/framework/volume_resource.go:181 @ 03/25/23 05:54:51.884 Mar 25 05:54:51.981: INFO: Deleting PersistentVolumeClaim "ebs.csi.aws.comthbr2" E0325 05:54:52.020720 7722 retrywatcher.go:130] "Watch failed" err="context canceled" Mar 25 05:54:52.033: INFO: Waiting up to 5m0s for PersistentVolume pvc-23423752-7929-437c-aa3e-6ab8716f7f69 to get deleted Mar 25 05:54:52.081: INFO: PersistentVolume pvc-23423752-7929-437c-aa3e-6ab8716f7f69 found and phase=Released (48.167397ms) E0325 05:54:53.020745 7722 retrywatcher.go:130] "Watch failed" err="context canceled" E0325 05:54:54.021149 7722 retrywatcher.go:130] "Watch failed" err="context canceled" E0325 05:54:55.021762 7722 retrywatcher.go:130] "Watch failed" err="context canceled" E0325 05:54:56.022444 7722 retrywatcher.go:130] "Watch failed" err="context canceled" E0325 05:54:57.023003 7722 retrywatcher.go:130] "Watch failed" err="context canceled" Mar 25 05:54:57.130: INFO: PersistentVolume pvc-23423752-7929-437c-aa3e-6ab8716f7f69 found and phase=Released (5.097275101s) E0325 05:54:58.023051 7722 retrywatcher.go:130] "Watch failed" err="context canceled" E0325 05:54:59.023925 7722 retrywatcher.go:130] "Watch failed" err="context canceled" E0325 05:55:00.024095 7722 retrywatcher.go:130] "Watch failed" err="context canceled" E0325 05:55:01.025027 7722 retrywatcher.go:130] "Watch failed" err="context canceled" E0325 05:55:02.025751 7722 retrywatcher.go:130] "Watch failed" err="context canceled" Mar 25 05:55:02.180: INFO: PersistentVolume pvc-23423752-7929-437c-aa3e-6ab8716f7f69 found and phase=Released (10.146754147s) E0325 05:55:03.026766 7722 retrywatcher.go:130] "Watch failed" err="context canceled" E0325 05:55:04.027010 7722 retrywatcher.go:130] "Watch failed" err="context canceled" E0325 05:55:05.027093 7722 retrywatcher.go:130] "Watch failed" err="context canceled" E0325 05:55:06.027696 7722 retrywatcher.go:130] "Watch failed" err="context canceled" E0325 05:55:07.028015 7722 retrywatcher.go:130] "Watch failed" err="context canceled" Mar 25 05:55:07.230: INFO: PersistentVolume pvc-23423752-7929-437c-aa3e-6ab8716f7f69 was removed STEP: Deleting sc - test/e2e/storage/framework/volume_resource.go:228 @ 03/25/23 05:55:07.23 < Exit [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/storage/testsuites/subpath.go:207 @ 03/25/23 05:55:07.281 (15.493s) > Enter [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/framework/metrics/init/init.go:35 @ 03/25/23 05:55:07.281 < Exit [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/framework/metrics/init/init.go:35 @ 03/25/23 05:55:07.281 (0s) > Enter [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] subPath - dump namespaces | framework.go:209 @ 03/25/23 05:55:07.281 STEP: dump namespace information after failure - test/e2e/framework/framework.go:288 @ 03/25/23 05:55:07.281 STEP: Collecting events from namespace "provisioning-4160". - test/e2e/framework/debug/dump.go:42 @ 03/25/23 05:55:07.281 STEP: Found 18 events. - test/e2e/framework/debug/dump.go:46 @ 03/25/23 05:55:07.331 Mar 25 05:55:07.331: INFO: At 2023-03-25 05:54:36 +0000 UTC - event for ebs.csi.aws.comthbr2: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding Mar 25 05:55:07.331: INFO: At 2023-03-25 05:54:37 +0000 UTC - event for ebs.csi.aws.comthbr2: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator Mar 25 05:55:07.331: INFO: At 2023-03-25 05:54:37 +0000 UTC - event for ebs.csi.aws.comthbr2: {ebs.csi.aws.com_i-0168f82458996cc53_0b589c46-3396-4499-94ba-eb51a48f7a58 } Provisioning: External provisioner is provisioning volume for claim "provisioning-4160/ebs.csi.aws.comthbr2" Mar 25 05:55:07.331: INFO: At 2023-03-25 05:54:40 +0000 UTC - event for ebs.csi.aws.comthbr2: {ebs.csi.aws.com_i-0168f82458996cc53_0b589c46-3396-4499-94ba-eb51a48f7a58 } ProvisioningSucceeded: Successfully provisioned volume pvc-23423752-7929-437c-aa3e-6ab8716f7f69 Mar 25 05:55:07.331: INFO: At 2023-03-25 05:54:41 +0000 UTC - event for pod-subpath-test-dynamicpv-72sq: {default-scheduler } Scheduled: Successfully assigned provisioning-4160/pod-subpath-test-dynamicpv-72sq to i-062679bb8c0ba4bce Mar 25 05:55:07.331: INFO: At 2023-03-25 05:54:43 +0000 UTC - event for pod-subpath-test-dynamicpv-72sq: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-23423752-7929-437c-aa3e-6ab8716f7f69" Mar 25 05:55:07.331: INFO: At 2023-03-25 05:54:44 +0000 UTC - event for pod-subpath-test-dynamicpv-72sq: {kubelet i-062679bb8c0ba4bce} Pulled: Container image "registry.k8s.io/e2e-test-images/busybox:1.29-4" already present on machine Mar 25 05:55:07.331: INFO: At 2023-03-25 05:54:45 +0000 UTC - event for pod-subpath-test-dynamicpv-72sq: {kubelet i-062679bb8c0ba4bce} Created: Created container init-volume-dynamicpv-72sq Mar 25 05:55:07.331: INFO: At 2023-03-25 05:54:45 +0000 UTC - event for pod-subpath-test-dynamicpv-72sq: {kubelet i-062679bb8c0ba4bce} Started: Started container init-volume-dynamicpv-72sq Mar 25 05:55:07.331: INFO: At 2023-03-25 05:54:45 +0000 UTC - event for pod-subpath-test-dynamicpv-72sq: {kubelet i-062679bb8c0ba4bce} Pulled: Container image "registry.k8s.io/e2e-test-images/agnhost:2.43" already present on machine Mar 25 05:55:07.331: INFO: At 2023-03-25 05:54:46 +0000 UTC - event for pod-subpath-test-dynamicpv-72sq: {kubelet i-062679bb8c0ba4bce} Created: Created container test-init-subpath-dynamicpv-72sq Mar 25 05:55:07.331: INFO: At 2023-03-25 05:54:46 +0000 UTC - event for pod-subpath-test-dynamicpv-72sq: {kubelet i-062679bb8c0ba4bce} Started: Started container test-init-subpath-dynamicpv-72sq Mar 25 05:55:07.331: INFO: At 2023-03-25 05:54:46 +0000 UTC - event for pod-subpath-test-dynamicpv-72sq: {kubelet i-062679bb8c0ba4bce} Pulled: Container image "registry.k8s.io/e2e-test-images/agnhost:2.43" already present on machine Mar 25 05:55:07.331: INFO: At 2023-03-25 05:54:47 +0000 UTC - event for pod-subpath-test-dynamicpv-72sq: {kubelet i-062679bb8c0ba4bce} Created: Created container test-container-subpath-dynamicpv-72sq Mar 25 05:55:07.331: INFO: At 2023-03-25 05:54:47 +0000 UTC - event for pod-subpath-test-dynamicpv-72sq: {kubelet i-062679bb8c0ba4bce} Started: Started container test-container-subpath-dynamicpv-72sq Mar 25 05:55:07.331: INFO: At 2023-03-25 05:54:47 +0000 UTC - event for pod-subpath-test-dynamicpv-72sq: {kubelet i-062679bb8c0ba4bce} Pulled: Container image "registry.k8s.io/e2e-test-images/agnhost:2.43" already present on machine Mar 25 05:55:07.331: INFO: At 2023-03-25 05:54:47 +0000 UTC - event for pod-subpath-test-dynamicpv-72sq: {kubelet i-062679bb8c0ba4bce} Created: Created container test-container-volume-dynamicpv-72sq Mar 25 05:55:07.331: INFO: At 2023-03-25 05:54:47 +0000 UTC - event for pod-subpath-test-dynamicpv-72sq: {kubelet i-062679bb8c0ba4bce} Started: Started container test-container-volume-dynamicpv-72sq Mar 25 05:55:07.381: INFO: POD NODE PHASE GRACE CONDITIONS Mar 25 05:55:07.381: INFO: Mar 25 05:55:07.436: INFO: Logging node info for node i-0168f82458996cc53 Mar 25 05:55:07.532: INFO: Node Info: &Node{ObjectMeta:{i-0168f82458996cc53 f4eb302c-c477-45ac-bcac-ffed65f6580f 56232 0 2023-03-25 05:18:57 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-1 failure-domain.beta.kubernetes.io/zone:us-west-1b kops.k8s.io/kops-controller-pki: kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0168f82458996cc53 kubernetes.io/os:linux node-role.kubernetes.io/control-plane: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:us-west-1b topology.kubernetes.io/region:us-west-1 topology.kubernetes.io/zone:us-west-1b] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0168f82458996cc53"} node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:172.20.48.81/19 projectcalico.org/IPv4IPIPTunnelAddr:100.100.215.192 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-03-25 05:18:57 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {protokube Update v1 2023-03-25 05:18:58 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {calico-node Update v1 2023-03-25 05:20:22 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4IPIPTunnelAddr":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {aws-cloud-controller-manager Update v1 2023-03-25 05:20:26 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-25 05:20:26 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2023-03-25 05:20:31 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.5.0/24\"":{}},"f:taints":{}}} } {kubelet Update v1 2023-03-25 05:52:18 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.5.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-1b/i-0168f82458996cc53,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/control-plane,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.5.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3892310016 0} {<nil>} 3801084Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3787452416 0} {<nil>} 3698684Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-03-25 05:20:22 +0000 UTC,LastTransitionTime:2023-03-25 05:20:22 +0000 UTC,Reason:CalicoIsUp,Message:Calico is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-25 05:52:18 +0000 UTC,LastTransitionTime:2023-03-25 05:18:50 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-25 05:52:18 +0000 UTC,LastTransitionTime:2023-03-25 05:18:50 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-25 05:52:18 +0000 UTC,LastTransitionTime:2023-03-25 05:18:50 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-25 05:52:18 +0000 UTC,LastTransitionTime:2023-03-25 05:20:29 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.48.81,},NodeAddress{Type:ExternalIP,Address:3.101.135.77,},NodeAddress{Type:InternalDNS,Address:i-0168f82458996cc53.us-west-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-0168f82458996cc53.us-west-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-3-101-135-77.us-west-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2d1ccc7ebf313420bd1f530a94fbfd,SystemUUID:ec2d1ccc-7ebf-3134-20bd-1f530a94fbfd,BootID:15cee321-56e7-4290-ab48-47b60b0bc657,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.28.0-alpha.0,KubeProxyVersion:v1.28.0-alpha.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcd@sha256:dd75ec974b0a2a6f6bb47001ba09207976e625db898d1b16735528c009cb171c registry.k8s.io/etcd:3.5.6-0],SizeBytes:102542580,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:6f72b851544986cb0921b53ea655ec04c36131248f16d4ad110cb3ca0c369dc1 registry.k8s.io/etcd:3.5.4-0],SizeBytes:102157811,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:13f53ed1d91e2e11aac476ee9a0269fdda6cc4874eba903efd40daf50c55eee5 registry.k8s.io/etcd:3.5.3-0],SizeBytes:102143581,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:51eae8381dcb1078289fa7b4f3df2630cdc18d09fb56f8e56b41c40e191d6c83 registry.k8s.io/etcd:3.5.7-0],SizeBytes:101639218,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:4afb99b4690b418ffc2ceb67e1a17376457e441c1f09ab55447f0aaf992fa646 registry.k8s.io/etcd:3.4.3-0],SizeBytes:100947667,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:9ce33ba33d8e738a5b85ed50b5080ac746deceed4a7496c550927a7a19ca3b6d registry.k8s.io/etcd:3.5.0-0],SizeBytes:99868722,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:64b9ea357325d5db9f8a723dcf503b5a449177b17ac87d69481e126bb724c263 registry.k8s.io/etcd:3.5.1-0],SizeBytes:98888614,},ContainerImage{Names:[docker.io/calico/cni@sha256:e282ea2914c806b5de2976330a17cfb5e6dcef47147bceb1432ca5c75fd46f50 docker.io/calico/cni:v3.24.5],SizeBytes:87458443,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:4ad90a11b55313b182afc186b9876c8e891531b8db4c9bf1541953021618d0e2 registry.k8s.io/etcd:3.4.13-0],SizeBytes:86742272,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:0d2ab6b15b4fe0b903b9e81192d942fb39c29be61fe341cd16db5d0165b0435c registry.k8s.io/etcd:3.3.17-0],SizeBytes:85524408,},ContainerImage{Names:[docker.io/calico/node@sha256:5972ad2bcbdc668564d3e26960c9c513b2d7b05581c704747cf7c62ef3a405a6 docker.io/calico/node:v3.24.5],SizeBytes:81584866,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:17da501f5d2a675be46040422a27b7cc21b8a43895ac998b171db1c346f361f7 registry.k8s.io/etcd:3.3.10-0],SizeBytes:76162358,},ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager-slim@sha256:8727adc059334e0e3329dbccce116a734da4a090da5e86c8cddccaf333caa666 registry.k8s.io/etcdadm/etcd-manager-slim:v3.0.20230201],SizeBytes:63926165,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:905d7ca17fd02bc24c0eba9a062753aba15db3e31422390bc3238eb762339b20 registry.k8s.io/etcd:3.2.24-1],SizeBytes:63423287,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.27.0-alpha.1],SizeBytes:45219545,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.27.0-alpha.1],SizeBytes:43628585,},ContainerImage{Names:[registry.k8s.io/kube-apiserver@sha256:49934a29c559e79ed19fc38b4697b18d8fe174b17bf56192588575f980c93956 registry.k8s.io/kube-apiserver:v1.28.0-alpha.0],SizeBytes:33212506,},ContainerImage{Names:[docker.io/calico/kube-controllers@sha256:2b6acd7f677f76ffe12ecf3ea7df92eb9b1bdb07336d1ac2a54c7631fb753f7e docker.io/calico/kube-controllers:v3.24.5],SizeBytes:31137169,},ContainerImage{Names:[registry.k8s.io/kube-controller-manager@sha256:974748384634982c73527fdb3c9ec0ec476bc0d52b3dd1d611a380b7f99af1f3 registry.k8s.io/kube-controller-manager:v1.28.0-alpha.0],SizeBytes:30820280,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/kube-proxy@sha256:8a03b3147cd7be8e8235ae1cd70cf69aca9ce03ba1cfaffea320540bb5afa496 registry.k8s.io/kube-proxy:v1.28.0-alpha.0],SizeBytes:23896850,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:23345856,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:22381475,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:22085298,},ContainerImage{Names:[registry.k8s.io/provider-aws/cloud-controller-manager@sha256:fdeb61e3e42ecd9cca868d550ebdb88dd6341d9e91fcfa9a37e227dab2ad22cb registry.k8s.io/provider-aws/cloud-controller-manager:v1.26.0],SizeBytes:20154862,},ContainerImage{Names:[registry.k8s.io/kube-scheduler@sha256:cbf4007abb076188f16b975bc2b74452603cc7b34c60a29d92f14c38ad02dc11 registry.k8s.io/kube-scheduler:v1.28.0-alpha.0],SizeBytes:18078674,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.27.0-alpha.1],SizeBytes:5074298,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Mar 25 05:55:07.533: INFO: Logging kubelet events for node i-0168f82458996cc53 Mar 25 05:55:07.688: INFO: Logging pods the kubelet thinks is on node i-0168f82458996cc53 Mar 25 05:55:07.754: INFO: kube-apiserver-i-0168f82458996cc53 started at 2023-03-25 05:17:36 +0000 UTC (0+2 container statuses recorded) Mar 25 05:55:07.754: INFO: Container healthcheck ready: true, restart count 0 Mar 25 05:55:07.754: INFO: Container kube-apiserver ready: true, restart count 2 Mar 25 05:55:07.754: INFO: calico-node-rdjq2 started at 2023-03-25 05:20:06 +0000 UTC (3+1 container statuses recorded) Mar 25 05:55:07.754: INFO: Init container upgrade-ipam ready: true, restart count 0 Mar 25 05:55:07.754: INFO: Init container install-cni ready: true, restart count 0 Mar 25 05:55:07.754: INFO: Init container mount-bpffs ready: true, restart count 0 Mar 25 05:55:07.754: INFO: Container calico-node ready: true, restart count 0 Mar 25 05:55:07.754: INFO: kube-controller-manager-i-0168f82458996cc53 started at 2023-03-25 05:17:36 +0000 UTC (0+1 container statuses recorded) Mar 25 05:55:07.754: INFO: Container kube-controller-manager ready: true, restart count 4 Mar 25 05:55:07.754: INFO: etcd-manager-events-i-0168f82458996cc53 started at 2023-03-25 05:17:36 +0000 UTC (11+1 container statuses recorded) Mar 25 05:55:07.754: INFO: Init container init-etcd-3-2-24 ready: true, restart count 0 Mar 25 05:55:07.754: INFO: Init container init-etcd-3-3-10 ready: true, restart count 0 Mar 25 05:55:07.754: INFO: Init container init-etcd-3-3-17 ready: true, restart count 0 Mar 25 05:55:07.754: INFO: Init container init-etcd-3-4-13 ready: true, restart count 0 Mar 25 05:55:07.754: INFO: Init container init-etcd-3-4-3 ready: true, restart count 0 Mar 25 05:55:07.754: INFO: Init container init-etcd-3-5-0 ready: true, restart count 0 Mar 25 05:55:07.754: INFO: Init container init-etcd-3-5-1 ready: true, restart count 0 Mar 25 05:55:07.754: INFO: Init container init-etcd-3-5-3 ready: true, restart count 0 Mar 25 05:55:07.754: INFO: Init container init-etcd-3-5-4 ready: true, restart count 0 Mar 25 05:55:07.754: INFO: Init container init-etcd-3-5-6 ready: true, restart count 0 Mar 25 05:55:07.754: INFO: Init container init-etcd-3-5-7 ready: true, restart count 0 Mar 25 05:55:07.754: INFO: Container etcd-manager ready: true, restart count 0 Mar 25 05:55:07.754: INFO: etcd-manager-main-i-0168f82458996cc53 started at 2023-03-25 05:17:36 +0000 UTC (11+1 container statuses recorded) Mar 25 05:55:07.754: INFO: Init container init-etcd-3-2-24 ready: true, restart count 0 Mar 25 05:55:07.754: INFO: Init container init-etcd-3-3-10 ready: true, restart count 0 Mar 25 05:55:07.754: INFO: Init container init-etcd-3-3-17 ready: true, restart count 0 Mar 25 05:55:07.754: INFO: Init container init-etcd-3-4-13 ready: true, restart count 0 Mar 25 05:55:07.754: INFO: Init container init-etcd-3-4-3 ready: true, restart count 0 Mar 25 05:55:07.754: INFO: Init container init-etcd-3-5-0 ready: true, restart count 0 Mar 25 05:55:07.754: INFO: Init container init-etcd-3-5-1 ready: true, restart count 0 Mar 25 05:55:07.754: INFO: Init container init-etcd-3-5-3 ready: true, restart count 0 Mar 25 05:55:07.754: INFO: Init container init-etcd-3-5-4 ready: true, restart count 0 Mar 25 05:55:07.754: INFO: Init container init-etcd-3-5-6 ready: true, restart count 0 Mar 25 05:55:07.754: INFO: Init container init-etcd-3-5-7 ready: true, restart count 0 Mar 25 05:55:07.754: INFO: Container etcd-manager ready: true, restart count 0 Mar 25 05:55:07.754: INFO: dns-controller-69987775c6-hbn69 started at 2023-03-25 05:19:47 +0000 UTC (0+1 container statuses recorded) Mar 25 05:55:07.754: INFO: Container dns-controller ready: true, restart count 0 Mar 25 05:55:07.754: INFO: aws-cloud-controller-manager-26vbv started at 2023-03-25 05:20:06 +0000 UTC (0+1 container statuses recorded) Mar 25 05:55:07.754: INFO: Container aws-cloud-controller-manager ready: true, restart count 0 Mar 25 05:55:07.754: INFO: ebs-csi-node-zjtj6 started at 2023-03-25 05:20:06 +0000 UTC (0+3 container statuses recorded) Mar 25 05:55:07.754: INFO: Container ebs-plugin ready: true, restart count 0 Mar 25 05:55:07.754: INFO: Container liveness-probe ready: true, restart count 0 Mar 25 05:55:07.754: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 25 05:55:07.754: INFO: kops-controller-cgw95 started at 2023-03-25 05:20:06 +0000 UTC (0+1 container statuses recorded) Mar 25 05:55:07.754: INFO: Container kops-controller ready: true, restart count 0 Mar 25 05:55:07.754: INFO: calico-kube-controllers-55f488656-rr8rz started at 2023-03-25 05:20:29 +0000 UTC (0+1 container statuses recorded) Mar 25 05:55:07.754: INFO: Container calico-kube-controllers ready: true, restart count 0 Mar 25 05:55:07.754: INFO: kube-proxy-i-0168f82458996cc53 started at 2023-03-25 05:17:36 +0000 UTC (0+1 container statuses recorded) Mar 25 05:55:07.754: INFO: Container kube-proxy ready: true, restart count 0 Mar 25 05:55:07.754: INFO: kube-scheduler-i-0168f82458996cc53 started at 2023-03-25 05:17:36 +0000 UTC (0+1 container statuses recorded) Mar 25 05:55:07.754: INFO: Container kube-scheduler ready: true, restart count 0 Mar 25 05:55:07.754: INFO: ebs-csi-controller-79f984545f-9q5ks started at 2023-03-25 05:21:26 +0000 UTC (0+5 container statuses recorded) Mar 25 05:55:07.754: INFO: Container csi-attacher ready: true, restart count 0 Mar 25 05:55:07.754: INFO: Container csi-provisioner ready: true, restart count 0 Mar 25 05:55:07.754: INFO: Container csi-resizer ready: true, restart count 0 Mar 25 05:55:07.754: INFO: Container ebs-plugin ready: true, restart count 0 Mar 25 05:55:07.754: INFO: Container liveness-probe ready: true, restart count 0 E0325 05:55:08.028718 7722 retrywatcher.go:130] "Watch failed" err="context canceled" Mar 25 05:55:08.101: INFO: Latency metrics for node i-0168f82458996cc53 Mar 25 05:55:08.101: INFO: Logging node info for node i-03d301e6f030748f1 Mar 25 05:55:08.152: INFO: Node Info: &Node{ObjectMeta:{i-03d301e6f030748f1 897f52bd-65e2-4107-843a-37e36aa74032 63377 0 2023-03-25 05:31:35 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-1 failure-domain.beta.kubernetes.io/zone:us-west-1b kubernetes.io/arch:amd64 kubernetes.io/hostname:i-03d301e6f030748f1 kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:us-west-1b topology.hostpath.csi/node:i-03d301e6f030748f1 topology.kubernetes.io/region:us-west-1 topology.kubernetes.io/zone:us-west-1b] map[csi.volume.kubernetes.io/nodeid:{"csi-mock-csi-mock-volumes-capacity-7853":"i-03d301e6f030748f1","ebs.csi.aws.com":"i-03d301e6f030748f1"} node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:172.20.50.199/19 projectcalico.org/IPv4IPIPTunnelAddr:100.110.167.64 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-03-25 05:31:35 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-25 05:31:35 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-03-25 05:31:35 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2023-03-25 05:31:35 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2023-03-25 05:31:48 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.8.0/24\"":{}}}} } {calico-node Update v1 2023-03-25 05:31:56 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4IPIPTunnelAddr":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2023-03-25 05:54:54 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status} {kube-controller-manager Update v1 2023-03-25 05:54:55 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.8.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-1b/i-03d301e6f030748f1,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.8.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-03-25 05:31:56 +0000 UTC,LastTransitionTime:2023-03-25 05:31:56 +0000 UTC,Reason:CalicoIsUp,Message:Calico is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-25 05:54:54 +0000 UTC,LastTransitionTime:2023-03-25 05:31:35 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-25 05:54:54 +0000 UTC,LastTransitionTime:2023-03-25 05:31:35 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-25 05:54:54 +0000 UTC,LastTransitionTime:2023-03-25 05:31:35 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-25 05:54:54 +0000 UTC,LastTransitionTime:2023-03-25 05:31:48 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.50.199,},NodeAddress{Type:ExternalIP,Address:52.53.245.63,},NodeAddress{Type:InternalDNS,Address:i-03d301e6f030748f1.us-west-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-03d301e6f030748f1.us-west-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-52-53-245-63.us-west-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec27152f7af9b9bfd1355ee19c256086,SystemUUID:ec27152f-7af9-b9bf-d135-5ee19c256086,BootID:1c303852-f2ce-45f9-ab14-6e8de97a0a3e,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.28.0-alpha.0,KubeProxyVersion:v1.28.0-alpha.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:112030336,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:51eae8381dcb1078289fa7b4f3df2630cdc18d09fb56f8e56b41c40e191d6c83 registry.k8s.io/etcd:3.5.7-0],SizeBytes:101639218,},ContainerImage{Names:[docker.io/calico/cni@sha256:e282ea2914c806b5de2976330a17cfb5e6dcef47147bceb1432ca5c75fd46f50 docker.io/calico/cni:v3.24.5],SizeBytes:87458443,},ContainerImage{Names:[docker.io/calico/node@sha256:5972ad2bcbdc668564d3e26960c9c513b2d7b05581c704747cf7c62ef3a405a6 docker.io/calico/node:v3.24.5],SizeBytes:81584866,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nautilus@sha256:80ba6c8c44f9623f06e868a1aa66026c8ec438ad814f9ec95e9333b415fe3550 registry.k8s.io/e2e-test-images/nautilus:1.7],SizeBytes:49641698,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41901587,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:e468dddcd275163a042ab297b2d8c2aca50d5e148d2d22f3b6ba119e2f31fa79 registry.k8s.io/sig-storage/csi-provisioner:v3.4.0],SizeBytes:27427836,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/sample-apiserver@sha256:8d70890151aa5d096f331cb9da1b9cd5be0412b7363fe67b5c3befdcaa2a28d0 registry.k8s.io/e2e-test-images/sample-apiserver:1.17.7],SizeBytes:25667066,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/kube-proxy@sha256:8a03b3147cd7be8e8235ae1cd70cf69aca9ce03ba1cfaffea320540bb5afa496 registry.k8s.io/kube-proxy:v1.28.0-alpha.0],SizeBytes:23896850,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:23881995,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18758628,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18233005,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonroot@sha256:ee9f50b3c64b174d296d91ca9f69a914ac30e59095dfb462b2b518ad28a63655 registry.k8s.io/e2e-test-images/nonroot:1.4],SizeBytes:17747885,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8688564,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:7000082,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6978614,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac registry.k8s.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[kubernetes.io/csi/ebs.csi.aws.com^vol-04e161d418db0c7b1],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-04e161d418db0c7b1,DevicePath:,},},Config:nil,},} Mar 25 05:55:08.153: INFO: Logging kubelet events for node i-03d301e6f030748f1 Mar 25 05:55:08.208: INFO: Logging pods the kubelet thinks is on node i-03d301e6f030748f1 Mar 25 05:55:08.262: INFO: pause-pod-1 started at 2023-03-25 05:54:22 +0000 UTC (0+1 container statuses recorded) Mar 25 05:55:08.262: INFO: Container agnhost-container ready: true, restart count 0 Mar 25 05:55:08.262: INFO: calico-node-h96vq started at 2023-03-25 05:31:35 +0000 UTC (3+1 container statuses recorded) Mar 25 05:55:08.262: INFO: Init container upgrade-ipam ready: true, restart count 0 Mar 25 05:55:08.262: INFO: Init container install-cni ready: true, restart count 0 Mar 25 05:55:08.262: INFO: Init container mount-bpffs ready: true, restart count 0 Mar 25 05:55:08.262: INFO: Container calico-node ready: true, restart count 0 Mar 25 05:55:08.262: INFO: csi-mockplugin-0 started at 2023-03-25 05:54:25 +0000 UTC (0+3 container statuses recorded) Mar 25 05:55:08.262: INFO: Container csi-provisioner ready: true, restart count 0 Mar 25 05:55:08.262: INFO: Container driver-registrar ready: true, restart count 0 Mar 25 05:55:08.262: INFO: Container mock ready: true, restart count 0 Mar 25 05:55:08.262: INFO: test-grpc-cbfe4b71-8b33-40b9-96de-a3f96633a513 started at 2023-03-25 05:53:14 +0000 UTC (0+1 container statuses recorded) Mar 25 05:55:08.262: INFO: Container agnhost ready: true, restart count 0 Mar 25 05:55:08.262: INFO: ss-1 started at 2023-03-25 05:55:07 +0000 UTC (0+1 container statuses recorded) Mar 25 05:55:08.262: INFO: Container webserver ready: false, restart count 0 Mar 25 05:55:08.262: INFO: pod-projected-configmaps-258cfcf3-2804-462b-9867-da2861a96093 started at 2023-03-25 05:52:50 +0000 UTC (0+1 container statuses recorded) Mar 25 05:55:08.262: INFO: Container agnhost-container ready: true, restart count 0 Mar 25 05:55:08.262: INFO: pfpod started at 2023-03-25 05:54:12 +0000 UTC (0+2 container statuses recorded) Mar 25 05:55:08.262: INFO: Container portforwardtester ready: false, restart count 0 Mar 25 05:55:08.262: INFO: Container readiness ready: false, restart count 0 Mar 25 05:55:08.262: INFO: kube-proxy-i-03d301e6f030748f1 started at 2023-03-25 05:31:35 +0000 UTC (0+1 container statuses recorded) Mar 25 05:55:08.262: INFO: Container kube-proxy ready: true, restart count 0 Mar 25 05:55:08.262: INFO: ebs-csi-node-lljwl started at 2023-03-25 05:31:35 +0000 UTC (0+3 container statuses recorded) Mar 25 05:55:08.262: INFO: Container ebs-plugin ready: true, restart count 0 Mar 25 05:55:08.262: INFO: Container liveness-probe ready: true, restart count 0 Mar 25 05:55:08.262: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 25 05:55:08.494: INFO: Latency metrics for node i-03d301e6f030748f1 Mar 25 05:55:08.494: INFO: Logging node info for node i-049fdb2c1cde894df Mar 25 05:55:08.542: INFO: Node Info: &Node{ObjectMeta:{i-049fdb2c1cde894df b43a51e6-dda7-4047-b368-72d515695b4e 63836 0 2023-03-25 05:23:34 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-1 failure-domain.beta.kubernetes.io/zone:us-west-1b io.kubernetes.storage.mock/node:some-mock-node kubernetes.io/arch:amd64 kubernetes.io/hostname:i-049fdb2c1cde894df kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:us-west-1b topology.hostpath.csi/node:i-049fdb2c1cde894df topology.kubernetes.io/region:us-west-1 topology.kubernetes.io/zone:us-west-1b] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-provisioning-7537":"i-049fdb2c1cde894df","csi-hostpath-volume-6086":"i-049fdb2c1cde894df","ebs.csi.aws.com":"i-049fdb2c1cde894df"} node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:172.20.43.61/19 projectcalico.org/IPv4IPIPTunnelAddr:100.97.18.64 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-03-25 05:23:34 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-25 05:23:34 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-03-25 05:23:34 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2023-03-25 05:23:34 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2023-03-25 05:23:45 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.6.0/24\"":{}}}} } {calico-node Update v1 2023-03-25 05:23:52 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4IPIPTunnelAddr":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2023-03-25 05:55:07 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:io.kubernetes.storage.mock/node":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.6.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-1b/i-049fdb2c1cde894df,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.6.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-03-25 05:23:52 +0000 UTC,LastTransitionTime:2023-03-25 05:23:52 +0000 UTC,Reason:CalicoIsUp,Message:Calico is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-25 05:55:03 +0000 UTC,LastTransitionTime:2023-03-25 05:23:34 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-25 05:55:03 +0000 UTC,LastTransitionTime:2023-03-25 05:23:34 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-25 05:55:03 +0000 UTC,LastTransitionTime:2023-03-25 05:23:34 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-25 05:55:03 +0000 UTC,LastTransitionTime:2023-03-25 05:23:45 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.43.61,},NodeAddress{Type:ExternalIP,Address:18.144.75.67,},NodeAddress{Type:InternalDNS,Address:i-049fdb2c1cde894df.us-west-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-049fdb2c1cde894df.us-west-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-18-144-75-67.us-west-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec252e592d19e1e6c583e87505431016,SystemUUID:ec252e59-2d19-e1e6-c583-e87505431016,BootID:45436b5d-1eb0-4b6f-a1d1-a1492b31b5d7,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.28.0-alpha.0,KubeProxyVersion:v1.28.0-alpha.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:112030336,},ContainerImage{Names:[docker.io/calico/cni@sha256:e282ea2914c806b5de2976330a17cfb5e6dcef47147bceb1432ca5c75fd46f50 docker.io/calico/cni:v3.24.5],SizeBytes:87458443,},ContainerImage{Names:[docker.io/calico/node@sha256:5972ad2bcbdc668564d3e26960c9c513b2d7b05581c704747cf7c62ef3a405a6 docker.io/calico/node:v3.24.5],SizeBytes:81584866,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41901587,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:e468dddcd275163a042ab297b2d8c2aca50d5e148d2d22f3b6ba119e2f31fa79 registry.k8s.io/sig-storage/csi-provisioner:v3.4.0],SizeBytes:27427836,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/kube-proxy@sha256:8a03b3147cd7be8e8235ae1cd70cf69aca9ce03ba1cfaffea320540bb5afa496 registry.k8s.io/kube-proxy:v1.28.0-alpha.0],SizeBytes:23896850,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:23881995,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:68d396900aeaa072c1f27289485fdac29834045a6f3ffe369bf389d830ef572d registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.6],SizeBytes:20293261,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18758628,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18233005,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:15273057,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8688564,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:7000082,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6978614,},ContainerImage{Names:[gcr.io/authenticated-image-pulling/alpine@sha256:7ff177862cb50c602bfe81f805969412e619c054a2bbead977d0c276988aa4a0 gcr.io/authenticated-image-pulling/alpine:3.7],SizeBytes:2110879,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Mar 25 05:55:08.543: INFO: Logging kubelet events for node i-049fdb2c1cde894df Mar 25 05:55:08.606: INFO: Logging pods the kubelet thinks is on node i-049fdb2c1cde894df Mar 25 05:55:08.662: INFO: calico-node-wt5p7 started at 2023-03-25 05:23:35 +0000 UTC (3+1 container statuses recorded) Mar 25 05:55:08.662: INFO: Init container upgrade-ipam ready: true, restart count 0 Mar 25 05:55:08.662: INFO: Init container install-cni ready: true, restart count 0 Mar 25 05:55:08.662: INFO: Init container mount-bpffs ready: true, restart count 0 Mar 25 05:55:08.662: INFO: Container calico-node ready: true, restart count 0 Mar 25 05:55:08.662: INFO: csi-hostpathplugin-0 started at 2023-03-25 05:54:14 +0000 UTC (0+7 container statuses recorded) Mar 25 05:55:08.662: INFO: Container csi-attacher ready: true, restart count 0 Mar 25 05:55:08.662: INFO: Container csi-provisioner ready: true, restart count 0 Mar 25 05:55:08.662: INFO: Container csi-resizer ready: true, restart count 0 Mar 25 05:55:08.662: INFO: Container csi-snapshotter ready: true, restart count 0 Mar 25 05:55:08.662: INFO: Container hostpath ready: true, restart count 0 Mar 25 05:55:08.662: INFO: Container liveness-probe ready: true, restart count 0 Mar 25 05:55:08.662: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 25 05:55:08.662: INFO: ebs-csi-node-m9zxz started at 2023-03-25 05:23:35 +0000 UTC (0+3 container statuses recorded) Mar 25 05:55:08.662: INFO: Container ebs-plugin ready: true, restart count 0 Mar 25 05:55:08.662: INFO: Container liveness-probe ready: true, restart count 0 Mar 25 05:55:08.662: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 25 05:55:08.662: INFO: coredns-559769c974-pgqhl started at 2023-03-25 05:24:36 +0000 UTC (0+1 container statuses recorded) Mar 25 05:55:08.662: INFO: Container coredns ready: true, restart count 0 Mar 25 05:55:08.662: INFO: kube-proxy-i-049fdb2c1cde894df started at 2023-03-25 05:23:35 +0000 UTC (0+1 container statuses recorded) Mar 25 05:55:08.662: INFO: Container kube-proxy ready: true, restart count 0 Mar 25 05:55:08.662: INFO: coredns-autoscaler-7cb5c5b969-h485x started at 2023-03-25 05:24:36 +0000 UTC (0+1 container statuses recorded) Mar 25 05:55:08.662: INFO: Container autoscaler ready: true, restart count 0 Mar 25 05:55:08.662: INFO: csi-hostpathplugin-0 started at 2023-03-25 05:54:39 +0000 UTC (0+7 container statuses recorded) Mar 25 05:55:08.662: INFO: Container csi-attacher ready: true, restart count 0 Mar 25 05:55:08.662: INFO: Container csi-provisioner ready: true, restart count 0 Mar 25 05:55:08.662: INFO: Container csi-resizer ready: true, restart count 0 Mar 25 05:55:08.662: INFO: Container csi-snapshotter ready: true, restart count 0 Mar 25 05:55:08.662: INFO: Container hostpath ready: true, restart count 0 Mar 25 05:55:08.662: INFO: Container liveness-probe ready: true, restart count 0 Mar 25 05:55:08.662: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 25 05:55:08.938: INFO: Latency metrics for node i-049fdb2c1cde894df Mar 25 05:55:08.938: INFO: Logging node info for node i-062679bb8c0ba4bce Mar 25 05:55:08.987: INFO: Node Info: &Node{ObjectMeta:{i-062679bb8c0ba4bce ee2da569-bb88-4297-9e6f-d2b15ba7ce86 63707 0 2023-03-25 05:27:31 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-1 failure-domain.beta.kubernetes.io/zone:us-west-1b kubernetes.io/arch:amd64 kubernetes.io/hostname:i-062679bb8c0ba4bce kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:us-west-1b topology.hostpath.csi/node:i-062679bb8c0ba4bce topology.kubernetes.io/region:us-west-1 topology.kubernetes.io/zone:us-west-1b] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-062679bb8c0ba4bce"} node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:172.20.45.43/19 projectcalico.org/IPv4IPIPTunnelAddr:100.106.79.0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-03-25 05:27:31 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-25 05:27:31 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-03-25 05:27:31 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2023-03-25 05:27:31 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2023-03-25 05:27:44 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.7.0/24\"":{}}}} } {calico-node Update v1 2023-03-25 05:27:50 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4IPIPTunnelAddr":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2023-03-25 05:55:03 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status} {kube-controller-manager Update v1 2023-03-25 05:55:05 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.7.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-1b/i-062679bb8c0ba4bce,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.7.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051689472 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946831872 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-03-25 05:27:50 +0000 UTC,LastTransitionTime:2023-03-25 05:27:50 +0000 UTC,Reason:CalicoIsUp,Message:Calico is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-25 05:55:03 +0000 UTC,LastTransitionTime:2023-03-25 05:27:31 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-25 05:55:03 +0000 UTC,LastTransitionTime:2023-03-25 05:27:31 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-25 05:55:03 +0000 UTC,LastTransitionTime:2023-03-25 05:27:31 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-25 05:55:03 +0000 UTC,LastTransitionTime:2023-03-25 05:27:44 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.45.43,},NodeAddress{Type:ExternalIP,Address:3.101.133.108,},NodeAddress{Type:InternalDNS,Address:i-062679bb8c0ba4bce.us-west-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-062679bb8c0ba4bce.us-west-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-3-101-133-108.us-west-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2528f599e9b1e199793fb5cd06d205,SystemUUID:ec2528f5-99e9-b1e1-9979-3fb5cd06d205,BootID:ff56b70b-c8c5-46db-8cc5-4353c8f535c6,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.28.0-alpha.0,KubeProxyVersion:v1.28.0-alpha.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:112030336,},ContainerImage{Names:[docker.io/calico/cni@sha256:e282ea2914c806b5de2976330a17cfb5e6dcef47147bceb1432ca5c75fd46f50 docker.io/calico/cni:v3.24.5],SizeBytes:87458443,},ContainerImage{Names:[docker.io/calico/node@sha256:5972ad2bcbdc668564d3e26960c9c513b2d7b05581c704747cf7c62ef3a405a6 docker.io/calico/node:v3.24.5],SizeBytes:81584866,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nautilus@sha256:80ba6c8c44f9623f06e868a1aa66026c8ec438ad814f9ec95e9333b415fe3550 registry.k8s.io/e2e-test-images/nautilus:1.7],SizeBytes:49641698,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41901587,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:e468dddcd275163a042ab297b2d8c2aca50d5e148d2d22f3b6ba119e2f31fa79 registry.k8s.io/sig-storage/csi-provisioner:v3.4.0],SizeBytes:27427836,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/kube-proxy@sha256:8a03b3147cd7be8e8235ae1cd70cf69aca9ce03ba1cfaffea320540bb5afa496 registry.k8s.io/kube-proxy:v1.28.0-alpha.0],SizeBytes:23896850,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:23881995,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18758628,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/regression-issue-74839@sha256:b4f1d8d61bdad84bd50442d161d5460e4019d53e989b64220fdbc62fc87d76bf registry.k8s.io/e2e-test-images/regression-issue-74839:1.2],SizeBytes:18651485,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18233005,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:15273057,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8688564,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/build-image/distroless-iptables@sha256:13ac9b3c476d7290a4451d65d09d6016a2cb89836ffbfa4eae55b72731a22080 registry.k8s.io/build-image/distroless-iptables:v0.2.2],SizeBytes:7729580,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:7000082,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6978614,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[kubernetes.io/csi/ebs.csi.aws.com^vol-0b0a70539f44dbc59 kubernetes.io/csi/ebs.csi.aws.com^vol-0ffe739c83a1d20d7],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-0ffe739c83a1d20d7,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-0b0a70539f44dbc59,DevicePath:,},},Config:nil,},} Mar 25 05:55:08.987: INFO: Logging kubelet events for node i-062679bb8c0ba4bce E0325 05:55:09.029530 7722 retrywatcher.go:130] "Watch failed" err="context canceled" Mar 25 05:55:09.047: INFO: Logging pods the kubelet thinks is on node i-062679bb8c0ba4bce Mar 25 05:55:09.104: INFO: coredns-559769c974-p9vg6 started at 2023-03-25 05:36:16 +0000 UTC (0+1 container statuses recorded) Mar 25 05:55:09.104: INFO: Container coredns ready: true, restart count 0 Mar 25 05:55:09.104: INFO: pfpod started at 2023-03-25 05:54:00 +0000 UTC (0+2 container statuses recorded) Mar 25 05:55:09.104: INFO: Container portforwardtester ready: false, restart count 0 Mar 25 05:55:09.104: INFO: Container readiness ready: false, restart count 0 Mar 25 05:55:09.104: INFO: pod-terminate-status-1-9 started at 2023-03-25 05:55:06 +0000 UTC (0+1 container statuses recorded) Mar 25 05:55:09.104: INFO: Container fail ready: false, restart count 0 Mar 25 05:55:09.104: INFO: ebs-csi-node-7twrq started at 2023-03-25 05:27:32 +0000 UTC (0+3 container statuses recorded) Mar 25 05:55:09.104: INFO: Container ebs-plugin ready: true, restart count 0 Mar 25 05:55:09.104: INFO: Container liveness-probe ready: true, restart count 0 Mar 25 05:55:09.104: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 25 05:55:09.104: INFO: pod-terminate-status-2-9 started at 2023-03-25 05:55:06 +0000 UTC (0+1 container statuses recorded) Mar 25 05:55:09.104: INFO: Container fail ready: false, restart count 0 Mar 25 05:55:09.104: INFO: pod-terminate-status-0-8 started at 2023-03-25 05:55:06 +0000 UTC (0+1 container statuses recorded) Mar 25 05:55:09.104: INFO: Container fail ready: false, restart count 0 Mar 25 05:55:09.104: INFO: ss-2 started at 2023-03-25 05:54:54 +0000 UTC (0+1 container statuses recorded) Mar 25 05:55:09.104: INFO: Container webserver ready: true, restart count 0 Mar 25 05:55:09.104: INFO: ss-0 started at 2023-03-25 05:54:04 +0000 UTC (0+1 container statuses recorded) Mar 25 05:55:09.104: INFO: Container webserver ready: true, restart count 0 Mar 25 05:55:09.104: INFO: kube-proxy-i-062679bb8c0ba4bce started at 2023-03-25 05:27:32 +0000 UTC (0+1 container statuses recorded) Mar 25 05:55:09.104: INFO: Container kube-proxy ready: true, restart count 0 Mar 25 05:55:09.104: INFO: calico-node-zwm5j started at 2023-03-25 05:27:32 +0000 UTC (3+1 container statuses recorded) Mar 25 05:55:09.104: INFO: Init container upgrade-ipam ready: true, restart count 0 Mar 25 05:55:09.104: INFO: Init container install-cni ready: true, restart count 0 Mar 25 05:55:09.104: INFO: Init container mount-bpffs ready: true, restart count 0 Mar 25 05:55:09.104: INFO: Container calico-node ready: true, restart count 0 Mar 25 05:55:09.364: INFO: Latency metrics for node i-062679bb8c0ba4bce Mar 25 05:55:09.364: INFO: Logging node info for node i-072608201d2c0f1ad Mar 25 05:55:09.412: INFO: Node Info: &Node{ObjectMeta:{i-072608201d2c0f1ad 264dfba1-3cf0-4c99-8e8c-098f0f792077 63701 0 2023-03-25 05:35:29 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-1 failure-domain.beta.kubernetes.io/zone:us-west-1b kubernetes.io/arch:amd64 kubernetes.io/hostname:i-072608201d2c0f1ad kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:us-west-1b topology.hostpath.csi/node:i-072608201d2c0f1ad topology.kubernetes.io/region:us-west-1 topology.kubernetes.io/zone:us-west-1b] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-7817":"i-072608201d2c0f1ad","ebs.csi.aws.com":"i-072608201d2c0f1ad"} node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:172.20.38.125/19 projectcalico.org/IPv4IPIPTunnelAddr:100.107.220.192 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-03-25 05:35:29 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-25 05:35:29 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-03-25 05:35:29 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2023-03-25 05:35:29 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2023-03-25 05:35:41 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.9.0/24\"":{}}}} } {calico-node Update v1 2023-03-25 05:35:48 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4IPIPTunnelAddr":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2023-03-25 05:54:55 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-03-25 05:55:05 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.9.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-1b/i-072608201d2c0f1ad,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.9.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051689472 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946831872 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-03-25 05:35:47 +0000 UTC,LastTransitionTime:2023-03-25 05:35:47 +0000 UTC,Reason:CalicoIsUp,Message:Calico is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-25 05:54:55 +0000 UTC,LastTransitionTime:2023-03-25 05:35:29 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-25 05:54:55 +0000 UTC,LastTransitionTime:2023-03-25 05:35:29 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-25 05:54:55 +0000 UTC,LastTransitionTime:2023-03-25 05:35:29 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-25 05:54:55 +0000 UTC,LastTransitionTime:2023-03-25 05:35:41 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.38.125,},NodeAddress{Type:ExternalIP,Address:3.101.88.10,},NodeAddress{Type:InternalDNS,Address:i-072608201d2c0f1ad.us-west-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-072608201d2c0f1ad.us-west-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-3-101-88-10.us-west-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2d3818c4f80a253dcc1a60e148dbc8,SystemUUID:ec2d3818-c4f8-0a25-3dcc-1a60e148dbc8,BootID:77d046cf-75ea-4a8c-970a-3190d64e019b,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.28.0-alpha.0,KubeProxyVersion:v1.28.0-alpha.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:112030336,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:51eae8381dcb1078289fa7b4f3df2630cdc18d09fb56f8e56b41c40e191d6c83 registry.k8s.io/etcd:3.5.7-0],SizeBytes:101639218,},ContainerImage{Names:[docker.io/calico/cni@sha256:e282ea2914c806b5de2976330a17cfb5e6dcef47147bceb1432ca5c75fd46f50 docker.io/calico/cni:v3.24.5],SizeBytes:87458443,},ContainerImage{Names:[docker.io/calico/node@sha256:5972ad2bcbdc668564d3e26960c9c513b2d7b05581c704747cf7c62ef3a405a6 docker.io/calico/node:v3.24.5],SizeBytes:81584866,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nautilus@sha256:80ba6c8c44f9623f06e868a1aa66026c8ec438ad814f9ec95e9333b415fe3550 registry.k8s.io/e2e-test-images/nautilus:1.7],SizeBytes:49641698,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41901587,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:e468dddcd275163a042ab297b2d8c2aca50d5e148d2d22f3b6ba119e2f31fa79 registry.k8s.io/sig-storage/csi-provisioner:v3.4.0],SizeBytes:27427836,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/sample-apiserver@sha256:8d70890151aa5d096f331cb9da1b9cd5be0412b7363fe67b5c3befdcaa2a28d0 registry.k8s.io/e2e-test-images/sample-apiserver:1.17.7],SizeBytes:25667066,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/kube-proxy@sha256:8a03b3147cd7be8e8235ae1cd70cf69aca9ce03ba1cfaffea320540bb5afa496 registry.k8s.io/kube-proxy:v1.28.0-alpha.0],SizeBytes:23896850,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:23881995,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18758628,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18233005,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonroot@sha256:ee9f50b3c64b174d296d91ca9f69a914ac30e59095dfb462b2b518ad28a63655 registry.k8s.io/e2e-test-images/nonroot:1.4],SizeBytes:17747885,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8688564,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:7000082,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6978614,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[kubernetes.io/csi/csi-hostpath-ephemeral-7817^4b20e30d-cad1-11ed-98df-264f88168798],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-7817^4b20e30d-cad1-11ed-98df-264f88168798,DevicePath:,},},Config:nil,},} Mar 25 05:55:09.413: INFO: Logging kubelet events for node i-072608201d2c0f1ad Mar 25 05:55:09.471: INFO: Logging pods the kubelet thinks is on node i-072608201d2c0f1ad Mar 25 05:55:09.524: INFO: calico-node-lc9zl started at 2023-03-25 05:35:30 +0000 UTC (3+1 container statuses recorded) Mar 25 05:55:09.524: INFO: Init container upgrade-ipam ready: true, restart count 0 Mar 25 05:55:09.524: INFO: Init container install-cni ready: true, restart count 0 Mar 25 05:55:09.524: INFO: Init container mount-bpffs ready: true, restart count 0 Mar 25 05:55:09.524: INFO: Container calico-node ready: true, restart count 0 Mar 25 05:55:09.524: INFO: kube-proxy-i-072608201d2c0f1ad started at 2023-03-25 05:35:30 +0000 UTC (0+1 container statuses recorded) Mar 25 05:55:09.524: INFO: Container kube-proxy ready: true, restart count 0 Mar 25 05:55:09.524: INFO: pod-secrets-c75f6630-caa4-40a1-b97c-358088342209 started at 2023-03-25 05:53:01 +0000 UTC (0+3 container statuses recorded) Mar 25 05:55:09.524: INFO: Container creates-volume-test ready: true, restart count 0 Mar 25 05:55:09.524: INFO: Container dels-volume-test ready: true, restart count 0 Mar 25 05:55:09.524: INFO: Container upds-volume-test ready: true, restart count 0 Mar 25 05:55:09.524: INFO: ebs-csi-node-r7mcm started at 2023-03-25 05:35:30 +0000 UTC (0+3 container statuses recorded) Mar 25 05:55:09.524: INFO: Container ebs-plugin ready: true, restart count 0 Mar 25 05:55:09.524: INFO: Container liveness-probe ready: true, restart count 0 Mar 25 05:55:09.524: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 25 05:55:09.524: INFO: csi-hostpathplugin-0 started at 2023-03-25 05:52:55 +0000 UTC (0+7 container statuses recorded) Mar 25 05:55:09.524: INFO: Container csi-attacher ready: true, restart count 0 Mar 25 05:55:09.524: INFO: Container csi-provisioner ready: true, restart count 0 Mar 25 05:55:09.524: INFO: Container csi-resizer ready: true, restart count 0 Mar 25 05:55:09.524: INFO: Container csi-snapshotter ready: true, restart count 0 Mar 25 05:55:09.524: INFO: Container hostpath ready: true, restart count 0 Mar 25 05:55:09.524: INFO: Container liveness-probe ready: true, restart count 0 Mar 25 05:55:09.524: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 25 05:55:09.802: INFO: Latency metrics for node i-072608201d2c0f1ad END STEP: dump namespace information after failure - test/e2e/framework/framework.go:288 @ 03/25/23 05:55:09.802 (2.52s) < Exit [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] subPath - dump namespaces | framework.go:209 @ 03/25/23 05:55:09.802 (2.52s) > Enter [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] subPath - tear down framework | framework.go:206 @ 03/25/23 05:55:09.802 STEP: Destroying namespace "provisioning-4160" for this suite. - test/e2e/framework/framework.go:351 @ 03/25/23 05:55:09.802 < Exit [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] subPath - tear down framework | framework.go:206 @ 03/25/23 05:55:09.853 (51ms) > Enter [ReportAfterEach] TOP-LEVEL - test/e2e/e2e_test.go:144 @ 03/25/23 05:55:09.853 < Exit [ReportAfterEach] TOP-LEVEL - test/e2e/e2e_test.go:144 @ 03/25/23 05:55:09.853 (0s)
Filter through log files | View test history on testgrid
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[It\]\sExternal\sStorage\s\[Driver\:\sebs\.csi\.aws\.com\]\s\[Testpattern\:\sDynamic\sPV\s\(default\sfs\)\]\ssubPath\sshould\ssupport\sexisting\ssingle\sfile\s\[LinuxOnly\]$'
[FAILED] failed to get logs from pod-subpath-test-dynamicpv-8296 for test-container-subpath-dynamicpv-8296: Get "https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/provisioning-6241/pods/pod-subpath-test-dynamicpv-8296/log?container=test-container-subpath-dynamicpv-8296&previous=false": stream error: stream ID 599; INTERNAL_ERROR; received from peer In [It] at: test/e2e/framework/pod/output/output.go:247 @ 03/25/23 05:39:29.607from junit_01.xml
> Enter [BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/storage/framework/testsuite.go:51 @ 03/25/23 05:39:10.261 < Exit [BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/storage/framework/testsuite.go:51 @ 03/25/23 05:39:10.261 (0s) > Enter [BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath - set up framework | framework.go:191 @ 03/25/23 05:39:10.261 STEP: Creating a kubernetes client - test/e2e/framework/framework.go:211 @ 03/25/23 05:39:10.261 Mar 25 05:39:10.261: INFO: >>> kubeConfig: /root/.kube/config STEP: Building a namespace api object, basename provisioning - test/e2e/framework/framework.go:250 @ 03/25/23 05:39:10.262 STEP: Waiting for a default service account to be provisioned in namespace - test/e2e/framework/framework.go:259 @ 03/25/23 05:39:10.411 STEP: Waiting for kube-root-ca.crt to be provisioned in namespace - test/e2e/framework/framework.go:262 @ 03/25/23 05:39:10.508 < Exit [BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath - set up framework | framework.go:191 @ 03/25/23 05:39:10.606 (345ms) > Enter [BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/framework/metrics/init/init.go:33 @ 03/25/23 05:39:10.606 < Exit [BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/framework/metrics/init/init.go:33 @ 03/25/23 05:39:10.606 (0s) > Enter [It] should support existing single file [LinuxOnly] - test/e2e/storage/testsuites/subpath.go:219 @ 03/25/23 05:39:10.606 Mar 25 05:39:10.606: INFO: Creating resource for dynamic PV Mar 25 05:39:10.606: INFO: Using claimSize:1Gi, test suite supported size:{ 1Mi}, driver(ebs.csi.aws.com) supported size:{ 1Mi} STEP: creating a StorageClass provisioning-6241-e2e-schsh94 - test/e2e/storage/framework/volume_resource.go:102 @ 03/25/23 05:39:10.606 STEP: creating a claim - test/e2e/storage/framework/volume_resource.go:294 @ 03/25/23 05:39:10.66 Mar 25 05:39:10.660: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil STEP: Creating pod pod-subpath-test-dynamicpv-8296 - test/e2e/storage/testsuites/subpath.go:691 @ 03/25/23 05:39:10.784 STEP: Creating a pod to test subpath - test/e2e/framework/pod/output/output.go:243 @ 03/25/23 05:39:10.784 STEP: Saw pod success - test/e2e/framework/pod/wait.go:409 @ 03/25/23 05:39:29.392 Mar 25 05:39:29.448: INFO: Trying to get logs from node i-049fdb2c1cde894df pod pod-subpath-test-dynamicpv-8296 container test-container-subpath-dynamicpv-8296: <nil> Mar 25 05:39:29.496: INFO: Failed to get logs from node "i-049fdb2c1cde894df" pod "pod-subpath-test-dynamicpv-8296" container "test-container-subpath-dynamicpv-8296". Get "https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/provisioning-6241/pods/pod-subpath-test-dynamicpv-8296/log?container=test-container-subpath-dynamicpv-8296&previous=false": stream error: stream ID 599; INTERNAL_ERROR; received from peer STEP: delete the pod - test/e2e/framework/pod/output/output.go:168 @ 03/25/23 05:39:29.497 Mar 25 05:39:29.606: INFO: Unexpected error: <*fmt.wrapError | 0xc00459f4a0>: failed to get logs from pod-subpath-test-dynamicpv-8296 for test-container-subpath-dynamicpv-8296: Get "https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/provisioning-6241/pods/pod-subpath-test-dynamicpv-8296/log?container=test-container-subpath-dynamicpv-8296&previous=false": stream error: stream ID 599; INTERNAL_ERROR; received from peer { msg: "failed to get logs from pod-subpath-test-dynamicpv-8296 for test-container-subpath-dynamicpv-8296: Get \"https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/provisioning-6241/pods/pod-subpath-test-dynamicpv-8296/log?container=test-container-subpath-dynamicpv-8296&previous=false\": stream error: stream ID 599; INTERNAL_ERROR; received from peer", err: <*url.Error | 0xc0037360c0>{ Op: "Get", URL: "https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/provisioning-6241/pods/pod-subpath-test-dynamicpv-8296/log?container=test-container-subpath-dynamicpv-8296&previous=false", Err: <http2.StreamError>{ StreamID: 599, Code: 2, Cause: <*errors.errorString | 0xc0001b5670>{ s: "received from peer", }, }, }, } [FAILED] failed to get logs from pod-subpath-test-dynamicpv-8296 for test-container-subpath-dynamicpv-8296: Get "https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/provisioning-6241/pods/pod-subpath-test-dynamicpv-8296/log?container=test-container-subpath-dynamicpv-8296&previous=false": stream error: stream ID 599; INTERNAL_ERROR; received from peer In [It] at: test/e2e/framework/pod/output/output.go:247 @ 03/25/23 05:39:29.607 < Exit [It] should support existing single file [LinuxOnly] - test/e2e/storage/testsuites/subpath.go:219 @ 03/25/23 05:39:29.607 (19.001s) > Enter [AfterEach] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/framework/node/init/init.go:33 @ 03/25/23 05:39:29.607 Mar 25 05:39:29.607: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready < Exit [AfterEach] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/framework/node/init/init.go:33 @ 03/25/23 05:39:29.656 (49ms) > Enter [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/storage/testsuites/subpath.go:221 @ 03/25/23 05:39:29.656 STEP: Deleting pod - test/e2e/storage/testsuites/subpath.go:172 @ 03/25/23 05:39:29.656 Mar 25 05:39:29.656: INFO: Deleting pod "pod-subpath-test-dynamicpv-8296" in namespace "provisioning-6241" STEP: Deleting pvc - test/e2e/storage/framework/volume_resource.go:181 @ 03/25/23 05:39:29.755 Mar 25 05:39:29.892: INFO: Deleting PersistentVolumeClaim "ebs.csi.aws.comm94jr" Mar 25 05:39:29.954: INFO: Waiting up to 5m0s for PersistentVolume pvc-829695f8-cf99-4188-84f0-868ecba75407 to get deleted Mar 25 05:39:30.002: INFO: PersistentVolume pvc-829695f8-cf99-4188-84f0-868ecba75407 found and phase=Released (48.252581ms) Mar 25 05:39:35.051: INFO: PersistentVolume pvc-829695f8-cf99-4188-84f0-868ecba75407 found and phase=Released (5.097313063s) Mar 25 05:39:40.104: INFO: PersistentVolume pvc-829695f8-cf99-4188-84f0-868ecba75407 found and phase=Released (10.150343465s) Mar 25 05:39:45.154: INFO: PersistentVolume pvc-829695f8-cf99-4188-84f0-868ecba75407 found and phase=Released (15.199700502s) Mar 25 05:39:50.217: INFO: PersistentVolume pvc-829695f8-cf99-4188-84f0-868ecba75407 was removed STEP: Deleting sc - test/e2e/storage/framework/volume_resource.go:228 @ 03/25/23 05:39:50.217 < Exit [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/storage/testsuites/subpath.go:221 @ 03/25/23 05:39:50.27 (20.614s) > Enter [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/framework/metrics/init/init.go:35 @ 03/25/23 05:39:50.27 < Exit [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/framework/metrics/init/init.go:35 @ 03/25/23 05:39:50.27 (0s) > Enter [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] subPath - dump namespaces | framework.go:209 @ 03/25/23 05:39:50.27 STEP: dump namespace information after failure - test/e2e/framework/framework.go:288 @ 03/25/23 05:39:50.27 STEP: Collecting events from namespace "provisioning-6241". - test/e2e/framework/debug/dump.go:42 @ 03/25/23 05:39:50.27 STEP: Found 12 events. - test/e2e/framework/debug/dump.go:46 @ 03/25/23 05:39:50.325 Mar 25 05:39:50.325: INFO: At 2023-03-25 05:39:10 +0000 UTC - event for ebs.csi.aws.comm94jr: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding Mar 25 05:39:50.325: INFO: At 2023-03-25 05:39:10 +0000 UTC - event for ebs.csi.aws.comm94jr: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator Mar 25 05:39:50.325: INFO: At 2023-03-25 05:39:10 +0000 UTC - event for ebs.csi.aws.comm94jr: {ebs.csi.aws.com_i-0168f82458996cc53_0b589c46-3396-4499-94ba-eb51a48f7a58 } Provisioning: External provisioner is provisioning volume for claim "provisioning-6241/ebs.csi.aws.comm94jr" Mar 25 05:39:50.325: INFO: At 2023-03-25 05:39:14 +0000 UTC - event for ebs.csi.aws.comm94jr: {ebs.csi.aws.com_i-0168f82458996cc53_0b589c46-3396-4499-94ba-eb51a48f7a58 } ProvisioningSucceeded: Successfully provisioned volume pvc-829695f8-cf99-4188-84f0-868ecba75407 Mar 25 05:39:50.325: INFO: At 2023-03-25 05:39:14 +0000 UTC - event for pod-subpath-test-dynamicpv-8296: {default-scheduler } Scheduled: Successfully assigned provisioning-6241/pod-subpath-test-dynamicpv-8296 to i-049fdb2c1cde894df Mar 25 05:39:50.325: INFO: At 2023-03-25 05:39:17 +0000 UTC - event for pod-subpath-test-dynamicpv-8296: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-829695f8-cf99-4188-84f0-868ecba75407" Mar 25 05:39:50.325: INFO: At 2023-03-25 05:39:24 +0000 UTC - event for pod-subpath-test-dynamicpv-8296: {kubelet i-049fdb2c1cde894df} Pulled: Container image "registry.k8s.io/e2e-test-images/busybox:1.29-4" already present on machine Mar 25 05:39:50.325: INFO: At 2023-03-25 05:39:24 +0000 UTC - event for pod-subpath-test-dynamicpv-8296: {kubelet i-049fdb2c1cde894df} Created: Created container init-volume-dynamicpv-8296 Mar 25 05:39:50.325: INFO: At 2023-03-25 05:39:24 +0000 UTC - event for pod-subpath-test-dynamicpv-8296: {kubelet i-049fdb2c1cde894df} Started: Started container init-volume-dynamicpv-8296 Mar 25 05:39:50.325: INFO: At 2023-03-25 05:39:25 +0000 UTC - event for pod-subpath-test-dynamicpv-8296: {kubelet i-049fdb2c1cde894df} Pulled: Container image "registry.k8s.io/e2e-test-images/agnhost:2.43" already present on machine Mar 25 05:39:50.325: INFO: At 2023-03-25 05:39:25 +0000 UTC - event for pod-subpath-test-dynamicpv-8296: {kubelet i-049fdb2c1cde894df} Created: Created container test-container-subpath-dynamicpv-8296 Mar 25 05:39:50.325: INFO: At 2023-03-25 05:39:25 +0000 UTC - event for pod-subpath-test-dynamicpv-8296: {kubelet i-049fdb2c1cde894df} Started: Started container test-container-subpath-dynamicpv-8296 Mar 25 05:39:50.377: INFO: POD NODE PHASE GRACE CONDITIONS Mar 25 05:39:50.377: INFO: Mar 25 05:39:50.444: INFO: Logging node info for node i-0168f82458996cc53 Mar 25 05:39:50.509: INFO: Node Info: &Node{ObjectMeta:{i-0168f82458996cc53 f4eb302c-c477-45ac-bcac-ffed65f6580f 5776 0 2023-03-25 05:18:57 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-1 failure-domain.beta.kubernetes.io/zone:us-west-1b kops.k8s.io/kops-controller-pki: kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0168f82458996cc53 kubernetes.io/os:linux node-role.kubernetes.io/control-plane: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:us-west-1b topology.kubernetes.io/region:us-west-1 topology.kubernetes.io/zone:us-west-1b] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0168f82458996cc53"} node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:172.20.48.81/19 projectcalico.org/IPv4IPIPTunnelAddr:100.100.215.192 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-03-25 05:18:57 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {protokube Update v1 2023-03-25 05:18:58 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {calico-node Update v1 2023-03-25 05:20:22 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4IPIPTunnelAddr":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {aws-cloud-controller-manager Update v1 2023-03-25 05:20:26 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-25 05:20:26 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2023-03-25 05:20:31 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.5.0/24\"":{}},"f:taints":{}}} } {kubelet Update v1 2023-03-25 05:36:59 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.5.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-1b/i-0168f82458996cc53,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/control-plane,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.5.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3892310016 0} {<nil>} 3801084Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3787452416 0} {<nil>} 3698684Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-03-25 05:20:22 +0000 UTC,LastTransitionTime:2023-03-25 05:20:22 +0000 UTC,Reason:CalicoIsUp,Message:Calico is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-25 05:36:59 +0000 UTC,LastTransitionTime:2023-03-25 05:18:50 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-25 05:36:59 +0000 UTC,LastTransitionTime:2023-03-25 05:18:50 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-25 05:36:59 +0000 UTC,LastTransitionTime:2023-03-25 05:18:50 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-25 05:36:59 +0000 UTC,LastTransitionTime:2023-03-25 05:20:29 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.48.81,},NodeAddress{Type:ExternalIP,Address:3.101.135.77,},NodeAddress{Type:InternalDNS,Address:i-0168f82458996cc53.us-west-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-0168f82458996cc53.us-west-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-3-101-135-77.us-west-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2d1ccc7ebf313420bd1f530a94fbfd,SystemUUID:ec2d1ccc-7ebf-3134-20bd-1f530a94fbfd,BootID:15cee321-56e7-4290-ab48-47b60b0bc657,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.28.0-alpha.0,KubeProxyVersion:v1.28.0-alpha.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcd@sha256:dd75ec974b0a2a6f6bb47001ba09207976e625db898d1b16735528c009cb171c registry.k8s.io/etcd:3.5.6-0],SizeBytes:102542580,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:6f72b851544986cb0921b53ea655ec04c36131248f16d4ad110cb3ca0c369dc1 registry.k8s.io/etcd:3.5.4-0],SizeBytes:102157811,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:13f53ed1d91e2e11aac476ee9a0269fdda6cc4874eba903efd40daf50c55eee5 registry.k8s.io/etcd:3.5.3-0],SizeBytes:102143581,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:51eae8381dcb1078289fa7b4f3df2630cdc18d09fb56f8e56b41c40e191d6c83 registry.k8s.io/etcd:3.5.7-0],SizeBytes:101639218,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:4afb99b4690b418ffc2ceb67e1a17376457e441c1f09ab55447f0aaf992fa646 registry.k8s.io/etcd:3.4.3-0],SizeBytes:100947667,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:9ce33ba33d8e738a5b85ed50b5080ac746deceed4a7496c550927a7a19ca3b6d registry.k8s.io/etcd:3.5.0-0],SizeBytes:99868722,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:64b9ea357325d5db9f8a723dcf503b5a449177b17ac87d69481e126bb724c263 registry.k8s.io/etcd:3.5.1-0],SizeBytes:98888614,},ContainerImage{Names:[docker.io/calico/cni@sha256:e282ea2914c806b5de2976330a17cfb5e6dcef47147bceb1432ca5c75fd46f50 docker.io/calico/cni:v3.24.5],SizeBytes:87458443,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:4ad90a11b55313b182afc186b9876c8e891531b8db4c9bf1541953021618d0e2 registry.k8s.io/etcd:3.4.13-0],SizeBytes:86742272,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:0d2ab6b15b4fe0b903b9e81192d942fb39c29be61fe341cd16db5d0165b0435c registry.k8s.io/etcd:3.3.17-0],SizeBytes:85524408,},ContainerImage{Names:[docker.io/calico/node@sha256:5972ad2bcbdc668564d3e26960c9c513b2d7b05581c704747cf7c62ef3a405a6 docker.io/calico/node:v3.24.5],SizeBytes:81584866,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:17da501f5d2a675be46040422a27b7cc21b8a43895ac998b171db1c346f361f7 registry.k8s.io/etcd:3.3.10-0],SizeBytes:76162358,},ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager-slim@sha256:8727adc059334e0e3329dbccce116a734da4a090da5e86c8cddccaf333caa666 registry.k8s.io/etcdadm/etcd-manager-slim:v3.0.20230201],SizeBytes:63926165,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:905d7ca17fd02bc24c0eba9a062753aba15db3e31422390bc3238eb762339b20 registry.k8s.io/etcd:3.2.24-1],SizeBytes:63423287,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.27.0-alpha.1],SizeBytes:45219545,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.27.0-alpha.1],SizeBytes:43628585,},ContainerImage{Names:[registry.k8s.io/kube-apiserver@sha256:49934a29c559e79ed19fc38b4697b18d8fe174b17bf56192588575f980c93956 registry.k8s.io/kube-apiserver:v1.28.0-alpha.0],SizeBytes:33212506,},ContainerImage{Names:[docker.io/calico/kube-controllers@sha256:2b6acd7f677f76ffe12ecf3ea7df92eb9b1bdb07336d1ac2a54c7631fb753f7e docker.io/calico/kube-controllers:v3.24.5],SizeBytes:31137169,},ContainerImage{Names:[registry.k8s.io/kube-controller-manager@sha256:974748384634982c73527fdb3c9ec0ec476bc0d52b3dd1d611a380b7f99af1f3 registry.k8s.io/kube-controller-manager:v1.28.0-alpha.0],SizeBytes:30820280,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/kube-proxy@sha256:8a03b3147cd7be8e8235ae1cd70cf69aca9ce03ba1cfaffea320540bb5afa496 registry.k8s.io/kube-proxy:v1.28.0-alpha.0],SizeBytes:23896850,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:23345856,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:22381475,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:22085298,},ContainerImage{Names:[registry.k8s.io/provider-aws/cloud-controller-manager@sha256:fdeb61e3e42ecd9cca868d550ebdb88dd6341d9e91fcfa9a37e227dab2ad22cb registry.k8s.io/provider-aws/cloud-controller-manager:v1.26.0],SizeBytes:20154862,},ContainerImage{Names:[registry.k8s.io/kube-scheduler@sha256:cbf4007abb076188f16b975bc2b74452603cc7b34c60a29d92f14c38ad02dc11 registry.k8s.io/kube-scheduler:v1.28.0-alpha.0],SizeBytes:18078674,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.27.0-alpha.1],SizeBytes:5074298,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Mar 25 05:39:50.509: INFO: Logging kubelet events for node i-0168f82458996cc53 Mar 25 05:39:50.585: INFO: Logging pods the kubelet thinks is on node i-0168f82458996cc53 Mar 25 05:39:50.646: INFO: kube-controller-manager-i-0168f82458996cc53 started at 2023-03-25 05:17:36 +0000 UTC (0+1 container statuses recorded) Mar 25 05:39:50.646: INFO: Container kube-controller-manager ready: true, restart count 4 Mar 25 05:39:50.646: INFO: etcd-manager-events-i-0168f82458996cc53 started at 2023-03-25 05:17:36 +0000 UTC (11+1 container statuses recorded) Mar 25 05:39:50.646: INFO: Init container init-etcd-3-2-24 ready: true, restart count 0 Mar 25 05:39:50.646: INFO: Init container init-etcd-3-3-10 ready: true, restart count 0 Mar 25 05:39:50.646: INFO: Init container init-etcd-3-3-17 ready: true, restart count 0 Mar 25 05:39:50.646: INFO: Init container init-etcd-3-4-13 ready: true, restart count 0 Mar 25 05:39:50.646: INFO: Init container init-etcd-3-4-3 ready: true, restart count 0 Mar 25 05:39:50.646: INFO: Init container init-etcd-3-5-0 ready: true, restart count 0 Mar 25 05:39:50.646: INFO: Init container init-etcd-3-5-1 ready: true, restart count 0 Mar 25 05:39:50.646: INFO: Init container init-etcd-3-5-3 ready: true, restart count 0 Mar 25 05:39:50.646: INFO: Init container init-etcd-3-5-4 ready: true, restart count 0 Mar 25 05:39:50.646: INFO: Init container init-etcd-3-5-6 ready: true, restart count 0 Mar 25 05:39:50.646: INFO: Init container init-etcd-3-5-7 ready: true, restart count 0 Mar 25 05:39:50.646: INFO: Container etcd-manager ready: true, restart count 0 Mar 25 05:39:50.646: INFO: kube-scheduler-i-0168f82458996cc53 started at 2023-03-25 05:17:36 +0000 UTC (0+1 container statuses recorded) Mar 25 05:39:50.646: INFO: Container kube-scheduler ready: true, restart count 0 Mar 25 05:39:50.646: INFO: etcd-manager-main-i-0168f82458996cc53 started at 2023-03-25 05:17:36 +0000 UTC (11+1 container statuses recorded) Mar 25 05:39:50.646: INFO: Init container init-etcd-3-2-24 ready: true, restart count 0 Mar 25 05:39:50.646: INFO: Init container init-etcd-3-3-10 ready: true, restart count 0 Mar 25 05:39:50.646: INFO: Init container init-etcd-3-3-17 ready: true, restart count 0 Mar 25 05:39:50.646: INFO: Init container init-etcd-3-4-13 ready: true, restart count 0 Mar 25 05:39:50.646: INFO: Init container init-etcd-3-4-3 ready: true, restart count 0 Mar 25 05:39:50.646: INFO: Init container init-etcd-3-5-0 ready: true, restart count 0 Mar 25 05:39:50.646: INFO: Init container init-etcd-3-5-1 ready: true, restart count 0 Mar 25 05:39:50.646: INFO: Init container init-etcd-3-5-3 ready: true, restart count 0 Mar 25 05:39:50.646: INFO: Init container init-etcd-3-5-4 ready: true, restart count 0 Mar 25 05:39:50.646: INFO: Init container init-etcd-3-5-6 ready: true, restart count 0 Mar 25 05:39:50.646: INFO: Init container init-etcd-3-5-7 ready: true, restart count 0 Mar 25 05:39:50.646: INFO: Container etcd-manager ready: true, restart count 0 Mar 25 05:39:50.646: INFO: dns-controller-69987775c6-hbn69 started at 2023-03-25 05:19:47 +0000 UTC (0+1 container statuses recorded) Mar 25 05:39:50.646: INFO: Container dns-controller ready: true, restart count 0 Mar 25 05:39:50.646: INFO: aws-cloud-controller-manager-26vbv started at 2023-03-25 05:20:06 +0000 UTC (0+1 container statuses recorded) Mar 25 05:39:50.646: INFO: Container aws-cloud-controller-manager ready: true, restart count 0 Mar 25 05:39:50.646: INFO: calico-node-rdjq2 started at 2023-03-25 05:20:06 +0000 UTC (3+1 container statuses recorded) Mar 25 05:39:50.646: INFO: Init container upgrade-ipam ready: true, restart count 0 Mar 25 05:39:50.646: INFO: Init container install-cni ready: true, restart count 0 Mar 25 05:39:50.646: INFO: Init container mount-bpffs ready: true, restart count 0 Mar 25 05:39:50.646: INFO: Container calico-node ready: true, restart count 0 Mar 25 05:39:50.646: INFO: ebs-csi-node-zjtj6 started at 2023-03-25 05:20:06 +0000 UTC (0+3 container statuses recorded) Mar 25 05:39:50.646: INFO: Container ebs-plugin ready: true, restart count 0 Mar 25 05:39:50.646: INFO: Container liveness-probe ready: true, restart count 0 Mar 25 05:39:50.646: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 25 05:39:50.646: INFO: kube-apiserver-i-0168f82458996cc53 started at 2023-03-25 05:17:36 +0000 UTC (0+2 container statuses recorded) Mar 25 05:39:50.646: INFO: Container healthcheck ready: true, restart count 0 Mar 25 05:39:50.646: INFO: Container kube-apiserver ready: true, restart count 2 Mar 25 05:39:50.646: INFO: kube-proxy-i-0168f82458996cc53 started at 2023-03-25 05:17:36 +0000 UTC (0+1 container statuses recorded) Mar 25 05:39:50.646: INFO: Container kube-proxy ready: true, restart count 0 Mar 25 05:39:50.646: INFO: ebs-csi-controller-79f984545f-9q5ks started at 2023-03-25 05:21:26 +0000 UTC (0+5 container statuses recorded) Mar 25 05:39:50.646: INFO: Container csi-attacher ready: true, restart count 0 Mar 25 05:39:50.646: INFO: Container csi-provisioner ready: true, restart count 0 Mar 25 05:39:50.646: INFO: Container csi-resizer ready: true, restart count 0 Mar 25 05:39:50.646: INFO: Container ebs-plugin ready: true, restart count 0 Mar 25 05:39:50.646: INFO: Container liveness-probe ready: true, restart count 0 Mar 25 05:39:50.646: INFO: kops-controller-cgw95 started at 2023-03-25 05:20:06 +0000 UTC (0+1 container statuses recorded) Mar 25 05:39:50.646: INFO: Container kops-controller ready: true, restart count 0 Mar 25 05:39:50.646: INFO: calico-kube-controllers-55f488656-rr8rz started at 2023-03-25 05:20:29 +0000 UTC (0+1 container statuses recorded) Mar 25 05:39:50.646: INFO: Container calico-kube-controllers ready: true, restart count 0 Mar 25 05:39:50.961: INFO: Latency metrics for node i-0168f82458996cc53 Mar 25 05:39:50.961: INFO: Logging node info for node i-03d301e6f030748f1 Mar 25 05:39:51.015: INFO: Node Info: &Node{ObjectMeta:{i-03d301e6f030748f1 897f52bd-65e2-4107-843a-37e36aa74032 13991 0 2023-03-25 05:31:35 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-1 failure-domain.beta.kubernetes.io/zone:us-west-1b kubernetes.io/arch:amd64 kubernetes.io/hostname:i-03d301e6f030748f1 kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:us-west-1b topology.hostpath.csi/node:i-03d301e6f030748f1 topology.kubernetes.io/region:us-west-1 topology.kubernetes.io/zone:us-west-1b] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-4807":"i-03d301e6f030748f1","ebs.csi.aws.com":"i-03d301e6f030748f1"} node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:172.20.50.199/19 projectcalico.org/IPv4IPIPTunnelAddr:100.110.167.64 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-03-25 05:31:35 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-25 05:31:35 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-03-25 05:31:35 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2023-03-25 05:31:35 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2023-03-25 05:31:48 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.8.0/24\"":{}}}} } {calico-node Update v1 2023-03-25 05:31:56 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4IPIPTunnelAddr":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2023-03-25 05:39:48 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.8.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-1b/i-03d301e6f030748f1,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.8.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-03-25 05:31:56 +0000 UTC,LastTransitionTime:2023-03-25 05:31:56 +0000 UTC,Reason:CalicoIsUp,Message:Calico is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-25 05:39:45 +0000 UTC,LastTransitionTime:2023-03-25 05:31:35 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-25 05:39:45 +0000 UTC,LastTransitionTime:2023-03-25 05:31:35 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-25 05:39:45 +0000 UTC,LastTransitionTime:2023-03-25 05:31:35 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-25 05:39:45 +0000 UTC,LastTransitionTime:2023-03-25 05:31:48 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.50.199,},NodeAddress{Type:ExternalIP,Address:52.53.245.63,},NodeAddress{Type:InternalDNS,Address:i-03d301e6f030748f1.us-west-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-03d301e6f030748f1.us-west-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-52-53-245-63.us-west-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec27152f7af9b9bfd1355ee19c256086,SystemUUID:ec27152f-7af9-b9bf-d135-5ee19c256086,BootID:1c303852-f2ce-45f9-ab14-6e8de97a0a3e,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.28.0-alpha.0,KubeProxyVersion:v1.28.0-alpha.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[docker.io/calico/cni@sha256:e282ea2914c806b5de2976330a17cfb5e6dcef47147bceb1432ca5c75fd46f50 docker.io/calico/cni:v3.24.5],SizeBytes:87458443,},ContainerImage{Names:[docker.io/calico/node@sha256:5972ad2bcbdc668564d3e26960c9c513b2d7b05581c704747cf7c62ef3a405a6 docker.io/calico/node:v3.24.5],SizeBytes:81584866,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nautilus@sha256:80ba6c8c44f9623f06e868a1aa66026c8ec438ad814f9ec95e9333b415fe3550 registry.k8s.io/e2e-test-images/nautilus:1.7],SizeBytes:49641698,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:e468dddcd275163a042ab297b2d8c2aca50d5e148d2d22f3b6ba119e2f31fa79 registry.k8s.io/sig-storage/csi-provisioner:v3.4.0],SizeBytes:27427836,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/kube-proxy@sha256:8a03b3147cd7be8e8235ae1cd70cf69aca9ce03ba1cfaffea320540bb5afa496 registry.k8s.io/kube-proxy:v1.28.0-alpha.0],SizeBytes:23896850,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:23881995,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18758628,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18233005,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8688564,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Mar 25 05:39:51.015: INFO: Logging kubelet events for node i-03d301e6f030748f1 Mar 25 05:39:51.100: INFO: Logging pods the kubelet thinks is on node i-03d301e6f030748f1 Mar 25 05:39:51.160: INFO: ebs-csi-node-lljwl started at 2023-03-25 05:31:35 +0000 UTC (0+3 container statuses recorded) Mar 25 05:39:51.160: INFO: Container ebs-plugin ready: true, restart count 0 Mar 25 05:39:51.160: INFO: Container liveness-probe ready: true, restart count 0 Mar 25 05:39:51.160: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 25 05:39:51.160: INFO: kube-proxy-i-03d301e6f030748f1 started at 2023-03-25 05:31:35 +0000 UTC (0+1 container statuses recorded) Mar 25 05:39:51.160: INFO: Container kube-proxy ready: true, restart count 0 Mar 25 05:39:51.160: INFO: pod-csi-inline-volumes started at 2023-03-25 05:38:08 +0000 UTC (0+1 container statuses recorded) Mar 25 05:39:51.160: INFO: Container pod-csi-inline-volumes ready: false, restart count 0 Mar 25 05:39:51.160: INFO: test-rolling-update-with-lb-69fcddc54f-m6d8k started at 2023-03-25 05:38:08 +0000 UTC (0+1 container statuses recorded) Mar 25 05:39:51.160: INFO: Container agnhost ready: true, restart count 0 Mar 25 05:39:51.160: INFO: pause-pod-1 started at 2023-03-25 05:39:50 +0000 UTC (0+1 container statuses recorded) Mar 25 05:39:51.160: INFO: Container agnhost-container ready: false, restart count 0 Mar 25 05:39:51.160: INFO: busybox-15fa3b67-2198-439c-9958-e0264653d9ed started at 2023-03-25 05:38:55 +0000 UTC (0+1 container statuses recorded) Mar 25 05:39:51.160: INFO: Container busybox ready: true, restart count 0 Mar 25 05:39:51.160: INFO: pod-5ba8bd16-ba61-4297-9534-d6108059fad2 started at 2023-03-25 05:39:49 +0000 UTC (0+1 container statuses recorded) Mar 25 05:39:51.160: INFO: Container write-pod ready: false, restart count 0 Mar 25 05:39:51.160: INFO: netserver-0 started at 2023-03-25 05:39:26 +0000 UTC (0+1 container statuses recorded) Mar 25 05:39:51.160: INFO: Container webserver ready: true, restart count 0 Mar 25 05:39:51.160: INFO: calico-node-h96vq started at 2023-03-25 05:31:35 +0000 UTC (3+1 container statuses recorded) Mar 25 05:39:51.160: INFO: Init container upgrade-ipam ready: true, restart count 0 Mar 25 05:39:51.160: INFO: Init container install-cni ready: true, restart count 0 Mar 25 05:39:51.160: INFO: Init container mount-bpffs ready: true, restart count 0 Mar 25 05:39:51.160: INFO: Container calico-node ready: true, restart count 0 Mar 25 05:39:51.160: INFO: csi-hostpathplugin-0 started at 2023-03-25 05:38:58 +0000 UTC (0+7 container statuses recorded) Mar 25 05:39:51.160: INFO: Container csi-attacher ready: true, restart count 0 Mar 25 05:39:51.160: INFO: Container csi-provisioner ready: true, restart count 0 Mar 25 05:39:51.160: INFO: Container csi-resizer ready: true, restart count 0 Mar 25 05:39:51.160: INFO: Container csi-snapshotter ready: true, restart count 0 Mar 25 05:39:51.160: INFO: Container hostpath ready: true, restart count 0 Mar 25 05:39:51.160: INFO: Container liveness-probe ready: true, restart count 0 Mar 25 05:39:51.160: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 25 05:39:51.484: INFO: Latency metrics for node i-03d301e6f030748f1 Mar 25 05:39:51.484: INFO: Logging node info for node i-049fdb2c1cde894df Mar 25 05:39:51.534: INFO: Node Info: &Node{ObjectMeta:{i-049fdb2c1cde894df b43a51e6-dda7-4047-b368-72d515695b4e 13967 0 2023-03-25 05:23:34 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-1 failure-domain.beta.kubernetes.io/zone:us-west-1b kubernetes.io/arch:amd64 kubernetes.io/hostname:i-049fdb2c1cde894df kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:us-west-1b topology.hostpath.csi/node:i-049fdb2c1cde894df topology.kubernetes.io/region:us-west-1 topology.kubernetes.io/zone:us-west-1b] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-049fdb2c1cde894df"} node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:172.20.43.61/19 projectcalico.org/IPv4IPIPTunnelAddr:100.97.18.64 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-03-25 05:23:34 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-25 05:23:34 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-03-25 05:23:34 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2023-03-25 05:23:34 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2023-03-25 05:23:45 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.6.0/24\"":{}}}} } {calico-node Update v1 2023-03-25 05:23:52 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4IPIPTunnelAddr":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2023-03-25 05:39:48 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.6.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-1b/i-049fdb2c1cde894df,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.6.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-03-25 05:23:52 +0000 UTC,LastTransitionTime:2023-03-25 05:23:52 +0000 UTC,Reason:CalicoIsUp,Message:Calico is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-25 05:39:43 +0000 UTC,LastTransitionTime:2023-03-25 05:23:34 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-25 05:39:43 +0000 UTC,LastTransitionTime:2023-03-25 05:23:34 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-25 05:39:43 +0000 UTC,LastTransitionTime:2023-03-25 05:23:34 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-25 05:39:43 +0000 UTC,LastTransitionTime:2023-03-25 05:23:45 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.43.61,},NodeAddress{Type:ExternalIP,Address:18.144.75.67,},NodeAddress{Type:InternalDNS,Address:i-049fdb2c1cde894df.us-west-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-049fdb2c1cde894df.us-west-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-18-144-75-67.us-west-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec252e592d19e1e6c583e87505431016,SystemUUID:ec252e59-2d19-e1e6-c583-e87505431016,BootID:45436b5d-1eb0-4b6f-a1d1-a1492b31b5d7,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.28.0-alpha.0,KubeProxyVersion:v1.28.0-alpha.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[docker.io/calico/cni@sha256:e282ea2914c806b5de2976330a17cfb5e6dcef47147bceb1432ca5c75fd46f50 docker.io/calico/cni:v3.24.5],SizeBytes:87458443,},ContainerImage{Names:[docker.io/calico/node@sha256:5972ad2bcbdc668564d3e26960c9c513b2d7b05581c704747cf7c62ef3a405a6 docker.io/calico/node:v3.24.5],SizeBytes:81584866,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:e468dddcd275163a042ab297b2d8c2aca50d5e148d2d22f3b6ba119e2f31fa79 registry.k8s.io/sig-storage/csi-provisioner:v3.4.0],SizeBytes:27427836,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/kube-proxy@sha256:8a03b3147cd7be8e8235ae1cd70cf69aca9ce03ba1cfaffea320540bb5afa496 registry.k8s.io/kube-proxy:v1.28.0-alpha.0],SizeBytes:23896850,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:23881995,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:68d396900aeaa072c1f27289485fdac29834045a6f3ffe369bf389d830ef572d registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.6],SizeBytes:20293261,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18758628,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18233005,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:15273057,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8688564,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6978614,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Mar 25 05:39:51.534: INFO: Logging kubelet events for node i-049fdb2c1cde894df Mar 25 05:39:51.628: INFO: Logging pods the kubelet thinks is on node i-049fdb2c1cde894df Mar 25 05:39:51.703: INFO: kube-proxy-i-049fdb2c1cde894df started at 2023-03-25 05:23:35 +0000 UTC (0+1 container statuses recorded) Mar 25 05:39:51.703: INFO: Container kube-proxy ready: true, restart count 0 Mar 25 05:39:51.703: INFO: ebs-csi-node-m9zxz started at 2023-03-25 05:23:35 +0000 UTC (0+3 container statuses recorded) Mar 25 05:39:51.703: INFO: Container ebs-plugin ready: true, restart count 0 Mar 25 05:39:51.703: INFO: Container liveness-probe ready: true, restart count 0 Mar 25 05:39:51.703: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 25 05:39:51.703: INFO: calico-node-wt5p7 started at 2023-03-25 05:23:35 +0000 UTC (3+1 container statuses recorded) Mar 25 05:39:51.703: INFO: Init container upgrade-ipam ready: true, restart count 0 Mar 25 05:39:51.703: INFO: Init container install-cni ready: true, restart count 0 Mar 25 05:39:51.703: INFO: Init container mount-bpffs ready: true, restart count 0 Mar 25 05:39:51.703: INFO: Container calico-node ready: true, restart count 0 Mar 25 05:39:51.703: INFO: coredns-559769c974-pgqhl started at 2023-03-25 05:24:36 +0000 UTC (0+1 container statuses recorded) Mar 25 05:39:51.703: INFO: Container coredns ready: true, restart count 0 Mar 25 05:39:51.703: INFO: suspend-false-to-true-vxmf9 started at 2023-03-25 05:39:41 +0000 UTC (0+1 container statuses recorded) Mar 25 05:39:51.703: INFO: Container c ready: true, restart count 0 Mar 25 05:39:51.703: INFO: httpd started at 2023-03-25 05:39:50 +0000 UTC (0+1 container statuses recorded) Mar 25 05:39:51.703: INFO: Container httpd ready: false, restart count 0 Mar 25 05:39:51.703: INFO: netserver-1 started at 2023-03-25 05:39:27 +0000 UTC (0+1 container statuses recorded) Mar 25 05:39:51.703: INFO: Container webserver ready: true, restart count 0 Mar 25 05:39:51.703: INFO: pod-0c48c0e8-3c8a-42b0-9b49-339ca3dfb2af started at 2023-03-25 05:39:39 +0000 UTC (0+1 container statuses recorded) Mar 25 05:39:51.703: INFO: Container write-pod ready: true, restart count 0 Mar 25 05:39:51.703: INFO: pod-subpath-test-configmap-qvpv started at 2023-03-25 05:39:45 +0000 UTC (0+1 container statuses recorded) Mar 25 05:39:51.703: INFO: Container test-container-subpath-configmap-qvpv ready: true, restart count 0 Mar 25 05:39:51.703: INFO: coredns-autoscaler-7cb5c5b969-h485x started at 2023-03-25 05:24:36 +0000 UTC (0+1 container statuses recorded) Mar 25 05:39:51.703: INFO: Container autoscaler ready: true, restart count 0 Mar 25 05:39:51.703: INFO: rand-non-local-2czp6 started at 2023-03-25 05:39:48 +0000 UTC (0+1 container statuses recorded) Mar 25 05:39:51.703: INFO: Container c ready: false, restart count 0 Mar 25 05:39:51.703: INFO: test-container-pod started at 2023-03-25 05:39:50 +0000 UTC (0+1 container statuses recorded) Mar 25 05:39:51.703: INFO: Container webserver ready: false, restart count 0 Mar 25 05:39:51.703: INFO: pod-terminate-status-0-2 started at 2023-03-25 05:39:50 +0000 UTC (1+1 container statuses recorded) Mar 25 05:39:51.703: INFO: Init container fail ready: false, restart count 0 Mar 25 05:39:51.703: INFO: Container blocked ready: false, restart count 0 Mar 25 05:39:51.703: INFO: pod-with-poststart-exec-hook started at 2023-03-25 05:38:01 +0000 UTC (0+1 container statuses recorded) Mar 25 05:39:51.703: INFO: Container pod-with-poststart-exec-hook ready: true, restart count 0 Mar 25 05:39:51.703: INFO: hostexec-i-049fdb2c1cde894df-2c6x5 started at 2023-03-25 05:39:33 +0000 UTC (0+1 container statuses recorded) Mar 25 05:39:51.703: INFO: Container agnhost-container ready: true, restart count 0 Mar 25 05:39:52.055: INFO: Latency metrics for node i-049fdb2c1cde894df Mar 25 05:39:52.055: INFO: Logging node info for node i-062679bb8c0ba4bce Mar 25 05:39:52.107: INFO: Node Info: &Node{ObjectMeta:{i-062679bb8c0ba4bce ee2da569-bb88-4297-9e6f-d2b15ba7ce86 13848 0 2023-03-25 05:27:31 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-1 failure-domain.beta.kubernetes.io/zone:us-west-1b kubernetes.io/arch:amd64 kubernetes.io/hostname:i-062679bb8c0ba4bce kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:us-west-1b topology.hostpath.csi/node:i-062679bb8c0ba4bce topology.kubernetes.io/region:us-west-1 topology.kubernetes.io/zone:us-west-1b] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-provisioning-2049":"i-062679bb8c0ba4bce","ebs.csi.aws.com":"i-062679bb8c0ba4bce"} node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:172.20.45.43/19 projectcalico.org/IPv4IPIPTunnelAddr:100.106.79.0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-03-25 05:27:31 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-25 05:27:31 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-03-25 05:27:31 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2023-03-25 05:27:31 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2023-03-25 05:27:44 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.7.0/24\"":{}}}} } {calico-node Update v1 2023-03-25 05:27:50 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4IPIPTunnelAddr":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2023-03-25 05:39:43 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-03-25 05:39:47 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.7.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-1b/i-062679bb8c0ba4bce,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.7.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051689472 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946831872 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-03-25 05:27:50 +0000 UTC,LastTransitionTime:2023-03-25 05:27:50 +0000 UTC,Reason:CalicoIsUp,Message:Calico is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-25 05:39:45 +0000 UTC,LastTransitionTime:2023-03-25 05:27:31 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-25 05:39:45 +0000 UTC,LastTransitionTime:2023-03-25 05:27:31 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-25 05:39:45 +0000 UTC,LastTransitionTime:2023-03-25 05:27:31 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-25 05:39:45 +0000 UTC,LastTransitionTime:2023-03-25 05:27:44 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.45.43,},NodeAddress{Type:ExternalIP,Address:3.101.133.108,},NodeAddress{Type:InternalDNS,Address:i-062679bb8c0ba4bce.us-west-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-062679bb8c0ba4bce.us-west-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-3-101-133-108.us-west-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2528f599e9b1e199793fb5cd06d205,SystemUUID:ec2528f5-99e9-b1e1-9979-3fb5cd06d205,BootID:ff56b70b-c8c5-46db-8cc5-4353c8f535c6,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.28.0-alpha.0,KubeProxyVersion:v1.28.0-alpha.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:112030336,},ContainerImage{Names:[docker.io/calico/cni@sha256:e282ea2914c806b5de2976330a17cfb5e6dcef47147bceb1432ca5c75fd46f50 docker.io/calico/cni:v3.24.5],SizeBytes:87458443,},ContainerImage{Names:[docker.io/calico/node@sha256:5972ad2bcbdc668564d3e26960c9c513b2d7b05581c704747cf7c62ef3a405a6 docker.io/calico/node:v3.24.5],SizeBytes:81584866,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nautilus@sha256:80ba6c8c44f9623f06e868a1aa66026c8ec438ad814f9ec95e9333b415fe3550 registry.k8s.io/e2e-test-images/nautilus:1.7],SizeBytes:49641698,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/kube-proxy@sha256:8a03b3147cd7be8e8235ae1cd70cf69aca9ce03ba1cfaffea320540bb5afa496 registry.k8s.io/kube-proxy:v1.28.0-alpha.0],SizeBytes:23896850,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:15273057,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6978614,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[kubernetes.io/csi/ebs.csi.aws.com^vol-05b5aad1e0ff17798],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-05b5aad1e0ff17798,DevicePath:,},},Config:nil,},} Mar 25 05:39:52.107: INFO: Logging kubelet events for node i-062679bb8c0ba4bce Mar 25 05:39:52.165: INFO: Logging pods the kubelet thinks is on node i-062679bb8c0ba4bce Mar 25 05:39:52.220: INFO: privileged-pod started at 2023-03-25 05:39:19 +0000 UTC (0+2 container statuses recorded) Mar 25 05:39:52.220: INFO: Container not-privileged-container ready: true, restart count 0 Mar 25 05:39:52.220: INFO: Container privileged-container ready: true, restart count 0 Mar 25 05:39:52.220: INFO: test-rolling-update-with-lb-69fcddc54f-hvwj4 started at 2023-03-25 05:38:08 +0000 UTC (0+1 container statuses recorded) Mar 25 05:39:52.220: INFO: Container agnhost ready: true, restart count 0 Mar 25 05:39:52.220: INFO: kube-proxy-i-062679bb8c0ba4bce started at 2023-03-25 05:27:32 +0000 UTC (0+1 container statuses recorded) Mar 25 05:39:52.220: INFO: Container kube-proxy ready: true, restart count 0 Mar 25 05:39:52.220: INFO: ebs-csi-node-7twrq started at 2023-03-25 05:27:32 +0000 UTC (0+3 container statuses recorded) Mar 25 05:39:52.220: INFO: Container ebs-plugin ready: true, restart count 0 Mar 25 05:39:52.220: INFO: Container liveness-probe ready: true, restart count 0 Mar 25 05:39:52.220: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 25 05:39:52.220: INFO: csi-hostpathplugin-0 started at 2023-03-25 05:39:42 +0000 UTC (0+7 container statuses recorded) Mar 25 05:39:52.220: INFO: Container csi-attacher ready: false, restart count 0 Mar 25 05:39:52.220: INFO: Container csi-provisioner ready: false, restart count 0 Mar 25 05:39:52.220: INFO: Container csi-resizer ready: false, restart count 0 Mar 25 05:39:52.220: INFO: Container csi-snapshotter ready: false, restart count 0 Mar 25 05:39:52.220: INFO: Container hostpath ready: false, restart count 0 Mar 25 05:39:52.220: INFO: Container liveness-probe ready: false, restart count 0 Mar 25 05:39:52.220: INFO: Container node-driver-registrar ready: false, restart count 0 Mar 25 05:39:52.220: INFO: calico-node-zwm5j started at 2023-03-25 05:27:32 +0000 UTC (3+1 container statuses recorded) Mar 25 05:39:52.220: INFO: Init container upgrade-ipam ready: true, restart count 0 Mar 25 05:39:52.220: INFO: Init container install-cni ready: true, restart count 0 Mar 25 05:39:52.220: INFO: Init container mount-bpffs ready: true, restart count 0 Mar 25 05:39:52.220: INFO: Container calico-node ready: true, restart count 0 Mar 25 05:39:52.220: INFO: startup-c479faf7-b2a9-4707-95f6-45b6b910a29d started at 2023-03-25 05:39:26 +0000 UTC (0+1 container statuses recorded) Mar 25 05:39:52.220: INFO: Container busybox ready: false, restart count 0 Mar 25 05:39:52.220: INFO: suspend-false-to-true-6f7v4 started at 2023-03-25 05:39:41 +0000 UTC (0+1 container statuses recorded) Mar 25 05:39:52.220: INFO: Container c ready: true, restart count 0 Mar 25 05:39:52.220: INFO: coredns-559769c974-p9vg6 started at 2023-03-25 05:36:16 +0000 UTC (0+1 container statuses recorded) Mar 25 05:39:52.220: INFO: Container coredns ready: true, restart count 0 Mar 25 05:39:52.220: INFO: netserver-2 started at 2023-03-25 05:39:27 +0000 UTC (0+1 container statuses recorded) Mar 25 05:39:52.220: INFO: Container webserver ready: true, restart count 0 Mar 25 05:39:52.220: INFO: run-log-test started at 2023-03-25 05:39:38 +0000 UTC (0+1 container statuses recorded) Mar 25 05:39:52.220: INFO: Container run-log-test ready: true, restart count 0 Mar 25 05:39:52.220: INFO: pod-subpath-test-inlinevolume-ln6g started at 2023-03-25 05:39:48 +0000 UTC (2+2 container statuses recorded) Mar 25 05:39:52.220: INFO: Init container init-volume-inlinevolume-ln6g ready: true, restart count 0 Mar 25 05:39:52.220: INFO: Init container test-init-subpath-inlinevolume-ln6g ready: true, restart count 0 Mar 25 05:39:52.220: INFO: Container test-container-subpath-inlinevolume-ln6g ready: false, restart count 0 Mar 25 05:39:52.220: INFO: Container test-container-volume-inlinevolume-ln6g ready: false, restart count 0 Mar 25 05:39:52.500: INFO: Latency metrics for node i-062679bb8c0ba4bce Mar 25 05:39:52.500: INFO: Logging node info for node i-072608201d2c0f1ad Mar 25 05:39:52.549: INFO: Node Info: &Node{ObjectMeta:{i-072608201d2c0f1ad 264dfba1-3cf0-4c99-8e8c-098f0f792077 11777 0 2023-03-25 05:35:29 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-1 failure-domain.beta.kubernetes.io/zone:us-west-1b kubernetes.io/arch:amd64 kubernetes.io/hostname:i-072608201d2c0f1ad kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:us-west-1b topology.hostpath.csi/node:i-072608201d2c0f1ad topology.kubernetes.io/region:us-west-1 topology.kubernetes.io/zone:us-west-1b] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-072608201d2c0f1ad"} node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:172.20.38.125/19 projectcalico.org/IPv4IPIPTunnelAddr:100.107.220.192 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-03-25 05:35:29 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-25 05:35:29 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-03-25 05:35:29 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2023-03-25 05:35:29 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2023-03-25 05:35:41 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.9.0/24\"":{}}}} } {calico-node Update v1 2023-03-25 05:35:48 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4IPIPTunnelAddr":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2023-03-25 05:39:08 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.9.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-1b/i-072608201d2c0f1ad,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.9.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051689472 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946831872 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-03-25 05:35:47 +0000 UTC,LastTransitionTime:2023-03-25 05:35:47 +0000 UTC,Reason:CalicoIsUp,Message:Calico is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-25 05:39:04 +0000 UTC,LastTransitionTime:2023-03-25 05:35:29 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-25 05:39:04 +0000 UTC,LastTransitionTime:2023-03-25 05:35:29 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-25 05:39:04 +0000 UTC,LastTransitionTime:2023-03-25 05:35:29 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-25 05:39:04 +0000 UTC,LastTransitionTime:2023-03-25 05:35:41 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.38.125,},NodeAddress{Type:ExternalIP,Address:3.101.88.10,},NodeAddress{Type:InternalDNS,Address:i-072608201d2c0f1ad.us-west-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-072608201d2c0f1ad.us-west-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-3-101-88-10.us-west-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2d3818c4f80a253dcc1a60e148dbc8,SystemUUID:ec2d3818-c4f8-0a25-3dcc-1a60e148dbc8,BootID:77d046cf-75ea-4a8c-970a-3190d64e019b,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.28.0-alpha.0,KubeProxyVersion:v1.28.0-alpha.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[docker.io/calico/cni@sha256:e282ea2914c806b5de2976330a17cfb5e6dcef47147bceb1432ca5c75fd46f50 docker.io/calico/cni:v3.24.5],SizeBytes:87458443,},ContainerImage{Names:[docker.io/calico/node@sha256:5972ad2bcbdc668564d3e26960c9c513b2d7b05581c704747cf7c62ef3a405a6 docker.io/calico/node:v3.24.5],SizeBytes:81584866,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:e468dddcd275163a042ab297b2d8c2aca50d5e148d2d22f3b6ba119e2f31fa79 registry.k8s.io/sig-storage/csi-provisioner:v3.4.0],SizeBytes:27427836,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/kube-proxy@sha256:8a03b3147cd7be8e8235ae1cd70cf69aca9ce03ba1cfaffea320540bb5afa496 registry.k8s.io/kube-proxy:v1.28.0-alpha.0],SizeBytes:23896850,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:23881995,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18758628,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18233005,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8688564,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6978614,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Mar 25 05:39:52.549: INFO: Logging kubelet events for node i-072608201d2c0f1ad Mar 25 05:39:52.605: INFO: Logging pods the kubelet thinks is on node i-072608201d2c0f1ad Mar 25 05:39:52.660: INFO: pod-projected-configmaps-61878de4-bfbc-4c4c-bb90-4daf9441ed77 started at 2023-03-25 05:39:21 +0000 UTC (0+3 container statuses recorded) Mar 25 05:39:52.660: INFO: Container createcm-volume-test ready: true, restart count 0 Mar 25 05:39:52.660: INFO: Container delcm-volume-test ready: true, restart count 0 Mar 25 05:39:52.660: INFO: Container updcm-volume-test ready: true, restart count 0 Mar 25 05:39:52.660: INFO: pod-handle-http-request started at 2023-03-25 05:37:59 +0000 UTC (0+2 container statuses recorded) Mar 25 05:39:52.660: INFO: Container container-handle-http-request ready: true, restart count 0 Mar 25 05:39:52.660: INFO: Container container-handle-https-request ready: true, restart count 0 Mar 25 05:39:52.660: INFO: calico-node-lc9zl started at 2023-03-25 05:35:30 +0000 UTC (3+1 container statuses recorded) Mar 25 05:39:52.660: INFO: Init container upgrade-ipam ready: true, restart count 0 Mar 25 05:39:52.660: INFO: Init container install-cni ready: true, restart count 0 Mar 25 05:39:52.660: INFO: Init container mount-bpffs ready: true, restart count 0 Mar 25 05:39:52.660: INFO: Container calico-node ready: true, restart count 0 Mar 25 05:39:52.660: INFO: inline-volume-tester-62x94 started at 2023-03-25 05:39:51 +0000 UTC (0+1 container statuses recorded) Mar 25 05:39:52.660: INFO: Container csi-volume-tester ready: false, restart count 0 Mar 25 05:39:52.660: INFO: kube-proxy-i-072608201d2c0f1ad started at 2023-03-25 05:35:30 +0000 UTC (0+1 container statuses recorded) Mar 25 05:39:52.660: INFO: Container kube-proxy ready: true, restart count 0 Mar 25 05:39:52.660: INFO: test-rolling-update-with-lb-69fcddc54f-gsptt started at 2023-03-25 05:38:08 +0000 UTC (0+1 container statuses recorded) Mar 25 05:39:52.660: INFO: Container agnhost ready: true, restart count 0 Mar 25 05:39:52.660: INFO: pod-subpath-test-projected-9zqc started at 2023-03-25 05:39:42 +0000 UTC (0+1 container statuses recorded) Mar 25 05:39:52.660: INFO: Container test-container-subpath-projected-9zqc ready: true, restart count 0 Mar 25 05:39:52.660: INFO: pod-update-activedeadlineseconds-2390442c-62f1-4aa0-98ff-3934301a3662 started at 2023-03-25 05:39:44 +0000 UTC (0+1 container statuses recorded) Mar 25 05:39:52.660: INFO: Container pause ready: false, restart count 0 Mar 25 05:39:52.660: INFO: webserver-pod started at 2023-03-25 05:39:46 +0000 UTC (0+1 container statuses recorded) Mar 25 05:39:52.660: INFO: Container agnhost ready: true, restart count 0 Mar 25 05:39:52.660: INFO: ebs-csi-node-r7mcm started at 2023-03-25 05:35:30 +0000 UTC (0+3 container statuses recorded) Mar 25 05:39:52.660: INFO: Container ebs-plugin ready: true, restart count 0 Mar 25 05:39:52.660: INFO: Container liveness-probe ready: true, restart count 0 Mar 25 05:39:52.660: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 25 05:39:52.660: INFO: pod-terminate-status-1-2 started at 2023-03-25 05:39:51 +0000 UTC (1+1 container statuses recorded) Mar 25 05:39:52.660: INFO: Init container fail ready: false, restart count 0 Mar 25 05:39:52.660: INFO: Container blocked ready: false, restart count 0 Mar 25 05:39:52.660: INFO: netserver-3 started at 2023-03-25 05:39:27 +0000 UTC (0+1 container statuses recorded) Mar 25 05:39:52.660: INFO: Container webserver ready: true, restart count 0 Mar 25 05:39:52.660: INFO: pod-terminate-status-2-2 started at 2023-03-25 05:39:51 +0000 UTC (1+1 container statuses recorded) Mar 25 05:39:52.660: INFO: Init container fail ready: false, restart count 0 Mar 25 05:39:52.660: INFO: Container blocked ready: false, restart count 0 Mar 25 05:39:52.900: INFO: Latency metrics for node i-072608201d2c0f1ad END STEP: dump namespace information after failure - test/e2e/framework/framework.go:288 @ 03/25/23 05:39:52.9 (2.629s) < Exit [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] subPath - dump namespaces | framework.go:209 @ 03/25/23 05:39:52.9 (2.629s) > Enter [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] subPath - tear down framework | framework.go:206 @ 03/25/23 05:39:52.9 STEP: Destroying namespace "provisioning-6241" for this suite. - test/e2e/framework/framework.go:351 @ 03/25/23 05:39:52.9 < Exit [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] subPath - tear down framework | framework.go:206 @ 03/25/23 05:39:52.951 (51ms) > Enter [ReportAfterEach] TOP-LEVEL - test/e2e/e2e_test.go:144 @ 03/25/23 05:39:52.951 < Exit [ReportAfterEach] TOP-LEVEL - test/e2e/e2e_test.go:144 @ 03/25/23 05:39:52.951 (0s)
Filter through log files | View test history on testgrid
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[It\]\sExternal\sStorage\s\[Driver\:\sebs\.csi\.aws\.com\]\s\[Testpattern\:\sDynamic\sPV\s\(default\sfs\)\]\ssubPath\sshould\ssupport\sfile\sas\ssubpath\s\[LinuxOnly\]$'
[FAILED] failed to get logs from pod-subpath-test-dynamicpv-z75w for test-container-subpath-dynamicpv-z75w: Get "https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/provisioning-4501/pods/pod-subpath-test-dynamicpv-z75w/log?container=test-container-subpath-dynamicpv-z75w&previous=false": stream error: stream ID 4023; INTERNAL_ERROR; received from peer In [It] at: test/e2e/framework/pod/output/output.go:247 @ 03/25/23 05:51:16.835from junit_01.xml
> Enter [BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/storage/framework/testsuite.go:51 @ 03/25/23 05:50:42.926 < Exit [BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/storage/framework/testsuite.go:51 @ 03/25/23 05:50:42.926 (0s) > Enter [BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath - set up framework | framework.go:191 @ 03/25/23 05:50:42.926 STEP: Creating a kubernetes client - test/e2e/framework/framework.go:211 @ 03/25/23 05:50:42.926 Mar 25 05:50:42.926: INFO: >>> kubeConfig: /root/.kube/config STEP: Building a namespace api object, basename provisioning - test/e2e/framework/framework.go:250 @ 03/25/23 05:50:42.928 STEP: Waiting for a default service account to be provisioned in namespace - test/e2e/framework/framework.go:259 @ 03/25/23 05:50:43.093 STEP: Waiting for kube-root-ca.crt to be provisioned in namespace - test/e2e/framework/framework.go:262 @ 03/25/23 05:50:43.213 < Exit [BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath - set up framework | framework.go:191 @ 03/25/23 05:50:43.313 (388ms) > Enter [BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/framework/metrics/init/init.go:33 @ 03/25/23 05:50:43.313 < Exit [BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/framework/metrics/init/init.go:33 @ 03/25/23 05:50:43.313 (0s) > Enter [It] should support file as subpath [LinuxOnly] - test/e2e/storage/testsuites/subpath.go:230 @ 03/25/23 05:50:43.313 Mar 25 05:50:43.313: INFO: Creating resource for dynamic PV Mar 25 05:50:43.313: INFO: Using claimSize:1Gi, test suite supported size:{ 1Mi}, driver(ebs.csi.aws.com) supported size:{ 1Mi} STEP: creating a StorageClass provisioning-4501-e2e-sc8bmm6 - test/e2e/storage/framework/volume_resource.go:102 @ 03/25/23 05:50:43.314 STEP: creating a claim - test/e2e/storage/framework/volume_resource.go:294 @ 03/25/23 05:50:43.381 Mar 25 05:50:43.381: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil STEP: Creating pod pod-subpath-test-dynamicpv-z75w - test/e2e/storage/testsuites/subpath.go:489 @ 03/25/23 05:50:43.521 STEP: Creating a pod to test atomic-volume-subpath - test/e2e/framework/pod/output/output.go:243 @ 03/25/23 05:50:43.521 STEP: Saw pod success - test/e2e/framework/pod/wait.go:409 @ 03/25/23 05:51:16.586 Mar 25 05:51:16.651: INFO: Trying to get logs from node i-03d301e6f030748f1 pod pod-subpath-test-dynamicpv-z75w container test-container-subpath-dynamicpv-z75w: <nil> Mar 25 05:51:16.701: INFO: Failed to get logs from node "i-03d301e6f030748f1" pod "pod-subpath-test-dynamicpv-z75w" container "test-container-subpath-dynamicpv-z75w". Get "https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/provisioning-4501/pods/pod-subpath-test-dynamicpv-z75w/log?container=test-container-subpath-dynamicpv-z75w&previous=false": stream error: stream ID 4023; INTERNAL_ERROR; received from peer STEP: delete the pod - test/e2e/framework/pod/output/output.go:168 @ 03/25/23 05:51:16.701 Mar 25 05:51:16.834: INFO: Unexpected error: <*fmt.wrapError | 0xc003f68900>: failed to get logs from pod-subpath-test-dynamicpv-z75w for test-container-subpath-dynamicpv-z75w: Get "https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/provisioning-4501/pods/pod-subpath-test-dynamicpv-z75w/log?container=test-container-subpath-dynamicpv-z75w&previous=false": stream error: stream ID 4023; INTERNAL_ERROR; received from peer { msg: "failed to get logs from pod-subpath-test-dynamicpv-z75w for test-container-subpath-dynamicpv-z75w: Get \"https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/provisioning-4501/pods/pod-subpath-test-dynamicpv-z75w/log?container=test-container-subpath-dynamicpv-z75w&previous=false\": stream error: stream ID 4023; INTERNAL_ERROR; received from peer", err: <*url.Error | 0xc003bd6120>{ Op: "Get", URL: "https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/provisioning-4501/pods/pod-subpath-test-dynamicpv-z75w/log?container=test-container-subpath-dynamicpv-z75w&previous=false", Err: <http2.StreamError>{ StreamID: 4023, Code: 2, Cause: <*errors.errorString | 0xc0001b3680>{ s: "received from peer", }, }, }, } [FAILED] failed to get logs from pod-subpath-test-dynamicpv-z75w for test-container-subpath-dynamicpv-z75w: Get "https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/provisioning-4501/pods/pod-subpath-test-dynamicpv-z75w/log?container=test-container-subpath-dynamicpv-z75w&previous=false": stream error: stream ID 4023; INTERNAL_ERROR; received from peer In [It] at: test/e2e/framework/pod/output/output.go:247 @ 03/25/23 05:51:16.835 < Exit [It] should support file as subpath [LinuxOnly] - test/e2e/storage/testsuites/subpath.go:230 @ 03/25/23 05:51:16.835 (33.521s) > Enter [AfterEach] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/framework/node/init/init.go:33 @ 03/25/23 05:51:16.835 Mar 25 05:51:16.835: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready < Exit [AfterEach] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/framework/node/init/init.go:33 @ 03/25/23 05:51:16.893 (58ms) > Enter [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/storage/testsuites/subpath.go:232 @ 03/25/23 05:51:16.893 STEP: Deleting pod - test/e2e/storage/testsuites/subpath.go:172 @ 03/25/23 05:51:16.893 Mar 25 05:51:16.893: INFO: Deleting pod "pod-subpath-test-dynamicpv-z75w" in namespace "provisioning-4501" STEP: Deleting pvc - test/e2e/storage/framework/volume_resource.go:181 @ 03/25/23 05:51:16.957 Mar 25 05:51:17.068: INFO: Deleting PersistentVolumeClaim "ebs.csi.aws.comvn5b2" Mar 25 05:51:17.144: INFO: Waiting up to 5m0s for PersistentVolume pvc-9aa5fe84-3860-4d5b-a7af-1f550a0bb130 to get deleted Mar 25 05:51:17.207: INFO: PersistentVolume pvc-9aa5fe84-3860-4d5b-a7af-1f550a0bb130 found and phase=Bound (62.272916ms) Mar 25 05:51:22.260: INFO: PersistentVolume pvc-9aa5fe84-3860-4d5b-a7af-1f550a0bb130 found and phase=Released (5.115337467s) Mar 25 05:51:27.309: INFO: PersistentVolume pvc-9aa5fe84-3860-4d5b-a7af-1f550a0bb130 found and phase=Released (10.164292588s) Mar 25 05:51:32.361: INFO: PersistentVolume pvc-9aa5fe84-3860-4d5b-a7af-1f550a0bb130 found and phase=Released (15.217123084s) Mar 25 05:51:37.413: INFO: PersistentVolume pvc-9aa5fe84-3860-4d5b-a7af-1f550a0bb130 was removed STEP: Deleting sc - test/e2e/storage/framework/volume_resource.go:228 @ 03/25/23 05:51:37.413 < Exit [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/storage/testsuites/subpath.go:232 @ 03/25/23 05:51:37.463 (20.57s) > Enter [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/framework/metrics/init/init.go:35 @ 03/25/23 05:51:37.463 < Exit [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/framework/metrics/init/init.go:35 @ 03/25/23 05:51:37.463 (0s) > Enter [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] subPath - dump namespaces | framework.go:209 @ 03/25/23 05:51:37.463 STEP: dump namespace information after failure - test/e2e/framework/framework.go:288 @ 03/25/23 05:51:37.463 STEP: Collecting events from namespace "provisioning-4501". - test/e2e/framework/debug/dump.go:42 @ 03/25/23 05:51:37.463 STEP: Found 12 events. - test/e2e/framework/debug/dump.go:46 @ 03/25/23 05:51:37.512 Mar 25 05:51:37.512: INFO: At 2023-03-25 05:50:43 +0000 UTC - event for ebs.csi.aws.comvn5b2: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding Mar 25 05:51:37.512: INFO: At 2023-03-25 05:50:43 +0000 UTC - event for ebs.csi.aws.comvn5b2: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator Mar 25 05:51:37.512: INFO: At 2023-03-25 05:50:43 +0000 UTC - event for ebs.csi.aws.comvn5b2: {ebs.csi.aws.com_i-0168f82458996cc53_0b589c46-3396-4499-94ba-eb51a48f7a58 } Provisioning: External provisioner is provisioning volume for claim "provisioning-4501/ebs.csi.aws.comvn5b2" Mar 25 05:51:37.512: INFO: At 2023-03-25 05:50:47 +0000 UTC - event for ebs.csi.aws.comvn5b2: {ebs.csi.aws.com_i-0168f82458996cc53_0b589c46-3396-4499-94ba-eb51a48f7a58 } ProvisioningSucceeded: Successfully provisioned volume pvc-9aa5fe84-3860-4d5b-a7af-1f550a0bb130 Mar 25 05:51:37.512: INFO: At 2023-03-25 05:50:47 +0000 UTC - event for pod-subpath-test-dynamicpv-z75w: {default-scheduler } Scheduled: Successfully assigned provisioning-4501/pod-subpath-test-dynamicpv-z75w to i-03d301e6f030748f1 Mar 25 05:51:37.512: INFO: At 2023-03-25 05:50:49 +0000 UTC - event for pod-subpath-test-dynamicpv-z75w: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-9aa5fe84-3860-4d5b-a7af-1f550a0bb130" Mar 25 05:51:37.512: INFO: At 2023-03-25 05:50:52 +0000 UTC - event for pod-subpath-test-dynamicpv-z75w: {kubelet i-03d301e6f030748f1} Pulled: Container image "registry.k8s.io/e2e-test-images/busybox:1.29-4" already present on machine Mar 25 05:51:37.512: INFO: At 2023-03-25 05:50:52 +0000 UTC - event for pod-subpath-test-dynamicpv-z75w: {kubelet i-03d301e6f030748f1} Created: Created container init-volume-dynamicpv-z75w Mar 25 05:51:37.512: INFO: At 2023-03-25 05:50:52 +0000 UTC - event for pod-subpath-test-dynamicpv-z75w: {kubelet i-03d301e6f030748f1} Started: Started container init-volume-dynamicpv-z75w Mar 25 05:51:37.512: INFO: At 2023-03-25 05:50:52 +0000 UTC - event for pod-subpath-test-dynamicpv-z75w: {kubelet i-03d301e6f030748f1} Pulled: Container image "registry.k8s.io/e2e-test-images/agnhost:2.43" already present on machine Mar 25 05:51:37.512: INFO: At 2023-03-25 05:50:52 +0000 UTC - event for pod-subpath-test-dynamicpv-z75w: {kubelet i-03d301e6f030748f1} Created: Created container test-container-subpath-dynamicpv-z75w Mar 25 05:51:37.512: INFO: At 2023-03-25 05:50:53 +0000 UTC - event for pod-subpath-test-dynamicpv-z75w: {kubelet i-03d301e6f030748f1} Started: Started container test-container-subpath-dynamicpv-z75w Mar 25 05:51:37.560: INFO: POD NODE PHASE GRACE CONDITIONS Mar 25 05:51:37.560: INFO: Mar 25 05:51:37.612: INFO: Logging node info for node i-0168f82458996cc53 Mar 25 05:51:37.661: INFO: Node Info: &Node{ObjectMeta:{i-0168f82458996cc53 f4eb302c-c477-45ac-bcac-ffed65f6580f 38654 0 2023-03-25 05:18:57 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-1 failure-domain.beta.kubernetes.io/zone:us-west-1b kops.k8s.io/kops-controller-pki: kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0168f82458996cc53 kubernetes.io/os:linux node-role.kubernetes.io/control-plane: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:us-west-1b topology.kubernetes.io/region:us-west-1 topology.kubernetes.io/zone:us-west-1b] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0168f82458996cc53"} node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:172.20.48.81/19 projectcalico.org/IPv4IPIPTunnelAddr:100.100.215.192 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-03-25 05:18:57 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {protokube Update v1 2023-03-25 05:18:58 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {calico-node Update v1 2023-03-25 05:20:22 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4IPIPTunnelAddr":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {aws-cloud-controller-manager Update v1 2023-03-25 05:20:26 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-25 05:20:26 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2023-03-25 05:20:31 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.5.0/24\"":{}},"f:taints":{}}} } {kubelet Update v1 2023-03-25 05:47:11 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.5.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-1b/i-0168f82458996cc53,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/control-plane,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.5.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3892310016 0} {<nil>} 3801084Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3787452416 0} {<nil>} 3698684Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-03-25 05:20:22 +0000 UTC,LastTransitionTime:2023-03-25 05:20:22 +0000 UTC,Reason:CalicoIsUp,Message:Calico is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-25 05:47:11 +0000 UTC,LastTransitionTime:2023-03-25 05:18:50 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-25 05:47:11 +0000 UTC,LastTransitionTime:2023-03-25 05:18:50 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-25 05:47:11 +0000 UTC,LastTransitionTime:2023-03-25 05:18:50 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-25 05:47:11 +0000 UTC,LastTransitionTime:2023-03-25 05:20:29 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.48.81,},NodeAddress{Type:ExternalIP,Address:3.101.135.77,},NodeAddress{Type:InternalDNS,Address:i-0168f82458996cc53.us-west-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-0168f82458996cc53.us-west-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-3-101-135-77.us-west-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2d1ccc7ebf313420bd1f530a94fbfd,SystemUUID:ec2d1ccc-7ebf-3134-20bd-1f530a94fbfd,BootID:15cee321-56e7-4290-ab48-47b60b0bc657,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.28.0-alpha.0,KubeProxyVersion:v1.28.0-alpha.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcd@sha256:dd75ec974b0a2a6f6bb47001ba09207976e625db898d1b16735528c009cb171c registry.k8s.io/etcd:3.5.6-0],SizeBytes:102542580,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:6f72b851544986cb0921b53ea655ec04c36131248f16d4ad110cb3ca0c369dc1 registry.k8s.io/etcd:3.5.4-0],SizeBytes:102157811,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:13f53ed1d91e2e11aac476ee9a0269fdda6cc4874eba903efd40daf50c55eee5 registry.k8s.io/etcd:3.5.3-0],SizeBytes:102143581,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:51eae8381dcb1078289fa7b4f3df2630cdc18d09fb56f8e56b41c40e191d6c83 registry.k8s.io/etcd:3.5.7-0],SizeBytes:101639218,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:4afb99b4690b418ffc2ceb67e1a17376457e441c1f09ab55447f0aaf992fa646 registry.k8s.io/etcd:3.4.3-0],SizeBytes:100947667,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:9ce33ba33d8e738a5b85ed50b5080ac746deceed4a7496c550927a7a19ca3b6d registry.k8s.io/etcd:3.5.0-0],SizeBytes:99868722,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:64b9ea357325d5db9f8a723dcf503b5a449177b17ac87d69481e126bb724c263 registry.k8s.io/etcd:3.5.1-0],SizeBytes:98888614,},ContainerImage{Names:[docker.io/calico/cni@sha256:e282ea2914c806b5de2976330a17cfb5e6dcef47147bceb1432ca5c75fd46f50 docker.io/calico/cni:v3.24.5],SizeBytes:87458443,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:4ad90a11b55313b182afc186b9876c8e891531b8db4c9bf1541953021618d0e2 registry.k8s.io/etcd:3.4.13-0],SizeBytes:86742272,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:0d2ab6b15b4fe0b903b9e81192d942fb39c29be61fe341cd16db5d0165b0435c registry.k8s.io/etcd:3.3.17-0],SizeBytes:85524408,},ContainerImage{Names:[docker.io/calico/node@sha256:5972ad2bcbdc668564d3e26960c9c513b2d7b05581c704747cf7c62ef3a405a6 docker.io/calico/node:v3.24.5],SizeBytes:81584866,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:17da501f5d2a675be46040422a27b7cc21b8a43895ac998b171db1c346f361f7 registry.k8s.io/etcd:3.3.10-0],SizeBytes:76162358,},ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager-slim@sha256:8727adc059334e0e3329dbccce116a734da4a090da5e86c8cddccaf333caa666 registry.k8s.io/etcdadm/etcd-manager-slim:v3.0.20230201],SizeBytes:63926165,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:905d7ca17fd02bc24c0eba9a062753aba15db3e31422390bc3238eb762339b20 registry.k8s.io/etcd:3.2.24-1],SizeBytes:63423287,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.27.0-alpha.1],SizeBytes:45219545,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.27.0-alpha.1],SizeBytes:43628585,},ContainerImage{Names:[registry.k8s.io/kube-apiserver@sha256:49934a29c559e79ed19fc38b4697b18d8fe174b17bf56192588575f980c93956 registry.k8s.io/kube-apiserver:v1.28.0-alpha.0],SizeBytes:33212506,},ContainerImage{Names:[docker.io/calico/kube-controllers@sha256:2b6acd7f677f76ffe12ecf3ea7df92eb9b1bdb07336d1ac2a54c7631fb753f7e docker.io/calico/kube-controllers:v3.24.5],SizeBytes:31137169,},ContainerImage{Names:[registry.k8s.io/kube-controller-manager@sha256:974748384634982c73527fdb3c9ec0ec476bc0d52b3dd1d611a380b7f99af1f3 registry.k8s.io/kube-controller-manager:v1.28.0-alpha.0],SizeBytes:30820280,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/kube-proxy@sha256:8a03b3147cd7be8e8235ae1cd70cf69aca9ce03ba1cfaffea320540bb5afa496 registry.k8s.io/kube-proxy:v1.28.0-alpha.0],SizeBytes:23896850,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:23345856,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:22381475,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:22085298,},ContainerImage{Names:[registry.k8s.io/provider-aws/cloud-controller-manager@sha256:fdeb61e3e42ecd9cca868d550ebdb88dd6341d9e91fcfa9a37e227dab2ad22cb registry.k8s.io/provider-aws/cloud-controller-manager:v1.26.0],SizeBytes:20154862,},ContainerImage{Names:[registry.k8s.io/kube-scheduler@sha256:cbf4007abb076188f16b975bc2b74452603cc7b34c60a29d92f14c38ad02dc11 registry.k8s.io/kube-scheduler:v1.28.0-alpha.0],SizeBytes:18078674,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.27.0-alpha.1],SizeBytes:5074298,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Mar 25 05:51:37.661: INFO: Logging kubelet events for node i-0168f82458996cc53 Mar 25 05:51:37.724: INFO: Logging pods the kubelet thinks is on node i-0168f82458996cc53 Mar 25 05:51:37.783: INFO: etcd-manager-events-i-0168f82458996cc53 started at 2023-03-25 05:17:36 +0000 UTC (11+1 container statuses recorded) Mar 25 05:51:37.783: INFO: Init container init-etcd-3-2-24 ready: true, restart count 0 Mar 25 05:51:37.783: INFO: Init container init-etcd-3-3-10 ready: true, restart count 0 Mar 25 05:51:37.783: INFO: Init container init-etcd-3-3-17 ready: true, restart count 0 Mar 25 05:51:37.783: INFO: Init container init-etcd-3-4-13 ready: true, restart count 0 Mar 25 05:51:37.783: INFO: Init container init-etcd-3-4-3 ready: true, restart count 0 Mar 25 05:51:37.783: INFO: Init container init-etcd-3-5-0 ready: true, restart count 0 Mar 25 05:51:37.783: INFO: Init container init-etcd-3-5-1 ready: true, restart count 0 Mar 25 05:51:37.783: INFO: Init container init-etcd-3-5-3 ready: true, restart count 0 Mar 25 05:51:37.783: INFO: Init container init-etcd-3-5-4 ready: true, restart count 0 Mar 25 05:51:37.783: INFO: Init container init-etcd-3-5-6 ready: true, restart count 0 Mar 25 05:51:37.783: INFO: Init container init-etcd-3-5-7 ready: true, restart count 0 Mar 25 05:51:37.783: INFO: Container etcd-manager ready: true, restart count 0 Mar 25 05:51:37.783: INFO: kube-controller-manager-i-0168f82458996cc53 started at 2023-03-25 05:17:36 +0000 UTC (0+1 container statuses recorded) Mar 25 05:51:37.783: INFO: Container kube-controller-manager ready: true, restart count 4 Mar 25 05:51:37.783: INFO: kube-scheduler-i-0168f82458996cc53 started at 2023-03-25 05:17:36 +0000 UTC (0+1 container statuses recorded) Mar 25 05:51:37.783: INFO: Container kube-scheduler ready: true, restart count 0 Mar 25 05:51:37.783: INFO: etcd-manager-main-i-0168f82458996cc53 started at 2023-03-25 05:17:36 +0000 UTC (11+1 container statuses recorded) Mar 25 05:51:37.783: INFO: Init container init-etcd-3-2-24 ready: true, restart count 0 Mar 25 05:51:37.783: INFO: Init container init-etcd-3-3-10 ready: true, restart count 0 Mar 25 05:51:37.783: INFO: Init container init-etcd-3-3-17 ready: true, restart count 0 Mar 25 05:51:37.783: INFO: Init container init-etcd-3-4-13 ready: true, restart count 0 Mar 25 05:51:37.783: INFO: Init container init-etcd-3-4-3 ready: true, restart count 0 Mar 25 05:51:37.783: INFO: Init container init-etcd-3-5-0 ready: true, restart count 0 Mar 25 05:51:37.783: INFO: Init container init-etcd-3-5-1 ready: true, restart count 0 Mar 25 05:51:37.783: INFO: Init container init-etcd-3-5-3 ready: true, restart count 0 Mar 25 05:51:37.783: INFO: Init container init-etcd-3-5-4 ready: true, restart count 0 Mar 25 05:51:37.783: INFO: Init container init-etcd-3-5-6 ready: true, restart count 0 Mar 25 05:51:37.783: INFO: Init container init-etcd-3-5-7 ready: true, restart count 0 Mar 25 05:51:37.783: INFO: Container etcd-manager ready: true, restart count 0 Mar 25 05:51:37.783: INFO: dns-controller-69987775c6-hbn69 started at 2023-03-25 05:19:47 +0000 UTC (0+1 container statuses recorded) Mar 25 05:51:37.783: INFO: Container dns-controller ready: true, restart count 0 Mar 25 05:51:37.783: INFO: aws-cloud-controller-manager-26vbv started at 2023-03-25 05:20:06 +0000 UTC (0+1 container statuses recorded) Mar 25 05:51:37.783: INFO: Container aws-cloud-controller-manager ready: true, restart count 0 Mar 25 05:51:37.783: INFO: ebs-csi-node-zjtj6 started at 2023-03-25 05:20:06 +0000 UTC (0+3 container statuses recorded) Mar 25 05:51:37.783: INFO: Container ebs-plugin ready: true, restart count 0 Mar 25 05:51:37.783: INFO: Container liveness-probe ready: true, restart count 0 Mar 25 05:51:37.783: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 25 05:51:37.783: INFO: kops-controller-cgw95 started at 2023-03-25 05:20:06 +0000 UTC (0+1 container statuses recorded) Mar 25 05:51:37.783: INFO: Container kops-controller ready: true, restart count 0 Mar 25 05:51:37.783: INFO: calico-kube-controllers-55f488656-rr8rz started at 2023-03-25 05:20:29 +0000 UTC (0+1 container statuses recorded) Mar 25 05:51:37.783: INFO: Container calico-kube-controllers ready: true, restart count 0 Mar 25 05:51:37.783: INFO: kube-proxy-i-0168f82458996cc53 started at 2023-03-25 05:17:36 +0000 UTC (0+1 container statuses recorded) Mar 25 05:51:37.783: INFO: Container kube-proxy ready: true, restart count 0 Mar 25 05:51:37.783: INFO: ebs-csi-controller-79f984545f-9q5ks started at 2023-03-25 05:21:26 +0000 UTC (0+5 container statuses recorded) Mar 25 05:51:37.783: INFO: Container csi-attacher ready: true, restart count 0 Mar 25 05:51:37.783: INFO: Container csi-provisioner ready: true, restart count 0 Mar 25 05:51:37.783: INFO: Container csi-resizer ready: true, restart count 0 Mar 25 05:51:37.783: INFO: Container ebs-plugin ready: true, restart count 0 Mar 25 05:51:37.783: INFO: Container liveness-probe ready: true, restart count 0 Mar 25 05:51:37.783: INFO: calico-node-rdjq2 started at 2023-03-25 05:20:06 +0000 UTC (3+1 container statuses recorded) Mar 25 05:51:37.783: INFO: Init container upgrade-ipam ready: true, restart count 0 Mar 25 05:51:37.783: INFO: Init container install-cni ready: true, restart count 0 Mar 25 05:51:37.783: INFO: Init container mount-bpffs ready: true, restart count 0 Mar 25 05:51:37.783: INFO: Container calico-node ready: true, restart count 0 Mar 25 05:51:37.783: INFO: kube-apiserver-i-0168f82458996cc53 started at 2023-03-25 05:17:36 +0000 UTC (0+2 container statuses recorded) Mar 25 05:51:37.783: INFO: Container healthcheck ready: true, restart count 0 Mar 25 05:51:37.783: INFO: Container kube-apiserver ready: true, restart count 2 Mar 25 05:51:38.146: INFO: Latency metrics for node i-0168f82458996cc53 Mar 25 05:51:38.146: INFO: Logging node info for node i-03d301e6f030748f1 Mar 25 05:51:38.194: INFO: Node Info: &Node{ObjectMeta:{i-03d301e6f030748f1 897f52bd-65e2-4107-843a-37e36aa74032 54004 0 2023-03-25 05:31:35 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-1 failure-domain.beta.kubernetes.io/zone:us-west-1b kubernetes.io/arch:amd64 kubernetes.io/hostname:i-03d301e6f030748f1 kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:us-west-1b topology.hostpath.csi/node:i-03d301e6f030748f1 topology.kubernetes.io/region:us-west-1 topology.kubernetes.io/zone:us-west-1b] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-volume-expand-2830":"i-03d301e6f030748f1","ebs.csi.aws.com":"i-03d301e6f030748f1"} node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:172.20.50.199/19 projectcalico.org/IPv4IPIPTunnelAddr:100.110.167.64 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-03-25 05:31:35 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-25 05:31:35 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-03-25 05:31:35 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2023-03-25 05:31:35 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2023-03-25 05:31:48 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.8.0/24\"":{}}}} } {calico-node Update v1 2023-03-25 05:31:56 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4IPIPTunnelAddr":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2023-03-25 05:51:27 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-03-25 05:51:31 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.8.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-1b/i-03d301e6f030748f1,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.8.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-03-25 05:31:56 +0000 UTC,LastTransitionTime:2023-03-25 05:31:56 +0000 UTC,Reason:CalicoIsUp,Message:Calico is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-25 05:51:31 +0000 UTC,LastTransitionTime:2023-03-25 05:31:35 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-25 05:51:31 +0000 UTC,LastTransitionTime:2023-03-25 05:31:35 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-25 05:51:31 +0000 UTC,LastTransitionTime:2023-03-25 05:31:35 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-25 05:51:31 +0000 UTC,LastTransitionTime:2023-03-25 05:31:48 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.50.199,},NodeAddress{Type:ExternalIP,Address:52.53.245.63,},NodeAddress{Type:InternalDNS,Address:i-03d301e6f030748f1.us-west-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-03d301e6f030748f1.us-west-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-52-53-245-63.us-west-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec27152f7af9b9bfd1355ee19c256086,SystemUUID:ec27152f-7af9-b9bf-d135-5ee19c256086,BootID:1c303852-f2ce-45f9-ab14-6e8de97a0a3e,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.28.0-alpha.0,KubeProxyVersion:v1.28.0-alpha.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:112030336,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:51eae8381dcb1078289fa7b4f3df2630cdc18d09fb56f8e56b41c40e191d6c83 registry.k8s.io/etcd:3.5.7-0],SizeBytes:101639218,},ContainerImage{Names:[docker.io/calico/cni@sha256:e282ea2914c806b5de2976330a17cfb5e6dcef47147bceb1432ca5c75fd46f50 docker.io/calico/cni:v3.24.5],SizeBytes:87458443,},ContainerImage{Names:[docker.io/calico/node@sha256:5972ad2bcbdc668564d3e26960c9c513b2d7b05581c704747cf7c62ef3a405a6 docker.io/calico/node:v3.24.5],SizeBytes:81584866,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nautilus@sha256:80ba6c8c44f9623f06e868a1aa66026c8ec438ad814f9ec95e9333b415fe3550 registry.k8s.io/e2e-test-images/nautilus:1.7],SizeBytes:49641698,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41901587,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:e468dddcd275163a042ab297b2d8c2aca50d5e148d2d22f3b6ba119e2f31fa79 registry.k8s.io/sig-storage/csi-provisioner:v3.4.0],SizeBytes:27427836,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/sample-apiserver@sha256:8d70890151aa5d096f331cb9da1b9cd5be0412b7363fe67b5c3befdcaa2a28d0 registry.k8s.io/e2e-test-images/sample-apiserver:1.17.7],SizeBytes:25667066,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/kube-proxy@sha256:8a03b3147cd7be8e8235ae1cd70cf69aca9ce03ba1cfaffea320540bb5afa496 registry.k8s.io/kube-proxy:v1.28.0-alpha.0],SizeBytes:23896850,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:23881995,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18758628,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18233005,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonroot@sha256:ee9f50b3c64b174d296d91ca9f69a914ac30e59095dfb462b2b518ad28a63655 registry.k8s.io/e2e-test-images/nonroot:1.4],SizeBytes:17747885,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8688564,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:7000082,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6978614,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac registry.k8s.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[kubernetes.io/csi/ebs.csi.aws.com^vol-00b32985831488b7c],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-00b32985831488b7c,DevicePath:,},},Config:nil,},} Mar 25 05:51:38.194: INFO: Logging kubelet events for node i-03d301e6f030748f1 Mar 25 05:51:38.257: INFO: Logging pods the kubelet thinks is on node i-03d301e6f030748f1 Mar 25 05:51:38.314: INFO: pod-with-poststart-http-hook started at 2023-03-25 05:50:55 +0000 UTC (0+1 container statuses recorded) Mar 25 05:51:38.314: INFO: Container pod-with-poststart-http-hook ready: true, restart count 0 Mar 25 05:51:38.314: INFO: ss-0 started at 2023-03-25 05:51:22 +0000 UTC (0+1 container statuses recorded) Mar 25 05:51:38.314: INFO: Container webserver ready: true, restart count 0 Mar 25 05:51:38.314: INFO: pod-handle-http-request started at 2023-03-25 05:50:51 +0000 UTC (0+2 container statuses recorded) Mar 25 05:51:38.314: INFO: Container container-handle-http-request ready: true, restart count 0 Mar 25 05:51:38.314: INFO: Container container-handle-https-request ready: true, restart count 0 Mar 25 05:51:38.314: INFO: pod-ephm-test-secret-m6m9 started at 2023-03-25 05:51:01 +0000 UTC (0+1 container statuses recorded) Mar 25 05:51:38.314: INFO: Container test-container-subpath-secret-m6m9 ready: false, restart count 0 Mar 25 05:51:38.314: INFO: pod-subpath-test-preprovisionedpv-hgcm started at 2023-03-25 05:51:37 +0000 UTC (2+1 container statuses recorded) Mar 25 05:51:38.314: INFO: Init container init-volume-preprovisionedpv-hgcm ready: false, restart count 0 Mar 25 05:51:38.314: INFO: Init container test-init-subpath-preprovisionedpv-hgcm ready: false, restart count 0 Mar 25 05:51:38.314: INFO: Container test-container-subpath-preprovisionedpv-hgcm ready: false, restart count 0 Mar 25 05:51:38.314: INFO: kube-proxy-i-03d301e6f030748f1 started at 2023-03-25 05:31:35 +0000 UTC (0+1 container statuses recorded) Mar 25 05:51:38.314: INFO: Container kube-proxy ready: true, restart count 0 Mar 25 05:51:38.314: INFO: ebs-csi-node-lljwl started at 2023-03-25 05:31:35 +0000 UTC (0+3 container statuses recorded) Mar 25 05:51:38.314: INFO: Container ebs-plugin ready: true, restart count 0 Mar 25 05:51:38.314: INFO: Container liveness-probe ready: true, restart count 0 Mar 25 05:51:38.314: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 25 05:51:38.314: INFO: hostexec-i-03d301e6f030748f1-dp26z started at 2023-03-25 05:51:22 +0000 UTC (0+1 container statuses recorded) Mar 25 05:51:38.314: INFO: Container agnhost-container ready: true, restart count 0 Mar 25 05:51:38.314: INFO: webserver-pod started at 2023-03-25 05:47:27 +0000 UTC (0+1 container statuses recorded) Mar 25 05:51:38.314: INFO: Container agnhost ready: false, restart count 0 Mar 25 05:51:38.314: INFO: startup-af050f25-aca5-4b88-833f-99d48995ab82 started at 2023-03-25 05:49:11 +0000 UTC (0+1 container statuses recorded) Mar 25 05:51:38.314: INFO: Container busybox ready: false, restart count 0 Mar 25 05:51:38.314: INFO: csi-hostpathplugin-0 started at 2023-03-25 05:51:23 +0000 UTC (0+7 container statuses recorded) Mar 25 05:51:38.314: INFO: Container csi-attacher ready: true, restart count 0 Mar 25 05:51:38.314: INFO: Container csi-provisioner ready: true, restart count 0 Mar 25 05:51:38.314: INFO: Container csi-resizer ready: true, restart count 0 Mar 25 05:51:38.314: INFO: Container csi-snapshotter ready: true, restart count 0 Mar 25 05:51:38.314: INFO: Container hostpath ready: true, restart count 0 Mar 25 05:51:38.314: INFO: Container liveness-probe ready: true, restart count 0 Mar 25 05:51:38.314: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 25 05:51:38.314: INFO: pod-configmaps-3fedbd34-8742-431b-9166-666b1586ef4e started at 2023-03-25 05:48:18 +0000 UTC (0+3 container statuses recorded) Mar 25 05:51:38.314: INFO: Container createcm-volume-test ready: true, restart count 0 Mar 25 05:51:38.314: INFO: Container delcm-volume-test ready: true, restart count 0 Mar 25 05:51:38.314: INFO: Container updcm-volume-test ready: true, restart count 0 Mar 25 05:51:38.314: INFO: pod-should-be-evicted7ebf5d43-7564-49c4-b50b-2029ea5c4e2b started at 2023-03-25 05:51:17 +0000 UTC (0+1 container statuses recorded) Mar 25 05:51:38.314: INFO: Container bar ready: true, restart count 0 Mar 25 05:51:38.314: INFO: calico-node-h96vq started at 2023-03-25 05:31:35 +0000 UTC (3+1 container statuses recorded) Mar 25 05:51:38.314: INFO: Init container upgrade-ipam ready: true, restart count 0 Mar 25 05:51:38.314: INFO: Init container install-cni ready: true, restart count 0 Mar 25 05:51:38.314: INFO: Init container mount-bpffs ready: true, restart count 0 Mar 25 05:51:38.314: INFO: Container calico-node ready: true, restart count 0 Mar 25 05:51:38.738: INFO: Latency metrics for node i-03d301e6f030748f1 Mar 25 05:51:38.738: INFO: Logging node info for node i-049fdb2c1cde894df Mar 25 05:51:38.787: INFO: Node Info: &Node{ObjectMeta:{i-049fdb2c1cde894df b43a51e6-dda7-4047-b368-72d515695b4e 54405 0 2023-03-25 05:23:34 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-1 failure-domain.beta.kubernetes.io/zone:us-west-1b io.kubernetes.storage.mock/node:some-mock-node kubernetes.io/arch:amd64 kubernetes.io/hostname:i-049fdb2c1cde894df kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:us-west-1b topology.hostpath.csi/node:i-049fdb2c1cde894df topology.kubernetes.io/region:us-west-1 topology.kubernetes.io/zone:us-west-1b] map[csi.volume.kubernetes.io/nodeid:{"csi-mock-csi-mock-volumes-expansion-8433":"i-049fdb2c1cde894df","ebs.csi.aws.com":"i-049fdb2c1cde894df"} node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:172.20.43.61/19 projectcalico.org/IPv4IPIPTunnelAddr:100.97.18.64 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-03-25 05:23:34 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-25 05:23:34 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-03-25 05:23:34 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2023-03-25 05:23:34 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2023-03-25 05:23:45 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.6.0/24\"":{}}}} } {calico-node Update v1 2023-03-25 05:23:52 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4IPIPTunnelAddr":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2023-03-25 05:51:29 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:io.kubernetes.storage.mock/node":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status} {kube-controller-manager Update v1 2023-03-25 05:51:38 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.6.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-1b/i-049fdb2c1cde894df,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.6.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-03-25 05:23:52 +0000 UTC,LastTransitionTime:2023-03-25 05:23:52 +0000 UTC,Reason:CalicoIsUp,Message:Calico is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-25 05:51:29 +0000 UTC,LastTransitionTime:2023-03-25 05:23:34 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-25 05:51:29 +0000 UTC,LastTransitionTime:2023-03-25 05:23:34 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-25 05:51:29 +0000 UTC,LastTransitionTime:2023-03-25 05:23:34 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-25 05:51:29 +0000 UTC,LastTransitionTime:2023-03-25 05:23:45 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.43.61,},NodeAddress{Type:ExternalIP,Address:18.144.75.67,},NodeAddress{Type:InternalDNS,Address:i-049fdb2c1cde894df.us-west-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-049fdb2c1cde894df.us-west-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-18-144-75-67.us-west-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec252e592d19e1e6c583e87505431016,SystemUUID:ec252e59-2d19-e1e6-c583-e87505431016,BootID:45436b5d-1eb0-4b6f-a1d1-a1492b31b5d7,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.28.0-alpha.0,KubeProxyVersion:v1.28.0-alpha.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:112030336,},ContainerImage{Names:[docker.io/calico/cni@sha256:e282ea2914c806b5de2976330a17cfb5e6dcef47147bceb1432ca5c75fd46f50 docker.io/calico/cni:v3.24.5],SizeBytes:87458443,},ContainerImage{Names:[docker.io/calico/node@sha256:5972ad2bcbdc668564d3e26960c9c513b2d7b05581c704747cf7c62ef3a405a6 docker.io/calico/node:v3.24.5],SizeBytes:81584866,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41901587,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:e468dddcd275163a042ab297b2d8c2aca50d5e148d2d22f3b6ba119e2f31fa79 registry.k8s.io/sig-storage/csi-provisioner:v3.4.0],SizeBytes:27427836,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/kube-proxy@sha256:8a03b3147cd7be8e8235ae1cd70cf69aca9ce03ba1cfaffea320540bb5afa496 registry.k8s.io/kube-proxy:v1.28.0-alpha.0],SizeBytes:23896850,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:23881995,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:68d396900aeaa072c1f27289485fdac29834045a6f3ffe369bf389d830ef572d registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.6],SizeBytes:20293261,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18758628,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18233005,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:15273057,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8688564,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:7000082,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6978614,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[kubernetes.io/csi/csi-mock-csi-mock-volumes-expansion-8433^6f2d5757-cad0-11ed-b4c8-3aceddafb7a9 kubernetes.io/csi/ebs.csi.aws.com^vol-07a89faa725070a64],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-mock-csi-mock-volumes-expansion-8433^6f2d5757-cad0-11ed-b4c8-3aceddafb7a9,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-0bdc719e4a134da12,DevicePath:,},},Config:nil,},} Mar 25 05:51:38.787: INFO: Logging kubelet events for node i-049fdb2c1cde894df Mar 25 05:51:38.843: INFO: Logging pods the kubelet thinks is on node i-049fdb2c1cde894df Mar 25 05:51:38.903: INFO: kube-proxy-i-049fdb2c1cde894df started at 2023-03-25 05:23:35 +0000 UTC (0+1 container statuses recorded) Mar 25 05:51:38.903: INFO: Container kube-proxy ready: true, restart count 0 Mar 25 05:51:38.903: INFO: coredns-autoscaler-7cb5c5b969-h485x started at 2023-03-25 05:24:36 +0000 UTC (0+1 container statuses recorded) Mar 25 05:51:38.903: INFO: Container autoscaler ready: true, restart count 0 Mar 25 05:51:38.903: INFO: calico-node-wt5p7 started at 2023-03-25 05:23:35 +0000 UTC (3+1 container statuses recorded) Mar 25 05:51:38.903: INFO: Init container upgrade-ipam ready: true, restart count 0 Mar 25 05:51:38.903: INFO: Init container install-cni ready: true, restart count 0 Mar 25 05:51:38.903: INFO: Init container mount-bpffs ready: true, restart count 0 Mar 25 05:51:38.903: INFO: Container calico-node ready: true, restart count 0 Mar 25 05:51:38.903: INFO: external-client started at 2023-03-25 05:51:27 +0000 UTC (0+1 container statuses recorded) Mar 25 05:51:38.903: INFO: Container external-client ready: false, restart count 0 Mar 25 05:51:38.903: INFO: csi-mockplugin-resizer-0 started at 2023-03-25 05:46:43 +0000 UTC (0+1 container statuses recorded) Mar 25 05:51:38.903: INFO: Container csi-resizer ready: true, restart count 0 Mar 25 05:51:38.903: INFO: csi-mockplugin-0 started at 2023-03-25 05:46:43 +0000 UTC (0+3 container statuses recorded) Mar 25 05:51:38.903: INFO: Container csi-provisioner ready: true, restart count 0 Mar 25 05:51:38.903: INFO: Container driver-registrar ready: true, restart count 0 Mar 25 05:51:38.903: INFO: Container mock ready: true, restart count 0 Mar 25 05:51:38.903: INFO: csi-hostpathplugin-0 started at 2023-03-25 05:51:38 +0000 UTC (0+7 container statuses recorded) Mar 25 05:51:38.903: INFO: Container csi-attacher ready: false, restart count 0 Mar 25 05:51:38.903: INFO: Container csi-provisioner ready: false, restart count 0 Mar 25 05:51:38.903: INFO: Container csi-resizer ready: false, restart count 0 Mar 25 05:51:38.903: INFO: Container csi-snapshotter ready: false, restart count 0 Mar 25 05:51:38.903: INFO: Container hostpath ready: false, restart count 0 Mar 25 05:51:38.903: INFO: Container liveness-probe ready: false, restart count 0 Mar 25 05:51:38.903: INFO: Container node-driver-registrar ready: false, restart count 0 Mar 25 05:51:38.903: INFO: csi-mockplugin-attacher-0 started at 2023-03-25 05:46:43 +0000 UTC (0+1 container statuses recorded) Mar 25 05:51:38.903: INFO: Container csi-attacher ready: true, restart count 0 Mar 25 05:51:38.903: INFO: ss-1 started at 2023-03-25 05:51:36 +0000 UTC (0+1 container statuses recorded) Mar 25 05:51:38.903: INFO: Container webserver ready: false, restart count 0 Mar 25 05:51:38.904: INFO: pvc-volume-tester-zz4pt started at 2023-03-25 05:46:48 +0000 UTC (0+1 container statuses recorded) Mar 25 05:51:38.904: INFO: Container volume-tester ready: true, restart count 0 Mar 25 05:51:38.904: INFO: ebs-csi-node-m9zxz started at 2023-03-25 05:23:35 +0000 UTC (0+3 container statuses recorded) Mar 25 05:51:38.904: INFO: Container ebs-plugin ready: true, restart count 0 Mar 25 05:51:38.904: INFO: Container liveness-probe ready: true, restart count 0 Mar 25 05:51:38.904: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 25 05:51:38.904: INFO: coredns-559769c974-pgqhl started at 2023-03-25 05:24:36 +0000 UTC (0+1 container statuses recorded) Mar 25 05:51:38.904: INFO: Container coredns ready: true, restart count 0 Mar 25 05:51:39.153: INFO: Latency metrics for node i-049fdb2c1cde894df Mar 25 05:51:39.153: INFO: Logging node info for node i-062679bb8c0ba4bce Mar 25 05:51:39.202: INFO: Node Info: &Node{ObjectMeta:{i-062679bb8c0ba4bce ee2da569-bb88-4297-9e6f-d2b15ba7ce86 54385 0 2023-03-25 05:27:31 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-1 failure-domain.beta.kubernetes.io/zone:us-west-1b kubernetes.io/arch:amd64 kubernetes.io/hostname:i-062679bb8c0ba4bce kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:us-west-1b topology.hostpath.csi/node:i-062679bb8c0ba4bce topology.kubernetes.io/region:us-west-1 topology.kubernetes.io/zone:us-west-1b] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-volume-expand-6509":"i-062679bb8c0ba4bce","csi-mock-csi-mock-volumes-workload-9268":"i-062679bb8c0ba4bce","ebs.csi.aws.com":"i-062679bb8c0ba4bce"} node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:172.20.45.43/19 projectcalico.org/IPv4IPIPTunnelAddr:100.106.79.0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-03-25 05:27:31 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-25 05:27:31 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-03-25 05:27:31 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2023-03-25 05:27:31 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2023-03-25 05:27:44 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.7.0/24\"":{}}}} } {calico-node Update v1 2023-03-25 05:27:50 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4IPIPTunnelAddr":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2023-03-25 05:51:32 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status} {kube-controller-manager Update v1 2023-03-25 05:51:37 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.7.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-1b/i-062679bb8c0ba4bce,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.7.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051689472 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946831872 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-03-25 05:27:50 +0000 UTC,LastTransitionTime:2023-03-25 05:27:50 +0000 UTC,Reason:CalicoIsUp,Message:Calico is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-25 05:51:29 +0000 UTC,LastTransitionTime:2023-03-25 05:27:31 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-25 05:51:29 +0000 UTC,LastTransitionTime:2023-03-25 05:27:31 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-25 05:51:29 +0000 UTC,LastTransitionTime:2023-03-25 05:27:31 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-25 05:51:29 +0000 UTC,LastTransitionTime:2023-03-25 05:27:44 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.45.43,},NodeAddress{Type:ExternalIP,Address:3.101.133.108,},NodeAddress{Type:InternalDNS,Address:i-062679bb8c0ba4bce.us-west-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-062679bb8c0ba4bce.us-west-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-3-101-133-108.us-west-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2528f599e9b1e199793fb5cd06d205,SystemUUID:ec2528f5-99e9-b1e1-9979-3fb5cd06d205,BootID:ff56b70b-c8c5-46db-8cc5-4353c8f535c6,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.28.0-alpha.0,KubeProxyVersion:v1.28.0-alpha.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:112030336,},ContainerImage{Names:[docker.io/calico/cni@sha256:e282ea2914c806b5de2976330a17cfb5e6dcef47147bceb1432ca5c75fd46f50 docker.io/calico/cni:v3.24.5],SizeBytes:87458443,},ContainerImage{Names:[docker.io/calico/node@sha256:5972ad2bcbdc668564d3e26960c9c513b2d7b05581c704747cf7c62ef3a405a6 docker.io/calico/node:v3.24.5],SizeBytes:81584866,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nautilus@sha256:80ba6c8c44f9623f06e868a1aa66026c8ec438ad814f9ec95e9333b415fe3550 registry.k8s.io/e2e-test-images/nautilus:1.7],SizeBytes:49641698,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41901587,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:e468dddcd275163a042ab297b2d8c2aca50d5e148d2d22f3b6ba119e2f31fa79 registry.k8s.io/sig-storage/csi-provisioner:v3.4.0],SizeBytes:27427836,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/kube-proxy@sha256:8a03b3147cd7be8e8235ae1cd70cf69aca9ce03ba1cfaffea320540bb5afa496 registry.k8s.io/kube-proxy:v1.28.0-alpha.0],SizeBytes:23896850,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:23881995,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18758628,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/regression-issue-74839@sha256:b4f1d8d61bdad84bd50442d161d5460e4019d53e989b64220fdbc62fc87d76bf registry.k8s.io/e2e-test-images/regression-issue-74839:1.2],SizeBytes:18651485,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18233005,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:15273057,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8688564,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/build-image/distroless-iptables@sha256:13ac9b3c476d7290a4451d65d09d6016a2cb89836ffbfa4eae55b72731a22080 registry.k8s.io/build-image/distroless-iptables:v0.2.2],SizeBytes:7729580,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:7000082,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6978614,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-volume-expand-6509^184bf3d4-cad1-11ed-bcec-62db49d9647a,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-06f363257c9807b4f,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-mock-csi-mock-volumes-workload-9268^1895dafe-cad1-11ed-95bc-d644bb3827b3,DevicePath:,},},Config:nil,},} Mar 25 05:51:39.202: INFO: Logging kubelet events for node i-062679bb8c0ba4bce Mar 25 05:51:39.262: INFO: Logging pods the kubelet thinks is on node i-062679bb8c0ba4bce Mar 25 05:51:39.337: INFO: calico-node-zwm5j started at 2023-03-25 05:27:32 +0000 UTC (3+1 container statuses recorded) Mar 25 05:51:39.337: INFO: Init container upgrade-ipam ready: true, restart count 0 Mar 25 05:51:39.337: INFO: Init container install-cni ready: true, restart count 0 Mar 25 05:51:39.337: INFO: Init container mount-bpffs ready: true, restart count 0 Mar 25 05:51:39.337: INFO: Container calico-node ready: true, restart count 0 Mar 25 05:51:39.337: INFO: coredns-559769c974-p9vg6 started at 2023-03-25 05:36:16 +0000 UTC (0+1 container statuses recorded) Mar 25 05:51:39.337: INFO: Container coredns ready: true, restart count 0 Mar 25 05:51:39.337: INFO: csi-hostpathplugin-0 started at 2023-03-25 05:51:30 +0000 UTC (0+7 container statuses recorded) Mar 25 05:51:39.337: INFO: Container csi-attacher ready: true, restart count 0 Mar 25 05:51:39.337: INFO: Container csi-provisioner ready: true, restart count 0 Mar 25 05:51:39.337: INFO: Container csi-resizer ready: true, restart count 0 Mar 25 05:51:39.337: INFO: Container csi-snapshotter ready: true, restart count 0 Mar 25 05:51:39.337: INFO: Container hostpath ready: true, restart count 0 Mar 25 05:51:39.337: INFO: Container liveness-probe ready: true, restart count 0 Mar 25 05:51:39.337: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 25 05:51:39.337: INFO: pvc-volume-tester-qv85d started at 2023-03-25 05:51:32 +0000 UTC (0+1 container statuses recorded) Mar 25 05:51:39.337: INFO: Container volume-tester ready: false, restart count 0 Mar 25 05:51:39.337: INFO: ebs-csi-node-7twrq started at 2023-03-25 05:27:32 +0000 UTC (0+3 container statuses recorded) Mar 25 05:51:39.337: INFO: Container ebs-plugin ready: true, restart count 0 Mar 25 05:51:39.337: INFO: Container liveness-probe ready: true, restart count 0 Mar 25 05:51:39.337: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 25 05:51:39.337: INFO: pod-disruption-failure-ignore-3-s48j5 started at 2023-03-25 05:51:36 +0000 UTC (0+1 container statuses recorded) Mar 25 05:51:39.337: INFO: Container c ready: false, restart count 0 Mar 25 05:51:39.337: INFO: ss-0 started at 2023-03-25 05:51:30 +0000 UTC (0+1 container statuses recorded) Mar 25 05:51:39.337: INFO: Container webserver ready: false, restart count 0 Mar 25 05:51:39.337: INFO: pod-disruption-failure-ignore-1-kwbfl started at 2023-03-25 05:51:23 +0000 UTC (0+1 container statuses recorded) Mar 25 05:51:39.337: INFO: Container c ready: false, restart count 0 Mar 25 05:51:39.337: INFO: pod-disruption-failure-ignore-0-dddnk started at 2023-03-25 05:51:23 +0000 UTC (0+1 container statuses recorded) Mar 25 05:51:39.337: INFO: Container c ready: true, restart count 0 Mar 25 05:51:39.337: INFO: pod-disruption-failure-ignore-1-6s2ng started at 2023-03-25 05:51:27 +0000 UTC (0+1 container statuses recorded) Mar 25 05:51:39.337: INFO: Container c ready: false, restart count 0 Mar 25 05:51:39.337: INFO: csi-mockplugin-0 started at 2023-03-25 05:51:27 +0000 UTC (0+3 container statuses recorded) Mar 25 05:51:39.337: INFO: Container csi-provisioner ready: true, restart count 0 Mar 25 05:51:39.337: INFO: Container driver-registrar ready: true, restart count 0 Mar 25 05:51:39.337: INFO: Container mock ready: true, restart count 0 Mar 25 05:51:39.337: INFO: csi-mockplugin-attacher-0 started at 2023-03-25 05:51:27 +0000 UTC (0+1 container statuses recorded) Mar 25 05:51:39.337: INFO: Container csi-attacher ready: true, restart count 0 Mar 25 05:51:39.337: INFO: pod-disruption-failure-ignore-2-tv67s started at 2023-03-25 05:51:32 +0000 UTC (0+1 container statuses recorded) Mar 25 05:51:39.337: INFO: Container c ready: false, restart count 0 Mar 25 05:51:39.337: INFO: pod-47a8fc62-d724-47f4-a865-f0444f9055fe started at 2023-03-25 05:51:34 +0000 UTC (0+1 container statuses recorded) Mar 25 05:51:39.337: INFO: Container write-pod ready: false, restart count 0 Mar 25 05:51:39.337: INFO: hostexec-i-062679bb8c0ba4bce-t7kj9 started at 2023-03-25 05:51:36 +0000 UTC (0+1 container statuses recorded) Mar 25 05:51:39.337: INFO: Container agnhost-container ready: true, restart count 0 Mar 25 05:51:39.337: INFO: kube-proxy-i-062679bb8c0ba4bce started at 2023-03-25 05:27:32 +0000 UTC (0+1 container statuses recorded) Mar 25 05:51:39.337: INFO: Container kube-proxy ready: true, restart count 0 Mar 25 05:51:39.969: INFO: Latency metrics for node i-062679bb8c0ba4bce Mar 25 05:51:39.969: INFO: Logging node info for node i-072608201d2c0f1ad Mar 25 05:51:40.019: INFO: Node Info: &Node{ObjectMeta:{i-072608201d2c0f1ad 264dfba1-3cf0-4c99-8e8c-098f0f792077 53679 0 2023-03-25 05:35:29 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-1 failure-domain.beta.kubernetes.io/zone:us-west-1b kubernetes.io/arch:amd64 kubernetes.io/hostname:i-072608201d2c0f1ad kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:us-west-1b topology.hostpath.csi/node:i-072608201d2c0f1ad topology.kubernetes.io/region:us-west-1 topology.kubernetes.io/zone:us-west-1b] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-5774":"i-072608201d2c0f1ad","ebs.csi.aws.com":"i-072608201d2c0f1ad"} node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:172.20.38.125/19 projectcalico.org/IPv4IPIPTunnelAddr:100.107.220.192 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-03-25 05:35:29 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-25 05:35:29 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-03-25 05:35:29 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2023-03-25 05:35:29 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2023-03-25 05:35:41 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.9.0/24\"":{}}}} } {calico-node Update v1 2023-03-25 05:35:48 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4IPIPTunnelAddr":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2023-03-25 05:51:26 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.9.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-1b/i-072608201d2c0f1ad,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.9.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051689472 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946831872 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-03-25 05:35:47 +0000 UTC,LastTransitionTime:2023-03-25 05:35:47 +0000 UTC,Reason:CalicoIsUp,Message:Calico is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-25 05:51:20 +0000 UTC,LastTransitionTime:2023-03-25 05:35:29 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-25 05:51:20 +0000 UTC,LastTransitionTime:2023-03-25 05:35:29 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-25 05:51:20 +0000 UTC,LastTransitionTime:2023-03-25 05:35:29 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-25 05:51:20 +0000 UTC,LastTransitionTime:2023-03-25 05:35:41 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.38.125,},NodeAddress{Type:ExternalIP,Address:3.101.88.10,},NodeAddress{Type:InternalDNS,Address:i-072608201d2c0f1ad.us-west-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-072608201d2c0f1ad.us-west-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-3-101-88-10.us-west-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2d3818c4f80a253dcc1a60e148dbc8,SystemUUID:ec2d3818-c4f8-0a25-3dcc-1a60e148dbc8,BootID:77d046cf-75ea-4a8c-970a-3190d64e019b,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.28.0-alpha.0,KubeProxyVersion:v1.28.0-alpha.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:112030336,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:51eae8381dcb1078289fa7b4f3df2630cdc18d09fb56f8e56b41c40e191d6c83 registry.k8s.io/etcd:3.5.7-0],SizeBytes:101639218,},ContainerImage{Names:[docker.io/calico/cni@sha256:e282ea2914c806b5de2976330a17cfb5e6dcef47147bceb1432ca5c75fd46f50 docker.io/calico/cni:v3.24.5],SizeBytes:87458443,},ContainerImage{Names:[docker.io/calico/node@sha256:5972ad2bcbdc668564d3e26960c9c513b2d7b05581c704747cf7c62ef3a405a6 docker.io/calico/node:v3.24.5],SizeBytes:81584866,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nautilus@sha256:80ba6c8c44f9623f06e868a1aa66026c8ec438ad814f9ec95e9333b415fe3550 registry.k8s.io/e2e-test-images/nautilus:1.7],SizeBytes:49641698,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41901587,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:e468dddcd275163a042ab297b2d8c2aca50d5e148d2d22f3b6ba119e2f31fa79 registry.k8s.io/sig-storage/csi-provisioner:v3.4.0],SizeBytes:27427836,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/sample-apiserver@sha256:8d70890151aa5d096f331cb9da1b9cd5be0412b7363fe67b5c3befdcaa2a28d0 registry.k8s.io/e2e-test-images/sample-apiserver:1.17.7],SizeBytes:25667066,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/kube-proxy@sha256:8a03b3147cd7be8e8235ae1cd70cf69aca9ce03ba1cfaffea320540bb5afa496 registry.k8s.io/kube-proxy:v1.28.0-alpha.0],SizeBytes:23896850,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:23881995,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18758628,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18233005,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonroot@sha256:ee9f50b3c64b174d296d91ca9f69a914ac30e59095dfb462b2b518ad28a63655 registry.k8s.io/e2e-test-images/nonroot:1.4],SizeBytes:17747885,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8688564,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:7000082,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6978614,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Mar 25 05:51:40.019: INFO: Logging kubelet events for node i-072608201d2c0f1ad Mar 25 05:51:40.081: INFO: Logging pods the kubelet thinks is on node i-072608201d2c0f1ad Mar 25 05:51:40.150: INFO: calico-node-lc9zl started at 2023-03-25 05:35:30 +0000 UTC (3+1 container statuses recorded) Mar 25 05:51:40.150: INFO: Init container upgrade-ipam ready: true, restart count 0 Mar 25 05:51:40.150: INFO: Init container install-cni ready: true, restart count 0 Mar 25 05:51:40.150: INFO: Init container mount-bpffs ready: true, restart count 0 Mar 25 05:51:40.150: INFO: Container calico-node ready: true, restart count 0 Mar 25 05:51:40.150: INFO: pod-configmaps-a8470e12-21d1-4453-85ec-2fef6a8b9767 started at 2023-03-25 05:48:12 +0000 UTC (0+1 container statuses recorded) Mar 25 05:51:40.150: INFO: Container agnhost-container ready: true, restart count 0 Mar 25 05:51:40.150: INFO: csi-hostpathplugin-0 started at 2023-03-25 05:51:05 +0000 UTC (0+7 container statuses recorded) Mar 25 05:51:40.150: INFO: Container csi-attacher ready: true, restart count 0 Mar 25 05:51:40.150: INFO: Container csi-provisioner ready: true, restart count 0 Mar 25 05:51:40.150: INFO: Container csi-resizer ready: true, restart count 0 Mar 25 05:51:40.150: INFO: Container csi-snapshotter ready: true, restart count 0 Mar 25 05:51:40.150: INFO: Container hostpath ready: true, restart count 0 Mar 25 05:51:40.150: INFO: Container liveness-probe ready: true, restart count 0 Mar 25 05:51:40.150: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 25 05:51:40.150: INFO: liveness-042b201c-6dcd-440e-ae49-84b93db25047 started at 2023-03-25 05:47:48 +0000 UTC (0+1 container statuses recorded) Mar 25 05:51:40.150: INFO: Container agnhost-container ready: true, restart count 0 Mar 25 05:51:40.150: INFO: kube-proxy-i-072608201d2c0f1ad started at 2023-03-25 05:35:30 +0000 UTC (0+1 container statuses recorded) Mar 25 05:51:40.150: INFO: Container kube-proxy ready: true, restart count 0 Mar 25 05:51:40.150: INFO: inline-volume-tester-s9fwf started at 2023-03-25 05:51:05 +0000 UTC (0+1 container statuses recorded) Mar 25 05:51:40.150: INFO: Container csi-volume-tester ready: true, restart count 0 Mar 25 05:51:40.150: INFO: ebs-csi-node-r7mcm started at 2023-03-25 05:35:30 +0000 UTC (0+3 container statuses recorded) Mar 25 05:51:40.150: INFO: Container ebs-plugin ready: true, restart count 0 Mar 25 05:51:40.150: INFO: Container liveness-probe ready: true, restart count 0 Mar 25 05:51:40.150: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 25 05:51:40.782: INFO: Latency metrics for node i-072608201d2c0f1ad END STEP: dump namespace information after failure - test/e2e/framework/framework.go:288 @ 03/25/23 05:51:40.782 (3.319s) < Exit [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] subPath - dump namespaces | framework.go:209 @ 03/25/23 05:51:40.782 (3.319s) > Enter [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] subPath - tear down framework | framework.go:206 @ 03/25/23 05:51:40.782 STEP: Destroying namespace "provisioning-4501" for this suite. - test/e2e/framework/framework.go:351 @ 03/25/23 05:51:40.782 < Exit [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] subPath - tear down framework | framework.go:206 @ 03/25/23 05:51:40.843 (61ms) > Enter [ReportAfterEach] TOP-LEVEL - test/e2e/e2e_test.go:144 @ 03/25/23 05:51:40.843 < Exit [ReportAfterEach] TOP-LEVEL - test/e2e/e2e_test.go:144 @ 03/25/23 05:51:40.843 (0s)
Filter through log files | View test history on testgrid
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[It\]\sExternal\sStorage\s\[Driver\:\sebs\.csi\.aws\.com\]\s\[Testpattern\:\sDynamic\sPV\s\(default\sfs\)\]\ssubPath\sshould\ssupport\snon\-existent\spath$'
[FAILED] failed to get logs from pod-subpath-test-dynamicpv-mnls for test-container-volume-dynamicpv-mnls: Get "https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/provisioning-5126/pods/pod-subpath-test-dynamicpv-mnls/log?container=test-container-volume-dynamicpv-mnls&previous=false": stream error: stream ID 3327; INTERNAL_ERROR; received from peer In [It] at: test/e2e/framework/pod/output/output.go:247 @ 03/25/23 05:54:41.901from junit_01.xml
> Enter [BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/storage/framework/testsuite.go:51 @ 03/25/23 05:54:28.742 < Exit [BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/storage/framework/testsuite.go:51 @ 03/25/23 05:54:28.742 (0s) > Enter [BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath - set up framework | framework.go:191 @ 03/25/23 05:54:28.742 STEP: Creating a kubernetes client - test/e2e/framework/framework.go:211 @ 03/25/23 05:54:28.742 Mar 25 05:54:28.742: INFO: >>> kubeConfig: /root/.kube/config STEP: Building a namespace api object, basename provisioning - test/e2e/framework/framework.go:250 @ 03/25/23 05:54:28.743 STEP: Waiting for a default service account to be provisioned in namespace - test/e2e/framework/framework.go:259 @ 03/25/23 05:54:28.898 STEP: Waiting for kube-root-ca.crt to be provisioned in namespace - test/e2e/framework/framework.go:262 @ 03/25/23 05:54:28.993 < Exit [BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath - set up framework | framework.go:191 @ 03/25/23 05:54:29.09 (348ms) > Enter [BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/framework/metrics/init/init.go:33 @ 03/25/23 05:54:29.091 < Exit [BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/framework/metrics/init/init.go:33 @ 03/25/23 05:54:29.091 (0s) > Enter [It] should support non-existent path - test/e2e/storage/testsuites/subpath.go:194 @ 03/25/23 05:54:29.091 Mar 25 05:54:29.091: INFO: Creating resource for dynamic PV Mar 25 05:54:29.091: INFO: Using claimSize:1Gi, test suite supported size:{ 1Mi}, driver(ebs.csi.aws.com) supported size:{ 1Mi} STEP: creating a StorageClass provisioning-5126-e2e-sctlxzg - test/e2e/storage/framework/volume_resource.go:102 @ 03/25/23 05:54:29.091 STEP: creating a claim - test/e2e/storage/framework/volume_resource.go:294 @ 03/25/23 05:54:29.142 Mar 25 05:54:29.142: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil STEP: Creating pod pod-subpath-test-dynamicpv-mnls - test/e2e/storage/testsuites/subpath.go:691 @ 03/25/23 05:54:29.249 STEP: Creating a pod to test subpath - test/e2e/framework/pod/output/output.go:243 @ 03/25/23 05:54:29.249 STEP: Saw pod success - test/e2e/framework/pod/wait.go:409 @ 03/25/23 05:54:41.676 Mar 25 05:54:41.726: INFO: Trying to get logs from node i-072608201d2c0f1ad pod pod-subpath-test-dynamicpv-mnls container test-container-volume-dynamicpv-mnls: <nil> Mar 25 05:54:41.777: INFO: Failed to get logs from node "i-072608201d2c0f1ad" pod "pod-subpath-test-dynamicpv-mnls" container "test-container-volume-dynamicpv-mnls". Get "https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/provisioning-5126/pods/pod-subpath-test-dynamicpv-mnls/log?container=test-container-volume-dynamicpv-mnls&previous=false": stream error: stream ID 3327; INTERNAL_ERROR; received from peer STEP: delete the pod - test/e2e/framework/pod/output/output.go:168 @ 03/25/23 05:54:41.778 Mar 25 05:54:41.901: INFO: Unexpected error: <*fmt.wrapError | 0xc004bbd2a0>: failed to get logs from pod-subpath-test-dynamicpv-mnls for test-container-volume-dynamicpv-mnls: Get "https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/provisioning-5126/pods/pod-subpath-test-dynamicpv-mnls/log?container=test-container-volume-dynamicpv-mnls&previous=false": stream error: stream ID 3327; INTERNAL_ERROR; received from peer { msg: "failed to get logs from pod-subpath-test-dynamicpv-mnls for test-container-volume-dynamicpv-mnls: Get \"https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/provisioning-5126/pods/pod-subpath-test-dynamicpv-mnls/log?container=test-container-volume-dynamicpv-mnls&previous=false\": stream error: stream ID 3327; INTERNAL_ERROR; received from peer", err: <*url.Error | 0xc003664780>{ Op: "Get", URL: "https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/provisioning-5126/pods/pod-subpath-test-dynamicpv-mnls/log?container=test-container-volume-dynamicpv-mnls&previous=false", Err: <http2.StreamError>{ StreamID: 3327, Code: 2, Cause: <*errors.errorString | 0xc0001b3650>{ s: "received from peer", }, }, }, } [FAILED] failed to get logs from pod-subpath-test-dynamicpv-mnls for test-container-volume-dynamicpv-mnls: Get "https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/provisioning-5126/pods/pod-subpath-test-dynamicpv-mnls/log?container=test-container-volume-dynamicpv-mnls&previous=false": stream error: stream ID 3327; INTERNAL_ERROR; received from peer In [It] at: test/e2e/framework/pod/output/output.go:247 @ 03/25/23 05:54:41.901 < Exit [It] should support non-existent path - test/e2e/storage/testsuites/subpath.go:194 @ 03/25/23 05:54:41.901 (12.811s) > Enter [AfterEach] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/framework/node/init/init.go:33 @ 03/25/23 05:54:41.901 Mar 25 05:54:41.901: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready < Exit [AfterEach] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/framework/node/init/init.go:33 @ 03/25/23 05:54:41.953 (51ms) > Enter [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/storage/testsuites/subpath.go:196 @ 03/25/23 05:54:41.953 STEP: Deleting pod - test/e2e/storage/testsuites/subpath.go:172 @ 03/25/23 05:54:41.953 Mar 25 05:54:41.953: INFO: Deleting pod "pod-subpath-test-dynamicpv-mnls" in namespace "provisioning-5126" STEP: Deleting pvc - test/e2e/storage/framework/volume_resource.go:181 @ 03/25/23 05:54:42.056 Mar 25 05:54:42.171: INFO: Deleting PersistentVolumeClaim "ebs.csi.aws.comkggv4" Mar 25 05:54:42.244: INFO: Waiting up to 5m0s for PersistentVolume pvc-57a58cf0-4d7f-4016-b1da-862b8e85e16b to get deleted Mar 25 05:54:42.293: INFO: PersistentVolume pvc-57a58cf0-4d7f-4016-b1da-862b8e85e16b found and phase=Released (49.122526ms) Mar 25 05:54:47.343: INFO: PersistentVolume pvc-57a58cf0-4d7f-4016-b1da-862b8e85e16b found and phase=Released (5.099081572s) Mar 25 05:54:52.392: INFO: PersistentVolume pvc-57a58cf0-4d7f-4016-b1da-862b8e85e16b found and phase=Released (10.148261466s) Mar 25 05:54:57.441: INFO: PersistentVolume pvc-57a58cf0-4d7f-4016-b1da-862b8e85e16b found and phase=Released (15.197536293s) Mar 25 05:55:02.490: INFO: PersistentVolume pvc-57a58cf0-4d7f-4016-b1da-862b8e85e16b was removed STEP: Deleting sc - test/e2e/storage/framework/volume_resource.go:228 @ 03/25/23 05:55:02.491 < Exit [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/storage/testsuites/subpath.go:196 @ 03/25/23 05:55:02.54 (20.588s) > Enter [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/framework/metrics/init/init.go:35 @ 03/25/23 05:55:02.54 < Exit [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/framework/metrics/init/init.go:35 @ 03/25/23 05:55:02.54 (0s) > Enter [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] subPath - dump namespaces | framework.go:209 @ 03/25/23 05:55:02.54 STEP: dump namespace information after failure - test/e2e/framework/framework.go:288 @ 03/25/23 05:55:02.54 STEP: Collecting events from namespace "provisioning-5126". - test/e2e/framework/debug/dump.go:42 @ 03/25/23 05:55:02.54 STEP: Found 15 events. - test/e2e/framework/debug/dump.go:46 @ 03/25/23 05:55:02.589 Mar 25 05:55:02.589: INFO: At 2023-03-25 05:54:29 +0000 UTC - event for ebs.csi.aws.comkggv4: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding Mar 25 05:55:02.589: INFO: At 2023-03-25 05:54:29 +0000 UTC - event for ebs.csi.aws.comkggv4: {ebs.csi.aws.com_i-0168f82458996cc53_0b589c46-3396-4499-94ba-eb51a48f7a58 } Provisioning: External provisioner is provisioning volume for claim "provisioning-5126/ebs.csi.aws.comkggv4" Mar 25 05:55:02.589: INFO: At 2023-03-25 05:54:29 +0000 UTC - event for ebs.csi.aws.comkggv4: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator Mar 25 05:55:02.589: INFO: At 2023-03-25 05:54:32 +0000 UTC - event for ebs.csi.aws.comkggv4: {ebs.csi.aws.com_i-0168f82458996cc53_0b589c46-3396-4499-94ba-eb51a48f7a58 } ProvisioningSucceeded: Successfully provisioned volume pvc-57a58cf0-4d7f-4016-b1da-862b8e85e16b Mar 25 05:55:02.589: INFO: At 2023-03-25 05:54:33 +0000 UTC - event for pod-subpath-test-dynamicpv-mnls: {default-scheduler } Scheduled: Successfully assigned provisioning-5126/pod-subpath-test-dynamicpv-mnls to i-072608201d2c0f1ad Mar 25 05:55:02.589: INFO: At 2023-03-25 05:54:35 +0000 UTC - event for pod-subpath-test-dynamicpv-mnls: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-57a58cf0-4d7f-4016-b1da-862b8e85e16b" Mar 25 05:55:02.589: INFO: At 2023-03-25 05:54:36 +0000 UTC - event for pod-subpath-test-dynamicpv-mnls: {kubelet i-072608201d2c0f1ad} Pulled: Container image "registry.k8s.io/e2e-test-images/agnhost:2.43" already present on machine Mar 25 05:55:02.589: INFO: At 2023-03-25 05:54:36 +0000 UTC - event for pod-subpath-test-dynamicpv-mnls: {kubelet i-072608201d2c0f1ad} Created: Created container test-init-subpath-dynamicpv-mnls Mar 25 05:55:02.589: INFO: At 2023-03-25 05:54:36 +0000 UTC - event for pod-subpath-test-dynamicpv-mnls: {kubelet i-072608201d2c0f1ad} Started: Started container test-init-subpath-dynamicpv-mnls Mar 25 05:55:02.589: INFO: At 2023-03-25 05:54:37 +0000 UTC - event for pod-subpath-test-dynamicpv-mnls: {kubelet i-072608201d2c0f1ad} Pulled: Container image "registry.k8s.io/e2e-test-images/agnhost:2.43" already present on machine Mar 25 05:55:02.589: INFO: At 2023-03-25 05:54:37 +0000 UTC - event for pod-subpath-test-dynamicpv-mnls: {kubelet i-072608201d2c0f1ad} Created: Created container test-container-subpath-dynamicpv-mnls Mar 25 05:55:02.589: INFO: At 2023-03-25 05:54:37 +0000 UTC - event for pod-subpath-test-dynamicpv-mnls: {kubelet i-072608201d2c0f1ad} Started: Started container test-container-subpath-dynamicpv-mnls Mar 25 05:55:02.589: INFO: At 2023-03-25 05:54:37 +0000 UTC - event for pod-subpath-test-dynamicpv-mnls: {kubelet i-072608201d2c0f1ad} Pulled: Container image "registry.k8s.io/e2e-test-images/agnhost:2.43" already present on machine Mar 25 05:55:02.589: INFO: At 2023-03-25 05:54:37 +0000 UTC - event for pod-subpath-test-dynamicpv-mnls: {kubelet i-072608201d2c0f1ad} Created: Created container test-container-volume-dynamicpv-mnls Mar 25 05:55:02.589: INFO: At 2023-03-25 05:54:37 +0000 UTC - event for pod-subpath-test-dynamicpv-mnls: {kubelet i-072608201d2c0f1ad} Started: Started container test-container-volume-dynamicpv-mnls Mar 25 05:55:02.638: INFO: POD NODE PHASE GRACE CONDITIONS Mar 25 05:55:02.638: INFO: Mar 25 05:55:02.688: INFO: Logging node info for node i-0168f82458996cc53 Mar 25 05:55:02.794: INFO: Node Info: &Node{ObjectMeta:{i-0168f82458996cc53 f4eb302c-c477-45ac-bcac-ffed65f6580f 56232 0 2023-03-25 05:18:57 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-1 failure-domain.beta.kubernetes.io/zone:us-west-1b kops.k8s.io/kops-controller-pki: kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0168f82458996cc53 kubernetes.io/os:linux node-role.kubernetes.io/control-plane: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:us-west-1b topology.kubernetes.io/region:us-west-1 topology.kubernetes.io/zone:us-west-1b] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0168f82458996cc53"} node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:172.20.48.81/19 projectcalico.org/IPv4IPIPTunnelAddr:100.100.215.192 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-03-25 05:18:57 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {protokube Update v1 2023-03-25 05:18:58 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {calico-node Update v1 2023-03-25 05:20:22 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4IPIPTunnelAddr":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {aws-cloud-controller-manager Update v1 2023-03-25 05:20:26 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-25 05:20:26 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2023-03-25 05:20:31 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.5.0/24\"":{}},"f:taints":{}}} } {kubelet Update v1 2023-03-25 05:52:18 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.5.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-1b/i-0168f82458996cc53,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/control-plane,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.5.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3892310016 0} {<nil>} 3801084Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3787452416 0} {<nil>} 3698684Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-03-25 05:20:22 +0000 UTC,LastTransitionTime:2023-03-25 05:20:22 +0000 UTC,Reason:CalicoIsUp,Message:Calico is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-25 05:52:18 +0000 UTC,LastTransitionTime:2023-03-25 05:18:50 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-25 05:52:18 +0000 UTC,LastTransitionTime:2023-03-25 05:18:50 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-25 05:52:18 +0000 UTC,LastTransitionTime:2023-03-25 05:18:50 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-25 05:52:18 +0000 UTC,LastTransitionTime:2023-03-25 05:20:29 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.48.81,},NodeAddress{Type:ExternalIP,Address:3.101.135.77,},NodeAddress{Type:InternalDNS,Address:i-0168f82458996cc53.us-west-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-0168f82458996cc53.us-west-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-3-101-135-77.us-west-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2d1ccc7ebf313420bd1f530a94fbfd,SystemUUID:ec2d1ccc-7ebf-3134-20bd-1f530a94fbfd,BootID:15cee321-56e7-4290-ab48-47b60b0bc657,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.28.0-alpha.0,KubeProxyVersion:v1.28.0-alpha.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcd@sha256:dd75ec974b0a2a6f6bb47001ba09207976e625db898d1b16735528c009cb171c registry.k8s.io/etcd:3.5.6-0],SizeBytes:102542580,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:6f72b851544986cb0921b53ea655ec04c36131248f16d4ad110cb3ca0c369dc1 registry.k8s.io/etcd:3.5.4-0],SizeBytes:102157811,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:13f53ed1d91e2e11aac476ee9a0269fdda6cc4874eba903efd40daf50c55eee5 registry.k8s.io/etcd:3.5.3-0],SizeBytes:102143581,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:51eae8381dcb1078289fa7b4f3df2630cdc18d09fb56f8e56b41c40e191d6c83 registry.k8s.io/etcd:3.5.7-0],SizeBytes:101639218,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:4afb99b4690b418ffc2ceb67e1a17376457e441c1f09ab55447f0aaf992fa646 registry.k8s.io/etcd:3.4.3-0],SizeBytes:100947667,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:9ce33ba33d8e738a5b85ed50b5080ac746deceed4a7496c550927a7a19ca3b6d registry.k8s.io/etcd:3.5.0-0],SizeBytes:99868722,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:64b9ea357325d5db9f8a723dcf503b5a449177b17ac87d69481e126bb724c263 registry.k8s.io/etcd:3.5.1-0],SizeBytes:98888614,},ContainerImage{Names:[docker.io/calico/cni@sha256:e282ea2914c806b5de2976330a17cfb5e6dcef47147bceb1432ca5c75fd46f50 docker.io/calico/cni:v3.24.5],SizeBytes:87458443,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:4ad90a11b55313b182afc186b9876c8e891531b8db4c9bf1541953021618d0e2 registry.k8s.io/etcd:3.4.13-0],SizeBytes:86742272,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:0d2ab6b15b4fe0b903b9e81192d942fb39c29be61fe341cd16db5d0165b0435c registry.k8s.io/etcd:3.3.17-0],SizeBytes:85524408,},ContainerImage{Names:[docker.io/calico/node@sha256:5972ad2bcbdc668564d3e26960c9c513b2d7b05581c704747cf7c62ef3a405a6 docker.io/calico/node:v3.24.5],SizeBytes:81584866,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:17da501f5d2a675be46040422a27b7cc21b8a43895ac998b171db1c346f361f7 registry.k8s.io/etcd:3.3.10-0],SizeBytes:76162358,},ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager-slim@sha256:8727adc059334e0e3329dbccce116a734da4a090da5e86c8cddccaf333caa666 registry.k8s.io/etcdadm/etcd-manager-slim:v3.0.20230201],SizeBytes:63926165,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:905d7ca17fd02bc24c0eba9a062753aba15db3e31422390bc3238eb762339b20 registry.k8s.io/etcd:3.2.24-1],SizeBytes:63423287,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.27.0-alpha.1],SizeBytes:45219545,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.27.0-alpha.1],SizeBytes:43628585,},ContainerImage{Names:[registry.k8s.io/kube-apiserver@sha256:49934a29c559e79ed19fc38b4697b18d8fe174b17bf56192588575f980c93956 registry.k8s.io/kube-apiserver:v1.28.0-alpha.0],SizeBytes:33212506,},ContainerImage{Names:[docker.io/calico/kube-controllers@sha256:2b6acd7f677f76ffe12ecf3ea7df92eb9b1bdb07336d1ac2a54c7631fb753f7e docker.io/calico/kube-controllers:v3.24.5],SizeBytes:31137169,},ContainerImage{Names:[registry.k8s.io/kube-controller-manager@sha256:974748384634982c73527fdb3c9ec0ec476bc0d52b3dd1d611a380b7f99af1f3 registry.k8s.io/kube-controller-manager:v1.28.0-alpha.0],SizeBytes:30820280,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/kube-proxy@sha256:8a03b3147cd7be8e8235ae1cd70cf69aca9ce03ba1cfaffea320540bb5afa496 registry.k8s.io/kube-proxy:v1.28.0-alpha.0],SizeBytes:23896850,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:23345856,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:22381475,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:22085298,},ContainerImage{Names:[registry.k8s.io/provider-aws/cloud-controller-manager@sha256:fdeb61e3e42ecd9cca868d550ebdb88dd6341d9e91fcfa9a37e227dab2ad22cb registry.k8s.io/provider-aws/cloud-controller-manager:v1.26.0],SizeBytes:20154862,},ContainerImage{Names:[registry.k8s.io/kube-scheduler@sha256:cbf4007abb076188f16b975bc2b74452603cc7b34c60a29d92f14c38ad02dc11 registry.k8s.io/kube-scheduler:v1.28.0-alpha.0],SizeBytes:18078674,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.27.0-alpha.1],SizeBytes:5074298,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Mar 25 05:55:02.794: INFO: Logging kubelet events for node i-0168f82458996cc53 Mar 25 05:55:02.871: INFO: Logging pods the kubelet thinks is on node i-0168f82458996cc53 Mar 25 05:55:02.928: INFO: aws-cloud-controller-manager-26vbv started at 2023-03-25 05:20:06 +0000 UTC (0+1 container statuses recorded) Mar 25 05:55:02.928: INFO: Container aws-cloud-controller-manager ready: true, restart count 0 Mar 25 05:55:02.928: INFO: ebs-csi-node-zjtj6 started at 2023-03-25 05:20:06 +0000 UTC (0+3 container statuses recorded) Mar 25 05:55:02.928: INFO: Container ebs-plugin ready: true, restart count 0 Mar 25 05:55:02.928: INFO: Container liveness-probe ready: true, restart count 0 Mar 25 05:55:02.928: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 25 05:55:02.928: INFO: kops-controller-cgw95 started at 2023-03-25 05:20:06 +0000 UTC (0+1 container statuses recorded) Mar 25 05:55:02.928: INFO: Container kops-controller ready: true, restart count 0 Mar 25 05:55:02.928: INFO: calico-kube-controllers-55f488656-rr8rz started at 2023-03-25 05:20:29 +0000 UTC (0+1 container statuses recorded) Mar 25 05:55:02.928: INFO: Container calico-kube-controllers ready: true, restart count 0 Mar 25 05:55:02.928: INFO: kube-proxy-i-0168f82458996cc53 started at 2023-03-25 05:17:36 +0000 UTC (0+1 container statuses recorded) Mar 25 05:55:02.928: INFO: Container kube-proxy ready: true, restart count 0 Mar 25 05:55:02.928: INFO: kube-scheduler-i-0168f82458996cc53 started at 2023-03-25 05:17:36 +0000 UTC (0+1 container statuses recorded) Mar 25 05:55:02.928: INFO: Container kube-scheduler ready: true, restart count 0 Mar 25 05:55:02.928: INFO: etcd-manager-main-i-0168f82458996cc53 started at 2023-03-25 05:17:36 +0000 UTC (11+1 container statuses recorded) Mar 25 05:55:02.928: INFO: Init container init-etcd-3-2-24 ready: true, restart count 0 Mar 25 05:55:02.928: INFO: Init container init-etcd-3-3-10 ready: true, restart count 0 Mar 25 05:55:02.928: INFO: Init container init-etcd-3-3-17 ready: true, restart count 0 Mar 25 05:55:02.928: INFO: Init container init-etcd-3-4-13 ready: true, restart count 0 Mar 25 05:55:02.928: INFO: Init container init-etcd-3-4-3 ready: true, restart count 0 Mar 25 05:55:02.928: INFO: Init container init-etcd-3-5-0 ready: true, restart count 0 Mar 25 05:55:02.928: INFO: Init container init-etcd-3-5-1 ready: true, restart count 0 Mar 25 05:55:02.928: INFO: Init container init-etcd-3-5-3 ready: true, restart count 0 Mar 25 05:55:02.928: INFO: Init container init-etcd-3-5-4 ready: true, restart count 0 Mar 25 05:55:02.928: INFO: Init container init-etcd-3-5-6 ready: true, restart count 0 Mar 25 05:55:02.928: INFO: Init container init-etcd-3-5-7 ready: true, restart count 0 Mar 25 05:55:02.928: INFO: Container etcd-manager ready: true, restart count 0 Mar 25 05:55:02.928: INFO: dns-controller-69987775c6-hbn69 started at 2023-03-25 05:19:47 +0000 UTC (0+1 container statuses recorded) Mar 25 05:55:02.928: INFO: Container dns-controller ready: true, restart count 0 Mar 25 05:55:02.928: INFO: ebs-csi-controller-79f984545f-9q5ks started at 2023-03-25 05:21:26 +0000 UTC (0+5 container statuses recorded) Mar 25 05:55:02.928: INFO: Container csi-attacher ready: true, restart count 0 Mar 25 05:55:02.928: INFO: Container csi-provisioner ready: true, restart count 0 Mar 25 05:55:02.928: INFO: Container csi-resizer ready: true, restart count 0 Mar 25 05:55:02.928: INFO: Container ebs-plugin ready: true, restart count 0 Mar 25 05:55:02.928: INFO: Container liveness-probe ready: true, restart count 0 Mar 25 05:55:02.928: INFO: kube-apiserver-i-0168f82458996cc53 started at 2023-03-25 05:17:36 +0000 UTC (0+2 container statuses recorded) Mar 25 05:55:02.928: INFO: Container healthcheck ready: true, restart count 0 Mar 25 05:55:02.928: INFO: Container kube-apiserver ready: true, restart count 2 Mar 25 05:55:02.928: INFO: calico-node-rdjq2 started at 2023-03-25 05:20:06 +0000 UTC (3+1 container statuses recorded) Mar 25 05:55:02.928: INFO: Init container upgrade-ipam ready: true, restart count 0 Mar 25 05:55:02.928: INFO: Init container install-cni ready: true, restart count 0 Mar 25 05:55:02.928: INFO: Init container mount-bpffs ready: true, restart count 0 Mar 25 05:55:02.928: INFO: Container calico-node ready: true, restart count 0 Mar 25 05:55:02.928: INFO: kube-controller-manager-i-0168f82458996cc53 started at 2023-03-25 05:17:36 +0000 UTC (0+1 container statuses recorded) Mar 25 05:55:02.928: INFO: Container kube-controller-manager ready: true, restart count 4 Mar 25 05:55:02.928: INFO: etcd-manager-events-i-0168f82458996cc53 started at 2023-03-25 05:17:36 +0000 UTC (11+1 container statuses recorded) Mar 25 05:55:02.928: INFO: Init container init-etcd-3-2-24 ready: true, restart count 0 Mar 25 05:55:02.928: INFO: Init container init-etcd-3-3-10 ready: true, restart count 0 Mar 25 05:55:02.928: INFO: Init container init-etcd-3-3-17 ready: true, restart count 0 Mar 25 05:55:02.928: INFO: Init container init-etcd-3-4-13 ready: true, restart count 0 Mar 25 05:55:02.928: INFO: Init container init-etcd-3-4-3 ready: true, restart count 0 Mar 25 05:55:02.928: INFO: Init container init-etcd-3-5-0 ready: true, restart count 0 Mar 25 05:55:02.928: INFO: Init container init-etcd-3-5-1 ready: true, restart count 0 Mar 25 05:55:02.928: INFO: Init container init-etcd-3-5-3 ready: true, restart count 0 Mar 25 05:55:02.928: INFO: Init container init-etcd-3-5-4 ready: true, restart count 0 Mar 25 05:55:02.928: INFO: Init container init-etcd-3-5-6 ready: true, restart count 0 Mar 25 05:55:02.928: INFO: Init container init-etcd-3-5-7 ready: true, restart count 0 Mar 25 05:55:02.928: INFO: Container etcd-manager ready: true, restart count 0 Mar 25 05:55:03.159: INFO: Latency metrics for node i-0168f82458996cc53 Mar 25 05:55:03.159: INFO: Logging node info for node i-03d301e6f030748f1 Mar 25 05:55:03.208: INFO: Node Info: &Node{ObjectMeta:{i-03d301e6f030748f1 897f52bd-65e2-4107-843a-37e36aa74032 63377 0 2023-03-25 05:31:35 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-1 failure-domain.beta.kubernetes.io/zone:us-west-1b kubernetes.io/arch:amd64 kubernetes.io/hostname:i-03d301e6f030748f1 kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:us-west-1b topology.hostpath.csi/node:i-03d301e6f030748f1 topology.kubernetes.io/region:us-west-1 topology.kubernetes.io/zone:us-west-1b] map[csi.volume.kubernetes.io/nodeid:{"csi-mock-csi-mock-volumes-capacity-7853":"i-03d301e6f030748f1","ebs.csi.aws.com":"i-03d301e6f030748f1"} node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:172.20.50.199/19 projectcalico.org/IPv4IPIPTunnelAddr:100.110.167.64 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-03-25 05:31:35 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-25 05:31:35 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-03-25 05:31:35 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2023-03-25 05:31:35 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2023-03-25 05:31:48 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.8.0/24\"":{}}}} } {calico-node Update v1 2023-03-25 05:31:56 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4IPIPTunnelAddr":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2023-03-25 05:54:54 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status} {kube-controller-manager Update v1 2023-03-25 05:54:55 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.8.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-1b/i-03d301e6f030748f1,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.8.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-03-25 05:31:56 +0000 UTC,LastTransitionTime:2023-03-25 05:31:56 +0000 UTC,Reason:CalicoIsUp,Message:Calico is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-25 05:54:54 +0000 UTC,LastTransitionTime:2023-03-25 05:31:35 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-25 05:54:54 +0000 UTC,LastTransitionTime:2023-03-25 05:31:35 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-25 05:54:54 +0000 UTC,LastTransitionTime:2023-03-25 05:31:35 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-25 05:54:54 +0000 UTC,LastTransitionTime:2023-03-25 05:31:48 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.50.199,},NodeAddress{Type:ExternalIP,Address:52.53.245.63,},NodeAddress{Type:InternalDNS,Address:i-03d301e6f030748f1.us-west-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-03d301e6f030748f1.us-west-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-52-53-245-63.us-west-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec27152f7af9b9bfd1355ee19c256086,SystemUUID:ec27152f-7af9-b9bf-d135-5ee19c256086,BootID:1c303852-f2ce-45f9-ab14-6e8de97a0a3e,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.28.0-alpha.0,KubeProxyVersion:v1.28.0-alpha.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:112030336,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:51eae8381dcb1078289fa7b4f3df2630cdc18d09fb56f8e56b41c40e191d6c83 registry.k8s.io/etcd:3.5.7-0],SizeBytes:101639218,},ContainerImage{Names:[docker.io/calico/cni@sha256:e282ea2914c806b5de2976330a17cfb5e6dcef47147bceb1432ca5c75fd46f50 docker.io/calico/cni:v3.24.5],SizeBytes:87458443,},ContainerImage{Names:[docker.io/calico/node@sha256:5972ad2bcbdc668564d3e26960c9c513b2d7b05581c704747cf7c62ef3a405a6 docker.io/calico/node:v3.24.5],SizeBytes:81584866,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nautilus@sha256:80ba6c8c44f9623f06e868a1aa66026c8ec438ad814f9ec95e9333b415fe3550 registry.k8s.io/e2e-test-images/nautilus:1.7],SizeBytes:49641698,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41901587,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:e468dddcd275163a042ab297b2d8c2aca50d5e148d2d22f3b6ba119e2f31fa79 registry.k8s.io/sig-storage/csi-provisioner:v3.4.0],SizeBytes:27427836,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/sample-apiserver@sha256:8d70890151aa5d096f331cb9da1b9cd5be0412b7363fe67b5c3befdcaa2a28d0 registry.k8s.io/e2e-test-images/sample-apiserver:1.17.7],SizeBytes:25667066,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/kube-proxy@sha256:8a03b3147cd7be8e8235ae1cd70cf69aca9ce03ba1cfaffea320540bb5afa496 registry.k8s.io/kube-proxy:v1.28.0-alpha.0],SizeBytes:23896850,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:23881995,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18758628,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18233005,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonroot@sha256:ee9f50b3c64b174d296d91ca9f69a914ac30e59095dfb462b2b518ad28a63655 registry.k8s.io/e2e-test-images/nonroot:1.4],SizeBytes:17747885,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8688564,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:7000082,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6978614,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac registry.k8s.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[kubernetes.io/csi/ebs.csi.aws.com^vol-04e161d418db0c7b1],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-04e161d418db0c7b1,DevicePath:,},},Config:nil,},} Mar 25 05:55:03.209: INFO: Logging kubelet events for node i-03d301e6f030748f1 Mar 25 05:55:03.277: INFO: Logging pods the kubelet thinks is on node i-03d301e6f030748f1 Mar 25 05:55:03.333: INFO: pfpod started at 2023-03-25 05:54:12 +0000 UTC (0+2 container statuses recorded) Mar 25 05:55:03.333: INFO: Container portforwardtester ready: false, restart count 0 Mar 25 05:55:03.333: INFO: Container readiness ready: false, restart count 0 Mar 25 05:55:03.333: INFO: kube-proxy-i-03d301e6f030748f1 started at 2023-03-25 05:31:35 +0000 UTC (0+1 container statuses recorded) Mar 25 05:55:03.333: INFO: Container kube-proxy ready: true, restart count 0 Mar 25 05:55:03.333: INFO: ebs-csi-node-lljwl started at 2023-03-25 05:31:35 +0000 UTC (0+3 container statuses recorded) Mar 25 05:55:03.333: INFO: Container ebs-plugin ready: true, restart count 0 Mar 25 05:55:03.333: INFO: Container liveness-probe ready: true, restart count 0 Mar 25 05:55:03.333: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 25 05:55:03.333: INFO: pause-pod-1 started at 2023-03-25 05:54:22 +0000 UTC (0+1 container statuses recorded) Mar 25 05:55:03.333: INFO: Container agnhost-container ready: true, restart count 0 Mar 25 05:55:03.333: INFO: calico-node-h96vq started at 2023-03-25 05:31:35 +0000 UTC (3+1 container statuses recorded) Mar 25 05:55:03.333: INFO: Init container upgrade-ipam ready: true, restart count 0 Mar 25 05:55:03.333: INFO: Init container install-cni ready: true, restart count 0 Mar 25 05:55:03.333: INFO: Init container mount-bpffs ready: true, restart count 0 Mar 25 05:55:03.333: INFO: Container calico-node ready: true, restart count 0 Mar 25 05:55:03.333: INFO: csi-mockplugin-0 started at 2023-03-25 05:54:25 +0000 UTC (0+3 container statuses recorded) Mar 25 05:55:03.333: INFO: Container csi-provisioner ready: true, restart count 0 Mar 25 05:55:03.333: INFO: Container driver-registrar ready: true, restart count 0 Mar 25 05:55:03.333: INFO: Container mock ready: true, restart count 0 Mar 25 05:55:03.333: INFO: test-grpc-cbfe4b71-8b33-40b9-96de-a3f96633a513 started at 2023-03-25 05:53:14 +0000 UTC (0+1 container statuses recorded) Mar 25 05:55:03.333: INFO: Container agnhost ready: true, restart count 0 Mar 25 05:55:03.333: INFO: ss-1 started at 2023-03-25 05:54:19 +0000 UTC (0+1 container statuses recorded) Mar 25 05:55:03.333: INFO: Container webserver ready: true, restart count 0 Mar 25 05:55:03.333: INFO: pod-projected-configmaps-258cfcf3-2804-462b-9867-da2861a96093 started at 2023-03-25 05:52:50 +0000 UTC (0+1 container statuses recorded) Mar 25 05:55:03.333: INFO: Container agnhost-container ready: true, restart count 0 Mar 25 05:55:03.710: INFO: Latency metrics for node i-03d301e6f030748f1 Mar 25 05:55:03.710: INFO: Logging node info for node i-049fdb2c1cde894df Mar 25 05:55:03.762: INFO: Node Info: &Node{ObjectMeta:{i-049fdb2c1cde894df b43a51e6-dda7-4047-b368-72d515695b4e 63324 0 2023-03-25 05:23:34 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-1 failure-domain.beta.kubernetes.io/zone:us-west-1b io.kubernetes.storage.mock/node:some-mock-node kubernetes.io/arch:amd64 kubernetes.io/hostname:i-049fdb2c1cde894df kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:us-west-1b topology.hostpath.csi/node:i-049fdb2c1cde894df topology.kubernetes.io/region:us-west-1 topology.kubernetes.io/zone:us-west-1b] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-provisioning-7537":"i-049fdb2c1cde894df","csi-hostpath-volume-6086":"i-049fdb2c1cde894df","csi-mock-csi-mock-volumes-capacity-8277":"i-049fdb2c1cde894df","ebs.csi.aws.com":"i-049fdb2c1cde894df"} node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:172.20.43.61/19 projectcalico.org/IPv4IPIPTunnelAddr:100.97.18.64 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-03-25 05:23:34 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-25 05:23:34 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-03-25 05:23:34 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2023-03-25 05:23:34 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2023-03-25 05:23:45 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.6.0/24\"":{}}}} } {calico-node Update v1 2023-03-25 05:23:52 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4IPIPTunnelAddr":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2023-03-25 05:54:53 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-03-25 05:54:53 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:io.kubernetes.storage.mock/node":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.6.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-1b/i-049fdb2c1cde894df,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.6.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-03-25 05:23:52 +0000 UTC,LastTransitionTime:2023-03-25 05:23:52 +0000 UTC,Reason:CalicoIsUp,Message:Calico is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-25 05:54:53 +0000 UTC,LastTransitionTime:2023-03-25 05:23:34 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-25 05:54:53 +0000 UTC,LastTransitionTime:2023-03-25 05:23:34 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-25 05:54:53 +0000 UTC,LastTransitionTime:2023-03-25 05:23:34 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-25 05:54:53 +0000 UTC,LastTransitionTime:2023-03-25 05:23:45 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.43.61,},NodeAddress{Type:ExternalIP,Address:18.144.75.67,},NodeAddress{Type:InternalDNS,Address:i-049fdb2c1cde894df.us-west-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-049fdb2c1cde894df.us-west-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-18-144-75-67.us-west-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec252e592d19e1e6c583e87505431016,SystemUUID:ec252e59-2d19-e1e6-c583-e87505431016,BootID:45436b5d-1eb0-4b6f-a1d1-a1492b31b5d7,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.28.0-alpha.0,KubeProxyVersion:v1.28.0-alpha.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:112030336,},ContainerImage{Names:[docker.io/calico/cni@sha256:e282ea2914c806b5de2976330a17cfb5e6dcef47147bceb1432ca5c75fd46f50 docker.io/calico/cni:v3.24.5],SizeBytes:87458443,},ContainerImage{Names:[docker.io/calico/node@sha256:5972ad2bcbdc668564d3e26960c9c513b2d7b05581c704747cf7c62ef3a405a6 docker.io/calico/node:v3.24.5],SizeBytes:81584866,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41901587,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:e468dddcd275163a042ab297b2d8c2aca50d5e148d2d22f3b6ba119e2f31fa79 registry.k8s.io/sig-storage/csi-provisioner:v3.4.0],SizeBytes:27427836,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/kube-proxy@sha256:8a03b3147cd7be8e8235ae1cd70cf69aca9ce03ba1cfaffea320540bb5afa496 registry.k8s.io/kube-proxy:v1.28.0-alpha.0],SizeBytes:23896850,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:23881995,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:68d396900aeaa072c1f27289485fdac29834045a6f3ffe369bf389d830ef572d registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.6],SizeBytes:20293261,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18758628,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18233005,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:15273057,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8688564,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:7000082,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6978614,},ContainerImage{Names:[gcr.io/authenticated-image-pulling/alpine@sha256:7ff177862cb50c602bfe81f805969412e619c054a2bbead977d0c276988aa4a0 gcr.io/authenticated-image-pulling/alpine:3.7],SizeBytes:2110879,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[kubernetes.io/csi/csi-hostpath-provisioning-7537^880be747-cad1-11ed-881d-2eebbe6b982f kubernetes.io/csi/csi-hostpath-volume-6086^88d37df3-cad1-11ed-9ea8-f6813064f795],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-provisioning-7537^880be747-cad1-11ed-881d-2eebbe6b982f,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-volume-6086^88d37df3-cad1-11ed-9ea8-f6813064f795,DevicePath:,},},Config:nil,},} Mar 25 05:55:03.763: INFO: Logging kubelet events for node i-049fdb2c1cde894df Mar 25 05:55:03.819: INFO: Logging pods the kubelet thinks is on node i-049fdb2c1cde894df Mar 25 05:55:03.883: INFO: csi-hostpathplugin-0 started at 2023-03-25 05:54:14 +0000 UTC (0+7 container statuses recorded) Mar 25 05:55:03.883: INFO: Container csi-attacher ready: true, restart count 0 Mar 25 05:55:03.883: INFO: Container csi-provisioner ready: true, restart count 0 Mar 25 05:55:03.883: INFO: Container csi-resizer ready: true, restart count 0 Mar 25 05:55:03.883: INFO: Container csi-snapshotter ready: true, restart count 0 Mar 25 05:55:03.883: INFO: Container hostpath ready: true, restart count 0 Mar 25 05:55:03.883: INFO: Container liveness-probe ready: true, restart count 0 Mar 25 05:55:03.883: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 25 05:55:03.883: INFO: ebs-csi-node-m9zxz started at 2023-03-25 05:23:35 +0000 UTC (0+3 container statuses recorded) Mar 25 05:55:03.883: INFO: Container ebs-plugin ready: true, restart count 0 Mar 25 05:55:03.883: INFO: Container liveness-probe ready: true, restart count 0 Mar 25 05:55:03.883: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 25 05:55:03.883: INFO: coredns-559769c974-pgqhl started at 2023-03-25 05:24:36 +0000 UTC (0+1 container statuses recorded) Mar 25 05:55:03.883: INFO: Container coredns ready: true, restart count 0 Mar 25 05:55:03.883: INFO: kube-proxy-i-049fdb2c1cde894df started at 2023-03-25 05:23:35 +0000 UTC (0+1 container statuses recorded) Mar 25 05:55:03.883: INFO: Container kube-proxy ready: true, restart count 0 Mar 25 05:55:03.883: INFO: csi-mockplugin-0 started at 2023-03-25 05:54:30 +0000 UTC (0+3 container statuses recorded) Mar 25 05:55:03.883: INFO: Container csi-provisioner ready: true, restart count 0 Mar 25 05:55:03.883: INFO: Container driver-registrar ready: true, restart count 0 Mar 25 05:55:03.883: INFO: Container mock ready: true, restart count 0 Mar 25 05:55:03.883: INFO: coredns-autoscaler-7cb5c5b969-h485x started at 2023-03-25 05:24:36 +0000 UTC (0+1 container statuses recorded) Mar 25 05:55:03.883: INFO: Container autoscaler ready: true, restart count 0 Mar 25 05:55:03.883: INFO: hostpath-client started at 2023-03-25 05:54:52 +0000 UTC (0+1 container statuses recorded) Mar 25 05:55:03.883: INFO: Container hostpath-client ready: false, restart count 0 Mar 25 05:55:03.883: INFO: csi-hostpathplugin-0 started at 2023-03-25 05:54:39 +0000 UTC (0+7 container statuses recorded) Mar 25 05:55:03.883: INFO: Container csi-attacher ready: true, restart count 0 Mar 25 05:55:03.883: INFO: Container csi-provisioner ready: true, restart count 0 Mar 25 05:55:03.883: INFO: Container csi-resizer ready: true, restart count 0 Mar 25 05:55:03.883: INFO: Container csi-snapshotter ready: true, restart count 0 Mar 25 05:55:03.883: INFO: Container hostpath ready: true, restart count 0 Mar 25 05:55:03.883: INFO: Container liveness-probe ready: true, restart count 0 Mar 25 05:55:03.883: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 25 05:55:03.883: INFO: calico-node-wt5p7 started at 2023-03-25 05:23:35 +0000 UTC (3+1 container statuses recorded) Mar 25 05:55:03.883: INFO: Init container upgrade-ipam ready: true, restart count 0 Mar 25 05:55:03.883: INFO: Init container install-cni ready: true, restart count 0 Mar 25 05:55:03.883: INFO: Init container mount-bpffs ready: true, restart count 0 Mar 25 05:55:03.883: INFO: Container calico-node ready: true, restart count 0 Mar 25 05:55:03.883: INFO: csi-mockplugin-attacher-0 started at 2023-03-25 05:54:30 +0000 UTC (0+1 container statuses recorded) Mar 25 05:55:03.883: INFO: Container csi-attacher ready: true, restart count 0 Mar 25 05:55:04.208: INFO: Latency metrics for node i-049fdb2c1cde894df Mar 25 05:55:04.208: INFO: Logging node info for node i-062679bb8c0ba4bce Mar 25 05:55:04.257: INFO: Node Info: &Node{ObjectMeta:{i-062679bb8c0ba4bce ee2da569-bb88-4297-9e6f-d2b15ba7ce86 63611 0 2023-03-25 05:27:31 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-1 failure-domain.beta.kubernetes.io/zone:us-west-1b kubernetes.io/arch:amd64 kubernetes.io/hostname:i-062679bb8c0ba4bce kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:us-west-1b topology.hostpath.csi/node:i-062679bb8c0ba4bce topology.kubernetes.io/region:us-west-1 topology.kubernetes.io/zone:us-west-1b] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-062679bb8c0ba4bce"} node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:172.20.45.43/19 projectcalico.org/IPv4IPIPTunnelAddr:100.106.79.0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-03-25 05:27:31 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-25 05:27:31 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-03-25 05:27:31 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2023-03-25 05:27:31 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2023-03-25 05:27:44 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.7.0/24\"":{}}}} } {calico-node Update v1 2023-03-25 05:27:50 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4IPIPTunnelAddr":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2023-03-25 05:54:53 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-03-25 05:55:03 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.7.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-1b/i-062679bb8c0ba4bce,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.7.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051689472 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946831872 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-03-25 05:27:50 +0000 UTC,LastTransitionTime:2023-03-25 05:27:50 +0000 UTC,Reason:CalicoIsUp,Message:Calico is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-25 05:55:03 +0000 UTC,LastTransitionTime:2023-03-25 05:27:31 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-25 05:55:03 +0000 UTC,LastTransitionTime:2023-03-25 05:27:31 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-25 05:55:03 +0000 UTC,LastTransitionTime:2023-03-25 05:27:31 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-25 05:55:03 +0000 UTC,LastTransitionTime:2023-03-25 05:27:44 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.45.43,},NodeAddress{Type:ExternalIP,Address:3.101.133.108,},NodeAddress{Type:InternalDNS,Address:i-062679bb8c0ba4bce.us-west-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-062679bb8c0ba4bce.us-west-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-3-101-133-108.us-west-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2528f599e9b1e199793fb5cd06d205,SystemUUID:ec2528f5-99e9-b1e1-9979-3fb5cd06d205,BootID:ff56b70b-c8c5-46db-8cc5-4353c8f535c6,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.28.0-alpha.0,KubeProxyVersion:v1.28.0-alpha.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:112030336,},ContainerImage{Names:[docker.io/calico/cni@sha256:e282ea2914c806b5de2976330a17cfb5e6dcef47147bceb1432ca5c75fd46f50 docker.io/calico/cni:v3.24.5],SizeBytes:87458443,},ContainerImage{Names:[docker.io/calico/node@sha256:5972ad2bcbdc668564d3e26960c9c513b2d7b05581c704747cf7c62ef3a405a6 docker.io/calico/node:v3.24.5],SizeBytes:81584866,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nautilus@sha256:80ba6c8c44f9623f06e868a1aa66026c8ec438ad814f9ec95e9333b415fe3550 registry.k8s.io/e2e-test-images/nautilus:1.7],SizeBytes:49641698,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41901587,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:e468dddcd275163a042ab297b2d8c2aca50d5e148d2d22f3b6ba119e2f31fa79 registry.k8s.io/sig-storage/csi-provisioner:v3.4.0],SizeBytes:27427836,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/kube-proxy@sha256:8a03b3147cd7be8e8235ae1cd70cf69aca9ce03ba1cfaffea320540bb5afa496 registry.k8s.io/kube-proxy:v1.28.0-alpha.0],SizeBytes:23896850,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:23881995,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18758628,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/regression-issue-74839@sha256:b4f1d8d61bdad84bd50442d161d5460e4019d53e989b64220fdbc62fc87d76bf registry.k8s.io/e2e-test-images/regression-issue-74839:1.2],SizeBytes:18651485,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18233005,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:15273057,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8688564,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/build-image/distroless-iptables@sha256:13ac9b3c476d7290a4451d65d09d6016a2cb89836ffbfa4eae55b72731a22080 registry.k8s.io/build-image/distroless-iptables:v0.2.2],SizeBytes:7729580,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:7000082,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6978614,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[kubernetes.io/csi/ebs.csi.aws.com^vol-0b0a70539f44dbc59 kubernetes.io/csi/ebs.csi.aws.com^vol-0ffe739c83a1d20d7],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-0ffe739c83a1d20d7,DevicePath:,},},Config:nil,},} Mar 25 05:55:04.258: INFO: Logging kubelet events for node i-062679bb8c0ba4bce Mar 25 05:55:04.316: INFO: Logging pods the kubelet thinks is on node i-062679bb8c0ba4bce Mar 25 05:55:04.398: INFO: ss-2 started at 2023-03-25 05:54:54 +0000 UTC (0+1 container statuses recorded) Mar 25 05:55:04.398: INFO: Container webserver ready: false, restart count 0 Mar 25 05:55:04.398: INFO: ss-0 started at 2023-03-25 05:54:04 +0000 UTC (0+1 container statuses recorded) Mar 25 05:55:04.398: INFO: Container webserver ready: true, restart count 0 Mar 25 05:55:04.398: INFO: kube-proxy-i-062679bb8c0ba4bce started at 2023-03-25 05:27:32 +0000 UTC (0+1 container statuses recorded) Mar 25 05:55:04.398: INFO: Container kube-proxy ready: true, restart count 0 Mar 25 05:55:04.398: INFO: pod-terminate-status-0-7 started at 2023-03-25 05:55:03 +0000 UTC (0+1 container statuses recorded) Mar 25 05:55:04.398: INFO: Container fail ready: false, restart count 0 Mar 25 05:55:04.398: INFO: calico-node-zwm5j started at 2023-03-25 05:27:32 +0000 UTC (3+1 container statuses recorded) Mar 25 05:55:04.398: INFO: Init container upgrade-ipam ready: true, restart count 0 Mar 25 05:55:04.398: INFO: Init container install-cni ready: true, restart count 0 Mar 25 05:55:04.398: INFO: Init container mount-bpffs ready: true, restart count 0 Mar 25 05:55:04.398: INFO: Container calico-node ready: true, restart count 0 Mar 25 05:55:04.398: INFO: coredns-559769c974-p9vg6 started at 2023-03-25 05:36:16 +0000 UTC (0+1 container statuses recorded) Mar 25 05:55:04.398: INFO: Container coredns ready: true, restart count 0 Mar 25 05:55:04.398: INFO: pod-terminate-status-2-8 started at 2023-03-25 05:55:03 +0000 UTC (0+1 container statuses recorded) Mar 25 05:55:04.398: INFO: Container fail ready: false, restart count 0 Mar 25 05:55:04.398: INFO: pfpod started at 2023-03-25 05:54:00 +0000 UTC (0+2 container statuses recorded) Mar 25 05:55:04.398: INFO: Container portforwardtester ready: false, restart count 0 Mar 25 05:55:04.398: INFO: Container readiness ready: false, restart count 0 Mar 25 05:55:04.398: INFO: ebs-csi-node-7twrq started at 2023-03-25 05:27:32 +0000 UTC (0+3 container statuses recorded) Mar 25 05:55:04.398: INFO: Container ebs-plugin ready: true, restart count 0 Mar 25 05:55:04.398: INFO: Container liveness-probe ready: true, restart count 0 Mar 25 05:55:04.398: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 25 05:55:04.398: INFO: pod-terminate-status-1-7 started at 2023-03-25 05:55:00 +0000 UTC (0+1 container statuses recorded) Mar 25 05:55:04.398: INFO: Container fail ready: false, restart count 0 Mar 25 05:55:04.723: INFO: Latency metrics for node i-062679bb8c0ba4bce Mar 25 05:55:04.723: INFO: Logging node info for node i-072608201d2c0f1ad Mar 25 05:55:04.772: INFO: Node Info: &Node{ObjectMeta:{i-072608201d2c0f1ad 264dfba1-3cf0-4c99-8e8c-098f0f792077 63657 0 2023-03-25 05:35:29 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-1 failure-domain.beta.kubernetes.io/zone:us-west-1b kubernetes.io/arch:amd64 kubernetes.io/hostname:i-072608201d2c0f1ad kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:us-west-1b topology.hostpath.csi/node:i-072608201d2c0f1ad topology.kubernetes.io/region:us-west-1 topology.kubernetes.io/zone:us-west-1b] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-1132":"i-072608201d2c0f1ad","csi-hostpath-ephemeral-7817":"i-072608201d2c0f1ad","ebs.csi.aws.com":"i-072608201d2c0f1ad"} node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:172.20.38.125/19 projectcalico.org/IPv4IPIPTunnelAddr:100.107.220.192 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-03-25 05:35:29 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-25 05:35:29 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-03-25 05:35:29 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2023-03-25 05:35:29 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2023-03-25 05:35:41 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.9.0/24\"":{}}}} } {calico-node Update v1 2023-03-25 05:35:48 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4IPIPTunnelAddr":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2023-03-25 05:54:55 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-03-25 05:55:04 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.9.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-1b/i-072608201d2c0f1ad,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.9.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051689472 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946831872 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-03-25 05:35:47 +0000 UTC,LastTransitionTime:2023-03-25 05:35:47 +0000 UTC,Reason:CalicoIsUp,Message:Calico is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-25 05:54:55 +0000 UTC,LastTransitionTime:2023-03-25 05:35:29 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-25 05:54:55 +0000 UTC,LastTransitionTime:2023-03-25 05:35:29 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-25 05:54:55 +0000 UTC,LastTransitionTime:2023-03-25 05:35:29 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-25 05:54:55 +0000 UTC,LastTransitionTime:2023-03-25 05:35:41 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.38.125,},NodeAddress{Type:ExternalIP,Address:3.101.88.10,},NodeAddress{Type:InternalDNS,Address:i-072608201d2c0f1ad.us-west-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-072608201d2c0f1ad.us-west-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-3-101-88-10.us-west-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2d3818c4f80a253dcc1a60e148dbc8,SystemUUID:ec2d3818-c4f8-0a25-3dcc-1a60e148dbc8,BootID:77d046cf-75ea-4a8c-970a-3190d64e019b,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.28.0-alpha.0,KubeProxyVersion:v1.28.0-alpha.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:112030336,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:51eae8381dcb1078289fa7b4f3df2630cdc18d09fb56f8e56b41c40e191d6c83 registry.k8s.io/etcd:3.5.7-0],SizeBytes:101639218,},ContainerImage{Names:[docker.io/calico/cni@sha256:e282ea2914c806b5de2976330a17cfb5e6dcef47147bceb1432ca5c75fd46f50 docker.io/calico/cni:v3.24.5],SizeBytes:87458443,},ContainerImage{Names:[docker.io/calico/node@sha256:5972ad2bcbdc668564d3e26960c9c513b2d7b05581c704747cf7c62ef3a405a6 docker.io/calico/node:v3.24.5],SizeBytes:81584866,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nautilus@sha256:80ba6c8c44f9623f06e868a1aa66026c8ec438ad814f9ec95e9333b415fe3550 registry.k8s.io/e2e-test-images/nautilus:1.7],SizeBytes:49641698,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41901587,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:e468dddcd275163a042ab297b2d8c2aca50d5e148d2d22f3b6ba119e2f31fa79 registry.k8s.io/sig-storage/csi-provisioner:v3.4.0],SizeBytes:27427836,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/sample-apiserver@sha256:8d70890151aa5d096f331cb9da1b9cd5be0412b7363fe67b5c3befdcaa2a28d0 registry.k8s.io/e2e-test-images/sample-apiserver:1.17.7],SizeBytes:25667066,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/kube-proxy@sha256:8a03b3147cd7be8e8235ae1cd70cf69aca9ce03ba1cfaffea320540bb5afa496 registry.k8s.io/kube-proxy:v1.28.0-alpha.0],SizeBytes:23896850,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:23881995,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18758628,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18233005,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonroot@sha256:ee9f50b3c64b174d296d91ca9f69a914ac30e59095dfb462b2b518ad28a63655 registry.k8s.io/e2e-test-images/nonroot:1.4],SizeBytes:17747885,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8688564,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:7000082,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6978614,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[kubernetes.io/csi/csi-hostpath-ephemeral-7817^4b20e30d-cad1-11ed-98df-264f88168798],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-7817^4b20e30d-cad1-11ed-98df-264f88168798,DevicePath:,},},Config:nil,},} Mar 25 05:55:04.772: INFO: Logging kubelet events for node i-072608201d2c0f1ad Mar 25 05:55:04.828: INFO: Logging pods the kubelet thinks is on node i-072608201d2c0f1ad Mar 25 05:55:04.884: INFO: csi-hostpathplugin-0 started at 2023-03-25 05:52:55 +0000 UTC (0+7 container statuses recorded) Mar 25 05:55:04.884: INFO: Container csi-attacher ready: true, restart count 0 Mar 25 05:55:04.884: INFO: Container csi-provisioner ready: true, restart count 0 Mar 25 05:55:04.884: INFO: Container csi-resizer ready: true, restart count 0 Mar 25 05:55:04.884: INFO: Container csi-snapshotter ready: true, restart count 0 Mar 25 05:55:04.884: INFO: Container hostpath ready: true, restart count 0 Mar 25 05:55:04.884: INFO: Container liveness-probe ready: true, restart count 0 Mar 25 05:55:04.884: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 25 05:55:04.884: INFO: calico-node-lc9zl started at 2023-03-25 05:35:30 +0000 UTC (3+1 container statuses recorded) Mar 25 05:55:04.884: INFO: Init container upgrade-ipam ready: true, restart count 0 Mar 25 05:55:04.884: INFO: Init container install-cni ready: true, restart count 0 Mar 25 05:55:04.884: INFO: Init container mount-bpffs ready: true, restart count 0 Mar 25 05:55:04.884: INFO: Container calico-node ready: true, restart count 0 Mar 25 05:55:04.884: INFO: csi-mockplugin-attacher-0 started at 2023-03-25 05:52:22 +0000 UTC (0+1 container statuses recorded) Mar 25 05:55:04.884: INFO: Container csi-attacher ready: true, restart count 0 Mar 25 05:55:04.884: INFO: csi-mockplugin-0 started at 2023-03-25 05:52:22 +0000 UTC (0+3 container statuses recorded) Mar 25 05:55:04.884: INFO: Container csi-provisioner ready: true, restart count 0 Mar 25 05:55:04.884: INFO: Container driver-registrar ready: true, restart count 0 Mar 25 05:55:04.884: INFO: Container mock ready: true, restart count 0 Mar 25 05:55:04.884: INFO: kube-proxy-i-072608201d2c0f1ad started at 2023-03-25 05:35:30 +0000 UTC (0+1 container statuses recorded) Mar 25 05:55:04.884: INFO: Container kube-proxy ready: true, restart count 0 Mar 25 05:55:04.884: INFO: inline-volume-tester-fdp2w started at 2023-03-25 05:52:58 +0000 UTC (0+1 container statuses recorded) Mar 25 05:55:04.884: INFO: Container csi-volume-tester ready: true, restart count 0 Mar 25 05:55:04.884: INFO: pod-secrets-c75f6630-caa4-40a1-b97c-358088342209 started at 2023-03-25 05:53:01 +0000 UTC (0+3 container statuses recorded) Mar 25 05:55:04.884: INFO: Container creates-volume-test ready: true, restart count 0 Mar 25 05:55:04.884: INFO: Container dels-volume-test ready: true, restart count 0 Mar 25 05:55:04.884: INFO: Container upds-volume-test ready: true, restart count 0 Mar 25 05:55:04.884: INFO: csi-hostpathplugin-0 started at 2023-03-25 05:53:24 +0000 UTC (0+7 container statuses recorded) Mar 25 05:55:04.884: INFO: Container csi-attacher ready: true, restart count 0 Mar 25 05:55:04.884: INFO: Container csi-provisioner ready: true, restart count 0 Mar 25 05:55:04.884: INFO: Container csi-resizer ready: true, restart count 0 Mar 25 05:55:04.884: INFO: Container csi-snapshotter ready: true, restart count 0 Mar 25 05:55:04.884: INFO: Container hostpath ready: true, restart count 0 Mar 25 05:55:04.884: INFO: Container liveness-probe ready: true, restart count 0 Mar 25 05:55:04.884: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 25 05:55:04.884: INFO: ebs-csi-node-r7mcm started at 2023-03-25 05:35:30 +0000 UTC (0+3 container statuses recorded) Mar 25 05:55:04.884: INFO: Container ebs-plugin ready: true, restart count 0 Mar 25 05:55:04.884: INFO: Container liveness-probe ready: true, restart count 0 Mar 25 05:55:04.884: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 25 05:55:05.231: INFO: Latency metrics for node i-072608201d2c0f1ad END STEP: dump namespace information after failure - test/e2e/framework/framework.go:288 @ 03/25/23 05:55:05.231 (2.69s) < Exit [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] subPath - dump namespaces | framework.go:209 @ 03/25/23 05:55:05.231 (2.69s) > Enter [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] subPath - tear down framework | framework.go:206 @ 03/25/23 05:55:05.231 STEP: Destroying namespace "provisioning-5126" for this suite. - test/e2e/framework/framework.go:351 @ 03/25/23 05:55:05.231 < Exit [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] subPath - tear down framework | framework.go:206 @ 03/25/23 05:55:05.281 (51ms) > Enter [ReportAfterEach] TOP-LEVEL - test/e2e/e2e_test.go:144 @ 03/25/23 05:55:05.282 < Exit [ReportAfterEach] TOP-LEVEL - test/e2e/e2e_test.go:144 @ 03/25/23 05:55:05.282 (0s)
Filter through log files | View test history on testgrid
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[It\]\sExternal\sStorage\s\[Driver\:\sebs\.csi\.aws\.com\]\s\[Testpattern\:\sDynamic\sPV\s\(default\sfs\)\]\ssubPath\sshould\ssupport\sreadOnly\sdirectory\sspecified\sin\sthe\svolumeMount$'
[FAILED] failed to get logs from pod-subpath-test-dynamicpv-4xtj for test-container-subpath-dynamicpv-4xtj: Get "https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/provisioning-7937/pods/pod-subpath-test-dynamicpv-4xtj/log?container=test-container-subpath-dynamicpv-4xtj&previous=false": stream error: stream ID 5387; INTERNAL_ERROR; received from peer In [It] at: test/e2e/framework/pod/output/output.go:247 @ 03/25/23 05:51:17.968from junit_01.xml
> Enter [BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/storage/framework/testsuite.go:51 @ 03/25/23 05:51:00.524 < Exit [BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/storage/framework/testsuite.go:51 @ 03/25/23 05:51:00.524 (0s) > Enter [BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath - set up framework | framework.go:191 @ 03/25/23 05:51:00.524 STEP: Creating a kubernetes client - test/e2e/framework/framework.go:211 @ 03/25/23 05:51:00.524 Mar 25 05:51:00.524: INFO: >>> kubeConfig: /root/.kube/config STEP: Building a namespace api object, basename provisioning - test/e2e/framework/framework.go:250 @ 03/25/23 05:51:00.525 STEP: Waiting for a default service account to be provisioned in namespace - test/e2e/framework/framework.go:259 @ 03/25/23 05:51:00.678 STEP: Waiting for kube-root-ca.crt to be provisioned in namespace - test/e2e/framework/framework.go:262 @ 03/25/23 05:51:00.78 < Exit [BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath - set up framework | framework.go:191 @ 03/25/23 05:51:00.904 (379ms) > Enter [BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/framework/metrics/init/init.go:33 @ 03/25/23 05:51:00.904 < Exit [BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/framework/metrics/init/init.go:33 @ 03/25/23 05:51:00.904 (0s) > Enter [It] should support readOnly directory specified in the volumeMount - test/e2e/storage/testsuites/subpath.go:367 @ 03/25/23 05:51:00.904 Mar 25 05:51:00.904: INFO: Creating resource for dynamic PV Mar 25 05:51:00.904: INFO: Using claimSize:1Gi, test suite supported size:{ 1Mi}, driver(ebs.csi.aws.com) supported size:{ 1Mi} STEP: creating a StorageClass provisioning-7937-e2e-scfrpqn - test/e2e/storage/framework/volume_resource.go:102 @ 03/25/23 05:51:00.904 STEP: creating a claim - test/e2e/storage/framework/volume_resource.go:294 @ 03/25/23 05:51:00.997 Mar 25 05:51:00.997: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil STEP: Creating pod pod-subpath-test-dynamicpv-4xtj - test/e2e/storage/testsuites/subpath.go:691 @ 03/25/23 05:51:01.115 STEP: Creating a pod to test subpath - test/e2e/framework/pod/output/output.go:243 @ 03/25/23 05:51:01.115 STEP: Saw pod success - test/e2e/framework/pod/wait.go:409 @ 03/25/23 05:51:17.731 Mar 25 05:51:17.782: INFO: Trying to get logs from node i-072608201d2c0f1ad pod pod-subpath-test-dynamicpv-4xtj container test-container-subpath-dynamicpv-4xtj: <nil> Mar 25 05:51:17.834: INFO: Failed to get logs from node "i-072608201d2c0f1ad" pod "pod-subpath-test-dynamicpv-4xtj" container "test-container-subpath-dynamicpv-4xtj". Get "https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/provisioning-7937/pods/pod-subpath-test-dynamicpv-4xtj/log?container=test-container-subpath-dynamicpv-4xtj&previous=false": stream error: stream ID 5387; INTERNAL_ERROR; received from peer STEP: delete the pod - test/e2e/framework/pod/output/output.go:168 @ 03/25/23 05:51:17.834 Mar 25 05:51:17.967: INFO: Unexpected error: <*fmt.wrapError | 0xc004e64c20>: failed to get logs from pod-subpath-test-dynamicpv-4xtj for test-container-subpath-dynamicpv-4xtj: Get "https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/provisioning-7937/pods/pod-subpath-test-dynamicpv-4xtj/log?container=test-container-subpath-dynamicpv-4xtj&previous=false": stream error: stream ID 5387; INTERNAL_ERROR; received from peer { msg: "failed to get logs from pod-subpath-test-dynamicpv-4xtj for test-container-subpath-dynamicpv-4xtj: Get \"https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/provisioning-7937/pods/pod-subpath-test-dynamicpv-4xtj/log?container=test-container-subpath-dynamicpv-4xtj&previous=false\": stream error: stream ID 5387; INTERNAL_ERROR; received from peer", err: <*url.Error | 0xc0051facf0>{ Op: "Get", URL: "https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/provisioning-7937/pods/pod-subpath-test-dynamicpv-4xtj/log?container=test-container-subpath-dynamicpv-4xtj&previous=false", Err: <http2.StreamError>{ StreamID: 5387, Code: 2, Cause: <*errors.errorString | 0xc0001676e0>{ s: "received from peer", }, }, }, } [FAILED] failed to get logs from pod-subpath-test-dynamicpv-4xtj for test-container-subpath-dynamicpv-4xtj: Get "https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/provisioning-7937/pods/pod-subpath-test-dynamicpv-4xtj/log?container=test-container-subpath-dynamicpv-4xtj&previous=false": stream error: stream ID 5387; INTERNAL_ERROR; received from peer In [It] at: test/e2e/framework/pod/output/output.go:247 @ 03/25/23 05:51:17.968 < Exit [It] should support readOnly directory specified in the volumeMount - test/e2e/storage/testsuites/subpath.go:367 @ 03/25/23 05:51:17.968 (17.064s) > Enter [AfterEach] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/framework/node/init/init.go:33 @ 03/25/23 05:51:17.968 Mar 25 05:51:17.968: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready < Exit [AfterEach] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/framework/node/init/init.go:33 @ 03/25/23 05:51:18.017 (50ms) > Enter [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/storage/testsuites/subpath.go:369 @ 03/25/23 05:51:18.017 STEP: Deleting pod - test/e2e/storage/testsuites/subpath.go:172 @ 03/25/23 05:51:18.017 Mar 25 05:51:18.017: INFO: Deleting pod "pod-subpath-test-dynamicpv-4xtj" in namespace "provisioning-7937" STEP: Deleting pvc - test/e2e/storage/framework/volume_resource.go:181 @ 03/25/23 05:51:18.112 Mar 25 05:51:18.221: INFO: Deleting PersistentVolumeClaim "ebs.csi.aws.comzzc9f" Mar 25 05:51:18.273: INFO: Waiting up to 5m0s for PersistentVolume pvc-e33b063e-1ae1-4474-92c5-d6b9575ff1ac to get deleted Mar 25 05:51:18.322: INFO: PersistentVolume pvc-e33b063e-1ae1-4474-92c5-d6b9575ff1ac found and phase=Released (48.727396ms) Mar 25 05:51:23.371: INFO: PersistentVolume pvc-e33b063e-1ae1-4474-92c5-d6b9575ff1ac found and phase=Released (5.098307837s) Mar 25 05:51:28.421: INFO: PersistentVolume pvc-e33b063e-1ae1-4474-92c5-d6b9575ff1ac found and phase=Released (10.147568852s) Mar 25 05:51:33.470: INFO: PersistentVolume pvc-e33b063e-1ae1-4474-92c5-d6b9575ff1ac found and phase=Released (15.196909089s) Mar 25 05:51:38.535: INFO: PersistentVolume pvc-e33b063e-1ae1-4474-92c5-d6b9575ff1ac was removed STEP: Deleting sc - test/e2e/storage/framework/volume_resource.go:228 @ 03/25/23 05:51:38.535 < Exit [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/storage/testsuites/subpath.go:369 @ 03/25/23 05:51:38.603 (20.586s) > Enter [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/framework/metrics/init/init.go:35 @ 03/25/23 05:51:38.603 < Exit [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/framework/metrics/init/init.go:35 @ 03/25/23 05:51:38.604 (0s) > Enter [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] subPath - dump namespaces | framework.go:209 @ 03/25/23 05:51:38.604 STEP: dump namespace information after failure - test/e2e/framework/framework.go:288 @ 03/25/23 05:51:38.604 STEP: Collecting events from namespace "provisioning-7937". - test/e2e/framework/debug/dump.go:42 @ 03/25/23 05:51:38.604 STEP: Found 15 events. - test/e2e/framework/debug/dump.go:46 @ 03/25/23 05:51:38.656 Mar 25 05:51:38.656: INFO: At 2023-03-25 05:51:01 +0000 UTC - event for ebs.csi.aws.comzzc9f: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding Mar 25 05:51:38.656: INFO: At 2023-03-25 05:51:01 +0000 UTC - event for ebs.csi.aws.comzzc9f: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator Mar 25 05:51:38.656: INFO: At 2023-03-25 05:51:01 +0000 UTC - event for ebs.csi.aws.comzzc9f: {ebs.csi.aws.com_i-0168f82458996cc53_0b589c46-3396-4499-94ba-eb51a48f7a58 } Provisioning: External provisioner is provisioning volume for claim "provisioning-7937/ebs.csi.aws.comzzc9f" Mar 25 05:51:38.656: INFO: At 2023-03-25 05:51:05 +0000 UTC - event for ebs.csi.aws.comzzc9f: {ebs.csi.aws.com_i-0168f82458996cc53_0b589c46-3396-4499-94ba-eb51a48f7a58 } ProvisioningSucceeded: Successfully provisioned volume pvc-e33b063e-1ae1-4474-92c5-d6b9575ff1ac Mar 25 05:51:38.656: INFO: At 2023-03-25 05:51:05 +0000 UTC - event for pod-subpath-test-dynamicpv-4xtj: {default-scheduler } Scheduled: Successfully assigned provisioning-7937/pod-subpath-test-dynamicpv-4xtj to i-072608201d2c0f1ad Mar 25 05:51:38.656: INFO: At 2023-03-25 05:51:07 +0000 UTC - event for pod-subpath-test-dynamicpv-4xtj: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-e33b063e-1ae1-4474-92c5-d6b9575ff1ac" Mar 25 05:51:38.656: INFO: At 2023-03-25 05:51:11 +0000 UTC - event for pod-subpath-test-dynamicpv-4xtj: {kubelet i-072608201d2c0f1ad} Pulled: Container image "registry.k8s.io/e2e-test-images/busybox:1.29-4" already present on machine Mar 25 05:51:38.656: INFO: At 2023-03-25 05:51:11 +0000 UTC - event for pod-subpath-test-dynamicpv-4xtj: {kubelet i-072608201d2c0f1ad} Created: Created container init-volume-dynamicpv-4xtj Mar 25 05:51:38.656: INFO: At 2023-03-25 05:51:11 +0000 UTC - event for pod-subpath-test-dynamicpv-4xtj: {kubelet i-072608201d2c0f1ad} Started: Started container init-volume-dynamicpv-4xtj Mar 25 05:51:38.656: INFO: At 2023-03-25 05:51:12 +0000 UTC - event for pod-subpath-test-dynamicpv-4xtj: {kubelet i-072608201d2c0f1ad} Pulled: Container image "registry.k8s.io/e2e-test-images/agnhost:2.43" already present on machine Mar 25 05:51:38.656: INFO: At 2023-03-25 05:51:12 +0000 UTC - event for pod-subpath-test-dynamicpv-4xtj: {kubelet i-072608201d2c0f1ad} Created: Created container test-init-volume-dynamicpv-4xtj Mar 25 05:51:38.656: INFO: At 2023-03-25 05:51:12 +0000 UTC - event for pod-subpath-test-dynamicpv-4xtj: {kubelet i-072608201d2c0f1ad} Started: Started container test-init-volume-dynamicpv-4xtj Mar 25 05:51:38.656: INFO: At 2023-03-25 05:51:13 +0000 UTC - event for pod-subpath-test-dynamicpv-4xtj: {kubelet i-072608201d2c0f1ad} Pulled: Container image "registry.k8s.io/e2e-test-images/agnhost:2.43" already present on machine Mar 25 05:51:38.656: INFO: At 2023-03-25 05:51:13 +0000 UTC - event for pod-subpath-test-dynamicpv-4xtj: {kubelet i-072608201d2c0f1ad} Created: Created container test-container-subpath-dynamicpv-4xtj Mar 25 05:51:38.656: INFO: At 2023-03-25 05:51:13 +0000 UTC - event for pod-subpath-test-dynamicpv-4xtj: {kubelet i-072608201d2c0f1ad} Started: Started container test-container-subpath-dynamicpv-4xtj Mar 25 05:51:38.707: INFO: POD NODE PHASE GRACE CONDITIONS Mar 25 05:51:38.707: INFO: Mar 25 05:51:38.757: INFO: Logging node info for node i-0168f82458996cc53 Mar 25 05:51:38.852: INFO: Node Info: &Node{ObjectMeta:{i-0168f82458996cc53 f4eb302c-c477-45ac-bcac-ffed65f6580f 38654 0 2023-03-25 05:18:57 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-1 failure-domain.beta.kubernetes.io/zone:us-west-1b kops.k8s.io/kops-controller-pki: kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0168f82458996cc53 kubernetes.io/os:linux node-role.kubernetes.io/control-plane: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:us-west-1b topology.kubernetes.io/region:us-west-1 topology.kubernetes.io/zone:us-west-1b] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0168f82458996cc53"} node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:172.20.48.81/19 projectcalico.org/IPv4IPIPTunnelAddr:100.100.215.192 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-03-25 05:18:57 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {protokube Update v1 2023-03-25 05:18:58 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {calico-node Update v1 2023-03-25 05:20:22 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4IPIPTunnelAddr":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {aws-cloud-controller-manager Update v1 2023-03-25 05:20:26 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-25 05:20:26 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2023-03-25 05:20:31 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.5.0/24\"":{}},"f:taints":{}}} } {kubelet Update v1 2023-03-25 05:47:11 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.5.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-1b/i-0168f82458996cc53,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/control-plane,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.5.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3892310016 0} {<nil>} 3801084Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3787452416 0} {<nil>} 3698684Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-03-25 05:20:22 +0000 UTC,LastTransitionTime:2023-03-25 05:20:22 +0000 UTC,Reason:CalicoIsUp,Message:Calico is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-25 05:47:11 +0000 UTC,LastTransitionTime:2023-03-25 05:18:50 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-25 05:47:11 +0000 UTC,LastTransitionTime:2023-03-25 05:18:50 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-25 05:47:11 +0000 UTC,LastTransitionTime:2023-03-25 05:18:50 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-25 05:47:11 +0000 UTC,LastTransitionTime:2023-03-25 05:20:29 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.48.81,},NodeAddress{Type:ExternalIP,Address:3.101.135.77,},NodeAddress{Type:InternalDNS,Address:i-0168f82458996cc53.us-west-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-0168f82458996cc53.us-west-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-3-101-135-77.us-west-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2d1ccc7ebf313420bd1f530a94fbfd,SystemUUID:ec2d1ccc-7ebf-3134-20bd-1f530a94fbfd,BootID:15cee321-56e7-4290-ab48-47b60b0bc657,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.28.0-alpha.0,KubeProxyVersion:v1.28.0-alpha.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcd@sha256:dd75ec974b0a2a6f6bb47001ba09207976e625db898d1b16735528c009cb171c registry.k8s.io/etcd:3.5.6-0],SizeBytes:102542580,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:6f72b851544986cb0921b53ea655ec04c36131248f16d4ad110cb3ca0c369dc1 registry.k8s.io/etcd:3.5.4-0],SizeBytes:102157811,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:13f53ed1d91e2e11aac476ee9a0269fdda6cc4874eba903efd40daf50c55eee5 registry.k8s.io/etcd:3.5.3-0],SizeBytes:102143581,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:51eae8381dcb1078289fa7b4f3df2630cdc18d09fb56f8e56b41c40e191d6c83 registry.k8s.io/etcd:3.5.7-0],SizeBytes:101639218,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:4afb99b4690b418ffc2ceb67e1a17376457e441c1f09ab55447f0aaf992fa646 registry.k8s.io/etcd:3.4.3-0],SizeBytes:100947667,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:9ce33ba33d8e738a5b85ed50b5080ac746deceed4a7496c550927a7a19ca3b6d registry.k8s.io/etcd:3.5.0-0],SizeBytes:99868722,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:64b9ea357325d5db9f8a723dcf503b5a449177b17ac87d69481e126bb724c263 registry.k8s.io/etcd:3.5.1-0],SizeBytes:98888614,},ContainerImage{Names:[docker.io/calico/cni@sha256:e282ea2914c806b5de2976330a17cfb5e6dcef47147bceb1432ca5c75fd46f50 docker.io/calico/cni:v3.24.5],SizeBytes:87458443,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:4ad90a11b55313b182afc186b9876c8e891531b8db4c9bf1541953021618d0e2 registry.k8s.io/etcd:3.4.13-0],SizeBytes:86742272,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:0d2ab6b15b4fe0b903b9e81192d942fb39c29be61fe341cd16db5d0165b0435c registry.k8s.io/etcd:3.3.17-0],SizeBytes:85524408,},ContainerImage{Names:[docker.io/calico/node@sha256:5972ad2bcbdc668564d3e26960c9c513b2d7b05581c704747cf7c62ef3a405a6 docker.io/calico/node:v3.24.5],SizeBytes:81584866,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:17da501f5d2a675be46040422a27b7cc21b8a43895ac998b171db1c346f361f7 registry.k8s.io/etcd:3.3.10-0],SizeBytes:76162358,},ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager-slim@sha256:8727adc059334e0e3329dbccce116a734da4a090da5e86c8cddccaf333caa666 registry.k8s.io/etcdadm/etcd-manager-slim:v3.0.20230201],SizeBytes:63926165,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:905d7ca17fd02bc24c0eba9a062753aba15db3e31422390bc3238eb762339b20 registry.k8s.io/etcd:3.2.24-1],SizeBytes:63423287,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.27.0-alpha.1],SizeBytes:45219545,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.27.0-alpha.1],SizeBytes:43628585,},ContainerImage{Names:[registry.k8s.io/kube-apiserver@sha256:49934a29c559e79ed19fc38b4697b18d8fe174b17bf56192588575f980c93956 registry.k8s.io/kube-apiserver:v1.28.0-alpha.0],SizeBytes:33212506,},ContainerImage{Names:[docker.io/calico/kube-controllers@sha256:2b6acd7f677f76ffe12ecf3ea7df92eb9b1bdb07336d1ac2a54c7631fb753f7e docker.io/calico/kube-controllers:v3.24.5],SizeBytes:31137169,},ContainerImage{Names:[registry.k8s.io/kube-controller-manager@sha256:974748384634982c73527fdb3c9ec0ec476bc0d52b3dd1d611a380b7f99af1f3 registry.k8s.io/kube-controller-manager:v1.28.0-alpha.0],SizeBytes:30820280,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/kube-proxy@sha256:8a03b3147cd7be8e8235ae1cd70cf69aca9ce03ba1cfaffea320540bb5afa496 registry.k8s.io/kube-proxy:v1.28.0-alpha.0],SizeBytes:23896850,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:23345856,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:22381475,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:22085298,},ContainerImage{Names:[registry.k8s.io/provider-aws/cloud-controller-manager@sha256:fdeb61e3e42ecd9cca868d550ebdb88dd6341d9e91fcfa9a37e227dab2ad22cb registry.k8s.io/provider-aws/cloud-controller-manager:v1.26.0],SizeBytes:20154862,},ContainerImage{Names:[registry.k8s.io/kube-scheduler@sha256:cbf4007abb076188f16b975bc2b74452603cc7b34c60a29d92f14c38ad02dc11 registry.k8s.io/kube-scheduler:v1.28.0-alpha.0],SizeBytes:18078674,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.27.0-alpha.1],SizeBytes:5074298,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Mar 25 05:51:38.852: INFO: Logging kubelet events for node i-0168f82458996cc53 Mar 25 05:51:38.909: INFO: Logging pods the kubelet thinks is on node i-0168f82458996cc53 Mar 25 05:51:38.974: INFO: kube-controller-manager-i-0168f82458996cc53 started at 2023-03-25 05:17:36 +0000 UTC (0+1 container statuses recorded) Mar 25 05:51:38.974: INFO: Container kube-controller-manager ready: true, restart count 4 Mar 25 05:51:38.974: INFO: etcd-manager-events-i-0168f82458996cc53 started at 2023-03-25 05:17:36 +0000 UTC (11+1 container statuses recorded) Mar 25 05:51:38.974: INFO: Init container init-etcd-3-2-24 ready: true, restart count 0 Mar 25 05:51:38.974: INFO: Init container init-etcd-3-3-10 ready: true, restart count 0 Mar 25 05:51:38.974: INFO: Init container init-etcd-3-3-17 ready: true, restart count 0 Mar 25 05:51:38.974: INFO: Init container init-etcd-3-4-13 ready: true, restart count 0 Mar 25 05:51:38.974: INFO: Init container init-etcd-3-4-3 ready: true, restart count 0 Mar 25 05:51:38.974: INFO: Init container init-etcd-3-5-0 ready: true, restart count 0 Mar 25 05:51:38.974: INFO: Init container init-etcd-3-5-1 ready: true, restart count 0 Mar 25 05:51:38.974: INFO: Init container init-etcd-3-5-3 ready: true, restart count 0 Mar 25 05:51:38.974: INFO: Init container init-etcd-3-5-4 ready: true, restart count 0 Mar 25 05:51:38.974: INFO: Init container init-etcd-3-5-6 ready: true, restart count 0 Mar 25 05:51:38.974: INFO: Init container init-etcd-3-5-7 ready: true, restart count 0 Mar 25 05:51:38.974: INFO: Container etcd-manager ready: true, restart count 0 Mar 25 05:51:38.974: INFO: kube-proxy-i-0168f82458996cc53 started at 2023-03-25 05:17:36 +0000 UTC (0+1 container statuses recorded) Mar 25 05:51:38.974: INFO: Container kube-proxy ready: true, restart count 0 Mar 25 05:51:38.974: INFO: kube-scheduler-i-0168f82458996cc53 started at 2023-03-25 05:17:36 +0000 UTC (0+1 container statuses recorded) Mar 25 05:51:38.974: INFO: Container kube-scheduler ready: true, restart count 0 Mar 25 05:51:38.974: INFO: etcd-manager-main-i-0168f82458996cc53 started at 2023-03-25 05:17:36 +0000 UTC (11+1 container statuses recorded) Mar 25 05:51:38.974: INFO: Init container init-etcd-3-2-24 ready: true, restart count 0 Mar 25 05:51:38.974: INFO: Init container init-etcd-3-3-10 ready: true, restart count 0 Mar 25 05:51:38.974: INFO: Init container init-etcd-3-3-17 ready: true, restart count 0 Mar 25 05:51:38.974: INFO: Init container init-etcd-3-4-13 ready: true, restart count 0 Mar 25 05:51:38.974: INFO: Init container init-etcd-3-4-3 ready: true, restart count 0 Mar 25 05:51:38.974: INFO: Init container init-etcd-3-5-0 ready: true, restart count 0 Mar 25 05:51:38.974: INFO: Init container init-etcd-3-5-1 ready: true, restart count 0 Mar 25 05:51:38.974: INFO: Init container init-etcd-3-5-3 ready: true, restart count 0 Mar 25 05:51:38.974: INFO: Init container init-etcd-3-5-4 ready: true, restart count 0 Mar 25 05:51:38.974: INFO: Init container init-etcd-3-5-6 ready: true, restart count 0 Mar 25 05:51:38.974: INFO: Init container init-etcd-3-5-7 ready: true, restart count 0 Mar 25 05:51:38.974: INFO: Container etcd-manager ready: true, restart count 0 Mar 25 05:51:38.974: INFO: dns-controller-69987775c6-hbn69 started at 2023-03-25 05:19:47 +0000 UTC (0+1 container statuses recorded) Mar 25 05:51:38.974: INFO: Container dns-controller ready: true, restart count 0 Mar 25 05:51:38.974: INFO: aws-cloud-controller-manager-26vbv started at 2023-03-25 05:20:06 +0000 UTC (0+1 container statuses recorded) Mar 25 05:51:38.974: INFO: Container aws-cloud-controller-manager ready: true, restart count 0 Mar 25 05:51:38.974: INFO: ebs-csi-node-zjtj6 started at 2023-03-25 05:20:06 +0000 UTC (0+3 container statuses recorded) Mar 25 05:51:38.974: INFO: Container ebs-plugin ready: true, restart count 0 Mar 25 05:51:38.974: INFO: Container liveness-probe ready: true, restart count 0 Mar 25 05:51:38.974: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 25 05:51:38.974: INFO: kops-controller-cgw95 started at 2023-03-25 05:20:06 +0000 UTC (0+1 container statuses recorded) Mar 25 05:51:38.974: INFO: Container kops-controller ready: true, restart count 0 Mar 25 05:51:38.974: INFO: calico-kube-controllers-55f488656-rr8rz started at 2023-03-25 05:20:29 +0000 UTC (0+1 container statuses recorded) Mar 25 05:51:38.974: INFO: Container calico-kube-controllers ready: true, restart count 0 Mar 25 05:51:38.974: INFO: ebs-csi-controller-79f984545f-9q5ks started at 2023-03-25 05:21:26 +0000 UTC (0+5 container statuses recorded) Mar 25 05:51:38.974: INFO: Container csi-attacher ready: true, restart count 0 Mar 25 05:51:38.974: INFO: Container csi-provisioner ready: true, restart count 0 Mar 25 05:51:38.974: INFO: Container csi-resizer ready: true, restart count 0 Mar 25 05:51:38.974: INFO: Container ebs-plugin ready: true, restart count 0 Mar 25 05:51:38.974: INFO: Container liveness-probe ready: true, restart count 0 Mar 25 05:51:38.974: INFO: kube-apiserver-i-0168f82458996cc53 started at 2023-03-25 05:17:36 +0000 UTC (0+2 container statuses recorded) Mar 25 05:51:38.974: INFO: Container healthcheck ready: true, restart count 0 Mar 25 05:51:38.974: INFO: Container kube-apiserver ready: true, restart count 2 Mar 25 05:51:38.974: INFO: calico-node-rdjq2 started at 2023-03-25 05:20:06 +0000 UTC (3+1 container statuses recorded) Mar 25 05:51:38.974: INFO: Init container upgrade-ipam ready: true, restart count 0 Mar 25 05:51:38.974: INFO: Init container install-cni ready: true, restart count 0 Mar 25 05:51:38.974: INFO: Init container mount-bpffs ready: true, restart count 0 Mar 25 05:51:38.974: INFO: Container calico-node ready: true, restart count 0 Mar 25 05:51:39.202: INFO: Latency metrics for node i-0168f82458996cc53 Mar 25 05:51:39.202: INFO: Logging node info for node i-03d301e6f030748f1 Mar 25 05:51:39.251: INFO: Node Info: &Node{ObjectMeta:{i-03d301e6f030748f1 897f52bd-65e2-4107-843a-37e36aa74032 54004 0 2023-03-25 05:31:35 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-1 failure-domain.beta.kubernetes.io/zone:us-west-1b kubernetes.io/arch:amd64 kubernetes.io/hostname:i-03d301e6f030748f1 kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:us-west-1b topology.hostpath.csi/node:i-03d301e6f030748f1 topology.kubernetes.io/region:us-west-1 topology.kubernetes.io/zone:us-west-1b] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-volume-expand-2830":"i-03d301e6f030748f1","ebs.csi.aws.com":"i-03d301e6f030748f1"} node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:172.20.50.199/19 projectcalico.org/IPv4IPIPTunnelAddr:100.110.167.64 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-03-25 05:31:35 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-25 05:31:35 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-03-25 05:31:35 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2023-03-25 05:31:35 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2023-03-25 05:31:48 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.8.0/24\"":{}}}} } {calico-node Update v1 2023-03-25 05:31:56 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4IPIPTunnelAddr":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2023-03-25 05:51:27 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-03-25 05:51:31 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.8.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-1b/i-03d301e6f030748f1,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.8.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-03-25 05:31:56 +0000 UTC,LastTransitionTime:2023-03-25 05:31:56 +0000 UTC,Reason:CalicoIsUp,Message:Calico is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-25 05:51:31 +0000 UTC,LastTransitionTime:2023-03-25 05:31:35 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-25 05:51:31 +0000 UTC,LastTransitionTime:2023-03-25 05:31:35 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-25 05:51:31 +0000 UTC,LastTransitionTime:2023-03-25 05:31:35 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-25 05:51:31 +0000 UTC,LastTransitionTime:2023-03-25 05:31:48 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.50.199,},NodeAddress{Type:ExternalIP,Address:52.53.245.63,},NodeAddress{Type:InternalDNS,Address:i-03d301e6f030748f1.us-west-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-03d301e6f030748f1.us-west-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-52-53-245-63.us-west-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec27152f7af9b9bfd1355ee19c256086,SystemUUID:ec27152f-7af9-b9bf-d135-5ee19c256086,BootID:1c303852-f2ce-45f9-ab14-6e8de97a0a3e,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.28.0-alpha.0,KubeProxyVersion:v1.28.0-alpha.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:112030336,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:51eae8381dcb1078289fa7b4f3df2630cdc18d09fb56f8e56b41c40e191d6c83 registry.k8s.io/etcd:3.5.7-0],SizeBytes:101639218,},ContainerImage{Names:[docker.io/calico/cni@sha256:e282ea2914c806b5de2976330a17cfb5e6dcef47147bceb1432ca5c75fd46f50 docker.io/calico/cni:v3.24.5],SizeBytes:87458443,},ContainerImage{Names:[docker.io/calico/node@sha256:5972ad2bcbdc668564d3e26960c9c513b2d7b05581c704747cf7c62ef3a405a6 docker.io/calico/node:v3.24.5],SizeBytes:81584866,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nautilus@sha256:80ba6c8c44f9623f06e868a1aa66026c8ec438ad814f9ec95e9333b415fe3550 registry.k8s.io/e2e-test-images/nautilus:1.7],SizeBytes:49641698,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41901587,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:e468dddcd275163a042ab297b2d8c2aca50d5e148d2d22f3b6ba119e2f31fa79 registry.k8s.io/sig-storage/csi-provisioner:v3.4.0],SizeBytes:27427836,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/sample-apiserver@sha256:8d70890151aa5d096f331cb9da1b9cd5be0412b7363fe67b5c3befdcaa2a28d0 registry.k8s.io/e2e-test-images/sample-apiserver:1.17.7],SizeBytes:25667066,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/kube-proxy@sha256:8a03b3147cd7be8e8235ae1cd70cf69aca9ce03ba1cfaffea320540bb5afa496 registry.k8s.io/kube-proxy:v1.28.0-alpha.0],SizeBytes:23896850,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:23881995,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18758628,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18233005,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonroot@sha256:ee9f50b3c64b174d296d91ca9f69a914ac30e59095dfb462b2b518ad28a63655 registry.k8s.io/e2e-test-images/nonroot:1.4],SizeBytes:17747885,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8688564,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:7000082,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6978614,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac registry.k8s.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[kubernetes.io/csi/ebs.csi.aws.com^vol-00b32985831488b7c],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-00b32985831488b7c,DevicePath:,},},Config:nil,},} Mar 25 05:51:39.251: INFO: Logging kubelet events for node i-03d301e6f030748f1 Mar 25 05:51:39.307: INFO: Logging pods the kubelet thinks is on node i-03d301e6f030748f1 Mar 25 05:51:39.363: INFO: kube-proxy-i-03d301e6f030748f1 started at 2023-03-25 05:31:35 +0000 UTC (0+1 container statuses recorded) Mar 25 05:51:39.363: INFO: Container kube-proxy ready: true, restart count 0 Mar 25 05:51:39.363: INFO: ebs-csi-node-lljwl started at 2023-03-25 05:31:35 +0000 UTC (0+3 container statuses recorded) Mar 25 05:51:39.363: INFO: Container ebs-plugin ready: true, restart count 0 Mar 25 05:51:39.363: INFO: Container liveness-probe ready: true, restart count 0 Mar 25 05:51:39.363: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 25 05:51:39.363: INFO: hostexec-i-03d301e6f030748f1-dp26z started at 2023-03-25 05:51:22 +0000 UTC (0+1 container statuses recorded) Mar 25 05:51:39.363: INFO: Container agnhost-container ready: true, restart count 0 Mar 25 05:51:39.363: INFO: webserver-pod started at 2023-03-25 05:47:27 +0000 UTC (0+1 container statuses recorded) Mar 25 05:51:39.363: INFO: Container agnhost ready: false, restart count 0 Mar 25 05:51:39.363: INFO: startup-af050f25-aca5-4b88-833f-99d48995ab82 started at 2023-03-25 05:49:11 +0000 UTC (0+1 container statuses recorded) Mar 25 05:51:39.363: INFO: Container busybox ready: false, restart count 0 Mar 25 05:51:39.363: INFO: csi-hostpathplugin-0 started at 2023-03-25 05:51:23 +0000 UTC (0+7 container statuses recorded) Mar 25 05:51:39.363: INFO: Container csi-attacher ready: true, restart count 0 Mar 25 05:51:39.363: INFO: Container csi-provisioner ready: true, restart count 0 Mar 25 05:51:39.363: INFO: Container csi-resizer ready: true, restart count 0 Mar 25 05:51:39.363: INFO: Container csi-snapshotter ready: true, restart count 0 Mar 25 05:51:39.363: INFO: Container hostpath ready: true, restart count 0 Mar 25 05:51:39.363: INFO: Container liveness-probe ready: true, restart count 0 Mar 25 05:51:39.363: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 25 05:51:39.363: INFO: pod-configmaps-3fedbd34-8742-431b-9166-666b1586ef4e started at 2023-03-25 05:48:18 +0000 UTC (0+3 container statuses recorded) Mar 25 05:51:39.363: INFO: Container createcm-volume-test ready: true, restart count 0 Mar 25 05:51:39.363: INFO: Container delcm-volume-test ready: true, restart count 0 Mar 25 05:51:39.363: INFO: Container updcm-volume-test ready: true, restart count 0 Mar 25 05:51:39.363: INFO: pod-should-be-evicted7ebf5d43-7564-49c4-b50b-2029ea5c4e2b started at 2023-03-25 05:51:17 +0000 UTC (0+1 container statuses recorded) Mar 25 05:51:39.363: INFO: Container bar ready: true, restart count 0 Mar 25 05:51:39.363: INFO: calico-node-h96vq started at 2023-03-25 05:31:35 +0000 UTC (3+1 container statuses recorded) Mar 25 05:51:39.363: INFO: Init container upgrade-ipam ready: true, restart count 0 Mar 25 05:51:39.363: INFO: Init container install-cni ready: true, restart count 0 Mar 25 05:51:39.363: INFO: Init container mount-bpffs ready: true, restart count 0 Mar 25 05:51:39.363: INFO: Container calico-node ready: true, restart count 0 Mar 25 05:51:39.363: INFO: pod-with-poststart-http-hook started at 2023-03-25 05:50:55 +0000 UTC (0+1 container statuses recorded) Mar 25 05:51:39.363: INFO: Container pod-with-poststart-http-hook ready: true, restart count 0 Mar 25 05:51:39.363: INFO: ss-0 started at 2023-03-25 05:51:22 +0000 UTC (0+1 container statuses recorded) Mar 25 05:51:39.363: INFO: Container webserver ready: true, restart count 0 Mar 25 05:51:39.363: INFO: pod-handle-http-request started at 2023-03-25 05:50:51 +0000 UTC (0+2 container statuses recorded) Mar 25 05:51:39.363: INFO: Container container-handle-http-request ready: true, restart count 0 Mar 25 05:51:39.363: INFO: Container container-handle-https-request ready: true, restart count 0 Mar 25 05:51:39.363: INFO: pod-ephm-test-secret-m6m9 started at 2023-03-25 05:51:01 +0000 UTC (0+1 container statuses recorded) Mar 25 05:51:39.363: INFO: Container test-container-subpath-secret-m6m9 ready: false, restart count 0 Mar 25 05:51:39.363: INFO: pod-subpath-test-preprovisionedpv-hgcm started at 2023-03-25 05:51:37 +0000 UTC (2+1 container statuses recorded) Mar 25 05:51:39.363: INFO: Init container init-volume-preprovisionedpv-hgcm ready: true, restart count 0 Mar 25 05:51:39.363: INFO: Init container test-init-subpath-preprovisionedpv-hgcm ready: false, restart count 0 Mar 25 05:51:39.363: INFO: Container test-container-subpath-preprovisionedpv-hgcm ready: false, restart count 0 Mar 25 05:51:39.728: INFO: Latency metrics for node i-03d301e6f030748f1 Mar 25 05:51:39.728: INFO: Logging node info for node i-049fdb2c1cde894df Mar 25 05:51:39.777: INFO: Node Info: &Node{ObjectMeta:{i-049fdb2c1cde894df b43a51e6-dda7-4047-b368-72d515695b4e 54433 0 2023-03-25 05:23:34 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-1 failure-domain.beta.kubernetes.io/zone:us-west-1b io.kubernetes.storage.mock/node:some-mock-node kubernetes.io/arch:amd64 kubernetes.io/hostname:i-049fdb2c1cde894df kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:us-west-1b topology.hostpath.csi/node:i-049fdb2c1cde894df topology.kubernetes.io/region:us-west-1 topology.kubernetes.io/zone:us-west-1b] map[csi.volume.kubernetes.io/nodeid:{"csi-mock-csi-mock-volumes-expansion-8433":"i-049fdb2c1cde894df","ebs.csi.aws.com":"i-049fdb2c1cde894df"} node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:172.20.43.61/19 projectcalico.org/IPv4IPIPTunnelAddr:100.97.18.64 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-03-25 05:23:34 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-25 05:23:34 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-03-25 05:23:34 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2023-03-25 05:23:34 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2023-03-25 05:23:45 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.6.0/24\"":{}}}} } {calico-node Update v1 2023-03-25 05:23:52 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4IPIPTunnelAddr":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2023-03-25 05:51:29 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:io.kubernetes.storage.mock/node":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status} {kube-controller-manager Update v1 2023-03-25 05:51:39 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.6.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-1b/i-049fdb2c1cde894df,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.6.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-03-25 05:23:52 +0000 UTC,LastTransitionTime:2023-03-25 05:23:52 +0000 UTC,Reason:CalicoIsUp,Message:Calico is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-25 05:51:29 +0000 UTC,LastTransitionTime:2023-03-25 05:23:34 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-25 05:51:29 +0000 UTC,LastTransitionTime:2023-03-25 05:23:34 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-25 05:51:29 +0000 UTC,LastTransitionTime:2023-03-25 05:23:34 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-25 05:51:29 +0000 UTC,LastTransitionTime:2023-03-25 05:23:45 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.43.61,},NodeAddress{Type:ExternalIP,Address:18.144.75.67,},NodeAddress{Type:InternalDNS,Address:i-049fdb2c1cde894df.us-west-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-049fdb2c1cde894df.us-west-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-18-144-75-67.us-west-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec252e592d19e1e6c583e87505431016,SystemUUID:ec252e59-2d19-e1e6-c583-e87505431016,BootID:45436b5d-1eb0-4b6f-a1d1-a1492b31b5d7,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.28.0-alpha.0,KubeProxyVersion:v1.28.0-alpha.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:112030336,},ContainerImage{Names:[docker.io/calico/cni@sha256:e282ea2914c806b5de2976330a17cfb5e6dcef47147bceb1432ca5c75fd46f50 docker.io/calico/cni:v3.24.5],SizeBytes:87458443,},ContainerImage{Names:[docker.io/calico/node@sha256:5972ad2bcbdc668564d3e26960c9c513b2d7b05581c704747cf7c62ef3a405a6 docker.io/calico/node:v3.24.5],SizeBytes:81584866,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41901587,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:e468dddcd275163a042ab297b2d8c2aca50d5e148d2d22f3b6ba119e2f31fa79 registry.k8s.io/sig-storage/csi-provisioner:v3.4.0],SizeBytes:27427836,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/kube-proxy@sha256:8a03b3147cd7be8e8235ae1cd70cf69aca9ce03ba1cfaffea320540bb5afa496 registry.k8s.io/kube-proxy:v1.28.0-alpha.0],SizeBytes:23896850,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:23881995,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:68d396900aeaa072c1f27289485fdac29834045a6f3ffe369bf389d830ef572d registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.6],SizeBytes:20293261,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18758628,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18233005,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:15273057,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8688564,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:7000082,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6978614,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[kubernetes.io/csi/csi-mock-csi-mock-volumes-expansion-8433^6f2d5757-cad0-11ed-b4c8-3aceddafb7a9 kubernetes.io/csi/ebs.csi.aws.com^vol-07a89faa725070a64],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-0bdc719e4a134da12,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-07a89faa725070a64,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-mock-csi-mock-volumes-expansion-8433^6f2d5757-cad0-11ed-b4c8-3aceddafb7a9,DevicePath:,},},Config:nil,},} Mar 25 05:51:39.777: INFO: Logging kubelet events for node i-049fdb2c1cde894df Mar 25 05:51:39.833: INFO: Logging pods the kubelet thinks is on node i-049fdb2c1cde894df Mar 25 05:51:39.896: INFO: coredns-autoscaler-7cb5c5b969-h485x started at 2023-03-25 05:24:36 +0000 UTC (0+1 container statuses recorded) Mar 25 05:51:39.896: INFO: Container autoscaler ready: true, restart count 0 Mar 25 05:51:39.896: INFO: calico-node-wt5p7 started at 2023-03-25 05:23:35 +0000 UTC (3+1 container statuses recorded) Mar 25 05:51:39.896: INFO: Init container upgrade-ipam ready: true, restart count 0 Mar 25 05:51:39.896: INFO: Init container install-cni ready: true, restart count 0 Mar 25 05:51:39.896: INFO: Init container mount-bpffs ready: true, restart count 0 Mar 25 05:51:39.896: INFO: Container calico-node ready: true, restart count 0 Mar 25 05:51:39.896: INFO: external-client started at 2023-03-25 05:51:27 +0000 UTC (0+1 container statuses recorded) Mar 25 05:51:39.896: INFO: Container external-client ready: false, restart count 0 Mar 25 05:51:39.896: INFO: csi-mockplugin-0 started at 2023-03-25 05:46:43 +0000 UTC (0+3 container statuses recorded) Mar 25 05:51:39.896: INFO: Container csi-provisioner ready: true, restart count 0 Mar 25 05:51:39.896: INFO: Container driver-registrar ready: true, restart count 0 Mar 25 05:51:39.896: INFO: Container mock ready: true, restart count 0 Mar 25 05:51:39.896: INFO: csi-mockplugin-resizer-0 started at 2023-03-25 05:46:43 +0000 UTC (0+1 container statuses recorded) Mar 25 05:51:39.896: INFO: Container csi-resizer ready: true, restart count 0 Mar 25 05:51:39.896: INFO: csi-mockplugin-attacher-0 started at 2023-03-25 05:46:43 +0000 UTC (0+1 container statuses recorded) Mar 25 05:51:39.896: INFO: Container csi-attacher ready: true, restart count 0 Mar 25 05:51:39.896: INFO: ss-1 started at 2023-03-25 05:51:36 +0000 UTC (0+1 container statuses recorded) Mar 25 05:51:39.896: INFO: Container webserver ready: false, restart count 0 Mar 25 05:51:39.896: INFO: csi-hostpathplugin-0 started at 2023-03-25 05:51:38 +0000 UTC (0+7 container statuses recorded) Mar 25 05:51:39.896: INFO: Container csi-attacher ready: false, restart count 0 Mar 25 05:51:39.896: INFO: Container csi-provisioner ready: false, restart count 0 Mar 25 05:51:39.896: INFO: Container csi-resizer ready: false, restart count 0 Mar 25 05:51:39.896: INFO: Container csi-snapshotter ready: false, restart count 0 Mar 25 05:51:39.896: INFO: Container hostpath ready: false, restart count 0 Mar 25 05:51:39.896: INFO: Container liveness-probe ready: false, restart count 0 Mar 25 05:51:39.896: INFO: Container node-driver-registrar ready: false, restart count 0 Mar 25 05:51:39.896: INFO: ebs-csi-node-m9zxz started at 2023-03-25 05:23:35 +0000 UTC (0+3 container statuses recorded) Mar 25 05:51:39.896: INFO: Container ebs-plugin ready: true, restart count 0 Mar 25 05:51:39.896: INFO: Container liveness-probe ready: true, restart count 0 Mar 25 05:51:39.896: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 25 05:51:39.896: INFO: coredns-559769c974-pgqhl started at 2023-03-25 05:24:36 +0000 UTC (0+1 container statuses recorded) Mar 25 05:51:39.896: INFO: Container coredns ready: true, restart count 0 Mar 25 05:51:39.896: INFO: pvc-volume-tester-zz4pt started at 2023-03-25 05:46:48 +0000 UTC (0+1 container statuses recorded) Mar 25 05:51:39.896: INFO: Container volume-tester ready: true, restart count 0 Mar 25 05:51:39.896: INFO: kube-proxy-i-049fdb2c1cde894df started at 2023-03-25 05:23:35 +0000 UTC (0+1 container statuses recorded) Mar 25 05:51:39.896: INFO: Container kube-proxy ready: true, restart count 0 Mar 25 05:51:40.268: INFO: Latency metrics for node i-049fdb2c1cde894df Mar 25 05:51:40.268: INFO: Logging node info for node i-062679bb8c0ba4bce Mar 25 05:51:40.317: INFO: Node Info: &Node{ObjectMeta:{i-062679bb8c0ba4bce ee2da569-bb88-4297-9e6f-d2b15ba7ce86 54450 0 2023-03-25 05:27:31 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-1 failure-domain.beta.kubernetes.io/zone:us-west-1b kubernetes.io/arch:amd64 kubernetes.io/hostname:i-062679bb8c0ba4bce kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:us-west-1b topology.hostpath.csi/node:i-062679bb8c0ba4bce topology.kubernetes.io/region:us-west-1 topology.kubernetes.io/zone:us-west-1b] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-volume-expand-6509":"i-062679bb8c0ba4bce","csi-mock-csi-mock-volumes-workload-9268":"i-062679bb8c0ba4bce","ebs.csi.aws.com":"i-062679bb8c0ba4bce"} node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:172.20.45.43/19 projectcalico.org/IPv4IPIPTunnelAddr:100.106.79.0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-03-25 05:27:31 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-25 05:27:31 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-03-25 05:27:31 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2023-03-25 05:27:31 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2023-03-25 05:27:44 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.7.0/24\"":{}}}} } {calico-node Update v1 2023-03-25 05:27:50 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4IPIPTunnelAddr":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2023-03-25 05:51:37 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-03-25 05:51:39 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.7.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-1b/i-062679bb8c0ba4bce,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.7.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051689472 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946831872 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-03-25 05:27:50 +0000 UTC,LastTransitionTime:2023-03-25 05:27:50 +0000 UTC,Reason:CalicoIsUp,Message:Calico is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-25 05:51:39 +0000 UTC,LastTransitionTime:2023-03-25 05:27:31 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-25 05:51:39 +0000 UTC,LastTransitionTime:2023-03-25 05:27:31 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-25 05:51:39 +0000 UTC,LastTransitionTime:2023-03-25 05:27:31 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-25 05:51:39 +0000 UTC,LastTransitionTime:2023-03-25 05:27:44 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.45.43,},NodeAddress{Type:ExternalIP,Address:3.101.133.108,},NodeAddress{Type:InternalDNS,Address:i-062679bb8c0ba4bce.us-west-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-062679bb8c0ba4bce.us-west-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-3-101-133-108.us-west-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2528f599e9b1e199793fb5cd06d205,SystemUUID:ec2528f5-99e9-b1e1-9979-3fb5cd06d205,BootID:ff56b70b-c8c5-46db-8cc5-4353c8f535c6,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.28.0-alpha.0,KubeProxyVersion:v1.28.0-alpha.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:112030336,},ContainerImage{Names:[docker.io/calico/cni@sha256:e282ea2914c806b5de2976330a17cfb5e6dcef47147bceb1432ca5c75fd46f50 docker.io/calico/cni:v3.24.5],SizeBytes:87458443,},ContainerImage{Names:[docker.io/calico/node@sha256:5972ad2bcbdc668564d3e26960c9c513b2d7b05581c704747cf7c62ef3a405a6 docker.io/calico/node:v3.24.5],SizeBytes:81584866,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nautilus@sha256:80ba6c8c44f9623f06e868a1aa66026c8ec438ad814f9ec95e9333b415fe3550 registry.k8s.io/e2e-test-images/nautilus:1.7],SizeBytes:49641698,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41901587,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:e468dddcd275163a042ab297b2d8c2aca50d5e148d2d22f3b6ba119e2f31fa79 registry.k8s.io/sig-storage/csi-provisioner:v3.4.0],SizeBytes:27427836,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/kube-proxy@sha256:8a03b3147cd7be8e8235ae1cd70cf69aca9ce03ba1cfaffea320540bb5afa496 registry.k8s.io/kube-proxy:v1.28.0-alpha.0],SizeBytes:23896850,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:23881995,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18758628,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/regression-issue-74839@sha256:b4f1d8d61bdad84bd50442d161d5460e4019d53e989b64220fdbc62fc87d76bf registry.k8s.io/e2e-test-images/regression-issue-74839:1.2],SizeBytes:18651485,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18233005,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:15273057,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8688564,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/build-image/distroless-iptables@sha256:13ac9b3c476d7290a4451d65d09d6016a2cb89836ffbfa4eae55b72731a22080 registry.k8s.io/build-image/distroless-iptables:v0.2.2],SizeBytes:7729580,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:7000082,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6978614,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[kubernetes.io/csi/csi-hostpath-volume-expand-6509^184bf3d4-cad1-11ed-bcec-62db49d9647a kubernetes.io/csi/csi-mock-csi-mock-volumes-workload-9268^1895dafe-cad1-11ed-95bc-d644bb3827b3 kubernetes.io/csi/ebs.csi.aws.com^vol-06f363257c9807b4f],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-volume-expand-6509^184bf3d4-cad1-11ed-bcec-62db49d9647a,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-06f363257c9807b4f,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-mock-csi-mock-volumes-workload-9268^1895dafe-cad1-11ed-95bc-d644bb3827b3,DevicePath:,},},Config:nil,},} Mar 25 05:51:40.317: INFO: Logging kubelet events for node i-062679bb8c0ba4bce Mar 25 05:51:40.373: INFO: Logging pods the kubelet thinks is on node i-062679bb8c0ba4bce Mar 25 05:51:40.432: INFO: csi-hostpathplugin-0 started at 2023-03-25 05:51:30 +0000 UTC (0+7 container statuses recorded) Mar 25 05:51:40.432: INFO: Container csi-attacher ready: true, restart count 0 Mar 25 05:51:40.432: INFO: Container csi-provisioner ready: true, restart count 0 Mar 25 05:51:40.432: INFO: Container csi-resizer ready: true, restart count 0 Mar 25 05:51:40.432: INFO: Container csi-snapshotter ready: true, restart count 0 Mar 25 05:51:40.432: INFO: Container hostpath ready: true, restart count 0 Mar 25 05:51:40.432: INFO: Container liveness-probe ready: true, restart count 0 Mar 25 05:51:40.432: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 25 05:51:40.432: INFO: pvc-volume-tester-qv85d started at 2023-03-25 05:51:32 +0000 UTC (0+1 container statuses recorded) Mar 25 05:51:40.432: INFO: Container volume-tester ready: false, restart count 0 Mar 25 05:51:40.432: INFO: ss-0 started at 2023-03-25 05:51:30 +0000 UTC (0+1 container statuses recorded) Mar 25 05:51:40.432: INFO: Container webserver ready: false, restart count 0 Mar 25 05:51:40.432: INFO: ebs-csi-node-7twrq started at 2023-03-25 05:27:32 +0000 UTC (0+3 container statuses recorded) Mar 25 05:51:40.432: INFO: Container ebs-plugin ready: true, restart count 0 Mar 25 05:51:40.432: INFO: Container liveness-probe ready: true, restart count 0 Mar 25 05:51:40.432: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 25 05:51:40.432: INFO: pod-disruption-failure-ignore-3-s48j5 started at 2023-03-25 05:51:36 +0000 UTC (0+1 container statuses recorded) Mar 25 05:51:40.432: INFO: Container c ready: false, restart count 0 Mar 25 05:51:40.432: INFO: pod-disruption-failure-ignore-1-kwbfl started at 2023-03-25 05:51:23 +0000 UTC (0+1 container statuses recorded) Mar 25 05:51:40.432: INFO: Container c ready: false, restart count 0 Mar 25 05:51:40.432: INFO: pod-disruption-failure-ignore-1-6s2ng started at 2023-03-25 05:51:27 +0000 UTC (0+1 container statuses recorded) Mar 25 05:51:40.432: INFO: Container c ready: false, restart count 0 Mar 25 05:51:40.432: INFO: pod-disruption-failure-ignore-0-dddnk started at 2023-03-25 05:51:23 +0000 UTC (0+1 container statuses recorded) Mar 25 05:51:40.432: INFO: Container c ready: true, restart count 0 Mar 25 05:51:40.432: INFO: pod-disruption-failure-ignore-2-tv67s started at 2023-03-25 05:51:32 +0000 UTC (0+1 container statuses recorded) Mar 25 05:51:40.432: INFO: Container c ready: false, restart count 0 Mar 25 05:51:40.432: INFO: pod-47a8fc62-d724-47f4-a865-f0444f9055fe started at 2023-03-25 05:51:34 +0000 UTC (0+1 container statuses recorded) Mar 25 05:51:40.432: INFO: Container write-pod ready: false, restart count 0 Mar 25 05:51:40.432: INFO: hostexec-i-062679bb8c0ba4bce-t7kj9 started at 2023-03-25 05:51:36 +0000 UTC (0+1 container statuses recorded) Mar 25 05:51:40.432: INFO: Container agnhost-container ready: true, restart count 0 Mar 25 05:51:40.432: INFO: csi-mockplugin-0 started at 2023-03-25 05:51:27 +0000 UTC (0+3 container statuses recorded) Mar 25 05:51:40.432: INFO: Container csi-provisioner ready: true, restart count 0 Mar 25 05:51:40.432: INFO: Container driver-registrar ready: true, restart count 0 Mar 25 05:51:40.432: INFO: Container mock ready: true, restart count 0 Mar 25 05:51:40.432: INFO: csi-mockplugin-attacher-0 started at 2023-03-25 05:51:27 +0000 UTC (0+1 container statuses recorded) Mar 25 05:51:40.432: INFO: Container csi-attacher ready: true, restart count 0 Mar 25 05:51:40.432: INFO: kube-proxy-i-062679bb8c0ba4bce started at 2023-03-25 05:27:32 +0000 UTC (0+1 container statuses recorded) Mar 25 05:51:40.432: INFO: Container kube-proxy ready: true, restart count 0 Mar 25 05:51:40.432: INFO: calico-node-zwm5j started at 2023-03-25 05:27:32 +0000 UTC (3+1 container statuses recorded) Mar 25 05:51:40.432: INFO: Init container upgrade-ipam ready: true, restart count 0 Mar 25 05:51:40.432: INFO: Init container install-cni ready: true, restart count 0 Mar 25 05:51:40.432: INFO: Init container mount-bpffs ready: true, restart count 0 Mar 25 05:51:40.432: INFO: Container calico-node ready: true, restart count 0 Mar 25 05:51:40.432: INFO: coredns-559769c974-p9vg6 started at 2023-03-25 05:36:16 +0000 UTC (0+1 container statuses recorded) Mar 25 05:51:40.432: INFO: Container coredns ready: true, restart count 0 Mar 25 05:51:40.825: INFO: Latency metrics for node i-062679bb8c0ba4bce Mar 25 05:51:40.825: INFO: Logging node info for node i-072608201d2c0f1ad Mar 25 05:51:40.884: INFO: Node Info: &Node{ObjectMeta:{i-072608201d2c0f1ad 264dfba1-3cf0-4c99-8e8c-098f0f792077 53679 0 2023-03-25 05:35:29 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-1 failure-domain.beta.kubernetes.io/zone:us-west-1b kubernetes.io/arch:amd64 kubernetes.io/hostname:i-072608201d2c0f1ad kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:us-west-1b topology.hostpath.csi/node:i-072608201d2c0f1ad topology.kubernetes.io/region:us-west-1 topology.kubernetes.io/zone:us-west-1b] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-5774":"i-072608201d2c0f1ad","ebs.csi.aws.com":"i-072608201d2c0f1ad"} node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:172.20.38.125/19 projectcalico.org/IPv4IPIPTunnelAddr:100.107.220.192 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-03-25 05:35:29 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-25 05:35:29 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-03-25 05:35:29 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2023-03-25 05:35:29 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2023-03-25 05:35:41 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.9.0/24\"":{}}}} } {calico-node Update v1 2023-03-25 05:35:48 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4IPIPTunnelAddr":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2023-03-25 05:51:26 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.9.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-1b/i-072608201d2c0f1ad,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.9.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051689472 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946831872 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-03-25 05:35:47 +0000 UTC,LastTransitionTime:2023-03-25 05:35:47 +0000 UTC,Reason:CalicoIsUp,Message:Calico is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-25 05:51:20 +0000 UTC,LastTransitionTime:2023-03-25 05:35:29 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-25 05:51:20 +0000 UTC,LastTransitionTime:2023-03-25 05:35:29 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-25 05:51:20 +0000 UTC,LastTransitionTime:2023-03-25 05:35:29 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-25 05:51:20 +0000 UTC,LastTransitionTime:2023-03-25 05:35:41 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.38.125,},NodeAddress{Type:ExternalIP,Address:3.101.88.10,},NodeAddress{Type:InternalDNS,Address:i-072608201d2c0f1ad.us-west-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-072608201d2c0f1ad.us-west-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-3-101-88-10.us-west-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2d3818c4f80a253dcc1a60e148dbc8,SystemUUID:ec2d3818-c4f8-0a25-3dcc-1a60e148dbc8,BootID:77d046cf-75ea-4a8c-970a-3190d64e019b,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.28.0-alpha.0,KubeProxyVersion:v1.28.0-alpha.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:112030336,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:51eae8381dcb1078289fa7b4f3df2630cdc18d09fb56f8e56b41c40e191d6c83 registry.k8s.io/etcd:3.5.7-0],SizeBytes:101639218,},ContainerImage{Names:[docker.io/calico/cni@sha256:e282ea2914c806b5de2976330a17cfb5e6dcef47147bceb1432ca5c75fd46f50 docker.io/calico/cni:v3.24.5],SizeBytes:87458443,},ContainerImage{Names:[docker.io/calico/node@sha256:5972ad2bcbdc668564d3e26960c9c513b2d7b05581c704747cf7c62ef3a405a6 docker.io/calico/node:v3.24.5],SizeBytes:81584866,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nautilus@sha256:80ba6c8c44f9623f06e868a1aa66026c8ec438ad814f9ec95e9333b415fe3550 registry.k8s.io/e2e-test-images/nautilus:1.7],SizeBytes:49641698,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41901587,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:e468dddcd275163a042ab297b2d8c2aca50d5e148d2d22f3b6ba119e2f31fa79 registry.k8s.io/sig-storage/csi-provisioner:v3.4.0],SizeBytes:27427836,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/sample-apiserver@sha256:8d70890151aa5d096f331cb9da1b9cd5be0412b7363fe67b5c3befdcaa2a28d0 registry.k8s.io/e2e-test-images/sample-apiserver:1.17.7],SizeBytes:25667066,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/kube-proxy@sha256:8a03b3147cd7be8e8235ae1cd70cf69aca9ce03ba1cfaffea320540bb5afa496 registry.k8s.io/kube-proxy:v1.28.0-alpha.0],SizeBytes:23896850,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:23881995,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18758628,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18233005,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonroot@sha256:ee9f50b3c64b174d296d91ca9f69a914ac30e59095dfb462b2b518ad28a63655 registry.k8s.io/e2e-test-images/nonroot:1.4],SizeBytes:17747885,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8688564,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:7000082,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6978614,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Mar 25 05:51:40.884: INFO: Logging kubelet events for node i-072608201d2c0f1ad Mar 25 05:51:40.958: INFO: Logging pods the kubelet thinks is on node i-072608201d2c0f1ad Mar 25 05:51:41.015: INFO: liveness-042b201c-6dcd-440e-ae49-84b93db25047 started at 2023-03-25 05:47:48 +0000 UTC (0+1 container statuses recorded) Mar 25 05:51:41.015: INFO: Container agnhost-container ready: true, restart count 0 Mar 25 05:51:41.015: INFO: kube-proxy-i-072608201d2c0f1ad started at 2023-03-25 05:35:30 +0000 UTC (0+1 container statuses recorded) Mar 25 05:51:41.015: INFO: Container kube-proxy ready: true, restart count 0 Mar 25 05:51:41.015: INFO: inline-volume-tester-s9fwf started at 2023-03-25 05:51:05 +0000 UTC (0+1 container statuses recorded) Mar 25 05:51:41.015: INFO: Container csi-volume-tester ready: true, restart count 0 Mar 25 05:51:41.015: INFO: ebs-csi-node-r7mcm started at 2023-03-25 05:35:30 +0000 UTC (0+3 container statuses recorded) Mar 25 05:51:41.015: INFO: Container ebs-plugin ready: true, restart count 0 Mar 25 05:51:41.015: INFO: Container liveness-probe ready: true, restart count 0 Mar 25 05:51:41.015: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 25 05:51:41.015: INFO: calico-node-lc9zl started at 2023-03-25 05:35:30 +0000 UTC (3+1 container statuses recorded) Mar 25 05:51:41.015: INFO: Init container upgrade-ipam ready: true, restart count 0 Mar 25 05:51:41.015: INFO: Init container install-cni ready: true, restart count 0 Mar 25 05:51:41.015: INFO: Init container mount-bpffs ready: true, restart count 0 Mar 25 05:51:41.015: INFO: Container calico-node ready: true, restart count 0 Mar 25 05:51:41.015: INFO: pod-configmaps-a8470e12-21d1-4453-85ec-2fef6a8b9767 started at 2023-03-25 05:48:12 +0000 UTC (0+1 container statuses recorded) Mar 25 05:51:41.015: INFO: Container agnhost-container ready: true, restart count 0 Mar 25 05:51:41.015: INFO: csi-hostpathplugin-0 started at 2023-03-25 05:51:05 +0000 UTC (0+7 container statuses recorded) Mar 25 05:51:41.015: INFO: Container csi-attacher ready: true, restart count 0 Mar 25 05:51:41.015: INFO: Container csi-provisioner ready: true, restart count 0 Mar 25 05:51:41.015: INFO: Container csi-resizer ready: true, restart count 0 Mar 25 05:51:41.015: INFO: Container csi-snapshotter ready: true, restart count 0 Mar 25 05:51:41.015: INFO: Container hostpath ready: true, restart count 0 Mar 25 05:51:41.015: INFO: Container liveness-probe ready: true, restart count 0 Mar 25 05:51:41.015: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 25 05:51:41.324: INFO: Latency metrics for node i-072608201d2c0f1ad END STEP: dump namespace information after failure - test/e2e/framework/framework.go:288 @ 03/25/23 05:51:41.324 (2.72s) < Exit [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] subPath - dump namespaces | framework.go:209 @ 03/25/23 05:51:41.324 (2.721s) > Enter [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] subPath - tear down framework | framework.go:206 @ 03/25/23 05:51:41.324 STEP: Destroying namespace "provisioning-7937" for this suite. - test/e2e/framework/framework.go:351 @ 03/25/23 05:51:41.324 < Exit [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] subPath - tear down framework | framework.go:206 @ 03/25/23 05:51:41.375 (51ms) > Enter [ReportAfterEach] TOP-LEVEL - test/e2e/e2e_test.go:144 @ 03/25/23 05:51:41.375 < Exit [ReportAfterEach] TOP-LEVEL - test/e2e/e2e_test.go:144 @ 03/25/23 05:51:41.375 (0s)
Filter through log files | View test history on testgrid
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[It\]\sExternal\sStorage\s\[Driver\:\sebs\.csi\.aws\.com\]\s\[Testpattern\:\sDynamic\sPV\s\(default\sfs\)\]\ssubPath\sshould\ssupport\sreadOnly\sfile\sspecified\sin\sthe\svolumeMount\s\[LinuxOnly\]$'
[FAILED] failed to get logs from pod-subpath-test-dynamicpv-lzxc for test-container-subpath-dynamicpv-lzxc: Get "https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/provisioning-5117/pods/pod-subpath-test-dynamicpv-lzxc/log?container=test-container-subpath-dynamicpv-lzxc&previous=false": stream error: stream ID 1039; INTERNAL_ERROR; received from peer In [It] at: test/e2e/framework/pod/output/output.go:247 @ 03/25/23 05:42:15.764from junit_01.xml
> Enter [BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/storage/framework/testsuite.go:51 @ 03/25/23 05:41:58.446 < Exit [BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/storage/framework/testsuite.go:51 @ 03/25/23 05:41:58.446 (0s) > Enter [BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath - set up framework | framework.go:191 @ 03/25/23 05:41:58.446 STEP: Creating a kubernetes client - test/e2e/framework/framework.go:211 @ 03/25/23 05:41:58.446 Mar 25 05:41:58.446: INFO: >>> kubeConfig: /root/.kube/config STEP: Building a namespace api object, basename provisioning - test/e2e/framework/framework.go:250 @ 03/25/23 05:41:58.447 STEP: Waiting for a default service account to be provisioned in namespace - test/e2e/framework/framework.go:259 @ 03/25/23 05:41:58.6 STEP: Waiting for kube-root-ca.crt to be provisioned in namespace - test/e2e/framework/framework.go:262 @ 03/25/23 05:41:58.699 < Exit [BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath - set up framework | framework.go:191 @ 03/25/23 05:41:58.798 (352ms) > Enter [BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/framework/metrics/init/init.go:33 @ 03/25/23 05:41:58.798 < Exit [BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/framework/metrics/init/init.go:33 @ 03/25/23 05:41:58.798 (0s) > Enter [It] should support readOnly file specified in the volumeMount [LinuxOnly] - test/e2e/storage/testsuites/subpath.go:382 @ 03/25/23 05:41:58.798 Mar 25 05:41:58.798: INFO: Creating resource for dynamic PV Mar 25 05:41:58.798: INFO: Using claimSize:1Gi, test suite supported size:{ 1Mi}, driver(ebs.csi.aws.com) supported size:{ 1Mi} STEP: creating a StorageClass provisioning-5117-e2e-scpbzlm - test/e2e/storage/framework/volume_resource.go:102 @ 03/25/23 05:41:58.799 STEP: creating a claim - test/e2e/storage/framework/volume_resource.go:294 @ 03/25/23 05:41:58.85 Mar 25 05:41:58.850: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil STEP: Creating pod pod-subpath-test-dynamicpv-lzxc - test/e2e/storage/testsuites/subpath.go:691 @ 03/25/23 05:41:58.956 STEP: Creating a pod to test subpath - test/e2e/framework/pod/output/output.go:243 @ 03/25/23 05:41:58.956 STEP: Saw pod success - test/e2e/framework/pod/wait.go:409 @ 03/25/23 05:42:15.492 Mar 25 05:42:15.547: INFO: Trying to get logs from node i-03d301e6f030748f1 pod pod-subpath-test-dynamicpv-lzxc container test-container-subpath-dynamicpv-lzxc: <nil> Mar 25 05:42:15.617: INFO: Failed to get logs from node "i-03d301e6f030748f1" pod "pod-subpath-test-dynamicpv-lzxc" container "test-container-subpath-dynamicpv-lzxc". Get "https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/provisioning-5117/pods/pod-subpath-test-dynamicpv-lzxc/log?container=test-container-subpath-dynamicpv-lzxc&previous=false": stream error: stream ID 1039; INTERNAL_ERROR; received from peer STEP: delete the pod - test/e2e/framework/pod/output/output.go:168 @ 03/25/23 05:42:15.617 Mar 25 05:42:15.763: INFO: Unexpected error: <*fmt.wrapError | 0xc004c14680>: failed to get logs from pod-subpath-test-dynamicpv-lzxc for test-container-subpath-dynamicpv-lzxc: Get "https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/provisioning-5117/pods/pod-subpath-test-dynamicpv-lzxc/log?container=test-container-subpath-dynamicpv-lzxc&previous=false": stream error: stream ID 1039; INTERNAL_ERROR; received from peer { msg: "failed to get logs from pod-subpath-test-dynamicpv-lzxc for test-container-subpath-dynamicpv-lzxc: Get \"https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/provisioning-5117/pods/pod-subpath-test-dynamicpv-lzxc/log?container=test-container-subpath-dynamicpv-lzxc&previous=false\": stream error: stream ID 1039; INTERNAL_ERROR; received from peer", err: <*url.Error | 0xc004c26ff0>{ Op: "Get", URL: "https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/provisioning-5117/pods/pod-subpath-test-dynamicpv-lzxc/log?container=test-container-subpath-dynamicpv-lzxc&previous=false", Err: <http2.StreamError>{ StreamID: 1039, Code: 2, Cause: <*errors.errorString | 0xc0001b3650>{ s: "received from peer", }, }, }, } [FAILED] failed to get logs from pod-subpath-test-dynamicpv-lzxc for test-container-subpath-dynamicpv-lzxc: Get "https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/provisioning-5117/pods/pod-subpath-test-dynamicpv-lzxc/log?container=test-container-subpath-dynamicpv-lzxc&previous=false": stream error: stream ID 1039; INTERNAL_ERROR; received from peer In [It] at: test/e2e/framework/pod/output/output.go:247 @ 03/25/23 05:42:15.764 < Exit [It] should support readOnly file specified in the volumeMount [LinuxOnly] - test/e2e/storage/testsuites/subpath.go:382 @ 03/25/23 05:42:15.764 (16.965s) > Enter [AfterEach] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/framework/node/init/init.go:33 @ 03/25/23 05:42:15.764 Mar 25 05:42:15.764: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready < Exit [AfterEach] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/framework/node/init/init.go:33 @ 03/25/23 05:42:15.819 (55ms) > Enter [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/storage/testsuites/subpath.go:384 @ 03/25/23 05:42:15.819 STEP: Deleting pod - test/e2e/storage/testsuites/subpath.go:172 @ 03/25/23 05:42:15.819 Mar 25 05:42:15.819: INFO: Deleting pod "pod-subpath-test-dynamicpv-lzxc" in namespace "provisioning-5117" STEP: Deleting pvc - test/e2e/storage/framework/volume_resource.go:181 @ 03/25/23 05:42:15.872 Mar 25 05:42:15.974: INFO: Deleting PersistentVolumeClaim "ebs.csi.aws.com7htm4" Mar 25 05:42:16.031: INFO: Waiting up to 5m0s for PersistentVolume pvc-15fbf43e-3c1c-4b12-acf2-5c3b4325c8ea to get deleted Mar 25 05:42:16.082: INFO: PersistentVolume pvc-15fbf43e-3c1c-4b12-acf2-5c3b4325c8ea found and phase=Released (50.943372ms) Mar 25 05:42:21.132: INFO: PersistentVolume pvc-15fbf43e-3c1c-4b12-acf2-5c3b4325c8ea found and phase=Released (5.101230759s) Mar 25 05:42:26.183: INFO: PersistentVolume pvc-15fbf43e-3c1c-4b12-acf2-5c3b4325c8ea found and phase=Released (10.152212463s) Mar 25 05:42:31.233: INFO: PersistentVolume pvc-15fbf43e-3c1c-4b12-acf2-5c3b4325c8ea was removed STEP: Deleting sc - test/e2e/storage/framework/volume_resource.go:228 @ 03/25/23 05:42:31.233 < Exit [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/storage/testsuites/subpath.go:384 @ 03/25/23 05:42:31.284 (15.465s) > Enter [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/framework/metrics/init/init.go:35 @ 03/25/23 05:42:31.284 < Exit [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/framework/metrics/init/init.go:35 @ 03/25/23 05:42:31.284 (0s) > Enter [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] subPath - dump namespaces | framework.go:209 @ 03/25/23 05:42:31.284 STEP: dump namespace information after failure - test/e2e/framework/framework.go:288 @ 03/25/23 05:42:31.284 STEP: Collecting events from namespace "provisioning-5117". - test/e2e/framework/debug/dump.go:42 @ 03/25/23 05:42:31.284 STEP: Found 15 events. - test/e2e/framework/debug/dump.go:46 @ 03/25/23 05:42:31.334 Mar 25 05:42:31.334: INFO: At 2023-03-25 05:41:58 +0000 UTC - event for ebs.csi.aws.com7htm4: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding Mar 25 05:42:31.334: INFO: At 2023-03-25 05:41:58 +0000 UTC - event for ebs.csi.aws.com7htm4: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator Mar 25 05:42:31.334: INFO: At 2023-03-25 05:41:58 +0000 UTC - event for ebs.csi.aws.com7htm4: {ebs.csi.aws.com_i-0168f82458996cc53_0b589c46-3396-4499-94ba-eb51a48f7a58 } Provisioning: External provisioner is provisioning volume for claim "provisioning-5117/ebs.csi.aws.com7htm4" Mar 25 05:42:31.334: INFO: At 2023-03-25 05:42:02 +0000 UTC - event for ebs.csi.aws.com7htm4: {ebs.csi.aws.com_i-0168f82458996cc53_0b589c46-3396-4499-94ba-eb51a48f7a58 } ProvisioningSucceeded: Successfully provisioned volume pvc-15fbf43e-3c1c-4b12-acf2-5c3b4325c8ea Mar 25 05:42:31.334: INFO: At 2023-03-25 05:42:03 +0000 UTC - event for pod-subpath-test-dynamicpv-lzxc: {default-scheduler } Scheduled: Successfully assigned provisioning-5117/pod-subpath-test-dynamicpv-lzxc to i-03d301e6f030748f1 Mar 25 05:42:31.334: INFO: At 2023-03-25 05:42:05 +0000 UTC - event for pod-subpath-test-dynamicpv-lzxc: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-15fbf43e-3c1c-4b12-acf2-5c3b4325c8ea" Mar 25 05:42:31.334: INFO: At 2023-03-25 05:42:09 +0000 UTC - event for pod-subpath-test-dynamicpv-lzxc: {kubelet i-03d301e6f030748f1} Pulled: Container image "registry.k8s.io/e2e-test-images/busybox:1.29-4" already present on machine Mar 25 05:42:31.334: INFO: At 2023-03-25 05:42:09 +0000 UTC - event for pod-subpath-test-dynamicpv-lzxc: {kubelet i-03d301e6f030748f1} Created: Created container init-volume-dynamicpv-lzxc Mar 25 05:42:31.334: INFO: At 2023-03-25 05:42:10 +0000 UTC - event for pod-subpath-test-dynamicpv-lzxc: {kubelet i-03d301e6f030748f1} Started: Started container init-volume-dynamicpv-lzxc Mar 25 05:42:31.334: INFO: At 2023-03-25 05:42:10 +0000 UTC - event for pod-subpath-test-dynamicpv-lzxc: {kubelet i-03d301e6f030748f1} Pulled: Container image "registry.k8s.io/e2e-test-images/agnhost:2.43" already present on machine Mar 25 05:42:31.334: INFO: At 2023-03-25 05:42:10 +0000 UTC - event for pod-subpath-test-dynamicpv-lzxc: {kubelet i-03d301e6f030748f1} Created: Created container test-init-volume-dynamicpv-lzxc Mar 25 05:42:31.334: INFO: At 2023-03-25 05:42:10 +0000 UTC - event for pod-subpath-test-dynamicpv-lzxc: {kubelet i-03d301e6f030748f1} Started: Started container test-init-volume-dynamicpv-lzxc Mar 25 05:42:31.334: INFO: At 2023-03-25 05:42:11 +0000 UTC - event for pod-subpath-test-dynamicpv-lzxc: {kubelet i-03d301e6f030748f1} Pulled: Container image "registry.k8s.io/e2e-test-images/agnhost:2.43" already present on machine Mar 25 05:42:31.334: INFO: At 2023-03-25 05:42:11 +0000 UTC - event for pod-subpath-test-dynamicpv-lzxc: {kubelet i-03d301e6f030748f1} Created: Created container test-container-subpath-dynamicpv-lzxc Mar 25 05:42:31.334: INFO: At 2023-03-25 05:42:11 +0000 UTC - event for pod-subpath-test-dynamicpv-lzxc: {kubelet i-03d301e6f030748f1} Started: Started container test-container-subpath-dynamicpv-lzxc Mar 25 05:42:31.384: INFO: POD NODE PHASE GRACE CONDITIONS Mar 25 05:42:31.384: INFO: Mar 25 05:42:31.442: INFO: Logging node info for node i-0168f82458996cc53 Mar 25 05:42:31.493: INFO: Node Info: &Node{ObjectMeta:{i-0168f82458996cc53 f4eb302c-c477-45ac-bcac-ffed65f6580f 21197 0 2023-03-25 05:18:57 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-1 failure-domain.beta.kubernetes.io/zone:us-west-1b kops.k8s.io/kops-controller-pki: kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0168f82458996cc53 kubernetes.io/os:linux node-role.kubernetes.io/control-plane: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:us-west-1b topology.kubernetes.io/region:us-west-1 topology.kubernetes.io/zone:us-west-1b] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0168f82458996cc53"} node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:172.20.48.81/19 projectcalico.org/IPv4IPIPTunnelAddr:100.100.215.192 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-03-25 05:18:57 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {protokube Update v1 2023-03-25 05:18:58 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {calico-node Update v1 2023-03-25 05:20:22 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4IPIPTunnelAddr":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {aws-cloud-controller-manager Update v1 2023-03-25 05:20:26 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-25 05:20:26 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2023-03-25 05:20:31 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.5.0/24\"":{}},"f:taints":{}}} } {kubelet Update v1 2023-03-25 05:42:05 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.5.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-1b/i-0168f82458996cc53,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/control-plane,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.5.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3892310016 0} {<nil>} 3801084Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3787452416 0} {<nil>} 3698684Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-03-25 05:20:22 +0000 UTC,LastTransitionTime:2023-03-25 05:20:22 +0000 UTC,Reason:CalicoIsUp,Message:Calico is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-25 05:42:05 +0000 UTC,LastTransitionTime:2023-03-25 05:18:50 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-25 05:42:05 +0000 UTC,LastTransitionTime:2023-03-25 05:18:50 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-25 05:42:05 +0000 UTC,LastTransitionTime:2023-03-25 05:18:50 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-25 05:42:05 +0000 UTC,LastTransitionTime:2023-03-25 05:20:29 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.48.81,},NodeAddress{Type:ExternalIP,Address:3.101.135.77,},NodeAddress{Type:InternalDNS,Address:i-0168f82458996cc53.us-west-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-0168f82458996cc53.us-west-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-3-101-135-77.us-west-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2d1ccc7ebf313420bd1f530a94fbfd,SystemUUID:ec2d1ccc-7ebf-3134-20bd-1f530a94fbfd,BootID:15cee321-56e7-4290-ab48-47b60b0bc657,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.28.0-alpha.0,KubeProxyVersion:v1.28.0-alpha.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcd@sha256:dd75ec974b0a2a6f6bb47001ba09207976e625db898d1b16735528c009cb171c registry.k8s.io/etcd:3.5.6-0],SizeBytes:102542580,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:6f72b851544986cb0921b53ea655ec04c36131248f16d4ad110cb3ca0c369dc1 registry.k8s.io/etcd:3.5.4-0],SizeBytes:102157811,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:13f53ed1d91e2e11aac476ee9a0269fdda6cc4874eba903efd40daf50c55eee5 registry.k8s.io/etcd:3.5.3-0],SizeBytes:102143581,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:51eae8381dcb1078289fa7b4f3df2630cdc18d09fb56f8e56b41c40e191d6c83 registry.k8s.io/etcd:3.5.7-0],SizeBytes:101639218,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:4afb99b4690b418ffc2ceb67e1a17376457e441c1f09ab55447f0aaf992fa646 registry.k8s.io/etcd:3.4.3-0],SizeBytes:100947667,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:9ce33ba33d8e738a5b85ed50b5080ac746deceed4a7496c550927a7a19ca3b6d registry.k8s.io/etcd:3.5.0-0],SizeBytes:99868722,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:64b9ea357325d5db9f8a723dcf503b5a449177b17ac87d69481e126bb724c263 registry.k8s.io/etcd:3.5.1-0],SizeBytes:98888614,},ContainerImage{Names:[docker.io/calico/cni@sha256:e282ea2914c806b5de2976330a17cfb5e6dcef47147bceb1432ca5c75fd46f50 docker.io/calico/cni:v3.24.5],SizeBytes:87458443,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:4ad90a11b55313b182afc186b9876c8e891531b8db4c9bf1541953021618d0e2 registry.k8s.io/etcd:3.4.13-0],SizeBytes:86742272,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:0d2ab6b15b4fe0b903b9e81192d942fb39c29be61fe341cd16db5d0165b0435c registry.k8s.io/etcd:3.3.17-0],SizeBytes:85524408,},ContainerImage{Names:[docker.io/calico/node@sha256:5972ad2bcbdc668564d3e26960c9c513b2d7b05581c704747cf7c62ef3a405a6 docker.io/calico/node:v3.24.5],SizeBytes:81584866,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:17da501f5d2a675be46040422a27b7cc21b8a43895ac998b171db1c346f361f7 registry.k8s.io/etcd:3.3.10-0],SizeBytes:76162358,},ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager-slim@sha256:8727adc059334e0e3329dbccce116a734da4a090da5e86c8cddccaf333caa666 registry.k8s.io/etcdadm/etcd-manager-slim:v3.0.20230201],SizeBytes:63926165,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:905d7ca17fd02bc24c0eba9a062753aba15db3e31422390bc3238eb762339b20 registry.k8s.io/etcd:3.2.24-1],SizeBytes:63423287,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.27.0-alpha.1],SizeBytes:45219545,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.27.0-alpha.1],SizeBytes:43628585,},ContainerImage{Names:[registry.k8s.io/kube-apiserver@sha256:49934a29c559e79ed19fc38b4697b18d8fe174b17bf56192588575f980c93956 registry.k8s.io/kube-apiserver:v1.28.0-alpha.0],SizeBytes:33212506,},ContainerImage{Names:[docker.io/calico/kube-controllers@sha256:2b6acd7f677f76ffe12ecf3ea7df92eb9b1bdb07336d1ac2a54c7631fb753f7e docker.io/calico/kube-controllers:v3.24.5],SizeBytes:31137169,},ContainerImage{Names:[registry.k8s.io/kube-controller-manager@sha256:974748384634982c73527fdb3c9ec0ec476bc0d52b3dd1d611a380b7f99af1f3 registry.k8s.io/kube-controller-manager:v1.28.0-alpha.0],SizeBytes:30820280,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/kube-proxy@sha256:8a03b3147cd7be8e8235ae1cd70cf69aca9ce03ba1cfaffea320540bb5afa496 registry.k8s.io/kube-proxy:v1.28.0-alpha.0],SizeBytes:23896850,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:23345856,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:22381475,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:22085298,},ContainerImage{Names:[registry.k8s.io/provider-aws/cloud-controller-manager@sha256:fdeb61e3e42ecd9cca868d550ebdb88dd6341d9e91fcfa9a37e227dab2ad22cb registry.k8s.io/provider-aws/cloud-controller-manager:v1.26.0],SizeBytes:20154862,},ContainerImage{Names:[registry.k8s.io/kube-scheduler@sha256:cbf4007abb076188f16b975bc2b74452603cc7b34c60a29d92f14c38ad02dc11 registry.k8s.io/kube-scheduler:v1.28.0-alpha.0],SizeBytes:18078674,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.27.0-alpha.1],SizeBytes:5074298,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Mar 25 05:42:31.493: INFO: Logging kubelet events for node i-0168f82458996cc53 Mar 25 05:42:31.558: INFO: Logging pods the kubelet thinks is on node i-0168f82458996cc53 Mar 25 05:42:31.626: INFO: kube-controller-manager-i-0168f82458996cc53 started at 2023-03-25 05:17:36 +0000 UTC (0+1 container statuses recorded) Mar 25 05:42:31.626: INFO: Container kube-controller-manager ready: true, restart count 4 Mar 25 05:42:31.626: INFO: etcd-manager-events-i-0168f82458996cc53 started at 2023-03-25 05:17:36 +0000 UTC (11+1 container statuses recorded) Mar 25 05:42:31.626: INFO: Init container init-etcd-3-2-24 ready: true, restart count 0 Mar 25 05:42:31.626: INFO: Init container init-etcd-3-3-10 ready: true, restart count 0 Mar 25 05:42:31.626: INFO: Init container init-etcd-3-3-17 ready: true, restart count 0 Mar 25 05:42:31.626: INFO: Init container init-etcd-3-4-13 ready: true, restart count 0 Mar 25 05:42:31.626: INFO: Init container init-etcd-3-4-3 ready: true, restart count 0 Mar 25 05:42:31.626: INFO: Init container init-etcd-3-5-0 ready: true, restart count 0 Mar 25 05:42:31.626: INFO: Init container init-etcd-3-5-1 ready: true, restart count 0 Mar 25 05:42:31.626: INFO: Init container init-etcd-3-5-3 ready: true, restart count 0 Mar 25 05:42:31.626: INFO: Init container init-etcd-3-5-4 ready: true, restart count 0 Mar 25 05:42:31.626: INFO: Init container init-etcd-3-5-6 ready: true, restart count 0 Mar 25 05:42:31.626: INFO: Init container init-etcd-3-5-7 ready: true, restart count 0 Mar 25 05:42:31.626: INFO: Container etcd-manager ready: true, restart count 0 Mar 25 05:42:31.626: INFO: dns-controller-69987775c6-hbn69 started at 2023-03-25 05:19:47 +0000 UTC (0+1 container statuses recorded) Mar 25 05:42:31.626: INFO: Container dns-controller ready: true, restart count 0 Mar 25 05:42:31.626: INFO: aws-cloud-controller-manager-26vbv started at 2023-03-25 05:20:06 +0000 UTC (0+1 container statuses recorded) Mar 25 05:42:31.626: INFO: Container aws-cloud-controller-manager ready: true, restart count 0 Mar 25 05:42:31.626: INFO: calico-node-rdjq2 started at 2023-03-25 05:20:06 +0000 UTC (3+1 container statuses recorded) Mar 25 05:42:31.626: INFO: Init container upgrade-ipam ready: true, restart count 0 Mar 25 05:42:31.626: INFO: Init container install-cni ready: true, restart count 0 Mar 25 05:42:31.626: INFO: Init container mount-bpffs ready: true, restart count 0 Mar 25 05:42:31.626: INFO: Container calico-node ready: true, restart count 0 Mar 25 05:42:31.626: INFO: ebs-csi-node-zjtj6 started at 2023-03-25 05:20:06 +0000 UTC (0+3 container statuses recorded) Mar 25 05:42:31.626: INFO: Container ebs-plugin ready: true, restart count 0 Mar 25 05:42:31.626: INFO: Container liveness-probe ready: true, restart count 0 Mar 25 05:42:31.626: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 25 05:42:31.626: INFO: kube-apiserver-i-0168f82458996cc53 started at 2023-03-25 05:17:36 +0000 UTC (0+2 container statuses recorded) Mar 25 05:42:31.626: INFO: Container healthcheck ready: true, restart count 0 Mar 25 05:42:31.626: INFO: Container kube-apiserver ready: true, restart count 2 Mar 25 05:42:31.626: INFO: kube-proxy-i-0168f82458996cc53 started at 2023-03-25 05:17:36 +0000 UTC (0+1 container statuses recorded) Mar 25 05:42:31.626: INFO: Container kube-proxy ready: true, restart count 0 Mar 25 05:42:31.626: INFO: kube-scheduler-i-0168f82458996cc53 started at 2023-03-25 05:17:36 +0000 UTC (0+1 container statuses recorded) Mar 25 05:42:31.626: INFO: Container kube-scheduler ready: true, restart count 0 Mar 25 05:42:31.626: INFO: etcd-manager-main-i-0168f82458996cc53 started at 2023-03-25 05:17:36 +0000 UTC (11+1 container statuses recorded) Mar 25 05:42:31.626: INFO: Init container init-etcd-3-2-24 ready: true, restart count 0 Mar 25 05:42:31.626: INFO: Init container init-etcd-3-3-10 ready: true, restart count 0 Mar 25 05:42:31.626: INFO: Init container init-etcd-3-3-17 ready: true, restart count 0 Mar 25 05:42:31.626: INFO: Init container init-etcd-3-4-13 ready: true, restart count 0 Mar 25 05:42:31.626: INFO: Init container init-etcd-3-4-3 ready: true, restart count 0 Mar 25 05:42:31.626: INFO: Init container init-etcd-3-5-0 ready: true, restart count 0 Mar 25 05:42:31.626: INFO: Init container init-etcd-3-5-1 ready: true, restart count 0 Mar 25 05:42:31.626: INFO: Init container init-etcd-3-5-3 ready: true, restart count 0 Mar 25 05:42:31.626: INFO: Init container init-etcd-3-5-4 ready: true, restart count 0 Mar 25 05:42:31.626: INFO: Init container init-etcd-3-5-6 ready: true, restart count 0 Mar 25 05:42:31.626: INFO: Init container init-etcd-3-5-7 ready: true, restart count 0 Mar 25 05:42:31.626: INFO: Container etcd-manager ready: true, restart count 0 Mar 25 05:42:31.626: INFO: kops-controller-cgw95 started at 2023-03-25 05:20:06 +0000 UTC (0+1 container statuses recorded) Mar 25 05:42:31.626: INFO: Container kops-controller ready: true, restart count 0 Mar 25 05:42:31.626: INFO: calico-kube-controllers-55f488656-rr8rz started at 2023-03-25 05:20:29 +0000 UTC (0+1 container statuses recorded) Mar 25 05:42:31.626: INFO: Container calico-kube-controllers ready: true, restart count 0 Mar 25 05:42:31.626: INFO: ebs-csi-controller-79f984545f-9q5ks started at 2023-03-25 05:21:26 +0000 UTC (0+5 container statuses recorded) Mar 25 05:42:31.626: INFO: Container csi-attacher ready: true, restart count 0 Mar 25 05:42:31.626: INFO: Container csi-provisioner ready: true, restart count 0 Mar 25 05:42:31.626: INFO: Container csi-resizer ready: true, restart count 0 Mar 25 05:42:31.626: INFO: Container ebs-plugin ready: true, restart count 0 Mar 25 05:42:31.626: INFO: Container liveness-probe ready: true, restart count 0 Mar 25 05:42:31.866: INFO: Latency metrics for node i-0168f82458996cc53 Mar 25 05:42:31.866: INFO: Logging node info for node i-03d301e6f030748f1 Mar 25 05:42:31.915: INFO: Node Info: &Node{ObjectMeta:{i-03d301e6f030748f1 897f52bd-65e2-4107-843a-37e36aa74032 22045 0 2023-03-25 05:31:35 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-1 failure-domain.beta.kubernetes.io/zone:us-west-1b kubernetes.io/arch:amd64 kubernetes.io/hostname:i-03d301e6f030748f1 kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:us-west-1b topology.hostpath.csi/node:i-03d301e6f030748f1 topology.kubernetes.io/region:us-west-1 topology.kubernetes.io/zone:us-west-1b] map[csi.volume.kubernetes.io/nodeid:{"csi-mock-csi-mock-volumes-fsgroup-mount-4340":"csi-mock-csi-mock-volumes-fsgroup-mount-4340","ebs.csi.aws.com":"i-03d301e6f030748f1"} node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:172.20.50.199/19 projectcalico.org/IPv4IPIPTunnelAddr:100.110.167.64 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-03-25 05:31:35 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-25 05:31:35 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-03-25 05:31:35 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2023-03-25 05:31:35 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2023-03-25 05:31:48 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.8.0/24\"":{}}}} } {calico-node Update v1 2023-03-25 05:31:56 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4IPIPTunnelAddr":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2023-03-25 05:42:18 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-03-25 05:42:26 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.8.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-1b/i-03d301e6f030748f1,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.8.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-03-25 05:31:56 +0000 UTC,LastTransitionTime:2023-03-25 05:31:56 +0000 UTC,Reason:CalicoIsUp,Message:Calico is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-25 05:42:18 +0000 UTC,LastTransitionTime:2023-03-25 05:31:35 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-25 05:42:18 +0000 UTC,LastTransitionTime:2023-03-25 05:31:35 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-25 05:42:18 +0000 UTC,LastTransitionTime:2023-03-25 05:31:35 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-25 05:42:18 +0000 UTC,LastTransitionTime:2023-03-25 05:31:48 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.50.199,},NodeAddress{Type:ExternalIP,Address:52.53.245.63,},NodeAddress{Type:InternalDNS,Address:i-03d301e6f030748f1.us-west-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-03d301e6f030748f1.us-west-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-52-53-245-63.us-west-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec27152f7af9b9bfd1355ee19c256086,SystemUUID:ec27152f-7af9-b9bf-d135-5ee19c256086,BootID:1c303852-f2ce-45f9-ab14-6e8de97a0a3e,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.28.0-alpha.0,KubeProxyVersion:v1.28.0-alpha.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[docker.io/calico/cni@sha256:e282ea2914c806b5de2976330a17cfb5e6dcef47147bceb1432ca5c75fd46f50 docker.io/calico/cni:v3.24.5],SizeBytes:87458443,},ContainerImage{Names:[docker.io/calico/node@sha256:5972ad2bcbdc668564d3e26960c9c513b2d7b05581c704747cf7c62ef3a405a6 docker.io/calico/node:v3.24.5],SizeBytes:81584866,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nautilus@sha256:80ba6c8c44f9623f06e868a1aa66026c8ec438ad814f9ec95e9333b415fe3550 registry.k8s.io/e2e-test-images/nautilus:1.7],SizeBytes:49641698,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41901587,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:e468dddcd275163a042ab297b2d8c2aca50d5e148d2d22f3b6ba119e2f31fa79 registry.k8s.io/sig-storage/csi-provisioner:v3.4.0],SizeBytes:27427836,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/kube-proxy@sha256:8a03b3147cd7be8e8235ae1cd70cf69aca9ce03ba1cfaffea320540bb5afa496 registry.k8s.io/kube-proxy:v1.28.0-alpha.0],SizeBytes:23896850,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:23881995,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18758628,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18233005,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonroot@sha256:ee9f50b3c64b174d296d91ca9f69a914ac30e59095dfb462b2b518ad28a63655 registry.k8s.io/e2e-test-images/nonroot:1.4],SizeBytes:17747885,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8688564,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:7000082,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6978614,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac registry.k8s.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[kubernetes.io/csi/ebs.csi.aws.com^vol-0730c32100e68928b kubernetes.io/csi/ebs.csi.aws.com^vol-0f3ac79afabe97ce1],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-0f3ac79afabe97ce1,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-0730c32100e68928b,DevicePath:,},},Config:nil,},} Mar 25 05:42:31.916: INFO: Logging kubelet events for node i-03d301e6f030748f1 Mar 25 05:42:31.977: INFO: Logging pods the kubelet thinks is on node i-03d301e6f030748f1 Mar 25 05:42:32.036: INFO: kube-proxy-i-03d301e6f030748f1 started at 2023-03-25 05:31:35 +0000 UTC (0+1 container statuses recorded) Mar 25 05:42:32.036: INFO: Container kube-proxy ready: true, restart count 0 Mar 25 05:42:32.036: INFO: ebs-csi-node-lljwl started at 2023-03-25 05:31:35 +0000 UTC (0+3 container statuses recorded) Mar 25 05:42:32.036: INFO: Container ebs-plugin ready: true, restart count 0 Mar 25 05:42:32.036: INFO: Container liveness-probe ready: true, restart count 0 Mar 25 05:42:32.036: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 25 05:42:32.036: INFO: pfpod started at 2023-03-25 05:41:19 +0000 UTC (0+2 container statuses recorded) Mar 25 05:42:32.036: INFO: Container portforwardtester ready: false, restart count 0 Mar 25 05:42:32.036: INFO: Container readiness ready: false, restart count 0 Mar 25 05:42:32.036: INFO: httpd started at 2023-03-25 05:42:12 +0000 UTC (0+1 container statuses recorded) Mar 25 05:42:32.036: INFO: Container httpd ready: true, restart count 0 Mar 25 05:42:32.036: INFO: replace-27995382-t6hbh started at 2023-03-25 05:42:00 +0000 UTC (0+1 container statuses recorded) Mar 25 05:42:32.036: INFO: Container c ready: false, restart count 0 Mar 25 05:42:32.036: INFO: pvc-volume-tester-8wx8b started at 2023-03-25 05:42:29 +0000 UTC (0+1 container statuses recorded) Mar 25 05:42:32.036: INFO: Container volume-tester ready: false, restart count 0 Mar 25 05:42:32.036: INFO: netserver-0 started at 2023-03-25 05:42:14 +0000 UTC (0+1 container statuses recorded) Mar 25 05:42:32.036: INFO: Container webserver ready: false, restart count 0 Mar 25 05:42:32.036: INFO: execpod6zh4h started at 2023-03-25 05:42:15 +0000 UTC (0+1 container statuses recorded) Mar 25 05:42:32.036: INFO: Container agnhost-container ready: true, restart count 0 Mar 25 05:42:32.036: INFO: pod-ephm-test-configmap-f48v started at 2023-03-25 05:40:32 +0000 UTC (0+1 container statuses recorded) Mar 25 05:42:32.036: INFO: Container test-container-subpath-configmap-f48v ready: false, restart count 0 Mar 25 05:42:32.036: INFO: calico-node-h96vq started at 2023-03-25 05:31:35 +0000 UTC (3+1 container statuses recorded) Mar 25 05:42:32.036: INFO: Init container upgrade-ipam ready: true, restart count 0 Mar 25 05:42:32.036: INFO: Init container install-cni ready: true, restart count 0 Mar 25 05:42:32.036: INFO: Init container mount-bpffs ready: true, restart count 0 Mar 25 05:42:32.036: INFO: Container calico-node ready: true, restart count 0 Mar 25 05:42:32.036: INFO: pod-8aa03cfb-1713-40aa-9799-8bf26b15abdc started at 2023-03-25 05:41:32 +0000 UTC (0+1 container statuses recorded) Mar 25 05:42:32.036: INFO: Container write-pod ready: true, restart count 0 Mar 25 05:42:32.036: INFO: csi-mockplugin-0 started at 2023-03-25 05:42:21 +0000 UTC (0+4 container statuses recorded) Mar 25 05:42:32.036: INFO: Container busybox ready: true, restart count 0 Mar 25 05:42:32.036: INFO: Container csi-provisioner ready: true, restart count 0 Mar 25 05:42:32.036: INFO: Container driver-registrar ready: true, restart count 0 Mar 25 05:42:32.036: INFO: Container mock ready: true, restart count 0 Mar 25 05:42:32.036: INFO: inline-volume-tester-tw9vd started at 2023-03-25 05:41:44 +0000 UTC (0+1 container statuses recorded) Mar 25 05:42:32.036: INFO: Container csi-volume-tester ready: true, restart count 0 Mar 25 05:42:32.036: INFO: nodeport-test-hv4m5 started at 2023-03-25 05:42:12 +0000 UTC (0+1 container statuses recorded) Mar 25 05:42:32.036: INFO: Container nodeport-test ready: true, restart count 0 Mar 25 05:42:32.036: INFO: liveness-1eeb7f13-f86d-4678-a5e1-d201118a9cbf started at 2023-03-25 05:42:13 +0000 UTC (0+1 container statuses recorded) Mar 25 05:42:32.036: INFO: Container agnhost-container ready: true, restart count 0 Mar 25 05:42:32.036: INFO: hostpath-symlink-prep-provisioning-1460 started at 2023-03-25 05:42:31 +0000 UTC (0+1 container statuses recorded) Mar 25 05:42:32.036: INFO: Container init-volume-provisioning-1460 ready: false, restart count 0 Mar 25 05:42:32.331: INFO: Latency metrics for node i-03d301e6f030748f1 Mar 25 05:42:32.331: INFO: Logging node info for node i-049fdb2c1cde894df Mar 25 05:42:32.382: INFO: Node Info: &Node{ObjectMeta:{i-049fdb2c1cde894df b43a51e6-dda7-4047-b368-72d515695b4e 21455 0 2023-03-25 05:23:34 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-1 failure-domain.beta.kubernetes.io/zone:us-west-1b io.kubernetes.storage.mock/node:some-mock-node kubernetes.io/arch:amd64 kubernetes.io/hostname:i-049fdb2c1cde894df kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:us-west-1b topology.hostpath.csi/node:i-049fdb2c1cde894df topology.kubernetes.io/region:us-west-1 topology.kubernetes.io/zone:us-west-1b] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-58":"i-049fdb2c1cde894df","csi-hostpath-ephemeral-8659":"i-049fdb2c1cde894df","ebs.csi.aws.com":"i-049fdb2c1cde894df"} node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:172.20.43.61/19 projectcalico.org/IPv4IPIPTunnelAddr:100.97.18.64 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-03-25 05:23:34 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-25 05:23:34 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-03-25 05:23:34 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2023-03-25 05:23:34 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2023-03-25 05:23:45 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.6.0/24\"":{}}}} } {calico-node Update v1 2023-03-25 05:23:52 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4IPIPTunnelAddr":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2023-03-25 05:41:56 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-03-25 05:42:10 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:io.kubernetes.storage.mock/node":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.6.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-1b/i-049fdb2c1cde894df,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.6.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-03-25 05:23:52 +0000 UTC,LastTransitionTime:2023-03-25 05:23:52 +0000 UTC,Reason:CalicoIsUp,Message:Calico is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-25 05:41:56 +0000 UTC,LastTransitionTime:2023-03-25 05:23:34 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-25 05:41:56 +0000 UTC,LastTransitionTime:2023-03-25 05:23:34 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-25 05:41:56 +0000 UTC,LastTransitionTime:2023-03-25 05:23:34 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-25 05:41:56 +0000 UTC,LastTransitionTime:2023-03-25 05:23:45 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.43.61,},NodeAddress{Type:ExternalIP,Address:18.144.75.67,},NodeAddress{Type:InternalDNS,Address:i-049fdb2c1cde894df.us-west-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-049fdb2c1cde894df.us-west-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-18-144-75-67.us-west-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec252e592d19e1e6c583e87505431016,SystemUUID:ec252e59-2d19-e1e6-c583-e87505431016,BootID:45436b5d-1eb0-4b6f-a1d1-a1492b31b5d7,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.28.0-alpha.0,KubeProxyVersion:v1.28.0-alpha.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[docker.io/calico/cni@sha256:e282ea2914c806b5de2976330a17cfb5e6dcef47147bceb1432ca5c75fd46f50 docker.io/calico/cni:v3.24.5],SizeBytes:87458443,},ContainerImage{Names:[docker.io/calico/node@sha256:5972ad2bcbdc668564d3e26960c9c513b2d7b05581c704747cf7c62ef3a405a6 docker.io/calico/node:v3.24.5],SizeBytes:81584866,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:e468dddcd275163a042ab297b2d8c2aca50d5e148d2d22f3b6ba119e2f31fa79 registry.k8s.io/sig-storage/csi-provisioner:v3.4.0],SizeBytes:27427836,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/kube-proxy@sha256:8a03b3147cd7be8e8235ae1cd70cf69aca9ce03ba1cfaffea320540bb5afa496 registry.k8s.io/kube-proxy:v1.28.0-alpha.0],SizeBytes:23896850,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:23881995,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:68d396900aeaa072c1f27289485fdac29834045a6f3ffe369bf389d830ef572d registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.6],SizeBytes:20293261,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18758628,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18233005,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:15273057,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8688564,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:7000082,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6978614,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[kubernetes.io/csi/csi-hostpath-ephemeral-58^bae5629b-cacf-11ed-b406-e6de5ba5691d],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-58^bae5629b-cacf-11ed-b406-e6de5ba5691d,DevicePath:,},},Config:nil,},} Mar 25 05:42:32.382: INFO: Logging kubelet events for node i-049fdb2c1cde894df Mar 25 05:42:32.487: INFO: Logging pods the kubelet thinks is on node i-049fdb2c1cde894df Mar 25 05:42:32.544: INFO: csi-hostpathplugin-0 started at 2023-03-25 05:41:20 +0000 UTC (0+7 container statuses recorded) Mar 25 05:42:32.544: INFO: Container csi-attacher ready: true, restart count 0 Mar 25 05:42:32.544: INFO: Container csi-provisioner ready: true, restart count 0 Mar 25 05:42:32.544: INFO: Container csi-resizer ready: true, restart count 0 Mar 25 05:42:32.544: INFO: Container csi-snapshotter ready: true, restart count 0 Mar 25 05:42:32.544: INFO: Container hostpath ready: true, restart count 0 Mar 25 05:42:32.544: INFO: Container liveness-probe ready: true, restart count 0 Mar 25 05:42:32.544: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 25 05:42:32.544: INFO: coredns-autoscaler-7cb5c5b969-h485x started at 2023-03-25 05:24:36 +0000 UTC (0+1 container statuses recorded) Mar 25 05:42:32.544: INFO: Container autoscaler ready: true, restart count 0 Mar 25 05:42:32.544: INFO: csi-hostpathplugin-0 started at 2023-03-25 05:41:43 +0000 UTC (0+7 container statuses recorded) Mar 25 05:42:32.544: INFO: Container csi-attacher ready: true, restart count 0 Mar 25 05:42:32.544: INFO: Container csi-provisioner ready: true, restart count 0 Mar 25 05:42:32.544: INFO: Container csi-resizer ready: true, restart count 0 Mar 25 05:42:32.544: INFO: Container csi-snapshotter ready: true, restart count 0 Mar 25 05:42:32.544: INFO: Container hostpath ready: true, restart count 0 Mar 25 05:42:32.544: INFO: Container liveness-probe ready: true, restart count 0 Mar 25 05:42:32.544: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 25 05:42:32.544: INFO: ebs-csi-node-m9zxz started at 2023-03-25 05:23:35 +0000 UTC (0+3 container statuses recorded) Mar 25 05:42:32.545: INFO: Container ebs-plugin ready: true, restart count 0 Mar 25 05:42:32.545: INFO: Container liveness-probe ready: true, restart count 0 Mar 25 05:42:32.545: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 25 05:42:32.545: INFO: calico-node-wt5p7 started at 2023-03-25 05:23:35 +0000 UTC (3+1 container statuses recorded) Mar 25 05:42:32.545: INFO: Init container upgrade-ipam ready: true, restart count 0 Mar 25 05:42:32.545: INFO: Init container install-cni ready: true, restart count 0 Mar 25 05:42:32.545: INFO: Init container mount-bpffs ready: true, restart count 0 Mar 25 05:42:32.545: INFO: Container calico-node ready: true, restart count 0 Mar 25 05:42:32.545: INFO: coredns-559769c974-pgqhl started at 2023-03-25 05:24:36 +0000 UTC (0+1 container statuses recorded) Mar 25 05:42:32.545: INFO: Container coredns ready: true, restart count 0 Mar 25 05:42:32.545: INFO: netserver-1 started at 2023-03-25 05:42:14 +0000 UTC (0+1 container statuses recorded) Mar 25 05:42:32.545: INFO: Container webserver ready: true, restart count 0 Mar 25 05:42:32.545: INFO: kube-proxy-i-049fdb2c1cde894df started at 2023-03-25 05:23:35 +0000 UTC (0+1 container statuses recorded) Mar 25 05:42:32.545: INFO: Container kube-proxy ready: true, restart count 0 Mar 25 05:42:32.829: INFO: Latency metrics for node i-049fdb2c1cde894df Mar 25 05:42:32.829: INFO: Logging node info for node i-062679bb8c0ba4bce Mar 25 05:42:32.879: INFO: Node Info: &Node{ObjectMeta:{i-062679bb8c0ba4bce ee2da569-bb88-4297-9e6f-d2b15ba7ce86 20969 0 2023-03-25 05:27:31 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-1 failure-domain.beta.kubernetes.io/zone:us-west-1b kubernetes.io/arch:amd64 kubernetes.io/hostname:i-062679bb8c0ba4bce kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:us-west-1b topology.hostpath.csi/node:i-062679bb8c0ba4bce topology.kubernetes.io/region:us-west-1 topology.kubernetes.io/zone:us-west-1b] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-062679bb8c0ba4bce"} node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:172.20.45.43/19 projectcalico.org/IPv4IPIPTunnelAddr:100.106.79.0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-03-25 05:27:31 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-25 05:27:31 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-03-25 05:27:31 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2023-03-25 05:27:31 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2023-03-25 05:27:44 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.7.0/24\"":{}}}} } {calico-node Update v1 2023-03-25 05:27:50 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4IPIPTunnelAddr":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2023-03-25 05:41:57 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status} {kube-controller-manager Update v1 2023-03-25 05:41:59 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.7.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-1b/i-062679bb8c0ba4bce,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.7.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051689472 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946831872 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-03-25 05:27:50 +0000 UTC,LastTransitionTime:2023-03-25 05:27:50 +0000 UTC,Reason:CalicoIsUp,Message:Calico is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-25 05:41:57 +0000 UTC,LastTransitionTime:2023-03-25 05:27:31 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-25 05:41:57 +0000 UTC,LastTransitionTime:2023-03-25 05:27:31 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-25 05:41:57 +0000 UTC,LastTransitionTime:2023-03-25 05:27:31 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-25 05:41:57 +0000 UTC,LastTransitionTime:2023-03-25 05:27:44 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.45.43,},NodeAddress{Type:ExternalIP,Address:3.101.133.108,},NodeAddress{Type:InternalDNS,Address:i-062679bb8c0ba4bce.us-west-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-062679bb8c0ba4bce.us-west-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-3-101-133-108.us-west-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2528f599e9b1e199793fb5cd06d205,SystemUUID:ec2528f5-99e9-b1e1-9979-3fb5cd06d205,BootID:ff56b70b-c8c5-46db-8cc5-4353c8f535c6,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.28.0-alpha.0,KubeProxyVersion:v1.28.0-alpha.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:112030336,},ContainerImage{Names:[docker.io/calico/cni@sha256:e282ea2914c806b5de2976330a17cfb5e6dcef47147bceb1432ca5c75fd46f50 docker.io/calico/cni:v3.24.5],SizeBytes:87458443,},ContainerImage{Names:[docker.io/calico/node@sha256:5972ad2bcbdc668564d3e26960c9c513b2d7b05581c704747cf7c62ef3a405a6 docker.io/calico/node:v3.24.5],SizeBytes:81584866,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nautilus@sha256:80ba6c8c44f9623f06e868a1aa66026c8ec438ad814f9ec95e9333b415fe3550 registry.k8s.io/e2e-test-images/nautilus:1.7],SizeBytes:49641698,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41901587,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:e468dddcd275163a042ab297b2d8c2aca50d5e148d2d22f3b6ba119e2f31fa79 registry.k8s.io/sig-storage/csi-provisioner:v3.4.0],SizeBytes:27427836,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/kube-proxy@sha256:8a03b3147cd7be8e8235ae1cd70cf69aca9ce03ba1cfaffea320540bb5afa496 registry.k8s.io/kube-proxy:v1.28.0-alpha.0],SizeBytes:23896850,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:23881995,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18758628,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18233005,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:15273057,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8688564,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:7000082,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6978614,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[kubernetes.io/csi/ebs.csi.aws.com^vol-09606fdadb4471680],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-09606fdadb4471680,DevicePath:,},},Config:nil,},} Mar 25 05:42:32.880: INFO: Logging kubelet events for node i-062679bb8c0ba4bce Mar 25 05:42:32.938: INFO: Logging pods the kubelet thinks is on node i-062679bb8c0ba4bce Mar 25 05:42:33.009: INFO: kube-proxy-i-062679bb8c0ba4bce started at 2023-03-25 05:27:32 +0000 UTC (0+1 container statuses recorded) Mar 25 05:42:33.009: INFO: Container kube-proxy ready: true, restart count 0 Mar 25 05:42:33.009: INFO: ebs-csi-node-7twrq started at 2023-03-25 05:27:32 +0000 UTC (0+3 container statuses recorded) Mar 25 05:42:33.009: INFO: Container ebs-plugin ready: true, restart count 0 Mar 25 05:42:33.009: INFO: Container liveness-probe ready: true, restart count 0 Mar 25 05:42:33.009: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 25 05:42:33.009: INFO: e2e-configmap-dns-server-ed687109-b32b-4dd2-bec0-a6cd92ca47e3 started at 2023-03-25 05:42:29 +0000 UTC (0+1 container statuses recorded) Mar 25 05:42:33.010: INFO: Container agnhost-container ready: true, restart count 0 Mar 25 05:42:33.010: INFO: pod-disruption-failure-ignore-0-5pgsm started at 2023-03-25 05:42:16 +0000 UTC (0+1 container statuses recorded) Mar 25 05:42:33.010: INFO: Container c ready: true, restart count 0 Mar 25 05:42:33.010: INFO: inline-volume-tester2-rhj5d started at 2023-03-25 05:41:57 +0000 UTC (0+1 container statuses recorded) Mar 25 05:42:33.010: INFO: Container csi-volume-tester ready: true, restart count 0 Mar 25 05:42:33.010: INFO: calico-node-zwm5j started at 2023-03-25 05:27:32 +0000 UTC (3+1 container statuses recorded) Mar 25 05:42:33.010: INFO: Init container upgrade-ipam ready: true, restart count 0 Mar 25 05:42:33.010: INFO: Init container install-cni ready: true, restart count 0 Mar 25 05:42:33.010: INFO: Init container mount-bpffs ready: true, restart count 0 Mar 25 05:42:33.010: INFO: Container calico-node ready: true, restart count 0 Mar 25 05:42:33.010: INFO: pod-disruption-failure-ignore-3-xg4dl started at 2023-03-25 05:42:23 +0000 UTC (0+1 container statuses recorded) Mar 25 05:42:33.010: INFO: Container c ready: false, restart count 0 Mar 25 05:42:33.010: INFO: nodeport-test-prqwz started at 2023-03-25 05:42:12 +0000 UTC (0+1 container statuses recorded) Mar 25 05:42:33.010: INFO: Container nodeport-test ready: true, restart count 0 Mar 25 05:42:33.010: INFO: coredns-559769c974-p9vg6 started at 2023-03-25 05:36:16 +0000 UTC (0+1 container statuses recorded) Mar 25 05:42:33.010: INFO: Container coredns ready: true, restart count 0 Mar 25 05:42:33.010: INFO: pod-disruption-failure-ignore-2-l968j started at 2023-03-25 05:42:20 +0000 UTC (0+1 container statuses recorded) Mar 25 05:42:33.010: INFO: Container c ready: false, restart count 0 Mar 25 05:42:33.010: INFO: pfpod started at 2023-03-25 05:42:31 +0000 UTC (0+2 container statuses recorded) Mar 25 05:42:33.010: INFO: Container portforwardtester ready: true, restart count 0 Mar 25 05:42:33.010: INFO: Container readiness ready: false, restart count 0 Mar 25 05:42:33.010: INFO: netserver-2 started at 2023-03-25 05:42:14 +0000 UTC (0+1 container statuses recorded) Mar 25 05:42:33.010: INFO: Container webserver ready: false, restart count 0 Mar 25 05:42:33.010: INFO: e2e-dns-utils started at 2023-03-25 05:42:31 +0000 UTC (0+1 container statuses recorded) Mar 25 05:42:33.010: INFO: Container agnhost-container ready: false, restart count 0 Mar 25 05:42:33.010: INFO: pod-disruption-failure-ignore-0-h55ln started at 2023-03-25 05:42:29 +0000 UTC (0+1 container statuses recorded) Mar 25 05:42:33.010: INFO: Container c ready: false, restart count 0 Mar 25 05:42:33.010: INFO: pod-disruption-failure-ignore-1-cmkcc started at 2023-03-25 05:42:16 +0000 UTC (0+1 container statuses recorded) Mar 25 05:42:33.010: INFO: Container c ready: false, restart count 0 Mar 25 05:42:33.325: INFO: Latency metrics for node i-062679bb8c0ba4bce Mar 25 05:42:33.325: INFO: Logging node info for node i-072608201d2c0f1ad Mar 25 05:42:33.377: INFO: Node Info: &Node{ObjectMeta:{i-072608201d2c0f1ad 264dfba1-3cf0-4c99-8e8c-098f0f792077 22082 0 2023-03-25 05:35:29 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-1 failure-domain.beta.kubernetes.io/zone:us-west-1b kubernetes.io/arch:amd64 kubernetes.io/hostname:i-072608201d2c0f1ad kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:us-west-1b topology.hostpath.csi/node:i-072608201d2c0f1ad topology.kubernetes.io/region:us-west-1 topology.kubernetes.io/zone:us-west-1b] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-8736":"i-072608201d2c0f1ad","csi-hostpath-provisioning-8197":"i-072608201d2c0f1ad","csi-mock-csi-mock-volumes-capacity-6274":"i-072608201d2c0f1ad","ebs.csi.aws.com":"i-072608201d2c0f1ad"} node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:172.20.38.125/19 projectcalico.org/IPv4IPIPTunnelAddr:100.107.220.192 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-03-25 05:35:29 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-25 05:35:29 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-03-25 05:35:29 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2023-03-25 05:35:29 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2023-03-25 05:35:41 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.9.0/24\"":{}}}} } {calico-node Update v1 2023-03-25 05:35:48 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4IPIPTunnelAddr":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2023-03-25 05:42:28 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-03-25 05:42:28 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.9.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-1b/i-072608201d2c0f1ad,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.9.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051689472 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946831872 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-03-25 05:35:47 +0000 UTC,LastTransitionTime:2023-03-25 05:35:47 +0000 UTC,Reason:CalicoIsUp,Message:Calico is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-25 05:42:28 +0000 UTC,LastTransitionTime:2023-03-25 05:35:29 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-25 05:42:28 +0000 UTC,LastTransitionTime:2023-03-25 05:35:29 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-25 05:42:28 +0000 UTC,LastTransitionTime:2023-03-25 05:35:29 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-25 05:42:28 +0000 UTC,LastTransitionTime:2023-03-25 05:35:41 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.38.125,},NodeAddress{Type:ExternalIP,Address:3.101.88.10,},NodeAddress{Type:InternalDNS,Address:i-072608201d2c0f1ad.us-west-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-072608201d2c0f1ad.us-west-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-3-101-88-10.us-west-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2d3818c4f80a253dcc1a60e148dbc8,SystemUUID:ec2d3818-c4f8-0a25-3dcc-1a60e148dbc8,BootID:77d046cf-75ea-4a8c-970a-3190d64e019b,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.28.0-alpha.0,KubeProxyVersion:v1.28.0-alpha.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:112030336,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:51eae8381dcb1078289fa7b4f3df2630cdc18d09fb56f8e56b41c40e191d6c83 registry.k8s.io/etcd:3.5.7-0],SizeBytes:101639218,},ContainerImage{Names:[docker.io/calico/cni@sha256:e282ea2914c806b5de2976330a17cfb5e6dcef47147bceb1432ca5c75fd46f50 docker.io/calico/cni:v3.24.5],SizeBytes:87458443,},ContainerImage{Names:[docker.io/calico/node@sha256:5972ad2bcbdc668564d3e26960c9c513b2d7b05581c704747cf7c62ef3a405a6 docker.io/calico/node:v3.24.5],SizeBytes:81584866,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41901587,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:e468dddcd275163a042ab297b2d8c2aca50d5e148d2d22f3b6ba119e2f31fa79 registry.k8s.io/sig-storage/csi-provisioner:v3.4.0],SizeBytes:27427836,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/sample-apiserver@sha256:8d70890151aa5d096f331cb9da1b9cd5be0412b7363fe67b5c3befdcaa2a28d0 registry.k8s.io/e2e-test-images/sample-apiserver:1.17.7],SizeBytes:25667066,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/kube-proxy@sha256:8a03b3147cd7be8e8235ae1cd70cf69aca9ce03ba1cfaffea320540bb5afa496 registry.k8s.io/kube-proxy:v1.28.0-alpha.0],SizeBytes:23896850,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:23881995,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18758628,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18233005,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8688564,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:7000082,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6978614,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[kubernetes.io/csi/csi-hostpath-ephemeral-8736^ce003867-cacf-11ed-8af4-f285849b7d5a kubernetes.io/csi/csi-hostpath-provisioning-8197^c3f5007d-cacf-11ed-954e-42ace337786e],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-provisioning-8197^c3f5007d-cacf-11ed-954e-42ace337786e,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-8736^ce003867-cacf-11ed-8af4-f285849b7d5a,DevicePath:,},},Config:nil,},} Mar 25 05:42:33.378: INFO: Logging kubelet events for node i-072608201d2c0f1ad Mar 25 05:42:33.436: INFO: Logging pods the kubelet thinks is on node i-072608201d2c0f1ad Mar 25 05:42:33.512: INFO: rs-jcfcz started at 2023-03-25 05:40:39 +0000 UTC (0+1 container statuses recorded) Mar 25 05:42:33.512: INFO: Container donothing ready: false, restart count 0 Mar 25 05:42:33.512: INFO: webserver-pod started at 2023-03-25 05:39:46 +0000 UTC (0+1 container statuses recorded) Mar 25 05:42:33.512: INFO: Container agnhost ready: false, restart count 0 Mar 25 05:42:33.512: INFO: csi-mockplugin-0 started at 2023-03-25 05:42:02 +0000 UTC (0+3 container statuses recorded) Mar 25 05:42:33.512: INFO: Container csi-provisioner ready: true, restart count 0 Mar 25 05:42:33.512: INFO: Container driver-registrar ready: true, restart count 0 Mar 25 05:42:33.512: INFO: Container mock ready: true, restart count 0 Mar 25 05:42:33.512: INFO: liveness-78b08cf2-0e88-4ba6-aa64-2fa547995658 started at 2023-03-25 05:40:42 +0000 UTC (0+1 container statuses recorded) Mar 25 05:42:33.512: INFO: Container agnhost-container ready: true, restart count 0 Mar 25 05:42:33.512: INFO: ebs-csi-node-r7mcm started at 2023-03-25 05:35:30 +0000 UTC (0+3 container statuses recorded) Mar 25 05:42:33.512: INFO: Container ebs-plugin ready: true, restart count 0 Mar 25 05:42:33.512: INFO: Container liveness-probe ready: true, restart count 0 Mar 25 05:42:33.512: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 25 05:42:33.512: INFO: pod-subpath-test-dynamicpv-bkhd started at 2023-03-25 05:42:02 +0000 UTC (1+1 container statuses recorded) Mar 25 05:42:33.512: INFO: Init container init-volume-dynamicpv-bkhd ready: true, restart count 0 Mar 25 05:42:33.512: INFO: Container test-container-subpath-dynamicpv-bkhd ready: false, restart count 0 Mar 25 05:42:33.512: INFO: pod-with-poststart-https-hook started at 2023-03-25 05:40:57 +0000 UTC (0+1 container statuses recorded) Mar 25 05:42:33.512: INFO: Container pod-with-poststart-https-hook ready: true, restart count 0 Mar 25 05:42:33.512: INFO: csi-hostpathplugin-0 started at 2023-03-25 05:42:14 +0000 UTC (0+7 container statuses recorded) Mar 25 05:42:33.512: INFO: Container csi-attacher ready: true, restart count 0 Mar 25 05:42:33.512: INFO: Container csi-provisioner ready: true, restart count 0 Mar 25 05:42:33.512: INFO: Container csi-resizer ready: true, restart count 0 Mar 25 05:42:33.512: INFO: Container csi-snapshotter ready: true, restart count 0 Mar 25 05:42:33.512: INFO: Container hostpath ready: true, restart count 0 Mar 25 05:42:33.512: INFO: Container liveness-probe ready: true, restart count 0 Mar 25 05:42:33.512: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 25 05:42:33.512: INFO: inline-volume-tester-j4ndf started at 2023-03-25 05:42:18 +0000 UTC (0+1 container statuses recorded) Mar 25 05:42:33.512: INFO: Container csi-volume-tester ready: true, restart count 0 Mar 25 05:42:33.512: INFO: csi-mockplugin-attacher-0 started at 2023-03-25 05:42:02 +0000 UTC (0+1 container statuses recorded) Mar 25 05:42:33.512: INFO: Container csi-attacher ready: true, restart count 0 Mar 25 05:42:33.512: INFO: calico-node-lc9zl started at 2023-03-25 05:35:30 +0000 UTC (3+1 container statuses recorded) Mar 25 05:42:33.512: INFO: Init container upgrade-ipam ready: true, restart count 0 Mar 25 05:42:33.512: INFO: Init container install-cni ready: true, restart count 0 Mar 25 05:42:33.512: INFO: Init container mount-bpffs ready: true, restart count 0 Mar 25 05:42:33.512: INFO: Container calico-node ready: true, restart count 0 Mar 25 05:42:33.512: INFO: pod-projected-configmaps-61878de4-bfbc-4c4c-bb90-4daf9441ed77 started at 2023-03-25 05:39:21 +0000 UTC (0+3 container statuses recorded) Mar 25 05:42:33.512: INFO: Container createcm-volume-test ready: true, restart count 0 Mar 25 05:42:33.512: INFO: Container delcm-volume-test ready: true, restart count 0 Mar 25 05:42:33.512: INFO: Container updcm-volume-test ready: true, restart count 0 Mar 25 05:42:33.512: INFO: pod-handle-http-request started at 2023-03-25 05:40:55 +0000 UTC (0+2 container statuses recorded) Mar 25 05:42:33.512: INFO: Container container-handle-http-request ready: true, restart count 0 Mar 25 05:42:33.512: INFO: Container container-handle-https-request ready: true, restart count 0 Mar 25 05:42:33.512: INFO: kube-proxy-i-072608201d2c0f1ad started at 2023-03-25 05:35:30 +0000 UTC (0+1 container statuses recorded) Mar 25 05:42:33.512: INFO: Container kube-proxy ready: true, restart count 0 Mar 25 05:42:33.512: INFO: csi-hostpathplugin-0 started at 2023-03-25 05:41:58 +0000 UTC (0+7 container statuses recorded) Mar 25 05:42:33.512: INFO: Container csi-attacher ready: true, restart count 0 Mar 25 05:42:33.512: INFO: Container csi-provisioner ready: true, restart count 0 Mar 25 05:42:33.512: INFO: Container csi-resizer ready: true, restart count 0 Mar 25 05:42:33.512: INFO: Container csi-snapshotter ready: true, restart count 0 Mar 25 05:42:33.512: INFO: Container hostpath ready: true, restart count 0 Mar 25 05:42:33.512: INFO: Container liveness-probe ready: true, restart count 0 Mar 25 05:42:33.512: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 25 05:42:33.512: INFO: netserver-3 started at 2023-03-25 05:42:14 +0000 UTC (0+1 container statuses recorded) Mar 25 05:42:33.512: INFO: Container webserver ready: false, restart count 0 Mar 25 05:42:33.893: INFO: Latency metrics for node i-072608201d2c0f1ad END STEP: dump namespace information after failure - test/e2e/framework/framework.go:288 @ 03/25/23 05:42:33.893 (2.609s) < Exit [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] subPath - dump namespaces | framework.go:209 @ 03/25/23 05:42:33.893 (2.609s) > Enter [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] subPath - tear down framework | framework.go:206 @ 03/25/23 05:42:33.893 STEP: Destroying namespace "provisioning-5117" for this suite. - test/e2e/framework/framework.go:351 @ 03/25/23 05:42:33.893 < Exit [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] subPath - tear down framework | framework.go:206 @ 03/25/23 05:42:33.946 (54ms) > Enter [ReportAfterEach] TOP-LEVEL - test/e2e/e2e_test.go:144 @ 03/25/23 05:42:33.947 < Exit [ReportAfterEach] TOP-LEVEL - test/e2e/e2e_test.go:144 @ 03/25/23 05:42:33.947 (0s)
Filter through log files | View test history on testgrid
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[It\]\sExternal\sStorage\s\[Driver\:\sebs\.csi\.aws\.com\]\s\[Testpattern\:\sDynamic\sPV\s\(default\sfs\)\]\svolumes\sshould\sallow\sexec\sof\sfiles\son\sthe\svolume$'
[FAILED] failed to get logs from exec-volume-test-dynamicpv-wtkb for exec-container-dynamicpv-wtkb: Get "https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/volume-4037/pods/exec-volume-test-dynamicpv-wtkb/log?container=exec-container-dynamicpv-wtkb&previous=false": stream error: stream ID 175; INTERNAL_ERROR; received from peer In [It] at: test/e2e/framework/pod/output/output.go:247 @ 03/25/23 05:39:52.03from junit_01.xml
> Enter [BeforeEach] [Testpattern: Dynamic PV (default fs)] volumes - test/e2e/storage/framework/testsuite.go:51 @ 03/25/23 05:39:36.804 < Exit [BeforeEach] [Testpattern: Dynamic PV (default fs)] volumes - test/e2e/storage/framework/testsuite.go:51 @ 03/25/23 05:39:36.804 (0s) > Enter [BeforeEach] [Testpattern: Dynamic PV (default fs)] volumes - set up framework | framework.go:191 @ 03/25/23 05:39:36.804 STEP: Creating a kubernetes client - test/e2e/framework/framework.go:211 @ 03/25/23 05:39:36.804 Mar 25 05:39:36.804: INFO: >>> kubeConfig: /root/.kube/config STEP: Building a namespace api object, basename volume - test/e2e/framework/framework.go:250 @ 03/25/23 05:39:36.805 STEP: Waiting for a default service account to be provisioned in namespace - test/e2e/framework/framework.go:259 @ 03/25/23 05:39:36.962 STEP: Waiting for kube-root-ca.crt to be provisioned in namespace - test/e2e/framework/framework.go:262 @ 03/25/23 05:39:37.062 < Exit [BeforeEach] [Testpattern: Dynamic PV (default fs)] volumes - set up framework | framework.go:191 @ 03/25/23 05:39:37.159 (354ms) > Enter [BeforeEach] [Testpattern: Dynamic PV (default fs)] volumes - test/e2e/framework/metrics/init/init.go:33 @ 03/25/23 05:39:37.159 < Exit [BeforeEach] [Testpattern: Dynamic PV (default fs)] volumes - test/e2e/framework/metrics/init/init.go:33 @ 03/25/23 05:39:37.159 (0s) > Enter [It] should allow exec of files on the volume - test/e2e/storage/testsuites/volumes.go:194 @ 03/25/23 05:39:37.159 Mar 25 05:39:37.159: INFO: Creating resource for dynamic PV Mar 25 05:39:37.159: INFO: Using claimSize:1Gi, test suite supported size:{ 1Mi}, driver(ebs.csi.aws.com) supported size:{ 1Mi} STEP: creating a StorageClass volume-4037-e2e-sczds4p - test/e2e/storage/framework/volume_resource.go:102 @ 03/25/23 05:39:37.159 STEP: creating a claim - test/e2e/storage/framework/volume_resource.go:294 @ 03/25/23 05:39:37.212 Mar 25 05:39:37.212: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil STEP: Creating pod exec-volume-test-dynamicpv-wtkb - test/e2e/storage/testsuites/volumes.go:253 @ 03/25/23 05:39:37.316 STEP: Creating a pod to test exec-volume-test - test/e2e/framework/pod/output/output.go:243 @ 03/25/23 05:39:37.316 STEP: Saw pod success - test/e2e/framework/pod/wait.go:409 @ 03/25/23 05:39:51.816 Mar 25 05:39:51.871: INFO: Trying to get logs from node i-062679bb8c0ba4bce pod exec-volume-test-dynamicpv-wtkb container exec-container-dynamicpv-wtkb: <nil> Mar 25 05:39:51.920: INFO: Failed to get logs from node "i-062679bb8c0ba4bce" pod "exec-volume-test-dynamicpv-wtkb" container "exec-container-dynamicpv-wtkb". Get "https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/volume-4037/pods/exec-volume-test-dynamicpv-wtkb/log?container=exec-container-dynamicpv-wtkb&previous=false": stream error: stream ID 175; INTERNAL_ERROR; received from peer STEP: delete the pod - test/e2e/framework/pod/output/output.go:168 @ 03/25/23 05:39:51.92 Mar 25 05:39:52.030: INFO: Unexpected error: <*fmt.wrapError | 0xc0044bb620>: failed to get logs from exec-volume-test-dynamicpv-wtkb for exec-container-dynamicpv-wtkb: Get "https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/volume-4037/pods/exec-volume-test-dynamicpv-wtkb/log?container=exec-container-dynamicpv-wtkb&previous=false": stream error: stream ID 175; INTERNAL_ERROR; received from peer { msg: "failed to get logs from exec-volume-test-dynamicpv-wtkb for exec-container-dynamicpv-wtkb: Get \"https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/volume-4037/pods/exec-volume-test-dynamicpv-wtkb/log?container=exec-container-dynamicpv-wtkb&previous=false\": stream error: stream ID 175; INTERNAL_ERROR; received from peer", err: <*url.Error | 0xc0044f0ed0>{ Op: "Get", URL: "https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/volume-4037/pods/exec-volume-test-dynamicpv-wtkb/log?container=exec-container-dynamicpv-wtkb&previous=false", Err: <http2.StreamError>{ StreamID: 175, Code: 2, Cause: <*errors.errorString | 0xc0001e9680>{ s: "received from peer", }, }, }, } [FAILED] failed to get logs from exec-volume-test-dynamicpv-wtkb for exec-container-dynamicpv-wtkb: Get "https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/volume-4037/pods/exec-volume-test-dynamicpv-wtkb/log?container=exec-container-dynamicpv-wtkb&previous=false": stream error: stream ID 175; INTERNAL_ERROR; received from peer In [It] at: test/e2e/framework/pod/output/output.go:247 @ 03/25/23 05:39:52.03 < Exit [It] should allow exec of files on the volume - test/e2e/storage/testsuites/volumes.go:194 @ 03/25/23 05:39:52.03 (14.872s) > Enter [AfterEach] [Testpattern: Dynamic PV (default fs)] volumes - test/e2e/framework/node/init/init.go:33 @ 03/25/23 05:39:52.03 Mar 25 05:39:52.030: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready < Exit [AfterEach] [Testpattern: Dynamic PV (default fs)] volumes - test/e2e/framework/node/init/init.go:33 @ 03/25/23 05:39:52.082 (51ms) > Enter [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] volumes - test/e2e/storage/testsuites/volumes.go:197 @ 03/25/23 05:39:52.082 STEP: Deleting pvc - test/e2e/storage/framework/volume_resource.go:181 @ 03/25/23 05:39:52.082 Mar 25 05:39:52.187: INFO: Deleting PersistentVolumeClaim "ebs.csi.aws.comqpmdc" Mar 25 05:39:52.253: INFO: Waiting up to 5m0s for PersistentVolume pvc-dc77c9a5-0443-4ac5-888d-ae032fc5ece0 to get deleted Mar 25 05:39:52.302: INFO: PersistentVolume pvc-dc77c9a5-0443-4ac5-888d-ae032fc5ece0 found and phase=Released (48.733061ms) Mar 25 05:39:57.353: INFO: PersistentVolume pvc-dc77c9a5-0443-4ac5-888d-ae032fc5ece0 found and phase=Released (5.099653218s) Mar 25 05:40:02.404: INFO: PersistentVolume pvc-dc77c9a5-0443-4ac5-888d-ae032fc5ece0 found and phase=Released (10.150999803s) Mar 25 05:40:07.454: INFO: PersistentVolume pvc-dc77c9a5-0443-4ac5-888d-ae032fc5ece0 was removed STEP: Deleting sc - test/e2e/storage/framework/volume_resource.go:228 @ 03/25/23 05:40:07.454 < Exit [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] volumes - test/e2e/storage/testsuites/volumes.go:197 @ 03/25/23 05:40:07.507 (15.425s) > Enter [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] volumes - test/e2e/framework/metrics/init/init.go:35 @ 03/25/23 05:40:07.507 < Exit [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] volumes - test/e2e/framework/metrics/init/init.go:35 @ 03/25/23 05:40:07.507 (0s) > Enter [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] volumes - dump namespaces | framework.go:209 @ 03/25/23 05:40:07.507 STEP: dump namespace information after failure - test/e2e/framework/framework.go:288 @ 03/25/23 05:40:07.507 STEP: Collecting events from namespace "volume-4037". - test/e2e/framework/debug/dump.go:42 @ 03/25/23 05:40:07.507 STEP: Found 9 events. - test/e2e/framework/debug/dump.go:46 @ 03/25/23 05:40:07.556 Mar 25 05:40:07.556: INFO: At 2023-03-25 05:39:37 +0000 UTC - event for ebs.csi.aws.comqpmdc: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding Mar 25 05:40:07.556: INFO: At 2023-03-25 05:39:37 +0000 UTC - event for ebs.csi.aws.comqpmdc: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator Mar 25 05:40:07.556: INFO: At 2023-03-25 05:39:37 +0000 UTC - event for ebs.csi.aws.comqpmdc: {ebs.csi.aws.com_i-0168f82458996cc53_0b589c46-3396-4499-94ba-eb51a48f7a58 } Provisioning: External provisioner is provisioning volume for claim "volume-4037/ebs.csi.aws.comqpmdc" Mar 25 05:40:07.556: INFO: At 2023-03-25 05:39:40 +0000 UTC - event for ebs.csi.aws.comqpmdc: {ebs.csi.aws.com_i-0168f82458996cc53_0b589c46-3396-4499-94ba-eb51a48f7a58 } ProvisioningSucceeded: Successfully provisioned volume pvc-dc77c9a5-0443-4ac5-888d-ae032fc5ece0 Mar 25 05:40:07.556: INFO: At 2023-03-25 05:39:41 +0000 UTC - event for exec-volume-test-dynamicpv-wtkb: {default-scheduler } Scheduled: Successfully assigned volume-4037/exec-volume-test-dynamicpv-wtkb to i-062679bb8c0ba4bce Mar 25 05:40:07.556: INFO: At 2023-03-25 05:39:43 +0000 UTC - event for exec-volume-test-dynamicpv-wtkb: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-dc77c9a5-0443-4ac5-888d-ae032fc5ece0" Mar 25 05:40:07.556: INFO: At 2023-03-25 05:39:47 +0000 UTC - event for exec-volume-test-dynamicpv-wtkb: {kubelet i-062679bb8c0ba4bce} Pulled: Container image "registry.k8s.io/e2e-test-images/nginx:1.14-4" already present on machine Mar 25 05:40:07.556: INFO: At 2023-03-25 05:39:47 +0000 UTC - event for exec-volume-test-dynamicpv-wtkb: {kubelet i-062679bb8c0ba4bce} Created: Created container exec-container-dynamicpv-wtkb Mar 25 05:40:07.556: INFO: At 2023-03-25 05:39:47 +0000 UTC - event for exec-volume-test-dynamicpv-wtkb: {kubelet i-062679bb8c0ba4bce} Started: Started container exec-container-dynamicpv-wtkb Mar 25 05:40:07.608: INFO: POD NODE PHASE GRACE CONDITIONS Mar 25 05:40:07.608: INFO: Mar 25 05:40:07.661: INFO: Logging node info for node i-0168f82458996cc53 Mar 25 05:40:07.711: INFO: Node Info: &Node{ObjectMeta:{i-0168f82458996cc53 f4eb302c-c477-45ac-bcac-ffed65f6580f 5776 0 2023-03-25 05:18:57 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-1 failure-domain.beta.kubernetes.io/zone:us-west-1b kops.k8s.io/kops-controller-pki: kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0168f82458996cc53 kubernetes.io/os:linux node-role.kubernetes.io/control-plane: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:us-west-1b topology.kubernetes.io/region:us-west-1 topology.kubernetes.io/zone:us-west-1b] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0168f82458996cc53"} node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:172.20.48.81/19 projectcalico.org/IPv4IPIPTunnelAddr:100.100.215.192 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-03-25 05:18:57 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {protokube Update v1 2023-03-25 05:18:58 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {calico-node Update v1 2023-03-25 05:20:22 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4IPIPTunnelAddr":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {aws-cloud-controller-manager Update v1 2023-03-25 05:20:26 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-25 05:20:26 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2023-03-25 05:20:31 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.5.0/24\"":{}},"f:taints":{}}} } {kubelet Update v1 2023-03-25 05:36:59 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.5.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-1b/i-0168f82458996cc53,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/control-plane,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.5.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3892310016 0} {<nil>} 3801084Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3787452416 0} {<nil>} 3698684Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-03-25 05:20:22 +0000 UTC,LastTransitionTime:2023-03-25 05:20:22 +0000 UTC,Reason:CalicoIsUp,Message:Calico is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-25 05:36:59 +0000 UTC,LastTransitionTime:2023-03-25 05:18:50 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-25 05:36:59 +0000 UTC,LastTransitionTime:2023-03-25 05:18:50 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-25 05:36:59 +0000 UTC,LastTransitionTime:2023-03-25 05:18:50 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-25 05:36:59 +0000 UTC,LastTransitionTime:2023-03-25 05:20:29 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.48.81,},NodeAddress{Type:ExternalIP,Address:3.101.135.77,},NodeAddress{Type:InternalDNS,Address:i-0168f82458996cc53.us-west-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-0168f82458996cc53.us-west-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-3-101-135-77.us-west-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2d1ccc7ebf313420bd1f530a94fbfd,SystemUUID:ec2d1ccc-7ebf-3134-20bd-1f530a94fbfd,BootID:15cee321-56e7-4290-ab48-47b60b0bc657,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.28.0-alpha.0,KubeProxyVersion:v1.28.0-alpha.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcd@sha256:dd75ec974b0a2a6f6bb47001ba09207976e625db898d1b16735528c009cb171c registry.k8s.io/etcd:3.5.6-0],SizeBytes:102542580,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:6f72b851544986cb0921b53ea655ec04c36131248f16d4ad110cb3ca0c369dc1 registry.k8s.io/etcd:3.5.4-0],SizeBytes:102157811,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:13f53ed1d91e2e11aac476ee9a0269fdda6cc4874eba903efd40daf50c55eee5 registry.k8s.io/etcd:3.5.3-0],SizeBytes:102143581,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:51eae8381dcb1078289fa7b4f3df2630cdc18d09fb56f8e56b41c40e191d6c83 registry.k8s.io/etcd:3.5.7-0],SizeBytes:101639218,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:4afb99b4690b418ffc2ceb67e1a17376457e441c1f09ab55447f0aaf992fa646 registry.k8s.io/etcd:3.4.3-0],SizeBytes:100947667,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:9ce33ba33d8e738a5b85ed50b5080ac746deceed4a7496c550927a7a19ca3b6d registry.k8s.io/etcd:3.5.0-0],SizeBytes:99868722,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:64b9ea357325d5db9f8a723dcf503b5a449177b17ac87d69481e126bb724c263 registry.k8s.io/etcd:3.5.1-0],SizeBytes:98888614,},ContainerImage{Names:[docker.io/calico/cni@sha256:e282ea2914c806b5de2976330a17cfb5e6dcef47147bceb1432ca5c75fd46f50 docker.io/calico/cni:v3.24.5],SizeBytes:87458443,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:4ad90a11b55313b182afc186b9876c8e891531b8db4c9bf1541953021618d0e2 registry.k8s.io/etcd:3.4.13-0],SizeBytes:86742272,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:0d2ab6b15b4fe0b903b9e81192d942fb39c29be61fe341cd16db5d0165b0435c registry.k8s.io/etcd:3.3.17-0],SizeBytes:85524408,},ContainerImage{Names:[docker.io/calico/node@sha256:5972ad2bcbdc668564d3e26960c9c513b2d7b05581c704747cf7c62ef3a405a6 docker.io/calico/node:v3.24.5],SizeBytes:81584866,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:17da501f5d2a675be46040422a27b7cc21b8a43895ac998b171db1c346f361f7 registry.k8s.io/etcd:3.3.10-0],SizeBytes:76162358,},ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager-slim@sha256:8727adc059334e0e3329dbccce116a734da4a090da5e86c8cddccaf333caa666 registry.k8s.io/etcdadm/etcd-manager-slim:v3.0.20230201],SizeBytes:63926165,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:905d7ca17fd02bc24c0eba9a062753aba15db3e31422390bc3238eb762339b20 registry.k8s.io/etcd:3.2.24-1],SizeBytes:63423287,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.27.0-alpha.1],SizeBytes:45219545,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.27.0-alpha.1],SizeBytes:43628585,},ContainerImage{Names:[registry.k8s.io/kube-apiserver@sha256:49934a29c559e79ed19fc38b4697b18d8fe174b17bf56192588575f980c93956 registry.k8s.io/kube-apiserver:v1.28.0-alpha.0],SizeBytes:33212506,},ContainerImage{Names:[docker.io/calico/kube-controllers@sha256:2b6acd7f677f76ffe12ecf3ea7df92eb9b1bdb07336d1ac2a54c7631fb753f7e docker.io/calico/kube-controllers:v3.24.5],SizeBytes:31137169,},ContainerImage{Names:[registry.k8s.io/kube-controller-manager@sha256:974748384634982c73527fdb3c9ec0ec476bc0d52b3dd1d611a380b7f99af1f3 registry.k8s.io/kube-controller-manager:v1.28.0-alpha.0],SizeBytes:30820280,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/kube-proxy@sha256:8a03b3147cd7be8e8235ae1cd70cf69aca9ce03ba1cfaffea320540bb5afa496 registry.k8s.io/kube-proxy:v1.28.0-alpha.0],SizeBytes:23896850,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:23345856,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:22381475,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:22085298,},ContainerImage{Names:[registry.k8s.io/provider-aws/cloud-controller-manager@sha256:fdeb61e3e42ecd9cca868d550ebdb88dd6341d9e91fcfa9a37e227dab2ad22cb registry.k8s.io/provider-aws/cloud-controller-manager:v1.26.0],SizeBytes:20154862,},ContainerImage{Names:[registry.k8s.io/kube-scheduler@sha256:cbf4007abb076188f16b975bc2b74452603cc7b34c60a29d92f14c38ad02dc11 registry.k8s.io/kube-scheduler:v1.28.0-alpha.0],SizeBytes:18078674,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.27.0-alpha.1],SizeBytes:5074298,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Mar 25 05:40:07.711: INFO: Logging kubelet events for node i-0168f82458996cc53 Mar 25 05:40:07.777: INFO: Logging pods the kubelet thinks is on node i-0168f82458996cc53 Mar 25 05:40:07.896: INFO: kube-controller-manager-i-0168f82458996cc53 started at 2023-03-25 05:17:36 +0000 UTC (0+1 container statuses recorded) Mar 25 05:40:07.896: INFO: Container kube-controller-manager ready: true, restart count 4 Mar 25 05:40:07.896: INFO: etcd-manager-events-i-0168f82458996cc53 started at 2023-03-25 05:17:36 +0000 UTC (11+1 container statuses recorded) Mar 25 05:40:07.896: INFO: Init container init-etcd-3-2-24 ready: true, restart count 0 Mar 25 05:40:07.896: INFO: Init container init-etcd-3-3-10 ready: true, restart count 0 Mar 25 05:40:07.896: INFO: Init container init-etcd-3-3-17 ready: true, restart count 0 Mar 25 05:40:07.896: INFO: Init container init-etcd-3-4-13 ready: true, restart count 0 Mar 25 05:40:07.896: INFO: Init container init-etcd-3-4-3 ready: true, restart count 0 Mar 25 05:40:07.896: INFO: Init container init-etcd-3-5-0 ready: true, restart count 0 Mar 25 05:40:07.896: INFO: Init container init-etcd-3-5-1 ready: true, restart count 0 Mar 25 05:40:07.896: INFO: Init container init-etcd-3-5-3 ready: true, restart count 0 Mar 25 05:40:07.896: INFO: Init container init-etcd-3-5-4 ready: true, restart count 0 Mar 25 05:40:07.896: INFO: Init container init-etcd-3-5-6 ready: true, restart count 0 Mar 25 05:40:07.896: INFO: Init container init-etcd-3-5-7 ready: true, restart count 0 Mar 25 05:40:07.896: INFO: Container etcd-manager ready: true, restart count 0 Mar 25 05:40:07.896: INFO: etcd-manager-main-i-0168f82458996cc53 started at 2023-03-25 05:17:36 +0000 UTC (11+1 container statuses recorded) Mar 25 05:40:07.896: INFO: Init container init-etcd-3-2-24 ready: true, restart count 0 Mar 25 05:40:07.896: INFO: Init container init-etcd-3-3-10 ready: true, restart count 0 Mar 25 05:40:07.896: INFO: Init container init-etcd-3-3-17 ready: true, restart count 0 Mar 25 05:40:07.896: INFO: Init container init-etcd-3-4-13 ready: true, restart count 0 Mar 25 05:40:07.896: INFO: Init container init-etcd-3-4-3 ready: true, restart count 0 Mar 25 05:40:07.896: INFO: Init container init-etcd-3-5-0 ready: true, restart count 0 Mar 25 05:40:07.896: INFO: Init container init-etcd-3-5-1 ready: true, restart count 0 Mar 25 05:40:07.896: INFO: Init container init-etcd-3-5-3 ready: true, restart count 0 Mar 25 05:40:07.896: INFO: Init container init-etcd-3-5-4 ready: true, restart count 0 Mar 25 05:40:07.896: INFO: Init container init-etcd-3-5-6 ready: true, restart count 0 Mar 25 05:40:07.896: INFO: Init container init-etcd-3-5-7 ready: true, restart count 0 Mar 25 05:40:07.896: INFO: Container etcd-manager ready: true, restart count 0 Mar 25 05:40:07.896: INFO: dns-controller-69987775c6-hbn69 started at 2023-03-25 05:19:47 +0000 UTC (0+1 container statuses recorded) Mar 25 05:40:07.896: INFO: Container dns-controller ready: true, restart count 0 Mar 25 05:40:07.896: INFO: aws-cloud-controller-manager-26vbv started at 2023-03-25 05:20:06 +0000 UTC (0+1 container statuses recorded) Mar 25 05:40:07.896: INFO: Container aws-cloud-controller-manager ready: true, restart count 0 Mar 25 05:40:07.896: INFO: calico-node-rdjq2 started at 2023-03-25 05:20:06 +0000 UTC (3+1 container statuses recorded) Mar 25 05:40:07.896: INFO: Init container upgrade-ipam ready: true, restart count 0 Mar 25 05:40:07.896: INFO: Init container install-cni ready: true, restart count 0 Mar 25 05:40:07.896: INFO: Init container mount-bpffs ready: true, restart count 0 Mar 25 05:40:07.896: INFO: Container calico-node ready: true, restart count 0 Mar 25 05:40:07.896: INFO: ebs-csi-node-zjtj6 started at 2023-03-25 05:20:06 +0000 UTC (0+3 container statuses recorded) Mar 25 05:40:07.896: INFO: Container ebs-plugin ready: true, restart count 0 Mar 25 05:40:07.896: INFO: Container liveness-probe ready: true, restart count 0 Mar 25 05:40:07.896: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 25 05:40:07.896: INFO: kube-apiserver-i-0168f82458996cc53 started at 2023-03-25 05:17:36 +0000 UTC (0+2 container statuses recorded) Mar 25 05:40:07.896: INFO: Container healthcheck ready: true, restart count 0 Mar 25 05:40:07.896: INFO: Container kube-apiserver ready: true, restart count 2 Mar 25 05:40:07.896: INFO: kube-proxy-i-0168f82458996cc53 started at 2023-03-25 05:17:36 +0000 UTC (0+1 container statuses recorded) Mar 25 05:40:07.896: INFO: Container kube-proxy ready: true, restart count 0 Mar 25 05:40:07.896: INFO: kube-scheduler-i-0168f82458996cc53 started at 2023-03-25 05:17:36 +0000 UTC (0+1 container statuses recorded) Mar 25 05:40:07.896: INFO: Container kube-scheduler ready: true, restart count 0 Mar 25 05:40:07.896: INFO: kops-controller-cgw95 started at 2023-03-25 05:20:06 +0000 UTC (0+1 container statuses recorded) Mar 25 05:40:07.896: INFO: Container kops-controller ready: true, restart count 0 Mar 25 05:40:07.896: INFO: calico-kube-controllers-55f488656-rr8rz started at 2023-03-25 05:20:29 +0000 UTC (0+1 container statuses recorded) Mar 25 05:40:07.896: INFO: Container calico-kube-controllers ready: true, restart count 0 Mar 25 05:40:07.896: INFO: ebs-csi-controller-79f984545f-9q5ks started at 2023-03-25 05:21:26 +0000 UTC (0+5 container statuses recorded) Mar 25 05:40:07.896: INFO: Container csi-attacher ready: true, restart count 0 Mar 25 05:40:07.896: INFO: Container csi-provisioner ready: true, restart count 0 Mar 25 05:40:07.896: INFO: Container csi-resizer ready: true, restart count 0 Mar 25 05:40:07.896: INFO: Container ebs-plugin ready: true, restart count 0 Mar 25 05:40:07.896: INFO: Container liveness-probe ready: true, restart count 0 Mar 25 05:40:08.331: INFO: Latency metrics for node i-0168f82458996cc53 Mar 25 05:40:08.331: INFO: Logging node info for node i-03d301e6f030748f1 Mar 25 05:40:08.383: INFO: Node Info: &Node{ObjectMeta:{i-03d301e6f030748f1 897f52bd-65e2-4107-843a-37e36aa74032 14883 0 2023-03-25 05:31:35 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-1 failure-domain.beta.kubernetes.io/zone:us-west-1b kubernetes.io/arch:amd64 kubernetes.io/hostname:i-03d301e6f030748f1 kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:us-west-1b topology.hostpath.csi/node:i-03d301e6f030748f1 topology.kubernetes.io/region:us-west-1 topology.kubernetes.io/zone:us-west-1b] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-03d301e6f030748f1"} node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:172.20.50.199/19 projectcalico.org/IPv4IPIPTunnelAddr:100.110.167.64 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-03-25 05:31:35 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-25 05:31:35 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-03-25 05:31:35 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2023-03-25 05:31:35 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2023-03-25 05:31:48 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.8.0/24\"":{}}}} } {calico-node Update v1 2023-03-25 05:31:56 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4IPIPTunnelAddr":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2023-03-25 05:40:05 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.8.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-1b/i-03d301e6f030748f1,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.8.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-03-25 05:31:56 +0000 UTC,LastTransitionTime:2023-03-25 05:31:56 +0000 UTC,Reason:CalicoIsUp,Message:Calico is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-25 05:40:05 +0000 UTC,LastTransitionTime:2023-03-25 05:31:35 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-25 05:40:05 +0000 UTC,LastTransitionTime:2023-03-25 05:31:35 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-25 05:40:05 +0000 UTC,LastTransitionTime:2023-03-25 05:31:35 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-25 05:40:05 +0000 UTC,LastTransitionTime:2023-03-25 05:31:48 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.50.199,},NodeAddress{Type:ExternalIP,Address:52.53.245.63,},NodeAddress{Type:InternalDNS,Address:i-03d301e6f030748f1.us-west-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-03d301e6f030748f1.us-west-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-52-53-245-63.us-west-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec27152f7af9b9bfd1355ee19c256086,SystemUUID:ec27152f-7af9-b9bf-d135-5ee19c256086,BootID:1c303852-f2ce-45f9-ab14-6e8de97a0a3e,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.28.0-alpha.0,KubeProxyVersion:v1.28.0-alpha.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[docker.io/calico/cni@sha256:e282ea2914c806b5de2976330a17cfb5e6dcef47147bceb1432ca5c75fd46f50 docker.io/calico/cni:v3.24.5],SizeBytes:87458443,},ContainerImage{Names:[docker.io/calico/node@sha256:5972ad2bcbdc668564d3e26960c9c513b2d7b05581c704747cf7c62ef3a405a6 docker.io/calico/node:v3.24.5],SizeBytes:81584866,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nautilus@sha256:80ba6c8c44f9623f06e868a1aa66026c8ec438ad814f9ec95e9333b415fe3550 registry.k8s.io/e2e-test-images/nautilus:1.7],SizeBytes:49641698,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:e468dddcd275163a042ab297b2d8c2aca50d5e148d2d22f3b6ba119e2f31fa79 registry.k8s.io/sig-storage/csi-provisioner:v3.4.0],SizeBytes:27427836,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/kube-proxy@sha256:8a03b3147cd7be8e8235ae1cd70cf69aca9ce03ba1cfaffea320540bb5afa496 registry.k8s.io/kube-proxy:v1.28.0-alpha.0],SizeBytes:23896850,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:23881995,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18758628,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18233005,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8688564,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6978614,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Mar 25 05:40:08.384: INFO: Logging kubelet events for node i-03d301e6f030748f1 Mar 25 05:40:08.448: INFO: Logging pods the kubelet thinks is on node i-03d301e6f030748f1 Mar 25 05:40:08.509: INFO: pod-terminate-status-0-6 started at 2023-03-25 05:40:07 +0000 UTC (1+1 container statuses recorded) Mar 25 05:40:08.509: INFO: Init container fail ready: false, restart count 0 Mar 25 05:40:08.509: INFO: Container blocked ready: false, restart count 0 Mar 25 05:40:08.509: INFO: deployment-shared-map-item-removal-5c89cddd56-v8tqd started at 2023-03-25 05:40:08 +0000 UTC (0+1 container statuses recorded) Mar 25 05:40:08.509: INFO: Container nginx ready: false, restart count 0 Mar 25 05:40:08.509: INFO: pvc-volume-tester-writer-dng6w started at 2023-03-25 05:40:08 +0000 UTC (0+1 container statuses recorded) Mar 25 05:40:08.509: INFO: Container volume-tester ready: false, restart count 0 Mar 25 05:40:08.509: INFO: kube-proxy-i-03d301e6f030748f1 started at 2023-03-25 05:31:35 +0000 UTC (0+1 container statuses recorded) Mar 25 05:40:08.509: INFO: Container kube-proxy ready: true, restart count 0 Mar 25 05:40:08.509: INFO: ebs-csi-node-lljwl started at 2023-03-25 05:31:35 +0000 UTC (0+3 container statuses recorded) Mar 25 05:40:08.509: INFO: Container ebs-plugin ready: true, restart count 0 Mar 25 05:40:08.509: INFO: Container liveness-probe ready: true, restart count 0 Mar 25 05:40:08.509: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 25 05:40:08.509: INFO: rand-non-local-rlgmz started at 2023-03-25 05:39:59 +0000 UTC (0+1 container statuses recorded) Mar 25 05:40:08.509: INFO: Container c ready: false, restart count 0 Mar 25 05:40:08.509: INFO: pod-terminate-status-1-6 started at 2023-03-25 05:40:06 +0000 UTC (1+1 container statuses recorded) Mar 25 05:40:08.509: INFO: Init container fail ready: false, restart count 0 Mar 25 05:40:08.509: INFO: Container blocked ready: false, restart count 0 Mar 25 05:40:08.509: INFO: pod-csi-inline-volumes started at 2023-03-25 05:38:08 +0000 UTC (0+1 container statuses recorded) Mar 25 05:40:08.509: INFO: Container pod-csi-inline-volumes ready: false, restart count 0 Mar 25 05:40:08.509: INFO: test-rolling-update-with-lb-69fcddc54f-m6d8k started at 2023-03-25 05:38:08 +0000 UTC (0+1 container statuses recorded) Mar 25 05:40:08.509: INFO: Container agnhost ready: true, restart count 0 Mar 25 05:40:08.509: INFO: pause-pod-1 started at 2023-03-25 05:39:50 +0000 UTC (0+1 container statuses recorded) Mar 25 05:40:08.509: INFO: Container agnhost-container ready: true, restart count 0 Mar 25 05:40:08.509: INFO: pod-c1c3c99e-0c22-459c-88c2-c280f3e1212e started at 2023-03-25 05:40:06 +0000 UTC (0+1 container statuses recorded) Mar 25 05:40:08.509: INFO: Container write-pod ready: false, restart count 0 Mar 25 05:40:08.509: INFO: calico-node-h96vq started at 2023-03-25 05:31:35 +0000 UTC (3+1 container statuses recorded) Mar 25 05:40:08.509: INFO: Init container upgrade-ipam ready: true, restart count 0 Mar 25 05:40:08.509: INFO: Init container install-cni ready: true, restart count 0 Mar 25 05:40:08.509: INFO: Init container mount-bpffs ready: true, restart count 0 Mar 25 05:40:08.509: INFO: Container calico-node ready: true, restart count 0 Mar 25 05:40:08.509: INFO: pod-terminate-status-2-6 started at 2023-03-25 05:40:06 +0000 UTC (1+1 container statuses recorded) Mar 25 05:40:08.509: INFO: Init container fail ready: false, restart count 0 Mar 25 05:40:08.509: INFO: Container blocked ready: false, restart count 0 Mar 25 05:40:08.509: INFO: implicit-nonroot-uid started at 2023-03-25 05:39:57 +0000 UTC (0+1 container statuses recorded) Mar 25 05:40:08.509: INFO: Container implicit-nonroot-uid ready: false, restart count 0 Mar 25 05:40:08.829: INFO: Latency metrics for node i-03d301e6f030748f1 Mar 25 05:40:08.829: INFO: Logging node info for node i-049fdb2c1cde894df Mar 25 05:40:08.879: INFO: Node Info: &Node{ObjectMeta:{i-049fdb2c1cde894df b43a51e6-dda7-4047-b368-72d515695b4e 14836 0 2023-03-25 05:23:34 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-1 failure-domain.beta.kubernetes.io/zone:us-west-1b kubernetes.io/arch:amd64 kubernetes.io/hostname:i-049fdb2c1cde894df kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:us-west-1b topology.hostpath.csi/node:i-049fdb2c1cde894df topology.kubernetes.io/region:us-west-1 topology.kubernetes.io/zone:us-west-1b] map[csi.volume.kubernetes.io/nodeid:{"csi-mock-csi-mock-volumes-expansion-3246":"csi-mock-csi-mock-volumes-expansion-3246","ebs.csi.aws.com":"i-049fdb2c1cde894df"} node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:172.20.43.61/19 projectcalico.org/IPv4IPIPTunnelAddr:100.97.18.64 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-03-25 05:23:34 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-25 05:23:34 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-03-25 05:23:34 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2023-03-25 05:23:34 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2023-03-25 05:23:45 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.6.0/24\"":{}}}} } {calico-node Update v1 2023-03-25 05:23:52 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4IPIPTunnelAddr":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2023-03-25 05:40:00 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-03-25 05:40:04 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.6.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-1b/i-049fdb2c1cde894df,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.6.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-03-25 05:23:52 +0000 UTC,LastTransitionTime:2023-03-25 05:23:52 +0000 UTC,Reason:CalicoIsUp,Message:Calico is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-25 05:40:04 +0000 UTC,LastTransitionTime:2023-03-25 05:23:34 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-25 05:40:04 +0000 UTC,LastTransitionTime:2023-03-25 05:23:34 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-25 05:40:04 +0000 UTC,LastTransitionTime:2023-03-25 05:23:34 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-25 05:40:04 +0000 UTC,LastTransitionTime:2023-03-25 05:23:45 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.43.61,},NodeAddress{Type:ExternalIP,Address:18.144.75.67,},NodeAddress{Type:InternalDNS,Address:i-049fdb2c1cde894df.us-west-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-049fdb2c1cde894df.us-west-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-18-144-75-67.us-west-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec252e592d19e1e6c583e87505431016,SystemUUID:ec252e59-2d19-e1e6-c583-e87505431016,BootID:45436b5d-1eb0-4b6f-a1d1-a1492b31b5d7,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.28.0-alpha.0,KubeProxyVersion:v1.28.0-alpha.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[docker.io/calico/cni@sha256:e282ea2914c806b5de2976330a17cfb5e6dcef47147bceb1432ca5c75fd46f50 docker.io/calico/cni:v3.24.5],SizeBytes:87458443,},ContainerImage{Names:[docker.io/calico/node@sha256:5972ad2bcbdc668564d3e26960c9c513b2d7b05581c704747cf7c62ef3a405a6 docker.io/calico/node:v3.24.5],SizeBytes:81584866,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:e468dddcd275163a042ab297b2d8c2aca50d5e148d2d22f3b6ba119e2f31fa79 registry.k8s.io/sig-storage/csi-provisioner:v3.4.0],SizeBytes:27427836,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/kube-proxy@sha256:8a03b3147cd7be8e8235ae1cd70cf69aca9ce03ba1cfaffea320540bb5afa496 registry.k8s.io/kube-proxy:v1.28.0-alpha.0],SizeBytes:23896850,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:23881995,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:68d396900aeaa072c1f27289485fdac29834045a6f3ffe369bf389d830ef572d registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.6],SizeBytes:20293261,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18758628,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18233005,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:15273057,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8688564,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6978614,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[kubernetes.io/csi/ebs.csi.aws.com^vol-0225877f54e65a091],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-0225877f54e65a091,DevicePath:,},},Config:nil,},} Mar 25 05:40:08.880: INFO: Logging kubelet events for node i-049fdb2c1cde894df Mar 25 05:40:08.943: INFO: Logging pods the kubelet thinks is on node i-049fdb2c1cde894df Mar 25 05:40:09.021: INFO: pod-0c48c0e8-3c8a-42b0-9b49-339ca3dfb2af started at 2023-03-25 05:39:39 +0000 UTC (0+1 container statuses recorded) Mar 25 05:40:09.021: INFO: Container write-pod ready: true, restart count 0 Mar 25 05:40:09.021: INFO: pod-subpath-test-configmap-qvpv started at 2023-03-25 05:39:45 +0000 UTC (0+1 container statuses recorded) Mar 25 05:40:09.021: INFO: Container test-container-subpath-configmap-qvpv ready: false, restart count 0 Mar 25 05:40:09.021: INFO: hostexec-i-049fdb2c1cde894df-99m4z started at 2023-03-25 05:39:56 +0000 UTC (0+1 container statuses recorded) Mar 25 05:40:09.021: INFO: Container agnhost-container ready: true, restart count 0 Mar 25 05:40:09.021: INFO: csi-hostpathplugin-0 started at 2023-03-25 05:40:07 +0000 UTC (0+7 container statuses recorded) Mar 25 05:40:09.021: INFO: Container csi-attacher ready: false, restart count 0 Mar 25 05:40:09.021: INFO: Container csi-provisioner ready: false, restart count 0 Mar 25 05:40:09.021: INFO: Container csi-resizer ready: false, restart count 0 Mar 25 05:40:09.021: INFO: Container csi-snapshotter ready: false, restart count 0 Mar 25 05:40:09.021: INFO: Container hostpath ready: false, restart count 0 Mar 25 05:40:09.021: INFO: Container liveness-probe ready: false, restart count 0 Mar 25 05:40:09.021: INFO: Container node-driver-registrar ready: false, restart count 0 Mar 25 05:40:09.021: INFO: rand-non-local-2czp6 started at 2023-03-25 05:39:48 +0000 UTC (0+1 container statuses recorded) Mar 25 05:40:09.021: INFO: Container c ready: false, restart count 0 Mar 25 05:40:09.021: INFO: exec-volume-test-preprovisionedpv-wwbz started at 2023-03-25 05:40:08 +0000 UTC (0+1 container statuses recorded) Mar 25 05:40:09.021: INFO: Container exec-container-preprovisionedpv-wwbz ready: false, restart count 0 Mar 25 05:40:09.021: INFO: pod-with-poststart-exec-hook started at 2023-03-25 05:38:01 +0000 UTC (0+1 container statuses recorded) Mar 25 05:40:09.021: INFO: Container pod-with-poststart-exec-hook ready: true, restart count 0 Mar 25 05:40:09.021: INFO: coredns-autoscaler-7cb5c5b969-h485x started at 2023-03-25 05:24:36 +0000 UTC (0+1 container statuses recorded) Mar 25 05:40:09.021: INFO: Container autoscaler ready: true, restart count 0 Mar 25 05:40:09.021: INFO: pod-9e7bafc9-4859-4926-9512-f7289fe85d3b started at 2023-03-25 05:39:57 +0000 UTC (0+1 container statuses recorded) Mar 25 05:40:09.021: INFO: Container write-pod ready: true, restart count 0 Mar 25 05:40:09.021: INFO: hostexec-i-049fdb2c1cde894df-2c6x5 started at 2023-03-25 05:39:33 +0000 UTC (0+1 container statuses recorded) Mar 25 05:40:09.021: INFO: Container agnhost-container ready: true, restart count 0 Mar 25 05:40:09.021: INFO: pvc-volume-tester-dlhbm started at 2023-03-25 05:40:03 +0000 UTC (0+1 container statuses recorded) Mar 25 05:40:09.021: INFO: Container volume-tester ready: true, restart count 0 Mar 25 05:40:09.021: INFO: hostexec-i-049fdb2c1cde894df-p6dhk started at 2023-03-25 05:40:04 +0000 UTC (0+1 container statuses recorded) Mar 25 05:40:09.021: INFO: Container agnhost-container ready: true, restart count 0 Mar 25 05:40:09.021: INFO: deployment-shared-map-item-removal-5c89cddd56-k9n85 started at 2023-03-25 05:40:08 +0000 UTC (0+1 container statuses recorded) Mar 25 05:40:09.021: INFO: Container nginx ready: false, restart count 0 Mar 25 05:40:09.021: INFO: csi-mockplugin-0 started at 2023-03-25 05:39:55 +0000 UTC (0+4 container statuses recorded) Mar 25 05:40:09.021: INFO: Container busybox ready: true, restart count 0 Mar 25 05:40:09.021: INFO: Container csi-provisioner ready: true, restart count 0 Mar 25 05:40:09.021: INFO: Container driver-registrar ready: true, restart count 0 Mar 25 05:40:09.021: INFO: Container mock ready: true, restart count 0 Mar 25 05:40:09.021: INFO: ebs-csi-node-m9zxz started at 2023-03-25 05:23:35 +0000 UTC (0+3 container statuses recorded) Mar 25 05:40:09.021: INFO: Container ebs-plugin ready: true, restart count 0 Mar 25 05:40:09.021: INFO: Container liveness-probe ready: true, restart count 0 Mar 25 05:40:09.021: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 25 05:40:09.021: INFO: calico-node-wt5p7 started at 2023-03-25 05:23:35 +0000 UTC (3+1 container statuses recorded) Mar 25 05:40:09.021: INFO: Init container upgrade-ipam ready: true, restart count 0 Mar 25 05:40:09.021: INFO: Init container install-cni ready: true, restart count 0 Mar 25 05:40:09.021: INFO: Init container mount-bpffs ready: true, restart count 0 Mar 25 05:40:09.021: INFO: Container calico-node ready: true, restart count 0 Mar 25 05:40:09.021: INFO: coredns-559769c974-pgqhl started at 2023-03-25 05:24:36 +0000 UTC (0+1 container statuses recorded) Mar 25 05:40:09.021: INFO: Container coredns ready: true, restart count 0 Mar 25 05:40:09.021: INFO: suspend-false-to-true-vxmf9 started at 2023-03-25 05:39:41 +0000 UTC (0+1 container statuses recorded) Mar 25 05:40:09.021: INFO: Container c ready: true, restart count 0 Mar 25 05:40:09.021: INFO: csi-mockplugin-resizer-0 started at 2023-03-25 05:39:55 +0000 UTC (0+1 container statuses recorded) Mar 25 05:40:09.021: INFO: Container csi-resizer ready: true, restart count 0 Mar 25 05:40:09.021: INFO: kube-proxy-i-049fdb2c1cde894df started at 2023-03-25 05:23:35 +0000 UTC (0+1 container statuses recorded) Mar 25 05:40:09.021: INFO: Container kube-proxy ready: true, restart count 0 Mar 25 05:40:09.391: INFO: Latency metrics for node i-049fdb2c1cde894df Mar 25 05:40:09.391: INFO: Logging node info for node i-062679bb8c0ba4bce Mar 25 05:40:09.441: INFO: Node Info: &Node{ObjectMeta:{i-062679bb8c0ba4bce ee2da569-bb88-4297-9e6f-d2b15ba7ce86 14903 0 2023-03-25 05:27:31 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-1 failure-domain.beta.kubernetes.io/zone:us-west-1b kubernetes.io/arch:amd64 kubernetes.io/hostname:i-062679bb8c0ba4bce kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:us-west-1b topology.hostpath.csi/node:i-062679bb8c0ba4bce topology.kubernetes.io/region:us-west-1 topology.kubernetes.io/zone:us-west-1b] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-provisioning-2049":"i-062679bb8c0ba4bce","ebs.csi.aws.com":"i-062679bb8c0ba4bce"} node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:172.20.45.43/19 projectcalico.org/IPv4IPIPTunnelAddr:100.106.79.0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-03-25 05:27:31 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-25 05:27:31 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-03-25 05:27:31 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2023-03-25 05:27:31 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2023-03-25 05:27:44 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.7.0/24\"":{}}}} } {calico-node Update v1 2023-03-25 05:27:50 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4IPIPTunnelAddr":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2023-03-25 05:39:59 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-03-25 05:40:06 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.7.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-1b/i-062679bb8c0ba4bce,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.7.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051689472 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946831872 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-03-25 05:27:50 +0000 UTC,LastTransitionTime:2023-03-25 05:27:50 +0000 UTC,Reason:CalicoIsUp,Message:Calico is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-25 05:40:06 +0000 UTC,LastTransitionTime:2023-03-25 05:27:31 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-25 05:40:06 +0000 UTC,LastTransitionTime:2023-03-25 05:27:31 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-25 05:40:06 +0000 UTC,LastTransitionTime:2023-03-25 05:27:31 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-25 05:40:06 +0000 UTC,LastTransitionTime:2023-03-25 05:27:44 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.45.43,},NodeAddress{Type:ExternalIP,Address:3.101.133.108,},NodeAddress{Type:InternalDNS,Address:i-062679bb8c0ba4bce.us-west-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-062679bb8c0ba4bce.us-west-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-3-101-133-108.us-west-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2528f599e9b1e199793fb5cd06d205,SystemUUID:ec2528f5-99e9-b1e1-9979-3fb5cd06d205,BootID:ff56b70b-c8c5-46db-8cc5-4353c8f535c6,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.28.0-alpha.0,KubeProxyVersion:v1.28.0-alpha.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:112030336,},ContainerImage{Names:[docker.io/calico/cni@sha256:e282ea2914c806b5de2976330a17cfb5e6dcef47147bceb1432ca5c75fd46f50 docker.io/calico/cni:v3.24.5],SizeBytes:87458443,},ContainerImage{Names:[docker.io/calico/node@sha256:5972ad2bcbdc668564d3e26960c9c513b2d7b05581c704747cf7c62ef3a405a6 docker.io/calico/node:v3.24.5],SizeBytes:81584866,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nautilus@sha256:80ba6c8c44f9623f06e868a1aa66026c8ec438ad814f9ec95e9333b415fe3550 registry.k8s.io/e2e-test-images/nautilus:1.7],SizeBytes:49641698,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:e468dddcd275163a042ab297b2d8c2aca50d5e148d2d22f3b6ba119e2f31fa79 registry.k8s.io/sig-storage/csi-provisioner:v3.4.0],SizeBytes:27427836,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/kube-proxy@sha256:8a03b3147cd7be8e8235ae1cd70cf69aca9ce03ba1cfaffea320540bb5afa496 registry.k8s.io/kube-proxy:v1.28.0-alpha.0],SizeBytes:23896850,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:23881995,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18233005,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:15273057,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8688564,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6978614,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[kubernetes.io/csi/csi-hostpath-provisioning-2049^779f1e33-cacf-11ed-84c7-060c6d94390c],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-provisioning-2049^779f1e33-cacf-11ed-84c7-060c6d94390c,DevicePath:,},},Config:nil,},} Mar 25 05:40:09.441: INFO: Logging kubelet events for node i-062679bb8c0ba4bce Mar 25 05:40:09.516: INFO: Logging pods the kubelet thinks is on node i-062679bb8c0ba4bce Mar 25 05:40:09.592: INFO: kube-proxy-i-062679bb8c0ba4bce started at 2023-03-25 05:27:32 +0000 UTC (0+1 container statuses recorded) Mar 25 05:40:09.592: INFO: Container kube-proxy ready: true, restart count 0 Mar 25 05:40:09.592: INFO: ebs-csi-node-7twrq started at 2023-03-25 05:27:32 +0000 UTC (0+3 container statuses recorded) Mar 25 05:40:09.592: INFO: Container ebs-plugin ready: true, restart count 0 Mar 25 05:40:09.592: INFO: Container liveness-probe ready: true, restart count 0 Mar 25 05:40:09.592: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 25 05:40:09.592: INFO: test-rolling-update-with-lb-69fcddc54f-hvwj4 started at 2023-03-25 05:38:08 +0000 UTC (0+1 container statuses recorded) Mar 25 05:40:09.592: INFO: Container agnhost ready: true, restart count 0 Mar 25 05:40:09.592: INFO: calico-node-zwm5j started at 2023-03-25 05:27:32 +0000 UTC (3+1 container statuses recorded) Mar 25 05:40:09.592: INFO: Init container upgrade-ipam ready: true, restart count 0 Mar 25 05:40:09.592: INFO: Init container install-cni ready: true, restart count 0 Mar 25 05:40:09.592: INFO: Init container mount-bpffs ready: true, restart count 0 Mar 25 05:40:09.592: INFO: Container calico-node ready: true, restart count 0 Mar 25 05:40:09.592: INFO: startup-c479faf7-b2a9-4707-95f6-45b6b910a29d started at 2023-03-25 05:39:26 +0000 UTC (0+1 container statuses recorded) Mar 25 05:40:09.592: INFO: Container busybox ready: false, restart count 0 Mar 25 05:40:09.592: INFO: hostpath-injector started at 2023-03-25 05:39:58 +0000 UTC (0+1 container statuses recorded) Mar 25 05:40:09.592: INFO: Container hostpath-injector ready: true, restart count 0 Mar 25 05:40:09.592: INFO: hostexec-i-062679bb8c0ba4bce-f5j5f started at 2023-03-25 05:40:02 +0000 UTC (0+1 container statuses recorded) Mar 25 05:40:09.592: INFO: Container agnhost-container ready: true, restart count 0 Mar 25 05:40:09.592: INFO: pod-subpath-test-preprovisionedpv-hsk5 started at 2023-03-25 05:40:07 +0000 UTC (1+2 container statuses recorded) Mar 25 05:40:09.592: INFO: Init container test-init-subpath-preprovisionedpv-hsk5 ready: true, restart count 0 Mar 25 05:40:09.592: INFO: Container test-container-subpath-preprovisionedpv-hsk5 ready: false, restart count 0 Mar 25 05:40:09.592: INFO: Container test-container-volume-preprovisionedpv-hsk5 ready: false, restart count 0 Mar 25 05:40:09.592: INFO: csi-hostpathplugin-0 started at 2023-03-25 05:39:42 +0000 UTC (0+7 container statuses recorded) Mar 25 05:40:09.592: INFO: Container csi-attacher ready: true, restart count 0 Mar 25 05:40:09.592: INFO: Container csi-provisioner ready: true, restart count 0 Mar 25 05:40:09.592: INFO: Container csi-resizer ready: true, restart count 0 Mar 25 05:40:09.592: INFO: Container csi-snapshotter ready: true, restart count 0 Mar 25 05:40:09.592: INFO: Container hostpath ready: true, restart count 0 Mar 25 05:40:09.592: INFO: Container liveness-probe ready: true, restart count 0 Mar 25 05:40:09.592: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 25 05:40:09.592: INFO: coredns-559769c974-p9vg6 started at 2023-03-25 05:36:16 +0000 UTC (0+1 container statuses recorded) Mar 25 05:40:09.592: INFO: Container coredns ready: true, restart count 0 Mar 25 05:40:09.592: INFO: suspend-false-to-true-6f7v4 started at 2023-03-25 05:39:41 +0000 UTC (0+1 container statuses recorded) Mar 25 05:40:09.592: INFO: Container c ready: true, restart count 0 Mar 25 05:40:09.915: INFO: Latency metrics for node i-062679bb8c0ba4bce Mar 25 05:40:09.915: INFO: Logging node info for node i-072608201d2c0f1ad Mar 25 05:40:09.964: INFO: Node Info: &Node{ObjectMeta:{i-072608201d2c0f1ad 264dfba1-3cf0-4c99-8e8c-098f0f792077 14440 0 2023-03-25 05:35:29 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-1 failure-domain.beta.kubernetes.io/zone:us-west-1b kubernetes.io/arch:amd64 kubernetes.io/hostname:i-072608201d2c0f1ad kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:us-west-1b topology.hostpath.csi/node:i-072608201d2c0f1ad topology.kubernetes.io/region:us-west-1 topology.kubernetes.io/zone:us-west-1b] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-072608201d2c0f1ad"} node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:172.20.38.125/19 projectcalico.org/IPv4IPIPTunnelAddr:100.107.220.192 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-03-25 05:35:29 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-25 05:35:29 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-03-25 05:35:29 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2023-03-25 05:35:29 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2023-03-25 05:35:41 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.9.0/24\"":{}}}} } {calico-node Update v1 2023-03-25 05:35:48 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4IPIPTunnelAddr":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2023-03-25 05:39:53 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-03-25 05:39:55 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.9.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-1b/i-072608201d2c0f1ad,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.9.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051689472 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946831872 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-03-25 05:35:47 +0000 UTC,LastTransitionTime:2023-03-25 05:35:47 +0000 UTC,Reason:CalicoIsUp,Message:Calico is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-25 05:39:55 +0000 UTC,LastTransitionTime:2023-03-25 05:35:29 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-25 05:39:55 +0000 UTC,LastTransitionTime:2023-03-25 05:35:29 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-25 05:39:55 +0000 UTC,LastTransitionTime:2023-03-25 05:35:29 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-25 05:39:55 +0000 UTC,LastTransitionTime:2023-03-25 05:35:41 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.38.125,},NodeAddress{Type:ExternalIP,Address:3.101.88.10,},NodeAddress{Type:InternalDNS,Address:i-072608201d2c0f1ad.us-west-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-072608201d2c0f1ad.us-west-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-3-101-88-10.us-west-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2d3818c4f80a253dcc1a60e148dbc8,SystemUUID:ec2d3818-c4f8-0a25-3dcc-1a60e148dbc8,BootID:77d046cf-75ea-4a8c-970a-3190d64e019b,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.28.0-alpha.0,KubeProxyVersion:v1.28.0-alpha.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[docker.io/calico/cni@sha256:e282ea2914c806b5de2976330a17cfb5e6dcef47147bceb1432ca5c75fd46f50 docker.io/calico/cni:v3.24.5],SizeBytes:87458443,},ContainerImage{Names:[docker.io/calico/node@sha256:5972ad2bcbdc668564d3e26960c9c513b2d7b05581c704747cf7c62ef3a405a6 docker.io/calico/node:v3.24.5],SizeBytes:81584866,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:e468dddcd275163a042ab297b2d8c2aca50d5e148d2d22f3b6ba119e2f31fa79 registry.k8s.io/sig-storage/csi-provisioner:v3.4.0],SizeBytes:27427836,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/kube-proxy@sha256:8a03b3147cd7be8e8235ae1cd70cf69aca9ce03ba1cfaffea320540bb5afa496 registry.k8s.io/kube-proxy:v1.28.0-alpha.0],SizeBytes:23896850,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:23881995,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18758628,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18233005,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8688564,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6978614,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[kubernetes.io/csi/ebs.csi.aws.com^vol-0a8712c0b538ed724],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-0a8712c0b538ed724,DevicePath:,},},Config:nil,},} Mar 25 05:40:09.964: INFO: Logging kubelet events for node i-072608201d2c0f1ad Mar 25 05:40:10.020: INFO: Logging pods the kubelet thinks is on node i-072608201d2c0f1ad Mar 25 05:40:10.075: INFO: hostexec-i-072608201d2c0f1ad-gjdks started at 2023-03-25 05:40:09 +0000 UTC (0+1 container statuses recorded) Mar 25 05:40:10.075: INFO: Container agnhost-container ready: false, restart count 0 Mar 25 05:40:10.075: INFO: kube-proxy-i-072608201d2c0f1ad started at 2023-03-25 05:35:30 +0000 UTC (0+1 container statuses recorded) Mar 25 05:40:10.075: INFO: Container kube-proxy ready: true, restart count 0 Mar 25 05:40:10.075: INFO: test-rolling-update-with-lb-69fcddc54f-gsptt started at 2023-03-25 05:38:08 +0000 UTC (0+1 container statuses recorded) Mar 25 05:40:10.075: INFO: Container agnhost ready: true, restart count 0 Mar 25 05:40:10.075: INFO: inline-volume-tester-62x94 started at 2023-03-25 05:39:51 +0000 UTC (0+1 container statuses recorded) Mar 25 05:40:10.075: INFO: Container csi-volume-tester ready: true, restart count 0 Mar 25 05:40:10.075: INFO: webserver-pod started at 2023-03-25 05:39:46 +0000 UTC (0+1 container statuses recorded) Mar 25 05:40:10.075: INFO: Container agnhost ready: true, restart count 0 Mar 25 05:40:10.075: INFO: ebs-csi-node-r7mcm started at 2023-03-25 05:35:30 +0000 UTC (0+3 container statuses recorded) Mar 25 05:40:10.075: INFO: Container ebs-plugin ready: true, restart count 0 Mar 25 05:40:10.075: INFO: Container liveness-probe ready: true, restart count 0 Mar 25 05:40:10.075: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 25 05:40:10.075: INFO: hostexec-i-072608201d2c0f1ad-vsl2c started at 2023-03-25 05:39:53 +0000 UTC (0+1 container statuses recorded) Mar 25 05:40:10.075: INFO: Container agnhost-container ready: true, restart count 0 Mar 25 05:40:10.075: INFO: deployment-shared-map-item-removal-5c89cddd56-d4dfm started at 2023-03-25 05:40:08 +0000 UTC (0+1 container statuses recorded) Mar 25 05:40:10.075: INFO: Container nginx ready: false, restart count 0 Mar 25 05:40:10.075: INFO: pod-handle-http-request started at 2023-03-25 05:37:59 +0000 UTC (0+2 container statuses recorded) Mar 25 05:40:10.075: INFO: Container container-handle-http-request ready: true, restart count 0 Mar 25 05:40:10.075: INFO: Container container-handle-https-request ready: true, restart count 0 Mar 25 05:40:10.075: INFO: calico-node-lc9zl started at 2023-03-25 05:35:30 +0000 UTC (3+1 container statuses recorded) Mar 25 05:40:10.075: INFO: Init container upgrade-ipam ready: true, restart count 0 Mar 25 05:40:10.075: INFO: Init container install-cni ready: true, restart count 0 Mar 25 05:40:10.075: INFO: Init container mount-bpffs ready: true, restart count 0 Mar 25 05:40:10.075: INFO: Container calico-node ready: true, restart count 0 Mar 25 05:40:10.075: INFO: pod-39c0f692-731a-40a6-abc0-a55dbe2183c3 started at 2023-03-25 05:40:06 +0000 UTC (0+1 container statuses recorded) Mar 25 05:40:10.075: INFO: Container write-pod ready: true, restart count 0 Mar 25 05:40:10.075: INFO: pod-projected-configmaps-61878de4-bfbc-4c4c-bb90-4daf9441ed77 started at 2023-03-25 05:39:21 +0000 UTC (0+3 container statuses recorded) Mar 25 05:40:10.075: INFO: Container createcm-volume-test ready: true, restart count 0 Mar 25 05:40:10.075: INFO: Container delcm-volume-test ready: true, restart count 0 Mar 25 05:40:10.075: INFO: Container updcm-volume-test ready: true, restart count 0 Mar 25 05:40:10.367: INFO: Latency metrics for node i-072608201d2c0f1ad END STEP: dump namespace information after failure - test/e2e/framework/framework.go:288 @ 03/25/23 05:40:10.367 (2.86s) < Exit [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] volumes - dump namespaces | framework.go:209 @ 03/25/23 05:40:10.367 (2.86s) > Enter [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] volumes - tear down framework | framework.go:206 @ 03/25/23 05:40:10.367 STEP: Destroying namespace "volume-4037" for this suite. - test/e2e/framework/framework.go:351 @ 03/25/23 05:40:10.367 < Exit [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] volumes - tear down framework | framework.go:206 @ 03/25/23 05:40:10.429 (62ms) > Enter [ReportAfterEach] TOP-LEVEL - test/e2e/e2e_test.go:144 @ 03/25/23 05:40:10.429 < Exit [ReportAfterEach] TOP-LEVEL - test/e2e/e2e_test.go:144 @ 03/25/23 05:40:10.429 (0s)
Filter through log files | View test history on testgrid
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[It\]\sExternal\sStorage\s\[Driver\:\sebs\.csi\.aws\.com\]\s\[Testpattern\:\sDynamic\sPV\s\(ext4\)\]\svolumes\sshould\sallow\sexec\sof\sfiles\son\sthe\svolume$'
[FAILED] failed to get logs from exec-volume-test-dynamicpv-gwfg for exec-container-dynamicpv-gwfg: Get "https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/volume-8318/pods/exec-volume-test-dynamicpv-gwfg/log?container=exec-container-dynamicpv-gwfg&previous=false": stream error: stream ID 2009; INTERNAL_ERROR; received from peer In [It] at: test/e2e/framework/pod/output/output.go:247 @ 03/25/23 05:45:29.05from junit_01.xml
> Enter [BeforeEach] [Testpattern: Dynamic PV (ext4)] volumes - test/e2e/storage/framework/testsuite.go:51 @ 03/25/23 05:45:15.853 < Exit [BeforeEach] [Testpattern: Dynamic PV (ext4)] volumes - test/e2e/storage/framework/testsuite.go:51 @ 03/25/23 05:45:15.853 (0s) > Enter [BeforeEach] [Testpattern: Dynamic PV (ext4)] volumes - set up framework | framework.go:191 @ 03/25/23 05:45:15.853 STEP: Creating a kubernetes client - test/e2e/framework/framework.go:211 @ 03/25/23 05:45:15.853 Mar 25 05:45:15.853: INFO: >>> kubeConfig: /root/.kube/config STEP: Building a namespace api object, basename volume - test/e2e/framework/framework.go:250 @ 03/25/23 05:45:15.854 STEP: Waiting for a default service account to be provisioned in namespace - test/e2e/framework/framework.go:259 @ 03/25/23 05:45:16.032 STEP: Waiting for kube-root-ca.crt to be provisioned in namespace - test/e2e/framework/framework.go:262 @ 03/25/23 05:45:16.128 < Exit [BeforeEach] [Testpattern: Dynamic PV (ext4)] volumes - set up framework | framework.go:191 @ 03/25/23 05:45:16.227 (375ms) > Enter [BeforeEach] [Testpattern: Dynamic PV (ext4)] volumes - test/e2e/framework/metrics/init/init.go:33 @ 03/25/23 05:45:16.227 < Exit [BeforeEach] [Testpattern: Dynamic PV (ext4)] volumes - test/e2e/framework/metrics/init/init.go:33 @ 03/25/23 05:45:16.227 (0s) > Enter [It] should allow exec of files on the volume - test/e2e/storage/testsuites/volumes.go:194 @ 03/25/23 05:45:16.227 Mar 25 05:45:16.227: INFO: Creating resource for dynamic PV Mar 25 05:45:16.228: INFO: Using claimSize:1Gi, test suite supported size:{ 1Mi}, driver(ebs.csi.aws.com) supported size:{ 1Mi} STEP: creating a StorageClass volume-8318-e2e-schjlrj - test/e2e/storage/framework/volume_resource.go:102 @ 03/25/23 05:45:16.228 STEP: creating a claim - test/e2e/storage/framework/volume_resource.go:294 @ 03/25/23 05:45:16.282 Mar 25 05:45:16.282: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil STEP: Creating pod exec-volume-test-dynamicpv-gwfg - test/e2e/storage/testsuites/volumes.go:253 @ 03/25/23 05:45:16.392 STEP: Creating a pod to test exec-volume-test - test/e2e/framework/pod/output/output.go:243 @ 03/25/23 05:45:16.392 STEP: Saw pod success - test/e2e/framework/pod/wait.go:409 @ 03/25/23 05:45:28.841 Mar 25 05:45:28.889: INFO: Trying to get logs from node i-049fdb2c1cde894df pod exec-volume-test-dynamicpv-gwfg container exec-container-dynamicpv-gwfg: <nil> Mar 25 05:45:28.938: INFO: Failed to get logs from node "i-049fdb2c1cde894df" pod "exec-volume-test-dynamicpv-gwfg" container "exec-container-dynamicpv-gwfg". Get "https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/volume-8318/pods/exec-volume-test-dynamicpv-gwfg/log?container=exec-container-dynamicpv-gwfg&previous=false": stream error: stream ID 2009; INTERNAL_ERROR; received from peer STEP: delete the pod - test/e2e/framework/pod/output/output.go:168 @ 03/25/23 05:45:28.938 Mar 25 05:45:29.050: INFO: Unexpected error: <*fmt.wrapError | 0xc00502ca80>: failed to get logs from exec-volume-test-dynamicpv-gwfg for exec-container-dynamicpv-gwfg: Get "https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/volume-8318/pods/exec-volume-test-dynamicpv-gwfg/log?container=exec-container-dynamicpv-gwfg&previous=false": stream error: stream ID 2009; INTERNAL_ERROR; received from peer { msg: "failed to get logs from exec-volume-test-dynamicpv-gwfg for exec-container-dynamicpv-gwfg: Get \"https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/volume-8318/pods/exec-volume-test-dynamicpv-gwfg/log?container=exec-container-dynamicpv-gwfg&previous=false\": stream error: stream ID 2009; INTERNAL_ERROR; received from peer", err: <*url.Error | 0xc0040c2360>{ Op: "Get", URL: "https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/volume-8318/pods/exec-volume-test-dynamicpv-gwfg/log?container=exec-container-dynamicpv-gwfg&previous=false", Err: <http2.StreamError>{ StreamID: 2009, Code: 2, Cause: <*errors.errorString | 0xc0001b3670>{ s: "received from peer", }, }, }, } [FAILED] failed to get logs from exec-volume-test-dynamicpv-gwfg for exec-container-dynamicpv-gwfg: Get "https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/volume-8318/pods/exec-volume-test-dynamicpv-gwfg/log?container=exec-container-dynamicpv-gwfg&previous=false": stream error: stream ID 2009; INTERNAL_ERROR; received from peer In [It] at: test/e2e/framework/pod/output/output.go:247 @ 03/25/23 05:45:29.05 < Exit [It] should allow exec of files on the volume - test/e2e/storage/testsuites/volumes.go:194 @ 03/25/23 05:45:29.05 (12.822s) > Enter [AfterEach] [Testpattern: Dynamic PV (ext4)] volumes - test/e2e/framework/node/init/init.go:33 @ 03/25/23 05:45:29.05 Mar 25 05:45:29.050: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready < Exit [AfterEach] [Testpattern: Dynamic PV (ext4)] volumes - test/e2e/framework/node/init/init.go:33 @ 03/25/23 05:45:29.099 (50ms) > Enter [DeferCleanup (Each)] [Testpattern: Dynamic PV (ext4)] volumes - test/e2e/storage/testsuites/volumes.go:197 @ 03/25/23 05:45:29.099 STEP: Deleting pvc - test/e2e/storage/framework/volume_resource.go:181 @ 03/25/23 05:45:29.099 Mar 25 05:45:29.197: INFO: Deleting PersistentVolumeClaim "ebs.csi.aws.comzjpm6" Mar 25 05:45:29.248: INFO: Waiting up to 5m0s for PersistentVolume pvc-ab680cbd-899c-4c09-a73e-61811dfae5c0 to get deleted Mar 25 05:45:29.296: INFO: PersistentVolume pvc-ab680cbd-899c-4c09-a73e-61811dfae5c0 found and phase=Released (48.013233ms) Mar 25 05:45:34.345: INFO: PersistentVolume pvc-ab680cbd-899c-4c09-a73e-61811dfae5c0 found and phase=Released (5.097101836s) Mar 25 05:45:39.394: INFO: PersistentVolume pvc-ab680cbd-899c-4c09-a73e-61811dfae5c0 found and phase=Released (10.146605813s) Mar 25 05:45:44.444: INFO: PersistentVolume pvc-ab680cbd-899c-4c09-a73e-61811dfae5c0 was removed STEP: Deleting sc - test/e2e/storage/framework/volume_resource.go:228 @ 03/25/23 05:45:44.444 < Exit [DeferCleanup (Each)] [Testpattern: Dynamic PV (ext4)] volumes - test/e2e/storage/testsuites/volumes.go:197 @ 03/25/23 05:45:44.495 (15.395s) > Enter [DeferCleanup (Each)] [Testpattern: Dynamic PV (ext4)] volumes - test/e2e/framework/metrics/init/init.go:35 @ 03/25/23 05:45:44.495 < Exit [DeferCleanup (Each)] [Testpattern: Dynamic PV (ext4)] volumes - test/e2e/framework/metrics/init/init.go:35 @ 03/25/23 05:45:44.495 (0s) > Enter [DeferCleanup (Each)] [Testpattern: Dynamic PV (ext4)] volumes - dump namespaces | framework.go:209 @ 03/25/23 05:45:44.495 STEP: dump namespace information after failure - test/e2e/framework/framework.go:288 @ 03/25/23 05:45:44.495 STEP: Collecting events from namespace "volume-8318". - test/e2e/framework/debug/dump.go:42 @ 03/25/23 05:45:44.495 STEP: Found 9 events. - test/e2e/framework/debug/dump.go:46 @ 03/25/23 05:45:44.543 Mar 25 05:45:44.543: INFO: At 2023-03-25 05:45:16 +0000 UTC - event for ebs.csi.aws.comzjpm6: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding Mar 25 05:45:44.543: INFO: At 2023-03-25 05:45:16 +0000 UTC - event for ebs.csi.aws.comzjpm6: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator Mar 25 05:45:44.543: INFO: At 2023-03-25 05:45:16 +0000 UTC - event for ebs.csi.aws.comzjpm6: {ebs.csi.aws.com_i-0168f82458996cc53_0b589c46-3396-4499-94ba-eb51a48f7a58 } Provisioning: External provisioner is provisioning volume for claim "volume-8318/ebs.csi.aws.comzjpm6" Mar 25 05:45:44.543: INFO: At 2023-03-25 05:45:20 +0000 UTC - event for ebs.csi.aws.comzjpm6: {ebs.csi.aws.com_i-0168f82458996cc53_0b589c46-3396-4499-94ba-eb51a48f7a58 } ProvisioningSucceeded: Successfully provisioned volume pvc-ab680cbd-899c-4c09-a73e-61811dfae5c0 Mar 25 05:45:44.543: INFO: At 2023-03-25 05:45:20 +0000 UTC - event for exec-volume-test-dynamicpv-gwfg: {default-scheduler } Scheduled: Successfully assigned volume-8318/exec-volume-test-dynamicpv-gwfg to i-049fdb2c1cde894df Mar 25 05:45:44.543: INFO: At 2023-03-25 05:45:24 +0000 UTC - event for exec-volume-test-dynamicpv-gwfg: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-ab680cbd-899c-4c09-a73e-61811dfae5c0" Mar 25 05:45:44.543: INFO: At 2023-03-25 05:45:25 +0000 UTC - event for exec-volume-test-dynamicpv-gwfg: {kubelet i-049fdb2c1cde894df} Pulled: Container image "registry.k8s.io/e2e-test-images/nginx:1.14-4" already present on machine Mar 25 05:45:44.543: INFO: At 2023-03-25 05:45:25 +0000 UTC - event for exec-volume-test-dynamicpv-gwfg: {kubelet i-049fdb2c1cde894df} Created: Created container exec-container-dynamicpv-gwfg Mar 25 05:45:44.543: INFO: At 2023-03-25 05:45:25 +0000 UTC - event for exec-volume-test-dynamicpv-gwfg: {kubelet i-049fdb2c1cde894df} Started: Started container exec-container-dynamicpv-gwfg Mar 25 05:45:44.592: INFO: POD NODE PHASE GRACE CONDITIONS Mar 25 05:45:44.592: INFO: Mar 25 05:45:44.641: INFO: Logging node info for node i-0168f82458996cc53 Mar 25 05:45:44.689: INFO: Node Info: &Node{ObjectMeta:{i-0168f82458996cc53 f4eb302c-c477-45ac-bcac-ffed65f6580f 21197 0 2023-03-25 05:18:57 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-1 failure-domain.beta.kubernetes.io/zone:us-west-1b kops.k8s.io/kops-controller-pki: kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0168f82458996cc53 kubernetes.io/os:linux node-role.kubernetes.io/control-plane: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:us-west-1b topology.kubernetes.io/region:us-west-1 topology.kubernetes.io/zone:us-west-1b] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0168f82458996cc53"} node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:172.20.48.81/19 projectcalico.org/IPv4IPIPTunnelAddr:100.100.215.192 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-03-25 05:18:57 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {protokube Update v1 2023-03-25 05:18:58 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {calico-node Update v1 2023-03-25 05:20:22 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4IPIPTunnelAddr":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {aws-cloud-controller-manager Update v1 2023-03-25 05:20:26 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-25 05:20:26 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2023-03-25 05:20:31 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.5.0/24\"":{}},"f:taints":{}}} } {kubelet Update v1 2023-03-25 05:42:05 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.5.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-1b/i-0168f82458996cc53,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/control-plane,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.5.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3892310016 0} {<nil>} 3801084Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3787452416 0} {<nil>} 3698684Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-03-25 05:20:22 +0000 UTC,LastTransitionTime:2023-03-25 05:20:22 +0000 UTC,Reason:CalicoIsUp,Message:Calico is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-25 05:42:05 +0000 UTC,LastTransitionTime:2023-03-25 05:18:50 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-25 05:42:05 +0000 UTC,LastTransitionTime:2023-03-25 05:18:50 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-25 05:42:05 +0000 UTC,LastTransitionTime:2023-03-25 05:18:50 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-25 05:42:05 +0000 UTC,LastTransitionTime:2023-03-25 05:20:29 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.48.81,},NodeAddress{Type:ExternalIP,Address:3.101.135.77,},NodeAddress{Type:InternalDNS,Address:i-0168f82458996cc53.us-west-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-0168f82458996cc53.us-west-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-3-101-135-77.us-west-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2d1ccc7ebf313420bd1f530a94fbfd,SystemUUID:ec2d1ccc-7ebf-3134-20bd-1f530a94fbfd,BootID:15cee321-56e7-4290-ab48-47b60b0bc657,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.28.0-alpha.0,KubeProxyVersion:v1.28.0-alpha.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcd@sha256:dd75ec974b0a2a6f6bb47001ba09207976e625db898d1b16735528c009cb171c registry.k8s.io/etcd:3.5.6-0],SizeBytes:102542580,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:6f72b851544986cb0921b53ea655ec04c36131248f16d4ad110cb3ca0c369dc1 registry.k8s.io/etcd:3.5.4-0],SizeBytes:102157811,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:13f53ed1d91e2e11aac476ee9a0269fdda6cc4874eba903efd40daf50c55eee5 registry.k8s.io/etcd:3.5.3-0],SizeBytes:102143581,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:51eae8381dcb1078289fa7b4f3df2630cdc18d09fb56f8e56b41c40e191d6c83 registry.k8s.io/etcd:3.5.7-0],SizeBytes:101639218,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:4afb99b4690b418ffc2ceb67e1a17376457e441c1f09ab55447f0aaf992fa646 registry.k8s.io/etcd:3.4.3-0],SizeBytes:100947667,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:9ce33ba33d8e738a5b85ed50b5080ac746deceed4a7496c550927a7a19ca3b6d registry.k8s.io/etcd:3.5.0-0],SizeBytes:99868722,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:64b9ea357325d5db9f8a723dcf503b5a449177b17ac87d69481e126bb724c263 registry.k8s.io/etcd:3.5.1-0],SizeBytes:98888614,},ContainerImage{Names:[docker.io/calico/cni@sha256:e282ea2914c806b5de2976330a17cfb5e6dcef47147bceb1432ca5c75fd46f50 docker.io/calico/cni:v3.24.5],SizeBytes:87458443,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:4ad90a11b55313b182afc186b9876c8e891531b8db4c9bf1541953021618d0e2 registry.k8s.io/etcd:3.4.13-0],SizeBytes:86742272,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:0d2ab6b15b4fe0b903b9e81192d942fb39c29be61fe341cd16db5d0165b0435c registry.k8s.io/etcd:3.3.17-0],SizeBytes:85524408,},ContainerImage{Names:[docker.io/calico/node@sha256:5972ad2bcbdc668564d3e26960c9c513b2d7b05581c704747cf7c62ef3a405a6 docker.io/calico/node:v3.24.5],SizeBytes:81584866,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:17da501f5d2a675be46040422a27b7cc21b8a43895ac998b171db1c346f361f7 registry.k8s.io/etcd:3.3.10-0],SizeBytes:76162358,},ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager-slim@sha256:8727adc059334e0e3329dbccce116a734da4a090da5e86c8cddccaf333caa666 registry.k8s.io/etcdadm/etcd-manager-slim:v3.0.20230201],SizeBytes:63926165,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:905d7ca17fd02bc24c0eba9a062753aba15db3e31422390bc3238eb762339b20 registry.k8s.io/etcd:3.2.24-1],SizeBytes:63423287,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.27.0-alpha.1],SizeBytes:45219545,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.27.0-alpha.1],SizeBytes:43628585,},ContainerImage{Names:[registry.k8s.io/kube-apiserver@sha256:49934a29c559e79ed19fc38b4697b18d8fe174b17bf56192588575f980c93956 registry.k8s.io/kube-apiserver:v1.28.0-alpha.0],SizeBytes:33212506,},ContainerImage{Names:[docker.io/calico/kube-controllers@sha256:2b6acd7f677f76ffe12ecf3ea7df92eb9b1bdb07336d1ac2a54c7631fb753f7e docker.io/calico/kube-controllers:v3.24.5],SizeBytes:31137169,},ContainerImage{Names:[registry.k8s.io/kube-controller-manager@sha256:974748384634982c73527fdb3c9ec0ec476bc0d52b3dd1d611a380b7f99af1f3 registry.k8s.io/kube-controller-manager:v1.28.0-alpha.0],SizeBytes:30820280,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/kube-proxy@sha256:8a03b3147cd7be8e8235ae1cd70cf69aca9ce03ba1cfaffea320540bb5afa496 registry.k8s.io/kube-proxy:v1.28.0-alpha.0],SizeBytes:23896850,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:23345856,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:22381475,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:22085298,},ContainerImage{Names:[registry.k8s.io/provider-aws/cloud-controller-manager@sha256:fdeb61e3e42ecd9cca868d550ebdb88dd6341d9e91fcfa9a37e227dab2ad22cb registry.k8s.io/provider-aws/cloud-controller-manager:v1.26.0],SizeBytes:20154862,},ContainerImage{Names:[registry.k8s.io/kube-scheduler@sha256:cbf4007abb076188f16b975bc2b74452603cc7b34c60a29d92f14c38ad02dc11 registry.k8s.io/kube-scheduler:v1.28.0-alpha.0],SizeBytes:18078674,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.27.0-alpha.1],SizeBytes:5074298,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Mar 25 05:45:44.689: INFO: Logging kubelet events for node i-0168f82458996cc53 Mar 25 05:45:44.747: INFO: Logging pods the kubelet thinks is on node i-0168f82458996cc53 Mar 25 05:45:44.803: INFO: kube-apiserver-i-0168f82458996cc53 started at 2023-03-25 05:17:36 +0000 UTC (0+2 container statuses recorded) Mar 25 05:45:44.803: INFO: Container healthcheck ready: true, restart count 0 Mar 25 05:45:44.803: INFO: Container kube-apiserver ready: true, restart count 2 Mar 25 05:45:44.803: INFO: calico-node-rdjq2 started at 2023-03-25 05:20:06 +0000 UTC (3+1 container statuses recorded) Mar 25 05:45:44.803: INFO: Init container upgrade-ipam ready: true, restart count 0 Mar 25 05:45:44.803: INFO: Init container install-cni ready: true, restart count 0 Mar 25 05:45:44.803: INFO: Init container mount-bpffs ready: true, restart count 0 Mar 25 05:45:44.803: INFO: Container calico-node ready: true, restart count 0 Mar 25 05:45:44.803: INFO: kube-controller-manager-i-0168f82458996cc53 started at 2023-03-25 05:17:36 +0000 UTC (0+1 container statuses recorded) Mar 25 05:45:44.803: INFO: Container kube-controller-manager ready: true, restart count 4 Mar 25 05:45:44.803: INFO: etcd-manager-events-i-0168f82458996cc53 started at 2023-03-25 05:17:36 +0000 UTC (11+1 container statuses recorded) Mar 25 05:45:44.803: INFO: Init container init-etcd-3-2-24 ready: true, restart count 0 Mar 25 05:45:44.803: INFO: Init container init-etcd-3-3-10 ready: true, restart count 0 Mar 25 05:45:44.803: INFO: Init container init-etcd-3-3-17 ready: true, restart count 0 Mar 25 05:45:44.803: INFO: Init container init-etcd-3-4-13 ready: true, restart count 0 Mar 25 05:45:44.803: INFO: Init container init-etcd-3-4-3 ready: true, restart count 0 Mar 25 05:45:44.803: INFO: Init container init-etcd-3-5-0 ready: true, restart count 0 Mar 25 05:45:44.803: INFO: Init container init-etcd-3-5-1 ready: true, restart count 0 Mar 25 05:45:44.803: INFO: Init container init-etcd-3-5-3 ready: true, restart count 0 Mar 25 05:45:44.803: INFO: Init container init-etcd-3-5-4 ready: true, restart count 0 Mar 25 05:45:44.803: INFO: Init container init-etcd-3-5-6 ready: true, restart count 0 Mar 25 05:45:44.803: INFO: Init container init-etcd-3-5-7 ready: true, restart count 0 Mar 25 05:45:44.803: INFO: Container etcd-manager ready: true, restart count 0 Mar 25 05:45:44.803: INFO: dns-controller-69987775c6-hbn69 started at 2023-03-25 05:19:47 +0000 UTC (0+1 container statuses recorded) Mar 25 05:45:44.803: INFO: Container dns-controller ready: true, restart count 0 Mar 25 05:45:44.803: INFO: aws-cloud-controller-manager-26vbv started at 2023-03-25 05:20:06 +0000 UTC (0+1 container statuses recorded) Mar 25 05:45:44.803: INFO: Container aws-cloud-controller-manager ready: true, restart count 0 Mar 25 05:45:44.803: INFO: ebs-csi-node-zjtj6 started at 2023-03-25 05:20:06 +0000 UTC (0+3 container statuses recorded) Mar 25 05:45:44.803: INFO: Container ebs-plugin ready: true, restart count 0 Mar 25 05:45:44.803: INFO: Container liveness-probe ready: true, restart count 0 Mar 25 05:45:44.803: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 25 05:45:44.803: INFO: kops-controller-cgw95 started at 2023-03-25 05:20:06 +0000 UTC (0+1 container statuses recorded) Mar 25 05:45:44.803: INFO: Container kops-controller ready: true, restart count 0 Mar 25 05:45:44.803: INFO: calico-kube-controllers-55f488656-rr8rz started at 2023-03-25 05:20:29 +0000 UTC (0+1 container statuses recorded) Mar 25 05:45:44.803: INFO: Container calico-kube-controllers ready: true, restart count 0 Mar 25 05:45:44.803: INFO: kube-proxy-i-0168f82458996cc53 started at 2023-03-25 05:17:36 +0000 UTC (0+1 container statuses recorded) Mar 25 05:45:44.803: INFO: Container kube-proxy ready: true, restart count 0 Mar 25 05:45:44.803: INFO: kube-scheduler-i-0168f82458996cc53 started at 2023-03-25 05:17:36 +0000 UTC (0+1 container statuses recorded) Mar 25 05:45:44.803: INFO: Container kube-scheduler ready: true, restart count 0 Mar 25 05:45:44.803: INFO: etcd-manager-main-i-0168f82458996cc53 started at 2023-03-25 05:17:36 +0000 UTC (11+1 container statuses recorded) Mar 25 05:45:44.803: INFO: Init container init-etcd-3-2-24 ready: true, restart count 0 Mar 25 05:45:44.803: INFO: Init container init-etcd-3-3-10 ready: true, restart count 0 Mar 25 05:45:44.803: INFO: Init container init-etcd-3-3-17 ready: true, restart count 0 Mar 25 05:45:44.803: INFO: Init container init-etcd-3-4-13 ready: true, restart count 0 Mar 25 05:45:44.803: INFO: Init container init-etcd-3-4-3 ready: true, restart count 0 Mar 25 05:45:44.803: INFO: Init container init-etcd-3-5-0 ready: true, restart count 0 Mar 25 05:45:44.803: INFO: Init container init-etcd-3-5-1 ready: true, restart count 0 Mar 25 05:45:44.803: INFO: Init container init-etcd-3-5-3 ready: true, restart count 0 Mar 25 05:45:44.803: INFO: Init container init-etcd-3-5-4 ready: true, restart count 0 Mar 25 05:45:44.803: INFO: Init container init-etcd-3-5-6 ready: true, restart count 0 Mar 25 05:45:44.803: INFO: Init container init-etcd-3-5-7 ready: true, restart count 0 Mar 25 05:45:44.803: INFO: Container etcd-manager ready: true, restart count 0 Mar 25 05:45:44.803: INFO: ebs-csi-controller-79f984545f-9q5ks started at 2023-03-25 05:21:26 +0000 UTC (0+5 container statuses recorded) Mar 25 05:45:44.803: INFO: Container csi-attacher ready: true, restart count 0 Mar 25 05:45:44.803: INFO: Container csi-provisioner ready: true, restart count 0 Mar 25 05:45:44.803: INFO: Container csi-resizer ready: true, restart count 0 Mar 25 05:45:44.803: INFO: Container ebs-plugin ready: true, restart count 0 Mar 25 05:45:44.803: INFO: Container liveness-probe ready: true, restart count 0 Mar 25 05:45:45.117: INFO: Latency metrics for node i-0168f82458996cc53 Mar 25 05:45:45.117: INFO: Logging node info for node i-03d301e6f030748f1 Mar 25 05:45:45.167: INFO: Node Info: &Node{ObjectMeta:{i-03d301e6f030748f1 897f52bd-65e2-4107-843a-37e36aa74032 31956 0 2023-03-25 05:31:35 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-1 failure-domain.beta.kubernetes.io/zone:us-west-1b kubernetes.io/arch:amd64 kubernetes.io/hostname:i-03d301e6f030748f1 kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:us-west-1b topology.hostpath.csi/node:i-03d301e6f030748f1 topology.kubernetes.io/region:us-west-1 topology.kubernetes.io/zone:us-west-1b] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-03d301e6f030748f1"} node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:172.20.50.199/19 projectcalico.org/IPv4IPIPTunnelAddr:100.110.167.64 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-03-25 05:31:35 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-25 05:31:35 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-03-25 05:31:35 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2023-03-25 05:31:35 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2023-03-25 05:31:48 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.8.0/24\"":{}}}} } {calico-node Update v1 2023-03-25 05:31:56 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4IPIPTunnelAddr":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2023-03-25 05:45:12 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-03-25 05:45:12 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.8.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-1b/i-03d301e6f030748f1,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.8.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-03-25 05:31:56 +0000 UTC,LastTransitionTime:2023-03-25 05:31:56 +0000 UTC,Reason:CalicoIsUp,Message:Calico is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-25 05:45:12 +0000 UTC,LastTransitionTime:2023-03-25 05:31:35 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-25 05:45:12 +0000 UTC,LastTransitionTime:2023-03-25 05:31:35 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-25 05:45:12 +0000 UTC,LastTransitionTime:2023-03-25 05:31:35 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-25 05:45:12 +0000 UTC,LastTransitionTime:2023-03-25 05:31:48 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.50.199,},NodeAddress{Type:ExternalIP,Address:52.53.245.63,},NodeAddress{Type:InternalDNS,Address:i-03d301e6f030748f1.us-west-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-03d301e6f030748f1.us-west-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-52-53-245-63.us-west-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec27152f7af9b9bfd1355ee19c256086,SystemUUID:ec27152f-7af9-b9bf-d135-5ee19c256086,BootID:1c303852-f2ce-45f9-ab14-6e8de97a0a3e,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.28.0-alpha.0,KubeProxyVersion:v1.28.0-alpha.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[docker.io/calico/cni@sha256:e282ea2914c806b5de2976330a17cfb5e6dcef47147bceb1432ca5c75fd46f50 docker.io/calico/cni:v3.24.5],SizeBytes:87458443,},ContainerImage{Names:[docker.io/calico/node@sha256:5972ad2bcbdc668564d3e26960c9c513b2d7b05581c704747cf7c62ef3a405a6 docker.io/calico/node:v3.24.5],SizeBytes:81584866,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nautilus@sha256:80ba6c8c44f9623f06e868a1aa66026c8ec438ad814f9ec95e9333b415fe3550 registry.k8s.io/e2e-test-images/nautilus:1.7],SizeBytes:49641698,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41901587,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:e468dddcd275163a042ab297b2d8c2aca50d5e148d2d22f3b6ba119e2f31fa79 registry.k8s.io/sig-storage/csi-provisioner:v3.4.0],SizeBytes:27427836,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/kube-proxy@sha256:8a03b3147cd7be8e8235ae1cd70cf69aca9ce03ba1cfaffea320540bb5afa496 registry.k8s.io/kube-proxy:v1.28.0-alpha.0],SizeBytes:23896850,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:23881995,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18758628,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18233005,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonroot@sha256:ee9f50b3c64b174d296d91ca9f69a914ac30e59095dfb462b2b518ad28a63655 registry.k8s.io/e2e-test-images/nonroot:1.4],SizeBytes:17747885,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8688564,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:7000082,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6978614,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac registry.k8s.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[kubernetes.io/csi/ebs.csi.aws.com^vol-0dec4dd6bb50184f2],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-0dec4dd6bb50184f2,DevicePath:,},},Config:nil,},} Mar 25 05:45:45.168: INFO: Logging kubelet events for node i-03d301e6f030748f1 Mar 25 05:45:45.227: INFO: Logging pods the kubelet thinks is on node i-03d301e6f030748f1 Mar 25 05:45:45.281: INFO: busybox-06aeea52-3822-43de-9b3c-1a9c004257fc started at 2023-03-25 05:45:16 +0000 UTC (0+1 container statuses recorded) Mar 25 05:45:45.281: INFO: Container busybox ready: true, restart count 0 Mar 25 05:45:45.281: INFO: ss2-1 started at 2023-03-25 05:45:33 +0000 UTC (0+1 container statuses recorded) Mar 25 05:45:45.281: INFO: Container webserver ready: false, restart count 0 Mar 25 05:45:45.281: INFO: startup-script started at 2023-03-25 05:44:37 +0000 UTC (0+1 container statuses recorded) Mar 25 05:45:45.281: INFO: Container startup-script ready: true, restart count 0 Mar 25 05:45:45.281: INFO: pod-configmaps-a93186b9-eae5-49fb-9839-a908e7cb30ca started at 2023-03-25 05:44:09 +0000 UTC (0+2 container statuses recorded) Mar 25 05:45:45.281: INFO: Container agnhost-container ready: true, restart count 0 Mar 25 05:45:45.281: INFO: Container configmap-volume-binary-test ready: false, restart count 0 Mar 25 05:45:45.281: INFO: test-pod started at 2023-03-25 05:45:44 +0000 UTC (0+1 container statuses recorded) Mar 25 05:45:45.281: INFO: Container test-container ready: false, restart count 0 Mar 25 05:45:45.281: INFO: pod-ephm-test-projected-rxqb started at 2023-03-25 05:44:02 +0000 UTC (0+1 container statuses recorded) Mar 25 05:45:45.281: INFO: Container test-container-subpath-projected-rxqb ready: false, restart count 0 Mar 25 05:45:45.281: INFO: inline-volume-tester-hdkbc started at 2023-03-25 05:45:44 +0000 UTC (0+1 container statuses recorded) Mar 25 05:45:45.281: INFO: Container csi-volume-tester ready: false, restart count 0 Mar 25 05:45:45.281: INFO: calico-node-h96vq started at 2023-03-25 05:31:35 +0000 UTC (3+1 container statuses recorded) Mar 25 05:45:45.281: INFO: Init container upgrade-ipam ready: true, restart count 0 Mar 25 05:45:45.281: INFO: Init container install-cni ready: true, restart count 0 Mar 25 05:45:45.281: INFO: Init container mount-bpffs ready: true, restart count 0 Mar 25 05:45:45.281: INFO: Container calico-node ready: true, restart count 0 Mar 25 05:45:45.281: INFO: ss-1 started at 2023-03-25 05:45:08 +0000 UTC (0+1 container statuses recorded) Mar 25 05:45:45.281: INFO: Container webserver ready: true, restart count 0 Mar 25 05:45:45.281: INFO: ebs-csi-node-lljwl started at 2023-03-25 05:31:35 +0000 UTC (0+3 container statuses recorded) Mar 25 05:45:45.281: INFO: Container ebs-plugin ready: true, restart count 0 Mar 25 05:45:45.281: INFO: Container liveness-probe ready: true, restart count 0 Mar 25 05:45:45.281: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 25 05:45:45.281: INFO: test-ss-0 started at 2023-03-25 05:45:16 +0000 UTC (0+1 container statuses recorded) Mar 25 05:45:45.281: INFO: Container webserver ready: true, restart count 0 Mar 25 05:45:45.281: INFO: kube-proxy-i-03d301e6f030748f1 started at 2023-03-25 05:31:35 +0000 UTC (0+1 container statuses recorded) Mar 25 05:45:45.281: INFO: Container kube-proxy ready: true, restart count 0 Mar 25 05:45:45.281: INFO: pfpod started at 2023-03-25 05:44:26 +0000 UTC (0+2 container statuses recorded) Mar 25 05:45:45.281: INFO: Container portforwardtester ready: false, restart count 0 Mar 25 05:45:45.281: INFO: Container readiness ready: false, restart count 0 Mar 25 05:45:45.545: INFO: Latency metrics for node i-03d301e6f030748f1 Mar 25 05:45:45.545: INFO: Logging node info for node i-049fdb2c1cde894df Mar 25 05:45:45.594: INFO: Node Info: &Node{ObjectMeta:{i-049fdb2c1cde894df b43a51e6-dda7-4047-b368-72d515695b4e 33250 0 2023-03-25 05:23:34 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-1 failure-domain.beta.kubernetes.io/zone:us-west-1b io.kubernetes.storage.mock/node:some-mock-node kubernetes.io/arch:amd64 kubernetes.io/hostname:i-049fdb2c1cde894df kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:us-west-1b topology.hostpath.csi/node:i-049fdb2c1cde894df topology.kubernetes.io/region:us-west-1 topology.kubernetes.io/zone:us-west-1b] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-volume-expand-5919":"i-049fdb2c1cde894df","csi-mock-csi-mock-volumes-expansion-1460":"i-049fdb2c1cde894df","ebs.csi.aws.com":"i-049fdb2c1cde894df"} node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:172.20.43.61/19 projectcalico.org/IPv4IPIPTunnelAddr:100.97.18.64 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-03-25 05:23:34 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-25 05:23:34 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-03-25 05:23:34 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2023-03-25 05:23:34 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2023-03-25 05:23:45 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.6.0/24\"":{}}}} } {calico-node Update v1 2023-03-25 05:23:52 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4IPIPTunnelAddr":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2023-03-25 05:45:32 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-03-25 05:45:39 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:io.kubernetes.storage.mock/node":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.6.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-1b/i-049fdb2c1cde894df,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.6.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-03-25 05:23:52 +0000 UTC,LastTransitionTime:2023-03-25 05:23:52 +0000 UTC,Reason:CalicoIsUp,Message:Calico is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-25 05:45:31 +0000 UTC,LastTransitionTime:2023-03-25 05:23:34 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-25 05:45:31 +0000 UTC,LastTransitionTime:2023-03-25 05:23:34 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-25 05:45:31 +0000 UTC,LastTransitionTime:2023-03-25 05:23:34 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-25 05:45:31 +0000 UTC,LastTransitionTime:2023-03-25 05:23:45 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.43.61,},NodeAddress{Type:ExternalIP,Address:18.144.75.67,},NodeAddress{Type:InternalDNS,Address:i-049fdb2c1cde894df.us-west-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-049fdb2c1cde894df.us-west-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-18-144-75-67.us-west-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec252e592d19e1e6c583e87505431016,SystemUUID:ec252e59-2d19-e1e6-c583-e87505431016,BootID:45436b5d-1eb0-4b6f-a1d1-a1492b31b5d7,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.28.0-alpha.0,KubeProxyVersion:v1.28.0-alpha.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:112030336,},ContainerImage{Names:[docker.io/calico/cni@sha256:e282ea2914c806b5de2976330a17cfb5e6dcef47147bceb1432ca5c75fd46f50 docker.io/calico/cni:v3.24.5],SizeBytes:87458443,},ContainerImage{Names:[docker.io/calico/node@sha256:5972ad2bcbdc668564d3e26960c9c513b2d7b05581c704747cf7c62ef3a405a6 docker.io/calico/node:v3.24.5],SizeBytes:81584866,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:e468dddcd275163a042ab297b2d8c2aca50d5e148d2d22f3b6ba119e2f31fa79 registry.k8s.io/sig-storage/csi-provisioner:v3.4.0],SizeBytes:27427836,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/kube-proxy@sha256:8a03b3147cd7be8e8235ae1cd70cf69aca9ce03ba1cfaffea320540bb5afa496 registry.k8s.io/kube-proxy:v1.28.0-alpha.0],SizeBytes:23896850,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:23881995,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:68d396900aeaa072c1f27289485fdac29834045a6f3ffe369bf389d830ef572d registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.6],SizeBytes:20293261,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18758628,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18233005,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:15273057,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8688564,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:7000082,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6978614,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[kubernetes.io/csi/ebs.csi.aws.com^vol-00e9b723211b131bc kubernetes.io/csi/ebs.csi.aws.com^vol-0bed540f628372c85],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-00e9b723211b131bc,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-0bed540f628372c85,DevicePath:,},},Config:nil,},} Mar 25 05:45:45.595: INFO: Logging kubelet events for node i-049fdb2c1cde894df Mar 25 05:45:45.655: INFO: Logging pods the kubelet thinks is on node i-049fdb2c1cde894df Mar 25 05:45:45.711: INFO: netserver-1 started at 2023-03-25 05:45:45 +0000 UTC (0+1 container statuses recorded) Mar 25 05:45:45.711: INFO: Container webserver ready: false, restart count 0 Mar 25 05:45:45.711: INFO: kube-proxy-i-049fdb2c1cde894df started at 2023-03-25 05:23:35 +0000 UTC (0+1 container statuses recorded) Mar 25 05:45:45.711: INFO: Container kube-proxy ready: true, restart count 0 Mar 25 05:45:45.711: INFO: ebs-csi-node-m9zxz started at 2023-03-25 05:23:35 +0000 UTC (0+3 container statuses recorded) Mar 25 05:45:45.711: INFO: Container ebs-plugin ready: true, restart count 0 Mar 25 05:45:45.711: INFO: Container liveness-probe ready: true, restart count 0 Mar 25 05:45:45.711: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 25 05:45:45.711: INFO: calico-node-wt5p7 started at 2023-03-25 05:23:35 +0000 UTC (3+1 container statuses recorded) Mar 25 05:45:45.711: INFO: Init container upgrade-ipam ready: true, restart count 0 Mar 25 05:45:45.711: INFO: Init container install-cni ready: true, restart count 0 Mar 25 05:45:45.711: INFO: Init container mount-bpffs ready: true, restart count 0 Mar 25 05:45:45.711: INFO: Container calico-node ready: true, restart count 0 Mar 25 05:45:45.711: INFO: coredns-559769c974-pgqhl started at 2023-03-25 05:24:36 +0000 UTC (0+1 container statuses recorded) Mar 25 05:45:45.711: INFO: Container coredns ready: true, restart count 0 Mar 25 05:45:45.711: INFO: bin-false92092514-d86f-443e-9f03-8be47f3e24e4 started at 2023-03-25 05:43:44 +0000 UTC (0+1 container statuses recorded) Mar 25 05:45:45.711: INFO: Container bin-false92092514-d86f-443e-9f03-8be47f3e24e4 ready: false, restart count 0 Mar 25 05:45:45.711: INFO: test-grpc-c9016762-a9d2-412b-87fb-e1be3cf11d20 started at 2023-03-25 05:45:01 +0000 UTC (0+1 container statuses recorded) Mar 25 05:45:45.711: INFO: Container agnhost ready: true, restart count 0 Mar 25 05:45:45.711: INFO: ss2-0 started at 2023-03-25 05:45:32 +0000 UTC (0+1 container statuses recorded) Mar 25 05:45:45.711: INFO: Container webserver ready: true, restart count 0 Mar 25 05:45:45.711: INFO: ss-0 started at 2023-03-25 05:44:46 +0000 UTC (0+1 container statuses recorded) Mar 25 05:45:45.711: INFO: Container webserver ready: true, restart count 0 Mar 25 05:45:45.711: INFO: csi-mockplugin-0 started at 2023-03-25 05:45:32 +0000 UTC (0+3 container statuses recorded) Mar 25 05:45:45.711: INFO: Container csi-provisioner ready: true, restart count 0 Mar 25 05:45:45.711: INFO: Container driver-registrar ready: true, restart count 0 Mar 25 05:45:45.711: INFO: Container mock ready: true, restart count 0 Mar 25 05:45:45.711: INFO: coredns-autoscaler-7cb5c5b969-h485x started at 2023-03-25 05:24:36 +0000 UTC (0+1 container statuses recorded) Mar 25 05:45:45.711: INFO: Container autoscaler ready: true, restart count 0 Mar 25 05:45:45.711: INFO: inline-volume-tester-5j9fs started at 2023-03-25 05:45:05 +0000 UTC (0+1 container statuses recorded) Mar 25 05:45:45.711: INFO: Container csi-volume-tester ready: true, restart count 0 Mar 25 05:45:45.711: INFO: csi-mockplugin-resizer-0 started at 2023-03-25 05:45:32 +0000 UTC (0+1 container statuses recorded) Mar 25 05:45:45.711: INFO: Container csi-resizer ready: true, restart count 0 Mar 25 05:45:45.711: INFO: csi-hostpathplugin-0 started at 2023-03-25 05:45:38 +0000 UTC (0+7 container statuses recorded) Mar 25 05:45:45.711: INFO: Container csi-attacher ready: true, restart count 0 Mar 25 05:45:45.711: INFO: Container csi-provisioner ready: true, restart count 0 Mar 25 05:45:45.711: INFO: Container csi-resizer ready: true, restart count 0 Mar 25 05:45:45.711: INFO: Container csi-snapshotter ready: true, restart count 0 Mar 25 05:45:45.711: INFO: Container hostpath ready: true, restart count 0 Mar 25 05:45:45.711: INFO: Container liveness-probe ready: true, restart count 0 Mar 25 05:45:45.711: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 25 05:45:45.711: INFO: labelsupdate8d6b443f-1225-422c-a63e-1271aef97f34 started at 2023-03-25 05:44:27 +0000 UTC (0+1 container statuses recorded) Mar 25 05:45:45.711: INFO: Container client-container ready: true, restart count 0 Mar 25 05:45:45.711: INFO: annotationupdatee5e5d8ca-80ad-4a9d-8c8d-8a995c448ccc started at 2023-03-25 05:43:58 +0000 UTC (0+1 container statuses recorded) Mar 25 05:45:45.711: INFO: Container client-container ready: true, restart count 0 Mar 25 05:45:46.087: INFO: Latency metrics for node i-049fdb2c1cde894df Mar 25 05:45:46.087: INFO: Logging node info for node i-062679bb8c0ba4bce Mar 25 05:45:46.142: INFO: Node Info: &Node{ObjectMeta:{i-062679bb8c0ba4bce ee2da569-bb88-4297-9e6f-d2b15ba7ce86 32934 0 2023-03-25 05:27:31 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-1 failure-domain.beta.kubernetes.io/zone:us-west-1b kubernetes.io/arch:amd64 kubernetes.io/hostname:i-062679bb8c0ba4bce kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:us-west-1b topology.hostpath.csi/node:i-062679bb8c0ba4bce topology.kubernetes.io/region:us-west-1 topology.kubernetes.io/zone:us-west-1b] map[csi.volume.kubernetes.io/nodeid:{"csi-mock-csi-mock-volumes-expansion-827":"csi-mock-csi-mock-volumes-expansion-827","ebs.csi.aws.com":"i-062679bb8c0ba4bce"} node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:172.20.45.43/19 projectcalico.org/IPv4IPIPTunnelAddr:100.106.79.0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-03-25 05:27:31 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-25 05:27:31 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-03-25 05:27:31 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2023-03-25 05:27:31 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2023-03-25 05:27:44 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.7.0/24\"":{}}}} } {calico-node Update v1 2023-03-25 05:27:50 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4IPIPTunnelAddr":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2023-03-25 05:45:32 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.7.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-1b/i-062679bb8c0ba4bce,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.7.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051689472 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946831872 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-03-25 05:27:50 +0000 UTC,LastTransitionTime:2023-03-25 05:27:50 +0000 UTC,Reason:CalicoIsUp,Message:Calico is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-25 05:45:32 +0000 UTC,LastTransitionTime:2023-03-25 05:27:31 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-25 05:45:32 +0000 UTC,LastTransitionTime:2023-03-25 05:27:31 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-25 05:45:32 +0000 UTC,LastTransitionTime:2023-03-25 05:27:31 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-25 05:45:32 +0000 UTC,LastTransitionTime:2023-03-25 05:27:44 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.45.43,},NodeAddress{Type:ExternalIP,Address:3.101.133.108,},NodeAddress{Type:InternalDNS,Address:i-062679bb8c0ba4bce.us-west-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-062679bb8c0ba4bce.us-west-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-3-101-133-108.us-west-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2528f599e9b1e199793fb5cd06d205,SystemUUID:ec2528f5-99e9-b1e1-9979-3fb5cd06d205,BootID:ff56b70b-c8c5-46db-8cc5-4353c8f535c6,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.28.0-alpha.0,KubeProxyVersion:v1.28.0-alpha.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:112030336,},ContainerImage{Names:[docker.io/calico/cni@sha256:e282ea2914c806b5de2976330a17cfb5e6dcef47147bceb1432ca5c75fd46f50 docker.io/calico/cni:v3.24.5],SizeBytes:87458443,},ContainerImage{Names:[docker.io/calico/node@sha256:5972ad2bcbdc668564d3e26960c9c513b2d7b05581c704747cf7c62ef3a405a6 docker.io/calico/node:v3.24.5],SizeBytes:81584866,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nautilus@sha256:80ba6c8c44f9623f06e868a1aa66026c8ec438ad814f9ec95e9333b415fe3550 registry.k8s.io/e2e-test-images/nautilus:1.7],SizeBytes:49641698,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41901587,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:e468dddcd275163a042ab297b2d8c2aca50d5e148d2d22f3b6ba119e2f31fa79 registry.k8s.io/sig-storage/csi-provisioner:v3.4.0],SizeBytes:27427836,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/kube-proxy@sha256:8a03b3147cd7be8e8235ae1cd70cf69aca9ce03ba1cfaffea320540bb5afa496 registry.k8s.io/kube-proxy:v1.28.0-alpha.0],SizeBytes:23896850,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:23881995,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18758628,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/regression-issue-74839@sha256:b4f1d8d61bdad84bd50442d161d5460e4019d53e989b64220fdbc62fc87d76bf registry.k8s.io/e2e-test-images/regression-issue-74839:1.2],SizeBytes:18651485,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18233005,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:15273057,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8688564,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/build-image/distroless-iptables@sha256:13ac9b3c476d7290a4451d65d09d6016a2cb89836ffbfa4eae55b72731a22080 registry.k8s.io/build-image/distroless-iptables:v0.2.2],SizeBytes:7729580,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:7000082,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6978614,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Mar 25 05:45:46.143: INFO: Logging kubelet events for node i-062679bb8c0ba4bce Mar 25 05:45:46.216: INFO: Logging pods the kubelet thinks is on node i-062679bb8c0ba4bce Mar 25 05:45:46.282: INFO: csi-mockplugin-0 started at 2023-03-25 05:42:40 +0000 UTC (0+4 container statuses recorded) Mar 25 05:45:46.282: INFO: Container busybox ready: true, restart count 0 Mar 25 05:45:46.282: INFO: Container csi-provisioner ready: true, restart count 0 Mar 25 05:45:46.282: INFO: Container driver-registrar ready: true, restart count 0 Mar 25 05:45:46.282: INFO: Container mock ready: true, restart count 0 Mar 25 05:45:46.282: INFO: ss2-2 started at 2023-03-25 05:45:34 +0000 UTC (0+1 container statuses recorded) Mar 25 05:45:46.282: INFO: Container webserver ready: true, restart count 0 Mar 25 05:45:46.282: INFO: netserver-2 started at 2023-03-25 05:45:45 +0000 UTC (0+1 container statuses recorded) Mar 25 05:45:46.282: INFO: Container webserver ready: false, restart count 0 Mar 25 05:45:46.282: INFO: csi-mockplugin-0 started at 2023-03-25 05:44:50 +0000 UTC (0+4 container statuses recorded) Mar 25 05:45:46.282: INFO: Container busybox ready: true, restart count 0 Mar 25 05:45:46.282: INFO: Container csi-provisioner ready: true, restart count 0 Mar 25 05:45:46.282: INFO: Container driver-registrar ready: true, restart count 0 Mar 25 05:45:46.282: INFO: Container mock ready: true, restart count 0 Mar 25 05:45:46.282: INFO: kube-proxy-i-062679bb8c0ba4bce started at 2023-03-25 05:27:32 +0000 UTC (0+1 container statuses recorded) Mar 25 05:45:46.282: INFO: Container kube-proxy ready: true, restart count 0 Mar 25 05:45:46.282: INFO: calico-node-zwm5j started at 2023-03-25 05:27:32 +0000 UTC (3+1 container statuses recorded) Mar 25 05:45:46.282: INFO: Init container upgrade-ipam ready: true, restart count 0 Mar 25 05:45:46.282: INFO: Init container install-cni ready: true, restart count 0 Mar 25 05:45:46.282: INFO: Init container mount-bpffs ready: true, restart count 0 Mar 25 05:45:46.282: INFO: Container calico-node ready: true, restart count 0 Mar 25 05:45:46.282: INFO: csi-mockplugin-attacher-0 started at 2023-03-25 05:44:50 +0000 UTC (0+1 container statuses recorded) Mar 25 05:45:46.282: INFO: Container csi-attacher ready: true, restart count 0 Mar 25 05:45:46.282: INFO: csi-mockplugin-resizer-0 started at 2023-03-25 05:44:50 +0000 UTC (0+1 container statuses recorded) Mar 25 05:45:46.282: INFO: Container csi-resizer ready: true, restart count 0 Mar 25 05:45:46.282: INFO: coredns-559769c974-p9vg6 started at 2023-03-25 05:36:16 +0000 UTC (0+1 container statuses recorded) Mar 25 05:45:46.282: INFO: Container coredns ready: true, restart count 0 Mar 25 05:45:46.282: INFO: ebs-csi-node-7twrq started at 2023-03-25 05:27:32 +0000 UTC (0+3 container statuses recorded) Mar 25 05:45:46.282: INFO: Container ebs-plugin ready: true, restart count 0 Mar 25 05:45:46.282: INFO: Container liveness-probe ready: true, restart count 0 Mar 25 05:45:46.282: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 25 05:45:46.282: INFO: boom-server started at 2023-03-25 05:44:33 +0000 UTC (0+1 container statuses recorded) Mar 25 05:45:46.282: INFO: Container boom-server ready: true, restart count 0 Mar 25 05:45:46.282: INFO: rs-l7jm2 started at 2023-03-25 05:44:57 +0000 UTC (0+1 container statuses recorded) Mar 25 05:45:46.282: INFO: Container donothing ready: false, restart count 0 Mar 25 05:45:46.542: INFO: Latency metrics for node i-062679bb8c0ba4bce Mar 25 05:45:46.542: INFO: Logging node info for node i-072608201d2c0f1ad Mar 25 05:45:46.590: INFO: Node Info: &Node{ObjectMeta:{i-072608201d2c0f1ad 264dfba1-3cf0-4c99-8e8c-098f0f792077 33373 0 2023-03-25 05:35:29 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-1 failure-domain.beta.kubernetes.io/zone:us-west-1b kubernetes.io/arch:amd64 kubernetes.io/hostname:i-072608201d2c0f1ad kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:us-west-1b topology.hostpath.csi/node:i-072608201d2c0f1ad topology.kubernetes.io/region:us-west-1 topology.kubernetes.io/zone:us-west-1b] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-volume-8577":"i-072608201d2c0f1ad","ebs.csi.aws.com":"i-072608201d2c0f1ad"} node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:172.20.38.125/19 projectcalico.org/IPv4IPIPTunnelAddr:100.107.220.192 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-03-25 05:35:29 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-25 05:35:29 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-03-25 05:35:29 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2023-03-25 05:35:29 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2023-03-25 05:35:41 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.9.0/24\"":{}}}} } {calico-node Update v1 2023-03-25 05:35:48 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4IPIPTunnelAddr":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2023-03-25 05:45:42 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-03-25 05:45:42 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.9.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-1b/i-072608201d2c0f1ad,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.9.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051689472 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946831872 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-03-25 05:35:47 +0000 UTC,LastTransitionTime:2023-03-25 05:35:47 +0000 UTC,Reason:CalicoIsUp,Message:Calico is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-25 05:45:42 +0000 UTC,LastTransitionTime:2023-03-25 05:35:29 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-25 05:45:42 +0000 UTC,LastTransitionTime:2023-03-25 05:35:29 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-25 05:45:42 +0000 UTC,LastTransitionTime:2023-03-25 05:35:29 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-25 05:45:42 +0000 UTC,LastTransitionTime:2023-03-25 05:35:41 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.38.125,},NodeAddress{Type:ExternalIP,Address:3.101.88.10,},NodeAddress{Type:InternalDNS,Address:i-072608201d2c0f1ad.us-west-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-072608201d2c0f1ad.us-west-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-3-101-88-10.us-west-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2d3818c4f80a253dcc1a60e148dbc8,SystemUUID:ec2d3818-c4f8-0a25-3dcc-1a60e148dbc8,BootID:77d046cf-75ea-4a8c-970a-3190d64e019b,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.28.0-alpha.0,KubeProxyVersion:v1.28.0-alpha.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:112030336,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:51eae8381dcb1078289fa7b4f3df2630cdc18d09fb56f8e56b41c40e191d6c83 registry.k8s.io/etcd:3.5.7-0],SizeBytes:101639218,},ContainerImage{Names:[docker.io/calico/cni@sha256:e282ea2914c806b5de2976330a17cfb5e6dcef47147bceb1432ca5c75fd46f50 docker.io/calico/cni:v3.24.5],SizeBytes:87458443,},ContainerImage{Names:[docker.io/calico/node@sha256:5972ad2bcbdc668564d3e26960c9c513b2d7b05581c704747cf7c62ef3a405a6 docker.io/calico/node:v3.24.5],SizeBytes:81584866,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41901587,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:e468dddcd275163a042ab297b2d8c2aca50d5e148d2d22f3b6ba119e2f31fa79 registry.k8s.io/sig-storage/csi-provisioner:v3.4.0],SizeBytes:27427836,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/sample-apiserver@sha256:8d70890151aa5d096f331cb9da1b9cd5be0412b7363fe67b5c3befdcaa2a28d0 registry.k8s.io/e2e-test-images/sample-apiserver:1.17.7],SizeBytes:25667066,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/kube-proxy@sha256:8a03b3147cd7be8e8235ae1cd70cf69aca9ce03ba1cfaffea320540bb5afa496 registry.k8s.io/kube-proxy:v1.28.0-alpha.0],SizeBytes:23896850,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:23881995,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18758628,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18233005,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8688564,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:7000082,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6978614,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[kubernetes.io/csi/ebs.csi.aws.com^vol-046462550b75078b8],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-046462550b75078b8,DevicePath:,},},Config:nil,},} Mar 25 05:45:46.591: INFO: Logging kubelet events for node i-072608201d2c0f1ad Mar 25 05:45:46.649: INFO: Logging pods the kubelet thinks is on node i-072608201d2c0f1ad Mar 25 05:45:46.704: INFO: dns-test-42c3860b-2f09-4ab6-b57e-35add40b3580 started at 2023-03-25 05:45:40 +0000 UTC (0+3 container statuses recorded) Mar 25 05:45:46.704: INFO: Container jessie-querier ready: true, restart count 0 Mar 25 05:45:46.704: INFO: Container querier ready: true, restart count 0 Mar 25 05:45:46.704: INFO: Container webserver ready: true, restart count 0 Mar 25 05:45:46.704: INFO: csi-hostpathplugin-0 started at 2023-03-25 05:45:02 +0000 UTC (0+7 container statuses recorded) Mar 25 05:45:46.704: INFO: Container csi-attacher ready: true, restart count 0 Mar 25 05:45:46.704: INFO: Container csi-provisioner ready: true, restart count 0 Mar 25 05:45:46.704: INFO: Container csi-resizer ready: true, restart count 0 Mar 25 05:45:46.704: INFO: Container csi-snapshotter ready: true, restart count 0 Mar 25 05:45:46.704: INFO: Container hostpath ready: true, restart count 0 Mar 25 05:45:46.704: INFO: Container liveness-probe ready: true, restart count 0 Mar 25 05:45:46.704: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 25 05:45:46.704: INFO: kube-proxy-i-072608201d2c0f1ad started at 2023-03-25 05:35:30 +0000 UTC (0+1 container statuses recorded) Mar 25 05:45:46.704: INFO: Container kube-proxy ready: true, restart count 0 Mar 25 05:45:46.704: INFO: netserver-3 started at 2023-03-25 05:45:45 +0000 UTC (0+1 container statuses recorded) Mar 25 05:45:46.704: INFO: Container webserver ready: false, restart count 0 Mar 25 05:45:46.704: INFO: ss-2 started at 2023-03-25 05:45:28 +0000 UTC (0+1 container statuses recorded) Mar 25 05:45:46.704: INFO: Container webserver ready: true, restart count 0 Mar 25 05:45:46.704: INFO: pod-fbbbcbf3-9851-406c-992e-726d4e6bae4c started at 2023-03-25 05:45:45 +0000 UTC (0+1 container statuses recorded) Mar 25 05:45:46.704: INFO: Container write-pod ready: false, restart count 0 Mar 25 05:45:46.704: INFO: ebs-csi-node-r7mcm started at 2023-03-25 05:35:30 +0000 UTC (0+3 container statuses recorded) Mar 25 05:45:46.704: INFO: Container ebs-plugin ready: true, restart count 0 Mar 25 05:45:46.704: INFO: Container liveness-probe ready: true, restart count 0 Mar 25 05:45:46.704: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 25 05:45:46.704: INFO: pod-handle-http-request started at 2023-03-25 05:45:36 +0000 UTC (0+2 container statuses recorded) Mar 25 05:45:46.704: INFO: Container container-handle-http-request ready: true, restart count 0 Mar 25 05:45:46.704: INFO: Container container-handle-https-request ready: true, restart count 0 Mar 25 05:45:46.704: INFO: calico-node-lc9zl started at 2023-03-25 05:35:30 +0000 UTC (3+1 container statuses recorded) Mar 25 05:45:46.704: INFO: Init container upgrade-ipam ready: true, restart count 0 Mar 25 05:45:46.704: INFO: Init container install-cni ready: true, restart count 0 Mar 25 05:45:46.704: INFO: Init container mount-bpffs ready: true, restart count 0 Mar 25 05:45:46.704: INFO: Container calico-node ready: true, restart count 0 Mar 25 05:45:47.094: INFO: Latency metrics for node i-072608201d2c0f1ad END STEP: dump namespace information after failure - test/e2e/framework/framework.go:288 @ 03/25/23 05:45:47.094 (2.599s) < Exit [DeferCleanup (Each)] [Testpattern: Dynamic PV (ext4)] volumes - dump namespaces | framework.go:209 @ 03/25/23 05:45:47.094 (2.599s) > Enter [DeferCleanup (Each)] [Testpattern: Dynamic PV (ext4)] volumes - tear down framework | framework.go:206 @ 03/25/23 05:45:47.094 STEP: Destroying namespace "volume-8318" for this suite. - test/e2e/framework/framework.go:351 @ 03/25/23 05:45:47.094 < Exit [DeferCleanup (Each)] [Testpattern: Dynamic PV (ext4)] volumes - tear down framework | framework.go:206 @ 03/25/23 05:45:47.145 (51ms) > Enter [ReportAfterEach] TOP-LEVEL - test/e2e/e2e_test.go:144 @ 03/25/23 05:45:47.146 < Exit [ReportAfterEach] TOP-LEVEL - test/e2e/e2e_test.go:144 @ 03/25/23 05:45:47.146 (0s)
Filter through log files | View test history on testgrid
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[It\]\s\[sig\-auth\]\sServiceAccounts\sServiceAccountIssuerDiscovery\sshould\ssupport\sOIDC\sdiscovery\sof\sservice\saccount\sissuer\s\[Conformance\]$'
[FAILED] Unexpected error getting pod logs: timed out waiting for the condition In [It] at: test/e2e/auth/service_accounts.go:630 @ 03/25/23 05:45:34.704from junit_01.xml
> Enter [BeforeEach] [sig-auth] ServiceAccounts - set up framework | framework.go:191 @ 03/25/23 05:43:29.953 STEP: Creating a kubernetes client - test/e2e/framework/framework.go:211 @ 03/25/23 05:43:29.953 Mar 25 05:43:29.953: INFO: >>> kubeConfig: /root/.kube/config STEP: Building a namespace api object, basename svcaccounts - test/e2e/framework/framework.go:250 @ 03/25/23 05:43:29.954 STEP: Waiting for a default service account to be provisioned in namespace - test/e2e/framework/framework.go:259 @ 03/25/23 05:43:30.101 STEP: Waiting for kube-root-ca.crt to be provisioned in namespace - test/e2e/framework/framework.go:262 @ 03/25/23 05:43:30.197 < Exit [BeforeEach] [sig-auth] ServiceAccounts - set up framework | framework.go:191 @ 03/25/23 05:43:30.293 (340ms) > Enter [BeforeEach] [sig-auth] ServiceAccounts - test/e2e/framework/metrics/init/init.go:33 @ 03/25/23 05:43:30.293 < Exit [BeforeEach] [sig-auth] ServiceAccounts - test/e2e/framework/metrics/init/init.go:33 @ 03/25/23 05:43:30.293 (0s) > Enter [It] ServiceAccountIssuerDiscovery should support OIDC discovery of service account issuer [Conformance] - test/e2e/auth/service_accounts.go:529 @ 03/25/23 05:43:30.293 Mar 25 05:43:30.398: INFO: created pod STEP: Saw pod success - test/e2e/framework/pod/wait.go:409 @ 03/25/23 05:43:34.556 Mar 25 05:44:04.557: INFO: polling logs Mar 25 05:44:04.605: INFO: Error pulling logs: Get "https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/svcaccounts-3613/pods/oidc-discovery-validator/log?container=oidc-discovery-validator&previous=false": stream error: stream ID 1701; INTERNAL_ERROR; received from peer Mar 25 05:44:34.557: INFO: polling logs Mar 25 05:44:34.605: INFO: Error pulling logs: Get "https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/svcaccounts-3613/pods/oidc-discovery-validator/log?container=oidc-discovery-validator&previous=false": stream error: stream ID 1703; INTERNAL_ERROR; received from peer Mar 25 05:45:04.557: INFO: polling logs Mar 25 05:45:04.611: INFO: Error pulling logs: Get "https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/svcaccounts-3613/pods/oidc-discovery-validator/log?container=oidc-discovery-validator&previous=false": stream error: stream ID 1705; INTERNAL_ERROR; received from peer Mar 25 05:45:34.557: INFO: polling logs Mar 25 05:45:34.605: INFO: Error pulling logs: Get "https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/svcaccounts-3613/pods/oidc-discovery-validator/log?container=oidc-discovery-validator&previous=false": stream error: stream ID 1707; INTERNAL_ERROR; received from peer Mar 25 05:45:34.605: INFO: polling logs Mar 25 05:45:34.653: INFO: Error pulling logs: Get "https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/svcaccounts-3613/pods/oidc-discovery-validator/log?container=oidc-discovery-validator&previous=false": stream error: stream ID 1709; INTERNAL_ERROR; received from peer [FAILED] Unexpected error getting pod logs: timed out waiting for the condition In [It] at: test/e2e/auth/service_accounts.go:630 @ 03/25/23 05:45:34.704 < Exit [It] ServiceAccountIssuerDiscovery should support OIDC discovery of service account issuer [Conformance] - test/e2e/auth/service_accounts.go:529 @ 03/25/23 05:45:34.704 (2m4.41s) > Enter [AfterEach] [sig-auth] ServiceAccounts - test/e2e/framework/node/init/init.go:33 @ 03/25/23 05:45:34.704 Mar 25 05:45:34.704: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready < Exit [AfterEach] [sig-auth] ServiceAccounts - test/e2e/framework/node/init/init.go:33 @ 03/25/23 05:45:34.753 (49ms) > Enter [DeferCleanup (Each)] [sig-auth] ServiceAccounts - test/e2e/framework/metrics/init/init.go:35 @ 03/25/23 05:45:34.753 < Exit [DeferCleanup (Each)] [sig-auth] ServiceAccounts - test/e2e/framework/metrics/init/init.go:35 @ 03/25/23 05:45:34.753 (0s) > Enter [DeferCleanup (Each)] [sig-auth] ServiceAccounts - dump namespaces | framework.go:209 @ 03/25/23 05:45:34.753 STEP: dump namespace information after failure - test/e2e/framework/framework.go:288 @ 03/25/23 05:45:34.753 STEP: Collecting events from namespace "svcaccounts-3613". - test/e2e/framework/debug/dump.go:42 @ 03/25/23 05:45:34.753 STEP: Found 4 events. - test/e2e/framework/debug/dump.go:46 @ 03/25/23 05:45:34.802 Mar 25 05:45:34.802: INFO: At 2023-03-25 05:43:30 +0000 UTC - event for oidc-discovery-validator: {default-scheduler } Scheduled: Successfully assigned svcaccounts-3613/oidc-discovery-validator to i-062679bb8c0ba4bce Mar 25 05:45:34.802: INFO: At 2023-03-25 05:43:31 +0000 UTC - event for oidc-discovery-validator: {kubelet i-062679bb8c0ba4bce} Pulled: Container image "registry.k8s.io/e2e-test-images/agnhost:2.43" already present on machine Mar 25 05:45:34.802: INFO: At 2023-03-25 05:43:31 +0000 UTC - event for oidc-discovery-validator: {kubelet i-062679bb8c0ba4bce} Created: Created container oidc-discovery-validator Mar 25 05:45:34.802: INFO: At 2023-03-25 05:43:31 +0000 UTC - event for oidc-discovery-validator: {kubelet i-062679bb8c0ba4bce} Started: Started container oidc-discovery-validator Mar 25 05:45:34.850: INFO: POD NODE PHASE GRACE CONDITIONS Mar 25 05:45:34.850: INFO: oidc-discovery-validator i-062679bb8c0ba4bce Succeeded [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2023-03-25 05:43:30 +0000 UTC PodCompleted } {Ready False 0001-01-01 00:00:00 +0000 UTC 2023-03-25 05:43:30 +0000 UTC PodCompleted } {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2023-03-25 05:43:30 +0000 UTC PodCompleted } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2023-03-25 05:43:30 +0000 UTC }] Mar 25 05:45:34.850: INFO: Mar 25 05:45:34.898: INFO: Unable to fetch svcaccounts-3613/oidc-discovery-validator/oidc-discovery-validator logs: Get "https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/svcaccounts-3613/pods/oidc-discovery-validator/log?container=oidc-discovery-validator&previous=false&tailLines=42": stream error: stream ID 1719; INTERNAL_ERROR; received from peer Mar 25 05:45:34.952: INFO: Logging node info for node i-0168f82458996cc53 Mar 25 05:45:35.001: INFO: Node Info: &Node{ObjectMeta:{i-0168f82458996cc53 f4eb302c-c477-45ac-bcac-ffed65f6580f 21197 0 2023-03-25 05:18:57 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-1 failure-domain.beta.kubernetes.io/zone:us-west-1b kops.k8s.io/kops-controller-pki: kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0168f82458996cc53 kubernetes.io/os:linux node-role.kubernetes.io/control-plane: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:us-west-1b topology.kubernetes.io/region:us-west-1 topology.kubernetes.io/zone:us-west-1b] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0168f82458996cc53"} node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:172.20.48.81/19 projectcalico.org/IPv4IPIPTunnelAddr:100.100.215.192 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-03-25 05:18:57 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {protokube Update v1 2023-03-25 05:18:58 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {calico-node Update v1 2023-03-25 05:20:22 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4IPIPTunnelAddr":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {aws-cloud-controller-manager Update v1 2023-03-25 05:20:26 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-25 05:20:26 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2023-03-25 05:20:31 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.5.0/24\"":{}},"f:taints":{}}} } {kubelet Update v1 2023-03-25 05:42:05 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.5.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-1b/i-0168f82458996cc53,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/control-plane,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.5.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3892310016 0} {<nil>} 3801084Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3787452416 0} {<nil>} 3698684Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-03-25 05:20:22 +0000 UTC,LastTransitionTime:2023-03-25 05:20:22 +0000 UTC,Reason:CalicoIsUp,Message:Calico is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-25 05:42:05 +0000 UTC,LastTransitionTime:2023-03-25 05:18:50 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-25 05:42:05 +0000 UTC,LastTransitionTime:2023-03-25 05:18:50 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-25 05:42:05 +0000 UTC,LastTransitionTime:2023-03-25 05:18:50 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-25 05:42:05 +0000 UTC,LastTransitionTime:2023-03-25 05:20:29 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.48.81,},NodeAddress{Type:ExternalIP,Address:3.101.135.77,},NodeAddress{Type:InternalDNS,Address:i-0168f82458996cc53.us-west-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-0168f82458996cc53.us-west-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-3-101-135-77.us-west-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2d1ccc7ebf313420bd1f530a94fbfd,SystemUUID:ec2d1ccc-7ebf-3134-20bd-1f530a94fbfd,BootID:15cee321-56e7-4290-ab48-47b60b0bc657,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.28.0-alpha.0,KubeProxyVersion:v1.28.0-alpha.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcd@sha256:dd75ec974b0a2a6f6bb47001ba09207976e625db898d1b16735528c009cb171c registry.k8s.io/etcd:3.5.6-0],SizeBytes:102542580,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:6f72b851544986cb0921b53ea655ec04c36131248f16d4ad110cb3ca0c369dc1 registry.k8s.io/etcd:3.5.4-0],SizeBytes:102157811,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:13f53ed1d91e2e11aac476ee9a0269fdda6cc4874eba903efd40daf50c55eee5 registry.k8s.io/etcd:3.5.3-0],SizeBytes:102143581,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:51eae8381dcb1078289fa7b4f3df2630cdc18d09fb56f8e56b41c40e191d6c83 registry.k8s.io/etcd:3.5.7-0],SizeBytes:101639218,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:4afb99b4690b418ffc2ceb67e1a17376457e441c1f09ab55447f0aaf992fa646 registry.k8s.io/etcd:3.4.3-0],SizeBytes:100947667,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:9ce33ba33d8e738a5b85ed50b5080ac746deceed4a7496c550927a7a19ca3b6d registry.k8s.io/etcd:3.5.0-0],SizeBytes:99868722,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:64b9ea357325d5db9f8a723dcf503b5a449177b17ac87d69481e126bb724c263 registry.k8s.io/etcd:3.5.1-0],SizeBytes:98888614,},ContainerImage{Names:[docker.io/calico/cni@sha256:e282ea2914c806b5de2976330a17cfb5e6dcef47147bceb1432ca5c75fd46f50 docker.io/calico/cni:v3.24.5],SizeBytes:87458443,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:4ad90a11b55313b182afc186b9876c8e891531b8db4c9bf1541953021618d0e2 registry.k8s.io/etcd:3.4.13-0],SizeBytes:86742272,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:0d2ab6b15b4fe0b903b9e81192d942fb39c29be61fe341cd16db5d0165b0435c registry.k8s.io/etcd:3.3.17-0],SizeBytes:85524408,},ContainerImage{Names:[docker.io/calico/node@sha256:5972ad2bcbdc668564d3e26960c9c513b2d7b05581c704747cf7c62ef3a405a6 docker.io/calico/node:v3.24.5],SizeBytes:81584866,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:17da501f5d2a675be46040422a27b7cc21b8a43895ac998b171db1c346f361f7 registry.k8s.io/etcd:3.3.10-0],SizeBytes:76162358,},ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager-slim@sha256:8727adc059334e0e3329dbccce116a734da4a090da5e86c8cddccaf333caa666 registry.k8s.io/etcdadm/etcd-manager-slim:v3.0.20230201],SizeBytes:63926165,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:905d7ca17fd02bc24c0eba9a062753aba15db3e31422390bc3238eb762339b20 registry.k8s.io/etcd:3.2.24-1],SizeBytes:63423287,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.27.0-alpha.1],SizeBytes:45219545,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.27.0-alpha.1],SizeBytes:43628585,},ContainerImage{Names:[registry.k8s.io/kube-apiserver@sha256:49934a29c559e79ed19fc38b4697b18d8fe174b17bf56192588575f980c93956 registry.k8s.io/kube-apiserver:v1.28.0-alpha.0],SizeBytes:33212506,},ContainerImage{Names:[docker.io/calico/kube-controllers@sha256:2b6acd7f677f76ffe12ecf3ea7df92eb9b1bdb07336d1ac2a54c7631fb753f7e docker.io/calico/kube-controllers:v3.24.5],SizeBytes:31137169,},ContainerImage{Names:[registry.k8s.io/kube-controller-manager@sha256:974748384634982c73527fdb3c9ec0ec476bc0d52b3dd1d611a380b7f99af1f3 registry.k8s.io/kube-controller-manager:v1.28.0-alpha.0],SizeBytes:30820280,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/kube-proxy@sha256:8a03b3147cd7be8e8235ae1cd70cf69aca9ce03ba1cfaffea320540bb5afa496 registry.k8s.io/kube-proxy:v1.28.0-alpha.0],SizeBytes:23896850,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:23345856,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:22381475,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:22085298,},ContainerImage{Names:[registry.k8s.io/provider-aws/cloud-controller-manager@sha256:fdeb61e3e42ecd9cca868d550ebdb88dd6341d9e91fcfa9a37e227dab2ad22cb registry.k8s.io/provider-aws/cloud-controller-manager:v1.26.0],SizeBytes:20154862,},ContainerImage{Names:[registry.k8s.io/kube-scheduler@sha256:cbf4007abb076188f16b975bc2b74452603cc7b34c60a29d92f14c38ad02dc11 registry.k8s.io/kube-scheduler:v1.28.0-alpha.0],SizeBytes:18078674,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.27.0-alpha.1],SizeBytes:5074298,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Mar 25 05:45:35.001: INFO: Logging kubelet events for node i-0168f82458996cc53 Mar 25 05:45:35.057: INFO: Logging pods the kubelet thinks is on node i-0168f82458996cc53 Mar 25 05:45:35.115: INFO: kube-proxy-i-0168f82458996cc53 started at 2023-03-25 05:17:36 +0000 UTC (0+1 container statuses recorded) Mar 25 05:45:35.115: INFO: Container kube-proxy ready: true, restart count 0 Mar 25 05:45:35.115: INFO: kube-scheduler-i-0168f82458996cc53 started at 2023-03-25 05:17:36 +0000 UTC (0+1 container statuses recorded) Mar 25 05:45:35.115: INFO: Container kube-scheduler ready: true, restart count 0 Mar 25 05:45:35.115: INFO: etcd-manager-main-i-0168f82458996cc53 started at 2023-03-25 05:17:36 +0000 UTC (11+1 container statuses recorded) Mar 25 05:45:35.115: INFO: Init container init-etcd-3-2-24 ready: true, restart count 0 Mar 25 05:45:35.116: INFO: Init container init-etcd-3-3-10 ready: true, restart count 0 Mar 25 05:45:35.116: INFO: Init container init-etcd-3-3-17 ready: true, restart count 0 Mar 25 05:45:35.116: INFO: Init container init-etcd-3-4-13 ready: true, restart count 0 Mar 25 05:45:35.116: INFO: Init container init-etcd-3-4-3 ready: true, restart count 0 Mar 25 05:45:35.116: INFO: Init container init-etcd-3-5-0 ready: true, restart count 0 Mar 25 05:45:35.116: INFO: Init container init-etcd-3-5-1 ready: true, restart count 0 Mar 25 05:45:35.116: INFO: Init container init-etcd-3-5-3 ready: true, restart count 0 Mar 25 05:45:35.116: INFO: Init container init-etcd-3-5-4 ready: true, restart count 0 Mar 25 05:45:35.116: INFO: Init container init-etcd-3-5-6 ready: true, restart count 0 Mar 25 05:45:35.116: INFO: Init container init-etcd-3-5-7 ready: true, restart count 0 Mar 25 05:45:35.116: INFO: Container etcd-manager ready: true, restart count 0 Mar 25 05:45:35.116: INFO: dns-controller-69987775c6-hbn69 started at 2023-03-25 05:19:47 +0000 UTC (0+1 container statuses recorded) Mar 25 05:45:35.116: INFO: Container dns-controller ready: true, restart count 0 Mar 25 05:45:35.116: INFO: aws-cloud-controller-manager-26vbv started at 2023-03-25 05:20:06 +0000 UTC (0+1 container statuses recorded) Mar 25 05:45:35.116: INFO: Container aws-cloud-controller-manager ready: true, restart count 0 Mar 25 05:45:35.116: INFO: ebs-csi-node-zjtj6 started at 2023-03-25 05:20:06 +0000 UTC (0+3 container statuses recorded) Mar 25 05:45:35.116: INFO: Container ebs-plugin ready: true, restart count 0 Mar 25 05:45:35.116: INFO: Container liveness-probe ready: true, restart count 0 Mar 25 05:45:35.116: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 25 05:45:35.116: INFO: kops-controller-cgw95 started at 2023-03-25 05:20:06 +0000 UTC (0+1 container statuses recorded) Mar 25 05:45:35.116: INFO: Container kops-controller ready: true, restart count 0 Mar 25 05:45:35.116: INFO: calico-kube-controllers-55f488656-rr8rz started at 2023-03-25 05:20:29 +0000 UTC (0+1 container statuses recorded) Mar 25 05:45:35.116: INFO: Container calico-kube-controllers ready: true, restart count 0 Mar 25 05:45:35.116: INFO: ebs-csi-controller-79f984545f-9q5ks started at 2023-03-25 05:21:26 +0000 UTC (0+5 container statuses recorded) Mar 25 05:45:35.116: INFO: Container csi-attacher ready: true, restart count 0 Mar 25 05:45:35.116: INFO: Container csi-provisioner ready: true, restart count 0 Mar 25 05:45:35.116: INFO: Container csi-resizer ready: true, restart count 0 Mar 25 05:45:35.116: INFO: Container ebs-plugin ready: true, restart count 0 Mar 25 05:45:35.116: INFO: Container liveness-probe ready: true, restart count 0 Mar 25 05:45:35.116: INFO: kube-apiserver-i-0168f82458996cc53 started at 2023-03-25 05:17:36 +0000 UTC (0+2 container statuses recorded) Mar 25 05:45:35.116: INFO: Container healthcheck ready: true, restart count 0 Mar 25 05:45:35.116: INFO: Container kube-apiserver ready: true, restart count 2 Mar 25 05:45:35.116: INFO: calico-node-rdjq2 started at 2023-03-25 05:20:06 +0000 UTC (3+1 container statuses recorded) Mar 25 05:45:35.116: INFO: Init container upgrade-ipam ready: true, restart count 0 Mar 25 05:45:35.116: INFO: Init container install-cni ready: true, restart count 0 Mar 25 05:45:35.116: INFO: Init container mount-bpffs ready: true, restart count 0 Mar 25 05:45:35.116: INFO: Container calico-node ready: true, restart count 0 Mar 25 05:45:35.116: INFO: kube-controller-manager-i-0168f82458996cc53 started at 2023-03-25 05:17:36 +0000 UTC (0+1 container statuses recorded) Mar 25 05:45:35.116: INFO: Container kube-controller-manager ready: true, restart count 4 Mar 25 05:45:35.116: INFO: etcd-manager-events-i-0168f82458996cc53 started at 2023-03-25 05:17:36 +0000 UTC (11+1 container statuses recorded) Mar 25 05:45:35.116: INFO: Init container init-etcd-3-2-24 ready: true, restart count 0 Mar 25 05:45:35.116: INFO: Init container init-etcd-3-3-10 ready: true, restart count 0 Mar 25 05:45:35.116: INFO: Init container init-etcd-3-3-17 ready: true, restart count 0 Mar 25 05:45:35.116: INFO: Init container init-etcd-3-4-13 ready: true, restart count 0 Mar 25 05:45:35.116: INFO: Init container init-etcd-3-4-3 ready: true, restart count 0 Mar 25 05:45:35.116: INFO: Init container init-etcd-3-5-0 ready: true, restart count 0 Mar 25 05:45:35.116: INFO: Init container init-etcd-3-5-1 ready: true, restart count 0 Mar 25 05:45:35.116: INFO: Init container init-etcd-3-5-3 ready: true, restart count 0 Mar 25 05:45:35.116: INFO: Init container init-etcd-3-5-4 ready: true, restart count 0 Mar 25 05:45:35.116: INFO: Init container init-etcd-3-5-6 ready: true, restart count 0 Mar 25 05:45:35.116: INFO: Init container init-etcd-3-5-7 ready: true, restart count 0 Mar 25 05:45:35.116: INFO: Container etcd-manager ready: true, restart count 0 Mar 25 05:45:35.339: INFO: Latency metrics for node i-0168f82458996cc53 Mar 25 05:45:35.339: INFO: Logging node info for node i-03d301e6f030748f1 Mar 25 05:45:35.389: INFO: Node Info: &Node{ObjectMeta:{i-03d301e6f030748f1 897f52bd-65e2-4107-843a-37e36aa74032 31956 0 2023-03-25 05:31:35 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-1 failure-domain.beta.kubernetes.io/zone:us-west-1b kubernetes.io/arch:amd64 kubernetes.io/hostname:i-03d301e6f030748f1 kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:us-west-1b topology.hostpath.csi/node:i-03d301e6f030748f1 topology.kubernetes.io/region:us-west-1 topology.kubernetes.io/zone:us-west-1b] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-03d301e6f030748f1"} node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:172.20.50.199/19 projectcalico.org/IPv4IPIPTunnelAddr:100.110.167.64 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-03-25 05:31:35 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-25 05:31:35 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-03-25 05:31:35 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2023-03-25 05:31:35 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2023-03-25 05:31:48 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.8.0/24\"":{}}}} } {calico-node Update v1 2023-03-25 05:31:56 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4IPIPTunnelAddr":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2023-03-25 05:45:12 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-03-25 05:45:12 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.8.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-1b/i-03d301e6f030748f1,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.8.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-03-25 05:31:56 +0000 UTC,LastTransitionTime:2023-03-25 05:31:56 +0000 UTC,Reason:CalicoIsUp,Message:Calico is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-25 05:45:12 +0000 UTC,LastTransitionTime:2023-03-25 05:31:35 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-25 05:45:12 +0000 UTC,LastTransitionTime:2023-03-25 05:31:35 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-25 05:45:12 +0000 UTC,LastTransitionTime:2023-03-25 05:31:35 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-25 05:45:12 +0000 UTC,LastTransitionTime:2023-03-25 05:31:48 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.50.199,},NodeAddress{Type:ExternalIP,Address:52.53.245.63,},NodeAddress{Type:InternalDNS,Address:i-03d301e6f030748f1.us-west-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-03d301e6f030748f1.us-west-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-52-53-245-63.us-west-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec27152f7af9b9bfd1355ee19c256086,SystemUUID:ec27152f-7af9-b9bf-d135-5ee19c256086,BootID:1c303852-f2ce-45f9-ab14-6e8de97a0a3e,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.28.0-alpha.0,KubeProxyVersion:v1.28.0-alpha.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[docker.io/calico/cni@sha256:e282ea2914c806b5de2976330a17cfb5e6dcef47147bceb1432ca5c75fd46f50 docker.io/calico/cni:v3.24.5],SizeBytes:87458443,},ContainerImage{Names:[docker.io/calico/node@sha256:5972ad2bcbdc668564d3e26960c9c513b2d7b05581c704747cf7c62ef3a405a6 docker.io/calico/node:v3.24.5],SizeBytes:81584866,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nautilus@sha256:80ba6c8c44f9623f06e868a1aa66026c8ec438ad814f9ec95e9333b415fe3550 registry.k8s.io/e2e-test-images/nautilus:1.7],SizeBytes:49641698,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41901587,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:e468dddcd275163a042ab297b2d8c2aca50d5e148d2d22f3b6ba119e2f31fa79 registry.k8s.io/sig-storage/csi-provisioner:v3.4.0],SizeBytes:27427836,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/kube-proxy@sha256:8a03b3147cd7be8e8235ae1cd70cf69aca9ce03ba1cfaffea320540bb5afa496 registry.k8s.io/kube-proxy:v1.28.0-alpha.0],SizeBytes:23896850,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:23881995,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18758628,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18233005,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonroot@sha256:ee9f50b3c64b174d296d91ca9f69a914ac30e59095dfb462b2b518ad28a63655 registry.k8s.io/e2e-test-images/nonroot:1.4],SizeBytes:17747885,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8688564,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:7000082,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6978614,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac registry.k8s.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[kubernetes.io/csi/ebs.csi.aws.com^vol-0dec4dd6bb50184f2],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-0dec4dd6bb50184f2,DevicePath:,},},Config:nil,},} Mar 25 05:45:35.389: INFO: Logging kubelet events for node i-03d301e6f030748f1 Mar 25 05:45:35.454: INFO: Logging pods the kubelet thinks is on node i-03d301e6f030748f1 Mar 25 05:45:35.512: INFO: downward-api-93f2a44f-812e-4c13-97f2-4a34a97fad36 started at 2023-03-25 05:45:33 +0000 UTC (0+1 container statuses recorded) Mar 25 05:45:35.512: INFO: Container dapi-container ready: false, restart count 0 Mar 25 05:45:35.512: INFO: ss-1 started at 2023-03-25 05:45:08 +0000 UTC (0+1 container statuses recorded) Mar 25 05:45:35.512: INFO: Container webserver ready: true, restart count 0 Mar 25 05:45:35.512: INFO: ebs-csi-node-lljwl started at 2023-03-25 05:31:35 +0000 UTC (0+3 container statuses recorded) Mar 25 05:45:35.512: INFO: Container ebs-plugin ready: true, restart count 0 Mar 25 05:45:35.512: INFO: Container liveness-probe ready: true, restart count 0 Mar 25 05:45:35.512: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 25 05:45:35.512: INFO: test-ss-0 started at 2023-03-25 05:45:16 +0000 UTC (0+1 container statuses recorded) Mar 25 05:45:35.512: INFO: Container webserver ready: true, restart count 0 Mar 25 05:45:35.512: INFO: kube-proxy-i-03d301e6f030748f1 started at 2023-03-25 05:31:35 +0000 UTC (0+1 container statuses recorded) Mar 25 05:45:35.512: INFO: Container kube-proxy ready: true, restart count 0 Mar 25 05:45:35.512: INFO: rc-test-mjwk8 started at 2023-03-25 05:45:34 +0000 UTC (0+1 container statuses recorded) Mar 25 05:45:35.512: INFO: Container rc-test ready: true, restart count 0 Mar 25 05:45:35.512: INFO: pfpod started at 2023-03-25 05:44:26 +0000 UTC (0+2 container statuses recorded) Mar 25 05:45:35.512: INFO: Container portforwardtester ready: false, restart count 0 Mar 25 05:45:35.512: INFO: Container readiness ready: false, restart count 0 Mar 25 05:45:35.512: INFO: busybox-06aeea52-3822-43de-9b3c-1a9c004257fc started at 2023-03-25 05:45:16 +0000 UTC (0+1 container statuses recorded) Mar 25 05:45:35.512: INFO: Container busybox ready: true, restart count 0 Mar 25 05:45:35.512: INFO: ss2-1 started at 2023-03-25 05:45:33 +0000 UTC (0+1 container statuses recorded) Mar 25 05:45:35.512: INFO: Container webserver ready: true, restart count 0 Mar 25 05:45:35.512: INFO: startup-script started at 2023-03-25 05:44:37 +0000 UTC (0+1 container statuses recorded) Mar 25 05:45:35.512: INFO: Container startup-script ready: true, restart count 0 Mar 25 05:45:35.512: INFO: pod-configmaps-a93186b9-eae5-49fb-9839-a908e7cb30ca started at 2023-03-25 05:44:09 +0000 UTC (0+2 container statuses recorded) Mar 25 05:45:35.512: INFO: Container agnhost-container ready: true, restart count 0 Mar 25 05:45:35.512: INFO: Container configmap-volume-binary-test ready: false, restart count 0 Mar 25 05:45:35.512: INFO: pod-ephm-test-projected-rxqb started at 2023-03-25 05:44:02 +0000 UTC (0+1 container statuses recorded) Mar 25 05:45:35.512: INFO: Container test-container-subpath-projected-rxqb ready: false, restart count 0 Mar 25 05:45:35.512: INFO: calico-node-h96vq started at 2023-03-25 05:31:35 +0000 UTC (3+1 container statuses recorded) Mar 25 05:45:35.512: INFO: Init container upgrade-ipam ready: true, restart count 0 Mar 25 05:45:35.512: INFO: Init container install-cni ready: true, restart count 0 Mar 25 05:45:35.512: INFO: Init container mount-bpffs ready: true, restart count 0 Mar 25 05:45:35.512: INFO: Container calico-node ready: true, restart count 0 Mar 25 05:45:35.747: INFO: Latency metrics for node i-03d301e6f030748f1 Mar 25 05:45:35.747: INFO: Logging node info for node i-049fdb2c1cde894df Mar 25 05:45:35.796: INFO: Node Info: &Node{ObjectMeta:{i-049fdb2c1cde894df b43a51e6-dda7-4047-b368-72d515695b4e 32870 0 2023-03-25 05:23:34 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-1 failure-domain.beta.kubernetes.io/zone:us-west-1b io.kubernetes.storage.mock/node:some-mock-node kubernetes.io/arch:amd64 kubernetes.io/hostname:i-049fdb2c1cde894df kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:us-west-1b topology.hostpath.csi/node:i-049fdb2c1cde894df topology.kubernetes.io/region:us-west-1 topology.kubernetes.io/zone:us-west-1b] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-049fdb2c1cde894df"} node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:172.20.43.61/19 projectcalico.org/IPv4IPIPTunnelAddr:100.97.18.64 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-03-25 05:23:34 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-25 05:23:34 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-03-25 05:23:34 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2023-03-25 05:23:34 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2023-03-25 05:23:45 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.6.0/24\"":{}}}} } {calico-node Update v1 2023-03-25 05:23:52 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4IPIPTunnelAddr":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2023-03-25 05:45:31 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:io.kubernetes.storage.mock/node":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status} {kube-controller-manager Update v1 2023-03-25 05:45:32 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.6.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-1b/i-049fdb2c1cde894df,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.6.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-03-25 05:23:52 +0000 UTC,LastTransitionTime:2023-03-25 05:23:52 +0000 UTC,Reason:CalicoIsUp,Message:Calico is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-25 05:45:31 +0000 UTC,LastTransitionTime:2023-03-25 05:23:34 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-25 05:45:31 +0000 UTC,LastTransitionTime:2023-03-25 05:23:34 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-25 05:45:31 +0000 UTC,LastTransitionTime:2023-03-25 05:23:34 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-25 05:45:31 +0000 UTC,LastTransitionTime:2023-03-25 05:23:45 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.43.61,},NodeAddress{Type:ExternalIP,Address:18.144.75.67,},NodeAddress{Type:InternalDNS,Address:i-049fdb2c1cde894df.us-west-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-049fdb2c1cde894df.us-west-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-18-144-75-67.us-west-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec252e592d19e1e6c583e87505431016,SystemUUID:ec252e59-2d19-e1e6-c583-e87505431016,BootID:45436b5d-1eb0-4b6f-a1d1-a1492b31b5d7,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.28.0-alpha.0,KubeProxyVersion:v1.28.0-alpha.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:112030336,},ContainerImage{Names:[docker.io/calico/cni@sha256:e282ea2914c806b5de2976330a17cfb5e6dcef47147bceb1432ca5c75fd46f50 docker.io/calico/cni:v3.24.5],SizeBytes:87458443,},ContainerImage{Names:[docker.io/calico/node@sha256:5972ad2bcbdc668564d3e26960c9c513b2d7b05581c704747cf7c62ef3a405a6 docker.io/calico/node:v3.24.5],SizeBytes:81584866,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:e468dddcd275163a042ab297b2d8c2aca50d5e148d2d22f3b6ba119e2f31fa79 registry.k8s.io/sig-storage/csi-provisioner:v3.4.0],SizeBytes:27427836,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/kube-proxy@sha256:8a03b3147cd7be8e8235ae1cd70cf69aca9ce03ba1cfaffea320540bb5afa496 registry.k8s.io/kube-proxy:v1.28.0-alpha.0],SizeBytes:23896850,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:23881995,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:68d396900aeaa072c1f27289485fdac29834045a6f3ffe369bf389d830ef572d registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.6],SizeBytes:20293261,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18758628,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18233005,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:15273057,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8688564,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:7000082,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6978614,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[kubernetes.io/csi/ebs.csi.aws.com^vol-00e9b723211b131bc kubernetes.io/csi/ebs.csi.aws.com^vol-0bed540f628372c85],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-00e9b723211b131bc,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-0bed540f628372c85,DevicePath:,},},Config:nil,},} Mar 25 05:45:35.796: INFO: Logging kubelet events for node i-049fdb2c1cde894df Mar 25 05:45:35.856: INFO: Logging pods the kubelet thinks is on node i-049fdb2c1cde894df Mar 25 05:45:35.932: INFO: coredns-autoscaler-7cb5c5b969-h485x started at 2023-03-25 05:24:36 +0000 UTC (0+1 container statuses recorded) Mar 25 05:45:35.932: INFO: Container autoscaler ready: true, restart count 0 Mar 25 05:45:35.932: INFO: inline-volume-tester-5j9fs started at 2023-03-25 05:45:05 +0000 UTC (0+1 container statuses recorded) Mar 25 05:45:35.932: INFO: Container csi-volume-tester ready: true, restart count 0 Mar 25 05:45:35.932: INFO: csi-mockplugin-resizer-0 started at 2023-03-25 05:45:32 +0000 UTC (0+1 container statuses recorded) Mar 25 05:45:35.932: INFO: Container csi-resizer ready: true, restart count 0 Mar 25 05:45:35.932: INFO: annotationupdatee5e5d8ca-80ad-4a9d-8c8d-8a995c448ccc started at 2023-03-25 05:43:58 +0000 UTC (0+1 container statuses recorded) Mar 25 05:45:35.932: INFO: Container client-container ready: true, restart count 0 Mar 25 05:45:35.932: INFO: labelsupdate8d6b443f-1225-422c-a63e-1271aef97f34 started at 2023-03-25 05:44:27 +0000 UTC (0+1 container statuses recorded) Mar 25 05:45:35.932: INFO: Container client-container ready: true, restart count 0 Mar 25 05:45:35.932: INFO: bin-false92092514-d86f-443e-9f03-8be47f3e24e4 started at 2023-03-25 05:43:44 +0000 UTC (0+1 container statuses recorded) Mar 25 05:45:35.932: INFO: Container bin-false92092514-d86f-443e-9f03-8be47f3e24e4 ready: false, restart count 0 Mar 25 05:45:35.932: INFO: test-grpc-c9016762-a9d2-412b-87fb-e1be3cf11d20 started at 2023-03-25 05:45:01 +0000 UTC (0+1 container statuses recorded) Mar 25 05:45:35.932: INFO: Container agnhost ready: true, restart count 0 Mar 25 05:45:35.932: INFO: ss2-0 started at 2023-03-25 05:45:32 +0000 UTC (0+1 container statuses recorded) Mar 25 05:45:35.932: INFO: Container webserver ready: true, restart count 0 Mar 25 05:45:35.932: INFO: rc-test-rj655 started at 2023-03-25 05:45:32 +0000 UTC (0+1 container statuses recorded) Mar 25 05:45:35.932: INFO: Container rc-test ready: true, restart count 0 Mar 25 05:45:35.932: INFO: kube-proxy-i-049fdb2c1cde894df started at 2023-03-25 05:23:35 +0000 UTC (0+1 container statuses recorded) Mar 25 05:45:35.932: INFO: Container kube-proxy ready: true, restart count 0 Mar 25 05:45:35.932: INFO: ebs-csi-node-m9zxz started at 2023-03-25 05:23:35 +0000 UTC (0+3 container statuses recorded) Mar 25 05:45:35.932: INFO: Container ebs-plugin ready: true, restart count 0 Mar 25 05:45:35.932: INFO: Container liveness-probe ready: true, restart count 0 Mar 25 05:45:35.932: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 25 05:45:35.932: INFO: calico-node-wt5p7 started at 2023-03-25 05:23:35 +0000 UTC (3+1 container statuses recorded) Mar 25 05:45:35.932: INFO: Init container upgrade-ipam ready: true, restart count 0 Mar 25 05:45:35.932: INFO: Init container install-cni ready: true, restart count 0 Mar 25 05:45:35.932: INFO: Init container mount-bpffs ready: true, restart count 0 Mar 25 05:45:35.932: INFO: Container calico-node ready: true, restart count 0 Mar 25 05:45:35.932: INFO: coredns-559769c974-pgqhl started at 2023-03-25 05:24:36 +0000 UTC (0+1 container statuses recorded) Mar 25 05:45:35.932: INFO: Container coredns ready: true, restart count 0 Mar 25 05:45:35.932: INFO: ss-0 started at 2023-03-25 05:44:46 +0000 UTC (0+1 container statuses recorded) Mar 25 05:45:35.932: INFO: Container webserver ready: true, restart count 0 Mar 25 05:45:35.932: INFO: csi-mockplugin-0 started at 2023-03-25 05:45:32 +0000 UTC (0+3 container statuses recorded) Mar 25 05:45:35.932: INFO: Container csi-provisioner ready: true, restart count 0 Mar 25 05:45:35.932: INFO: Container driver-registrar ready: true, restart count 0 Mar 25 05:45:35.932: INFO: Container mock ready: true, restart count 0 Mar 25 05:45:36.198: INFO: Latency metrics for node i-049fdb2c1cde894df Mar 25 05:45:36.198: INFO: Logging node info for node i-062679bb8c0ba4bce Mar 25 05:45:36.246: INFO: Node Info: &Node{ObjectMeta:{i-062679bb8c0ba4bce ee2da569-bb88-4297-9e6f-d2b15ba7ce86 32934 0 2023-03-25 05:27:31 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-1 failure-domain.beta.kubernetes.io/zone:us-west-1b kubernetes.io/arch:amd64 kubernetes.io/hostname:i-062679bb8c0ba4bce kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:us-west-1b topology.hostpath.csi/node:i-062679bb8c0ba4bce topology.kubernetes.io/region:us-west-1 topology.kubernetes.io/zone:us-west-1b] map[csi.volume.kubernetes.io/nodeid:{"csi-mock-csi-mock-volumes-expansion-827":"csi-mock-csi-mock-volumes-expansion-827","ebs.csi.aws.com":"i-062679bb8c0ba4bce"} node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:172.20.45.43/19 projectcalico.org/IPv4IPIPTunnelAddr:100.106.79.0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-03-25 05:27:31 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-25 05:27:31 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-03-25 05:27:31 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2023-03-25 05:27:31 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2023-03-25 05:27:44 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.7.0/24\"":{}}}} } {calico-node Update v1 2023-03-25 05:27:50 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4IPIPTunnelAddr":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2023-03-25 05:45:32 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.7.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-1b/i-062679bb8c0ba4bce,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.7.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051689472 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946831872 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-03-25 05:27:50 +0000 UTC,LastTransitionTime:2023-03-25 05:27:50 +0000 UTC,Reason:CalicoIsUp,Message:Calico is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-25 05:45:32 +0000 UTC,LastTransitionTime:2023-03-25 05:27:31 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-25 05:45:32 +0000 UTC,LastTransitionTime:2023-03-25 05:27:31 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-25 05:45:32 +0000 UTC,LastTransitionTime:2023-03-25 05:27:31 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-25 05:45:32 +0000 UTC,LastTransitionTime:2023-03-25 05:27:44 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.45.43,},NodeAddress{Type:ExternalIP,Address:3.101.133.108,},NodeAddress{Type:InternalDNS,Address:i-062679bb8c0ba4bce.us-west-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-062679bb8c0ba4bce.us-west-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-3-101-133-108.us-west-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2528f599e9b1e199793fb5cd06d205,SystemUUID:ec2528f5-99e9-b1e1-9979-3fb5cd06d205,BootID:ff56b70b-c8c5-46db-8cc5-4353c8f535c6,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.28.0-alpha.0,KubeProxyVersion:v1.28.0-alpha.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:112030336,},ContainerImage{Names:[docker.io/calico/cni@sha256:e282ea2914c806b5de2976330a17cfb5e6dcef47147bceb1432ca5c75fd46f50 docker.io/calico/cni:v3.24.5],SizeBytes:87458443,},ContainerImage{Names:[docker.io/calico/node@sha256:5972ad2bcbdc668564d3e26960c9c513b2d7b05581c704747cf7c62ef3a405a6 docker.io/calico/node:v3.24.5],SizeBytes:81584866,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nautilus@sha256:80ba6c8c44f9623f06e868a1aa66026c8ec438ad814f9ec95e9333b415fe3550 registry.k8s.io/e2e-test-images/nautilus:1.7],SizeBytes:49641698,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41901587,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:e468dddcd275163a042ab297b2d8c2aca50d5e148d2d22f3b6ba119e2f31fa79 registry.k8s.io/sig-storage/csi-provisioner:v3.4.0],SizeBytes:27427836,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/kube-proxy@sha256:8a03b3147cd7be8e8235ae1cd70cf69aca9ce03ba1cfaffea320540bb5afa496 registry.k8s.io/kube-proxy:v1.28.0-alpha.0],SizeBytes:23896850,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:23881995,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18758628,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/regression-issue-74839@sha256:b4f1d8d61bdad84bd50442d161d5460e4019d53e989b64220fdbc62fc87d76bf registry.k8s.io/e2e-test-images/regression-issue-74839:1.2],SizeBytes:18651485,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18233005,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:15273057,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8688564,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/build-image/distroless-iptables@sha256:13ac9b3c476d7290a4451d65d09d6016a2cb89836ffbfa4eae55b72731a22080 registry.k8s.io/build-image/distroless-iptables:v0.2.2],SizeBytes:7729580,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:7000082,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6978614,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Mar 25 05:45:36.247: INFO: Logging kubelet events for node i-062679bb8c0ba4bce Mar 25 05:45:36.302: INFO: Logging pods the kubelet thinks is on node i-062679bb8c0ba4bce Mar 25 05:45:36.358: INFO: ss2-2 started at 2023-03-25 05:45:34 +0000 UTC (0+1 container statuses recorded) Mar 25 05:45:36.358: INFO: Container webserver ready: true, restart count 0 Mar 25 05:45:36.358: INFO: csi-mockplugin-0 started at 2023-03-25 05:44:50 +0000 UTC (0+4 container statuses recorded) Mar 25 05:45:36.358: INFO: Container busybox ready: true, restart count 0 Mar 25 05:45:36.358: INFO: Container csi-provisioner ready: true, restart count 0 Mar 25 05:45:36.358: INFO: Container driver-registrar ready: true, restart count 0 Mar 25 05:45:36.358: INFO: Container mock ready: true, restart count 0 Mar 25 05:45:36.358: INFO: oidc-discovery-validator started at 2023-03-25 05:43:30 +0000 UTC (0+1 container statuses recorded) Mar 25 05:45:36.358: INFO: Container oidc-discovery-validator ready: false, restart count 0 Mar 25 05:45:36.358: INFO: kube-proxy-i-062679bb8c0ba4bce started at 2023-03-25 05:27:32 +0000 UTC (0+1 container statuses recorded) Mar 25 05:45:36.358: INFO: Container kube-proxy ready: true, restart count 0 Mar 25 05:45:36.358: INFO: calico-node-zwm5j started at 2023-03-25 05:27:32 +0000 UTC (3+1 container statuses recorded) Mar 25 05:45:36.358: INFO: Init container upgrade-ipam ready: true, restart count 0 Mar 25 05:45:36.358: INFO: Init container install-cni ready: true, restart count 0 Mar 25 05:45:36.358: INFO: Init container mount-bpffs ready: true, restart count 0 Mar 25 05:45:36.358: INFO: Container calico-node ready: true, restart count 0 Mar 25 05:45:36.358: INFO: csi-mockplugin-attacher-0 started at 2023-03-25 05:44:50 +0000 UTC (0+1 container statuses recorded) Mar 25 05:45:36.358: INFO: Container csi-attacher ready: true, restart count 0 Mar 25 05:45:36.358: INFO: csi-mockplugin-resizer-0 started at 2023-03-25 05:44:50 +0000 UTC (0+1 container statuses recorded) Mar 25 05:45:36.358: INFO: Container csi-resizer ready: true, restart count 0 Mar 25 05:45:36.358: INFO: coredns-559769c974-p9vg6 started at 2023-03-25 05:36:16 +0000 UTC (0+1 container statuses recorded) Mar 25 05:45:36.358: INFO: Container coredns ready: true, restart count 0 Mar 25 05:45:36.358: INFO: ebs-csi-node-7twrq started at 2023-03-25 05:27:32 +0000 UTC (0+3 container statuses recorded) Mar 25 05:45:36.358: INFO: Container ebs-plugin ready: true, restart count 0 Mar 25 05:45:36.358: INFO: Container liveness-probe ready: true, restart count 0 Mar 25 05:45:36.358: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 25 05:45:36.358: INFO: boom-server started at 2023-03-25 05:44:33 +0000 UTC (0+1 container statuses recorded) Mar 25 05:45:36.358: INFO: Container boom-server ready: true, restart count 0 Mar 25 05:45:36.358: INFO: rs-l7jm2 started at 2023-03-25 05:44:57 +0000 UTC (0+1 container statuses recorded) Mar 25 05:45:36.358: INFO: Container donothing ready: false, restart count 0 Mar 25 05:45:36.358: INFO: csi-mockplugin-0 started at 2023-03-25 05:42:40 +0000 UTC (0+4 container statuses recorded) Mar 25 05:45:36.358: INFO: Container busybox ready: true, restart count 0 Mar 25 05:45:36.358: INFO: Container csi-provisioner ready: true, restart count 0 Mar 25 05:45:36.358: INFO: Container driver-registrar ready: true, restart count 0 Mar 25 05:45:36.358: INFO: Container mock ready: true, restart count 0 Mar 25 05:45:36.587: INFO: Latency metrics for node i-062679bb8c0ba4bce Mar 25 05:45:36.587: INFO: Logging node info for node i-072608201d2c0f1ad Mar 25 05:45:36.642: INFO: Node Info: &Node{ObjectMeta:{i-072608201d2c0f1ad 264dfba1-3cf0-4c99-8e8c-098f0f792077 32906 0 2023-03-25 05:35:29 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-1 failure-domain.beta.kubernetes.io/zone:us-west-1b kubernetes.io/arch:amd64 kubernetes.io/hostname:i-072608201d2c0f1ad kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:us-west-1b topology.hostpath.csi/node:i-072608201d2c0f1ad topology.kubernetes.io/region:us-west-1 topology.kubernetes.io/zone:us-west-1b] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-provisioning-1196":"i-072608201d2c0f1ad","csi-hostpath-volume-8577":"i-072608201d2c0f1ad","ebs.csi.aws.com":"i-072608201d2c0f1ad"} node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:172.20.38.125/19 projectcalico.org/IPv4IPIPTunnelAddr:100.107.220.192 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-03-25 05:35:29 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-25 05:35:29 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-03-25 05:35:29 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2023-03-25 05:35:29 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2023-03-25 05:35:41 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.9.0/24\"":{}}}} } {calico-node Update v1 2023-03-25 05:35:48 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4IPIPTunnelAddr":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2023-03-25 05:45:32 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-03-25 05:45:32 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.9.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-1b/i-072608201d2c0f1ad,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.9.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051689472 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946831872 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-03-25 05:35:47 +0000 UTC,LastTransitionTime:2023-03-25 05:35:47 +0000 UTC,Reason:CalicoIsUp,Message:Calico is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-25 05:45:32 +0000 UTC,LastTransitionTime:2023-03-25 05:35:29 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-25 05:45:32 +0000 UTC,LastTransitionTime:2023-03-25 05:35:29 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-25 05:45:32 +0000 UTC,LastTransitionTime:2023-03-25 05:35:29 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-25 05:45:32 +0000 UTC,LastTransitionTime:2023-03-25 05:35:41 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.38.125,},NodeAddress{Type:ExternalIP,Address:3.101.88.10,},NodeAddress{Type:InternalDNS,Address:i-072608201d2c0f1ad.us-west-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-072608201d2c0f1ad.us-west-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-3-101-88-10.us-west-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2d3818c4f80a253dcc1a60e148dbc8,SystemUUID:ec2d3818-c4f8-0a25-3dcc-1a60e148dbc8,BootID:77d046cf-75ea-4a8c-970a-3190d64e019b,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.28.0-alpha.0,KubeProxyVersion:v1.28.0-alpha.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:112030336,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:51eae8381dcb1078289fa7b4f3df2630cdc18d09fb56f8e56b41c40e191d6c83 registry.k8s.io/etcd:3.5.7-0],SizeBytes:101639218,},ContainerImage{Names:[docker.io/calico/cni@sha256:e282ea2914c806b5de2976330a17cfb5e6dcef47147bceb1432ca5c75fd46f50 docker.io/calico/cni:v3.24.5],SizeBytes:87458443,},ContainerImage{Names:[docker.io/calico/node@sha256:5972ad2bcbdc668564d3e26960c9c513b2d7b05581c704747cf7c62ef3a405a6 docker.io/calico/node:v3.24.5],SizeBytes:81584866,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41901587,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:e468dddcd275163a042ab297b2d8c2aca50d5e148d2d22f3b6ba119e2f31fa79 registry.k8s.io/sig-storage/csi-provisioner:v3.4.0],SizeBytes:27427836,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/sample-apiserver@sha256:8d70890151aa5d096f331cb9da1b9cd5be0412b7363fe67b5c3befdcaa2a28d0 registry.k8s.io/e2e-test-images/sample-apiserver:1.17.7],SizeBytes:25667066,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/kube-proxy@sha256:8a03b3147cd7be8e8235ae1cd70cf69aca9ce03ba1cfaffea320540bb5afa496 registry.k8s.io/kube-proxy:v1.28.0-alpha.0],SizeBytes:23896850,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:23881995,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18758628,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18233005,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8688564,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:7000082,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6978614,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[kubernetes.io/csi/csi-hostpath-volume-8577^31b8707e-cad0-11ed-bd01-ea3526134491 kubernetes.io/csi/ebs.csi.aws.com^vol-046462550b75078b8],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-046462550b75078b8,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-volume-8577^31b8707e-cad0-11ed-bd01-ea3526134491,DevicePath:,},},Config:nil,},} Mar 25 05:45:36.642: INFO: Logging kubelet events for node i-072608201d2c0f1ad Mar 25 05:45:36.728: INFO: Logging pods the kubelet thinks is on node i-072608201d2c0f1ad Mar 25 05:45:36.800: INFO: ss-2 started at 2023-03-25 05:45:28 +0000 UTC (0+1 container statuses recorded) Mar 25 05:45:36.800: INFO: Container webserver ready: true, restart count 0 Mar 25 05:45:36.801: INFO: ebs-csi-node-r7mcm started at 2023-03-25 05:35:30 +0000 UTC (0+3 container statuses recorded) Mar 25 05:45:36.801: INFO: Container ebs-plugin ready: true, restart count 0 Mar 25 05:45:36.801: INFO: Container liveness-probe ready: true, restart count 0 Mar 25 05:45:36.801: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 25 05:45:36.801: INFO: pod-handle-http-request started at 2023-03-25 05:45:36 +0000 UTC (0+2 container statuses recorded) Mar 25 05:45:36.801: INFO: Container container-handle-http-request ready: false, restart count 0 Mar 25 05:45:36.801: INFO: Container container-handle-https-request ready: false, restart count 0 Mar 25 05:45:36.801: INFO: csi-hostpathplugin-0 started at 2023-03-25 05:45:02 +0000 UTC (0+7 container statuses recorded) Mar 25 05:45:36.801: INFO: Container csi-attacher ready: true, restart count 0 Mar 25 05:45:36.801: INFO: Container csi-provisioner ready: true, restart count 0 Mar 25 05:45:36.801: INFO: Container csi-resizer ready: true, restart count 0 Mar 25 05:45:36.801: INFO: Container csi-snapshotter ready: true, restart count 0 Mar 25 05:45:36.801: INFO: Container hostpath ready: true, restart count 0 Mar 25 05:45:36.801: INFO: Container liveness-probe ready: true, restart count 0 Mar 25 05:45:36.801: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 25 05:45:36.801: INFO: calico-node-lc9zl started at 2023-03-25 05:35:30 +0000 UTC (3+1 container statuses recorded) Mar 25 05:45:36.801: INFO: Init container upgrade-ipam ready: true, restart count 0 Mar 25 05:45:36.801: INFO: Init container install-cni ready: true, restart count 0 Mar 25 05:45:36.801: INFO: Init container mount-bpffs ready: true, restart count 0 Mar 25 05:45:36.801: INFO: Container calico-node ready: true, restart count 0 Mar 25 05:45:36.801: INFO: csi-hostpathplugin-0 started at 2023-03-25 05:45:02 +0000 UTC (0+7 container statuses recorded) Mar 25 05:45:36.801: INFO: Container csi-attacher ready: true, restart count 0 Mar 25 05:45:36.801: INFO: Container csi-provisioner ready: true, restart count 0 Mar 25 05:45:36.801: INFO: Container csi-resizer ready: true, restart count 0 Mar 25 05:45:36.801: INFO: Container csi-snapshotter ready: true, restart count 0 Mar 25 05:45:36.801: INFO: Container hostpath ready: true, restart count 0 Mar 25 05:45:36.801: INFO: Container liveness-probe ready: true, restart count 0 Mar 25 05:45:36.801: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 25 05:45:36.801: INFO: kube-proxy-i-072608201d2c0f1ad started at 2023-03-25 05:35:30 +0000 UTC (0+1 container statuses recorded) Mar 25 05:45:36.801: INFO: Container kube-proxy ready: true, restart count 0 Mar 25 05:45:37.153: INFO: Latency metrics for node i-072608201d2c0f1ad END STEP: dump namespace information after failure - test/e2e/framework/framework.go:288 @ 03/25/23 05:45:37.153 (2.4s) < Exit [DeferCleanup (Each)] [sig-auth] ServiceAccounts - dump namespaces | framework.go:209 @ 03/25/23 05:45:37.153 (2.4s) > Enter [DeferCleanup (Each)] [sig-auth] ServiceAccounts - tear down framework | framework.go:206 @ 03/25/23 05:45:37.153 STEP: Destroying namespace "svcaccounts-3613" for this suite. - test/e2e/framework/framework.go:351 @ 03/25/23 05:45:37.153 < Exit [DeferCleanup (Each)] [sig-auth] ServiceAccounts - tear down framework | framework.go:206 @ 03/25/23 05:45:37.21 (57ms) > Enter [ReportAfterEach] TOP-LEVEL - test/e2e/e2e_test.go:144 @ 03/25/23 05:45:37.21 < Exit [ReportAfterEach] TOP-LEVEL - test/e2e/e2e_test.go:144 @ 03/25/23 05:45:37.21 (0s)
Find logs mentions in log files | View test history on testgrid
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[It\]\s\[sig\-auth\]\sServiceAccounts\sshould\smount\sprojected\sservice\saccount\stoken\s\[Conformance\]$'
[FAILED] failed to get logs from test-pod-d961f1cf-6d1b-491f-969c-43d3ab0c1cbd for agnhost-container: Get "https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/svcaccounts-2502/pods/test-pod-d961f1cf-6d1b-491f-969c-43d3ab0c1cbd/log?container=agnhost-container&previous=false": stream error: stream ID 1265; INTERNAL_ERROR; received from peer In [It] at: test/e2e/framework/pod/output/output.go:247 @ 03/25/23 05:47:50.157
> Enter [BeforeEach] [sig-auth] ServiceAccounts - set up framework | framework.go:191 @ 03/25/23 05:47:45.383 STEP: Creating a kubernetes client - test/e2e/framework/framework.go:211 @ 03/25/23 05:47:45.383 Mar 25 05:47:45.383: INFO: >>> kubeConfig: /root/.kube/config STEP: Building a namespace api object, basename svcaccounts - test/e2e/framework/framework.go:250 @ 03/25/23 05:47:45.384 STEP: Waiting for a default service account to be provisioned in namespace - test/e2e/framework/framework.go:259 @ 03/25/23 05:47:45.537 STEP: Waiting for kube-root-ca.crt to be provisioned in namespace - test/e2e/framework/framework.go:262 @ 03/25/23 05:47:45.634 < Exit [BeforeEach] [sig-auth] ServiceAccounts - set up framework | framework.go:191 @ 03/25/23 05:47:45.731 (347ms) > Enter [BeforeEach] [sig-auth] ServiceAccounts - test/e2e/framework/metrics/init/init.go:33 @ 03/25/23 05:47:45.731 < Exit [BeforeEach] [sig-auth] ServiceAccounts - test/e2e/framework/metrics/init/init.go:33 @ 03/25/23 05:47:45.731 (0s) > Enter [It] should mount projected service account token [Conformance] - test/e2e/auth/service_accounts.go:275 @ 03/25/23 05:47:45.731 STEP: Creating a pod to test service account token: - test/e2e/framework/pod/output/output.go:243 @ 03/25/23 05:47:45.731 STEP: Saw pod success - test/e2e/framework/pod/wait.go:409 @ 03/25/23 05:47:49.954 Mar 25 05:47:50.003: INFO: Trying to get logs from node i-072608201d2c0f1ad pod test-pod-d961f1cf-6d1b-491f-969c-43d3ab0c1cbd container agnhost-container: <nil> Mar 25 05:47:50.052: INFO: Failed to get logs from node "i-072608201d2c0f1ad" pod "test-pod-d961f1cf-6d1b-491f-969c-43d3ab0c1cbd" container "agnhost-container". Get "https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/svcaccounts-2502/pods/test-pod-d961f1cf-6d1b-491f-969c-43d3ab0c1cbd/log?container=agnhost-container&previous=false": stream error: stream ID 1265; INTERNAL_ERROR; received from peer STEP: delete the pod - test/e2e/framework/pod/output/output.go:168 @ 03/25/23 05:47:50.052 Mar 25 05:47:50.157: INFO: Unexpected error: <*fmt.wrapError | 0xc0003c4940>: failed to get logs from test-pod-d961f1cf-6d1b-491f-969c-43d3ab0c1cbd for agnhost-container: Get "https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/svcaccounts-2502/pods/test-pod-d961f1cf-6d1b-491f-969c-43d3ab0c1cbd/log?container=agnhost-container&previous=false": stream error: stream ID 1265; INTERNAL_ERROR; received from peer { msg: "failed to get logs from test-pod-d961f1cf-6d1b-491f-969c-43d3ab0c1cbd for agnhost-container: Get \"https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/svcaccounts-2502/pods/test-pod-d961f1cf-6d1b-491f-969c-43d3ab0c1cbd/log?container=agnhost-container&previous=false\": stream error: stream ID 1265; INTERNAL_ERROR; received from peer", err: <*url.Error | 0xc003b9c7e0>{ Op: "Get", URL: "https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/svcaccounts-2502/pods/test-pod-d961f1cf-6d1b-491f-969c-43d3ab0c1cbd/log?container=agnhost-container&previous=false", Err: <http2.StreamError>{ StreamID: 1265, Code: 2, Cause: <*errors.errorString | 0xc000233650>{ s: "received from peer", }, }, }, } [FAILED] failed to get logs from test-pod-d961f1cf-6d1b-491f-969c-43d3ab0c1cbd for agnhost-container: Get "https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/svcaccounts-2502/pods/test-pod-d961f1cf-6d1b-491f-969c-43d3ab0c1cbd/log?container=agnhost-container&previous=false": stream error: stream ID 1265; INTERNAL_ERROR; received from peer In [It] at: test/e2e/framework/pod/output/output.go:247 @ 03/25/23 05:47:50.157 < Exit [It] should mount projected service account token [Conformance] - test/e2e/auth/service_accounts.go:275 @ 03/25/23 05:47:50.157 (4.427s) > Enter [AfterEach] [sig-auth] ServiceAccounts - test/e2e/framework/node/init/init.go:33 @ 03/25/23 05:47:50.157 Mar 25 05:47:50.157: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready < Exit [AfterEach] [sig-auth] ServiceAccounts - test/e2e/framework/node/init/init.go:33 @ 03/25/23 05:47:50.207 (50ms) > Enter [DeferCleanup (Each)] [sig-auth] ServiceAccounts - test/e2e/framework/metrics/init/init.go:35 @ 03/25/23 05:47:50.207 < Exit [DeferCleanup (Each)] [sig-auth] ServiceAccounts - test/e2e/framework/metrics/init/init.go:35 @ 03/25/23 05:47:50.207 (0s) > Enter [DeferCleanup (Each)] [sig-auth] ServiceAccounts - dump namespaces | framework.go:209 @ 03/25/23 05:47:50.207 STEP: dump namespace information after failure - test/e2e/framework/framework.go:288 @ 03/25/23 05:47:50.207 STEP: Collecting events from namespace "svcaccounts-2502". - test/e2e/framework/debug/dump.go:42 @ 03/25/23 05:47:50.207 STEP: Found 4 events. - test/e2e/framework/debug/dump.go:46 @ 03/25/23 05:47:50.305 Mar 25 05:47:50.305: INFO: At 2023-03-25 05:47:45 +0000 UTC - event for test-pod-d961f1cf-6d1b-491f-969c-43d3ab0c1cbd: {default-scheduler } Scheduled: Successfully assigned svcaccounts-2502/test-pod-d961f1cf-6d1b-491f-969c-43d3ab0c1cbd to i-072608201d2c0f1ad Mar 25 05:47:50.305: INFO: At 2023-03-25 05:47:46 +0000 UTC - event for test-pod-d961f1cf-6d1b-491f-969c-43d3ab0c1cbd: {kubelet i-072608201d2c0f1ad} Pulled: Container image "registry.k8s.io/e2e-test-images/agnhost:2.43" already present on machine Mar 25 05:47:50.305: INFO: At 2023-03-25 05:47:46 +0000 UTC - event for test-pod-d961f1cf-6d1b-491f-969c-43d3ab0c1cbd: {kubelet i-072608201d2c0f1ad} Created: Created container agnhost-container Mar 25 05:47:50.305: INFO: At 2023-03-25 05:47:46 +0000 UTC - event for test-pod-d961f1cf-6d1b-491f-969c-43d3ab0c1cbd: {kubelet i-072608201d2c0f1ad} Started: Started container agnhost-container Mar 25 05:47:50.354: INFO: POD NODE PHASE GRACE CONDITIONS Mar 25 05:47:50.354: INFO: Mar 25 05:47:50.404: INFO: Logging node info for node i-0168f82458996cc53 Mar 25 05:47:50.455: INFO: Node Info: &Node{ObjectMeta:{i-0168f82458996cc53 f4eb302c-c477-45ac-bcac-ffed65f6580f 38654 0 2023-03-25 05:18:57 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-1 failure-domain.beta.kubernetes.io/zone:us-west-1b kops.k8s.io/kops-controller-pki: kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0168f82458996cc53 kubernetes.io/os:linux node-role.kubernetes.io/control-plane: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:us-west-1b topology.kubernetes.io/region:us-west-1 topology.kubernetes.io/zone:us-west-1b] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0168f82458996cc53"} node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:172.20.48.81/19 projectcalico.org/IPv4IPIPTunnelAddr:100.100.215.192 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-03-25 05:18:57 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {protokube Update v1 2023-03-25 05:18:58 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {calico-node Update v1 2023-03-25 05:20:22 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4IPIPTunnelAddr":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {aws-cloud-controller-manager Update v1 2023-03-25 05:20:26 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-25 05:20:26 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2023-03-25 05:20:31 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.5.0/24\"":{}},"f:taints":{}}} } {kubelet Update v1 2023-03-25 05:47:11 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.5.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-1b/i-0168f82458996cc53,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/control-plane,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.5.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3892310016 0} {<nil>} 3801084Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3787452416 0} {<nil>} 3698684Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-03-25 05:20:22 +0000 UTC,LastTransitionTime:2023-03-25 05:20:22 +0000 UTC,Reason:CalicoIsUp,Message:Calico is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-25 05:47:11 +0000 UTC,LastTransitionTime:2023-03-25 05:18:50 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-25 05:47:11 +0000 UTC,LastTransitionTime:2023-03-25 05:18:50 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-25 05:47:11 +0000 UTC,LastTransitionTime:2023-03-25 05:18:50 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-25 05:47:11 +0000 UTC,LastTransitionTime:2023-03-25 05:20:29 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.48.81,},NodeAddress{Type:ExternalIP,Address:3.101.135.77,},NodeAddress{Type:InternalDNS,Address:i-0168f82458996cc53.us-west-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-0168f82458996cc53.us-west-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-3-101-135-77.us-west-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2d1ccc7ebf313420bd1f530a94fbfd,SystemUUID:ec2d1ccc-7ebf-3134-20bd-1f530a94fbfd,BootID:15cee321-56e7-4290-ab48-47b60b0bc657,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.28.0-alpha.0,KubeProxyVersion:v1.28.0-alpha.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcd@sha256:dd75ec974b0a2a6f6bb47001ba09207976e625db898d1b16735528c009cb171c registry.k8s.io/etcd:3.5.6-0],SizeBytes:102542580,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:6f72b851544986cb0921b53ea655ec04c36131248f16d4ad110cb3ca0c369dc1 registry.k8s.io/etcd:3.5.4-0],SizeBytes:102157811,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:13f53ed1d91e2e11aac476ee9a0269fdda6cc4874eba903efd40daf50c55eee5 registry.k8s.io/etcd:3.5.3-0],SizeBytes:102143581,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:51eae8381dcb1078289fa7b4f3df2630cdc18d09fb56f8e56b41c40e191d6c83 registry.k8s.io/etcd:3.5.7-0],SizeBytes:101639218,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:4afb99b4690b418ffc2ceb67e1a17376457e441c1f09ab55447f0aaf992fa646 registry.k8s.io/etcd:3.4.3-0],SizeBytes:100947667,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:9ce33ba33d8e738a5b85ed50b5080ac746deceed4a7496c550927a7a19ca3b6d registry.k8s.io/etcd:3.5.0-0],SizeBytes:99868722,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:64b9ea357325d5db9f8a723dcf503b5a449177b17ac87d69481e126bb724c263 registry.k8s.io/etcd:3.5.1-0],SizeBytes:98888614,},ContainerImage{Names:[docker.io/calico/cni@sha256:e282ea2914c806b5de2976330a17cfb5e6dcef47147bceb1432ca5c75fd46f50 docker.io/calico/cni:v3.24.5],SizeBytes:87458443,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:4ad90a11b55313b182afc186b9876c8e891531b8db4c9bf1541953021618d0e2 registry.k8s.io/etcd:3.4.13-0],SizeBytes:86742272,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:0d2ab6b15b4fe0b903b9e81192d942fb39c29be61fe341cd16db5d0165b0435c registry.k8s.io/etcd:3.3.17-0],SizeBytes:85524408,},ContainerImage{Names:[docker.io/calico/node@sha256:5972ad2bcbdc668564d3e26960c9c513b2d7b05581c704747cf7c62ef3a405a6 docker.io/calico/node:v3.24.5],SizeBytes:81584866,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:17da501f5d2a675be46040422a27b7cc21b8a43895ac998b171db1c346f361f7 registry.k8s.io/etcd:3.3.10-0],SizeBytes:76162358,},ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager-slim@sha256:8727adc059334e0e3329dbccce116a734da4a090da5e86c8cddccaf333caa666 registry.k8s.io/etcdadm/etcd-manager-slim:v3.0.20230201],SizeBytes:63926165,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:905d7ca17fd02bc24c0eba9a062753aba15db3e31422390bc3238eb762339b20 registry.k8s.io/etcd:3.2.24-1],SizeBytes:63423287,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.27.0-alpha.1],SizeBytes:45219545,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.27.0-alpha.1],SizeBytes:43628585,},ContainerImage{Names:[registry.k8s.io/kube-apiserver@sha256:49934a29c559e79ed19fc38b4697b18d8fe174b17bf56192588575f980c93956 registry.k8s.io/kube-apiserver:v1.28.0-alpha.0],SizeBytes:33212506,},ContainerImage{Names:[docker.io/calico/kube-controllers@sha256:2b6acd7f677f76ffe12ecf3ea7df92eb9b1bdb07336d1ac2a54c7631fb753f7e docker.io/calico/kube-controllers:v3.24.5],SizeBytes:31137169,},ContainerImage{Names:[registry.k8s.io/kube-controller-manager@sha256:974748384634982c73527fdb3c9ec0ec476bc0d52b3dd1d611a380b7f99af1f3 registry.k8s.io/kube-controller-manager:v1.28.0-alpha.0],SizeBytes:30820280,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/kube-proxy@sha256:8a03b3147cd7be8e8235ae1cd70cf69aca9ce03ba1cfaffea320540bb5afa496 registry.k8s.io/kube-proxy:v1.28.0-alpha.0],SizeBytes:23896850,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:23345856,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:22381475,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:22085298,},ContainerImage{Names:[registry.k8s.io/provider-aws/cloud-controller-manager@sha256:fdeb61e3e42ecd9cca868d550ebdb88dd6341d9e91fcfa9a37e227dab2ad22cb registry.k8s.io/provider-aws/cloud-controller-manager:v1.26.0],SizeBytes:20154862,},ContainerImage{Names:[registry.k8s.io/kube-scheduler@sha256:cbf4007abb076188f16b975bc2b74452603cc7b34c60a29d92f14c38ad02dc11 registry.k8s.io/kube-scheduler:v1.28.0-alpha.0],SizeBytes:18078674,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.27.0-alpha.1],SizeBytes:5074298,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Mar 25 05:47:50.455: INFO: Logging kubelet events for node i-0168f82458996cc53 Mar 25 05:47:50.523: INFO: Logging pods the kubelet thinks is on node i-0168f82458996cc53 Mar 25 05:47:50.584: INFO: kube-apiserver-i-0168f82458996cc53 started at 2023-03-25 05:17:36 +0000 UTC (0+2 container statuses recorded) Mar 25 05:47:50.584: INFO: Container healthcheck ready: true, restart count 0 Mar 25 05:47:50.584: INFO: Container kube-apiserver ready: true, restart count 2 Mar 25 05:47:50.584: INFO: calico-node-rdjq2 started at 2023-03-25 05:20:06 +0000 UTC (3+1 container statuses recorded) Mar 25 05:47:50.584: INFO: Init container upgrade-ipam ready: true, restart count 0 Mar 25 05:47:50.584: INFO: Init container install-cni ready: true, restart count 0 Mar 25 05:47:50.584: INFO: Init container mount-bpffs ready: true, restart count 0 Mar 25 05:47:50.584: INFO: Container calico-node ready: true, restart count 0 Mar 25 05:47:50.584: INFO: kube-controller-manager-i-0168f82458996cc53 started at 2023-03-25 05:17:36 +0000 UTC (0+1 container statuses recorded) Mar 25 05:47:50.584: INFO: Container kube-controller-manager ready: true, restart count 4 Mar 25 05:47:50.584: INFO: etcd-manager-events-i-0168f82458996cc53 started at 2023-03-25 05:17:36 +0000 UTC (11+1 container statuses recorded) Mar 25 05:47:50.584: INFO: Init container init-etcd-3-2-24 ready: true, restart count 0 Mar 25 05:47:50.584: INFO: Init container init-etcd-3-3-10 ready: true, restart count 0 Mar 25 05:47:50.584: INFO: Init container init-etcd-3-3-17 ready: true, restart count 0 Mar 25 05:47:50.584: INFO: Init container init-etcd-3-4-13 ready: true, restart count 0 Mar 25 05:47:50.584: INFO: Init container init-etcd-3-4-3 ready: true, restart count 0 Mar 25 05:47:50.584: INFO: Init container init-etcd-3-5-0 ready: true, restart count 0 Mar 25 05:47:50.584: INFO: Init container init-etcd-3-5-1 ready: true, restart count 0 Mar 25 05:47:50.584: INFO: Init container init-etcd-3-5-3 ready: true, restart count 0 Mar 25 05:47:50.584: INFO: Init container init-etcd-3-5-4 ready: true, restart count 0 Mar 25 05:47:50.584: INFO: Init container init-etcd-3-5-6 ready: true, restart count 0 Mar 25 05:47:50.584: INFO: Init container init-etcd-3-5-7 ready: true, restart count 0 Mar 25 05:47:50.584: INFO: Container etcd-manager ready: true, restart count 0 Mar 25 05:47:50.584: INFO: calico-kube-controllers-55f488656-rr8rz started at 2023-03-25 05:20:29 +0000 UTC (0+1 container statuses recorded) Mar 25 05:47:50.584: INFO: Container calico-kube-controllers ready: true, restart count 0 Mar 25 05:47:50.584: INFO: kube-proxy-i-0168f82458996cc53 started at 2023-03-25 05:17:36 +0000 UTC (0+1 container statuses recorded) Mar 25 05:47:50.584: INFO: Container kube-proxy ready: true, restart count 0 Mar 25 05:47:50.584: INFO: kube-scheduler-i-0168f82458996cc53 started at 2023-03-25 05:17:36 +0000 UTC (0+1 container statuses recorded) Mar 25 05:47:50.584: INFO: Container kube-scheduler ready: true, restart count 0 Mar 25 05:47:50.584: INFO: etcd-manager-main-i-0168f82458996cc53 started at 2023-03-25 05:17:36 +0000 UTC (11+1 container statuses recorded) Mar 25 05:47:50.584: INFO: Init container init-etcd-3-2-24 ready: true, restart count 0 Mar 25 05:47:50.584: INFO: Init container init-etcd-3-3-10 ready: true, restart count 0 Mar 25 05:47:50.584: INFO: Init container init-etcd-3-3-17 ready: true, restart count 0 Mar 25 05:47:50.584: INFO: Init container init-etcd-3-4-13 ready: true, restart count 0 Mar 25 05:47:50.584: INFO: Init container init-etcd-3-4-3 ready: true, restart count 0 Mar 25 05:47:50.584: INFO: Init container init-etcd-3-5-0 ready: true, restart count 0 Mar 25 05:47:50.584: INFO: Init container init-etcd-3-5-1 ready: true, restart count 0 Mar 25 05:47:50.584: INFO: Init container init-etcd-3-5-3 ready: true, restart count 0 Mar 25 05:47:50.584: INFO: Init container init-etcd-3-5-4 ready: true, restart count 0 Mar 25 05:47:50.584: INFO: Init container init-etcd-3-5-6 ready: true, restart count 0 Mar 25 05:47:50.584: INFO: Init container init-etcd-3-5-7 ready: true, restart count 0 Mar 25 05:47:50.584: INFO: Container etcd-manager ready: true, restart count 0 Mar 25 05:47:50.584: INFO: dns-controller-69987775c6-hbn69 started at 2023-03-25 05:19:47 +0000 UTC (0+1 container statuses recorded) Mar 25 05:47:50.584: INFO: Container dns-controller ready: true, restart count 0 Mar 25 05:47:50.584: INFO: aws-cloud-controller-manager-26vbv started at 2023-03-25 05:20:06 +0000 UTC (0+1 container statuses recorded) Mar 25 05:47:50.584: INFO: Container aws-cloud-controller-manager ready: true, restart count 0 Mar 25 05:47:50.584: INFO: ebs-csi-node-zjtj6 started at 2023-03-25 05:20:06 +0000 UTC (0+3 container statuses recorded) Mar 25 05:47:50.584: INFO: Container ebs-plugin ready: true, restart count 0 Mar 25 05:47:50.584: INFO: Container liveness-probe ready: true, restart count 0 Mar 25 05:47:50.584: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 25 05:47:50.584: INFO: kops-controller-cgw95 started at 2023-03-25 05:20:06 +0000 UTC (0+1 container statuses recorded) Mar 25 05:47:50.584: INFO: Container kops-controller ready: true, restart count 0 Mar 25 05:47:50.584: INFO: ebs-csi-controller-79f984545f-9q5ks started at 2023-03-25 05:21:26 +0000 UTC (0+5 container statuses recorded) Mar 25 05:47:50.584: INFO: Container csi-attacher ready: true, restart count 0 Mar 25 05:47:50.584: INFO: Container csi-provisioner ready: true, restart count 0 Mar 25 05:47:50.584: INFO: Container csi-resizer ready: true, restart count 0 Mar 25 05:47:50.584: INFO: Container ebs-plugin ready: true, restart count 0 Mar 25 05:47:50.584: INFO: Container liveness-probe ready: true, restart count 0 Mar 25 05:47:50.914: INFO: Latency metrics for node i-0168f82458996cc53 Mar 25 05:47:50.914: INFO: Logging node info for node i-03d301e6f030748f1 Mar 25 05:47:50.967: INFO: Node Info: &Node{ObjectMeta:{i-03d301e6f030748f1 897f52bd-65e2-4107-843a-37e36aa74032 40326 0 2023-03-25 05:31:35 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-1 failure-domain.beta.kubernetes.io/zone:us-west-1b kubernetes.io/arch:amd64 kubernetes.io/hostname:i-03d301e6f030748f1 kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:us-west-1b topology.hostpath.csi/node:i-03d301e6f030748f1 topology.kubernetes.io/region:us-west-1 topology.kubernetes.io/zone:us-west-1b] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-03d301e6f030748f1"} node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:172.20.50.199/19 projectcalico.org/IPv4IPIPTunnelAddr:100.110.167.64 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-03-25 05:31:35 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-25 05:31:35 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-03-25 05:31:35 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2023-03-25 05:31:35 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2023-03-25 05:31:48 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.8.0/24\"":{}}}} } {calico-node Update v1 2023-03-25 05:31:56 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4IPIPTunnelAddr":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2023-03-25 05:47:16 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-03-25 05:47:38 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.8.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-1b/i-03d301e6f030748f1,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.8.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-03-25 05:31:56 +0000 UTC,LastTransitionTime:2023-03-25 05:31:56 +0000 UTC,Reason:CalicoIsUp,Message:Calico is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-25 05:47:16 +0000 UTC,LastTransitionTime:2023-03-25 05:31:35 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-25 05:47:16 +0000 UTC,LastTransitionTime:2023-03-25 05:31:35 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-25 05:47:16 +0000 UTC,LastTransitionTime:2023-03-25 05:31:35 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-25 05:47:16 +0000 UTC,LastTransitionTime:2023-03-25 05:31:48 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.50.199,},NodeAddress{Type:ExternalIP,Address:52.53.245.63,},NodeAddress{Type:InternalDNS,Address:i-03d301e6f030748f1.us-west-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-03d301e6f030748f1.us-west-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-52-53-245-63.us-west-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec27152f7af9b9bfd1355ee19c256086,SystemUUID:ec27152f-7af9-b9bf-d135-5ee19c256086,BootID:1c303852-f2ce-45f9-ab14-6e8de97a0a3e,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.28.0-alpha.0,KubeProxyVersion:v1.28.0-alpha.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[docker.io/calico/cni@sha256:e282ea2914c806b5de2976330a17cfb5e6dcef47147bceb1432ca5c75fd46f50 docker.io/calico/cni:v3.24.5],SizeBytes:87458443,},ContainerImage{Names:[docker.io/calico/node@sha256:5972ad2bcbdc668564d3e26960c9c513b2d7b05581c704747cf7c62ef3a405a6 docker.io/calico/node:v3.24.5],SizeBytes:81584866,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nautilus@sha256:80ba6c8c44f9623f06e868a1aa66026c8ec438ad814f9ec95e9333b415fe3550 registry.k8s.io/e2e-test-images/nautilus:1.7],SizeBytes:49641698,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41901587,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:e468dddcd275163a042ab297b2d8c2aca50d5e148d2d22f3b6ba119e2f31fa79 registry.k8s.io/sig-storage/csi-provisioner:v3.4.0],SizeBytes:27427836,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/kube-proxy@sha256:8a03b3147cd7be8e8235ae1cd70cf69aca9ce03ba1cfaffea320540bb5afa496 registry.k8s.io/kube-proxy:v1.28.0-alpha.0],SizeBytes:23896850,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:23881995,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18758628,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18233005,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonroot@sha256:ee9f50b3c64b174d296d91ca9f69a914ac30e59095dfb462b2b518ad28a63655 registry.k8s.io/e2e-test-images/nonroot:1.4],SizeBytes:17747885,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8688564,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:7000082,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6978614,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac registry.k8s.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[kubernetes.io/csi/ebs.csi.aws.com^vol-091e496fb964dde60],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-091e496fb964dde60,DevicePath:,},},Config:nil,},} Mar 25 05:47:50.967: INFO: Logging kubelet events for node i-03d301e6f030748f1 Mar 25 05:47:51.045: INFO: Logging pods the kubelet thinks is on node i-03d301e6f030748f1 Mar 25 05:47:51.101: INFO: pod-failure-ignore-r9rt5 started at 2023-03-25 05:47:46 +0000 UTC (0+1 container statuses recorded) Mar 25 05:47:51.101: INFO: Container c ready: false, restart count 0 Mar 25 05:47:51.101: INFO: foo-jhtvd started at 2023-03-25 05:47:46 +0000 UTC (0+1 container statuses recorded) Mar 25 05:47:51.101: INFO: Container c ready: true, restart count 0 Mar 25 05:47:51.101: INFO: calico-node-h96vq started at 2023-03-25 05:31:35 +0000 UTC (3+1 container statuses recorded) Mar 25 05:47:51.101: INFO: Init container upgrade-ipam ready: true, restart count 0 Mar 25 05:47:51.101: INFO: Init container install-cni ready: true, restart count 0 Mar 25 05:47:51.101: INFO: Init container mount-bpffs ready: true, restart count 0 Mar 25 05:47:51.101: INFO: Container calico-node ready: true, restart count 0 Mar 25 05:47:51.101: INFO: ebs-csi-node-lljwl started at 2023-03-25 05:31:35 +0000 UTC (0+3 container statuses recorded) Mar 25 05:47:51.101: INFO: Container ebs-plugin ready: true, restart count 0 Mar 25 05:47:51.101: INFO: Container liveness-probe ready: true, restart count 0 Mar 25 05:47:51.101: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 25 05:47:51.101: INFO: agnhost-primary-76wps started at 2023-03-25 05:47:45 +0000 UTC (0+1 container statuses recorded) Mar 25 05:47:51.101: INFO: Container agnhost-primary ready: false, restart count 0 Mar 25 05:47:51.101: INFO: test-rs-pdlfp started at 2023-03-25 05:47:47 +0000 UTC (0+1 container statuses recorded) Mar 25 05:47:51.101: INFO: Container httpd ready: true, restart count 0 Mar 25 05:47:51.101: INFO: kube-proxy-i-03d301e6f030748f1 started at 2023-03-25 05:31:35 +0000 UTC (0+1 container statuses recorded) Mar 25 05:47:51.101: INFO: Container kube-proxy ready: true, restart count 0 Mar 25 05:47:51.101: INFO: pod-failure-ignore-h88dm started at 2023-03-25 05:47:38 +0000 UTC (0+1 container statuses recorded) Mar 25 05:47:51.101: INFO: Container c ready: false, restart count 0 Mar 25 05:47:51.101: INFO: agnhost-primary-g5rzg started at 2023-03-25 05:47:43 +0000 UTC (0+1 container statuses recorded) Mar 25 05:47:51.101: INFO: Container agnhost-primary ready: false, restart count 0 Mar 25 05:47:51.101: INFO: pod-ready started at 2023-03-25 05:47:48 +0000 UTC (0+1 container statuses recorded) Mar 25 05:47:51.101: INFO: Container pod-readiness-gate ready: true, restart count 0 Mar 25 05:47:51.101: INFO: webserver-pod started at 2023-03-25 05:47:27 +0000 UTC (0+1 container statuses recorded) Mar 25 05:47:51.101: INFO: Container agnhost ready: true, restart count 0 Mar 25 05:47:51.101: INFO: pod-failure-ignore-tnxsp started at 2023-03-25 05:47:38 +0000 UTC (0+1 container statuses recorded) Mar 25 05:47:51.101: INFO: Container c ready: false, restart count 0 Mar 25 05:47:51.101: INFO: pod-failure-ignore-x5pb6 started at 2023-03-25 05:47:43 +0000 UTC (0+1 container statuses recorded) Mar 25 05:47:51.101: INFO: Container c ready: false, restart count 0 Mar 25 05:47:51.101: INFO: pod-configmaps-a93186b9-eae5-49fb-9839-a908e7cb30ca started at 2023-03-25 05:44:09 +0000 UTC (0+2 container statuses recorded) Mar 25 05:47:51.101: INFO: Container agnhost-container ready: true, restart count 0 Mar 25 05:47:51.101: INFO: Container configmap-volume-binary-test ready: false, restart count 0 Mar 25 05:47:51.101: INFO: test-webserver-710e0aeb-9cd0-4068-966f-436eebd19d01 started at 2023-03-25 05:47:46 +0000 UTC (0+1 container statuses recorded) Mar 25 05:47:51.101: INFO: Container test-webserver ready: false, restart count 0 Mar 25 05:47:51.101: INFO: affinity-clusterip-r6l9z started at 2023-03-25 05:47:49 +0000 UTC (0+1 container statuses recorded) Mar 25 05:47:51.101: INFO: Container affinity-clusterip ready: false, restart count 0 Mar 25 05:47:51.101: INFO: pod-failure-ignore-4jnxc started at 2023-03-25 05:47:42 +0000 UTC (0+1 container statuses recorded) Mar 25 05:47:51.101: INFO: Container c ready: false, restart count 0 Mar 25 05:47:51.101: INFO: inline-volume-tester-fplkr started at 2023-03-25 05:46:13 +0000 UTC (0+1 container statuses recorded) Mar 25 05:47:51.101: INFO: Container csi-volume-tester ready: true, restart count 0 Mar 25 05:47:51.101: INFO: webserver-pod started at 2023-03-25 05:46:54 +0000 UTC (0+1 container statuses recorded) Mar 25 05:47:51.101: INFO: Container agnhost-container ready: true, restart count 0 Mar 25 05:47:51.348: INFO: Latency metrics for node i-03d301e6f030748f1 Mar 25 05:47:51.348: INFO: Logging node info for node i-049fdb2c1cde894df Mar 25 05:47:51.405: INFO: Node Info: &Node{ObjectMeta:{i-049fdb2c1cde894df b43a51e6-dda7-4047-b368-72d515695b4e 40640 0 2023-03-25 05:23:34 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-1 failure-domain.beta.kubernetes.io/zone:us-west-1b io.kubernetes.storage.mock/node:some-mock-node kubernetes.io/arch:amd64 kubernetes.io/hostname:i-049fdb2c1cde894df kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:us-west-1b topology.hostpath.csi/node:i-049fdb2c1cde894df topology.kubernetes.io/region:us-west-1 topology.kubernetes.io/zone:us-west-1b] map[csi.volume.kubernetes.io/nodeid:{"csi-mock-csi-mock-volumes-expansion-3185":"i-049fdb2c1cde894df","csi-mock-csi-mock-volumes-expansion-8433":"i-049fdb2c1cde894df","ebs.csi.aws.com":"i-049fdb2c1cde894df"} node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:172.20.43.61/19 projectcalico.org/IPv4IPIPTunnelAddr:100.97.18.64 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-03-25 05:23:34 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-25 05:23:34 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-03-25 05:23:34 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2023-03-25 05:23:34 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2023-03-25 05:23:45 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.6.0/24\"":{}}}} } {calico-node Update v1 2023-03-25 05:23:52 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4IPIPTunnelAddr":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2023-03-25 05:47:44 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-03-25 05:47:44 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:io.kubernetes.storage.mock/node":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.6.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-1b/i-049fdb2c1cde894df,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.6.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-03-25 05:23:52 +0000 UTC,LastTransitionTime:2023-03-25 05:23:52 +0000 UTC,Reason:CalicoIsUp,Message:Calico is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-25 05:47:44 +0000 UTC,LastTransitionTime:2023-03-25 05:23:34 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-25 05:47:44 +0000 UTC,LastTransitionTime:2023-03-25 05:23:34 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-25 05:47:44 +0000 UTC,LastTransitionTime:2023-03-25 05:23:34 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-25 05:47:44 +0000 UTC,LastTransitionTime:2023-03-25 05:23:45 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.43.61,},NodeAddress{Type:ExternalIP,Address:18.144.75.67,},NodeAddress{Type:InternalDNS,Address:i-049fdb2c1cde894df.us-west-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-049fdb2c1cde894df.us-west-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-18-144-75-67.us-west-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec252e592d19e1e6c583e87505431016,SystemUUID:ec252e59-2d19-e1e6-c583-e87505431016,BootID:45436b5d-1eb0-4b6f-a1d1-a1492b31b5d7,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.28.0-alpha.0,KubeProxyVersion:v1.28.0-alpha.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:112030336,},ContainerImage{Names:[docker.io/calico/cni@sha256:e282ea2914c806b5de2976330a17cfb5e6dcef47147bceb1432ca5c75fd46f50 docker.io/calico/cni:v3.24.5],SizeBytes:87458443,},ContainerImage{Names:[docker.io/calico/node@sha256:5972ad2bcbdc668564d3e26960c9c513b2d7b05581c704747cf7c62ef3a405a6 docker.io/calico/node:v3.24.5],SizeBytes:81584866,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:e468dddcd275163a042ab297b2d8c2aca50d5e148d2d22f3b6ba119e2f31fa79 registry.k8s.io/sig-storage/csi-provisioner:v3.4.0],SizeBytes:27427836,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/kube-proxy@sha256:8a03b3147cd7be8e8235ae1cd70cf69aca9ce03ba1cfaffea320540bb5afa496 registry.k8s.io/kube-proxy:v1.28.0-alpha.0],SizeBytes:23896850,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:23881995,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:68d396900aeaa072c1f27289485fdac29834045a6f3ffe369bf389d830ef572d registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.6],SizeBytes:20293261,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18758628,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18233005,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:15273057,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8688564,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:7000082,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6978614,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[kubernetes.io/csi/csi-mock-csi-mock-volumes-expansion-8433^6f2d5757-cad0-11ed-b4c8-3aceddafb7a9],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-mock-csi-mock-volumes-expansion-8433^6f2d5757-cad0-11ed-b4c8-3aceddafb7a9,DevicePath:,},},Config:nil,},} Mar 25 05:47:51.406: INFO: Logging kubelet events for node i-049fdb2c1cde894df Mar 25 05:47:51.487: INFO: Logging pods the kubelet thinks is on node i-049fdb2c1cde894df Mar 25 05:47:51.546: INFO: csi-mockplugin-attacher-0 started at 2023-03-25 05:46:43 +0000 UTC (0+1 container statuses recorded) Mar 25 05:47:51.546: INFO: Container csi-attacher ready: true, restart count 0 Mar 25 05:47:51.546: INFO: calico-node-wt5p7 started at 2023-03-25 05:23:35 +0000 UTC (3+1 container statuses recorded) Mar 25 05:47:51.546: INFO: Init container upgrade-ipam ready: true, restart count 0 Mar 25 05:47:51.546: INFO: Init container install-cni ready: true, restart count 0 Mar 25 05:47:51.546: INFO: Init container mount-bpffs ready: true, restart count 0 Mar 25 05:47:51.546: INFO: Container calico-node ready: true, restart count 0 Mar 25 05:47:51.546: INFO: coredns-559769c974-pgqhl started at 2023-03-25 05:24:36 +0000 UTC (0+1 container statuses recorded) Mar 25 05:47:51.546: INFO: Container coredns ready: true, restart count 0 Mar 25 05:47:51.546: INFO: pvc-volume-tester-zz4pt started at 2023-03-25 05:46:48 +0000 UTC (0+1 container statuses recorded) Mar 25 05:47:51.546: INFO: Container volume-tester ready: true, restart count 0 Mar 25 05:47:51.546: INFO: ebs-csi-node-m9zxz started at 2023-03-25 05:23:35 +0000 UTC (0+3 container statuses recorded) Mar 25 05:47:51.546: INFO: Container ebs-plugin ready: true, restart count 0 Mar 25 05:47:51.546: INFO: Container liveness-probe ready: true, restart count 0 Mar 25 05:47:51.546: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 25 05:47:51.546: INFO: csi-mockplugin-0 started at 2023-03-25 05:47:19 +0000 UTC (0+3 container statuses recorded) Mar 25 05:47:51.546: INFO: Container csi-provisioner ready: true, restart count 0 Mar 25 05:47:51.546: INFO: Container driver-registrar ready: true, restart count 0 Mar 25 05:47:51.546: INFO: Container mock ready: true, restart count 0 Mar 25 05:47:51.546: INFO: csi-mockplugin-resizer-0 started at 2023-03-25 05:47:19 +0000 UTC (0+1 container statuses recorded) Mar 25 05:47:51.546: INFO: Container csi-resizer ready: true, restart count 0 Mar 25 05:47:51.546: INFO: csi-mockplugin-attacher-0 started at 2023-03-25 05:47:50 +0000 UTC (0+1 container statuses recorded) Mar 25 05:47:51.546: INFO: Container csi-attacher ready: false, restart count 0 Mar 25 05:47:51.546: INFO: csi-mockplugin-attacher-0 started at 2023-03-25 05:47:19 +0000 UTC (0+1 container statuses recorded) Mar 25 05:47:51.546: INFO: Container csi-attacher ready: true, restart count 0 Mar 25 05:47:51.546: INFO: pause-pod-1 started at 2023-03-25 05:47:30 +0000 UTC (0+1 container statuses recorded) Mar 25 05:47:51.546: INFO: Container agnhost-container ready: true, restart count 0 Mar 25 05:47:51.546: INFO: kube-proxy-i-049fdb2c1cde894df started at 2023-03-25 05:23:35 +0000 UTC (0+1 container statuses recorded) Mar 25 05:47:51.546: INFO: Container kube-proxy ready: true, restart count 0 Mar 25 05:47:51.546: INFO: csi-mockplugin-0 started at 2023-03-25 05:47:50 +0000 UTC (0+3 container statuses recorded) Mar 25 05:47:51.546: INFO: Container csi-provisioner ready: false, restart count 0 Mar 25 05:47:51.546: INFO: Container driver-registrar ready: false, restart count 0 Mar 25 05:47:51.546: INFO: Container mock ready: false, restart count 0 Mar 25 05:47:51.546: INFO: test-rs-sj8h7 started at 2023-03-25 05:47:49 +0000 UTC (0+2 container statuses recorded) Mar 25 05:47:51.546: INFO: Container httpd ready: true, restart count 0 Mar 25 05:47:51.546: INFO: Container test-rs ready: true, restart count 0 Mar 25 05:47:51.546: INFO: csi-mockplugin-0 started at 2023-03-25 05:46:43 +0000 UTC (0+3 container statuses recorded) Mar 25 05:47:51.546: INFO: Container csi-provisioner ready: true, restart count 0 Mar 25 05:47:51.546: INFO: Container driver-registrar ready: true, restart count 0 Mar 25 05:47:51.546: INFO: Container mock ready: true, restart count 0 Mar 25 05:47:51.546: INFO: csi-mockplugin-resizer-0 started at 2023-03-25 05:46:43 +0000 UTC (0+1 container statuses recorded) Mar 25 05:47:51.546: INFO: Container csi-resizer ready: true, restart count 0 Mar 25 05:47:51.546: INFO: coredns-autoscaler-7cb5c5b969-h485x started at 2023-03-25 05:24:36 +0000 UTC (0+1 container statuses recorded) Mar 25 05:47:51.546: INFO: Container autoscaler ready: true, restart count 0 Mar 25 05:47:51.906: INFO: Latency metrics for node i-049fdb2c1cde894df Mar 25 05:47:51.906: INFO: Logging node info for node i-062679bb8c0ba4bce Mar 25 05:47:51.955: INFO: Node Info: &Node{ObjectMeta:{i-062679bb8c0ba4bce ee2da569-bb88-4297-9e6f-d2b15ba7ce86 39783 0 2023-03-25 05:27:31 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-1 failure-domain.beta.kubernetes.io/zone:us-west-1b kubernetes.io/arch:amd64 kubernetes.io/hostname:i-062679bb8c0ba4bce kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:us-west-1b topology.hostpath.csi/node:i-062679bb8c0ba4bce topology.kubernetes.io/region:us-west-1 topology.kubernetes.io/zone:us-west-1b] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-2327":"i-062679bb8c0ba4bce","ebs.csi.aws.com":"i-062679bb8c0ba4bce"} node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:172.20.45.43/19 projectcalico.org/IPv4IPIPTunnelAddr:100.106.79.0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-03-25 05:27:31 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-25 05:27:31 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-03-25 05:27:31 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2023-03-25 05:27:31 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2023-03-25 05:27:44 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.7.0/24\"":{}}}} } {calico-node Update v1 2023-03-25 05:27:50 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4IPIPTunnelAddr":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2023-03-25 05:47:25 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-03-25 05:47:25 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.7.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-1b/i-062679bb8c0ba4bce,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.7.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051689472 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946831872 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-03-25 05:27:50 +0000 UTC,LastTransitionTime:2023-03-25 05:27:50 +0000 UTC,Reason:CalicoIsUp,Message:Calico is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-25 05:47:25 +0000 UTC,LastTransitionTime:2023-03-25 05:27:31 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-25 05:47:25 +0000 UTC,LastTransitionTime:2023-03-25 05:27:31 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-25 05:47:25 +0000 UTC,LastTransitionTime:2023-03-25 05:27:31 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-25 05:47:25 +0000 UTC,LastTransitionTime:2023-03-25 05:27:44 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.45.43,},NodeAddress{Type:ExternalIP,Address:3.101.133.108,},NodeAddress{Type:InternalDNS,Address:i-062679bb8c0ba4bce.us-west-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-062679bb8c0ba4bce.us-west-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-3-101-133-108.us-west-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2528f599e9b1e199793fb5cd06d205,SystemUUID:ec2528f5-99e9-b1e1-9979-3fb5cd06d205,BootID:ff56b70b-c8c5-46db-8cc5-4353c8f535c6,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.28.0-alpha.0,KubeProxyVersion:v1.28.0-alpha.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:112030336,},ContainerImage{Names:[docker.io/calico/cni@sha256:e282ea2914c806b5de2976330a17cfb5e6dcef47147bceb1432ca5c75fd46f50 docker.io/calico/cni:v3.24.5],SizeBytes:87458443,},ContainerImage{Names:[docker.io/calico/node@sha256:5972ad2bcbdc668564d3e26960c9c513b2d7b05581c704747cf7c62ef3a405a6 docker.io/calico/node:v3.24.5],SizeBytes:81584866,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nautilus@sha256:80ba6c8c44f9623f06e868a1aa66026c8ec438ad814f9ec95e9333b415fe3550 registry.k8s.io/e2e-test-images/nautilus:1.7],SizeBytes:49641698,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41901587,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:e468dddcd275163a042ab297b2d8c2aca50d5e148d2d22f3b6ba119e2f31fa79 registry.k8s.io/sig-storage/csi-provisioner:v3.4.0],SizeBytes:27427836,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/kube-proxy@sha256:8a03b3147cd7be8e8235ae1cd70cf69aca9ce03ba1cfaffea320540bb5afa496 registry.k8s.io/kube-proxy:v1.28.0-alpha.0],SizeBytes:23896850,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:23881995,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.i