periodic-cluster-api-provider-azure-capi-e2e-v1beta1-minus-1 #1613218635327737856
Recent runs || View in Spyglass
Recent runs || View in Spyglass
| Result | FAILURE |
| Tests | 0 failed / 8 succeeded |
| Started | |
| Elapsed | 4h15m |
| Revision | release-1.5 |
capz-e2e Running the Cluster API E2E tests Running the quick-start spec Should create a workload cluster
capz-e2e Running the Cluster API E2E tests Running the self-hosted spec Should pivot the bootstrap cluster to a self-hosted cluster
capz-e2e Running the Cluster API E2E tests Should adopt up-to-date control plane Machines without modification Should adopt up-to-date control plane Machines without modification
capz-e2e Running the Cluster API E2E tests Should successfully exercise machine pools Should successfully create a cluster with machine pool machines
capz-e2e Running the Cluster API E2E tests Should successfully remediate unhealthy machines with MachineHealthCheck Should successfully trigger KCP remediation
capz-e2e Running the Cluster API E2E tests Should successfully remediate unhealthy machines with MachineHealthCheck Should successfully trigger machine deployment remediation
capz-e2e Running the Cluster API E2E tests Should successfully scale out and scale in a MachineDeployment Should successfully scale a MachineDeployment up and down upon changes to the MachineDeployment replica count
capz-e2e Running the Cluster API E2E tests Should successfully set and use node drain timeout A node should be forcefully removed if it cannot be drained in time
capz-e2e Conformance Tests conformance-tests
capz-e2e Running the Cluster API E2E tests API Version Upgrade upgrade from v1alpha4 to v1beta1, and scale workload clusters created in v1alpha4 Should create a management cluster and then upgrade all the providers
capz-e2e Running the Cluster API E2E tests Running KCP upgrade in a HA cluster [K8s-Upgrade] Should create and upgrade a workload cluster and eventually run kubetest
capz-e2e Running the Cluster API E2E tests Running KCP upgrade in a HA cluster using scale in rollout [K8s-Upgrade] Should create and upgrade a workload cluster and eventually run kubetest
capz-e2e Running the Cluster API E2E tests Running the workload cluster upgrade spec [K8s-Upgrade] Should create and upgrade a workload cluster and eventually run kubetest
capz-e2e Workload cluster creation Creating a GPU-enabled cluster [OPTIONAL] with a single control plane node and 1 node
capz-e2e Workload cluster creation Creating a VMSS cluster [REQUIRED] with a single control plane node and an AzureMachinePool with 2 Linux and 2 Windows worker nodes
capz-e2e Workload cluster creation Creating a cluster that uses the external cloud provider and external azurediskcsi driver [OPTIONAL] with a 1 control plane nodes and 2 worker nodes
capz-e2e Workload cluster creation Creating a dual-stack cluster [OPTIONAL] With dual-stack worker node
capz-e2e Workload cluster creation Creating a highly available cluster [REQUIRED] With 3 control-plane nodes and 2 Linux and 2 Windows worker nodes
capz-e2e Workload cluster creation Creating a ipv6 control-plane cluster [REQUIRED] With ipv6 worker node
capz-e2e Workload cluster creation Creating a private cluster [REQUIRED] Creates a public management cluster in a custom vnet
capz-e2e Workload cluster creation Creating an AKS cluster [EXPERIMENTAL][Managed Kubernetes] with a single control plane node and 1 node
capz-e2e Workload cluster creation Creating clusters using clusterclass [OPTIONAL] with a single control plane node, one linux worker node, and one windows worker node
capz-e2e [K8s-Upgrade] Running the workload cluster upgrade tests [CSI Migration] Running CSI migration test CSI=external CCM=external AzureDiskCSIMigration=true: upgrade to v1.23 should create volumes dynamically with intree cloud provider
capz-e2e [K8s-Upgrade] Running the workload cluster upgrade tests [CSI Migration] Running CSI migration test CSI=external CCM=internal AzureDiskCSIMigration=true: upgrade to v1.23 should create volumes dynamically with intree cloud provider
capz-e2e [K8s-Upgrade] Running the workload cluster upgrade tests [CSI Migration] Running CSI migration test CSI=internal CCM=internal AzureDiskCSIMigration=false: upgrade to v1.23 should create volumes dynamically with intree cloud provider
... skipping 573 lines ... [1mSTEP[0m: Dumping workload cluster kcp-adoption-qj97z5/kcp-adoption-2nuus8 kube-system pod logs [1mSTEP[0m: Fetching kube-system pod logs took 421.619043ms [1mSTEP[0m: Dumping workload cluster kcp-adoption-qj97z5/kcp-adoption-2nuus8 Azure activity log [1mSTEP[0m: Creating log watcher for controller kube-system/kube-apiserver-kcp-adoption-2nuus8-control-plane-0, container kube-apiserver [1mSTEP[0m: Collecting events for Pod kube-system/etcd-kcp-adoption-2nuus8-control-plane-0 [1mSTEP[0m: Collecting events for Pod kube-system/kube-apiserver-kcp-adoption-2nuus8-control-plane-0 [1mSTEP[0m: failed to find events of Pod "kube-apiserver-kcp-adoption-2nuus8-control-plane-0" [1mSTEP[0m: Creating log watcher for controller kube-system/kube-controller-manager-kcp-adoption-2nuus8-control-plane-0, container kube-controller-manager [1mSTEP[0m: Collecting events for Pod kube-system/coredns-bd6b6df9f-s77f4 [1mSTEP[0m: Creating log watcher for controller kube-system/coredns-bd6b6df9f-x9zz2, container coredns [1mSTEP[0m: Collecting events for Pod kube-system/coredns-bd6b6df9f-x9zz2 [1mSTEP[0m: failed to find events of Pod "etcd-kcp-adoption-2nuus8-control-plane-0" [1mSTEP[0m: Creating log watcher for controller kube-system/calico-node-4q5bd, container calico-node [1mSTEP[0m: Collecting events for Pod kube-system/kube-scheduler-kcp-adoption-2nuus8-control-plane-0 [1mSTEP[0m: Collecting events for Pod kube-system/calico-kube-controllers-85f479877b-4rztb [1mSTEP[0m: Creating log watcher for controller kube-system/kube-proxy-82tcj, container kube-proxy [1mSTEP[0m: failed to find events of Pod "kube-scheduler-kcp-adoption-2nuus8-control-plane-0" [1mSTEP[0m: Creating log watcher for controller kube-system/calico-kube-controllers-85f479877b-4rztb, container calico-kube-controllers [1mSTEP[0m: Collecting events for Pod kube-system/kube-proxy-82tcj [1mSTEP[0m: Creating log watcher for controller kube-system/kube-scheduler-kcp-adoption-2nuus8-control-plane-0, container kube-scheduler [1mSTEP[0m: Collecting events for Pod kube-system/kube-controller-manager-kcp-adoption-2nuus8-control-plane-0 [1mSTEP[0m: failed to find events of Pod "kube-controller-manager-kcp-adoption-2nuus8-control-plane-0" [1mSTEP[0m: Collecting events for Pod kube-system/calico-node-4q5bd [1mSTEP[0m: Creating log watcher for controller kube-system/etcd-kcp-adoption-2nuus8-control-plane-0, container etcd [1mSTEP[0m: Creating log watcher for controller kube-system/coredns-bd6b6df9f-s77f4, container coredns [1mSTEP[0m: Fetching activity logs took 1.785320781s [1mSTEP[0m: Dumping all the Cluster API resources in the "kcp-adoption-qj97z5" namespace [1mSTEP[0m: Deleting cluster kcp-adoption-qj97z5/kcp-adoption-2nuus8 ... skipping 73 lines ... [1mSTEP[0m: Dumping workload cluster mhc-remediation-k8ua3v/mhc-remediation-qauw00 kube-system pod logs [1mSTEP[0m: Fetching kube-system pod logs took 467.353484ms [1mSTEP[0m: Creating log watcher for controller kube-system/calico-node-mkdkt, container calico-node [1mSTEP[0m: Creating log watcher for controller kube-system/coredns-bd6b6df9f-fql8n, container coredns [1mSTEP[0m: Collecting events for Pod kube-system/etcd-mhc-remediation-qauw00-control-plane-dlk6g [1mSTEP[0m: failed to find events of Pod "etcd-mhc-remediation-qauw00-control-plane-dlk6g" [1mSTEP[0m: Dumping workload cluster mhc-remediation-k8ua3v/mhc-remediation-qauw00 Azure activity log [1mSTEP[0m: Creating log watcher for controller kube-system/calico-kube-controllers-85f479877b-vffxq, container calico-kube-controllers [1mSTEP[0m: Creating log watcher for controller kube-system/kube-apiserver-mhc-remediation-qauw00-control-plane-dlk6g, container kube-apiserver [1mSTEP[0m: Collecting events for Pod kube-system/coredns-bd6b6df9f-fql8n [1mSTEP[0m: Collecting events for Pod kube-system/calico-node-mkdkt [1mSTEP[0m: Collecting events for Pod kube-system/kube-apiserver-mhc-remediation-qauw00-control-plane-dlk6g ... skipping 6 lines ... [1mSTEP[0m: Collecting events for Pod kube-system/kube-proxy-fhmch [1mSTEP[0m: Collecting events for Pod kube-system/calico-node-w2nlh [1mSTEP[0m: Creating log watcher for controller kube-system/kube-scheduler-mhc-remediation-qauw00-control-plane-dlk6g, container kube-scheduler [1mSTEP[0m: Creating log watcher for controller kube-system/coredns-bd6b6df9f-66thz, container coredns [1mSTEP[0m: Collecting events for Pod kube-system/coredns-bd6b6df9f-66thz [1mSTEP[0m: Collecting events for Pod kube-system/kube-scheduler-mhc-remediation-qauw00-control-plane-dlk6g [1mSTEP[0m: failed to find events of Pod "kube-scheduler-mhc-remediation-qauw00-control-plane-dlk6g" [1mSTEP[0m: Creating log watcher for controller kube-system/etcd-mhc-remediation-qauw00-control-plane-dlk6g, container etcd [1mSTEP[0m: Collecting events for Pod kube-system/calico-kube-controllers-85f479877b-vffxq [1mSTEP[0m: failed to find events of Pod "kube-controller-manager-mhc-remediation-qauw00-control-plane-dlk6g" [1mSTEP[0m: failed to find events of Pod "kube-apiserver-mhc-remediation-qauw00-control-plane-dlk6g" [1mSTEP[0m: Fetching activity logs took 1.270989614s [1mSTEP[0m: Dumping all the Cluster API resources in the "mhc-remediation-k8ua3v" namespace [1mSTEP[0m: Deleting cluster mhc-remediation-k8ua3v/mhc-remediation-qauw00 [1mSTEP[0m: Deleting cluster mhc-remediation-qauw00 INFO: Waiting for the Cluster mhc-remediation-k8ua3v/mhc-remediation-qauw00 to be deleted [1mSTEP[0m: Waiting for cluster mhc-remediation-qauw00 to be deleted ... skipping 16 lines ... [1mShould pivot the bootstrap cluster to a self-hosted cluster[0m [37m/home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure/test/e2e/azure_selfhosted.go:107[0m INFO: "Should pivot the bootstrap cluster to a self-hosted cluster" started at Wed, 11 Jan 2023 17:07:35 UTC on Ginkgo node 5 of 10 [1mSTEP[0m: Creating namespace "self-hosted" for hosting the cluster Jan 11 17:07:35.046: INFO: starting to create namespace for hosting the "self-hosted" test spec 2023/01/11 17:07:35 failed trying to get namespace (self-hosted):namespaces "self-hosted" not found INFO: Creating namespace self-hosted INFO: Creating event watcher for namespace "self-hosted" [1mSTEP[0m: Creating a workload cluster INFO: Creating the workload cluster with name "self-hosted-ejjg61" using the "management" template (Kubernetes v1.23.15, 1 control-plane machines, 1 worker machines) INFO: Getting the cluster template yaml INFO: clusterctl config cluster self-hosted-ejjg61 --infrastructure (default) --kubernetes-version v1.23.15 --control-plane-machine-count 1 --worker-machine-count 1 --flavor management ... skipping 60 lines ... [1mSTEP[0m: Dumping workload cluster self-hosted/self-hosted-ejjg61 kube-system pod logs [1mSTEP[0m: Fetching kube-system pod logs took 391.705661ms [1mSTEP[0m: Dumping workload cluster self-hosted/self-hosted-ejjg61 Azure activity log [1mSTEP[0m: Creating log watcher for controller kube-system/etcd-self-hosted-ejjg61-control-plane-4ggqn, container etcd [1mSTEP[0m: Collecting events for Pod kube-system/etcd-self-hosted-ejjg61-control-plane-4ggqn [1mSTEP[0m: failed to find events of Pod "etcd-self-hosted-ejjg61-control-plane-4ggqn" [1mSTEP[0m: Creating log watcher for controller kube-system/kube-apiserver-self-hosted-ejjg61-control-plane-4ggqn, container kube-apiserver [1mSTEP[0m: Creating log watcher for controller kube-system/calico-node-gkdkj, container calico-node [1mSTEP[0m: Creating log watcher for controller kube-system/kube-controller-manager-self-hosted-ejjg61-control-plane-4ggqn, container kube-controller-manager [1mSTEP[0m: Collecting events for Pod kube-system/kube-proxy-rc7kg [1mSTEP[0m: Creating log watcher for controller kube-system/kube-proxy-4gwhw, container kube-proxy [1mSTEP[0m: Creating log watcher for controller kube-system/calico-kube-controllers-85f479877b-wcxjf, container calico-kube-controllers [1mSTEP[0m: Creating log watcher for controller kube-system/kube-scheduler-self-hosted-ejjg61-control-plane-4ggqn, container kube-scheduler [1mSTEP[0m: Collecting events for Pod kube-system/kube-controller-manager-self-hosted-ejjg61-control-plane-4ggqn [1mSTEP[0m: failed to find events of Pod "kube-controller-manager-self-hosted-ejjg61-control-plane-4ggqn" [1mSTEP[0m: Collecting events for Pod kube-system/kube-proxy-4gwhw [1mSTEP[0m: Collecting events for Pod kube-system/calico-kube-controllers-85f479877b-wcxjf [1mSTEP[0m: Creating log watcher for controller kube-system/kube-proxy-rc7kg, container kube-proxy [1mSTEP[0m: Collecting events for Pod kube-system/kube-apiserver-self-hosted-ejjg61-control-plane-4ggqn [1mSTEP[0m: failed to find events of Pod "kube-apiserver-self-hosted-ejjg61-control-plane-4ggqn" [1mSTEP[0m: Collecting events for Pod kube-system/coredns-bd6b6df9f-4tm5m [1mSTEP[0m: Creating log watcher for controller kube-system/coredns-bd6b6df9f-7mw2g, container coredns [1mSTEP[0m: Creating log watcher for controller kube-system/coredns-bd6b6df9f-4tm5m, container coredns [1mSTEP[0m: Collecting events for Pod kube-system/coredns-bd6b6df9f-7mw2g [1mSTEP[0m: Collecting events for Pod kube-system/calico-node-h6vfb [1mSTEP[0m: Collecting events for Pod kube-system/calico-node-gkdkj [1mSTEP[0m: Collecting events for Pod kube-system/kube-scheduler-self-hosted-ejjg61-control-plane-4ggqn [1mSTEP[0m: failed to find events of Pod "kube-scheduler-self-hosted-ejjg61-control-plane-4ggqn" [1mSTEP[0m: Creating log watcher for controller kube-system/calico-node-h6vfb, container calico-node [1mSTEP[0m: Fetching activity logs took 1.423660936s Jan 11 17:17:42.974: INFO: Dumping all the Cluster API resources in the "self-hosted" namespace Jan 11 17:17:43.370: INFO: Deleting all clusters in the self-hosted namespace [1mSTEP[0m: Deleting cluster self-hosted-ejjg61 INFO: Waiting for the Cluster self-hosted/self-hosted-ejjg61 to be deleted ... skipping 211 lines ... Jan 11 17:16:23.014: INFO: Collecting logs for Windows node quick-sta-vl572 in cluster quick-start-7rz2i5 in namespace quick-start-atjlip Jan 11 17:19:07.284: INFO: Attempting to copy file /c:/crashdumps.tar on node quick-sta-vl572 to /logs/artifacts/clusters/quick-start-7rz2i5/machines/quick-start-7rz2i5-md-win-5d8d647fdd-kvv45/crashdumps.tar Jan 11 17:19:09.632: INFO: Collecting boot logs for AzureMachine quick-start-7rz2i5-md-win-vl572 Failed to get logs for machine quick-start-7rz2i5-md-win-5d8d647fdd-kvv45, cluster quick-start-atjlip/quick-start-7rz2i5: [running command "Get-Content "C:\\cni.log"": Process exited with status 1, running command "$p = 'c:\localdumps' ; if (Test-Path $p) { tar.exe -cvzf c:\crashdumps.tar $p *>&1 | %{ Write-Output "$_"} } else { Write-Host "No crash dumps found at $p" }": Process exited with status 1] Jan 11 17:19:10.476: INFO: Collecting logs for Windows node quick-sta-hqjrw in cluster quick-start-7rz2i5 in namespace quick-start-atjlip Jan 11 17:21:53.073: INFO: Attempting to copy file /c:/crashdumps.tar on node quick-sta-hqjrw to /logs/artifacts/clusters/quick-start-7rz2i5/machines/quick-start-7rz2i5-md-win-5d8d647fdd-r6xhd/crashdumps.tar Jan 11 17:21:55.442: INFO: Collecting boot logs for AzureMachine quick-start-7rz2i5-md-win-hqjrw Failed to get logs for machine quick-start-7rz2i5-md-win-5d8d647fdd-r6xhd, cluster quick-start-atjlip/quick-start-7rz2i5: [running command "Get-Content "C:\\cni.log"": Process exited with status 1, running command "$p = 'c:\localdumps' ; if (Test-Path $p) { tar.exe -cvzf c:\crashdumps.tar $p *>&1 | %{ Write-Output "$_"} } else { Write-Host "No crash dumps found at $p" }": Process exited with status 1] [1mSTEP[0m: Dumping workload cluster quick-start-atjlip/quick-start-7rz2i5 kube-system pod logs [1mSTEP[0m: Creating log watcher for controller kube-system/calico-node-windows-hkst8, container calico-node-felix [1mSTEP[0m: Collecting events for Pod kube-system/kube-proxy-windows-w48sr [1mSTEP[0m: Collecting events for Pod kube-system/csi-proxy-nvjwp [1mSTEP[0m: Creating log watcher for controller kube-system/kube-proxy-zvs7k, container kube-proxy [1mSTEP[0m: Creating log watcher for controller kube-system/etcd-quick-start-7rz2i5-control-plane-mb7jj, container etcd [1mSTEP[0m: Creating log watcher for controller kube-system/calico-node-8pkvf, container calico-node [1mSTEP[0m: Collecting events for Pod kube-system/kube-proxy-zvs7k [1mSTEP[0m: Creating log watcher for controller kube-system/kube-scheduler-quick-start-7rz2i5-control-plane-mb7jj, container kube-scheduler [1mSTEP[0m: Collecting events for Pod kube-system/calico-node-8pkvf [1mSTEP[0m: Creating log watcher for controller kube-system/calico-node-windows-hkst8, container calico-node-startup [1mSTEP[0m: Collecting events for Pod kube-system/etcd-quick-start-7rz2i5-control-plane-mb7jj [1mSTEP[0m: failed to find events of Pod "etcd-quick-start-7rz2i5-control-plane-mb7jj" [1mSTEP[0m: Creating log watcher for controller kube-system/kube-apiserver-quick-start-7rz2i5-control-plane-mb7jj, container kube-apiserver [1mSTEP[0m: Fetching kube-system pod logs took 666.615889ms [1mSTEP[0m: Dumping workload cluster quick-start-atjlip/quick-start-7rz2i5 Azure activity log [1mSTEP[0m: Collecting events for Pod kube-system/kube-apiserver-quick-start-7rz2i5-control-plane-mb7jj [1mSTEP[0m: Creating log watcher for controller kube-system/kube-controller-manager-quick-start-7rz2i5-control-plane-mb7jj, container kube-controller-manager [1mSTEP[0m: failed to find events of Pod "kube-apiserver-quick-start-7rz2i5-control-plane-mb7jj" [1mSTEP[0m: Collecting events for Pod kube-system/kube-scheduler-quick-start-7rz2i5-control-plane-mb7jj [1mSTEP[0m: failed to find events of Pod "kube-scheduler-quick-start-7rz2i5-control-plane-mb7jj" [1mSTEP[0m: Collecting events for Pod kube-system/calico-node-windows-hkst8 [1mSTEP[0m: Creating log watcher for controller kube-system/calico-node-windows-mmdcq, container calico-node-startup [1mSTEP[0m: Collecting events for Pod kube-system/calico-node-54mj8 [1mSTEP[0m: Creating log watcher for controller kube-system/calico-node-windows-mmdcq, container calico-node-felix [1mSTEP[0m: Collecting events for Pod kube-system/calico-node-windows-mmdcq [1mSTEP[0m: Creating log watcher for controller kube-system/containerd-logger-bcqst, container containerd-logger ... skipping 7 lines ... [1mSTEP[0m: Creating log watcher for controller kube-system/coredns-bd6b6df9f-6g8gm, container coredns [1mSTEP[0m: Creating log watcher for controller kube-system/calico-node-54mj8, container calico-node [1mSTEP[0m: Collecting events for Pod kube-system/coredns-bd6b6df9f-6g8gm [1mSTEP[0m: Creating log watcher for controller kube-system/kube-proxy-d9dtw, container kube-proxy [1mSTEP[0m: Collecting events for Pod kube-system/kube-controller-manager-quick-start-7rz2i5-control-plane-mb7jj [1mSTEP[0m: Collecting events for Pod kube-system/kube-proxy-d9dtw [1mSTEP[0m: failed to find events of Pod "kube-controller-manager-quick-start-7rz2i5-control-plane-mb7jj" [1mSTEP[0m: Creating log watcher for controller kube-system/csi-proxy-8qzqw, container csi-proxy [1mSTEP[0m: Creating log watcher for controller kube-system/kube-proxy-windows-9w5gs, container kube-proxy [1mSTEP[0m: Collecting events for Pod kube-system/kube-proxy-windows-9w5gs [1mSTEP[0m: Collecting events for Pod kube-system/csi-proxy-8qzqw [1mSTEP[0m: Creating log watcher for controller kube-system/kube-proxy-windows-w48sr, container kube-proxy [1mSTEP[0m: Creating log watcher for controller kube-system/csi-proxy-nvjwp, container csi-proxy ... skipping 106 lines ... [1mSTEP[0m: Collecting events for Pod kube-system/calico-node-9q2js [1mSTEP[0m: Collecting events for Pod kube-system/calico-kube-controllers-85f479877b-ghch8 [1mSTEP[0m: Creating log watcher for controller kube-system/calico-node-qzxzq, container calico-node [1mSTEP[0m: Collecting events for Pod kube-system/coredns-bd6b6df9f-ch5tm [1mSTEP[0m: Creating log watcher for controller kube-system/etcd-node-drain-z93dd4-control-plane-sjfqb, container etcd [1mSTEP[0m: Creating log watcher for controller kube-system/coredns-bd6b6df9f-tqwqq, container coredns [1mSTEP[0m: Error starting logs stream for pod kube-system/kube-scheduler-node-drain-z93dd4-control-plane-sjfqb, container kube-scheduler: pods "node-drain-z93dd4-control-plane-sjfqb" not found [1mSTEP[0m: Error starting logs stream for pod kube-system/kube-apiserver-node-drain-z93dd4-control-plane-sjfqb, container kube-apiserver: pods "node-drain-z93dd4-control-plane-sjfqb" not found [1mSTEP[0m: Error starting logs stream for pod kube-system/etcd-node-drain-z93dd4-control-plane-sjfqb, container etcd: pods "node-drain-z93dd4-control-plane-sjfqb" not found [1mSTEP[0m: Error starting logs stream for pod kube-system/calico-node-qzxzq, container calico-node: pods "node-drain-z93dd4-control-plane-sjfqb" not found [1mSTEP[0m: Error starting logs stream for pod kube-system/kube-controller-manager-node-drain-z93dd4-control-plane-sjfqb, container kube-controller-manager: pods "node-drain-z93dd4-control-plane-sjfqb" not found [1mSTEP[0m: Error starting logs stream for pod kube-system/kube-proxy-svk7r, container kube-proxy: pods "node-drain-z93dd4-control-plane-sjfqb" not found [1mSTEP[0m: Fetching activity logs took 7.555918523s [1mSTEP[0m: Dumping all the Cluster API resources in the "node-drain-7epbvb" namespace [1mSTEP[0m: Deleting cluster node-drain-7epbvb/node-drain-z93dd4 [1mSTEP[0m: Deleting cluster node-drain-z93dd4 INFO: Waiting for the Cluster node-drain-7epbvb/node-drain-z93dd4 to be deleted [1mSTEP[0m: Waiting for cluster node-drain-z93dd4 to be deleted ... skipping 79 lines ... [1mSTEP[0m: Dumping workload cluster machine-pool-5e09q8/machine-pool-zvnmbq Azure activity log [1mSTEP[0m: Collecting events for Pod kube-system/calico-node-windows-2fgrv [1mSTEP[0m: Collecting events for Pod kube-system/calico-node-7t2g4 [1mSTEP[0m: Creating log watcher for controller kube-system/calico-node-windows-9g7g6, container calico-node-startup [1mSTEP[0m: Creating log watcher for controller kube-system/kube-apiserver-machine-pool-zvnmbq-control-plane-5bpbl, container kube-apiserver [1mSTEP[0m: Collecting events for Pod kube-system/kube-apiserver-machine-pool-zvnmbq-control-plane-5bpbl [1mSTEP[0m: failed to find events of Pod "kube-apiserver-machine-pool-zvnmbq-control-plane-5bpbl" [1mSTEP[0m: Collecting events for Pod kube-system/calico-node-r229v [1mSTEP[0m: Creating log watcher for controller kube-system/kube-proxy-prjkk, container kube-proxy [1mSTEP[0m: Collecting events for Pod kube-system/kube-proxy-windows-wk2c6 [1mSTEP[0m: Creating log watcher for controller kube-system/kube-controller-manager-machine-pool-zvnmbq-control-plane-5bpbl, container kube-controller-manager [1mSTEP[0m: Collecting events for Pod kube-system/kube-scheduler-machine-pool-zvnmbq-control-plane-5bpbl [1mSTEP[0m: Collecting events for Pod kube-system/kube-proxy-prjkk [1mSTEP[0m: Creating log watcher for controller kube-system/calico-node-windows-9g7g6, container calico-node-felix [1mSTEP[0m: Collecting events for Pod kube-system/kube-proxy-vrpsc [1mSTEP[0m: failed to find events of Pod "kube-scheduler-machine-pool-zvnmbq-control-plane-5bpbl" [1mSTEP[0m: Creating log watcher for controller kube-system/kube-proxy-windows-zctsm, container kube-proxy [1mSTEP[0m: Creating log watcher for controller kube-system/kube-proxy-windows-tfqwq, container kube-proxy [1mSTEP[0m: Creating log watcher for controller kube-system/calico-node-windows-2fgrv, container calico-node-felix [1mSTEP[0m: Creating log watcher for controller kube-system/calico-node-7t2g4, container calico-node [1mSTEP[0m: Collecting events for Pod kube-system/calico-node-windows-9g7g6 [1mSTEP[0m: Creating log watcher for controller kube-system/calico-node-windows-fcc66, container calico-node-startup ... skipping 2 lines ... [1mSTEP[0m: Creating log watcher for controller kube-system/kube-scheduler-machine-pool-zvnmbq-control-plane-5bpbl, container kube-scheduler [1mSTEP[0m: Collecting events for Pod kube-system/calico-kube-controllers-85f479877b-t9lqs [1mSTEP[0m: Collecting events for Pod kube-system/kube-proxy-windows-tfqwq [1mSTEP[0m: Creating log watcher for controller kube-system/calico-kube-controllers-85f479877b-t9lqs, container calico-kube-controllers [1mSTEP[0m: Creating log watcher for controller kube-system/coredns-bd6b6df9f-8wwjk, container coredns [1mSTEP[0m: Collecting events for Pod kube-system/kube-controller-manager-machine-pool-zvnmbq-control-plane-5bpbl [1mSTEP[0m: failed to find events of Pod "kube-controller-manager-machine-pool-zvnmbq-control-plane-5bpbl" [1mSTEP[0m: Collecting events for Pod kube-system/coredns-bd6b6df9f-8wwjk [1mSTEP[0m: Creating log watcher for controller kube-system/etcd-machine-pool-zvnmbq-control-plane-5bpbl, container etcd [1mSTEP[0m: Creating log watcher for controller kube-system/calico-node-r229v, container calico-node [1mSTEP[0m: Creating log watcher for controller kube-system/kube-proxy-vrpsc, container kube-proxy [1mSTEP[0m: Collecting events for Pod kube-system/calico-node-windows-fcc66 [1mSTEP[0m: Creating log watcher for controller kube-system/coredns-bd6b6df9f-2bsrb, container coredns [1mSTEP[0m: Collecting events for Pod kube-system/etcd-machine-pool-zvnmbq-control-plane-5bpbl [1mSTEP[0m: failed to find events of Pod "etcd-machine-pool-zvnmbq-control-plane-5bpbl" [1mSTEP[0m: Creating log watcher for controller kube-system/kube-proxy-windows-wk2c6, container kube-proxy [1mSTEP[0m: Creating log watcher for controller kube-system/calico-node-windows-fcc66, container calico-node-felix [1mSTEP[0m: Collecting events for Pod kube-system/coredns-bd6b6df9f-2bsrb [1mSTEP[0m: Error starting logs stream for pod kube-system/calico-node-windows-9g7g6, container calico-node-startup: pods "win-p-win000002" not found [1mSTEP[0m: Error starting logs stream for pod kube-system/calico-node-windows-9g7g6, container calico-node-felix: pods "win-p-win000002" not found [1mSTEP[0m: Error starting logs stream for pod kube-system/kube-proxy-windows-tfqwq, container kube-proxy: pods "win-p-win000000" not found [1mSTEP[0m: Error starting logs stream for pod kube-system/kube-proxy-vrpsc, container kube-proxy: pods "machine-pool-zvnmbq-mp-0000002" not found [1mSTEP[0m: Error starting logs stream for pod kube-system/kube-proxy-windows-wk2c6, container kube-proxy: pods "win-p-win000001" not found [1mSTEP[0m: Error starting logs stream for pod kube-system/calico-node-windows-fcc66, container calico-node-startup: pods "win-p-win000001" not found [1mSTEP[0m: Error starting logs stream for pod kube-system/kube-proxy-windows-zctsm, container kube-proxy: pods "win-p-win000002" not found [1mSTEP[0m: Error starting logs stream for pod kube-system/calico-node-windows-2fgrv, container calico-node-felix: pods "win-p-win000000" not found [1mSTEP[0m: Error starting logs stream for pod kube-system/calico-node-windows-2fgrv, container calico-node-startup: pods "win-p-win000000" not found [1mSTEP[0m: Error starting logs stream for pod kube-system/calico-node-r229v, container calico-node: pods "machine-pool-zvnmbq-mp-0000002" not found [1mSTEP[0m: Error starting logs stream for pod kube-system/calico-node-windows-fcc66, container calico-node-felix: pods "win-p-win000001" not found [1mSTEP[0m: Fetching activity logs took 1.651086171s [1mSTEP[0m: Dumping all the Cluster API resources in the "machine-pool-5e09q8" namespace [1mSTEP[0m: Deleting cluster machine-pool-5e09q8/machine-pool-zvnmbq [1mSTEP[0m: Deleting cluster machine-pool-zvnmbq INFO: Waiting for the Cluster machine-pool-5e09q8/machine-pool-zvnmbq to be deleted [1mSTEP[0m: Waiting for cluster machine-pool-zvnmbq to be deleted ... skipping 78 lines ... Jan 11 17:23:41.901: INFO: Collecting logs for Windows node md-scale-ww584 in cluster md-scale-lw7vx5 in namespace md-scale-8bj2l2 Jan 11 17:26:24.516: INFO: Attempting to copy file /c:/crashdumps.tar on node md-scale-ww584 to /logs/artifacts/clusters/md-scale-lw7vx5/machines/md-scale-lw7vx5-md-win-744c867896-lvxgz/crashdumps.tar Jan 11 17:26:26.957: INFO: Collecting boot logs for AzureMachine md-scale-lw7vx5-md-win-ww584 Failed to get logs for machine md-scale-lw7vx5-md-win-744c867896-lvxgz, cluster md-scale-8bj2l2/md-scale-lw7vx5: [running command "Get-Content "C:\\cni.log"": Process exited with status 1, running command "$p = 'c:\localdumps' ; if (Test-Path $p) { tar.exe -cvzf c:\crashdumps.tar $p *>&1 | %{ Write-Output "$_"} } else { Write-Host "No crash dumps found at $p" }": Process exited with status 1] Jan 11 17:26:28.036: INFO: Collecting logs for Windows node md-scale-vpqg4 in cluster md-scale-lw7vx5 in namespace md-scale-8bj2l2 Jan 11 17:29:08.778: INFO: Attempting to copy file /c:/crashdumps.tar on node md-scale-vpqg4 to /logs/artifacts/clusters/md-scale-lw7vx5/machines/md-scale-lw7vx5-md-win-744c867896-vp64b/crashdumps.tar Jan 11 17:29:10.799: INFO: Collecting boot logs for AzureMachine md-scale-lw7vx5-md-win-vpqg4 Failed to get logs for machine md-scale-lw7vx5-md-win-744c867896-vp64b, cluster md-scale-8bj2l2/md-scale-lw7vx5: [running command "Get-Content "C:\\cni.log"": Process exited with status 1, running command "$p = 'c:\localdumps' ; if (Test-Path $p) { tar.exe -cvzf c:\crashdumps.tar $p *>&1 | %{ Write-Output "$_"} } else { Write-Host "No crash dumps found at $p" }": Process exited with status 1] [1mSTEP[0m: Dumping workload cluster md-scale-8bj2l2/md-scale-lw7vx5 kube-system pod logs [1mSTEP[0m: Creating log watcher for controller kube-system/containerd-logger-thm5l, container containerd-logger [1mSTEP[0m: Collecting events for Pod kube-system/csi-proxy-vl7lj [1mSTEP[0m: Collecting events for Pod kube-system/kube-proxy-gv4gz [1mSTEP[0m: Collecting events for Pod kube-system/kube-apiserver-md-scale-lw7vx5-control-plane-95wcl [1mSTEP[0m: Creating log watcher for controller kube-system/calico-node-windows-5dksv, container calico-node-startup [1mSTEP[0m: Creating log watcher for controller kube-system/calico-node-windows-scppz, container calico-node-felix [1mSTEP[0m: Collecting events for Pod kube-system/kube-proxy-windows-6lj8d [1mSTEP[0m: Collecting events for Pod kube-system/kube-proxy-tl5nv [1mSTEP[0m: Creating log watcher for controller kube-system/kube-proxy-windows-hcdnb, container kube-proxy [1mSTEP[0m: Creating log watcher for controller kube-system/kube-controller-manager-md-scale-lw7vx5-control-plane-95wcl, container kube-controller-manager [1mSTEP[0m: failed to find events of Pod "kube-apiserver-md-scale-lw7vx5-control-plane-95wcl" [1mSTEP[0m: Creating log watcher for controller kube-system/etcd-md-scale-lw7vx5-control-plane-95wcl, container etcd [1mSTEP[0m: Creating log watcher for controller kube-system/kube-proxy-windows-6lj8d, container kube-proxy [1mSTEP[0m: Collecting events for Pod kube-system/kube-proxy-windows-hcdnb [1mSTEP[0m: Creating log watcher for controller kube-system/calico-kube-controllers-85f479877b-q95td, container calico-kube-controllers [1mSTEP[0m: Creating log watcher for controller kube-system/kube-apiserver-md-scale-lw7vx5-control-plane-95wcl, container kube-apiserver [1mSTEP[0m: Creating log watcher for controller kube-system/kube-scheduler-md-scale-lw7vx5-control-plane-95wcl, container kube-scheduler ... skipping 2 lines ... [1mSTEP[0m: Collecting events for Pod kube-system/containerd-logger-dcrqn [1mSTEP[0m: Creating log watcher for controller kube-system/calico-node-windows-5dksv, container calico-node-felix [1mSTEP[0m: Collecting events for Pod kube-system/calico-kube-controllers-85f479877b-q95td [1mSTEP[0m: Creating log watcher for controller kube-system/calico-node-hbhjt, container calico-node [1mSTEP[0m: Creating log watcher for controller kube-system/kube-proxy-tl5nv, container kube-proxy [1mSTEP[0m: Collecting events for Pod kube-system/kube-controller-manager-md-scale-lw7vx5-control-plane-95wcl [1mSTEP[0m: failed to find events of Pod "kube-controller-manager-md-scale-lw7vx5-control-plane-95wcl" [1mSTEP[0m: Fetching kube-system pod logs took 673.425768ms [1mSTEP[0m: Collecting events for Pod kube-system/calico-node-windows-5dksv [1mSTEP[0m: Dumping workload cluster md-scale-8bj2l2/md-scale-lw7vx5 Azure activity log [1mSTEP[0m: Creating log watcher for controller kube-system/calico-node-windows-scppz, container calico-node-startup [1mSTEP[0m: Collecting events for Pod kube-system/calico-node-n4h5n [1mSTEP[0m: Creating log watcher for controller kube-system/calico-node-n4h5n, container calico-node [1mSTEP[0m: Collecting events for Pod kube-system/etcd-md-scale-lw7vx5-control-plane-95wcl [1mSTEP[0m: failed to find events of Pod "etcd-md-scale-lw7vx5-control-plane-95wcl" [1mSTEP[0m: Collecting events for Pod kube-system/coredns-bd6b6df9f-kbfj7 [1mSTEP[0m: Creating log watcher for controller kube-system/coredns-bd6b6df9f-qqrnv, container coredns [1mSTEP[0m: Collecting events for Pod kube-system/containerd-logger-thm5l [1mSTEP[0m: Creating log watcher for controller kube-system/csi-proxy-h9qmr, container csi-proxy [1mSTEP[0m: Creating log watcher for controller kube-system/coredns-bd6b6df9f-kbfj7, container coredns [1mSTEP[0m: Collecting events for Pod kube-system/coredns-bd6b6df9f-qqrnv [1mSTEP[0m: Collecting events for Pod kube-system/calico-node-windows-scppz [1mSTEP[0m: Creating log watcher for controller kube-system/containerd-logger-dcrqn, container containerd-logger [1mSTEP[0m: Collecting events for Pod kube-system/csi-proxy-h9qmr [1mSTEP[0m: Creating log watcher for controller kube-system/csi-proxy-vl7lj, container csi-proxy [1mSTEP[0m: Collecting events for Pod kube-system/kube-scheduler-md-scale-lw7vx5-control-plane-95wcl [1mSTEP[0m: failed to find events of Pod "kube-scheduler-md-scale-lw7vx5-control-plane-95wcl" [1mSTEP[0m: Fetching activity logs took 6.391835032s [1mSTEP[0m: Dumping all the Cluster API resources in the "md-scale-8bj2l2" namespace [1mSTEP[0m: Deleting cluster md-scale-8bj2l2/md-scale-lw7vx5 [1mSTEP[0m: Deleting cluster md-scale-lw7vx5 INFO: Waiting for the Cluster md-scale-8bj2l2/md-scale-lw7vx5 to be deleted [1mSTEP[0m: Waiting for cluster md-scale-lw7vx5 to be deleted ... skipping 9 lines ... [90m/home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure/test/e2e/capi_test.go:45[0m Should successfully scale out and scale in a MachineDeployment [90m/home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure/test/e2e/capi_test.go:183[0m Should successfully scale a MachineDeployment up and down upon changes to the MachineDeployment replica count [90m/home/prow/go/pkg/mod/sigs.k8s.io/cluster-api/test@v1.2.1/e2e/md_scale.go:71[0m [90m------------------------------[0m {"component":"entrypoint","file":"k8s.io/test-infra/prow/entrypoint/run.go:164","func":"k8s.io/test-infra/prow/entrypoint.Options.ExecuteProcess","level":"error","msg":"Process did not finish before 4h0m0s timeout","severity":"error","time":"2023-01-11T20:58:01Z"} ++ early_exit_handler ++ '[' -n 162 ']' ++ kill -TERM 162 ++ cleanup_dind ++ [[ true == \t\r\u\e ]] ++ echo 'Cleaning up after docker' ... skipping 12 lines ... Cleaning up after docker Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running? Stopping Docker: dockerProgram process in pidfile '/var/run/docker-ssd.pid', 1 process(es), refused to die. ================================================================================ Done cleaning up after docker in docker. All sensitive variables are redacted {"component":"entrypoint","file":"k8s.io/test-infra/prow/entrypoint/run.go:254","func":"k8s.io/test-infra/prow/entrypoint.gracefullyTerminate","level":"error","msg":"Process did not exit before 15m0s grace period","severity":"error","time":"2023-01-11T21:13:01Z"} {"component":"entrypoint","error":"os: process already finished","file":"k8s.io/test-infra/prow/entrypoint/run.go:256","func":"k8s.io/test-infra/prow/entrypoint.gracefullyTerminate","level":"error","msg":"Could not kill process after grace period","severity":"error","time":"2023-01-11T21:13:01Z"}