This job view page is being replaced by Spyglass soon. Check out the new job view.
ResultFAILURE
Tests 0 failed / 8 succeeded
Started2022-09-29 20:37
Elapsed4h15m
Revisionrelease-1.5

No Test Failures!


Show 8 Passed Tests

Show 11 Skipped Tests

Error lines from build-log.txt

... skipping 686 lines ...
Sep 29 20:55:31.865: INFO: Collecting boot logs for AzureMachine mhc-remediation-ba5sji-md-0-nx24g

STEP: Dumping workload cluster mhc-remediation-w8f9kt/mhc-remediation-ba5sji kube-system pod logs
STEP: Creating log watcher for controller kube-system/etcd-mhc-remediation-ba5sji-control-plane-jv5kq, container etcd
STEP: Collecting events for Pod kube-system/kube-controller-manager-mhc-remediation-ba5sji-control-plane-jv5kq
STEP: Collecting events for Pod kube-system/kube-proxy-pnvt9
STEP: failed to find events of Pod "kube-controller-manager-mhc-remediation-ba5sji-control-plane-jv5kq"
STEP: Creating log watcher for controller kube-system/calico-node-zkwmj, container calico-node
STEP: Collecting events for Pod kube-system/coredns-64897985d-ltq9w
STEP: Collecting events for Pod kube-system/calico-node-zkwmj
STEP: Creating log watcher for controller kube-system/coredns-64897985d-ltq9w, container coredns
STEP: Fetching kube-system pod logs took 455.211416ms
STEP: Dumping workload cluster mhc-remediation-w8f9kt/mhc-remediation-ba5sji Azure activity log
STEP: Creating log watcher for controller kube-system/coredns-64897985d-m7nvr, container coredns
STEP: Collecting events for Pod kube-system/kube-proxy-wnpfp
STEP: Collecting events for Pod kube-system/etcd-mhc-remediation-ba5sji-control-plane-jv5kq
STEP: Creating log watcher for controller kube-system/kube-scheduler-mhc-remediation-ba5sji-control-plane-jv5kq, container kube-scheduler
STEP: failed to find events of Pod "etcd-mhc-remediation-ba5sji-control-plane-jv5kq"
STEP: Creating log watcher for controller kube-system/kube-apiserver-mhc-remediation-ba5sji-control-plane-jv5kq, container kube-apiserver
STEP: Collecting events for Pod kube-system/kube-scheduler-mhc-remediation-ba5sji-control-plane-jv5kq
STEP: failed to find events of Pod "kube-scheduler-mhc-remediation-ba5sji-control-plane-jv5kq"
STEP: Creating log watcher for controller kube-system/kube-controller-manager-mhc-remediation-ba5sji-control-plane-jv5kq, container kube-controller-manager
STEP: Collecting events for Pod kube-system/calico-kube-controllers-85f479877b-lc5h8
STEP: Collecting events for Pod kube-system/coredns-64897985d-m7nvr
STEP: Collecting events for Pod kube-system/kube-apiserver-mhc-remediation-ba5sji-control-plane-jv5kq
STEP: Creating log watcher for controller kube-system/calico-node-jjsvz, container calico-node
STEP: failed to find events of Pod "kube-apiserver-mhc-remediation-ba5sji-control-plane-jv5kq"
STEP: Creating log watcher for controller kube-system/calico-kube-controllers-85f479877b-lc5h8, container calico-kube-controllers
STEP: Creating log watcher for controller kube-system/kube-proxy-wnpfp, container kube-proxy
STEP: Collecting events for Pod kube-system/calico-node-jjsvz
STEP: Creating log watcher for controller kube-system/kube-proxy-pnvt9, container kube-proxy
STEP: Fetching activity logs took 2.771338759s
STEP: Dumping all the Cluster API resources in the "mhc-remediation-w8f9kt" namespace
... skipping 20 lines ...
  Should pivot the bootstrap cluster to a self-hosted cluster
  /home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure/test/e2e/azure_selfhosted.go:107

INFO: "Should pivot the bootstrap cluster to a self-hosted cluster" started at Thu, 29 Sep 2022 20:46:23 UTC on Ginkgo node 8 of 10
STEP: Creating namespace "self-hosted" for hosting the cluster
Sep 29 20:46:23.807: INFO: starting to create namespace for hosting the "self-hosted" test spec
2022/09/29 20:46:23 failed trying to get namespace (self-hosted):namespaces "self-hosted" not found
INFO: Creating namespace self-hosted
INFO: Creating event watcher for namespace "self-hosted"
STEP: Creating a workload cluster
INFO: Creating the workload cluster with name "self-hosted-vvzexf" using the "management" template (Kubernetes v1.23.12, 1 control-plane machines, 1 worker machines)
INFO: Getting the cluster template yaml
INFO: clusterctl config cluster self-hosted-vvzexf --infrastructure (default) --kubernetes-version v1.23.12 --control-plane-machine-count 1 --worker-machine-count 1 --flavor management
... skipping 67 lines ...
STEP: Collecting events for Pod kube-system/calico-node-ptplf
STEP: Collecting events for Pod kube-system/kube-proxy-pwxqh
STEP: Collecting events for Pod kube-system/kube-scheduler-self-hosted-vvzexf-control-plane-pps58
STEP: Creating log watcher for controller kube-system/calico-kube-controllers-85f479877b-wqbtn, container calico-kube-controllers
STEP: Creating log watcher for controller kube-system/coredns-64897985d-gmpst, container coredns
STEP: Creating log watcher for controller kube-system/kube-proxy-lpznj, container kube-proxy
STEP: failed to find events of Pod "kube-scheduler-self-hosted-vvzexf-control-plane-pps58"
STEP: Creating log watcher for controller kube-system/kube-scheduler-self-hosted-vvzexf-control-plane-pps58, container kube-scheduler
STEP: Creating log watcher for controller kube-system/calico-node-ptplf, container calico-node
STEP: Collecting events for Pod kube-system/coredns-64897985d-wqssx
STEP: Collecting events for Pod kube-system/coredns-64897985d-gmpst
STEP: Creating log watcher for controller kube-system/coredns-64897985d-wqssx, container coredns
STEP: Collecting events for Pod kube-system/kube-proxy-lpznj
STEP: Creating log watcher for controller kube-system/kube-proxy-pwxqh, container kube-proxy
STEP: Creating log watcher for controller kube-system/etcd-self-hosted-vvzexf-control-plane-pps58, container etcd
STEP: Collecting events for Pod kube-system/etcd-self-hosted-vvzexf-control-plane-pps58
STEP: failed to find events of Pod "etcd-self-hosted-vvzexf-control-plane-pps58"
STEP: Creating log watcher for controller kube-system/calico-node-nmr6r, container calico-node
STEP: Collecting events for Pod kube-system/calico-node-nmr6r
STEP: Collecting events for Pod kube-system/kube-controller-manager-self-hosted-vvzexf-control-plane-pps58
STEP: Collecting events for Pod kube-system/calico-kube-controllers-85f479877b-wqbtn
STEP: failed to find events of Pod "kube-controller-manager-self-hosted-vvzexf-control-plane-pps58"
STEP: Fetching activity logs took 3.35447877s
Sep 29 20:56:01.039: INFO: Dumping all the Cluster API resources in the "self-hosted" namespace
Sep 29 20:56:01.367: INFO: Deleting all clusters in the self-hosted namespace
STEP: Deleting cluster self-hosted-vvzexf
INFO: Waiting for the Cluster self-hosted/self-hosted-vvzexf to be deleted
STEP: Waiting for cluster self-hosted-vvzexf to be deleted
INFO: Got error while streaming logs for pod capi-kubeadm-control-plane-system/capi-kubeadm-control-plane-controller-manager-74b6b6b77f-fw994, container manager: http2: client connection lost
INFO: Got error while streaming logs for pod capi-kubeadm-bootstrap-system/capi-kubeadm-bootstrap-controller-manager-6c76c59d6b-wh8rp, container manager: http2: client connection lost
INFO: Got error while streaming logs for pod capz-system/capz-controller-manager-78ddbfbdf9-qt4xn, container manager: http2: client connection lost
INFO: Got error while streaming logs for pod capi-system/capi-controller-manager-7df9bc44b4-xjd6z, container manager: http2: client connection lost
Sep 29 21:03:41.679: INFO: Deleting namespace used for hosting the "self-hosted" test spec
INFO: Deleting namespace self-hosted
Sep 29 21:03:41.698: INFO: Checking if any resources are left over in Azure for spec "self-hosted"
STEP: Redacting sensitive information from logs
Sep 29 21:04:26.013: INFO: Cleaning up after "Running the Cluster API E2E tests Running the self-hosted spec Should pivot the bootstrap cluster to a self-hosted cluster" spec
STEP: Redacting sensitive information from logs
... skipping 204 lines ...

Sep 29 20:55:16.456: INFO: Collecting logs for Windows node quick-sta-mrg7c in cluster quick-start-jcsm4k in namespace quick-start-geareb

Sep 29 20:58:00.893: INFO: Attempting to copy file /c:/crashdumps.tar on node quick-sta-mrg7c to /logs/artifacts/clusters/quick-start-jcsm4k/machines/quick-start-jcsm4k-md-win-c7c5dcf8b-hqk5g/crashdumps.tar
Sep 29 20:58:03.373: INFO: Collecting boot logs for AzureMachine quick-start-jcsm4k-md-win-mrg7c

Failed to get logs for machine quick-start-jcsm4k-md-win-c7c5dcf8b-hqk5g, cluster quick-start-geareb/quick-start-jcsm4k: [running command "Get-Content "C:\\cni.log"": Process exited with status 1, running command "$p = 'c:\localdumps' ; if (Test-Path $p) { tar.exe -cvzf c:\crashdumps.tar $p *>&1 | %{ Write-Output "$_"} } else { Write-Host "No crash dumps found at $p" }": Process exited with status 1]
Sep 29 20:58:04.252: INFO: Collecting logs for Windows node quick-sta-vnr6t in cluster quick-start-jcsm4k in namespace quick-start-geareb

Sep 29 21:00:49.393: INFO: Attempting to copy file /c:/crashdumps.tar on node quick-sta-vnr6t to /logs/artifacts/clusters/quick-start-jcsm4k/machines/quick-start-jcsm4k-md-win-c7c5dcf8b-sskvr/crashdumps.tar
Sep 29 21:00:51.364: INFO: Collecting boot logs for AzureMachine quick-start-jcsm4k-md-win-vnr6t

Failed to get logs for machine quick-start-jcsm4k-md-win-c7c5dcf8b-sskvr, cluster quick-start-geareb/quick-start-jcsm4k: [running command "Get-Content "C:\\cni.log"": Process exited with status 1, running command "$p = 'c:\localdumps' ; if (Test-Path $p) { tar.exe -cvzf c:\crashdumps.tar $p *>&1 | %{ Write-Output "$_"} } else { Write-Host "No crash dumps found at $p" }": Process exited with status 1]
STEP: Dumping workload cluster quick-start-geareb/quick-start-jcsm4k kube-system pod logs
STEP: Fetching kube-system pod logs took 618.719067ms
STEP: Collecting events for Pod kube-system/calico-node-wdf84
STEP: Creating log watcher for controller kube-system/kube-controller-manager-quick-start-jcsm4k-control-plane-j95p5, container kube-controller-manager
STEP: Creating log watcher for controller kube-system/containerd-logger-s94fw, container containerd-logger
STEP: Collecting events for Pod kube-system/coredns-64897985d-gcm7n
STEP: Creating log watcher for controller kube-system/kube-proxy-windows-zls94, container kube-proxy
STEP: Collecting events for Pod kube-system/kube-proxy-xj6mz
STEP: Creating log watcher for controller kube-system/kube-scheduler-quick-start-jcsm4k-control-plane-j95p5, container kube-scheduler
STEP: Collecting events for Pod kube-system/kube-proxy-windows-zls94
STEP: Dumping workload cluster quick-start-geareb/quick-start-jcsm4k Azure activity log
STEP: Creating log watcher for controller kube-system/kube-proxy-xj6mz, container kube-proxy
STEP: Collecting events for Pod kube-system/kube-controller-manager-quick-start-jcsm4k-control-plane-j95p5
STEP: failed to find events of Pod "kube-controller-manager-quick-start-jcsm4k-control-plane-j95p5"
STEP: Creating log watcher for controller kube-system/kube-proxy-jlmft, container kube-proxy
STEP: Collecting events for Pod kube-system/kube-scheduler-quick-start-jcsm4k-control-plane-j95p5
STEP: Creating log watcher for controller kube-system/calico-node-windows-58dhp, container calico-node-startup
STEP: failed to find events of Pod "kube-scheduler-quick-start-jcsm4k-control-plane-j95p5"
STEP: Creating log watcher for controller kube-system/csi-proxy-qxzks, container csi-proxy
STEP: Collecting events for Pod kube-system/csi-proxy-qxzks
STEP: Collecting events for Pod kube-system/containerd-logger-4wp4w
STEP: Creating log watcher for controller kube-system/etcd-quick-start-jcsm4k-control-plane-j95p5, container etcd
STEP: Collecting events for Pod kube-system/calico-kube-controllers-85f479877b-lrlfc
STEP: Creating log watcher for controller kube-system/calico-node-dj78z, container calico-node
... skipping 14 lines ...
STEP: Creating log watcher for controller kube-system/coredns-64897985d-gcm7n, container coredns
STEP: Creating log watcher for controller kube-system/calico-node-windows-k5kss, container calico-node-startup
STEP: Creating log watcher for controller kube-system/kube-apiserver-quick-start-jcsm4k-control-plane-j95p5, container kube-apiserver
STEP: Collecting events for Pod kube-system/etcd-quick-start-jcsm4k-control-plane-j95p5
STEP: Creating log watcher for controller kube-system/containerd-logger-4wp4w, container containerd-logger
STEP: Collecting events for Pod kube-system/kube-apiserver-quick-start-jcsm4k-control-plane-j95p5
STEP: failed to find events of Pod "kube-apiserver-quick-start-jcsm4k-control-plane-j95p5"
STEP: Collecting events for Pod kube-system/kube-proxy-windows-wm2kt
STEP: failed to find events of Pod "etcd-quick-start-jcsm4k-control-plane-j95p5"
STEP: Fetching activity logs took 3.759954081s
STEP: Dumping all the Cluster API resources in the "quick-start-geareb" namespace
STEP: Deleting cluster quick-start-geareb/quick-start-jcsm4k
STEP: Deleting cluster quick-start-jcsm4k
INFO: Waiting for the Cluster quick-start-geareb/quick-start-jcsm4k to be deleted
STEP: Waiting for cluster quick-start-jcsm4k to be deleted
... skipping 101 lines ...
STEP: Collecting events for Pod kube-system/kube-proxy-5prhq
STEP: Collecting events for Pod kube-system/kube-proxy-windows-wtjh2
STEP: Creating log watcher for controller kube-system/kube-controller-manager-machine-pool-cyih5y-control-plane-nbpwj, container kube-controller-manager
STEP: Creating log watcher for controller kube-system/kube-scheduler-machine-pool-cyih5y-control-plane-nbpwj, container kube-scheduler
STEP: Collecting events for Pod kube-system/etcd-machine-pool-cyih5y-control-plane-nbpwj
STEP: Creating log watcher for controller kube-system/kube-proxy-5prhq, container kube-proxy
STEP: Error starting logs stream for pod kube-system/kube-proxy-5prhq, container kube-proxy: pods "machine-pool-cyih5y-mp-0000002" not found
STEP: Error starting logs stream for pod kube-system/calico-node-windows-cw2md, container calico-node-startup: pods "win-p-win000002" not found
STEP: Error starting logs stream for pod kube-system/calico-node-56grx, container calico-node: pods "machine-pool-cyih5y-mp-0000002" not found
STEP: Error starting logs stream for pod kube-system/kube-proxy-windows-wtjh2, container kube-proxy: pods "win-p-win000002" not found
STEP: Error starting logs stream for pod kube-system/calico-node-windows-cw2md, container calico-node-felix: pods "win-p-win000002" not found
STEP: Fetching activity logs took 2.256652899s
STEP: Dumping all the Cluster API resources in the "machine-pool-k3uuez" namespace
STEP: Deleting cluster machine-pool-k3uuez/machine-pool-cyih5y
STEP: Deleting cluster machine-pool-cyih5y
INFO: Waiting for the Cluster machine-pool-k3uuez/machine-pool-cyih5y to be deleted
STEP: Waiting for cluster machine-pool-cyih5y to be deleted
... skipping 78 lines ...

Sep 29 20:57:33.839: INFO: Collecting logs for Windows node md-scale-rrjrb in cluster md-scale-ibj501 in namespace md-scale-kxlwtq

Sep 29 21:00:14.500: INFO: Attempting to copy file /c:/crashdumps.tar on node md-scale-rrjrb to /logs/artifacts/clusters/md-scale-ibj501/machines/md-scale-ibj501-md-win-69cd6845b7-6kgxq/crashdumps.tar
Sep 29 21:00:16.696: INFO: Collecting boot logs for AzureMachine md-scale-ibj501-md-win-rrjrb

Failed to get logs for machine md-scale-ibj501-md-win-69cd6845b7-6kgxq, cluster md-scale-kxlwtq/md-scale-ibj501: [running command "Get-Content "C:\\cni.log"": Process exited with status 1, running command "$p = 'c:\localdumps' ; if (Test-Path $p) { tar.exe -cvzf c:\crashdumps.tar $p *>&1 | %{ Write-Output "$_"} } else { Write-Host "No crash dumps found at $p" }": Process exited with status 1]
Sep 29 21:00:18.169: INFO: Collecting logs for Windows node md-scale-vjfs2 in cluster md-scale-ibj501 in namespace md-scale-kxlwtq

Sep 29 21:03:00.057: INFO: Attempting to copy file /c:/crashdumps.tar on node md-scale-vjfs2 to /logs/artifacts/clusters/md-scale-ibj501/machines/md-scale-ibj501-md-win-69cd6845b7-9cvgm/crashdumps.tar
Sep 29 21:03:02.296: INFO: Collecting boot logs for AzureMachine md-scale-ibj501-md-win-vjfs2

Failed to get logs for machine md-scale-ibj501-md-win-69cd6845b7-9cvgm, cluster md-scale-kxlwtq/md-scale-ibj501: [running command "Get-Content "C:\\cni.log"": Process exited with status 1, running command "$p = 'c:\localdumps' ; if (Test-Path $p) { tar.exe -cvzf c:\crashdumps.tar $p *>&1 | %{ Write-Output "$_"} } else { Write-Host "No crash dumps found at $p" }": Process exited with status 1]
STEP: Dumping workload cluster md-scale-kxlwtq/md-scale-ibj501 kube-system pod logs
STEP: Creating log watcher for controller kube-system/calico-node-7spbq, container calico-node
STEP: Creating log watcher for controller kube-system/calico-kube-controllers-85f479877b-wnxfh, container calico-kube-controllers
STEP: Creating log watcher for controller kube-system/calico-node-8l7hr, container calico-node
STEP: Creating log watcher for controller kube-system/kube-proxy-dzjnh, container kube-proxy
STEP: Collecting events for Pod kube-system/etcd-md-scale-ibj501-control-plane-74jcs
... skipping 11 lines ...
STEP: Collecting events for Pod kube-system/kube-proxy-windows-7ch55
STEP: Collecting events for Pod kube-system/kube-proxy-hncxh
STEP: Creating log watcher for controller kube-system/kube-proxy-windows-7ch55, container kube-proxy
STEP: Collecting events for Pod kube-system/kube-proxy-windows-lq2kl
STEP: Creating log watcher for controller kube-system/kube-proxy-windows-lq2kl, container kube-proxy
STEP: Collecting events for Pod kube-system/kube-controller-manager-md-scale-ibj501-control-plane-74jcs
STEP: failed to find events of Pod "kube-controller-manager-md-scale-ibj501-control-plane-74jcs"
STEP: Collecting events for Pod kube-system/kube-scheduler-md-scale-ibj501-control-plane-74jcs
STEP: Collecting events for Pod kube-system/calico-node-8l7hr
STEP: Creating log watcher for controller kube-system/calico-node-windows-pc695, container calico-node-startup
STEP: failed to find events of Pod "kube-scheduler-md-scale-ibj501-control-plane-74jcs"
STEP: Collecting events for Pod kube-system/calico-node-7spbq
STEP: Creating log watcher for controller kube-system/coredns-64897985d-h22nj, container coredns
STEP: Creating log watcher for controller kube-system/kube-scheduler-md-scale-ibj501-control-plane-74jcs, container kube-scheduler
STEP: Creating log watcher for controller kube-system/containerd-logger-7fwrq, container containerd-logger
STEP: Creating log watcher for controller kube-system/coredns-64897985d-s9h9c, container coredns
STEP: Collecting events for Pod kube-system/containerd-logger-7fwrq
... skipping 84 lines ...
STEP: Dumping logs from the "node-drain-408q19" workload cluster
STEP: Dumping workload cluster node-drain-0l8lzp/node-drain-408q19 logs
Sep 29 21:01:12.822: INFO: Collecting logs for Linux node node-drain-408q19-control-plane-wbjhz in cluster node-drain-408q19 in namespace node-drain-0l8lzp

Sep 29 21:07:46.743: INFO: Collecting boot logs for AzureMachine node-drain-408q19-control-plane-wbjhz

Failed to get logs for machine node-drain-408q19-control-plane-p68qk, cluster node-drain-0l8lzp/node-drain-408q19: dialing public load balancer at node-drain-408q19-acfea8c8.westus2.cloudapp.azure.com: dial tcp 20.109.134.100:22: connect: connection timed out
STEP: Dumping workload cluster node-drain-0l8lzp/node-drain-408q19 kube-system pod logs
STEP: Fetching kube-system pod logs took 610.243898ms
STEP: Collecting events for Pod kube-system/calico-node-vvqdw
STEP: Collecting events for Pod kube-system/coredns-64897985d-cr6vs
STEP: Creating log watcher for controller kube-system/calico-node-vvqdw, container calico-node
STEP: Dumping workload cluster node-drain-0l8lzp/node-drain-408q19 Azure activity log
... skipping 30 lines ...
/home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure/test/e2e/capi_test.go:45
  Should successfully set and use node drain timeout
  /home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure/test/e2e/capi_test.go:195
    A node should be forcefully removed if it cannot be drained in time
    /home/prow/go/pkg/mod/sigs.k8s.io/cluster-api/test@v1.2.1/e2e/node_drain_timeout.go:83
------------------------------
{"component":"entrypoint","file":"k8s.io/test-infra/prow/entrypoint/run.go:165","func":"k8s.io/test-infra/prow/entrypoint.Options.ExecuteProcess","level":"error","msg":"Process did not finish before 4h0m0s timeout","severity":"error","time":"2022-09-30T00:37:40Z"}
++ early_exit_handler
++ '[' -n 160 ']'
++ kill -TERM 160
++ cleanup_dind
++ [[ true == \t\r\u\e ]]
++ echo 'Cleaning up after docker'
... skipping 12 lines ...
Cleaning up after docker
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Stopping Docker: dockerProgram process in pidfile '/var/run/docker-ssd.pid', 1 process(es), refused to die.
================================================================================
Done cleaning up after docker in docker.
All sensitive variables are redacted
{"component":"entrypoint","file":"k8s.io/test-infra/prow/entrypoint/run.go:255","func":"k8s.io/test-infra/prow/entrypoint.gracefullyTerminate","level":"error","msg":"Process did not exit before 15m0s grace period","severity":"error","time":"2022-09-30T00:52:40Z"}
{"component":"entrypoint","error":"os: process already finished","file":"k8s.io/test-infra/prow/entrypoint/run.go:257","func":"k8s.io/test-infra/prow/entrypoint.gracefullyTerminate","level":"error","msg":"Could not kill process after grace period","severity":"error","time":"2022-09-30T00:52:40Z"}