This job view page is being replaced by Spyglass soon. Check out the new job view.
ResultFAILURE
Tests 0 failed / 6 succeeded
Started2023-01-18 17:00
Elapsed4h15m
Revisionrelease-1.6

No Test Failures!


Show 6 Passed Tests

Show 14 Skipped Tests

Error lines from build-log.txt

... skipping 602 lines ...
STEP: Collecting events for Pod kube-system/kube-scheduler-mhc-remediation-a0b5ms-control-plane-htfrc
STEP: Creating log watcher for controller kube-system/etcd-mhc-remediation-a0b5ms-control-plane-htfrc, container etcd
STEP: Creating log watcher for controller kube-system/coredns-bd6b6df9f-gh89h, container coredns
STEP: Creating log watcher for controller kube-system/kube-apiserver-mhc-remediation-a0b5ms-control-plane-htfrc, container kube-apiserver
STEP: Creating log watcher for controller kube-system/coredns-bd6b6df9f-nqd24, container coredns
STEP: Collecting events for Pod kube-system/etcd-mhc-remediation-a0b5ms-control-plane-htfrc
STEP: failed to find events of Pod "etcd-mhc-remediation-a0b5ms-control-plane-htfrc"
STEP: Collecting events for Pod kube-system/kube-apiserver-mhc-remediation-a0b5ms-control-plane-htfrc
STEP: Collecting events for Pod kube-system/calico-node-2fxcm
STEP: Creating log watcher for controller kube-system/kube-controller-manager-mhc-remediation-a0b5ms-control-plane-htfrc, container kube-controller-manager
STEP: Creating log watcher for controller kube-system/calico-node-5h8mg, container calico-node
STEP: Collecting events for Pod kube-system/coredns-bd6b6df9f-nqd24
STEP: Collecting events for Pod kube-system/kube-proxy-9vdgx
... skipping 22 lines ...
  Should pivot the bootstrap cluster to a self-hosted cluster
  /home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure/test/e2e/azure_selfhosted.go:107

INFO: "Should pivot the bootstrap cluster to a self-hosted cluster" started at Wed, 18 Jan 2023 17:10:26 UTC on Ginkgo node 9 of 10
STEP: Creating namespace "self-hosted" for hosting the cluster
Jan 18 17:10:26.962: INFO: starting to create namespace for hosting the "self-hosted" test spec
2023/01/18 17:10:26 failed trying to get namespace (self-hosted):namespaces "self-hosted" not found
INFO: Creating namespace self-hosted
INFO: Creating event watcher for namespace "self-hosted"
STEP: Creating a workload cluster
INFO: Creating the workload cluster with name "self-hosted-v49arj" using the "management" template (Kubernetes v1.23.15, 1 control-plane machines, 1 worker machines)
INFO: Getting the cluster template yaml
INFO: clusterctl config cluster self-hosted-v49arj --infrastructure (default) --kubernetes-version v1.23.15 --control-plane-machine-count 1 --worker-machine-count 1 --flavor management
... skipping 68 lines ...
STEP: Collecting events for Pod kube-system/kube-scheduler-self-hosted-v49arj-control-plane-hhdmf
STEP: Collecting events for Pod kube-system/kube-proxy-p5tjv
STEP: Collecting events for Pod kube-system/kube-proxy-qfmhf
STEP: Creating log watcher for controller kube-system/kube-proxy-qfmhf, container kube-proxy
STEP: Collecting events for Pod kube-system/coredns-bd6b6df9f-jkfw9
STEP: Collecting events for Pod kube-system/kube-controller-manager-self-hosted-v49arj-control-plane-hhdmf
STEP: failed to find events of Pod "kube-controller-manager-self-hosted-v49arj-control-plane-hhdmf"
STEP: Collecting events for Pod kube-system/coredns-bd6b6df9f-wnzj4
STEP: Creating log watcher for controller kube-system/kube-proxy-p5tjv, container kube-proxy
STEP: Creating log watcher for controller kube-system/etcd-self-hosted-v49arj-control-plane-hhdmf, container etcd
STEP: Collecting events for Pod kube-system/kube-apiserver-self-hosted-v49arj-control-plane-hhdmf
STEP: Creating log watcher for controller kube-system/kube-scheduler-self-hosted-v49arj-control-plane-hhdmf, container kube-scheduler
STEP: failed to find events of Pod "kube-scheduler-self-hosted-v49arj-control-plane-hhdmf"
STEP: Creating log watcher for controller kube-system/calico-node-576fm, container calico-node
STEP: Creating log watcher for controller kube-system/calico-kube-controllers-85f479877b-94ntl, container calico-kube-controllers
STEP: failed to find events of Pod "kube-apiserver-self-hosted-v49arj-control-plane-hhdmf"
STEP: Collecting events for Pod kube-system/calico-node-wz7mg
STEP: Creating log watcher for controller kube-system/coredns-bd6b6df9f-wnzj4, container coredns
STEP: Creating log watcher for controller kube-system/coredns-bd6b6df9f-jkfw9, container coredns
STEP: Collecting events for Pod kube-system/calico-kube-controllers-85f479877b-94ntl
STEP: Collecting events for Pod kube-system/etcd-self-hosted-v49arj-control-plane-hhdmf
STEP: failed to find events of Pod "etcd-self-hosted-v49arj-control-plane-hhdmf"
STEP: Fetching activity logs took 1.612944008s
Jan 18 17:21:34.793: INFO: Dumping all the Cluster API resources in the "self-hosted" namespace
Jan 18 17:21:35.115: INFO: Deleting all clusters in the self-hosted namespace
STEP: Deleting cluster self-hosted-v49arj
INFO: Waiting for the Cluster self-hosted/self-hosted-v49arj to be deleted
STEP: Waiting for cluster self-hosted-v49arj to be deleted
... skipping 74 lines ...

Jan 18 17:19:49.560: INFO: Collecting logs for Windows node quick-sta-95pkw in cluster quick-start-mnp8hb in namespace quick-start-5rzasq

Jan 18 17:22:27.111: INFO: Attempting to copy file /c:/crashdumps.tar on node quick-sta-95pkw to /logs/artifacts/clusters/quick-start-mnp8hb/machines/quick-start-mnp8hb-md-win-95cb8cb86-485gr/crashdumps.tar
Jan 18 17:22:30.454: INFO: Collecting boot logs for AzureMachine quick-start-mnp8hb-md-win-95pkw

Failed to get logs for machine quick-start-mnp8hb-md-win-95cb8cb86-485gr, cluster quick-start-5rzasq/quick-start-mnp8hb: [running command "Get-Content "C:\\cni.log"": Process exited with status 1, running command "$p = 'c:\localdumps' ; if (Test-Path $p) { tar.exe -cvzf c:\crashdumps.tar $p *>&1 | %{ Write-Output "$_"} } else { Write-Host "No crash dumps found at $p" }": Process exited with status 1]
Jan 18 17:22:31.959: INFO: Collecting logs for Windows node quick-sta-rcvz9 in cluster quick-start-mnp8hb in namespace quick-start-5rzasq

Jan 18 17:25:08.848: INFO: Attempting to copy file /c:/crashdumps.tar on node quick-sta-rcvz9 to /logs/artifacts/clusters/quick-start-mnp8hb/machines/quick-start-mnp8hb-md-win-95cb8cb86-7xf77/crashdumps.tar
Jan 18 17:25:12.157: INFO: Collecting boot logs for AzureMachine quick-start-mnp8hb-md-win-rcvz9

Failed to get logs for machine quick-start-mnp8hb-md-win-95cb8cb86-7xf77, cluster quick-start-5rzasq/quick-start-mnp8hb: [running command "Get-Content "C:\\cni.log"": Process exited with status 1, running command "$p = 'c:\localdumps' ; if (Test-Path $p) { tar.exe -cvzf c:\crashdumps.tar $p *>&1 | %{ Write-Output "$_"} } else { Write-Host "No crash dumps found at $p" }": Process exited with status 1]
STEP: Dumping workload cluster quick-start-5rzasq/quick-start-mnp8hb kube-system pod logs
STEP: Fetching kube-system pod logs took 1.157556476s
STEP: Collecting events for Pod kube-system/calico-node-windows-d8gsm
STEP: Collecting events for Pod kube-system/csi-proxy-hlrjr
STEP: Collecting events for Pod kube-system/kube-scheduler-quick-start-mnp8hb-control-plane-crlxl
STEP: Collecting events for Pod kube-system/calico-node-zc56r
STEP: Dumping workload cluster quick-start-5rzasq/quick-start-mnp8hb Azure activity log
STEP: Creating log watcher for controller kube-system/csi-proxy-ps7pz, container csi-proxy
STEP: failed to find events of Pod "kube-scheduler-quick-start-mnp8hb-control-plane-crlxl"
STEP: Collecting events for Pod kube-system/containerd-logger-tvvxt
STEP: Collecting events for Pod kube-system/kube-proxy-windows-hqrxq
STEP: Collecting events for Pod kube-system/containerd-logger-4lv2h
STEP: Collecting events for Pod kube-system/calico-kube-controllers-85f479877b-p7z82
STEP: Creating log watcher for controller kube-system/containerd-logger-tvvxt, container containerd-logger
STEP: Creating log watcher for controller kube-system/calico-node-fljwh, container calico-node
... skipping 6 lines ...
STEP: Collecting events for Pod kube-system/calico-node-fljwh
STEP: Creating log watcher for controller kube-system/calico-node-windows-d8gsm, container calico-node-startup
STEP: Creating log watcher for controller kube-system/calico-kube-controllers-85f479877b-p7z82, container calico-kube-controllers
STEP: Collecting events for Pod kube-system/kube-apiserver-quick-start-mnp8hb-control-plane-crlxl
STEP: Collecting events for Pod kube-system/etcd-quick-start-mnp8hb-control-plane-crlxl
STEP: Creating log watcher for controller kube-system/kube-controller-manager-quick-start-mnp8hb-control-plane-crlxl, container kube-controller-manager
STEP: failed to find events of Pod "etcd-quick-start-mnp8hb-control-plane-crlxl"
STEP: failed to find events of Pod "kube-apiserver-quick-start-mnp8hb-control-plane-crlxl"
STEP: Creating log watcher for controller kube-system/kube-proxy-hsvg2, container kube-proxy
STEP: Creating log watcher for controller kube-system/calico-node-windows-d8gsm, container calico-node-felix
STEP: Collecting events for Pod kube-system/kube-controller-manager-quick-start-mnp8hb-control-plane-crlxl
STEP: Collecting events for Pod kube-system/coredns-bd6b6df9f-cccdg
STEP: Collecting events for Pod kube-system/kube-proxy-hsvg2
STEP: Collecting events for Pod kube-system/kube-proxy-st54f
... skipping 3 lines ...
STEP: Creating log watcher for controller kube-system/kube-proxy-windows-jp56d, container kube-proxy
STEP: Creating log watcher for controller kube-system/calico-node-windows-q5xcv, container calico-node-felix
STEP: Collecting events for Pod kube-system/kube-proxy-windows-jp56d
STEP: Collecting events for Pod kube-system/calico-node-windows-q5xcv
STEP: Creating log watcher for controller kube-system/kube-scheduler-quick-start-mnp8hb-control-plane-crlxl, container kube-scheduler
STEP: Creating log watcher for controller kube-system/calico-node-zc56r, container calico-node
STEP: failed to find events of Pod "kube-controller-manager-quick-start-mnp8hb-control-plane-crlxl"
STEP: Creating log watcher for controller kube-system/kube-proxy-st54f, container kube-proxy
STEP: Creating log watcher for controller kube-system/calico-node-windows-q5xcv, container calico-node-startup
STEP: Fetching activity logs took 1.534257849s
STEP: Dumping all the Cluster API resources in the "quick-start-5rzasq" namespace
STEP: Deleting cluster quick-start-5rzasq/quick-start-mnp8hb
STEP: Deleting cluster quick-start-mnp8hb
... skipping 86 lines ...
STEP: Creating log watcher for controller kube-system/calico-node-windows-ddxsl, container calico-node-startup
STEP: Creating log watcher for controller kube-system/kube-controller-manager-machine-pool-2lv7at-control-plane-r5m4c, container kube-controller-manager
STEP: Creating log watcher for controller kube-system/calico-kube-controllers-85f479877b-9g7xf, container calico-kube-controllers
STEP: Collecting events for Pod kube-system/kube-proxy-vg7hb
STEP: Creating log watcher for controller kube-system/kube-scheduler-machine-pool-2lv7at-control-plane-r5m4c, container kube-scheduler
STEP: Collecting events for Pod kube-system/kube-controller-manager-machine-pool-2lv7at-control-plane-r5m4c
STEP: failed to find events of Pod "kube-controller-manager-machine-pool-2lv7at-control-plane-r5m4c"
STEP: Creating log watcher for controller kube-system/kube-proxy-pxc9r, container kube-proxy
STEP: Collecting events for Pod kube-system/kube-scheduler-machine-pool-2lv7at-control-plane-r5m4c
STEP: failed to find events of Pod "kube-scheduler-machine-pool-2lv7at-control-plane-r5m4c"
STEP: Collecting events for Pod kube-system/calico-kube-controllers-85f479877b-9g7xf
STEP: Creating log watcher for controller kube-system/coredns-bd6b6df9f-m5sq4, container coredns
STEP: Collecting events for Pod kube-system/coredns-bd6b6df9f-m5sq4
STEP: Creating log watcher for controller kube-system/etcd-machine-pool-2lv7at-control-plane-r5m4c, container etcd
STEP: Creating log watcher for controller kube-system/kube-proxy-windows-x7rkc, container kube-proxy
STEP: Collecting events for Pod kube-system/kube-proxy-windows-x7rkc
STEP: Collecting events for Pod kube-system/calico-node-windows-ddxsl
STEP: Creating log watcher for controller kube-system/calico-node-windows-ddxsl, container calico-node-felix
STEP: Collecting events for Pod kube-system/kube-proxy-pxc9r
STEP: Creating log watcher for controller kube-system/coredns-bd6b6df9f-95gkj, container coredns
STEP: Creating log watcher for controller kube-system/kube-apiserver-machine-pool-2lv7at-control-plane-r5m4c, container kube-apiserver
STEP: Creating log watcher for controller kube-system/kube-proxy-vg7hb, container kube-proxy
STEP: Collecting events for Pod kube-system/kube-apiserver-machine-pool-2lv7at-control-plane-r5m4c
STEP: failed to find events of Pod "kube-apiserver-machine-pool-2lv7at-control-plane-r5m4c"
STEP: Collecting events for Pod kube-system/etcd-machine-pool-2lv7at-control-plane-r5m4c
STEP: failed to find events of Pod "etcd-machine-pool-2lv7at-control-plane-r5m4c"
STEP: Collecting events for Pod kube-system/coredns-bd6b6df9f-95gkj
STEP: Error starting logs stream for pod kube-system/calico-node-7jjg7, container calico-node: pods "machine-pool-2lv7at-mp-0000002" not found
STEP: Error starting logs stream for pod kube-system/calico-node-windows-ddxsl, container calico-node-startup: pods "win-p-win000002" not found
STEP: Error starting logs stream for pod kube-system/calico-node-windows-ddxsl, container calico-node-felix: pods "win-p-win000002" not found
STEP: Error starting logs stream for pod kube-system/kube-proxy-windows-x7rkc, container kube-proxy: pods "win-p-win000002" not found
STEP: Error starting logs stream for pod kube-system/kube-proxy-pxc9r, container kube-proxy: pods "machine-pool-2lv7at-mp-0000002" not found
STEP: Fetching activity logs took 2.256936645s
STEP: Dumping all the Cluster API resources in the "machine-pool-ryc4wu" namespace
STEP: Deleting cluster machine-pool-ryc4wu/machine-pool-2lv7at
STEP: Deleting cluster machine-pool-2lv7at
INFO: Waiting for the Cluster machine-pool-ryc4wu/machine-pool-2lv7at to be deleted
STEP: Waiting for cluster machine-pool-2lv7at to be deleted
... skipping 214 lines ...

Jan 18 17:23:01.231: INFO: Collecting logs for Windows node md-scale-ghb9q in cluster md-scale-ht9d7r in namespace md-scale-afr0u8

Jan 18 17:25:40.093: INFO: Attempting to copy file /c:/crashdumps.tar on node md-scale-ghb9q to /logs/artifacts/clusters/md-scale-ht9d7r/machines/md-scale-ht9d7r-md-win-56c54cd9cc-2v6c8/crashdumps.tar
Jan 18 17:25:43.456: INFO: Collecting boot logs for AzureMachine md-scale-ht9d7r-md-win-ghb9q

Failed to get logs for machine md-scale-ht9d7r-md-win-56c54cd9cc-2v6c8, cluster md-scale-afr0u8/md-scale-ht9d7r: [running command "Get-Content "C:\\cni.log"": Process exited with status 1, running command "$p = 'c:\localdumps' ; if (Test-Path $p) { tar.exe -cvzf c:\crashdumps.tar $p *>&1 | %{ Write-Output "$_"} } else { Write-Host "No crash dumps found at $p" }": Process exited with status 1]
Jan 18 17:25:44.984: INFO: Collecting logs for Windows node md-scale-kb4st in cluster md-scale-ht9d7r in namespace md-scale-afr0u8

Jan 18 17:28:23.189: INFO: Attempting to copy file /c:/crashdumps.tar on node md-scale-kb4st to /logs/artifacts/clusters/md-scale-ht9d7r/machines/md-scale-ht9d7r-md-win-56c54cd9cc-g5kts/crashdumps.tar
Jan 18 17:28:26.589: INFO: Collecting boot logs for AzureMachine md-scale-ht9d7r-md-win-kb4st

Failed to get logs for machine md-scale-ht9d7r-md-win-56c54cd9cc-g5kts, cluster md-scale-afr0u8/md-scale-ht9d7r: [running command "Get-Content "C:\\cni.log"": Process exited with status 1, running command "$p = 'c:\localdumps' ; if (Test-Path $p) { tar.exe -cvzf c:\crashdumps.tar $p *>&1 | %{ Write-Output "$_"} } else { Write-Host "No crash dumps found at $p" }": Process exited with status 1]
STEP: Dumping workload cluster md-scale-afr0u8/md-scale-ht9d7r kube-system pod logs
STEP: Fetching kube-system pod logs took 1.104785798s
STEP: Dumping workload cluster md-scale-afr0u8/md-scale-ht9d7r Azure activity log
STEP: Collecting events for Pod kube-system/calico-node-windows-vzkhn
STEP: Collecting events for Pod kube-system/etcd-md-scale-ht9d7r-control-plane-dcc6k
STEP: Creating log watcher for controller kube-system/calico-kube-controllers-85f479877b-74b7g, container calico-kube-controllers
... skipping 110 lines ...
STEP: Dumping logs from the "node-drain-kj1vr5" workload cluster
STEP: Dumping workload cluster node-drain-dmvb25/node-drain-kj1vr5 logs
Jan 18 17:28:27.529: INFO: Collecting logs for Linux node node-drain-kj1vr5-control-plane-r2d4g in cluster node-drain-kj1vr5 in namespace node-drain-dmvb25

Jan 18 17:35:01.965: INFO: Collecting boot logs for AzureMachine node-drain-kj1vr5-control-plane-r2d4g

Failed to get logs for machine node-drain-kj1vr5-control-plane-82npc, cluster node-drain-dmvb25/node-drain-kj1vr5: dialing public load balancer at node-drain-kj1vr5-5a98ca87.uksouth.cloudapp.azure.com: dial tcp 20.108.113.31:22: connect: connection timed out
STEP: Dumping workload cluster node-drain-dmvb25/node-drain-kj1vr5 kube-system pod logs
STEP: Fetching kube-system pod logs took 1.08534499s
STEP: Dumping workload cluster node-drain-dmvb25/node-drain-kj1vr5 Azure activity log
STEP: Creating log watcher for controller kube-system/calico-kube-controllers-85f479877b-czqk9, container calico-kube-controllers
STEP: Creating log watcher for controller kube-system/kube-apiserver-node-drain-kj1vr5-control-plane-r2d4g, container kube-apiserver
STEP: Collecting events for Pod kube-system/kube-apiserver-node-drain-kj1vr5-control-plane-r2d4g
... skipping 30 lines ...
/home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure/test/e2e/capi_test.go:45
  Should successfully set and use node drain timeout
  /home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure/test/e2e/capi_test.go:183
    A node should be forcefully removed if it cannot be drained in time
    /home/prow/go/pkg/mod/sigs.k8s.io/cluster-api/test@v1.2.9/e2e/node_drain_timeout.go:83
------------------------------
{"component":"entrypoint","file":"k8s.io/test-infra/prow/entrypoint/run.go:164","func":"k8s.io/test-infra/prow/entrypoint.Options.ExecuteProcess","level":"error","msg":"Process did not finish before 4h0m0s timeout","severity":"error","time":"2023-01-18T21:00:29Z"}
++ early_exit_handler
++ '[' -n 162 ']'
++ kill -TERM 162
++ cleanup_dind
++ [[ true == \t\r\u\e ]]
++ echo 'Cleaning up after docker'
... skipping 12 lines ...
Cleaning up after docker
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Stopping Docker: dockerProgram process in pidfile '/var/run/docker-ssd.pid', 1 process(es), refused to die.
================================================================================
Done cleaning up after docker in docker.
All sensitive variables are redacted
{"component":"entrypoint","file":"k8s.io/test-infra/prow/entrypoint/run.go:254","func":"k8s.io/test-infra/prow/entrypoint.gracefullyTerminate","level":"error","msg":"Process did not exit before 15m0s grace period","severity":"error","time":"2023-01-18T21:15:29Z"}
{"component":"entrypoint","error":"os: process already finished","file":"k8s.io/test-infra/prow/entrypoint/run.go:256","func":"k8s.io/test-infra/prow/entrypoint.gracefullyTerminate","level":"error","msg":"Could not kill process after grace period","severity":"error","time":"2023-01-18T21:15:29Z"}