This job view page is being replaced by Spyglass soon. Check out the new job view.
ResultFAILURE
Tests 1 failed / 27 succeeded
Started2023-01-19 21:13
Elapsed1h23m
Revisionrelease-1.7

Test Failures

capz-e2e [It] Running the Cluster API E2E tests Should successfully remediate unhealthy machines with MachineHealthCheck Should successfully trigger KCP remediation 1h11m

go run hack/e2e.go -v --test --test_args='--ginkgo.focus=capz\-e2e\s\[It\]\sRunning\sthe\sCluster\sAPI\sE2E\stests\sShould\ssuccessfully\sremediate\sunhealthy\smachines\swith\sMachineHealthCheck\sShould\ssuccessfully\strigger\sKCP\sremediation$'
[FAILED] Timed out after 1800.001s.
Expected
    <bool>: false
to be true
In [It] at: /home/prow/go/pkg/mod/sigs.k8s.io/cluster-api/test@v1.3.1/framework/machinehealthcheck_helpers.go:168 @ 01/19/23 22:01:57.066

There were additional failures detected after the initial failure. These are visible in the timeline

				
				Click to see stdout/stderrfrom junit.e2e_suite.1.xml

Filter through log files

Show 27 Passed Tests
Show 18 Skipped Tests

Error lines from build-log.txt

... skipping 785 lines ...
  Jan 19 21:33:17.504: INFO: Collecting events for Pod kube-system/kube-apiserver-mhc-remediation-5ymxn6-control-plane-s7x99
  Jan 19 21:33:17.501: INFO: Collecting events for Pod kube-system/coredns-57575c5f89-ctz7c
  Jan 19 21:33:17.504: INFO: Creating log watcher for controller kube-system/kube-controller-manager-mhc-remediation-5ymxn6-control-plane-s7x99, container kube-controller-manager
  Jan 19 21:33:17.505: INFO: Creating log watcher for controller kube-system/kube-proxy-4hxds, container kube-proxy
  Jan 19 21:33:17.506: INFO: Collecting events for Pod kube-system/kube-proxy-4hxds
  Jan 19 21:33:17.506: INFO: Collecting events for Pod kube-system/csi-azuredisk-controller-545d478dbf-g55j7
  Jan 19 21:33:17.570: INFO: Error starting logs stream for pod calico-system/csi-node-driver-pj7wq, container csi-node-driver-registrar: container "csi-node-driver-registrar" in pod "csi-node-driver-pj7wq" is waiting to start: ContainerCreating
  Jan 19 21:33:17.571: INFO: Error starting logs stream for pod calico-system/calico-node-zcmp7, container calico-node: container "calico-node" in pod "calico-node-zcmp7" is waiting to start: PodInitializing
  Jan 19 21:33:17.581: INFO: Error starting logs stream for pod calico-system/csi-node-driver-pj7wq, container calico-csi: container "calico-csi" in pod "csi-node-driver-pj7wq" is waiting to start: ContainerCreating
  Jan 19 21:33:17.584: INFO: Fetching kube-system pod logs took 702.266385ms
  Jan 19 21:33:17.584: INFO: Dumping workload cluster mhc-remediation-yujpmj/mhc-remediation-5ymxn6 Azure activity log
  Jan 19 21:33:17.584: INFO: Creating log watcher for controller tigera-operator/tigera-operator-65d6bf4d4f-8mfxm, container tigera-operator
  Jan 19 21:33:17.584: INFO: Collecting events for Pod tigera-operator/tigera-operator-65d6bf4d4f-8mfxm
  Jan 19 21:33:19.871: INFO: Fetching activity logs took 2.28658284s
  STEP: Dumping all the Cluster API resources in the "mhc-remediation-yujpmj" namespace @ 01/19/23 21:33:19.871
... skipping 14 lines ...
------------------------------
• [1016.410 seconds]
Running the Cluster API E2E tests Running the self-hosted spec Should pivot the bootstrap cluster to a self-hosted cluster
/home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure/test/e2e/azure_selfhosted.go:108

  Captured StdOut/StdErr Output >>
  2023/01/19 21:23:11 failed trying to get namespace (self-hosted):namespaces "self-hosted" not found
  kubeadmconfigtemplate.bootstrap.cluster.x-k8s.io/self-hosted-fk5l4z-md-0 created
  cluster.cluster.x-k8s.io/self-hosted-fk5l4z created
  machinedeployment.cluster.x-k8s.io/self-hosted-fk5l4z-md-0 created
  kubeadmcontrolplane.controlplane.cluster.x-k8s.io/self-hosted-fk5l4z-control-plane created
  azurecluster.infrastructure.cluster.x-k8s.io/self-hosted-fk5l4z created
  azureclusteridentity.infrastructure.cluster.x-k8s.io/cluster-identity-sp created
... skipping 236 lines ...
  azureclusteridentity.infrastructure.cluster.x-k8s.io/cluster-identity-sp created
  azuremachinetemplate.infrastructure.cluster.x-k8s.io/node-drain-i9w5pm-control-plane created
  azuremachinetemplate.infrastructure.cluster.x-k8s.io/node-drain-i9w5pm-md-0 created

  felixconfiguration.crd.projectcalico.org/default configured

  Failed to get logs for Machine node-drain-i9w5pm-control-plane-n7zvw, Cluster node-drain-dl6m4h/node-drain-i9w5pm: dialing public load balancer at node-drain-i9w5pm-dda2cb04.westus3.cloudapp.azure.com: dial tcp 20.25.170.6:22: connect: connection timed out
  << Captured StdOut/StdErr Output

  Timeline >>
  INFO: "" started at Thu, 19 Jan 2023 21:23:11 UTC on Ginkgo node 6 of 10 and junit test report to file /logs/artifacts/test_e2e_junit.e2e_suite.1.xml
  STEP: Creating a namespace for hosting the "node-drain" test spec @ 01/19/23 21:23:11.301
  INFO: Creating namespace node-drain-dl6m4h
... skipping 200 lines ...
  configmap/cni-quick-start-s4i3b1-calico-windows created
  configmap/csi-proxy-addon created
  configmap/containerd-logger-quick-start-s4i3b1 created

  felixconfiguration.crd.projectcalico.org/default created

  Failed to get logs for Machine quick-start-s4i3b1-md-win-56869465bb-l5jll, Cluster quick-start-14yz1t/quick-start-s4i3b1: [running command "Get-Content "C:\\cni.log"": Process exited with status 1, running command "$p = 'c:\localdumps' ; if (Test-Path $p) { tar.exe -cvzf c:\crashdumps.tar $p *>&1 | %{ Write-Output "$_"} } else { Write-Host "No crash dumps found at $p" }": Process exited with status 1]
  Failed to get logs for Machine quick-start-s4i3b1-md-win-56869465bb-pbjv4, Cluster quick-start-14yz1t/quick-start-s4i3b1: [running command "Get-Content "C:\\cni.log"": Process exited with status 1, running command "$p = 'c:\localdumps' ; if (Test-Path $p) { tar.exe -cvzf c:\crashdumps.tar $p *>&1 | %{ Write-Output "$_"} } else { Write-Host "No crash dumps found at $p" }": Process exited with status 1]
  << Captured StdOut/StdErr Output

  Timeline >>
  INFO: "" started at Thu, 19 Jan 2023 21:23:11 UTC on Ginkgo node 1 of 10 and junit test report to file /logs/artifacts/test_e2e_junit.e2e_suite.1.xml
  STEP: Creating a namespace for hosting the "quick-start" test spec @ 01/19/23 21:23:11.137
  INFO: Creating namespace quick-start-14yz1t
... skipping 231 lines ...
  configmap/cni-md-scale-8udbpz-calico-windows created
  configmap/csi-proxy-addon created
  configmap/containerd-logger-md-scale-8udbpz created

  felixconfiguration.crd.projectcalico.org/default configured

  Failed to get logs for Machine md-scale-8udbpz-md-win-9d7f5b5dc-2ql2d, Cluster md-scale-opky54/md-scale-8udbpz: [running command "Get-Content "C:\\cni.log"": Process exited with status 1, running command "$p = 'c:\localdumps' ; if (Test-Path $p) { tar.exe -cvzf c:\crashdumps.tar $p *>&1 | %{ Write-Output "$_"} } else { Write-Host "No crash dumps found at $p" }": Process exited with status 1]
  Failed to get logs for Machine md-scale-8udbpz-md-win-9d7f5b5dc-7wp8w, Cluster md-scale-opky54/md-scale-8udbpz: [running command "Get-Content "C:\\cni.log"": Process exited with status 1, running command "$p = 'c:\localdumps' ; if (Test-Path $p) { tar.exe -cvzf c:\crashdumps.tar $p *>&1 | %{ Write-Output "$_"} } else { Write-Host "No crash dumps found at $p" }": Process exited with status 1]
  << Captured StdOut/StdErr Output

  Timeline >>
  INFO: "" started at Thu, 19 Jan 2023 21:23:11 UTC on Ginkgo node 8 of 10 and junit test report to file /logs/artifacts/test_e2e_junit.e2e_suite.1.xml
  STEP: Creating a namespace for hosting the "md-scale" test spec @ 01/19/23 21:23:11.381
  INFO: Creating namespace md-scale-opky54
... skipping 383 lines ...
  Jan 19 21:57:35.743: INFO: Creating log watcher for controller calico-system/calico-node-windows-5567f, container calico-node-startup
  Jan 19 21:57:35.744: INFO: Creating log watcher for controller calico-system/csi-node-driver-b5fjj, container csi-node-driver-registrar
  Jan 19 21:57:35.744: INFO: Creating log watcher for controller calico-system/csi-node-driver-hhpmr, container csi-node-driver-registrar
  Jan 19 21:57:35.744: INFO: Collecting events for Pod calico-system/csi-node-driver-hhpmr
  Jan 19 21:57:35.744: INFO: Creating log watcher for controller calico-system/calico-kube-controllers-594d54f99-qflp7, container calico-kube-controllers
  Jan 19 21:57:35.745: INFO: Creating log watcher for controller calico-system/calico-node-lhbkd, container calico-node
  Jan 19 21:57:35.809: INFO: Error starting logs stream for pod calico-system/csi-node-driver-hhpmr, container csi-node-driver-registrar: pods "machine-pool-efnoqe-mp-0000002" not found
  Jan 19 21:57:35.811: INFO: Error starting logs stream for pod calico-system/calico-node-dpcpc, container calico-node: pods "machine-pool-efnoqe-mp-0000002" not found
  Jan 19 21:57:35.811: INFO: Error starting logs stream for pod calico-system/calico-node-windows-5567f, container calico-node-felix: pods "win-p-win000002" not found
  Jan 19 21:57:35.833: INFO: Error starting logs stream for pod calico-system/csi-node-driver-hhpmr, container calico-csi: pods "machine-pool-efnoqe-mp-0000002" not found
  Jan 19 21:57:35.833: INFO: Error starting logs stream for pod calico-system/calico-node-windows-5567f, container calico-node-startup: pods "win-p-win000002" not found
  Jan 19 21:57:35.840: INFO: Creating log watcher for controller kube-system/coredns-57575c5f89-87lt4, container coredns
  Jan 19 21:57:35.840: INFO: Collecting events for Pod kube-system/coredns-57575c5f89-jmf5q
  Jan 19 21:57:35.840: INFO: Collecting events for Pod kube-system/coredns-57575c5f89-87lt4
  Jan 19 21:57:35.840: INFO: Creating log watcher for controller kube-system/coredns-57575c5f89-jmf5q, container coredns
  Jan 19 21:57:35.840: INFO: Creating log watcher for controller kube-system/csi-azuredisk-node-x6cw2, container node-driver-registrar
  Jan 19 21:57:35.840: INFO: Creating log watcher for controller kube-system/kube-controller-manager-machine-pool-efnoqe-control-plane-75pdm, container kube-controller-manager
... skipping 29 lines ...
  Jan 19 21:57:35.844: INFO: Creating log watcher for controller kube-system/csi-azuredisk-controller-545d478dbf-m8kqt, container csi-resizer
  Jan 19 21:57:35.844: INFO: Creating log watcher for controller kube-system/csi-azuredisk-node-win-kbsh2, container liveness-probe
  Jan 19 21:57:35.844: INFO: Creating log watcher for controller kube-system/csi-azuredisk-controller-545d478dbf-m8kqt, container azuredisk
  Jan 19 21:57:35.845: INFO: Collecting events for Pod kube-system/csi-azuredisk-controller-545d478dbf-m8kqt
  Jan 19 21:57:35.845: INFO: Creating log watcher for controller kube-system/csi-azuredisk-controller-545d478dbf-m8kqt, container csi-attacher
  Jan 19 21:57:35.845: INFO: Creating log watcher for controller kube-system/csi-azuredisk-node-qh9x8, container liveness-probe
  Jan 19 21:57:35.990: INFO: Error starting logs stream for pod kube-system/csi-azuredisk-node-x6cw2, container node-driver-registrar: pods "machine-pool-efnoqe-mp-0000002" not found
  Jan 19 21:57:35.990: INFO: Error starting logs stream for pod kube-system/csi-azuredisk-node-win-kbsh2, container liveness-probe: pods "win-p-win000002" not found
  Jan 19 21:57:35.990: INFO: Error starting logs stream for pod kube-system/csi-azuredisk-node-win-kbsh2, container azuredisk: pods "win-p-win000002" not found
  Jan 19 21:57:35.990: INFO: Error starting logs stream for pod kube-system/csi-azuredisk-node-win-kbsh2, container node-driver-registrar: pods "win-p-win000002" not found
  Jan 19 21:57:35.990: INFO: Error starting logs stream for pod kube-system/kube-proxy-windows-vv788, container kube-proxy: pods "win-p-win000002" not found
  Jan 19 21:57:35.990: INFO: Error starting logs stream for pod kube-system/csi-azuredisk-node-x6cw2, container liveness-probe: pods "machine-pool-efnoqe-mp-0000002" not found
  Jan 19 21:57:35.990: INFO: Error starting logs stream for pod kube-system/csi-proxy-v2zjx, container csi-proxy: pods "win-p-win000002" not found
  Jan 19 21:57:35.990: INFO: Error starting logs stream for pod kube-system/kube-proxy-nnmm9, container kube-proxy: pods "machine-pool-efnoqe-mp-0000002" not found
  Jan 19 21:57:35.990: INFO: Error starting logs stream for pod kube-system/csi-azuredisk-node-x6cw2, container azuredisk: pods "machine-pool-efnoqe-mp-0000002" not found
  Jan 19 21:57:35.990: INFO: Error starting logs stream for pod kube-system/containerd-logger-6sp2j, container containerd-logger: pods "win-p-win000002" not found
  Jan 19 21:57:35.991: INFO: Fetching kube-system pod logs took 949.257315ms
  Jan 19 21:57:35.991: INFO: Dumping workload cluster machine-pool-2pwnt7/machine-pool-efnoqe Azure activity log
  Jan 19 21:57:35.991: INFO: Creating log watcher for controller tigera-operator/tigera-operator-65d6bf4d4f-cpvrs, container tigera-operator
  Jan 19 21:57:35.991: INFO: Collecting events for Pod tigera-operator/tigera-operator-65d6bf4d4f-cpvrs
  Jan 19 21:57:38.884: INFO: Fetching activity logs took 2.893468251s
  STEP: Dumping all the Cluster API resources in the "machine-pool-2pwnt7" namespace @ 01/19/23 21:57:38.884
... skipping 35 lines ...
  configmap/cni-md-rollout-yeqwuo-calico-windows created
  configmap/csi-proxy-addon created
  configmap/containerd-logger-md-rollout-yeqwuo created

  felixconfiguration.crd.projectcalico.org/default configured

  Failed to get logs for Machine md-rollout-yeqwuo-md-win-6d64964dd7-72n77, Cluster md-rollout-u27nd2/md-rollout-yeqwuo: [dialing from control plane to target node at md-rollou-5wjml: ssh: rejected: connect failed (Temporary failure in name resolution), Unable to collect VM Boot Diagnostic logs: failed to get boot diagnostics data: compute.VirtualMachinesClient#RetrieveBootDiagnosticsData: Failure responding to request: StatusCode=404 -- Original Error: autorest/azure: Service returned an error. Status=404 Code="ResourceNotFound" Message="The Resource 'Microsoft.Compute/virtualMachines/md-rollou-5wjml' under resource group 'capz-e2e-jvbqb3' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"]
  Failed to get logs for Machine md-rollout-yeqwuo-md-win-6d64964dd7-s54jn, Cluster md-rollout-u27nd2/md-rollout-yeqwuo: [running command "Get-Content "C:\\cni.log"": Process exited with status 1, running command "$p = 'c:\localdumps' ; if (Test-Path $p) { tar.exe -cvzf c:\crashdumps.tar $p *>&1 | %{ Write-Output "$_"} } else { Write-Host "No crash dumps found at $p" }": Process exited with status 1]
  Failed to get logs for Machine md-rollout-yeqwuo-md-win-7789c9c8f4-drnj9, Cluster md-rollout-u27nd2/md-rollout-yeqwuo: [running command "Get-Content "C:\\cni.log"": Process exited with status 1, running command "$p = 'c:\localdumps' ; if (Test-Path $p) { tar.exe -cvzf c:\crashdumps.tar $p *>&1 | %{ Write-Output "$_"} } else { Write-Host "No crash dumps found at $p" }": Process exited with status 1]
  << Captured StdOut/StdErr Output

  Timeline >>
  INFO: "" started at Thu, 19 Jan 2023 21:23:11 UTC on Ginkgo node 9 of 10 and junit test report to file /logs/artifacts/test_e2e_junit.e2e_suite.1.xml
  STEP: Creating a namespace for hosting the "md-rollout" test spec @ 01/19/23 21:23:11.219
  INFO: Creating namespace md-rollout-u27nd2
... skipping 225 lines ...
  << Timeline
------------------------------
[SynchronizedAfterSuite] PASSED [0.000 seconds]
[SynchronizedAfterSuite] 
/home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure/test/e2e/e2e_suite_test.go:116
------------------------------
• [FAILED] [4274.186 seconds]
Running the Cluster API E2E tests Should successfully remediate unhealthy machines with MachineHealthCheck [It] Should successfully trigger KCP remediation
/home/prow/go/pkg/mod/sigs.k8s.io/cluster-api/test@v1.3.1/e2e/mhc_remediations.go:116

  Captured StdOut/StdErr Output >>
  kubeadmconfigtemplate.bootstrap.cluster.x-k8s.io/mhc-remediation-ea0ks1-md-0 created
  cluster.cluster.x-k8s.io/mhc-remediation-ea0ks1 created
... skipping 104 lines ...
  Discovering machine health check resources
  Ensuring there is at least 1 Machine that MachineHealthCheck is matching
  Patching MachineHealthCheck unhealthy condition to one of the nodes
  INFO: Patching the node condition to the node
  Waiting for remediation
  Waiting until the node with unhealthy node condition is remediated
  [FAILED] in [It] - /home/prow/go/pkg/mod/sigs.k8s.io/cluster-api/test@v1.3.1/framework/machinehealthcheck_helpers.go:168 @ 01/19/23 22:01:57.066
  STEP: Dumping logs from the "mhc-remediation-ea0ks1" workload cluster @ 01/19/23 22:01:57.066
  Jan 19 22:01:57.066: INFO: Dumping workload cluster mhc-remediation-u1144g/mhc-remediation-ea0ks1 logs
  Jan 19 22:01:57.115: INFO: Collecting logs for Linux node mhc-remediation-ea0ks1-control-plane-4dngj in cluster mhc-remediation-ea0ks1 in namespace mhc-remediation-u1144g

  Jan 19 22:02:11.253: INFO: Collecting boot logs for AzureMachine mhc-remediation-ea0ks1-control-plane-4dngj

... skipping 106 lines ...
  Jan 19 22:02:49.860: INFO: Fetching activity logs took 4.385936556s
  STEP: Dumping all the Cluster API resources in the "mhc-remediation-u1144g" namespace @ 01/19/23 22:02:49.86
  STEP: Deleting cluster mhc-remediation-u1144g/mhc-remediation-ea0ks1 @ 01/19/23 22:02:50.177
  STEP: Deleting cluster mhc-remediation-ea0ks1 @ 01/19/23 22:02:50.193
  INFO: Waiting for the Cluster mhc-remediation-u1144g/mhc-remediation-ea0ks1 to be deleted
  STEP: Waiting for cluster mhc-remediation-ea0ks1 to be deleted @ 01/19/23 22:02:50.206
  [FAILED] in [AfterEach] - /home/prow/go/pkg/mod/sigs.k8s.io/cluster-api/test@v1.3.1/framework/cluster_helpers.go:176 @ 01/19/23 22:32:50.208
  Jan 19 22:32:50.208: INFO: FAILED!
  Jan 19 22:32:50.208: INFO: Cleaning up after "Running the Cluster API E2E tests Should successfully remediate unhealthy machines with MachineHealthCheck Should successfully trigger KCP remediation" spec
  STEP: Redacting sensitive information from logs @ 01/19/23 22:32:50.208
  INFO: "Should successfully trigger KCP remediation" started at Thu, 19 Jan 2023 22:34:25 UTC on Ginkgo node 5 of 10 and junit test report to file /logs/artifacts/test_e2e_junit.e2e_suite.1.xml
  << Timeline

  [FAILED] Timed out after 1800.001s.
  Expected
      <bool>: false
  to be true
  In [It] at: /home/prow/go/pkg/mod/sigs.k8s.io/cluster-api/test@v1.3.1/framework/machinehealthcheck_helpers.go:168 @ 01/19/23 22:01:57.066

  Full Stack Trace
... skipping 21 lines ...
[ReportAfterSuite] PASSED [0.011 seconds]
[ReportAfterSuite] Autogenerated ReportAfterSuite for --junit-report
autogenerated by Ginkgo
------------------------------

Summarizing 1 Failure:
  [FAIL] Running the Cluster API E2E tests Should successfully remediate unhealthy machines with MachineHealthCheck [It] Should successfully trigger KCP remediation
  /home/prow/go/pkg/mod/sigs.k8s.io/cluster-api/test@v1.3.1/framework/machinehealthcheck_helpers.go:168

Ran 8 of 26 Specs in 4436.533 seconds
FAIL! -- 7 Passed | 1 Failed | 0 Pending | 18 Skipped

You're using deprecated Ginkgo functionality:
=============================================
  CurrentGinkgoTestDescription() is deprecated in Ginkgo V2.  Use CurrentSpecReport() instead.
  Learn more at: https://onsi.github.io/ginkgo/MIGRATING_TO_V2#changed-currentginkgotestdescription
    /home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure/test/e2e/helpers.go:423
... skipping 29 lines ...
    /home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure/test/e2e/common.go:281
    /home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure/test/e2e/helpers.go:423

To silence deprecations that can be silenced set the following environment variable:
  ACK_GINKGO_DEPRECATIONS=2.6.0

--- FAIL: TestE2E (4435.10s)
FAIL

You're using deprecated Ginkgo functionality:
=============================================
  CurrentGinkgoTestDescription() is deprecated in Ginkgo V2.  Use CurrentSpecReport() instead.
  Learn more at: https://onsi.github.io/ginkgo/MIGRATING_TO_V2#changed-currentginkgotestdescription
    /home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure/test/e2e/helpers.go:423
... skipping 62 lines ...

PASS


Ginkgo ran 1 suite in 1h16m14.794743546s

Test Suite Failed
make[1]: *** [Makefile:655: test-e2e-run] Error 1
make[1]: Leaving directory '/home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure'
make: *** [Makefile:664: test-e2e] Error 2
================ REDACTING LOGS ================
All sensitive variables are redacted
+ EXIT_VALUE=2
+ set +o xtrace
Cleaning up after docker in docker.
================================================================================
... skipping 5 lines ...