periodic-cluster-api-e2e-workload-upgrade-1-21-1-22-main #1611323322300108800
Recent runs || View in Spyglass
Recent runs || View in Spyglass
| Result | FAILURE |
| Tests | 1 failed / 6 succeeded |
| Started | |
| Elapsed | 30m46s |
| Revision | main |
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=capi\-e2e\s\[It\]\sWhen\supgrading\sa\sworkload\scluster\susing\sClusterClass\sand\stesting\sK8S\sconformance\s\[Conformance\]\s\[K8s\-Upgrade\]\s\[ClusterClass\]\sShould\screate\sand\supgrade\sa\sworkload\scluster\sand\seventually\srun\skubetest$'
[FAILED] Timed out after 390.132s.
Timed out waiting for all MachinePool k8s-upgrade-and-conformance-yi0chr/k8s-upgrade-and-conformance-44o336-mp-0 instances to be upgraded to Kubernetes version v1.22.17
Error: function returned error: old version instances remain
<*fmt.wrapError | 0xc000470360>: {
msg: "function returned error: old version instances remain",
err: <*errors.fundamental | 0xc0009d3308>{
msg: "old version instances remain",
stack: [0x180d55a, 0x4dad65, 0x4da25c, 0x88447a, 0x885382, 0x882ced, 0x180d364, 0x180ba85, 0x1cb420a, 0x86241b, 0x8754b8, 0x4704e1],
},
}
In [It] at: /home/prow/go/src/sigs.k8s.io/cluster-api/test/framework/machinepool_helpers.go:268 @ 01/06/23 11:59:04.084
clusterclass.cluster.x-k8s.io/quick-start created
dockerclustertemplate.infrastructure.cluster.x-k8s.io/quick-start-cluster created
kubeadmcontrolplanetemplate.controlplane.cluster.x-k8s.io/quick-start-control-plane created
dockermachinetemplate.infrastructure.cluster.x-k8s.io/quick-start-control-plane created
dockermachinetemplate.infrastructure.cluster.x-k8s.io/quick-start-default-worker-machinetemplate created
kubeadmconfigtemplate.bootstrap.cluster.x-k8s.io/quick-start-default-worker-bootstraptemplate created
configmap/cni-k8s-upgrade-and-conformance-44o336-crs-0 created
clusterresourceset.addons.cluster.x-k8s.io/k8s-upgrade-and-conformance-44o336-crs-0 created
kubeadmconfig.bootstrap.cluster.x-k8s.io/k8s-upgrade-and-conformance-44o336-mp-0-config created
kubeadmconfig.bootstrap.cluster.x-k8s.io/k8s-upgrade-and-conformance-44o336-mp-0-config-cgroupfs created
cluster.cluster.x-k8s.io/k8s-upgrade-and-conformance-44o336 created
machinepool.cluster.x-k8s.io/k8s-upgrade-and-conformance-44o336-mp-0 created
dockermachinepool.infrastructure.cluster.x-k8s.io/k8s-upgrade-and-conformance-44o336-dmp-0 created
Failed to get logs for Machine k8s-upgrade-and-conformance-44o336-md-0-w9kft-58d5c7654c-b7nvh, Cluster k8s-upgrade-and-conformance-yi0chr/k8s-upgrade-and-conformance-44o336: exit status 2
Failed to get logs for Machine k8s-upgrade-and-conformance-44o336-md-0-w9kft-58d5c7654c-ttvrt, Cluster k8s-upgrade-and-conformance-yi0chr/k8s-upgrade-and-conformance-44o336: exit status 2
Failed to get logs for Machine k8s-upgrade-and-conformance-44o336-rf778-5vzgq, Cluster k8s-upgrade-and-conformance-yi0chr/k8s-upgrade-and-conformance-44o336: exit status 2
Failed to get logs for MachinePool k8s-upgrade-and-conformance-44o336-mp-0, Cluster k8s-upgrade-and-conformance-yi0chr/k8s-upgrade-and-conformance-44o336: exit status 2
> Enter [BeforeEach] When upgrading a workload cluster using ClusterClass and testing K8S conformance [Conformance] [K8s-Upgrade] [ClusterClass] - /home/prow/go/src/sigs.k8s.io/cluster-api/test/e2e/cluster_upgrade.go:84 @ 01/06/23 11:46:10.676
STEP: Creating a namespace for hosting the "k8s-upgrade-and-conformance" test spec - /home/prow/go/src/sigs.k8s.io/cluster-api/test/e2e/common.go:51 @ 01/06/23 11:46:10.676
INFO: Creating namespace k8s-upgrade-and-conformance-yi0chr
INFO: Creating event watcher for namespace "k8s-upgrade-and-conformance-yi0chr"
< Exit [BeforeEach] When upgrading a workload cluster using ClusterClass and testing K8S conformance [Conformance] [K8s-Upgrade] [ClusterClass] - /home/prow/go/src/sigs.k8s.io/cluster-api/test/e2e/cluster_upgrade.go:84 @ 01/06/23 11:46:10.708 (32ms)
> Enter [It] Should create and upgrade a workload cluster and eventually run kubetest - /home/prow/go/src/sigs.k8s.io/cluster-api/test/e2e/cluster_upgrade.go:118 @ 01/06/23 11:46:10.708
STEP: Creating a workload cluster - /home/prow/go/src/sigs.k8s.io/cluster-api/test/e2e/cluster_upgrade.go:119 @ 01/06/23 11:46:10.708
INFO: Creating the workload cluster with name "k8s-upgrade-and-conformance-44o336" using the "upgrades-cgroupfs" template (Kubernetes v1.21.14, 1 control-plane machines, 2 worker machines)
INFO: Getting the cluster template yaml
INFO: clusterctl config cluster k8s-upgrade-and-conformance-44o336 --infrastructure (default) --kubernetes-version v1.21.14 --control-plane-machine-count 1 --worker-machine-count 2 --flavor upgrades-cgroupfs
INFO: Applying the cluster template yaml to the cluster
INFO: Waiting for the cluster infrastructure to be provisioned
STEP: Waiting for cluster to enter the provisioned phase - /home/prow/go/src/sigs.k8s.io/cluster-api/test/framework/cluster_helpers.go:134 @ 01/06/23 11:46:12.915
INFO: Waiting for control plane to be initialized
INFO: Waiting for the first control plane machine managed by k8s-upgrade-and-conformance-yi0chr/k8s-upgrade-and-conformance-44o336-rf778 to be provisioned
STEP: Waiting for one control plane node to exist - /home/prow/go/src/sigs.k8s.io/cluster-api/test/framework/controlplane_helpers.go:133 @ 01/06/23 11:46:22.973
INFO: Waiting for control plane to be ready
INFO: Waiting for control plane k8s-upgrade-and-conformance-yi0chr/k8s-upgrade-and-conformance-44o336-rf778 to be ready (implies underlying nodes to be ready as well)
STEP: Waiting for the control plane to be ready - /home/prow/go/src/sigs.k8s.io/cluster-api/test/framework/controlplane_helpers.go:165 @ 01/06/23 11:46:53.021
STEP: Checking all the control plane machines are in the expected failure domains - /home/prow/go/src/sigs.k8s.io/cluster-api/test/framework/controlplane_helpers.go:196 @ 01/06/23 11:47:13.038
INFO: Waiting for the machine deployments to be provisioned
STEP: Waiting for the workload nodes to exist - /home/prow/go/src/sigs.k8s.io/cluster-api/test/framework/machinedeployment_helpers.go:102 @ 01/06/23 11:47:13.062
STEP: Checking all the machines controlled by k8s-upgrade-and-conformance-44o336-md-0-w9kft are in the "fd4" failure domain - /home/prow/go/src/sigs.k8s.io/cluster-api/test/framework/ginkgoextensions/output.go:35 @ 01/06/23 11:47:43.114
INFO: Waiting for the machine pools to be provisioned
STEP: Waiting for the machine pool workload nodes - /home/prow/go/src/sigs.k8s.io/cluster-api/test/framework/machinepool_helpers.go:79 @ 01/06/23 11:47:43.17
STEP: Upgrading the Cluster topology - /home/prow/go/src/sigs.k8s.io/cluster-api/test/e2e/cluster_upgrade.go:144 @ 01/06/23 11:48:53.216
INFO: Patching the new Kubernetes version to Cluster topology
INFO: Waiting for control-plane machines to have the upgraded Kubernetes version
STEP: Ensuring all control-plane machines have upgraded kubernetes version v1.22.17 - /home/prow/go/src/sigs.k8s.io/cluster-api/test/framework/ginkgoextensions/output.go:35 @ 01/06/23 11:48:53.287
INFO: Waiting for kube-proxy to have the upgraded Kubernetes version
STEP: Ensuring kube-proxy has the correct image - /home/prow/go/src/sigs.k8s.io/cluster-api/test/framework/daemonset_helpers.go:40 @ 01/06/23 11:50:43.719
INFO: Waiting for CoreDNS to have the upgraded image tag
STEP: Ensuring CoreDNS has the correct image - /home/prow/go/src/sigs.k8s.io/cluster-api/test/framework/deployment_helpers.go:276 @ 01/06/23 11:50:53.733
INFO: Waiting for etcd to have the upgraded image tag
INFO: Waiting for Kubernetes versions of machines in MachineDeployment k8s-upgrade-and-conformance-yi0chr/k8s-upgrade-and-conformance-44o336-md-0-w9kft to be upgraded to v1.22.17
INFO: Ensuring all MachineDeployment Machines have upgraded kubernetes version v1.22.17
STEP: Upgrading the machinepool instances - /home/prow/go/src/sigs.k8s.io/cluster-api/test/e2e/cluster_upgrade.go:203 @ 01/06/23 11:52:33.867
INFO: Patching the new Kubernetes version to Machine Pool k8s-upgrade-and-conformance-yi0chr/k8s-upgrade-and-conformance-44o336-mp-0
INFO: Waiting for Kubernetes versions of machines in MachinePool k8s-upgrade-and-conformance-yi0chr/k8s-upgrade-and-conformance-44o336-mp-0 to be upgraded from v1.21.14 to v1.22.17
INFO: Ensuring all MachinePool Instances have upgraded kubernetes version v1.22.17
Automatically polling progress:
When upgrading a workload cluster using ClusterClass and testing K8S conformance [Conformance] [K8s-Upgrade] [ClusterClass] Should create and upgrade a workload cluster and eventually run kubetest (Spec Runtime: 10m0.032s)
/home/prow/go/src/sigs.k8s.io/cluster-api/test/e2e/cluster_upgrade.go:118
In [It] (Node Runtime: 10m0s)
/home/prow/go/src/sigs.k8s.io/cluster-api/test/e2e/cluster_upgrade.go:118
At [By Step] Upgrading the machinepool instances (Step Runtime: 3m36.841s)
/home/prow/go/src/sigs.k8s.io/cluster-api/test/e2e/cluster_upgrade.go:203
Spec Goroutine
goroutine 22345 [select]
k8s.io/apimachinery/pkg/util/wait.WaitForWithContext({0x266ae00, 0xc00019c000}, 0xc00241a168, 0x1196e0a?)
/home/prow/go/pkg/mod/k8s.io/apimachinery@v0.25.0/pkg/util/wait/wait.go:660
k8s.io/apimachinery/pkg/util/wait.poll({0x266ae00, 0xc00019c000}, 0x48?, 0x1195bc5?, 0x50?)
/home/prow/go/pkg/mod/k8s.io/apimachinery@v0.25.0/pkg/util/wait/wait.go:596
k8s.io/apimachinery/pkg/util/wait.PollImmediateWithContext({0x266ae00, 0xc00019c000}, 0x265c578?, 0xc0012a4898?, 0x40fac7?)
/home/prow/go/pkg/mod/k8s.io/apimachinery@v0.25.0/pkg/util/wait/wait.go:528
k8s.io/apimachinery/pkg/util/wait.PollImmediate(0xc001ac6580?, 0x265c578?, 0x38f95b8?)
/home/prow/go/pkg/mod/k8s.io/apimachinery@v0.25.0/pkg/util/wait/wait.go:514
sigs.k8s.io/cluster-api/test/framework.getMachinePoolInstanceVersions({0x266adc8?, 0xc0001130c0}, {{0x7f66886b81e0, 0xc00002afc0}, {0xc000855350, 0x22}, 0xc000746000})
/home/prow/go/src/sigs.k8s.io/cluster-api/test/framework/machinepool_helpers.go:289
sigs.k8s.io/cluster-api/test/framework.WaitForMachinePoolInstancesToBeUpgraded.func1()
/home/prow/go/src/sigs.k8s.io/cluster-api/test/framework/machinepool_helpers.go:250
reflect.Value.call({0x1e86820?, 0xc0008eca80?, 0x60?}, {0x22c4f45, 0x4}, {0x38f95b8, 0x0, 0x0?})
/usr/local/go/src/reflect/value.go:584
reflect.Value.Call({0x1e86820?, 0xc0008eca80?, 0x1?}, {0x38f95b8?, 0x0?, 0xc00008c800?})
/usr/local/go/src/reflect/value.go:368
github.com/onsi/gomega/internal.(*AsyncAssertion).buildActualPoller.func3()
/home/prow/go/pkg/mod/github.com/onsi/gomega@v1.24.2/internal/async_assertion.go:281
github.com/onsi/gomega/internal.(*AsyncAssertion).match(0xc00002b030, {0x265c698?, 0xc00023d610}, 0x1, {0xc000c1da70, 0x3, 0x3})
/home/prow/go/pkg/mod/github.com/onsi/gomega@v1.24.2/internal/async_assertion.go:440
github.com/onsi/gomega/internal.(*AsyncAssertion).Should(0xc00002b030, {0x265c698, 0xc00023d610}, {0xc000c1da70, 0x3, 0x3})
/home/prow/go/pkg/mod/github.com/onsi/gomega@v1.24.2/internal/async_assertion.go:121
sigs.k8s.io/cluster-api/test/framework.WaitForMachinePoolInstancesToBeUpgraded({0x266adc8?, 0xc0001130c0}, {{0x7f66886b81e0, 0xc0004ea380}, {0x7f66886b81e0, 0xc00002afc0}, 0xc001580b60, {0xc0000542be, 0x8}, 0x2, ...}, ...)
/home/prow/go/src/sigs.k8s.io/cluster-api/test/framework/machinepool_helpers.go:268
sigs.k8s.io/cluster-api/test/framework.UpgradeMachinePoolAndWait({0x266adc8?, 0xc0001130c0}, {{0x26794c8, 0xc001c998c0}, 0xc001580b60, {0xc0000542be, 0x8}, {0xc000bb2430, 0x1, 0x1}, ...})
/home/prow/go/src/sigs.k8s.io/cluster-api/test/framework/machinepool_helpers.go:176
> sigs.k8s.io/cluster-api/test/e2e.ClusterUpgradeConformanceSpec.func2()
/home/prow/go/src/sigs.k8s.io/cluster-api/test/e2e/cluster_upgrade.go:204
| if len(clusterResources.MachinePools) > 0 && workerMachineCount > 0 {
| By("Upgrading the machinepool instances")
> framework.UpgradeMachinePoolAndWait(ctx, framework.UpgradeMachinePoolAndWaitInput{
| ClusterProxy: input.BootstrapClusterProxy,
| Cluster: clusterResources.Cluster,
github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func3({0x1397cc0, 0xc002222b30})
/home/prow/go/pkg/mod/github.com/onsi/ginkgo/v2@v2.6.1/internal/node.go:445
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
/home/prow/go/pkg/mod/github.com/onsi/ginkgo/v2@v2.6.1/internal/suite.go:847
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode
/home/prow/go/pkg/mod/github.com/onsi/ginkgo/v2@v2.6.1/internal/suite.go:834
Automatically polling progress:
When upgrading a workload cluster using ClusterClass and testing K8S conformance [Conformance] [K8s-Upgrade] [ClusterClass] Should create and upgrade a workload cluster and eventually run kubetest (Spec Runtime: 11m0.035s)
/home/prow/go/src/sigs.k8s.io/cluster-api/test/e2e/cluster_upgrade.go:118
In [It] (Node Runtime: 11m0.003s)
/home/prow/go/src/sigs.k8s.io/cluster-api/test/e2e/cluster_upgrade.go:118
At [By Step] Upgrading the machinepool instances (Step Runtime: 4m36.844s)
/home/prow/go/src/sigs.k8s.io/cluster-api/test/e2e/cluster_upgrade.go:203
Spec Goroutine
goroutine 22345 [select]
k8s.io/apimachinery/pkg/util/wait.WaitForWithContext({0x266ae00, 0xc00019c000}, 0xc00241a168, 0x1196e0a?)
/home/prow/go/pkg/mod/k8s.io/apimachinery@v0.25.0/pkg/util/wait/wait.go:660
k8s.io/apimachinery/pkg/util/wait.poll({0x266ae00, 0xc00019c000}, 0x48?, 0x1195bc5?, 0x50?)
/home/prow/go/pkg/mod/k8s.io/apimachinery@v0.25.0/pkg/util/wait/wait.go:596
k8s.io/apimachinery/pkg/util/wait.PollImmediateWithContext({0x266ae00, 0xc00019c000}, 0x265c578?, 0xc0012a4898?, 0x40fac7?)
/home/prow/go/pkg/mod/k8s.io/apimachinery@v0.25.0/pkg/util/wait/wait.go:528
k8s.io/apimachinery/pkg/util/wait.PollImmediate(0xc001ac6580?, 0x265c578?, 0x38f95b8?)
/home/prow/go/pkg/mod/k8s.io/apimachinery@v0.25.0/pkg/util/wait/wait.go:514
sigs.k8s.io/cluster-api/test/framework.getMachinePoolInstanceVersions({0x266adc8?, 0xc0001130c0}, {{0x7f66886b81e0, 0xc00002afc0}, {0xc000855350, 0x22}, 0xc000746000})
/home/prow/go/src/sigs.k8s.io/cluster-api/test/framework/machinepool_helpers.go:289
sigs.k8s.io/cluster-api/test/framework.WaitForMachinePoolInstancesToBeUpgraded.func1()
/home/prow/go/src/sigs.k8s.io/cluster-api/test/framework/machinepool_helpers.go:250
reflect.Value.call({0x1e86820?, 0xc0008eca80?, 0x60?}, {0x22c4f45, 0x4}, {0x38f95b8, 0x0, 0x0?})
/usr/local/go/src/reflect/value.go:584
reflect.Value.Call({0x1e86820?, 0xc0008eca80?, 0x1?}, {0x38f95b8?, 0x0?, 0xc00008c800?})
/usr/local/go/src/reflect/value.go:368
github.com/onsi/gomega/internal.(*AsyncAssertion).buildActualPoller.func3()
/home/prow/go/pkg/mod/github.com/onsi/gomega@v1.24.2/internal/async_assertion.go:281
github.com/onsi/gomega/internal.(*AsyncAssertion).match(0xc00002b030, {0x265c698?, 0xc00023d610}, 0x1, {0xc000c1da70, 0x3, 0x3})
/home/prow/go/pkg/mod/github.com/onsi/gomega@v1.24.2/internal/async_assertion.go:440
github.com/onsi/gomega/internal.(*AsyncAssertion).Should(0xc00002b030, {0x265c698, 0xc00023d610}, {0xc000c1da70, 0x3, 0x3})
/home/prow/go/pkg/mod/github.com/onsi/gomega@v1.24.2/internal/async_assertion.go:121
sigs.k8s.io/cluster-api/test/framework.WaitForMachinePoolInstancesToBeUpgraded({0x266adc8?, 0xc0001130c0}, {{0x7f66886b81e0, 0xc0004ea380}, {0x7f66886b81e0, 0xc00002afc0}, 0xc001580b60, {0xc0000542be, 0x8}, 0x2, ...}, ...)
/home/prow/go/src/sigs.k8s.io/cluster-api/test/framework/machinepool_helpers.go:268
sigs.k8s.io/cluster-api/test/framework.UpgradeMachinePoolAndWait({0x266adc8?, 0xc0001130c0}, {{0x26794c8, 0xc001c998c0}, 0xc001580b60, {0xc0000542be, 0x8}, {0xc000bb2430, 0x1, 0x1}, ...})
/home/prow/go/src/sigs.k8s.io/cluster-api/test/framework/machinepool_helpers.go:176
> sigs.k8s.io/cluster-api/test/e2e.ClusterUpgradeConformanceSpec.func2()
/home/prow/go/src/sigs.k8s.io/cluster-api/test/e2e/cluster_upgrade.go:204
| if len(clusterResources.MachinePools) > 0 && workerMachineCount > 0 {
| By("Upgrading the machinepool instances")
> framework.UpgradeMachinePoolAndWait(ctx, framework.UpgradeMachinePoolAndWaitInput{
| ClusterProxy: input.BootstrapClusterProxy,
| Cluster: clusterResources.Cluster,
github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func3({0x1397cc0, 0xc002222b30})
/home/prow/go/pkg/mod/github.com/onsi/ginkgo/v2@v2.6.1/internal/node.go:445
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
/home/prow/go/pkg/mod/github.com/onsi/ginkgo/v2@v2.6.1/internal/suite.go:847
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode
/home/prow/go/pkg/mod/github.com/onsi/ginkgo/v2@v2.6.1/internal/suite.go:834
Automatically polling progress:
When upgrading a workload cluster using ClusterClass and testing K8S conformance [Conformance] [K8s-Upgrade] [ClusterClass] Should create and upgrade a workload cluster and eventually run kubetest (Spec Runtime: 12m0.038s)
/home/prow/go/src/sigs.k8s.io/cluster-api/test/e2e/cluster_upgrade.go:118
In [It] (Node Runtime: 12m0.006s)
/home/prow/go/src/sigs.k8s.io/cluster-api/test/e2e/cluster_upgrade.go:118
At [By Step] Upgrading the machinepool instances (Step Runtime: 5m36.847s)
/home/prow/go/src/sigs.k8s.io/cluster-api/test/e2e/cluster_upgrade.go:203
Spec Goroutine
goroutine 22345 [select]
k8s.io/apimachinery/pkg/util/wait.WaitForWithContext({0x266ae00, 0xc00019c000}, 0xc00241a168, 0x1196e0a?)
/home/prow/go/pkg/mod/k8s.io/apimachinery@v0.25.0/pkg/util/wait/wait.go:660
k8s.io/apimachinery/pkg/util/wait.poll({0x266ae00, 0xc00019c000}, 0x48?, 0x1195bc5?, 0x50?)
/home/prow/go/pkg/mod/k8s.io/apimachinery@v0.25.0/pkg/util/wait/wait.go:596
k8s.io/apimachinery/pkg/util/wait.PollImmediateWithContext({0x266ae00, 0xc00019c000}, 0x265c578?, 0xc0012a4898?, 0x40fac7?)
/home/prow/go/pkg/mod/k8s.io/apimachinery@v0.25.0/pkg/util/wait/wait.go:528
k8s.io/apimachinery/pkg/util/wait.PollImmediate(0xc001ac6580?, 0x265c578?, 0x38f95b8?)
/home/prow/go/pkg/mod/k8s.io/apimachinery@v0.25.0/pkg/util/wait/wait.go:514
sigs.k8s.io/cluster-api/test/framework.getMachinePoolInstanceVersions({0x266adc8?, 0xc0001130c0}, {{0x7f66886b81e0, 0xc00002afc0}, {0xc000855350, 0x22}, 0xc000746000})
/home/prow/go/src/sigs.k8s.io/cluster-api/test/framework/machinepool_helpers.go:289
sigs.k8s.io/cluster-api/test/framework.WaitForMachinePoolInstancesToBeUpgraded.func1()
/home/prow/go/src/sigs.k8s.io/cluster-api/test/framework/machinepool_helpers.go:250
reflect.Value.call({0x1e86820?, 0xc0008eca80?, 0x60?}, {0x22c4f45, 0x4}, {0x38f95b8, 0x0, 0x0?})
/usr/local/go/src/reflect/value.go:584
reflect.Value.Call({0x1e86820?, 0xc0008eca80?, 0x1?}, {0x38f95b8?, 0x0?, 0xc00008c800?})
/usr/local/go/src/reflect/value.go:368
github.com/onsi/gomega/internal.(*AsyncAssertion).buildActualPoller.func3()
/home/prow/go/pkg/mod/github.com/onsi/gomega@v1.24.2/internal/async_assertion.go:281
github.com/onsi/gomega/internal.(*AsyncAssertion).match(0xc00002b030, {0x265c698?, 0xc00023d610}, 0x1, {0xc000c1da70, 0x3, 0x3})
/home/prow/go/pkg/mod/github.com/onsi/gomega@v1.24.2/internal/async_assertion.go:440
github.com/onsi/gomega/internal.(*AsyncAssertion).Should(0xc00002b030, {0x265c698, 0xc00023d610}, {0xc000c1da70, 0x3, 0x3})
/home/prow/go/pkg/mod/github.com/onsi/gomega@v1.24.2/internal/async_assertion.go:121
sigs.k8s.io/cluster-api/test/framework.WaitForMachinePoolInstancesToBeUpgraded({0x266adc8?, 0xc0001130c0}, {{0x7f66886b81e0, 0xc0004ea380}, {0x7f66886b81e0, 0xc00002afc0}, 0xc001580b60, {0xc0000542be, 0x8}, 0x2, ...}, ...)
/home/prow/go/src/sigs.k8s.io/cluster-api/test/framework/machinepool_helpers.go:268
sigs.k8s.io/cluster-api/test/framework.UpgradeMachinePoolAndWait({0x266adc8?, 0xc0001130c0}, {{0x26794c8, 0xc001c998c0}, 0xc001580b60, {0xc0000542be, 0x8}, {0xc000bb2430, 0x1, 0x1}, ...})
/home/prow/go/src/sigs.k8s.io/cluster-api/test/framework/machinepool_helpers.go:176
> sigs.k8s.io/cluster-api/test/e2e.ClusterUpgradeConformanceSpec.func2()
/home/prow/go/src/sigs.k8s.io/cluster-api/test/e2e/cluster_upgrade.go:204
| if len(clusterResources.MachinePools) > 0 && workerMachineCount > 0 {
| By("Upgrading the machinepool instances")
> framework.UpgradeMachinePoolAndWait(ctx, framework.UpgradeMachinePoolAndWaitInput{
| ClusterProxy: input.BootstrapClusterProxy,
| Cluster: clusterResources.Cluster,
github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func3({0x1397cc0, 0xc002222b30})
/home/prow/go/pkg/mod/github.com/onsi/ginkgo/v2@v2.6.1/internal/node.go:445
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
/home/prow/go/pkg/mod/github.com/onsi/ginkgo/v2@v2.6.1/internal/suite.go:847
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode
/home/prow/go/pkg/mod/github.com/onsi/ginkgo/v2@v2.6.1/internal/suite.go:834
[FAILED] Timed out after 390.132s.
Timed out waiting for all MachinePool k8s-upgrade-and-conformance-yi0chr/k8s-upgrade-and-conformance-44o336-mp-0 instances to be upgraded to Kubernetes version v1.22.17
Error: function returned error: old version instances remain
<*fmt.wrapError | 0xc000470360>: {
msg: "function returned error: old version instances remain",
err: <*errors.fundamental | 0xc0009d3308>{
msg: "old version instances remain",
stack: [0x180d55a, 0x4dad65, 0x4da25c, 0x88447a, 0x885382, 0x882ced, 0x180d364, 0x180ba85, 0x1cb420a, 0x86241b, 0x8754b8, 0x4704e1],
},
}
In [It] at: /home/prow/go/src/sigs.k8s.io/cluster-api/test/framework/machinepool_helpers.go:268 @ 01/06/23 11:59:04.084
< Exit [It] Should create and upgrade a workload cluster and eventually run kubetest - /home/prow/go/src/sigs.k8s.io/cluster-api/test/e2e/cluster_upgrade.go:118 @ 01/06/23 11:59:04.084 (12m53.376s)
> Enter [AfterEach] When upgrading a workload cluster using ClusterClass and testing K8S conformance [Conformance] [K8s-Upgrade] [ClusterClass] - /home/prow/go/src/sigs.k8s.io/cluster-api/test/e2e/cluster_upgrade.go:242 @ 01/06/23 11:59:04.084
STEP: Dumping logs from the "k8s-upgrade-and-conformance-44o336" workload cluster - /home/prow/go/src/sigs.k8s.io/cluster-api/test/e2e/common.go:51 @ 01/06/23 11:59:04.084
STEP: Dumping all the Cluster API resources in the "k8s-upgrade-and-conformance-yi0chr" namespace - /home/prow/go/src/sigs.k8s.io/cluster-api/test/e2e/common.go:51 @ 01/06/23 11:59:07.271
STEP: Deleting cluster k8s-upgrade-and-conformance-yi0chr/k8s-upgrade-and-conformance-44o336 - /home/prow/go/src/sigs.k8s.io/cluster-api/test/e2e/common.go:51 @ 01/06/23 11:59:07.563
STEP: Deleting cluster k8s-upgrade-and-conformance-44o336 - /home/prow/go/src/sigs.k8s.io/cluster-api/test/framework/ginkgoextensions/output.go:35 @ 01/06/23 11:59:07.587
INFO: Waiting for the Cluster k8s-upgrade-and-conformance-yi0chr/k8s-upgrade-and-conformance-44o336 to be deleted
STEP: Waiting for cluster k8s-upgrade-and-conformance-44o336 to be deleted - /home/prow/go/src/sigs.k8s.io/cluster-api/test/framework/ginkgoextensions/output.go:35 @ 01/06/23 11:59:07.605
STEP: Deleting namespace used for hosting the "k8s-upgrade-and-conformance" test spec - /home/prow/go/src/sigs.k8s.io/cluster-api/test/e2e/common.go:51 @ 01/06/23 11:59:17.613
INFO: Deleting namespace k8s-upgrade-and-conformance-yi0chr
< Exit [AfterEach] When upgrading a workload cluster using ClusterClass and testing K8S conformance [Conformance] [K8s-Upgrade] [ClusterClass] - /home/prow/go/src/sigs.k8s.io/cluster-api/test/e2e/cluster_upgrade.go:242 @ 01/06/23 11:59:17.634 (13.551s)
from junit.e2e_suite.1.xml
Filter through log files | View test history on testgrid
capi-e2e [SynchronizedAfterSuite]
capi-e2e [SynchronizedAfterSuite]
capi-e2e [SynchronizedAfterSuite]
capi-e2e [SynchronizedBeforeSuite]
capi-e2e [SynchronizedBeforeSuite]
capi-e2e [SynchronizedBeforeSuite]
capi-e2e [It] When following the Cluster API quick-start [PR-Blocking] Should create a workload cluster
capi-e2e [It] When following the Cluster API quick-start with ClusterClass [PR-Informing] [ClusterClass] Should create a workload cluster
capi-e2e [It] When following the Cluster API quick-start with IPv6 [IPv6] [PR-Informing] Should create a workload cluster
capi-e2e [It] When following the Cluster API quick-start with Ignition Should create a workload cluster
capi-e2e [It] When testing Cluster API working on self-hosted clusters Should pivot the bootstrap cluster to a self-hosted cluster
capi-e2e [It] When testing Cluster API working on self-hosted clusters using ClusterClass [ClusterClass] Should pivot the bootstrap cluster to a self-hosted cluster
capi-e2e [It] When testing Cluster API working on self-hosted clusters using ClusterClass with a HA control plane [ClusterClass] Should pivot the bootstrap cluster to a self-hosted cluster
capi-e2e [It] When testing Cluster API working on single-node self-hosted clusters using ClusterClass [ClusterClass] Should pivot the bootstrap cluster to a self-hosted cluster
capi-e2e [It] When testing ClusterClass changes [ClusterClass] Should successfully rollout the managed topology upon changes to the ClusterClass
capi-e2e [It] When testing K8S conformance [Conformance] Should create a workload cluster and run kubetest
capi-e2e [It] When testing KCP adoption Should adopt up-to-date control plane Machines without modification
capi-e2e [It] When testing MachineDeployment rolling upgrades Should successfully upgrade Machines upon changes in relevant MachineDeployment fields
capi-e2e [It] When testing MachineDeployment scale out/in Should successfully scale a MachineDeployment up and down upon changes to the MachineDeployment replica count
capi-e2e [It] When testing MachinePools Should successfully create a cluster with machine pool machines
capi-e2e [It] When testing clusterctl upgrades (v0.3=>current) Should create a management cluster and then upgrade all the providers
capi-e2e [It] When testing clusterctl upgrades (v0.4=>current) Should create a management cluster and then upgrade all the providers
capi-e2e [It] When testing clusterctl upgrades (v1.2=>current) Should create a management cluster and then upgrade all the providers
capi-e2e [It] When testing clusterctl upgrades (v1.3=>current) Should create a management cluster and then upgrade all the providers
capi-e2e [It] When testing clusterctl upgrades using ClusterClass (v1.2=>current) [ClusterClass] Should create a management cluster and then upgrade all the providers
capi-e2e [It] When testing clusterctl upgrades using ClusterClass (v1.3=>current) [ClusterClass] Should create a management cluster and then upgrade all the providers
capi-e2e [It] When testing node drain timeout A node should be forcefully removed if it cannot be drained in time
capi-e2e [It] When testing unhealthy machines remediation Should successfully trigger KCP remediation
capi-e2e [It] When testing unhealthy machines remediation Should successfully trigger machine deployment remediation
capi-e2e [It] When upgrading a workload cluster using ClusterClass [ClusterClass] Should create and upgrade a workload cluster and eventually run kubetest
capi-e2e [It] When upgrading a workload cluster using ClusterClass with RuntimeSDK [PR-Informing] [ClusterClass] Should create, upgrade and delete a workload cluster
capi-e2e [It] When upgrading a workload cluster using ClusterClass with a HA control plane [ClusterClass] Should create and upgrade a workload cluster and eventually run kubetest
capi-e2e [It] When upgrading a workload cluster using ClusterClass with a HA control plane using scale-in rollout [ClusterClass] Should create and upgrade a workload cluster and eventually run kubetest
... skipping 612 lines ... Updating kustomize pull policy file for manager resources sed -i'' -e 's@imagePullPolicy: .*@imagePullPolicy: '"IfNotPresent"'@' ./test/extension/config/default/manager_pull_policy.yaml make[2]: Leaving directory '/home/prow/go/src/sigs.k8s.io/cluster-api' make[1]: Leaving directory '/home/prow/go/src/sigs.k8s.io/cluster-api' + KUBERNETES_VERSION_UPGRADE_TO="stable-1.22" resolved to "v1.22.17" + Pulling kindest/node:v1.22.17 Error response from daemon: manifest for kindest/node:v1.22.17 not found: manifest unknown: manifest unknown + image for Kuberentes v1.22.17 is not available in docker hub, trying local build KUBE_ROOT /home/prow/go/src/k8s.io/kubernetes + Checkout branch for Kubernetes v1.22.17 + checkout tag v1.22.17 Switched to a new branch 'v1.22.17-branch' + Setting version for Kubernetes build to v1.22.17 ... skipping 191 lines ... /home/prow/go/src/sigs.k8s.io/cluster-api/hack/tools/bin/kustomize-v4.5.2 build test/e2e/data/infrastructure-docker/v1beta1/main/cluster-template-ignition --load-restrictor LoadRestrictionsNone > test/e2e/data/infrastructure-docker/v1beta1/main/cluster-template-ignition.yaml mkdir -p test/e2e/data/test-extension /home/prow/go/src/sigs.k8s.io/cluster-api/hack/tools/bin/kustomize-v4.5.2 build test/extension/config/default > test/e2e/data/test-extension/deployment.yaml /home/prow/go/src/sigs.k8s.io/cluster-api/hack/tools/bin/ginkgo-v2.6.1 -v --trace -poll-progress-after=10m \ -poll-progress-interval=1m --tags=e2e --focus="\[K8s-Upgrade\]" \ --nodes=3 --timeout=2h --no-color=true \ --output-dir="/logs/artifacts" --junit-report="junit.e2e_suite.1.xml" --fail-fast /home/prow/go/src/sigs.k8s.io/cluster-api/test/e2e -- \ -e2e.artifacts-folder="/logs/artifacts" \ -e2e.config="/home/prow/go/src/sigs.k8s.io/cluster-api/test/e2e/config/docker.yaml" \ -e2e.skip-resource-cleanup=false -e2e.use-existing-cluster=false go: downloading github.com/blang/semver v3.5.1+incompatible go: downloading k8s.io/api v0.25.0 go: downloading k8s.io/apimachinery v0.25.0 ... skipping 386 lines ... /home/prow/go/pkg/mod/github.com/onsi/ginkgo/v2@v2.6.1/internal/node.go:445 github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3() /home/prow/go/pkg/mod/github.com/onsi/ginkgo/v2@v2.6.1/internal/suite.go:847 github.com/onsi/ginkgo/v2/internal.(*Suite).runNode /home/prow/go/pkg/mod/github.com/onsi/ginkgo/v2@v2.6.1/internal/suite.go:834 ------------------------------ • [FAILED] [786.959 seconds] When upgrading a workload cluster using ClusterClass and testing K8S conformance [Conformance] [K8s-Upgrade] [ClusterClass] [It] Should create and upgrade a workload cluster and eventually run kubetest /home/prow/go/src/sigs.k8s.io/cluster-api/test/e2e/cluster_upgrade.go:118 Captured StdOut/StdErr Output >> clusterclass.cluster.x-k8s.io/quick-start created dockerclustertemplate.infrastructure.cluster.x-k8s.io/quick-start-cluster created ... skipping 6 lines ... kubeadmconfig.bootstrap.cluster.x-k8s.io/k8s-upgrade-and-conformance-44o336-mp-0-config created kubeadmconfig.bootstrap.cluster.x-k8s.io/k8s-upgrade-and-conformance-44o336-mp-0-config-cgroupfs created cluster.cluster.x-k8s.io/k8s-upgrade-and-conformance-44o336 created machinepool.cluster.x-k8s.io/k8s-upgrade-and-conformance-44o336-mp-0 created dockermachinepool.infrastructure.cluster.x-k8s.io/k8s-upgrade-and-conformance-44o336-dmp-0 created Failed to get logs for Machine k8s-upgrade-and-conformance-44o336-md-0-w9kft-58d5c7654c-b7nvh, Cluster k8s-upgrade-and-conformance-yi0chr/k8s-upgrade-and-conformance-44o336: exit status 2 Failed to get logs for Machine k8s-upgrade-and-conformance-44o336-md-0-w9kft-58d5c7654c-ttvrt, Cluster k8s-upgrade-and-conformance-yi0chr/k8s-upgrade-and-conformance-44o336: exit status 2 Failed to get logs for Machine k8s-upgrade-and-conformance-44o336-rf778-5vzgq, Cluster k8s-upgrade-and-conformance-yi0chr/k8s-upgrade-and-conformance-44o336: exit status 2 Failed to get logs for MachinePool k8s-upgrade-and-conformance-44o336-mp-0, Cluster k8s-upgrade-and-conformance-yi0chr/k8s-upgrade-and-conformance-44o336: exit status 2 << Captured StdOut/StdErr Output Timeline >> STEP: Creating a namespace for hosting the "k8s-upgrade-and-conformance" test spec @ 01/06/23 11:46:10.676 INFO: Creating namespace k8s-upgrade-and-conformance-yi0chr INFO: Creating event watcher for namespace "k8s-upgrade-and-conformance-yi0chr" ... skipping 175 lines ... github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func3({0x1397cc0, 0xc002222b30}) /home/prow/go/pkg/mod/github.com/onsi/ginkgo/v2@v2.6.1/internal/node.go:445 github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3() /home/prow/go/pkg/mod/github.com/onsi/ginkgo/v2@v2.6.1/internal/suite.go:847 github.com/onsi/ginkgo/v2/internal.(*Suite).runNode /home/prow/go/pkg/mod/github.com/onsi/ginkgo/v2@v2.6.1/internal/suite.go:834 [FAILED] in [It] - /home/prow/go/src/sigs.k8s.io/cluster-api/test/framework/machinepool_helpers.go:268 @ 01/06/23 11:59:04.084 STEP: Dumping logs from the "k8s-upgrade-and-conformance-44o336" workload cluster @ 01/06/23 11:59:04.084 STEP: Dumping all the Cluster API resources in the "k8s-upgrade-and-conformance-yi0chr" namespace @ 01/06/23 11:59:07.271 STEP: Deleting cluster k8s-upgrade-and-conformance-yi0chr/k8s-upgrade-and-conformance-44o336 @ 01/06/23 11:59:07.563 STEP: Deleting cluster k8s-upgrade-and-conformance-44o336 @ 01/06/23 11:59:07.587 INFO: Waiting for the Cluster k8s-upgrade-and-conformance-yi0chr/k8s-upgrade-and-conformance-44o336 to be deleted STEP: Waiting for cluster k8s-upgrade-and-conformance-44o336 to be deleted @ 01/06/23 11:59:07.605 STEP: Deleting namespace used for hosting the "k8s-upgrade-and-conformance" test spec @ 01/06/23 11:59:17.613 INFO: Deleting namespace k8s-upgrade-and-conformance-yi0chr << Timeline [FAILED] Timed out after 390.132s. Timed out waiting for all MachinePool k8s-upgrade-and-conformance-yi0chr/k8s-upgrade-and-conformance-44o336-mp-0 instances to be upgraded to Kubernetes version v1.22.17 Error: function returned error: old version instances remain <*fmt.wrapError | 0xc000470360>: { msg: "function returned error: old version instances remain", err: <*errors.fundamental | 0xc0009d3308>{ msg: "old version instances remain", stack: [0x180d55a, 0x4dad65, 0x4da25c, 0x88447a, 0x885382, 0x882ced, 0x180d364, 0x180ba85, 0x1cb420a, 0x86241b, 0x8754b8, 0x4704e1], }, } In [It] at: /home/prow/go/src/sigs.k8s.io/cluster-api/test/framework/machinepool_helpers.go:268 @ 01/06/23 11:59:04.084 ... skipping 23 lines ... [ReportAfterSuite] PASSED [0.006 seconds] [ReportAfterSuite] Autogenerated ReportAfterSuite for --junit-report autogenerated by Ginkgo ------------------------------ Summarizing 2 Failures: [FAIL] When upgrading a workload cluster using ClusterClass and testing K8S conformance [Conformance] [K8s-Upgrade] [ClusterClass] [It] Should create and upgrade a workload cluster and eventually run kubetest /home/prow/go/src/sigs.k8s.io/cluster-api/test/framework/machinepool_helpers.go:268 [INTERRUPTED] [SynchronizedAfterSuite] /home/prow/go/src/sigs.k8s.io/cluster-api/test/e2e/e2e_suite_test.go:169 Ran 1 of 28 Specs in 884.008 seconds FAIL! - Interrupted by Other Ginkgo Process -- 0 Passed | 1 Failed | 0 Pending | 27 Skipped Ginkgo ran 1 suite in 15m51.008084036s Test Suite Failed make: *** [Makefile:790: test-e2e] Error 1 WARNING: No swap limit support + EXIT_VALUE=2 + set +o xtrace Cleaning up after docker in docker. ================================================================================ Cleaning up after docker ... skipping 6 lines ...