e2e-kops-gce-leader-migration #1620690577093824512
Recent runs || View in Spyglass
| Result | FAILURE |
| Tests | 1 failed / 1 succeeded |
| Started | |
| Elapsed | 21m0s |
| Revision | master |
exit status 1
from junit_runner.xml
Filter through log files | View test history on testgrid
kubetest2 Down
... skipping 156 lines ... ................Updated [https://www.googleapis.com/compute/v1/projects/k8s-boskos-gce-project-12]. ..done. WARNING: No host aliases were added to your SSH configs because you do not have any running instances. Try running this command again after running some instances. I0201 07:50:30.830096 5789 up.go:44] Cleaning up any leaked resources from previous cluster I0201 07:50:30.830248 5789 dumplogs.go:45] /home/prow/go/src/k8s.io/kops/_rundir/d72d5fa8-a204-11ed-9efc-5e7820dba16b/kops toolbox dump --name ha-migration.k8s.local --dir /logs/artifacts --private-key /tmp/kops-ssh563587820/key --ssh-user prow I0201 07:50:30.830291 5789 local.go:42] ⚙️ /home/prow/go/src/k8s.io/kops/_rundir/d72d5fa8-a204-11ed-9efc-5e7820dba16b/kops toolbox dump --name ha-migration.k8s.local --dir /logs/artifacts --private-key /tmp/kops-ssh563587820/key --ssh-user prow Error: Cluster.kops.k8s.io "ha-migration.k8s.local" not found W0201 07:50:31.133030 5789 down.go:34] Dumping cluster logs at the start of Down() failed: exit status 1 I0201 07:50:31.133083 5789 down.go:48] /home/prow/go/src/k8s.io/kops/_rundir/d72d5fa8-a204-11ed-9efc-5e7820dba16b/kops delete cluster --name ha-migration.k8s.local --yes I0201 07:50:31.133094 5789 local.go:42] ⚙️ /home/prow/go/src/k8s.io/kops/_rundir/d72d5fa8-a204-11ed-9efc-5e7820dba16b/kops delete cluster --name ha-migration.k8s.local --yes Error: error reading cluster configuration: Cluster.kops.k8s.io "ha-migration.k8s.local" not found I0201 07:50:31.370058 5789 gcs.go:51] gsutil ls -b -p k8s-boskos-gce-project-12 gs://k8s-boskos-gce-project-12-state-d7 I0201 07:50:31.370113 5789 local.go:42] ⚙️ gsutil ls -b -p k8s-boskos-gce-project-12 gs://k8s-boskos-gce-project-12-state-d7 I0201 07:50:33.341300 5789 gcs.go:70] gsutil mb -p k8s-boskos-gce-project-12 gs://k8s-boskos-gce-project-12-state-d7 I0201 07:50:33.341341 5789 local.go:42] ⚙️ gsutil mb -p k8s-boskos-gce-project-12 gs://k8s-boskos-gce-project-12-state-d7 Creating gs://k8s-boskos-gce-project-12-state-d7/... I0201 07:50:35.962019 5789 http.go:37] curl http://metadata.google.internal/computeMetadata/v1/instance/network-interfaces/0/access-configs/0/external-ip 2023/02/01 07:50:35 failed to get external ip from metadata service: http://metadata.google.internal/computeMetadata/v1/instance/network-interfaces/0/access-configs/0/external-ip returned 404 I0201 07:50:35.976216 5789 http.go:37] curl https://ip.jsb.workers.dev I0201 07:50:36.068732 5789 up.go:167] /home/prow/go/src/k8s.io/kops/_rundir/d72d5fa8-a204-11ed-9efc-5e7820dba16b/kops create cluster --name ha-migration.k8s.local --cloud gce --kubernetes-version v1.23.16 --ssh-public-key /tmp/kops-ssh563587820/key.pub --set cluster.spec.nodePortAccess=0.0.0.0/0 --channel=alpha --node-count=1 --master-count=3 --zones=us-central1-a,us-central1-b,us-central1-c --master-zones=us-central1-a,us-central1-b,us-central1-c --gce-service-account=default --admin-access 35.194.23.235/32 --master-volume-size 48 --node-volume-size 48 --master-size e2-standard-2 --project k8s-boskos-gce-project-12 I0201 07:50:36.068768 5789 local.go:42] ⚙️ /home/prow/go/src/k8s.io/kops/_rundir/d72d5fa8-a204-11ed-9efc-5e7820dba16b/kops create cluster --name ha-migration.k8s.local --cloud gce --kubernetes-version v1.23.16 --ssh-public-key /tmp/kops-ssh563587820/key.pub --set cluster.spec.nodePortAccess=0.0.0.0/0 --channel=alpha --node-count=1 --master-count=3 --zones=us-central1-a,us-central1-b,us-central1-c --master-zones=us-central1-a,us-central1-b,us-central1-c --gce-service-account=default --admin-access 35.194.23.235/32 --master-volume-size 48 --node-volume-size 48 --master-size e2-standard-2 --project k8s-boskos-gce-project-12 I0201 07:50:36.117538 6107 create_cluster.go:882] Using SSH public key: /tmp/kops-ssh563587820/key.pub I0201 07:50:36.325513 6107 new_cluster.go:573] VMs will be configured to use specified Service Account: default I0201 07:50:36.325626 6107 new_cluster.go:1339] Cloud Provider ID: "gce" ... skipping 527 lines ... I0201 07:50:48.485431 6126 keypair.go:226] Issuing new certificate: "etcd-peers-ca-events" I0201 07:50:48.491310 6126 keypair.go:226] Issuing new certificate: "etcd-clients-ca" W0201 07:50:48.680047 6126 vfs_keystorereader.go:143] CA private key was not found I0201 07:50:48.822646 6126 keypair.go:226] Issuing new certificate: "service-account" I0201 07:50:48.883237 6126 keypair.go:226] Issuing new certificate: "kubernetes-ca" I0201 07:50:59.250761 6126 executor.go:111] Tasks: 50 done / 91 total; 23 can run W0201 07:51:09.038045 6126 executor.go:139] error running task "ForwardingRule/api-ha-migration-k8s-local" (9m50s remaining to succeed): error creating ForwardingRule "api-ha-migration-k8s-local": googleapi: Error 400: The resource 'projects/k8s-boskos-gce-project-12/regions/us-central1/targetPools/api-ha-migration-k8s-local' is not ready, resourceNotReady I0201 07:51:09.038119 6126 executor.go:111] Tasks: 72 done / 91 total; 13 can run I0201 07:51:15.816095 6126 executor.go:111] Tasks: 85 done / 91 total; 6 can run I0201 07:51:29.912803 6126 executor.go:111] Tasks: 91 done / 91 total; 0 can run I0201 07:51:29.979569 6126 update_cluster.go:323] Exporting kubeconfig for cluster kOps has set your kubectl context to ha-migration.k8s.local ... skipping 7 lines ... * read about installing addons at: https://kops.sigs.k8s.io/addons. I0201 07:51:40.253247 5789 up.go:251] /home/prow/go/src/k8s.io/kops/_rundir/d72d5fa8-a204-11ed-9efc-5e7820dba16b/kops validate cluster --name ha-migration.k8s.local --count 10 --wait 15m0s I0201 07:51:40.253318 5789 local.go:42] ⚙️ /home/prow/go/src/k8s.io/kops/_rundir/d72d5fa8-a204-11ed-9efc-5e7820dba16b/kops validate cluster --name ha-migration.k8s.local --count 10 --wait 15m0s Validating cluster ha-migration.k8s.local W0201 07:52:10.716686 6144 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://34.172.225.17/api/v1/nodes": dial tcp 34.172.225.17:443: i/o timeout W0201 07:52:23.763485 6144 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://34.172.225.17/api/v1/nodes": dial tcp 34.172.225.17:443: connect: connection refused W0201 07:52:33.769328 6144 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://34.172.225.17/api/v1/nodes": dial tcp 34.172.225.17:443: connect: connection refused W0201 07:52:43.773450 6144 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://34.172.225.17/api/v1/nodes": dial tcp 34.172.225.17:443: connect: connection refused W0201 07:52:53.778744 6144 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://34.172.225.17/api/v1/nodes": dial tcp 34.172.225.17:443: connect: connection refused W0201 07:53:03.784845 6144 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://34.172.225.17/api/v1/nodes": dial tcp 34.172.225.17:443: connect: connection refused W0201 07:53:13.790665 6144 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://34.172.225.17/api/v1/nodes": dial tcp 34.172.225.17:443: connect: connection refused W0201 07:53:23.795857 6144 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://34.172.225.17/api/v1/nodes": dial tcp 34.172.225.17:443: connect: connection refused W0201 07:53:33.800182 6144 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://34.172.225.17/api/v1/nodes": dial tcp 34.172.225.17:443: connect: connection refused W0201 07:53:43.806120 6144 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://34.172.225.17/api/v1/nodes": dial tcp 34.172.225.17:443: connect: connection refused W0201 07:53:53.812194 6144 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://34.172.225.17/api/v1/nodes": dial tcp 34.172.225.17:443: connect: connection refused W0201 07:54:03.815271 6144 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://34.172.225.17/api/v1/nodes": dial tcp 34.172.225.17:443: connect: connection refused W0201 07:54:23.818804 6144 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://34.172.225.17/api/v1/nodes": net/http: TLS handshake timeout W0201 07:54:43.824025 6144 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://34.172.225.17/api/v1/nodes": net/http: TLS handshake timeout W0201 07:55:03.828500 6144 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://34.172.225.17/api/v1/nodes": net/http: TLS handshake timeout W0201 07:55:13.831158 6144 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://34.172.225.17/api/v1/nodes": dial tcp 34.172.225.17:443: connect: connection refused I0201 07:55:24.073577 5789 boskos.go:86] Sending heartbeat to Boskos W0201 07:55:33.839430 6144 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://34.172.225.17/api/v1/nodes": net/http: TLS handshake timeout W0201 07:55:43.842393 6144 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://34.172.225.17/api/v1/nodes": dial tcp 34.172.225.17:443: connect: connection refused W0201 07:55:53.846138 6144 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://34.172.225.17/api/v1/nodes": dial tcp 34.172.225.17:443: connect: connection refused W0201 07:56:13.851944 6144 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://34.172.225.17/api/v1/nodes": net/http: TLS handshake timeout W0201 07:56:23.856186 6144 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://34.172.225.17/api/v1/nodes": dial tcp 34.172.225.17:443: connect: connection refused W0201 07:56:33.859395 6144 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://34.172.225.17/api/v1/nodes": dial tcp 34.172.225.17:443: connect: connection refused W0201 07:56:43.865880 6144 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://34.172.225.17/api/v1/nodes": dial tcp 34.172.225.17:443: connect: connection refused W0201 07:56:53.872158 6144 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://34.172.225.17/api/v1/nodes": dial tcp 34.172.225.17:443: connect: connection refused W0201 07:57:03.875579 6144 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://34.172.225.17/api/v1/nodes": dial tcp 34.172.225.17:443: connect: connection refused I0201 07:57:14.273016 6144 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-c ControlPlane e2-standard-2 1 1 us-central1 ... skipping 9 lines ... KIND NAME MESSAGE Machine https://www.googleapis.com/compute/v1/projects/k8s-boskos-gce-project-12/zones/us-central1-a/instances/control-plane-us-central1-a-fbtr machine "https://www.googleapis.com/compute/v1/projects/k8s-boskos-gce-project-12/zones/us-central1-a/instances/control-plane-us-central1-a-fbtr" has not yet joined cluster Machine https://www.googleapis.com/compute/v1/projects/k8s-boskos-gce-project-12/zones/us-central1-a/instances/nodes-us-central1-a-vx63 machine "https://www.googleapis.com/compute/v1/projects/k8s-boskos-gce-project-12/zones/us-central1-a/instances/nodes-us-central1-a-vx63" has not yet joined cluster Machine https://www.googleapis.com/compute/v1/projects/k8s-boskos-gce-project-12/zones/us-central1-b/instances/control-plane-us-central1-b-b08t machine "https://www.googleapis.com/compute/v1/projects/k8s-boskos-gce-project-12/zones/us-central1-b/instances/control-plane-us-central1-b-b08t" has not yet joined cluster Node control-plane-us-central1-c-5j0j node "control-plane-us-central1-c-5j0j" of role "control-plane" is not ready Validation Failed W0201 07:57:15.156338 6144 validate_cluster.go:232] (will retry): cluster not yet healthy I0201 07:57:25.597941 6144 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 12 lines ... KIND NAME MESSAGE Machine https://www.googleapis.com/compute/v1/projects/k8s-boskos-gce-project-12/zones/us-central1-a/instances/nodes-us-central1-a-vx63 machine "https://www.googleapis.com/compute/v1/projects/k8s-boskos-gce-project-12/zones/us-central1-a/instances/nodes-us-central1-a-vx63" has not yet joined cluster Node control-plane-us-central1-a-fbtr node "control-plane-us-central1-a-fbtr" of role "control-plane" is not ready Node control-plane-us-central1-b-b08t node "control-plane-us-central1-b-b08t" of role "control-plane" is not ready Node control-plane-us-central1-c-5j0j node "control-plane-us-central1-c-5j0j" of role "control-plane" is not ready Validation Failed W0201 07:57:26.482844 6144 validate_cluster.go:232] (will retry): cluster not yet healthy I0201 07:57:36.800870 6144 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 13 lines ... Machine https://www.googleapis.com/compute/v1/projects/k8s-boskos-gce-project-12/zones/us-central1-a/instances/nodes-us-central1-a-vx63 machine "https://www.googleapis.com/compute/v1/projects/k8s-boskos-gce-project-12/zones/us-central1-a/instances/nodes-us-central1-a-vx63" has not yet joined cluster Node control-plane-us-central1-a-fbtr node "control-plane-us-central1-a-fbtr" of role "control-plane" is not ready Node control-plane-us-central1-b-b08t node "control-plane-us-central1-b-b08t" of role "control-plane" is not ready Node control-plane-us-central1-c-5j0j node "control-plane-us-central1-c-5j0j" of role "control-plane" is not ready Pod kube-system/kube-controller-manager-control-plane-us-central1-b-b08t system-cluster-critical pod "kube-controller-manager-control-plane-us-central1-b-b08t" is pending Validation Failed W0201 07:57:37.702583 6144 validate_cluster.go:232] (will retry): cluster not yet healthy I0201 07:57:48.058977 6144 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 23 lines ... Pod kube-system/coredns-autoscaler-557ccb4c66-4j724 system-cluster-critical pod "coredns-autoscaler-557ccb4c66-4j724" is pending Pod kube-system/dns-controller-6f69f47785-tb4ft system-cluster-critical pod "dns-controller-6f69f47785-tb4ft" is pending Pod kube-system/kops-controller-5pt2f system-cluster-critical pod "kops-controller-5pt2f" is pending Pod kube-system/kops-controller-7gc24 system-cluster-critical pod "kops-controller-7gc24" is pending Pod kube-system/kops-controller-x2mr5 system-cluster-critical pod "kops-controller-x2mr5" is pending Validation Failed W0201 07:57:49.067978 6144 validate_cluster.go:232] (will retry): cluster not yet healthy I0201 07:57:59.466029 6144 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 23 lines ... Pod kube-system/coredns-autoscaler-557ccb4c66-4j724 system-cluster-critical pod "coredns-autoscaler-557ccb4c66-4j724" is pending Pod kube-system/dns-controller-6f69f47785-tb4ft system-cluster-critical pod "dns-controller-6f69f47785-tb4ft" is pending Pod kube-system/kops-controller-5pt2f system-cluster-critical pod "kops-controller-5pt2f" is pending Pod kube-system/kops-controller-7gc24 system-cluster-critical pod "kops-controller-7gc24" is pending Pod kube-system/kops-controller-x2mr5 system-cluster-critical pod "kops-controller-x2mr5" is pending Validation Failed W0201 07:58:00.362286 6144 validate_cluster.go:232] (will retry): cluster not yet healthy I0201 07:58:10.682197 6144 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 20 lines ... Pod kube-system/cilium-operator-b4b4c5b48-knc5d system-cluster-critical pod "cilium-operator-b4b4c5b48-knc5d" is pending Pod kube-system/cilium-q4x64 system-node-critical pod "cilium-q4x64" is pending Pod kube-system/coredns-5d479c4f64-x5lf8 system-cluster-critical pod "coredns-5d479c4f64-x5lf8" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-4j724 system-cluster-critical pod "coredns-autoscaler-557ccb4c66-4j724" is pending Pod kube-system/dns-controller-6f69f47785-tb4ft system-cluster-critical pod "dns-controller-6f69f47785-tb4ft" is pending Validation Failed W0201 07:58:11.495993 6144 validate_cluster.go:232] (will retry): cluster not yet healthy I0201 07:58:22.017988 6144 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 20 lines ... Pod kube-system/cilium-q4x64 system-node-critical pod "cilium-q4x64" is pending Pod kube-system/coredns-5d479c4f64-x5lf8 system-cluster-critical pod "coredns-5d479c4f64-x5lf8" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-4j724 system-cluster-critical pod "coredns-autoscaler-557ccb4c66-4j724" is pending Pod kube-system/dns-controller-6f69f47785-tb4ft system-cluster-critical pod "dns-controller-6f69f47785-tb4ft" is pending Pod kube-system/kube-scheduler-control-plane-us-central1-c-5j0j system-cluster-critical pod "kube-scheduler-control-plane-us-central1-c-5j0j" is pending Validation Failed W0201 07:58:22.854195 6144 validate_cluster.go:232] (will retry): cluster not yet healthy I0201 07:58:33.204136 6144 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 20 lines ... Pod kube-system/coredns-5d479c4f64-x5lf8 system-cluster-critical pod "coredns-5d479c4f64-x5lf8" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-4j724 system-cluster-critical pod "coredns-autoscaler-557ccb4c66-4j724" is pending Pod kube-system/dns-controller-6f69f47785-tb4ft system-cluster-critical pod "dns-controller-6f69f47785-tb4ft" is pending Pod kube-system/etcd-manager-events-control-plane-us-central1-b-b08t system-cluster-critical pod "etcd-manager-events-control-plane-us-central1-b-b08t" is pending Pod kube-system/kube-apiserver-control-plane-us-central1-c-5j0j system-cluster-critical pod "kube-apiserver-control-plane-us-central1-c-5j0j" is pending Validation Failed W0201 07:58:34.157934 6144 validate_cluster.go:232] (will retry): cluster not yet healthy I0201 07:58:44.572324 6144 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 19 lines ... Pod kube-system/cilium-q4x64 system-node-critical pod "cilium-q4x64" is not ready (cilium-agent) Pod kube-system/coredns-5d479c4f64-x5lf8 system-cluster-critical pod "coredns-5d479c4f64-x5lf8" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-4j724 system-cluster-critical pod "coredns-autoscaler-557ccb4c66-4j724" is pending Pod kube-system/dns-controller-6f69f47785-tb4ft system-cluster-critical pod "dns-controller-6f69f47785-tb4ft" is pending Pod kube-system/etcd-manager-main-control-plane-us-central1-a-fbtr system-cluster-critical pod "etcd-manager-main-control-plane-us-central1-a-fbtr" is pending Validation Failed W0201 07:58:45.480070 6144 validate_cluster.go:232] (will retry): cluster not yet healthy I0201 07:58:55.779651 6144 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-b08t node "control-plane-us-central1-b-b08t" of role "control-plane" is not ready Node control-plane-us-central1-c-5j0j node "control-plane-us-central1-c-5j0j" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-x5lf8 system-cluster-critical pod "coredns-5d479c4f64-x5lf8" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-4j724 system-cluster-critical pod "coredns-autoscaler-557ccb4c66-4j724" is pending Pod kube-system/dns-controller-6f69f47785-tb4ft system-cluster-critical pod "dns-controller-6f69f47785-tb4ft" is pending Validation Failed W0201 07:58:56.667111 6144 validate_cluster.go:232] (will retry): cluster not yet healthy I0201 07:59:07.029017 6144 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-b08t node "control-plane-us-central1-b-b08t" of role "control-plane" is not ready Node control-plane-us-central1-c-5j0j node "control-plane-us-central1-c-5j0j" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-x5lf8 system-cluster-critical pod "coredns-5d479c4f64-x5lf8" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-4j724 system-cluster-critical pod "coredns-autoscaler-557ccb4c66-4j724" is pending Pod kube-system/dns-controller-6f69f47785-tb4ft system-cluster-critical pod "dns-controller-6f69f47785-tb4ft" is pending Validation Failed W0201 07:59:07.897113 6144 validate_cluster.go:232] (will retry): cluster not yet healthy I0201 07:59:18.214039 6144 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-b08t node "control-plane-us-central1-b-b08t" of role "control-plane" is not ready Node control-plane-us-central1-c-5j0j node "control-plane-us-central1-c-5j0j" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-x5lf8 system-cluster-critical pod "coredns-5d479c4f64-x5lf8" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-4j724 system-cluster-critical pod "coredns-autoscaler-557ccb4c66-4j724" is pending Pod kube-system/dns-controller-6f69f47785-tb4ft system-cluster-critical pod "dns-controller-6f69f47785-tb4ft" is pending Validation Failed W0201 07:59:19.090080 6144 validate_cluster.go:232] (will retry): cluster not yet healthy I0201 07:59:29.426070 6144 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-b08t node "control-plane-us-central1-b-b08t" of role "control-plane" is not ready Node control-plane-us-central1-c-5j0j node "control-plane-us-central1-c-5j0j" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-x5lf8 system-cluster-critical pod "coredns-5d479c4f64-x5lf8" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-4j724 system-cluster-critical pod "coredns-autoscaler-557ccb4c66-4j724" is pending Pod kube-system/dns-controller-6f69f47785-tb4ft system-cluster-critical pod "dns-controller-6f69f47785-tb4ft" is pending Validation Failed W0201 07:59:30.276631 6144 validate_cluster.go:232] (will retry): cluster not yet healthy I0201 07:59:40.594968 6144 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-b08t node "control-plane-us-central1-b-b08t" of role "control-plane" is not ready Node control-plane-us-central1-c-5j0j node "control-plane-us-central1-c-5j0j" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-x5lf8 system-cluster-critical pod "coredns-5d479c4f64-x5lf8" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-4j724 system-cluster-critical pod "coredns-autoscaler-557ccb4c66-4j724" is pending Pod kube-system/dns-controller-6f69f47785-tb4ft system-cluster-critical pod "dns-controller-6f69f47785-tb4ft" is pending Validation Failed W0201 07:59:41.392529 6144 validate_cluster.go:232] (will retry): cluster not yet healthy I0201 07:59:51.855610 6144 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-b08t node "control-plane-us-central1-b-b08t" of role "control-plane" is not ready Node control-plane-us-central1-c-5j0j node "control-plane-us-central1-c-5j0j" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-x5lf8 system-cluster-critical pod "coredns-5d479c4f64-x5lf8" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-4j724 system-cluster-critical pod "coredns-autoscaler-557ccb4c66-4j724" is pending Pod kube-system/dns-controller-6f69f47785-tb4ft system-cluster-critical pod "dns-controller-6f69f47785-tb4ft" is pending Validation Failed W0201 07:59:52.711909 6144 validate_cluster.go:232] (will retry): cluster not yet healthy I0201 08:00:03.082756 6144 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-b08t node "control-plane-us-central1-b-b08t" of role "control-plane" is not ready Node control-plane-us-central1-c-5j0j node "control-plane-us-central1-c-5j0j" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-x5lf8 system-cluster-critical pod "coredns-5d479c4f64-x5lf8" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-4j724 system-cluster-critical pod "coredns-autoscaler-557ccb4c66-4j724" is pending Pod kube-system/dns-controller-6f69f47785-tb4ft system-cluster-critical pod "dns-controller-6f69f47785-tb4ft" is pending Validation Failed W0201 08:00:03.949092 6144 validate_cluster.go:232] (will retry): cluster not yet healthy I0201 08:00:14.293587 6144 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-b08t node "control-plane-us-central1-b-b08t" of role "control-plane" is not ready Node control-plane-us-central1-c-5j0j node "control-plane-us-central1-c-5j0j" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-x5lf8 system-cluster-critical pod "coredns-5d479c4f64-x5lf8" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-4j724 system-cluster-critical pod "coredns-autoscaler-557ccb4c66-4j724" is pending Pod kube-system/dns-controller-6f69f47785-tb4ft system-cluster-critical pod "dns-controller-6f69f47785-tb4ft" is pending Validation Failed W0201 08:00:15.201412 6144 validate_cluster.go:232] (will retry): cluster not yet healthy I0201 08:00:24.083748 5789 boskos.go:86] Sending heartbeat to Boskos I0201 08:00:25.631074 6144 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 ... skipping 16 lines ... Node control-plane-us-central1-b-b08t node "control-plane-us-central1-b-b08t" of role "control-plane" is not ready Node control-plane-us-central1-c-5j0j node "control-plane-us-central1-c-5j0j" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-x5lf8 system-cluster-critical pod "coredns-5d479c4f64-x5lf8" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-4j724 system-cluster-critical pod "coredns-autoscaler-557ccb4c66-4j724" is pending Pod kube-system/dns-controller-6f69f47785-tb4ft system-cluster-critical pod "dns-controller-6f69f47785-tb4ft" is pending Validation Failed W0201 08:00:26.499110 6144 validate_cluster.go:232] (will retry): cluster not yet healthy I0201 08:00:36.886039 6144 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-b08t node "control-plane-us-central1-b-b08t" of role "control-plane" is not ready Node control-plane-us-central1-c-5j0j node "control-plane-us-central1-c-5j0j" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-x5lf8 system-cluster-critical pod "coredns-5d479c4f64-x5lf8" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-4j724 system-cluster-critical pod "coredns-autoscaler-557ccb4c66-4j724" is pending Pod kube-system/dns-controller-6f69f47785-tb4ft system-cluster-critical pod "dns-controller-6f69f47785-tb4ft" is pending Validation Failed W0201 08:00:37.711694 6144 validate_cluster.go:232] (will retry): cluster not yet healthy I0201 08:00:48.055080 6144 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-b08t node "control-plane-us-central1-b-b08t" of role "control-plane" is not ready Node control-plane-us-central1-c-5j0j node "control-plane-us-central1-c-5j0j" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-x5lf8 system-cluster-critical pod "coredns-5d479c4f64-x5lf8" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-4j724 system-cluster-critical pod "coredns-autoscaler-557ccb4c66-4j724" is pending Pod kube-system/dns-controller-6f69f47785-tb4ft system-cluster-critical pod "dns-controller-6f69f47785-tb4ft" is pending Validation Failed W0201 08:00:48.904325 6144 validate_cluster.go:232] (will retry): cluster not yet healthy I0201 08:00:59.348579 6144 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-b08t node "control-plane-us-central1-b-b08t" of role "control-plane" is not ready Node control-plane-us-central1-c-5j0j node "control-plane-us-central1-c-5j0j" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-x5lf8 system-cluster-critical pod "coredns-5d479c4f64-x5lf8" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-4j724 system-cluster-critical pod "coredns-autoscaler-557ccb4c66-4j724" is pending Pod kube-system/dns-controller-6f69f47785-tb4ft system-cluster-critical pod "dns-controller-6f69f47785-tb4ft" is pending Validation Failed W0201 08:02:14.853683 6144 validate_cluster.go:232] (will retry): cluster not yet healthy I0201 08:02:25.290150 6144 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-b08t node "control-plane-us-central1-b-b08t" of role "control-plane" is not ready Node control-plane-us-central1-c-5j0j node "control-plane-us-central1-c-5j0j" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-x5lf8 system-cluster-critical pod "coredns-5d479c4f64-x5lf8" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-4j724 system-cluster-critical pod "coredns-autoscaler-557ccb4c66-4j724" is pending Pod kube-system/dns-controller-6f69f47785-tb4ft system-cluster-critical pod "dns-controller-6f69f47785-tb4ft" is pending Validation Failed W0201 08:02:26.174669 6144 validate_cluster.go:232] (will retry): cluster not yet healthy I0201 08:02:36.535591 6144 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-b08t node "control-plane-us-central1-b-b08t" of role "control-plane" is not ready Node control-plane-us-central1-c-5j0j node "control-plane-us-central1-c-5j0j" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-x5lf8 system-cluster-critical pod "coredns-5d479c4f64-x5lf8" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-4j724 system-cluster-critical pod "coredns-autoscaler-557ccb4c66-4j724" is pending Pod kube-system/dns-controller-6f69f47785-tb4ft system-cluster-critical pod "dns-controller-6f69f47785-tb4ft" is pending Validation Failed W0201 08:02:37.406753 6144 validate_cluster.go:232] (will retry): cluster not yet healthy I0201 08:02:47.782431 6144 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-b08t node "control-plane-us-central1-b-b08t" of role "control-plane" is not ready Node control-plane-us-central1-c-5j0j node "control-plane-us-central1-c-5j0j" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-x5lf8 system-cluster-critical pod "coredns-5d479c4f64-x5lf8" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-4j724 system-cluster-critical pod "coredns-autoscaler-557ccb4c66-4j724" is pending Pod kube-system/dns-controller-6f69f47785-tb4ft system-cluster-critical pod "dns-controller-6f69f47785-tb4ft" is pending Validation Failed W0201 08:02:48.683266 6144 validate_cluster.go:232] (will retry): cluster not yet healthy I0201 08:02:59.046998 6144 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-b08t node "control-plane-us-central1-b-b08t" of role "control-plane" is not ready Node control-plane-us-central1-c-5j0j node "control-plane-us-central1-c-5j0j" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-x5lf8 system-cluster-critical pod "coredns-5d479c4f64-x5lf8" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-4j724 system-cluster-critical pod "coredns-autoscaler-557ccb4c66-4j724" is pending Pod kube-system/dns-controller-6f69f47785-tb4ft system-cluster-critical pod "dns-controller-6f69f47785-tb4ft" is pending Validation Failed W0201 08:02:59.890436 6144 validate_cluster.go:232] (will retry): cluster not yet healthy I0201 08:03:10.333686 6144 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-b08t node "control-plane-us-central1-b-b08t" of role "control-plane" is not ready Node control-plane-us-central1-c-5j0j node "control-plane-us-central1-c-5j0j" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-x5lf8 system-cluster-critical pod "coredns-5d479c4f64-x5lf8" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-4j724 system-cluster-critical pod "coredns-autoscaler-557ccb4c66-4j724" is pending Pod kube-system/dns-controller-6f69f47785-tb4ft system-cluster-critical pod "dns-controller-6f69f47785-tb4ft" is pending Validation Failed W0201 08:03:11.184500 6144 validate_cluster.go:232] (will retry): cluster not yet healthy I0201 08:03:21.576065 6144 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-b08t node "control-plane-us-central1-b-b08t" of role "control-plane" is not ready Node control-plane-us-central1-c-5j0j node "control-plane-us-central1-c-5j0j" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-x5lf8 system-cluster-critical pod "coredns-5d479c4f64-x5lf8" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-4j724 system-cluster-critical pod "coredns-autoscaler-557ccb4c66-4j724" is pending Pod kube-system/dns-controller-6f69f47785-tb4ft system-cluster-critical pod "dns-controller-6f69f47785-tb4ft" is pending Validation Failed W0201 08:03:22.402749 6144 validate_cluster.go:232] (will retry): cluster not yet healthy I0201 08:03:32.737404 6144 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-b08t node "control-plane-us-central1-b-b08t" of role "control-plane" is not ready Node control-plane-us-central1-c-5j0j node "control-plane-us-central1-c-5j0j" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-x5lf8 system-cluster-critical pod "coredns-5d479c4f64-x5lf8" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-4j724 system-cluster-critical pod "coredns-autoscaler-557ccb4c66-4j724" is pending Pod kube-system/dns-controller-6f69f47785-tb4ft system-cluster-critical pod "dns-controller-6f69f47785-tb4ft" is pending Validation Failed W0201 08:03:33.603287 6144 validate_cluster.go:232] (will retry): cluster not yet healthy I0201 08:03:44.112035 6144 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-b08t node "control-plane-us-central1-b-b08t" of role "control-plane" is not ready Node control-plane-us-central1-c-5j0j node "control-plane-us-central1-c-5j0j" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-x5lf8 system-cluster-critical pod "coredns-5d479c4f64-x5lf8" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-4j724 system-cluster-critical pod "coredns-autoscaler-557ccb4c66-4j724" is pending Pod kube-system/dns-controller-6f69f47785-tb4ft system-cluster-critical pod "dns-controller-6f69f47785-tb4ft" is pending Validation Failed W0201 08:03:44.916394 6144 validate_cluster.go:232] (will retry): cluster not yet healthy I0201 08:03:55.309251 6144 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-b08t node "control-plane-us-central1-b-b08t" of role "control-plane" is not ready Node control-plane-us-central1-c-5j0j node "control-plane-us-central1-c-5j0j" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-x5lf8 system-cluster-critical pod "coredns-5d479c4f64-x5lf8" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-4j724 system-cluster-critical pod "coredns-autoscaler-557ccb4c66-4j724" is pending Pod kube-system/dns-controller-6f69f47785-tb4ft system-cluster-critical pod "dns-controller-6f69f47785-tb4ft" is pending Validation Failed W0201 08:03:56.171304 6144 validate_cluster.go:232] (will retry): cluster not yet healthy I0201 08:04:06.570842 6144 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-b08t node "control-plane-us-central1-b-b08t" of role "control-plane" is not ready Node control-plane-us-central1-c-5j0j node "control-plane-us-central1-c-5j0j" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-x5lf8 system-cluster-critical pod "coredns-5d479c4f64-x5lf8" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-4j724 system-cluster-critical pod "coredns-autoscaler-557ccb4c66-4j724" is pending Pod kube-system/dns-controller-6f69f47785-tb4ft system-cluster-critical pod "dns-controller-6f69f47785-tb4ft" is pending Validation Failed W0201 08:04:07.478103 6144 validate_cluster.go:232] (will retry): cluster not yet healthy I0201 08:04:17.905245 6144 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-b08t node "control-plane-us-central1-b-b08t" of role "control-plane" is not ready Node control-plane-us-central1-c-5j0j node "control-plane-us-central1-c-5j0j" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-x5lf8 system-cluster-critical pod "coredns-5d479c4f64-x5lf8" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-4j724 system-cluster-critical pod "coredns-autoscaler-557ccb4c66-4j724" is pending Pod kube-system/dns-controller-6f69f47785-tb4ft system-cluster-critical pod "dns-controller-6f69f47785-tb4ft" is pending Validation Failed W0201 08:04:18.687006 6144 validate_cluster.go:232] (will retry): cluster not yet healthy I0201 08:04:29.011371 6144 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] I0201 08:05:24.093610 5789 boskos.go:86] Sending heartbeat to Boskos INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 ... skipping 16 lines ... Node control-plane-us-central1-b-b08t node "control-plane-us-central1-b-b08t" of role "control-plane" is not ready Node control-plane-us-central1-c-5j0j node "control-plane-us-central1-c-5j0j" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-x5lf8 system-cluster-critical pod "coredns-5d479c4f64-x5lf8" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-4j724 system-cluster-critical pod "coredns-autoscaler-557ccb4c66-4j724" is pending Pod kube-system/dns-controller-6f69f47785-tb4ft system-cluster-critical pod "dns-controller-6f69f47785-tb4ft" is pending Validation Failed W0201 08:06:07.524817 6144 validate_cluster.go:232] (will retry): cluster not yet healthy I0201 08:06:17.924354 6144 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-b08t node "control-plane-us-central1-b-b08t" of role "control-plane" is not ready Node control-plane-us-central1-c-5j0j node "control-plane-us-central1-c-5j0j" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-x5lf8 system-cluster-critical pod "coredns-5d479c4f64-x5lf8" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-4j724 system-cluster-critical pod "coredns-autoscaler-557ccb4c66-4j724" is pending Pod kube-system/dns-controller-6f69f47785-tb4ft system-cluster-critical pod "dns-controller-6f69f47785-tb4ft" is pending Validation Failed W0201 08:06:18.758218 6144 validate_cluster.go:232] (will retry): cluster not yet healthy I0201 08:06:29.122044 6144 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-b08t node "control-plane-us-central1-b-b08t" of role "control-plane" is not ready Node control-plane-us-central1-c-5j0j node "control-plane-us-central1-c-5j0j" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-x5lf8 system-cluster-critical pod "coredns-5d479c4f64-x5lf8" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-4j724 system-cluster-critical pod "coredns-autoscaler-557ccb4c66-4j724" is pending Pod kube-system/dns-controller-6f69f47785-tb4ft system-cluster-critical pod "dns-controller-6f69f47785-tb4ft" is pending Validation Failed W0201 08:06:29.988415 6144 validate_cluster.go:232] (will retry): cluster not yet healthy I0201 08:06:40.348476 6144 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-b08t node "control-plane-us-central1-b-b08t" of role "control-plane" is not ready Node control-plane-us-central1-c-5j0j node "control-plane-us-central1-c-5j0j" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-x5lf8 system-cluster-critical pod "coredns-5d479c4f64-x5lf8" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-4j724 system-cluster-critical pod "coredns-autoscaler-557ccb4c66-4j724" is pending Pod kube-system/dns-controller-6f69f47785-tb4ft system-cluster-critical pod "dns-controller-6f69f47785-tb4ft" is pending Validation Failed W0201 08:06:41.220576 6144 validate_cluster.go:232] (will retry): cluster not yet healthy Error: validation failed: wait time exceeded during validation I0201 08:06:51.230062 5789 dumplogs.go:45] /home/prow/go/src/k8s.io/kops/_rundir/d72d5fa8-a204-11ed-9efc-5e7820dba16b/kops toolbox dump --name ha-migration.k8s.local --dir /logs/artifacts --private-key /tmp/kops-ssh563587820/key --ssh-user prow I0201 08:06:51.230111 5789 local.go:42] ⚙️ /home/prow/go/src/k8s.io/kops/_rundir/d72d5fa8-a204-11ed-9efc-5e7820dba16b/kops toolbox dump --name ha-migration.k8s.local --dir /logs/artifacts --private-key /tmp/kops-ssh563587820/key --ssh-user prow I0201 08:06:51.661196 6154 gce.go:99] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] 2023/02/01 08:06:56 Dumping node control-plane-us-central1-a-fbtr 2023/02/01 08:07:00 error dumping node control-plane-us-central1-a-fbtr: error executing command "if command -v kubectl &> /dev/null; then kubectl logs -n kube-system --all-containers -l \"k8s-app=external-dns\"; fi": Process exited with status 1 2023/02/01 08:07:00 error dumping node control-plane-us-central1-a-fbtr: error executing command "if command -v kubectl &> /dev/null; then kubectl logs -n kube-system --all-containers -l \"k8s-app=dns-controller\"; fi": Process exited with status 1 2023/02/01 08:07:00 Dumping node control-plane-us-central1-b-b08t 2023/02/01 08:07:05 error dumping node control-plane-us-central1-b-b08t: error executing command "if command -v kubectl &> /dev/null; then kubectl logs -n kube-system --all-containers -l \"k8s-app=external-dns\"; fi": Process exited with status 1 2023/02/01 08:07:05 error dumping node control-plane-us-central1-b-b08t: error executing command "if command -v kubectl &> /dev/null; then kubectl logs -n kube-system --all-containers -l \"k8s-app=dns-controller\"; fi": Process exited with status 1 2023/02/01 08:07:05 Dumping node control-plane-us-central1-c-5j0j 2023/02/01 08:07:09 error dumping node control-plane-us-central1-c-5j0j: error executing command "if command -v kubectl &> /dev/null; then kubectl logs -n kube-system --all-containers -l \"k8s-app=external-dns\"; fi": Process exited with status 1 2023/02/01 08:07:09 error dumping node control-plane-us-central1-c-5j0j: error executing command "if command -v kubectl &> /dev/null; then kubectl logs -n kube-system --all-containers -l \"k8s-app=dns-controller\"; fi": Process exited with status 1 2023/02/01 08:07:09 dumping node not registered in kubernetes: 34.68.78.116 2023/02/01 08:07:09 Dumping node 34.68.78.116 I0201 08:07:12.037257 5789 dumplogs.go:79] /home/prow/go/src/k8s.io/kops/_rundir/d72d5fa8-a204-11ed-9efc-5e7820dba16b/kops get cluster --name ha-migration.k8s.local -o yaml I0201 08:07:12.037300 5789 local.go:42] ⚙️ /home/prow/go/src/k8s.io/kops/_rundir/d72d5fa8-a204-11ed-9efc-5e7820dba16b/kops get cluster --name ha-migration.k8s.local -o yaml I0201 08:07:12.178178 5789 dumplogs.go:79] /home/prow/go/src/k8s.io/kops/_rundir/d72d5fa8-a204-11ed-9efc-5e7820dba16b/kops get instancegroups --name ha-migration.k8s.local -o yaml I0201 08:07:12.178328 5789 local.go:42] ⚙️ /home/prow/go/src/k8s.io/kops/_rundir/d72d5fa8-a204-11ed-9efc-5e7820dba16b/kops get instancegroups --name ha-migration.k8s.local -o yaml ... skipping 167 lines ... Deleted cluster: "ha-migration.k8s.local" I0201 08:10:19.707179 5789 gcs.go:88] gsutil -u k8s-boskos-gce-project-12 rm -r gs://k8s-boskos-gce-project-12-state-d7 I0201 08:10:19.707256 5789 local.go:42] ⚙️ gsutil -u k8s-boskos-gce-project-12 rm -r gs://k8s-boskos-gce-project-12-state-d7 Removing gs://k8s-boskos-gce-project-12-state-d7/... I0201 08:10:22.238213 5789 down.go:62] releasing boskos project I0201 08:10:22.245843 5789 boskos.go:83] Boskos heartbeat func received signal to close Error: exit status 1 + EXIT_VALUE=1 + set +o xtrace