e2e-kops-gce-leader-migration #1619693937658695680
Recent runs || View in Spyglass
| Result | FAILURE |
| Tests | 1 failed / 1 succeeded |
| Started | |
| Elapsed | 20m18s |
| Revision | master |
exit status 1
from junit_runner.xml
Filter through log files | View test history on testgrid
kubetest2 Down
... skipping 156 lines ... ..................................Updated [https://www.googleapis.com/compute/v1/projects/gce-cvm-upg-1-3-lat-ctl-skew]. ..done. WARNING: No host aliases were added to your SSH configs because you do not have any running instances. Try running this command again after running some instances. I0129 13:50:29.358817 5750 up.go:44] Cleaning up any leaked resources from previous cluster I0129 13:50:29.358956 5750 dumplogs.go:45] /home/prow/go/src/k8s.io/kops/_rundir/9831df3a-9fdb-11ed-9a0c-f25a29267a91/kops toolbox dump --name ha-migration.k8s.local --dir /logs/artifacts --private-key /tmp/kops-ssh32606979/key --ssh-user prow I0129 13:50:29.358971 5750 local.go:42] ⚙️ /home/prow/go/src/k8s.io/kops/_rundir/9831df3a-9fdb-11ed-9a0c-f25a29267a91/kops toolbox dump --name ha-migration.k8s.local --dir /logs/artifacts --private-key /tmp/kops-ssh32606979/key --ssh-user prow Error: Cluster.kops.k8s.io "ha-migration.k8s.local" not found W0129 13:50:29.648602 5750 down.go:34] Dumping cluster logs at the start of Down() failed: exit status 1 I0129 13:50:29.648717 5750 down.go:48] /home/prow/go/src/k8s.io/kops/_rundir/9831df3a-9fdb-11ed-9a0c-f25a29267a91/kops delete cluster --name ha-migration.k8s.local --yes I0129 13:50:29.648730 5750 local.go:42] ⚙️ /home/prow/go/src/k8s.io/kops/_rundir/9831df3a-9fdb-11ed-9a0c-f25a29267a91/kops delete cluster --name ha-migration.k8s.local --yes Error: error reading cluster configuration: Cluster.kops.k8s.io "ha-migration.k8s.local" not found I0129 13:50:29.886411 5750 gcs.go:51] gsutil ls -b -p gce-cvm-upg-1-3-lat-ctl-skew gs://gce-cvm-upg-1-3-lat-ctl-skew-state-98 I0129 13:50:29.886454 5750 local.go:42] ⚙️ gsutil ls -b -p gce-cvm-upg-1-3-lat-ctl-skew gs://gce-cvm-upg-1-3-lat-ctl-skew-state-98 I0129 13:50:32.122577 5750 gcs.go:70] gsutil mb -p gce-cvm-upg-1-3-lat-ctl-skew gs://gce-cvm-upg-1-3-lat-ctl-skew-state-98 I0129 13:50:32.122616 5750 local.go:42] ⚙️ gsutil mb -p gce-cvm-upg-1-3-lat-ctl-skew gs://gce-cvm-upg-1-3-lat-ctl-skew-state-98 Creating gs://gce-cvm-upg-1-3-lat-ctl-skew-state-98/... I0129 13:50:34.965803 5750 http.go:37] curl http://metadata.google.internal/computeMetadata/v1/instance/network-interfaces/0/access-configs/0/external-ip 2023/01/29 13:50:34 failed to get external ip from metadata service: http://metadata.google.internal/computeMetadata/v1/instance/network-interfaces/0/access-configs/0/external-ip returned 404 I0129 13:50:34.981180 5750 http.go:37] curl https://ip.jsb.workers.dev I0129 13:50:35.075524 5750 up.go:167] /home/prow/go/src/k8s.io/kops/_rundir/9831df3a-9fdb-11ed-9a0c-f25a29267a91/kops create cluster --name ha-migration.k8s.local --cloud gce --kubernetes-version v1.23.16 --ssh-public-key /tmp/kops-ssh32606979/key.pub --set cluster.spec.nodePortAccess=0.0.0.0/0 --channel=alpha --node-count=1 --master-count=3 --zones=us-central1-a,us-central1-b,us-central1-c --master-zones=us-central1-a,us-central1-b,us-central1-c --gce-service-account=default --admin-access 104.198.245.181/32 --master-volume-size 48 --node-volume-size 48 --master-size e2-standard-2 --project gce-cvm-upg-1-3-lat-ctl-skew I0129 13:50:35.075571 5750 local.go:42] ⚙️ /home/prow/go/src/k8s.io/kops/_rundir/9831df3a-9fdb-11ed-9a0c-f25a29267a91/kops create cluster --name ha-migration.k8s.local --cloud gce --kubernetes-version v1.23.16 --ssh-public-key /tmp/kops-ssh32606979/key.pub --set cluster.spec.nodePortAccess=0.0.0.0/0 --channel=alpha --node-count=1 --master-count=3 --zones=us-central1-a,us-central1-b,us-central1-c --master-zones=us-central1-a,us-central1-b,us-central1-c --gce-service-account=default --admin-access 104.198.245.181/32 --master-volume-size 48 --node-volume-size 48 --master-size e2-standard-2 --project gce-cvm-upg-1-3-lat-ctl-skew I0129 13:50:35.139060 6069 create_cluster.go:882] Using SSH public key: /tmp/kops-ssh32606979/key.pub I0129 13:50:35.436537 6069 new_cluster.go:573] VMs will be configured to use specified Service Account: default I0129 13:50:35.436676 6069 new_cluster.go:1338] Cloud Provider ID: "gce" ... skipping 527 lines ... I0129 13:50:49.011619 6089 keypair.go:226] Issuing new certificate: "etcd-peers-ca-main" I0129 13:50:49.063780 6089 keypair.go:226] Issuing new certificate: "etcd-clients-ca" W0129 13:50:49.179364 6089 vfs_keystorereader.go:143] CA private key was not found I0129 13:50:49.383378 6089 keypair.go:226] Issuing new certificate: "kubernetes-ca" I0129 13:50:49.383553 6089 keypair.go:226] Issuing new certificate: "service-account" I0129 13:51:04.689579 6089 executor.go:111] Tasks: 50 done / 91 total; 23 can run W0129 13:51:14.564653 6089 executor.go:139] error running task "ForwardingRule/api-ha-migration-k8s-local" (9m50s remaining to succeed): error creating ForwardingRule "api-ha-migration-k8s-local": googleapi: Error 400: The resource 'projects/gce-cvm-upg-1-3-lat-ctl-skew/regions/us-central1/targetPools/api-ha-migration-k8s-local' is not ready, resourceNotReady I0129 13:51:14.564736 6089 executor.go:111] Tasks: 72 done / 91 total; 13 can run I0129 13:51:21.474323 6089 executor.go:111] Tasks: 85 done / 91 total; 6 can run I0129 13:51:35.497130 6089 executor.go:111] Tasks: 91 done / 91 total; 0 can run I0129 13:51:35.569770 6089 update_cluster.go:323] Exporting kubeconfig for cluster kOps has set your kubectl context to ha-migration.k8s.local ... skipping 7 lines ... * read about installing addons at: https://kops.sigs.k8s.io/addons. I0129 13:51:45.917599 5750 up.go:251] /home/prow/go/src/k8s.io/kops/_rundir/9831df3a-9fdb-11ed-9a0c-f25a29267a91/kops validate cluster --name ha-migration.k8s.local --count 10 --wait 15m0s I0129 13:51:45.917686 5750 local.go:42] ⚙️ /home/prow/go/src/k8s.io/kops/_rundir/9831df3a-9fdb-11ed-9a0c-f25a29267a91/kops validate cluster --name ha-migration.k8s.local --count 10 --wait 15m0s Validating cluster ha-migration.k8s.local W0129 13:52:16.520849 6109 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://35.202.2.44/api/v1/nodes": dial tcp 35.202.2.44:443: i/o timeout W0129 13:52:41.892453 6109 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://35.202.2.44/api/v1/nodes": dial tcp 35.202.2.44:443: connect: connection refused W0129 13:52:51.898582 6109 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://35.202.2.44/api/v1/nodes": dial tcp 35.202.2.44:443: connect: connection refused W0129 13:53:01.907637 6109 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://35.202.2.44/api/v1/nodes": dial tcp 35.202.2.44:443: connect: connection refused W0129 13:53:11.916279 6109 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://35.202.2.44/api/v1/nodes": dial tcp 35.202.2.44:443: connect: connection refused W0129 13:53:21.921585 6109 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://35.202.2.44/api/v1/nodes": dial tcp 35.202.2.44:443: connect: connection refused W0129 13:53:31.924748 6109 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://35.202.2.44/api/v1/nodes": dial tcp 35.202.2.44:443: connect: connection refused W0129 13:53:41.934188 6109 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://35.202.2.44/api/v1/nodes": dial tcp 35.202.2.44:443: connect: connection refused W0129 13:53:51.940170 6109 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://35.202.2.44/api/v1/nodes": dial tcp 35.202.2.44:443: connect: connection refused W0129 13:54:01.948991 6109 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://35.202.2.44/api/v1/nodes": dial tcp 35.202.2.44:443: connect: connection refused W0129 13:54:21.963803 6109 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://35.202.2.44/api/v1/nodes": net/http: TLS handshake timeout W0129 13:54:59.793180 6109 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://35.202.2.44/api/v1/nodes": net/http: TLS handshake timeout - error from a previous attempt: read tcp 10.60.167.72:59448->35.202.2.44:443: read: connection reset by peer I0129 13:55:18.454207 5750 boskos.go:86] Sending heartbeat to Boskos W0129 13:55:19.803071 6109 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://35.202.2.44/api/v1/nodes": net/http: TLS handshake timeout W0129 13:55:39.812740 6109 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://35.202.2.44/api/v1/nodes": net/http: TLS handshake timeout W0129 13:55:49.819460 6109 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://35.202.2.44/api/v1/nodes": dial tcp 35.202.2.44:443: connect: connection refused W0129 13:55:59.822881 6109 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://35.202.2.44/api/v1/nodes": dial tcp 35.202.2.44:443: connect: connection refused W0129 13:56:09.829630 6109 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://35.202.2.44/api/v1/nodes": dial tcp 35.202.2.44:443: connect: connection refused I0129 13:56:29.353649 6109 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-c ControlPlane e2-standard-2 1 1 us-central1 ... skipping 8 lines ... KIND NAME MESSAGE Machine https://www.googleapis.com/compute/v1/projects/gce-cvm-upg-1-3-lat-ctl-skew/zones/us-central1-a/instances/control-plane-us-central1-a-tnc1 machine "https://www.googleapis.com/compute/v1/projects/gce-cvm-upg-1-3-lat-ctl-skew/zones/us-central1-a/instances/control-plane-us-central1-a-tnc1" has not yet joined cluster Machine https://www.googleapis.com/compute/v1/projects/gce-cvm-upg-1-3-lat-ctl-skew/zones/us-central1-a/instances/nodes-us-central1-a-h2xb machine "https://www.googleapis.com/compute/v1/projects/gce-cvm-upg-1-3-lat-ctl-skew/zones/us-central1-a/instances/nodes-us-central1-a-h2xb" has not yet joined cluster Machine https://www.googleapis.com/compute/v1/projects/gce-cvm-upg-1-3-lat-ctl-skew/zones/us-central1-b/instances/control-plane-us-central1-b-qrn9 machine "https://www.googleapis.com/compute/v1/projects/gce-cvm-upg-1-3-lat-ctl-skew/zones/us-central1-b/instances/control-plane-us-central1-b-qrn9" has not yet joined cluster Machine https://www.googleapis.com/compute/v1/projects/gce-cvm-upg-1-3-lat-ctl-skew/zones/us-central1-c/instances/control-plane-us-central1-c-mcc0 machine "https://www.googleapis.com/compute/v1/projects/gce-cvm-upg-1-3-lat-ctl-skew/zones/us-central1-c/instances/control-plane-us-central1-c-mcc0" has not yet joined cluster Validation Failed W0129 13:56:30.329134 6109 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 13:56:40.838193 6109 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 10 lines ... KIND NAME MESSAGE Machine https://www.googleapis.com/compute/v1/projects/gce-cvm-upg-1-3-lat-ctl-skew/zones/us-central1-a/instances/nodes-us-central1-a-h2xb machine "https://www.googleapis.com/compute/v1/projects/gce-cvm-upg-1-3-lat-ctl-skew/zones/us-central1-a/instances/nodes-us-central1-a-h2xb" has not yet joined cluster Machine https://www.googleapis.com/compute/v1/projects/gce-cvm-upg-1-3-lat-ctl-skew/zones/us-central1-b/instances/control-plane-us-central1-b-qrn9 machine "https://www.googleapis.com/compute/v1/projects/gce-cvm-upg-1-3-lat-ctl-skew/zones/us-central1-b/instances/control-plane-us-central1-b-qrn9" has not yet joined cluster Machine https://www.googleapis.com/compute/v1/projects/gce-cvm-upg-1-3-lat-ctl-skew/zones/us-central1-c/instances/control-plane-us-central1-c-mcc0 machine "https://www.googleapis.com/compute/v1/projects/gce-cvm-upg-1-3-lat-ctl-skew/zones/us-central1-c/instances/control-plane-us-central1-c-mcc0" has not yet joined cluster Node control-plane-us-central1-a-tnc1 node "control-plane-us-central1-a-tnc1" of role "control-plane" is not ready Validation Failed W0129 13:56:41.704811 6109 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 13:56:52.096123 6109 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 10 lines ... KIND NAME MESSAGE Machine https://www.googleapis.com/compute/v1/projects/gce-cvm-upg-1-3-lat-ctl-skew/zones/us-central1-a/instances/nodes-us-central1-a-h2xb machine "https://www.googleapis.com/compute/v1/projects/gce-cvm-upg-1-3-lat-ctl-skew/zones/us-central1-a/instances/nodes-us-central1-a-h2xb" has not yet joined cluster Machine https://www.googleapis.com/compute/v1/projects/gce-cvm-upg-1-3-lat-ctl-skew/zones/us-central1-b/instances/control-plane-us-central1-b-qrn9 machine "https://www.googleapis.com/compute/v1/projects/gce-cvm-upg-1-3-lat-ctl-skew/zones/us-central1-b/instances/control-plane-us-central1-b-qrn9" has not yet joined cluster Machine https://www.googleapis.com/compute/v1/projects/gce-cvm-upg-1-3-lat-ctl-skew/zones/us-central1-c/instances/control-plane-us-central1-c-mcc0 machine "https://www.googleapis.com/compute/v1/projects/gce-cvm-upg-1-3-lat-ctl-skew/zones/us-central1-c/instances/control-plane-us-central1-c-mcc0" has not yet joined cluster Node control-plane-us-central1-a-tnc1 node "control-plane-us-central1-a-tnc1" of role "control-plane" is not ready Validation Failed W0129 13:56:52.928865 6109 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 13:57:03.324813 6109 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 13 lines ... Machine https://www.googleapis.com/compute/v1/projects/gce-cvm-upg-1-3-lat-ctl-skew/zones/us-central1-c/instances/control-plane-us-central1-c-mcc0 machine "https://www.googleapis.com/compute/v1/projects/gce-cvm-upg-1-3-lat-ctl-skew/zones/us-central1-c/instances/control-plane-us-central1-c-mcc0" has not yet joined cluster Node control-plane-us-central1-a-tnc1 node "control-plane-us-central1-a-tnc1" of role "control-plane" is not ready Node control-plane-us-central1-b-qrn9 node "control-plane-us-central1-b-qrn9" of role "control-plane" is not ready Pod kube-system/etcd-manager-events-control-plane-us-central1-a-tnc1 system-cluster-critical pod "etcd-manager-events-control-plane-us-central1-a-tnc1" is pending Pod kube-system/kube-controller-manager-control-plane-us-central1-a-tnc1 system-cluster-critical pod "kube-controller-manager-control-plane-us-central1-a-tnc1" is not ready (kube-controller-manager) Validation Failed W0129 13:57:04.188140 6109 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 13:57:14.741769 6109 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 13 lines ... Machine https://www.googleapis.com/compute/v1/projects/gce-cvm-upg-1-3-lat-ctl-skew/zones/us-central1-c/instances/control-plane-us-central1-c-mcc0 machine "https://www.googleapis.com/compute/v1/projects/gce-cvm-upg-1-3-lat-ctl-skew/zones/us-central1-c/instances/control-plane-us-central1-c-mcc0" has not yet joined cluster Node control-plane-us-central1-a-tnc1 node "control-plane-us-central1-a-tnc1" of role "control-plane" is not ready Node control-plane-us-central1-b-qrn9 node "control-plane-us-central1-b-qrn9" of role "control-plane" is not ready Pod kube-system/kube-controller-manager-control-plane-us-central1-a-tnc1 system-cluster-critical pod "kube-controller-manager-control-plane-us-central1-a-tnc1" is not ready (kube-controller-manager) Pod kube-system/kube-controller-manager-control-plane-us-central1-b-qrn9 system-cluster-critical pod "kube-controller-manager-control-plane-us-central1-b-qrn9" is pending Validation Failed W0129 13:57:15.566858 6109 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 13:57:26.022493 6109 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 13 lines ... Machine https://www.googleapis.com/compute/v1/projects/gce-cvm-upg-1-3-lat-ctl-skew/zones/us-central1-a/instances/nodes-us-central1-a-h2xb machine "https://www.googleapis.com/compute/v1/projects/gce-cvm-upg-1-3-lat-ctl-skew/zones/us-central1-a/instances/nodes-us-central1-a-h2xb" has not yet joined cluster Node control-plane-us-central1-a-tnc1 node "control-plane-us-central1-a-tnc1" of role "control-plane" is not ready Node control-plane-us-central1-b-qrn9 node "control-plane-us-central1-b-qrn9" of role "control-plane" is not ready Node control-plane-us-central1-c-mcc0 node "control-plane-us-central1-c-mcc0" of role "control-plane" is not ready Pod kube-system/kube-controller-manager-control-plane-us-central1-b-qrn9 system-cluster-critical pod "kube-controller-manager-control-plane-us-central1-b-qrn9" is not ready (kube-controller-manager) Validation Failed W0129 13:57:26.930308 6109 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 13:57:37.317962 6109 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 14 lines ... Node control-plane-us-central1-a-tnc1 node "control-plane-us-central1-a-tnc1" of role "control-plane" is not ready Node control-plane-us-central1-b-qrn9 node "control-plane-us-central1-b-qrn9" of role "control-plane" is not ready Node control-plane-us-central1-c-mcc0 node "control-plane-us-central1-c-mcc0" of role "control-plane" is not ready Pod kube-system/kube-controller-manager-control-plane-us-central1-b-qrn9 system-cluster-critical pod "kube-controller-manager-control-plane-us-central1-b-qrn9" is not ready (kube-controller-manager) Pod kube-system/kube-controller-manager-control-plane-us-central1-c-mcc0 system-cluster-critical pod "kube-controller-manager-control-plane-us-central1-c-mcc0" is pending Validation Failed W0129 13:57:38.159817 6109 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 13:57:48.603288 6109 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 21 lines ... Pod kube-system/coredns-5d479c4f64-fjgf9 system-cluster-critical pod "coredns-5d479c4f64-fjgf9" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-k5wn2 system-cluster-critical pod "coredns-autoscaler-557ccb4c66-k5wn2" is pending Pod kube-system/dns-controller-6f69f47785-fz5ft system-cluster-critical pod "dns-controller-6f69f47785-fz5ft" is pending Pod kube-system/etcd-manager-main-control-plane-us-central1-c-mcc0 system-cluster-critical pod "etcd-manager-main-control-plane-us-central1-c-mcc0" is pending Pod kube-system/kube-controller-manager-control-plane-us-central1-c-mcc0 system-cluster-critical pod "kube-controller-manager-control-plane-us-central1-c-mcc0" is not ready (kube-controller-manager) Validation Failed W0129 13:57:49.527515 6109 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 13:57:59.923664 6109 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 19 lines ... Pod kube-system/cilium-rfbkh system-node-critical pod "cilium-rfbkh" is not ready (cilium-agent) Pod kube-system/coredns-5d479c4f64-fjgf9 system-cluster-critical pod "coredns-5d479c4f64-fjgf9" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-k5wn2 system-cluster-critical pod "coredns-autoscaler-557ccb4c66-k5wn2" is pending Pod kube-system/dns-controller-6f69f47785-fz5ft system-cluster-critical pod "dns-controller-6f69f47785-fz5ft" is pending Pod kube-system/etcd-manager-events-control-plane-us-central1-c-mcc0 system-cluster-critical pod "etcd-manager-events-control-plane-us-central1-c-mcc0" is pending Validation Failed W0129 13:58:00.933946 6109 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 13:58:11.351976 6109 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 19 lines ... Pod kube-system/cilium-rfbkh system-node-critical pod "cilium-rfbkh" is not ready (cilium-agent) Pod kube-system/coredns-5d479c4f64-fjgf9 system-cluster-critical pod "coredns-5d479c4f64-fjgf9" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-k5wn2 system-cluster-critical pod "coredns-autoscaler-557ccb4c66-k5wn2" is pending Pod kube-system/dns-controller-6f69f47785-fz5ft system-cluster-critical pod "dns-controller-6f69f47785-fz5ft" is pending Pod kube-system/etcd-manager-events-control-plane-us-central1-b-qrn9 system-cluster-critical pod "etcd-manager-events-control-plane-us-central1-b-qrn9" is pending Validation Failed W0129 13:58:12.226419 6109 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 13:58:22.680613 6109 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 16 lines ... Node control-plane-us-central1-c-mcc0 node "control-plane-us-central1-c-mcc0" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-fjgf9 system-cluster-critical pod "coredns-5d479c4f64-fjgf9" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-k5wn2 system-cluster-critical pod "coredns-autoscaler-557ccb4c66-k5wn2" is pending Pod kube-system/dns-controller-6f69f47785-fz5ft system-cluster-critical pod "dns-controller-6f69f47785-fz5ft" is pending Pod kube-system/kube-apiserver-control-plane-us-central1-b-qrn9 system-cluster-critical pod "kube-apiserver-control-plane-us-central1-b-qrn9" is pending Validation Failed W0129 13:58:23.573219 6109 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 13:58:34.034605 6109 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-qrn9 node "control-plane-us-central1-b-qrn9" of role "control-plane" is not ready Node control-plane-us-central1-c-mcc0 node "control-plane-us-central1-c-mcc0" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-fjgf9 system-cluster-critical pod "coredns-5d479c4f64-fjgf9" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-k5wn2 system-cluster-critical pod "coredns-autoscaler-557ccb4c66-k5wn2" is pending Pod kube-system/dns-controller-6f69f47785-fz5ft system-cluster-critical pod "dns-controller-6f69f47785-fz5ft" is pending Validation Failed W0129 13:58:34.853153 6109 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 13:58:45.269645 6109 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-qrn9 node "control-plane-us-central1-b-qrn9" of role "control-plane" is not ready Node control-plane-us-central1-c-mcc0 node "control-plane-us-central1-c-mcc0" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-fjgf9 system-cluster-critical pod "coredns-5d479c4f64-fjgf9" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-k5wn2 system-cluster-critical pod "coredns-autoscaler-557ccb4c66-k5wn2" is pending Pod kube-system/dns-controller-6f69f47785-fz5ft system-cluster-critical pod "dns-controller-6f69f47785-fz5ft" is pending Validation Failed W0129 13:58:46.132692 6109 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 13:58:56.625127 6109 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-qrn9 node "control-plane-us-central1-b-qrn9" of role "control-plane" is not ready Node control-plane-us-central1-c-mcc0 node "control-plane-us-central1-c-mcc0" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-fjgf9 system-cluster-critical pod "coredns-5d479c4f64-fjgf9" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-k5wn2 system-cluster-critical pod "coredns-autoscaler-557ccb4c66-k5wn2" is pending Pod kube-system/dns-controller-6f69f47785-fz5ft system-cluster-critical pod "dns-controller-6f69f47785-fz5ft" is pending Validation Failed W0129 13:58:57.486080 6109 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 13:59:07.905921 6109 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-qrn9 node "control-plane-us-central1-b-qrn9" of role "control-plane" is not ready Node control-plane-us-central1-c-mcc0 node "control-plane-us-central1-c-mcc0" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-fjgf9 system-cluster-critical pod "coredns-5d479c4f64-fjgf9" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-k5wn2 system-cluster-critical pod "coredns-autoscaler-557ccb4c66-k5wn2" is pending Pod kube-system/dns-controller-6f69f47785-fz5ft system-cluster-critical pod "dns-controller-6f69f47785-fz5ft" is pending Validation Failed W0129 13:59:08.660043 6109 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 13:59:19.082935 6109 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-qrn9 node "control-plane-us-central1-b-qrn9" of role "control-plane" is not ready Node control-plane-us-central1-c-mcc0 node "control-plane-us-central1-c-mcc0" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-fjgf9 system-cluster-critical pod "coredns-5d479c4f64-fjgf9" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-k5wn2 system-cluster-critical pod "coredns-autoscaler-557ccb4c66-k5wn2" is pending Pod kube-system/dns-controller-6f69f47785-fz5ft system-cluster-critical pod "dns-controller-6f69f47785-fz5ft" is pending Validation Failed W0129 13:59:19.915692 6109 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 13:59:30.386876 6109 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-qrn9 node "control-plane-us-central1-b-qrn9" of role "control-plane" is not ready Node control-plane-us-central1-c-mcc0 node "control-plane-us-central1-c-mcc0" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-fjgf9 system-cluster-critical pod "coredns-5d479c4f64-fjgf9" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-k5wn2 system-cluster-critical pod "coredns-autoscaler-557ccb4c66-k5wn2" is pending Pod kube-system/dns-controller-6f69f47785-fz5ft system-cluster-critical pod "dns-controller-6f69f47785-fz5ft" is pending Validation Failed W0129 13:59:31.259213 6109 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 13:59:41.667439 6109 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-qrn9 node "control-plane-us-central1-b-qrn9" of role "control-plane" is not ready Node control-plane-us-central1-c-mcc0 node "control-plane-us-central1-c-mcc0" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-fjgf9 system-cluster-critical pod "coredns-5d479c4f64-fjgf9" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-k5wn2 system-cluster-critical pod "coredns-autoscaler-557ccb4c66-k5wn2" is pending Pod kube-system/dns-controller-6f69f47785-fz5ft system-cluster-critical pod "dns-controller-6f69f47785-fz5ft" is pending Validation Failed W0129 13:59:42.545022 6109 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 13:59:52.916046 6109 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-qrn9 node "control-plane-us-central1-b-qrn9" of role "control-plane" is not ready Node control-plane-us-central1-c-mcc0 node "control-plane-us-central1-c-mcc0" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-fjgf9 system-cluster-critical pod "coredns-5d479c4f64-fjgf9" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-k5wn2 system-cluster-critical pod "coredns-autoscaler-557ccb4c66-k5wn2" is pending Pod kube-system/dns-controller-6f69f47785-fz5ft system-cluster-critical pod "dns-controller-6f69f47785-fz5ft" is pending Validation Failed W0129 13:59:53.822747 6109 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 14:00:04.188498 6109 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-qrn9 node "control-plane-us-central1-b-qrn9" of role "control-plane" is not ready Node control-plane-us-central1-c-mcc0 node "control-plane-us-central1-c-mcc0" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-fjgf9 system-cluster-critical pod "coredns-5d479c4f64-fjgf9" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-k5wn2 system-cluster-critical pod "coredns-autoscaler-557ccb4c66-k5wn2" is pending Pod kube-system/dns-controller-6f69f47785-fz5ft system-cluster-critical pod "dns-controller-6f69f47785-fz5ft" is pending Validation Failed W0129 14:00:05.029512 6109 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 14:00:15.528583 6109 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-qrn9 node "control-plane-us-central1-b-qrn9" of role "control-plane" is not ready Node control-plane-us-central1-c-mcc0 node "control-plane-us-central1-c-mcc0" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-fjgf9 system-cluster-critical pod "coredns-5d479c4f64-fjgf9" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-k5wn2 system-cluster-critical pod "coredns-autoscaler-557ccb4c66-k5wn2" is pending Pod kube-system/dns-controller-6f69f47785-fz5ft system-cluster-critical pod "dns-controller-6f69f47785-fz5ft" is pending Validation Failed W0129 14:00:16.374327 6109 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 14:00:18.468270 5750 boskos.go:86] Sending heartbeat to Boskos I0129 14:00:26.787625 6109 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 ... skipping 16 lines ... Node control-plane-us-central1-b-qrn9 node "control-plane-us-central1-b-qrn9" of role "control-plane" is not ready Node control-plane-us-central1-c-mcc0 node "control-plane-us-central1-c-mcc0" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-fjgf9 system-cluster-critical pod "coredns-5d479c4f64-fjgf9" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-k5wn2 system-cluster-critical pod "coredns-autoscaler-557ccb4c66-k5wn2" is pending Pod kube-system/dns-controller-6f69f47785-fz5ft system-cluster-critical pod "dns-controller-6f69f47785-fz5ft" is pending Validation Failed W0129 14:00:27.643505 6109 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 14:00:38.051426 6109 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-qrn9 node "control-plane-us-central1-b-qrn9" of role "control-plane" is not ready Node control-plane-us-central1-c-mcc0 node "control-plane-us-central1-c-mcc0" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-fjgf9 system-cluster-critical pod "coredns-5d479c4f64-fjgf9" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-k5wn2 system-cluster-critical pod "coredns-autoscaler-557ccb4c66-k5wn2" is pending Pod kube-system/dns-controller-6f69f47785-fz5ft system-cluster-critical pod "dns-controller-6f69f47785-fz5ft" is pending Validation Failed W0129 14:00:38.820316 6109 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 14:00:49.190052 6109 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-qrn9 node "control-plane-us-central1-b-qrn9" of role "control-plane" is not ready Node control-plane-us-central1-c-mcc0 node "control-plane-us-central1-c-mcc0" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-fjgf9 system-cluster-critical pod "coredns-5d479c4f64-fjgf9" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-k5wn2 system-cluster-critical pod "coredns-autoscaler-557ccb4c66-k5wn2" is pending Pod kube-system/dns-controller-6f69f47785-fz5ft system-cluster-critical pod "dns-controller-6f69f47785-fz5ft" is pending Validation Failed W0129 14:00:49.993430 6109 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 14:01:00.374562 6109 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-qrn9 node "control-plane-us-central1-b-qrn9" of role "control-plane" is not ready Node control-plane-us-central1-c-mcc0 node "control-plane-us-central1-c-mcc0" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-fjgf9 system-cluster-critical pod "coredns-5d479c4f64-fjgf9" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-k5wn2 system-cluster-critical pod "coredns-autoscaler-557ccb4c66-k5wn2" is pending Pod kube-system/dns-controller-6f69f47785-fz5ft system-cluster-critical pod "dns-controller-6f69f47785-fz5ft" is pending Validation Failed W0129 14:01:01.280694 6109 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 14:01:11.711334 6109 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-qrn9 node "control-plane-us-central1-b-qrn9" of role "control-plane" is not ready Node control-plane-us-central1-c-mcc0 node "control-plane-us-central1-c-mcc0" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-fjgf9 system-cluster-critical pod "coredns-5d479c4f64-fjgf9" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-k5wn2 system-cluster-critical pod "coredns-autoscaler-557ccb4c66-k5wn2" is pending Pod kube-system/dns-controller-6f69f47785-fz5ft system-cluster-critical pod "dns-controller-6f69f47785-fz5ft" is pending Validation Failed W0129 14:01:12.580225 6109 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 14:01:23.003526 6109 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-qrn9 node "control-plane-us-central1-b-qrn9" of role "control-plane" is not ready Node control-plane-us-central1-c-mcc0 node "control-plane-us-central1-c-mcc0" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-fjgf9 system-cluster-critical pod "coredns-5d479c4f64-fjgf9" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-k5wn2 system-cluster-critical pod "coredns-autoscaler-557ccb4c66-k5wn2" is pending Pod kube-system/dns-controller-6f69f47785-fz5ft system-cluster-critical pod "dns-controller-6f69f47785-fz5ft" is pending Validation Failed W0129 14:01:23.866878 6109 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 14:01:34.271369 6109 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-qrn9 node "control-plane-us-central1-b-qrn9" of role "control-plane" is not ready Node control-plane-us-central1-c-mcc0 node "control-plane-us-central1-c-mcc0" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-fjgf9 system-cluster-critical pod "coredns-5d479c4f64-fjgf9" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-k5wn2 system-cluster-critical pod "coredns-autoscaler-557ccb4c66-k5wn2" is pending Pod kube-system/dns-controller-6f69f47785-fz5ft system-cluster-critical pod "dns-controller-6f69f47785-fz5ft" is pending Validation Failed W0129 14:01:35.086117 6109 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 14:01:45.477888 6109 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-qrn9 node "control-plane-us-central1-b-qrn9" of role "control-plane" is not ready Node control-plane-us-central1-c-mcc0 node "control-plane-us-central1-c-mcc0" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-fjgf9 system-cluster-critical pod "coredns-5d479c4f64-fjgf9" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-k5wn2 system-cluster-critical pod "coredns-autoscaler-557ccb4c66-k5wn2" is pending Pod kube-system/dns-controller-6f69f47785-fz5ft system-cluster-critical pod "dns-controller-6f69f47785-fz5ft" is pending Validation Failed W0129 14:01:46.316186 6109 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 14:01:56.755728 6109 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-qrn9 node "control-plane-us-central1-b-qrn9" of role "control-plane" is not ready Node control-plane-us-central1-c-mcc0 node "control-plane-us-central1-c-mcc0" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-fjgf9 system-cluster-critical pod "coredns-5d479c4f64-fjgf9" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-k5wn2 system-cluster-critical pod "coredns-autoscaler-557ccb4c66-k5wn2" is pending Pod kube-system/dns-controller-6f69f47785-fz5ft system-cluster-critical pod "dns-controller-6f69f47785-fz5ft" is pending Validation Failed W0129 14:01:57.509557 6109 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 14:02:07.940523 6109 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-qrn9 node "control-plane-us-central1-b-qrn9" of role "control-plane" is not ready Node control-plane-us-central1-c-mcc0 node "control-plane-us-central1-c-mcc0" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-fjgf9 system-cluster-critical pod "coredns-5d479c4f64-fjgf9" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-k5wn2 system-cluster-critical pod "coredns-autoscaler-557ccb4c66-k5wn2" is pending Pod kube-system/dns-controller-6f69f47785-fz5ft system-cluster-critical pod "dns-controller-6f69f47785-fz5ft" is pending Validation Failed W0129 14:02:08.807913 6109 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 14:02:19.207188 6109 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-qrn9 node "control-plane-us-central1-b-qrn9" of role "control-plane" is not ready Node control-plane-us-central1-c-mcc0 node "control-plane-us-central1-c-mcc0" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-fjgf9 system-cluster-critical pod "coredns-5d479c4f64-fjgf9" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-k5wn2 system-cluster-critical pod "coredns-autoscaler-557ccb4c66-k5wn2" is pending Pod kube-system/dns-controller-6f69f47785-fz5ft system-cluster-critical pod "dns-controller-6f69f47785-fz5ft" is pending Validation Failed W0129 14:02:20.037975 6109 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 14:02:30.416675 6109 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-qrn9 node "control-plane-us-central1-b-qrn9" of role "control-plane" is not ready Node control-plane-us-central1-c-mcc0 node "control-plane-us-central1-c-mcc0" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-fjgf9 system-cluster-critical pod "coredns-5d479c4f64-fjgf9" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-k5wn2 system-cluster-critical pod "coredns-autoscaler-557ccb4c66-k5wn2" is pending Pod kube-system/dns-controller-6f69f47785-fz5ft system-cluster-critical pod "dns-controller-6f69f47785-fz5ft" is pending Validation Failed W0129 14:02:31.285543 6109 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 14:02:41.720228 6109 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-qrn9 node "control-plane-us-central1-b-qrn9" of role "control-plane" is not ready Node control-plane-us-central1-c-mcc0 node "control-plane-us-central1-c-mcc0" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-fjgf9 system-cluster-critical pod "coredns-5d479c4f64-fjgf9" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-k5wn2 system-cluster-critical pod "coredns-autoscaler-557ccb4c66-k5wn2" is pending Pod kube-system/dns-controller-6f69f47785-fz5ft system-cluster-critical pod "dns-controller-6f69f47785-fz5ft" is pending Validation Failed W0129 14:02:42.572691 6109 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 14:02:52.986638 6109 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-qrn9 node "control-plane-us-central1-b-qrn9" of role "control-plane" is not ready Node control-plane-us-central1-c-mcc0 node "control-plane-us-central1-c-mcc0" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-fjgf9 system-cluster-critical pod "coredns-5d479c4f64-fjgf9" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-k5wn2 system-cluster-critical pod "coredns-autoscaler-557ccb4c66-k5wn2" is pending Pod kube-system/dns-controller-6f69f47785-fz5ft system-cluster-critical pod "dns-controller-6f69f47785-fz5ft" is pending Validation Failed W0129 14:02:53.844514 6109 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 14:03:04.271972 6109 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-qrn9 node "control-plane-us-central1-b-qrn9" of role "control-plane" is not ready Node control-plane-us-central1-c-mcc0 node "control-plane-us-central1-c-mcc0" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-fjgf9 system-cluster-critical pod "coredns-5d479c4f64-fjgf9" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-k5wn2 system-cluster-critical pod "coredns-autoscaler-557ccb4c66-k5wn2" is pending Pod kube-system/dns-controller-6f69f47785-fz5ft system-cluster-critical pod "dns-controller-6f69f47785-fz5ft" is pending Validation Failed W0129 14:03:05.047045 6109 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 14:03:15.443158 6109 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-qrn9 node "control-plane-us-central1-b-qrn9" of role "control-plane" is not ready Node control-plane-us-central1-c-mcc0 node "control-plane-us-central1-c-mcc0" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-fjgf9 system-cluster-critical pod "coredns-5d479c4f64-fjgf9" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-k5wn2 system-cluster-critical pod "coredns-autoscaler-557ccb4c66-k5wn2" is pending Pod kube-system/dns-controller-6f69f47785-fz5ft system-cluster-critical pod "dns-controller-6f69f47785-fz5ft" is pending Validation Failed W0129 14:03:16.279846 6109 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 14:03:26.771646 6109 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-qrn9 node "control-plane-us-central1-b-qrn9" of role "control-plane" is not ready Node control-plane-us-central1-c-mcc0 node "control-plane-us-central1-c-mcc0" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-fjgf9 system-cluster-critical pod "coredns-5d479c4f64-fjgf9" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-k5wn2 system-cluster-critical pod "coredns-autoscaler-557ccb4c66-k5wn2" is pending Pod kube-system/dns-controller-6f69f47785-fz5ft system-cluster-critical pod "dns-controller-6f69f47785-fz5ft" is pending Validation Failed W0129 14:03:27.647266 6109 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 14:03:38.109856 6109 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-qrn9 node "control-plane-us-central1-b-qrn9" of role "control-plane" is not ready Node control-plane-us-central1-c-mcc0 node "control-plane-us-central1-c-mcc0" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-fjgf9 system-cluster-critical pod "coredns-5d479c4f64-fjgf9" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-k5wn2 system-cluster-critical pod "coredns-autoscaler-557ccb4c66-k5wn2" is pending Pod kube-system/dns-controller-6f69f47785-fz5ft system-cluster-critical pod "dns-controller-6f69f47785-fz5ft" is pending Validation Failed W0129 14:03:38.949895 6109 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 14:03:49.347489 6109 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-qrn9 node "control-plane-us-central1-b-qrn9" of role "control-plane" is not ready Node control-plane-us-central1-c-mcc0 node "control-plane-us-central1-c-mcc0" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-fjgf9 system-cluster-critical pod "coredns-5d479c4f64-fjgf9" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-k5wn2 system-cluster-critical pod "coredns-autoscaler-557ccb4c66-k5wn2" is pending Pod kube-system/dns-controller-6f69f47785-fz5ft system-cluster-critical pod "dns-controller-6f69f47785-fz5ft" is pending Validation Failed W0129 14:03:50.278263 6109 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 14:04:00.738599 6109 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-qrn9 node "control-plane-us-central1-b-qrn9" of role "control-plane" is not ready Node control-plane-us-central1-c-mcc0 node "control-plane-us-central1-c-mcc0" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-fjgf9 system-cluster-critical pod "coredns-5d479c4f64-fjgf9" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-k5wn2 system-cluster-critical pod "coredns-autoscaler-557ccb4c66-k5wn2" is pending Pod kube-system/dns-controller-6f69f47785-fz5ft system-cluster-critical pod "dns-controller-6f69f47785-fz5ft" is pending Validation Failed W0129 14:04:01.720660 6109 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 14:04:12.114227 6109 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-qrn9 node "control-plane-us-central1-b-qrn9" of role "control-plane" is not ready Node control-plane-us-central1-c-mcc0 node "control-plane-us-central1-c-mcc0" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-fjgf9 system-cluster-critical pod "coredns-5d479c4f64-fjgf9" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-k5wn2 system-cluster-critical pod "coredns-autoscaler-557ccb4c66-k5wn2" is pending Pod kube-system/dns-controller-6f69f47785-fz5ft system-cluster-critical pod "dns-controller-6f69f47785-fz5ft" is pending Validation Failed W0129 14:04:12.991777 6109 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 14:04:23.396112 6109 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-qrn9 node "control-plane-us-central1-b-qrn9" of role "control-plane" is not ready Node control-plane-us-central1-c-mcc0 node "control-plane-us-central1-c-mcc0" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-fjgf9 system-cluster-critical pod "coredns-5d479c4f64-fjgf9" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-k5wn2 system-cluster-critical pod "coredns-autoscaler-557ccb4c66-k5wn2" is pending Pod kube-system/dns-controller-6f69f47785-fz5ft system-cluster-critical pod "dns-controller-6f69f47785-fz5ft" is pending Validation Failed W0129 14:04:24.176919 6109 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 14:04:34.620354 6109 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-qrn9 node "control-plane-us-central1-b-qrn9" of role "control-plane" is not ready Node control-plane-us-central1-c-mcc0 node "control-plane-us-central1-c-mcc0" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-fjgf9 system-cluster-critical pod "coredns-5d479c4f64-fjgf9" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-k5wn2 system-cluster-critical pod "coredns-autoscaler-557ccb4c66-k5wn2" is pending Pod kube-system/dns-controller-6f69f47785-fz5ft system-cluster-critical pod "dns-controller-6f69f47785-fz5ft" is pending Validation Failed W0129 14:04:35.417903 6109 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 14:04:45.847889 6109 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-qrn9 node "control-plane-us-central1-b-qrn9" of role "control-plane" is not ready Node control-plane-us-central1-c-mcc0 node "control-plane-us-central1-c-mcc0" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-fjgf9 system-cluster-critical pod "coredns-5d479c4f64-fjgf9" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-k5wn2 system-cluster-critical pod "coredns-autoscaler-557ccb4c66-k5wn2" is pending Pod kube-system/dns-controller-6f69f47785-fz5ft system-cluster-critical pod "dns-controller-6f69f47785-fz5ft" is pending Validation Failed W0129 14:04:46.694739 6109 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 14:04:57.127727 6109 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-qrn9 node "control-plane-us-central1-b-qrn9" of role "control-plane" is not ready Node control-plane-us-central1-c-mcc0 node "control-plane-us-central1-c-mcc0" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-fjgf9 system-cluster-critical pod "coredns-5d479c4f64-fjgf9" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-k5wn2 system-cluster-critical pod "coredns-autoscaler-557ccb4c66-k5wn2" is pending Pod kube-system/dns-controller-6f69f47785-fz5ft system-cluster-critical pod "dns-controller-6f69f47785-fz5ft" is pending Validation Failed W0129 14:04:57.916186 6109 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 14:05:08.378061 6109 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-qrn9 node "control-plane-us-central1-b-qrn9" of role "control-plane" is not ready Node control-plane-us-central1-c-mcc0 node "control-plane-us-central1-c-mcc0" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-fjgf9 system-cluster-critical pod "coredns-5d479c4f64-fjgf9" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-k5wn2 system-cluster-critical pod "coredns-autoscaler-557ccb4c66-k5wn2" is pending Pod kube-system/dns-controller-6f69f47785-fz5ft system-cluster-critical pod "dns-controller-6f69f47785-fz5ft" is pending Validation Failed W0129 14:05:09.149781 6109 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 14:05:18.481749 5750 boskos.go:86] Sending heartbeat to Boskos I0129 14:05:19.579087 6109 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 ... skipping 16 lines ... Node control-plane-us-central1-b-qrn9 node "control-plane-us-central1-b-qrn9" of role "control-plane" is not ready Node control-plane-us-central1-c-mcc0 node "control-plane-us-central1-c-mcc0" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-fjgf9 system-cluster-critical pod "coredns-5d479c4f64-fjgf9" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-k5wn2 system-cluster-critical pod "coredns-autoscaler-557ccb4c66-k5wn2" is pending Pod kube-system/dns-controller-6f69f47785-fz5ft system-cluster-critical pod "dns-controller-6f69f47785-fz5ft" is pending Validation Failed W0129 14:05:20.435859 6109 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 14:05:30.919842 6109 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-qrn9 node "control-plane-us-central1-b-qrn9" of role "control-plane" is not ready Node control-plane-us-central1-c-mcc0 node "control-plane-us-central1-c-mcc0" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-fjgf9 system-cluster-critical pod "coredns-5d479c4f64-fjgf9" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-k5wn2 system-cluster-critical pod "coredns-autoscaler-557ccb4c66-k5wn2" is pending Pod kube-system/dns-controller-6f69f47785-fz5ft system-cluster-critical pod "dns-controller-6f69f47785-fz5ft" is pending Validation Failed W0129 14:05:31.740477 6109 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 14:05:42.184790 6109 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-qrn9 node "control-plane-us-central1-b-qrn9" of role "control-plane" is not ready Node control-plane-us-central1-c-mcc0 node "control-plane-us-central1-c-mcc0" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-fjgf9 system-cluster-critical pod "coredns-5d479c4f64-fjgf9" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-k5wn2 system-cluster-critical pod "coredns-autoscaler-557ccb4c66-k5wn2" is pending Pod kube-system/dns-controller-6f69f47785-fz5ft system-cluster-critical pod "dns-controller-6f69f47785-fz5ft" is pending Validation Failed W0129 14:05:43.057937 6109 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 14:05:53.480103 6109 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-qrn9 node "control-plane-us-central1-b-qrn9" of role "control-plane" is not ready Node control-plane-us-central1-c-mcc0 node "control-plane-us-central1-c-mcc0" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-fjgf9 system-cluster-critical pod "coredns-5d479c4f64-fjgf9" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-k5wn2 system-cluster-critical pod "coredns-autoscaler-557ccb4c66-k5wn2" is pending Pod kube-system/dns-controller-6f69f47785-fz5ft system-cluster-critical pod "dns-controller-6f69f47785-fz5ft" is pending Validation Failed W0129 14:05:54.266850 6109 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 14:06:04.656882 6109 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-qrn9 node "control-plane-us-central1-b-qrn9" of role "control-plane" is not ready Node control-plane-us-central1-c-mcc0 node "control-plane-us-central1-c-mcc0" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-fjgf9 system-cluster-critical pod "coredns-5d479c4f64-fjgf9" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-k5wn2 system-cluster-critical pod "coredns-autoscaler-557ccb4c66-k5wn2" is pending Pod kube-system/dns-controller-6f69f47785-fz5ft system-cluster-critical pod "dns-controller-6f69f47785-fz5ft" is pending Validation Failed W0129 14:06:05.461801 6109 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 14:06:15.933139 6109 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-qrn9 node "control-plane-us-central1-b-qrn9" of role "control-plane" is not ready Node control-plane-us-central1-c-mcc0 node "control-plane-us-central1-c-mcc0" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-fjgf9 system-cluster-critical pod "coredns-5d479c4f64-fjgf9" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-k5wn2 system-cluster-critical pod "coredns-autoscaler-557ccb4c66-k5wn2" is pending Pod kube-system/dns-controller-6f69f47785-fz5ft system-cluster-critical pod "dns-controller-6f69f47785-fz5ft" is pending Validation Failed W0129 14:06:16.761399 6109 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 14:06:27.212389 6109 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-qrn9 node "control-plane-us-central1-b-qrn9" of role "control-plane" is not ready Node control-plane-us-central1-c-mcc0 node "control-plane-us-central1-c-mcc0" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-fjgf9 system-cluster-critical pod "coredns-5d479c4f64-fjgf9" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-k5wn2 system-cluster-critical pod "coredns-autoscaler-557ccb4c66-k5wn2" is pending Pod kube-system/dns-controller-6f69f47785-fz5ft system-cluster-critical pod "dns-controller-6f69f47785-fz5ft" is pending Validation Failed W0129 14:06:28.065991 6109 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 14:06:38.452739 6109 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-qrn9 node "control-plane-us-central1-b-qrn9" of role "control-plane" is not ready Node control-plane-us-central1-c-mcc0 node "control-plane-us-central1-c-mcc0" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-fjgf9 system-cluster-critical pod "coredns-5d479c4f64-fjgf9" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-k5wn2 system-cluster-critical pod "coredns-autoscaler-557ccb4c66-k5wn2" is pending Pod kube-system/dns-controller-6f69f47785-fz5ft system-cluster-critical pod "dns-controller-6f69f47785-fz5ft" is pending Validation Failed W0129 14:06:39.267490 6109 validate_cluster.go:232] (will retry): cluster not yet healthy Error: validation failed: wait time exceeded during validation I0129 14:06:49.276073 5750 dumplogs.go:45] /home/prow/go/src/k8s.io/kops/_rundir/9831df3a-9fdb-11ed-9a0c-f25a29267a91/kops toolbox dump --name ha-migration.k8s.local --dir /logs/artifacts --private-key /tmp/kops-ssh32606979/key --ssh-user prow I0129 14:06:49.276127 5750 local.go:42] ⚙️ /home/prow/go/src/k8s.io/kops/_rundir/9831df3a-9fdb-11ed-9a0c-f25a29267a91/kops toolbox dump --name ha-migration.k8s.local --dir /logs/artifacts --private-key /tmp/kops-ssh32606979/key --ssh-user prow I0129 14:06:49.737686 6119 gce.go:99] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] 2023/01/29 14:06:54 Dumping node control-plane-us-central1-a-tnc1 2023/01/29 14:06:57 error dumping node control-plane-us-central1-a-tnc1: error executing command "if command -v kubectl &> /dev/null; then kubectl logs -n kube-system --all-containers -l \"k8s-app=external-dns\"; fi": Process exited with status 1 2023/01/29 14:06:57 error dumping node control-plane-us-central1-a-tnc1: error executing command "if command -v kubectl &> /dev/null; then kubectl logs -n kube-system --all-containers -l \"k8s-app=dns-controller\"; fi": Process exited with status 1 2023/01/29 14:06:57 Dumping node control-plane-us-central1-b-qrn9 2023/01/29 14:07:01 error dumping node control-plane-us-central1-b-qrn9: error executing command "if command -v kubectl &> /dev/null; then kubectl logs -n kube-system --all-containers -l \"k8s-app=external-dns\"; fi": Process exited with status 1 2023/01/29 14:07:01 error dumping node control-plane-us-central1-b-qrn9: error executing command "if command -v kubectl &> /dev/null; then kubectl logs -n kube-system --all-containers -l \"k8s-app=dns-controller\"; fi": Process exited with status 1 2023/01/29 14:07:01 Dumping node control-plane-us-central1-c-mcc0 2023/01/29 14:07:05 error dumping node control-plane-us-central1-c-mcc0: error executing command "if command -v kubectl &> /dev/null; then kubectl logs -n kube-system --all-containers -l \"k8s-app=external-dns\"; fi": Process exited with status 1 2023/01/29 14:07:05 error dumping node control-plane-us-central1-c-mcc0: error executing command "if command -v kubectl &> /dev/null; then kubectl logs -n kube-system --all-containers -l \"k8s-app=dns-controller\"; fi": Process exited with status 1 2023/01/29 14:07:05 dumping node not registered in kubernetes: 34.70.186.46 2023/01/29 14:07:05 Dumping node 34.70.186.46 I0129 14:07:07.153058 5750 dumplogs.go:79] /home/prow/go/src/k8s.io/kops/_rundir/9831df3a-9fdb-11ed-9a0c-f25a29267a91/kops get cluster --name ha-migration.k8s.local -o yaml I0129 14:07:07.153100 5750 local.go:42] ⚙️ /home/prow/go/src/k8s.io/kops/_rundir/9831df3a-9fdb-11ed-9a0c-f25a29267a91/kops get cluster --name ha-migration.k8s.local -o yaml I0129 14:07:07.328151 5750 dumplogs.go:79] /home/prow/go/src/k8s.io/kops/_rundir/9831df3a-9fdb-11ed-9a0c-f25a29267a91/kops get instancegroups --name ha-migration.k8s.local -o yaml I0129 14:07:07.328192 5750 local.go:42] ⚙️ /home/prow/go/src/k8s.io/kops/_rundir/9831df3a-9fdb-11ed-9a0c-f25a29267a91/kops get instancegroups --name ha-migration.k8s.local -o yaml ... skipping 167 lines ... Deleted cluster: "ha-migration.k8s.local" I0129 14:09:26.332510 5750 gcs.go:88] gsutil -u gce-cvm-upg-1-3-lat-ctl-skew rm -r gs://gce-cvm-upg-1-3-lat-ctl-skew-state-98 I0129 14:09:26.332595 5750 local.go:42] ⚙️ gsutil -u gce-cvm-upg-1-3-lat-ctl-skew rm -r gs://gce-cvm-upg-1-3-lat-ctl-skew-state-98 Removing gs://gce-cvm-upg-1-3-lat-ctl-skew-state-98/... I0129 14:09:29.035062 5750 down.go:62] releasing boskos project I0129 14:09:29.044995 5750 boskos.go:83] Boskos heartbeat func received signal to close Error: exit status 1 + EXIT_VALUE=1 + set +o xtrace