e2e-kops-gce-leader-migration #1619739236745875456
Recent runs || View in Spyglass
| Result | FAILURE |
| Tests | 1 failed / 1 succeeded |
| Started | |
| Elapsed | 20m19s |
| Revision | master |
exit status 1
from junit_runner.xml
Filter through log files | View test history on testgrid
kubetest2 Down
... skipping 156 lines ... ...................................Updated [https://www.googleapis.com/compute/v1/projects/k8s-jkns-pr-kubeadm]. ...done. WARNING: No host aliases were added to your SSH configs because you do not have any running instances. Try running this command again after running some instances. I0129 16:50:25.056204 5743 up.go:44] Cleaning up any leaked resources from previous cluster I0129 16:50:25.056351 5743 dumplogs.go:45] /home/prow/go/src/k8s.io/kops/_rundir/bd94eb0b-9ff4-11ed-9a0c-f25a29267a91/kops toolbox dump --name ha-migration.k8s.local --dir /logs/artifacts --private-key /tmp/kops-ssh540320256/key --ssh-user prow I0129 16:50:25.056370 5743 local.go:42] ⚙️ /home/prow/go/src/k8s.io/kops/_rundir/bd94eb0b-9ff4-11ed-9a0c-f25a29267a91/kops toolbox dump --name ha-migration.k8s.local --dir /logs/artifacts --private-key /tmp/kops-ssh540320256/key --ssh-user prow Error: Cluster.kops.k8s.io "ha-migration.k8s.local" not found W0129 16:50:25.318955 5743 down.go:34] Dumping cluster logs at the start of Down() failed: exit status 1 I0129 16:50:25.319056 5743 down.go:48] /home/prow/go/src/k8s.io/kops/_rundir/bd94eb0b-9ff4-11ed-9a0c-f25a29267a91/kops delete cluster --name ha-migration.k8s.local --yes I0129 16:50:25.319070 5743 local.go:42] ⚙️ /home/prow/go/src/k8s.io/kops/_rundir/bd94eb0b-9ff4-11ed-9a0c-f25a29267a91/kops delete cluster --name ha-migration.k8s.local --yes Error: error reading cluster configuration: Cluster.kops.k8s.io "ha-migration.k8s.local" not found I0129 16:50:25.496487 5743 gcs.go:51] gsutil ls -b -p k8s-jkns-pr-kubeadm gs://k8s-jkns-pr-kubeadm-state-bd I0129 16:50:25.496530 5743 local.go:42] ⚙️ gsutil ls -b -p k8s-jkns-pr-kubeadm gs://k8s-jkns-pr-kubeadm-state-bd I0129 16:50:27.570874 5743 gcs.go:70] gsutil mb -p k8s-jkns-pr-kubeadm gs://k8s-jkns-pr-kubeadm-state-bd I0129 16:50:27.570952 5743 local.go:42] ⚙️ gsutil mb -p k8s-jkns-pr-kubeadm gs://k8s-jkns-pr-kubeadm-state-bd Creating gs://k8s-jkns-pr-kubeadm-state-bd/... I0129 16:50:30.261603 5743 http.go:37] curl http://metadata.google.internal/computeMetadata/v1/instance/network-interfaces/0/access-configs/0/external-ip 2023/01/29 16:50:30 failed to get external ip from metadata service: http://metadata.google.internal/computeMetadata/v1/instance/network-interfaces/0/access-configs/0/external-ip returned 404 I0129 16:50:30.276892 5743 http.go:37] curl https://ip.jsb.workers.dev I0129 16:50:30.376211 5743 up.go:167] /home/prow/go/src/k8s.io/kops/_rundir/bd94eb0b-9ff4-11ed-9a0c-f25a29267a91/kops create cluster --name ha-migration.k8s.local --cloud gce --kubernetes-version v1.23.16 --ssh-public-key /tmp/kops-ssh540320256/key.pub --set cluster.spec.nodePortAccess=0.0.0.0/0 --channel=alpha --node-count=1 --master-count=3 --zones=us-central1-a,us-central1-b,us-central1-c --master-zones=us-central1-a,us-central1-b,us-central1-c --gce-service-account=default --admin-access 35.192.168.24/32 --master-volume-size 48 --node-volume-size 48 --master-size e2-standard-2 --project k8s-jkns-pr-kubeadm I0129 16:50:30.376251 5743 local.go:42] ⚙️ /home/prow/go/src/k8s.io/kops/_rundir/bd94eb0b-9ff4-11ed-9a0c-f25a29267a91/kops create cluster --name ha-migration.k8s.local --cloud gce --kubernetes-version v1.23.16 --ssh-public-key /tmp/kops-ssh540320256/key.pub --set cluster.spec.nodePortAccess=0.0.0.0/0 --channel=alpha --node-count=1 --master-count=3 --zones=us-central1-a,us-central1-b,us-central1-c --master-zones=us-central1-a,us-central1-b,us-central1-c --gce-service-account=default --admin-access 35.192.168.24/32 --master-volume-size 48 --node-volume-size 48 --master-size e2-standard-2 --project k8s-jkns-pr-kubeadm I0129 16:50:30.427847 6062 create_cluster.go:882] Using SSH public key: /tmp/kops-ssh540320256/key.pub I0129 16:50:30.734553 6062 new_cluster.go:573] VMs will be configured to use specified Service Account: default I0129 16:50:30.734678 6062 new_cluster.go:1338] Cloud Provider ID: "gce" ... skipping 527 lines ... I0129 16:50:42.969635 6083 keypair.go:226] Issuing new certificate: "etcd-clients-ca" I0129 16:50:42.976261 6083 keypair.go:226] Issuing new certificate: "etcd-peers-ca-main" W0129 16:50:43.078027 6083 vfs_keystorereader.go:143] CA private key was not found I0129 16:50:43.178465 6083 keypair.go:226] Issuing new certificate: "kubernetes-ca" I0129 16:50:43.283108 6083 keypair.go:226] Issuing new certificate: "service-account" I0129 16:50:58.783294 6083 executor.go:111] Tasks: 50 done / 91 total; 23 can run W0129 16:51:11.767048 6083 executor.go:139] error running task "ForwardingRule/api-ha-migration-k8s-local" (9m47s remaining to succeed): error creating ForwardingRule "api-ha-migration-k8s-local": googleapi: Error 400: The resource 'projects/k8s-jkns-pr-kubeadm/regions/us-central1/targetPools/api-ha-migration-k8s-local' is not ready, resourceNotReady I0129 16:51:11.767111 6083 executor.go:111] Tasks: 72 done / 91 total; 13 can run I0129 16:51:18.563892 6083 executor.go:111] Tasks: 85 done / 91 total; 6 can run I0129 16:51:32.637181 6083 executor.go:111] Tasks: 91 done / 91 total; 0 can run I0129 16:51:32.714399 6083 update_cluster.go:323] Exporting kubeconfig for cluster kOps has set your kubectl context to ha-migration.k8s.local ... skipping 7 lines ... * read about installing addons at: https://kops.sigs.k8s.io/addons. I0129 16:51:43.104012 5743 up.go:251] /home/prow/go/src/k8s.io/kops/_rundir/bd94eb0b-9ff4-11ed-9a0c-f25a29267a91/kops validate cluster --name ha-migration.k8s.local --count 10 --wait 15m0s I0129 16:51:43.104086 5743 local.go:42] ⚙️ /home/prow/go/src/k8s.io/kops/_rundir/bd94eb0b-9ff4-11ed-9a0c-f25a29267a91/kops validate cluster --name ha-migration.k8s.local --count 10 --wait 15m0s Validating cluster ha-migration.k8s.local W0129 16:52:13.723358 6103 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://34.171.29.170/api/v1/nodes": dial tcp 34.171.29.170:443: i/o timeout W0129 16:52:39.074551 6103 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://34.171.29.170/api/v1/nodes": dial tcp 34.171.29.170:443: connect: connection refused W0129 16:52:49.077988 6103 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://34.171.29.170/api/v1/nodes": dial tcp 34.171.29.170:443: connect: connection refused W0129 16:52:59.087169 6103 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://34.171.29.170/api/v1/nodes": dial tcp 34.171.29.170:443: connect: connection refused W0129 16:53:09.096472 6103 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://34.171.29.170/api/v1/nodes": dial tcp 34.171.29.170:443: connect: connection refused W0129 16:53:19.102264 6103 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://34.171.29.170/api/v1/nodes": dial tcp 34.171.29.170:443: connect: connection refused W0129 16:53:29.111999 6103 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://34.171.29.170/api/v1/nodes": dial tcp 34.171.29.170:443: connect: connection refused W0129 16:53:39.115428 6103 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://34.171.29.170/api/v1/nodes": dial tcp 34.171.29.170:443: connect: connection refused W0129 16:53:59.621340 6103 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://34.171.29.170/api/v1/nodes": dial tcp 34.171.29.170:443: connect: connection refused - error from a previous attempt: read tcp 10.60.101.175:54048->34.171.29.170:443: read: connection reset by peer W0129 16:54:09.631003 6103 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://34.171.29.170/api/v1/nodes": dial tcp 34.171.29.170:443: connect: connection refused W0129 16:54:29.646920 6103 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://34.171.29.170/api/v1/nodes": net/http: TLS handshake timeout W0129 16:54:39.656269 6103 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://34.171.29.170/api/v1/nodes": dial tcp 34.171.29.170:443: connect: connection refused W0129 16:54:55.328665 6103 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://34.171.29.170/api/v1/nodes": dial tcp 34.171.29.170:443: connect: connection refused - error from a previous attempt: read tcp 10.60.101.175:34892->34.171.29.170:443: read: connection reset by peer W0129 16:55:05.332108 6103 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://34.171.29.170/api/v1/nodes": dial tcp 34.171.29.170:443: connect: connection refused I0129 16:55:12.901897 5743 boskos.go:86] Sending heartbeat to Boskos W0129 16:55:15.337525 6103 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://34.171.29.170/api/v1/nodes": dial tcp 34.171.29.170:443: connect: connection refused W0129 16:55:35.341421 6103 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://34.171.29.170/api/v1/nodes": net/http: TLS handshake timeout W0129 16:55:45.345919 6103 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://34.171.29.170/api/v1/nodes": dial tcp 34.171.29.170:443: connect: connection refused W0129 16:56:05.352357 6103 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://34.171.29.170/api/v1/nodes": net/http: TLS handshake timeout I0129 16:56:21.631082 6103 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-c ControlPlane e2-standard-2 1 1 us-central1 ... skipping 9 lines ... KIND NAME MESSAGE Machine https://www.googleapis.com/compute/v1/projects/k8s-jkns-pr-kubeadm/zones/us-central1-a/instances/control-plane-us-central1-a-jvxb machine "https://www.googleapis.com/compute/v1/projects/k8s-jkns-pr-kubeadm/zones/us-central1-a/instances/control-plane-us-central1-a-jvxb" has not yet joined cluster Machine https://www.googleapis.com/compute/v1/projects/k8s-jkns-pr-kubeadm/zones/us-central1-a/instances/nodes-us-central1-a-2667 machine "https://www.googleapis.com/compute/v1/projects/k8s-jkns-pr-kubeadm/zones/us-central1-a/instances/nodes-us-central1-a-2667" has not yet joined cluster Machine https://www.googleapis.com/compute/v1/projects/k8s-jkns-pr-kubeadm/zones/us-central1-b/instances/control-plane-us-central1-b-qrmf machine "https://www.googleapis.com/compute/v1/projects/k8s-jkns-pr-kubeadm/zones/us-central1-b/instances/control-plane-us-central1-b-qrmf" has not yet joined cluster Node control-plane-us-central1-c-bgbm node "control-plane-us-central1-c-bgbm" of role "control-plane" is not ready Validation Failed W0129 16:56:22.531125 6103 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 16:56:32.926580 6103 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 13 lines ... Machine https://www.googleapis.com/compute/v1/projects/k8s-jkns-pr-kubeadm/zones/us-central1-a/instances/nodes-us-central1-a-2667 machine "https://www.googleapis.com/compute/v1/projects/k8s-jkns-pr-kubeadm/zones/us-central1-a/instances/nodes-us-central1-a-2667" has not yet joined cluster Node control-plane-us-central1-a-jvxb node "control-plane-us-central1-a-jvxb" of role "control-plane" is not ready Node control-plane-us-central1-b-qrmf node "control-plane-us-central1-b-qrmf" of role "control-plane" is not ready Node control-plane-us-central1-c-bgbm node "control-plane-us-central1-c-bgbm" of role "control-plane" is not ready Pod kube-system/kube-controller-manager-control-plane-us-central1-c-bgbm system-cluster-critical pod "kube-controller-manager-control-plane-us-central1-c-bgbm" is not ready (kube-controller-manager) Validation Failed W0129 16:56:33.830624 6103 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 16:56:44.185598 6103 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 14 lines ... Node control-plane-us-central1-a-jvxb node "control-plane-us-central1-a-jvxb" of role "control-plane" is not ready Node control-plane-us-central1-b-qrmf node "control-plane-us-central1-b-qrmf" of role "control-plane" is not ready Node control-plane-us-central1-c-bgbm node "control-plane-us-central1-c-bgbm" of role "control-plane" is not ready Pod kube-system/etcd-manager-main-control-plane-us-central1-c-bgbm system-cluster-critical pod "etcd-manager-main-control-plane-us-central1-c-bgbm" is pending Pod kube-system/kube-controller-manager-control-plane-us-central1-c-bgbm system-cluster-critical pod "kube-controller-manager-control-plane-us-central1-c-bgbm" is not ready (kube-controller-manager) Validation Failed W0129 16:56:45.075614 6103 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 16:56:55.481872 6103 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 13 lines ... Machine https://www.googleapis.com/compute/v1/projects/k8s-jkns-pr-kubeadm/zones/us-central1-a/instances/nodes-us-central1-a-2667 machine "https://www.googleapis.com/compute/v1/projects/k8s-jkns-pr-kubeadm/zones/us-central1-a/instances/nodes-us-central1-a-2667" has not yet joined cluster Node control-plane-us-central1-a-jvxb node "control-plane-us-central1-a-jvxb" of role "control-plane" is not ready Node control-plane-us-central1-b-qrmf node "control-plane-us-central1-b-qrmf" of role "control-plane" is not ready Node control-plane-us-central1-c-bgbm node "control-plane-us-central1-c-bgbm" of role "control-plane" is not ready Pod kube-system/kube-controller-manager-control-plane-us-central1-c-bgbm system-cluster-critical pod "kube-controller-manager-control-plane-us-central1-c-bgbm" is not ready (kube-controller-manager) Validation Failed W0129 16:56:56.371523 6103 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 16:57:06.816818 6103 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 14 lines ... Node control-plane-us-central1-a-jvxb node "control-plane-us-central1-a-jvxb" of role "control-plane" is not ready Node control-plane-us-central1-b-qrmf node "control-plane-us-central1-b-qrmf" of role "control-plane" is not ready Node control-plane-us-central1-c-bgbm node "control-plane-us-central1-c-bgbm" of role "control-plane" is not ready Pod kube-system/etcd-manager-events-control-plane-us-central1-c-bgbm system-cluster-critical pod "etcd-manager-events-control-plane-us-central1-c-bgbm" is pending Pod kube-system/kube-controller-manager-control-plane-us-central1-c-bgbm system-cluster-critical pod "kube-controller-manager-control-plane-us-central1-c-bgbm" is not ready (kube-controller-manager) Validation Failed W0129 16:57:07.691190 6103 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 16:57:18.155978 6103 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 16 lines ... Node control-plane-us-central1-c-bgbm node "control-plane-us-central1-c-bgbm" of role "control-plane" is not ready Pod kube-system/kube-apiserver-control-plane-us-central1-a-jvxb system-cluster-critical pod "kube-apiserver-control-plane-us-central1-a-jvxb" is pending Pod kube-system/kube-controller-manager-control-plane-us-central1-a-jvxb system-cluster-critical pod "kube-controller-manager-control-plane-us-central1-a-jvxb" is pending Pod kube-system/kube-controller-manager-control-plane-us-central1-b-qrmf system-cluster-critical pod "kube-controller-manager-control-plane-us-central1-b-qrmf" is pending Pod kube-system/kube-controller-manager-control-plane-us-central1-c-bgbm system-cluster-critical pod "kube-controller-manager-control-plane-us-central1-c-bgbm" is not ready (kube-controller-manager) Validation Failed W0129 16:57:18.980098 6103 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 16:57:29.416537 6103 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 18 lines ... Pod kube-system/etcd-manager-main-control-plane-us-central1-b-qrmf system-cluster-critical pod "etcd-manager-main-control-plane-us-central1-b-qrmf" is pending Pod kube-system/kube-controller-manager-control-plane-us-central1-a-jvxb system-cluster-critical pod "kube-controller-manager-control-plane-us-central1-a-jvxb" is not ready (kube-controller-manager) Pod kube-system/kube-controller-manager-control-plane-us-central1-b-qrmf system-cluster-critical pod "kube-controller-manager-control-plane-us-central1-b-qrmf" is not ready (kube-controller-manager) Pod kube-system/kube-controller-manager-control-plane-us-central1-c-bgbm system-cluster-critical pod "kube-controller-manager-control-plane-us-central1-c-bgbm" is not ready (kube-controller-manager) Pod kube-system/kube-scheduler-control-plane-us-central1-a-jvxb system-cluster-critical pod "kube-scheduler-control-plane-us-central1-a-jvxb" is pending Validation Failed W0129 16:57:30.292631 6103 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 16:57:40.745441 6103 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-qrmf node "control-plane-us-central1-b-qrmf" of role "control-plane" is not ready Node control-plane-us-central1-c-bgbm node "control-plane-us-central1-c-bgbm" of role "control-plane" is not ready Pod kube-system/kube-controller-manager-control-plane-us-central1-a-jvxb system-cluster-critical pod "kube-controller-manager-control-plane-us-central1-a-jvxb" is not ready (kube-controller-manager) Pod kube-system/kube-controller-manager-control-plane-us-central1-b-qrmf system-cluster-critical pod "kube-controller-manager-control-plane-us-central1-b-qrmf" is not ready (kube-controller-manager) Pod kube-system/kube-controller-manager-control-plane-us-central1-c-bgbm system-cluster-critical pod "kube-controller-manager-control-plane-us-central1-c-bgbm" is not ready (kube-controller-manager) Validation Failed W0129 16:57:41.732679 6103 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 16:57:52.114135 6103 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-qrmf node "control-plane-us-central1-b-qrmf" of role "control-plane" is not ready Node control-plane-us-central1-c-bgbm node "control-plane-us-central1-c-bgbm" of role "control-plane" is not ready Pod kube-system/kube-apiserver-control-plane-us-central1-b-qrmf system-cluster-critical pod "kube-apiserver-control-plane-us-central1-b-qrmf" is pending Pod kube-system/kube-controller-manager-control-plane-us-central1-b-qrmf system-cluster-critical pod "kube-controller-manager-control-plane-us-central1-b-qrmf" is not ready (kube-controller-manager) Pod kube-system/kube-controller-manager-control-plane-us-central1-c-bgbm system-cluster-critical pod "kube-controller-manager-control-plane-us-central1-c-bgbm" is not ready (kube-controller-manager) Validation Failed W0129 16:57:53.050189 6103 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 16:58:03.463753 6103 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 22 lines ... Pod kube-system/coredns-5d479c4f64-bg4qz system-cluster-critical pod "coredns-5d479c4f64-bg4qz" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-nrghx system-cluster-critical pod "coredns-autoscaler-557ccb4c66-nrghx" is pending Pod kube-system/dns-controller-6f69f47785-xxrdt system-cluster-critical pod "dns-controller-6f69f47785-xxrdt" is pending Pod kube-system/etcd-manager-events-control-plane-us-central1-a-jvxb system-cluster-critical pod "etcd-manager-events-control-plane-us-central1-a-jvxb" is pending Pod kube-system/kube-controller-manager-control-plane-us-central1-b-qrmf system-cluster-critical pod "kube-controller-manager-control-plane-us-central1-b-qrmf" is not ready (kube-controller-manager) Validation Failed W0129 16:58:04.373762 6103 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 16:58:14.748758 6103 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 19 lines ... Pod kube-system/cilium-wvlj4 system-node-critical pod "cilium-wvlj4" is not ready (cilium-agent) Pod kube-system/coredns-5d479c4f64-bg4qz system-cluster-critical pod "coredns-5d479c4f64-bg4qz" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-nrghx system-cluster-critical pod "coredns-autoscaler-557ccb4c66-nrghx" is pending Pod kube-system/dns-controller-6f69f47785-xxrdt system-cluster-critical pod "dns-controller-6f69f47785-xxrdt" is pending Pod kube-system/kube-controller-manager-control-plane-us-central1-b-qrmf system-cluster-critical pod "kube-controller-manager-control-plane-us-central1-b-qrmf" is not ready (kube-controller-manager) Validation Failed W0129 16:58:15.613456 6103 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 16:58:26.114243 6103 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 19 lines ... Pod kube-system/cilium-wvlj4 system-node-critical pod "cilium-wvlj4" is not ready (cilium-agent) Pod kube-system/coredns-5d479c4f64-bg4qz system-cluster-critical pod "coredns-5d479c4f64-bg4qz" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-nrghx system-cluster-critical pod "coredns-autoscaler-557ccb4c66-nrghx" is pending Pod kube-system/dns-controller-6f69f47785-xxrdt system-cluster-critical pod "dns-controller-6f69f47785-xxrdt" is pending Pod kube-system/kube-controller-manager-control-plane-us-central1-b-qrmf system-cluster-critical pod "kube-controller-manager-control-plane-us-central1-b-qrmf" is not ready (kube-controller-manager) Validation Failed W0129 16:58:27.013341 6103 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 16:58:37.417297 6103 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 17 lines ... Pod kube-system/cilium-5qhkh system-node-critical pod "cilium-5qhkh" is not ready (cilium-agent) Pod kube-system/coredns-5d479c4f64-bg4qz system-cluster-critical pod "coredns-5d479c4f64-bg4qz" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-nrghx system-cluster-critical pod "coredns-autoscaler-557ccb4c66-nrghx" is pending Pod kube-system/dns-controller-6f69f47785-xxrdt system-cluster-critical pod "dns-controller-6f69f47785-xxrdt" is pending Pod kube-system/kube-controller-manager-control-plane-us-central1-b-qrmf system-cluster-critical pod "kube-controller-manager-control-plane-us-central1-b-qrmf" is not ready (kube-controller-manager) Validation Failed W0129 16:58:38.253155 6103 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 16:58:48.629716 6103 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 16 lines ... Node control-plane-us-central1-c-bgbm node "control-plane-us-central1-c-bgbm" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-bg4qz system-cluster-critical pod "coredns-5d479c4f64-bg4qz" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-nrghx system-cluster-critical pod "coredns-autoscaler-557ccb4c66-nrghx" is pending Pod kube-system/dns-controller-6f69f47785-xxrdt system-cluster-critical pod "dns-controller-6f69f47785-xxrdt" is pending Pod kube-system/kube-controller-manager-control-plane-us-central1-b-qrmf system-cluster-critical pod "kube-controller-manager-control-plane-us-central1-b-qrmf" is not ready (kube-controller-manager) Validation Failed W0129 16:58:49.493348 6103 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 16:58:59.903963 6103 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 16 lines ... Node control-plane-us-central1-c-bgbm node "control-plane-us-central1-c-bgbm" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-bg4qz system-cluster-critical pod "coredns-5d479c4f64-bg4qz" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-nrghx system-cluster-critical pod "coredns-autoscaler-557ccb4c66-nrghx" is pending Pod kube-system/dns-controller-6f69f47785-xxrdt system-cluster-critical pod "dns-controller-6f69f47785-xxrdt" is pending Pod kube-system/kube-controller-manager-control-plane-us-central1-b-qrmf system-cluster-critical pod "kube-controller-manager-control-plane-us-central1-b-qrmf" is not ready (kube-controller-manager) Validation Failed W0129 16:59:00.800072 6103 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 16:59:11.190232 6103 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-qrmf node "control-plane-us-central1-b-qrmf" of role "control-plane" is not ready Node control-plane-us-central1-c-bgbm node "control-plane-us-central1-c-bgbm" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-bg4qz system-cluster-critical pod "coredns-5d479c4f64-bg4qz" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-nrghx system-cluster-critical pod "coredns-autoscaler-557ccb4c66-nrghx" is pending Pod kube-system/dns-controller-6f69f47785-xxrdt system-cluster-critical pod "dns-controller-6f69f47785-xxrdt" is pending Validation Failed W0129 16:59:12.059951 6103 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 16:59:22.406863 6103 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-qrmf node "control-plane-us-central1-b-qrmf" of role "control-plane" is not ready Node control-plane-us-central1-c-bgbm node "control-plane-us-central1-c-bgbm" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-bg4qz system-cluster-critical pod "coredns-5d479c4f64-bg4qz" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-nrghx system-cluster-critical pod "coredns-autoscaler-557ccb4c66-nrghx" is pending Pod kube-system/dns-controller-6f69f47785-xxrdt system-cluster-critical pod "dns-controller-6f69f47785-xxrdt" is pending Validation Failed W0129 16:59:23.223057 6103 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 16:59:33.649265 6103 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-qrmf node "control-plane-us-central1-b-qrmf" of role "control-plane" is not ready Node control-plane-us-central1-c-bgbm node "control-plane-us-central1-c-bgbm" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-bg4qz system-cluster-critical pod "coredns-5d479c4f64-bg4qz" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-nrghx system-cluster-critical pod "coredns-autoscaler-557ccb4c66-nrghx" is pending Pod kube-system/dns-controller-6f69f47785-xxrdt system-cluster-critical pod "dns-controller-6f69f47785-xxrdt" is pending Validation Failed W0129 16:59:34.482902 6103 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 16:59:44.944278 6103 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-qrmf node "control-plane-us-central1-b-qrmf" of role "control-plane" is not ready Node control-plane-us-central1-c-bgbm node "control-plane-us-central1-c-bgbm" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-bg4qz system-cluster-critical pod "coredns-5d479c4f64-bg4qz" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-nrghx system-cluster-critical pod "coredns-autoscaler-557ccb4c66-nrghx" is pending Pod kube-system/dns-controller-6f69f47785-xxrdt system-cluster-critical pod "dns-controller-6f69f47785-xxrdt" is pending Validation Failed W0129 16:59:45.794982 6103 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 16:59:56.272193 6103 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-qrmf node "control-plane-us-central1-b-qrmf" of role "control-plane" is not ready Node control-plane-us-central1-c-bgbm node "control-plane-us-central1-c-bgbm" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-bg4qz system-cluster-critical pod "coredns-5d479c4f64-bg4qz" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-nrghx system-cluster-critical pod "coredns-autoscaler-557ccb4c66-nrghx" is pending Pod kube-system/dns-controller-6f69f47785-xxrdt system-cluster-critical pod "dns-controller-6f69f47785-xxrdt" is pending Validation Failed W0129 16:59:57.176188 6103 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 17:00:07.661778 6103 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-qrmf node "control-plane-us-central1-b-qrmf" of role "control-plane" is not ready Node control-plane-us-central1-c-bgbm node "control-plane-us-central1-c-bgbm" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-bg4qz system-cluster-critical pod "coredns-5d479c4f64-bg4qz" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-nrghx system-cluster-critical pod "coredns-autoscaler-557ccb4c66-nrghx" is pending Pod kube-system/dns-controller-6f69f47785-xxrdt system-cluster-critical pod "dns-controller-6f69f47785-xxrdt" is pending Validation Failed W0129 17:00:08.628739 6103 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 17:00:12.912625 5743 boskos.go:86] Sending heartbeat to Boskos I0129 17:00:19.042011 6103 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 ... skipping 16 lines ... Node control-plane-us-central1-b-qrmf node "control-plane-us-central1-b-qrmf" of role "control-plane" is not ready Node control-plane-us-central1-c-bgbm node "control-plane-us-central1-c-bgbm" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-bg4qz system-cluster-critical pod "coredns-5d479c4f64-bg4qz" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-nrghx system-cluster-critical pod "coredns-autoscaler-557ccb4c66-nrghx" is pending Pod kube-system/dns-controller-6f69f47785-xxrdt system-cluster-critical pod "dns-controller-6f69f47785-xxrdt" is pending Validation Failed W0129 17:00:19.940829 6103 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 17:00:30.476685 6103 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-qrmf node "control-plane-us-central1-b-qrmf" of role "control-plane" is not ready Node control-plane-us-central1-c-bgbm node "control-plane-us-central1-c-bgbm" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-bg4qz system-cluster-critical pod "coredns-5d479c4f64-bg4qz" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-nrghx system-cluster-critical pod "coredns-autoscaler-557ccb4c66-nrghx" is pending Pod kube-system/dns-controller-6f69f47785-xxrdt system-cluster-critical pod "dns-controller-6f69f47785-xxrdt" is pending Validation Failed W0129 17:00:31.478726 6103 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 17:00:41.885869 6103 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-qrmf node "control-plane-us-central1-b-qrmf" of role "control-plane" is not ready Node control-plane-us-central1-c-bgbm node "control-plane-us-central1-c-bgbm" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-bg4qz system-cluster-critical pod "coredns-5d479c4f64-bg4qz" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-nrghx system-cluster-critical pod "coredns-autoscaler-557ccb4c66-nrghx" is pending Pod kube-system/dns-controller-6f69f47785-xxrdt system-cluster-critical pod "dns-controller-6f69f47785-xxrdt" is pending Validation Failed W0129 17:00:42.777114 6103 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 17:00:53.244314 6103 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-qrmf node "control-plane-us-central1-b-qrmf" of role "control-plane" is not ready Node control-plane-us-central1-c-bgbm node "control-plane-us-central1-c-bgbm" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-bg4qz system-cluster-critical pod "coredns-5d479c4f64-bg4qz" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-nrghx system-cluster-critical pod "coredns-autoscaler-557ccb4c66-nrghx" is pending Pod kube-system/dns-controller-6f69f47785-xxrdt system-cluster-critical pod "dns-controller-6f69f47785-xxrdt" is pending Validation Failed W0129 17:00:54.137640 6103 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 17:01:04.536341 6103 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-qrmf node "control-plane-us-central1-b-qrmf" of role "control-plane" is not ready Node control-plane-us-central1-c-bgbm node "control-plane-us-central1-c-bgbm" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-bg4qz system-cluster-critical pod "coredns-5d479c4f64-bg4qz" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-nrghx system-cluster-critical pod "coredns-autoscaler-557ccb4c66-nrghx" is pending Pod kube-system/dns-controller-6f69f47785-xxrdt system-cluster-critical pod "dns-controller-6f69f47785-xxrdt" is pending Validation Failed W0129 17:01:05.421608 6103 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 17:01:15.867658 6103 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-qrmf node "control-plane-us-central1-b-qrmf" of role "control-plane" is not ready Node control-plane-us-central1-c-bgbm node "control-plane-us-central1-c-bgbm" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-bg4qz system-cluster-critical pod "coredns-5d479c4f64-bg4qz" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-nrghx system-cluster-critical pod "coredns-autoscaler-557ccb4c66-nrghx" is pending Pod kube-system/dns-controller-6f69f47785-xxrdt system-cluster-critical pod "dns-controller-6f69f47785-xxrdt" is pending Validation Failed W0129 17:01:16.708144 6103 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 17:01:27.168932 6103 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-qrmf node "control-plane-us-central1-b-qrmf" of role "control-plane" is not ready Node control-plane-us-central1-c-bgbm node "control-plane-us-central1-c-bgbm" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-bg4qz system-cluster-critical pod "coredns-5d479c4f64-bg4qz" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-nrghx system-cluster-critical pod "coredns-autoscaler-557ccb4c66-nrghx" is pending Pod kube-system/dns-controller-6f69f47785-xxrdt system-cluster-critical pod "dns-controller-6f69f47785-xxrdt" is pending Validation Failed W0129 17:01:28.053197 6103 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 17:01:38.497848 6103 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-qrmf node "control-plane-us-central1-b-qrmf" of role "control-plane" is not ready Node control-plane-us-central1-c-bgbm node "control-plane-us-central1-c-bgbm" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-bg4qz system-cluster-critical pod "coredns-5d479c4f64-bg4qz" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-nrghx system-cluster-critical pod "coredns-autoscaler-557ccb4c66-nrghx" is pending Pod kube-system/dns-controller-6f69f47785-xxrdt system-cluster-critical pod "dns-controller-6f69f47785-xxrdt" is pending Validation Failed W0129 17:01:39.374937 6103 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 17:01:49.732167 6103 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-qrmf node "control-plane-us-central1-b-qrmf" of role "control-plane" is not ready Node control-plane-us-central1-c-bgbm node "control-plane-us-central1-c-bgbm" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-bg4qz system-cluster-critical pod "coredns-5d479c4f64-bg4qz" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-nrghx system-cluster-critical pod "coredns-autoscaler-557ccb4c66-nrghx" is pending Pod kube-system/dns-controller-6f69f47785-xxrdt system-cluster-critical pod "dns-controller-6f69f47785-xxrdt" is pending Validation Failed W0129 17:01:50.560003 6103 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 17:02:00.955473 6103 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-qrmf node "control-plane-us-central1-b-qrmf" of role "control-plane" is not ready Node control-plane-us-central1-c-bgbm node "control-plane-us-central1-c-bgbm" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-bg4qz system-cluster-critical pod "coredns-5d479c4f64-bg4qz" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-nrghx system-cluster-critical pod "coredns-autoscaler-557ccb4c66-nrghx" is pending Pod kube-system/dns-controller-6f69f47785-xxrdt system-cluster-critical pod "dns-controller-6f69f47785-xxrdt" is pending Validation Failed W0129 17:02:01.817247 6103 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 17:02:12.290401 6103 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-qrmf node "control-plane-us-central1-b-qrmf" of role "control-plane" is not ready Node control-plane-us-central1-c-bgbm node "control-plane-us-central1-c-bgbm" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-bg4qz system-cluster-critical pod "coredns-5d479c4f64-bg4qz" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-nrghx system-cluster-critical pod "coredns-autoscaler-557ccb4c66-nrghx" is pending Pod kube-system/dns-controller-6f69f47785-xxrdt system-cluster-critical pod "dns-controller-6f69f47785-xxrdt" is pending Validation Failed W0129 17:02:13.091417 6103 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 17:02:23.510169 6103 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-qrmf node "control-plane-us-central1-b-qrmf" of role "control-plane" is not ready Node control-plane-us-central1-c-bgbm node "control-plane-us-central1-c-bgbm" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-bg4qz system-cluster-critical pod "coredns-5d479c4f64-bg4qz" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-nrghx system-cluster-critical pod "coredns-autoscaler-557ccb4c66-nrghx" is pending Pod kube-system/dns-controller-6f69f47785-xxrdt system-cluster-critical pod "dns-controller-6f69f47785-xxrdt" is pending Validation Failed W0129 17:02:24.420940 6103 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 17:02:34.818633 6103 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-qrmf node "control-plane-us-central1-b-qrmf" of role "control-plane" is not ready Node control-plane-us-central1-c-bgbm node "control-plane-us-central1-c-bgbm" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-bg4qz system-cluster-critical pod "coredns-5d479c4f64-bg4qz" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-nrghx system-cluster-critical pod "coredns-autoscaler-557ccb4c66-nrghx" is pending Pod kube-system/dns-controller-6f69f47785-xxrdt system-cluster-critical pod "dns-controller-6f69f47785-xxrdt" is pending Validation Failed W0129 17:02:35.735605 6103 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 17:02:46.109557 6103 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-qrmf node "control-plane-us-central1-b-qrmf" of role "control-plane" is not ready Node control-plane-us-central1-c-bgbm node "control-plane-us-central1-c-bgbm" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-bg4qz system-cluster-critical pod "coredns-5d479c4f64-bg4qz" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-nrghx system-cluster-critical pod "coredns-autoscaler-557ccb4c66-nrghx" is pending Pod kube-system/dns-controller-6f69f47785-xxrdt system-cluster-critical pod "dns-controller-6f69f47785-xxrdt" is pending Validation Failed W0129 17:02:47.008175 6103 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 17:02:57.383142 6103 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-qrmf node "control-plane-us-central1-b-qrmf" of role "control-plane" is not ready Node control-plane-us-central1-c-bgbm node "control-plane-us-central1-c-bgbm" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-bg4qz system-cluster-critical pod "coredns-5d479c4f64-bg4qz" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-nrghx system-cluster-critical pod "coredns-autoscaler-557ccb4c66-nrghx" is pending Pod kube-system/dns-controller-6f69f47785-xxrdt system-cluster-critical pod "dns-controller-6f69f47785-xxrdt" is pending Validation Failed W0129 17:02:58.221312 6103 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 17:03:08.648087 6103 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-qrmf node "control-plane-us-central1-b-qrmf" of role "control-plane" is not ready Node control-plane-us-central1-c-bgbm node "control-plane-us-central1-c-bgbm" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-bg4qz system-cluster-critical pod "coredns-5d479c4f64-bg4qz" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-nrghx system-cluster-critical pod "coredns-autoscaler-557ccb4c66-nrghx" is pending Pod kube-system/dns-controller-6f69f47785-xxrdt system-cluster-critical pod "dns-controller-6f69f47785-xxrdt" is pending Validation Failed W0129 17:03:09.470363 6103 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 17:03:19.904918 6103 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-qrmf node "control-plane-us-central1-b-qrmf" of role "control-plane" is not ready Node control-plane-us-central1-c-bgbm node "control-plane-us-central1-c-bgbm" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-bg4qz system-cluster-critical pod "coredns-5d479c4f64-bg4qz" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-nrghx system-cluster-critical pod "coredns-autoscaler-557ccb4c66-nrghx" is pending Pod kube-system/dns-controller-6f69f47785-xxrdt system-cluster-critical pod "dns-controller-6f69f47785-xxrdt" is pending Validation Failed W0129 17:03:20.787041 6103 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 17:03:31.299297 6103 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-qrmf node "control-plane-us-central1-b-qrmf" of role "control-plane" is not ready Node control-plane-us-central1-c-bgbm node "control-plane-us-central1-c-bgbm" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-bg4qz system-cluster-critical pod "coredns-5d479c4f64-bg4qz" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-nrghx system-cluster-critical pod "coredns-autoscaler-557ccb4c66-nrghx" is pending Pod kube-system/dns-controller-6f69f47785-xxrdt system-cluster-critical pod "dns-controller-6f69f47785-xxrdt" is pending Validation Failed W0129 17:03:32.181013 6103 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 17:03:42.689385 6103 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-qrmf node "control-plane-us-central1-b-qrmf" of role "control-plane" is not ready Node control-plane-us-central1-c-bgbm node "control-plane-us-central1-c-bgbm" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-bg4qz system-cluster-critical pod "coredns-5d479c4f64-bg4qz" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-nrghx system-cluster-critical pod "coredns-autoscaler-557ccb4c66-nrghx" is pending Pod kube-system/dns-controller-6f69f47785-xxrdt system-cluster-critical pod "dns-controller-6f69f47785-xxrdt" is pending Validation Failed W0129 17:03:43.524864 6103 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 17:03:53.998836 6103 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-qrmf node "control-plane-us-central1-b-qrmf" of role "control-plane" is not ready Node control-plane-us-central1-c-bgbm node "control-plane-us-central1-c-bgbm" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-bg4qz system-cluster-critical pod "coredns-5d479c4f64-bg4qz" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-nrghx system-cluster-critical pod "coredns-autoscaler-557ccb4c66-nrghx" is pending Pod kube-system/dns-controller-6f69f47785-xxrdt system-cluster-critical pod "dns-controller-6f69f47785-xxrdt" is pending Validation Failed W0129 17:03:54.893169 6103 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 17:04:05.371157 6103 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-qrmf node "control-plane-us-central1-b-qrmf" of role "control-plane" is not ready Node control-plane-us-central1-c-bgbm node "control-plane-us-central1-c-bgbm" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-bg4qz system-cluster-critical pod "coredns-5d479c4f64-bg4qz" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-nrghx system-cluster-critical pod "coredns-autoscaler-557ccb4c66-nrghx" is pending Pod kube-system/dns-controller-6f69f47785-xxrdt system-cluster-critical pod "dns-controller-6f69f47785-xxrdt" is pending Validation Failed W0129 17:04:06.322779 6103 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 17:04:16.734197 6103 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-qrmf node "control-plane-us-central1-b-qrmf" of role "control-plane" is not ready Node control-plane-us-central1-c-bgbm node "control-plane-us-central1-c-bgbm" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-bg4qz system-cluster-critical pod "coredns-5d479c4f64-bg4qz" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-nrghx system-cluster-critical pod "coredns-autoscaler-557ccb4c66-nrghx" is pending Pod kube-system/dns-controller-6f69f47785-xxrdt system-cluster-critical pod "dns-controller-6f69f47785-xxrdt" is pending Validation Failed W0129 17:04:17.542222 6103 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 17:04:27.936134 6103 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-qrmf node "control-plane-us-central1-b-qrmf" of role "control-plane" is not ready Node control-plane-us-central1-c-bgbm node "control-plane-us-central1-c-bgbm" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-bg4qz system-cluster-critical pod "coredns-5d479c4f64-bg4qz" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-nrghx system-cluster-critical pod "coredns-autoscaler-557ccb4c66-nrghx" is pending Pod kube-system/dns-controller-6f69f47785-xxrdt system-cluster-critical pod "dns-controller-6f69f47785-xxrdt" is pending Validation Failed W0129 17:04:28.828434 6103 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 17:04:39.278676 6103 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-qrmf node "control-plane-us-central1-b-qrmf" of role "control-plane" is not ready Node control-plane-us-central1-c-bgbm node "control-plane-us-central1-c-bgbm" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-bg4qz system-cluster-critical pod "coredns-5d479c4f64-bg4qz" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-nrghx system-cluster-critical pod "coredns-autoscaler-557ccb4c66-nrghx" is pending Pod kube-system/dns-controller-6f69f47785-xxrdt system-cluster-critical pod "dns-controller-6f69f47785-xxrdt" is pending Validation Failed W0129 17:04:40.053609 6103 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 17:04:50.438862 6103 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-qrmf node "control-plane-us-central1-b-qrmf" of role "control-plane" is not ready Node control-plane-us-central1-c-bgbm node "control-plane-us-central1-c-bgbm" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-bg4qz system-cluster-critical pod "coredns-5d479c4f64-bg4qz" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-nrghx system-cluster-critical pod "coredns-autoscaler-557ccb4c66-nrghx" is pending Pod kube-system/dns-controller-6f69f47785-xxrdt system-cluster-critical pod "dns-controller-6f69f47785-xxrdt" is pending Validation Failed W0129 17:04:51.343734 6103 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 17:05:01.740089 6103 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-qrmf node "control-plane-us-central1-b-qrmf" of role "control-plane" is not ready Node control-plane-us-central1-c-bgbm node "control-plane-us-central1-c-bgbm" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-bg4qz system-cluster-critical pod "coredns-5d479c4f64-bg4qz" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-nrghx system-cluster-critical pod "coredns-autoscaler-557ccb4c66-nrghx" is pending Pod kube-system/dns-controller-6f69f47785-xxrdt system-cluster-critical pod "dns-controller-6f69f47785-xxrdt" is pending Validation Failed W0129 17:05:02.654012 6103 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 17:05:12.923914 5743 boskos.go:86] Sending heartbeat to Boskos I0129 17:05:13.148304 6103 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 ... skipping 16 lines ... Node control-plane-us-central1-b-qrmf node "control-plane-us-central1-b-qrmf" of role "control-plane" is not ready Node control-plane-us-central1-c-bgbm node "control-plane-us-central1-c-bgbm" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-bg4qz system-cluster-critical pod "coredns-5d479c4f64-bg4qz" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-nrghx system-cluster-critical pod "coredns-autoscaler-557ccb4c66-nrghx" is pending Pod kube-system/dns-controller-6f69f47785-xxrdt system-cluster-critical pod "dns-controller-6f69f47785-xxrdt" is pending Validation Failed W0129 17:05:14.026071 6103 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 17:05:24.418352 6103 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-qrmf node "control-plane-us-central1-b-qrmf" of role "control-plane" is not ready Node control-plane-us-central1-c-bgbm node "control-plane-us-central1-c-bgbm" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-bg4qz system-cluster-critical pod "coredns-5d479c4f64-bg4qz" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-nrghx system-cluster-critical pod "coredns-autoscaler-557ccb4c66-nrghx" is pending Pod kube-system/dns-controller-6f69f47785-xxrdt system-cluster-critical pod "dns-controller-6f69f47785-xxrdt" is pending Validation Failed W0129 17:05:25.250848 6103 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 17:05:35.575919 6103 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-qrmf node "control-plane-us-central1-b-qrmf" of role "control-plane" is not ready Node control-plane-us-central1-c-bgbm node "control-plane-us-central1-c-bgbm" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-bg4qz system-cluster-critical pod "coredns-5d479c4f64-bg4qz" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-nrghx system-cluster-critical pod "coredns-autoscaler-557ccb4c66-nrghx" is pending Pod kube-system/dns-controller-6f69f47785-xxrdt system-cluster-critical pod "dns-controller-6f69f47785-xxrdt" is pending Validation Failed W0129 17:05:36.498625 6103 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 17:05:46.894077 6103 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-qrmf node "control-plane-us-central1-b-qrmf" of role "control-plane" is not ready Node control-plane-us-central1-c-bgbm node "control-plane-us-central1-c-bgbm" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-bg4qz system-cluster-critical pod "coredns-5d479c4f64-bg4qz" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-nrghx system-cluster-critical pod "coredns-autoscaler-557ccb4c66-nrghx" is pending Pod kube-system/dns-controller-6f69f47785-xxrdt system-cluster-critical pod "dns-controller-6f69f47785-xxrdt" is pending Validation Failed W0129 17:05:47.780711 6103 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 17:05:58.263279 6103 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-qrmf node "control-plane-us-central1-b-qrmf" of role "control-plane" is not ready Node control-plane-us-central1-c-bgbm node "control-plane-us-central1-c-bgbm" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-bg4qz system-cluster-critical pod "coredns-5d479c4f64-bg4qz" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-nrghx system-cluster-critical pod "coredns-autoscaler-557ccb4c66-nrghx" is pending Pod kube-system/dns-controller-6f69f47785-xxrdt system-cluster-critical pod "dns-controller-6f69f47785-xxrdt" is pending Validation Failed W0129 17:05:59.141416 6103 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 17:06:09.596081 6103 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-qrmf node "control-plane-us-central1-b-qrmf" of role "control-plane" is not ready Node control-plane-us-central1-c-bgbm node "control-plane-us-central1-c-bgbm" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-bg4qz system-cluster-critical pod "coredns-5d479c4f64-bg4qz" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-nrghx system-cluster-critical pod "coredns-autoscaler-557ccb4c66-nrghx" is pending Pod kube-system/dns-controller-6f69f47785-xxrdt system-cluster-critical pod "dns-controller-6f69f47785-xxrdt" is pending Validation Failed W0129 17:06:10.489126 6103 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 17:06:20.932735 6103 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-qrmf node "control-plane-us-central1-b-qrmf" of role "control-plane" is not ready Node control-plane-us-central1-c-bgbm node "control-plane-us-central1-c-bgbm" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-bg4qz system-cluster-critical pod "coredns-5d479c4f64-bg4qz" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-nrghx system-cluster-critical pod "coredns-autoscaler-557ccb4c66-nrghx" is pending Pod kube-system/dns-controller-6f69f47785-xxrdt system-cluster-critical pod "dns-controller-6f69f47785-xxrdt" is pending Validation Failed W0129 17:06:21.833250 6103 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 17:06:32.271311 6103 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-qrmf node "control-plane-us-central1-b-qrmf" of role "control-plane" is not ready Node control-plane-us-central1-c-bgbm node "control-plane-us-central1-c-bgbm" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-bg4qz system-cluster-critical pod "coredns-5d479c4f64-bg4qz" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-nrghx system-cluster-critical pod "coredns-autoscaler-557ccb4c66-nrghx" is pending Pod kube-system/dns-controller-6f69f47785-xxrdt system-cluster-critical pod "dns-controller-6f69f47785-xxrdt" is pending Validation Failed W0129 17:06:33.153326 6103 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 17:06:43.595755 6103 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-qrmf node "control-plane-us-central1-b-qrmf" of role "control-plane" is not ready Node control-plane-us-central1-c-bgbm node "control-plane-us-central1-c-bgbm" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-bg4qz system-cluster-critical pod "coredns-5d479c4f64-bg4qz" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-nrghx system-cluster-critical pod "coredns-autoscaler-557ccb4c66-nrghx" is pending Pod kube-system/dns-controller-6f69f47785-xxrdt system-cluster-critical pod "dns-controller-6f69f47785-xxrdt" is pending Validation Failed W0129 17:06:44.456509 6103 validate_cluster.go:232] (will retry): cluster not yet healthy Error: validation failed: wait time exceeded during validation I0129 17:06:54.466571 5743 dumplogs.go:45] /home/prow/go/src/k8s.io/kops/_rundir/bd94eb0b-9ff4-11ed-9a0c-f25a29267a91/kops toolbox dump --name ha-migration.k8s.local --dir /logs/artifacts --private-key /tmp/kops-ssh540320256/key --ssh-user prow I0129 17:06:54.466652 5743 local.go:42] ⚙️ /home/prow/go/src/k8s.io/kops/_rundir/bd94eb0b-9ff4-11ed-9a0c-f25a29267a91/kops toolbox dump --name ha-migration.k8s.local --dir /logs/artifacts --private-key /tmp/kops-ssh540320256/key --ssh-user prow I0129 17:06:54.951358 6113 gce.go:99] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] 2023/01/29 17:06:59 Dumping node control-plane-us-central1-a-jvxb 2023/01/29 17:07:03 error dumping node control-plane-us-central1-a-jvxb: error executing command "if command -v kubectl &> /dev/null; then kubectl logs -n kube-system --all-containers -l \"k8s-app=external-dns\"; fi": Process exited with status 1 2023/01/29 17:07:03 error dumping node control-plane-us-central1-a-jvxb: error executing command "if command -v kubectl &> /dev/null; then kubectl logs -n kube-system --all-containers -l \"k8s-app=dns-controller\"; fi": Process exited with status 1 2023/01/29 17:07:03 Dumping node control-plane-us-central1-b-qrmf 2023/01/29 17:07:06 error dumping node control-plane-us-central1-b-qrmf: error executing command "if command -v kubectl &> /dev/null; then kubectl logs -n kube-system --all-containers -l \"k8s-app=external-dns\"; fi": Process exited with status 1 2023/01/29 17:07:06 error dumping node control-plane-us-central1-b-qrmf: error executing command "if command -v kubectl &> /dev/null; then kubectl logs -n kube-system --all-containers -l \"k8s-app=dns-controller\"; fi": Process exited with status 1 2023/01/29 17:07:06 Dumping node control-plane-us-central1-c-bgbm 2023/01/29 17:07:09 error dumping node control-plane-us-central1-c-bgbm: error executing command "if command -v kubectl &> /dev/null; then kubectl logs -n kube-system --all-containers -l \"k8s-app=external-dns\"; fi": Process exited with status 1 2023/01/29 17:07:09 error dumping node control-plane-us-central1-c-bgbm: error executing command "if command -v kubectl &> /dev/null; then kubectl logs -n kube-system --all-containers -l \"k8s-app=dns-controller\"; fi": Process exited with status 1 2023/01/29 17:07:09 dumping node not registered in kubernetes: 35.238.149.99 2023/01/29 17:07:09 Dumping node 35.238.149.99 I0129 17:07:11.694625 5743 dumplogs.go:79] /home/prow/go/src/k8s.io/kops/_rundir/bd94eb0b-9ff4-11ed-9a0c-f25a29267a91/kops get cluster --name ha-migration.k8s.local -o yaml I0129 17:07:11.694685 5743 local.go:42] ⚙️ /home/prow/go/src/k8s.io/kops/_rundir/bd94eb0b-9ff4-11ed-9a0c-f25a29267a91/kops get cluster --name ha-migration.k8s.local -o yaml I0129 17:07:11.867452 5743 dumplogs.go:79] /home/prow/go/src/k8s.io/kops/_rundir/bd94eb0b-9ff4-11ed-9a0c-f25a29267a91/kops get instancegroups --name ha-migration.k8s.local -o yaml I0129 17:07:11.867495 5743 local.go:42] ⚙️ /home/prow/go/src/k8s.io/kops/_rundir/bd94eb0b-9ff4-11ed-9a0c-f25a29267a91/kops get instancegroups --name ha-migration.k8s.local -o yaml ... skipping 166 lines ... Deleted cluster: "ha-migration.k8s.local" I0129 17:09:23.019969 5743 gcs.go:88] gsutil -u k8s-jkns-pr-kubeadm rm -r gs://k8s-jkns-pr-kubeadm-state-bd I0129 17:09:23.020034 5743 local.go:42] ⚙️ gsutil -u k8s-jkns-pr-kubeadm rm -r gs://k8s-jkns-pr-kubeadm-state-bd Removing gs://k8s-jkns-pr-kubeadm-state-bd/... I0129 17:09:25.970976 5743 down.go:62] releasing boskos project I0129 17:09:25.980074 5743 boskos.go:83] Boskos heartbeat func received signal to close Error: exit status 1 + EXIT_VALUE=1 + set +o xtrace