e2e-kops-gce-leader-migration #1619648638818979840
Recent runs || View in Spyglass
| Result | FAILURE |
| Tests | 1 failed / 1 succeeded |
| Started | |
| Elapsed | 20m8s |
| Revision | master |
exit status 1
from junit_runner.xml
Filter through log files | View test history on testgrid
kubetest2 Down
... skipping 156 lines ... ..........................................Updated [https://www.googleapis.com/compute/v1/projects/k8s-jkns-e2e-bazel]. .done. WARNING: No host aliases were added to your SSH configs because you do not have any running instances. Try running this command again after running some instances. I0129 10:50:23.240812 5746 up.go:44] Cleaning up any leaked resources from previous cluster I0129 10:50:23.240930 5746 dumplogs.go:45] /home/prow/go/src/k8s.io/kops/_rundir/72d7c71b-9fc2-11ed-9a0c-f25a29267a91/kops toolbox dump --name ha-migration.k8s.local --dir /logs/artifacts --private-key /tmp/kops-ssh786086317/key --ssh-user prow I0129 10:50:23.240945 5746 local.go:42] ⚙️ /home/prow/go/src/k8s.io/kops/_rundir/72d7c71b-9fc2-11ed-9a0c-f25a29267a91/kops toolbox dump --name ha-migration.k8s.local --dir /logs/artifacts --private-key /tmp/kops-ssh786086317/key --ssh-user prow Error: Cluster.kops.k8s.io "ha-migration.k8s.local" not found W0129 10:50:23.593718 5746 down.go:34] Dumping cluster logs at the start of Down() failed: exit status 1 I0129 10:50:23.593755 5746 down.go:48] /home/prow/go/src/k8s.io/kops/_rundir/72d7c71b-9fc2-11ed-9a0c-f25a29267a91/kops delete cluster --name ha-migration.k8s.local --yes I0129 10:50:23.593765 5746 local.go:42] ⚙️ /home/prow/go/src/k8s.io/kops/_rundir/72d7c71b-9fc2-11ed-9a0c-f25a29267a91/kops delete cluster --name ha-migration.k8s.local --yes Error: error reading cluster configuration: Cluster.kops.k8s.io "ha-migration.k8s.local" not found I0129 10:50:23.856593 5746 gcs.go:51] gsutil ls -b -p k8s-jkns-e2e-bazel gs://k8s-jkns-e2e-bazel-state-72 I0129 10:50:23.856637 5746 local.go:42] ⚙️ gsutil ls -b -p k8s-jkns-e2e-bazel gs://k8s-jkns-e2e-bazel-state-72 I0129 10:50:25.919083 5746 gcs.go:70] gsutil mb -p k8s-jkns-e2e-bazel gs://k8s-jkns-e2e-bazel-state-72 I0129 10:50:25.919112 5746 local.go:42] ⚙️ gsutil mb -p k8s-jkns-e2e-bazel gs://k8s-jkns-e2e-bazel-state-72 Creating gs://k8s-jkns-e2e-bazel-state-72/... I0129 10:50:28.585303 5746 http.go:37] curl http://metadata.google.internal/computeMetadata/v1/instance/network-interfaces/0/access-configs/0/external-ip 2023/01/29 10:50:28 failed to get external ip from metadata service: http://metadata.google.internal/computeMetadata/v1/instance/network-interfaces/0/access-configs/0/external-ip returned 404 I0129 10:50:28.600604 5746 http.go:37] curl https://ip.jsb.workers.dev I0129 10:50:28.695399 5746 up.go:167] /home/prow/go/src/k8s.io/kops/_rundir/72d7c71b-9fc2-11ed-9a0c-f25a29267a91/kops create cluster --name ha-migration.k8s.local --cloud gce --kubernetes-version v1.23.16 --ssh-public-key /tmp/kops-ssh786086317/key.pub --set cluster.spec.nodePortAccess=0.0.0.0/0 --channel=alpha --node-count=1 --master-count=3 --zones=us-central1-a,us-central1-b,us-central1-c --master-zones=us-central1-a,us-central1-b,us-central1-c --gce-service-account=default --admin-access 34.123.225.205/32 --master-volume-size 48 --node-volume-size 48 --master-size e2-standard-2 --project k8s-jkns-e2e-bazel I0129 10:50:28.695444 5746 local.go:42] ⚙️ /home/prow/go/src/k8s.io/kops/_rundir/72d7c71b-9fc2-11ed-9a0c-f25a29267a91/kops create cluster --name ha-migration.k8s.local --cloud gce --kubernetes-version v1.23.16 --ssh-public-key /tmp/kops-ssh786086317/key.pub --set cluster.spec.nodePortAccess=0.0.0.0/0 --channel=alpha --node-count=1 --master-count=3 --zones=us-central1-a,us-central1-b,us-central1-c --master-zones=us-central1-a,us-central1-b,us-central1-c --gce-service-account=default --admin-access 34.123.225.205/32 --master-volume-size 48 --node-volume-size 48 --master-size e2-standard-2 --project k8s-jkns-e2e-bazel I0129 10:50:28.746756 6065 create_cluster.go:882] Using SSH public key: /tmp/kops-ssh786086317/key.pub I0129 10:50:29.023761 6065 new_cluster.go:573] VMs will be configured to use specified Service Account: default I0129 10:50:29.023901 6065 new_cluster.go:1338] Cloud Provider ID: "gce" ... skipping 527 lines ... I0129 10:50:40.587837 6084 keypair.go:226] Issuing new certificate: "apiserver-aggregator-ca" I0129 10:50:40.587954 6084 keypair.go:226] Issuing new certificate: "etcd-peers-ca-events" W0129 10:50:40.624548 6084 vfs_keystorereader.go:143] CA private key was not found I0129 10:50:40.778589 6084 keypair.go:226] Issuing new certificate: "kubernetes-ca" I0129 10:50:40.778636 6084 keypair.go:226] Issuing new certificate: "service-account" I0129 10:50:52.837063 6084 executor.go:111] Tasks: 50 done / 91 total; 23 can run W0129 10:51:02.543766 6084 executor.go:139] error running task "ForwardingRule/api-ha-migration-k8s-local" (9m50s remaining to succeed): error creating ForwardingRule "api-ha-migration-k8s-local": googleapi: Error 400: The resource 'projects/k8s-jkns-e2e-bazel/regions/us-central1/targetPools/api-ha-migration-k8s-local' is not ready, resourceNotReady I0129 10:51:02.543996 6084 executor.go:111] Tasks: 72 done / 91 total; 13 can run I0129 10:51:09.335526 6084 executor.go:111] Tasks: 85 done / 91 total; 6 can run I0129 10:51:23.114164 6084 executor.go:111] Tasks: 91 done / 91 total; 0 can run I0129 10:51:23.186534 6084 update_cluster.go:323] Exporting kubeconfig for cluster kOps has set your kubectl context to ha-migration.k8s.local ... skipping 7 lines ... * read about installing addons at: https://kops.sigs.k8s.io/addons. I0129 10:51:33.582032 5746 up.go:251] /home/prow/go/src/k8s.io/kops/_rundir/72d7c71b-9fc2-11ed-9a0c-f25a29267a91/kops validate cluster --name ha-migration.k8s.local --count 10 --wait 15m0s I0129 10:51:33.582082 5746 local.go:42] ⚙️ /home/prow/go/src/k8s.io/kops/_rundir/72d7c71b-9fc2-11ed-9a0c-f25a29267a91/kops validate cluster --name ha-migration.k8s.local --count 10 --wait 15m0s Validating cluster ha-migration.k8s.local W0129 10:52:04.092432 6103 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://34.27.236.186/api/v1/nodes": dial tcp 34.27.236.186:443: i/o timeout W0129 10:52:14.095671 6103 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://34.27.236.186/api/v1/nodes": dial tcp 34.27.236.186:443: connect: connection refused W0129 10:52:24.098328 6103 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://34.27.236.186/api/v1/nodes": dial tcp 34.27.236.186:443: connect: connection refused W0129 10:52:34.101488 6103 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://34.27.236.186/api/v1/nodes": dial tcp 34.27.236.186:443: connect: connection refused W0129 10:52:44.104043 6103 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://34.27.236.186/api/v1/nodes": dial tcp 34.27.236.186:443: connect: connection refused W0129 10:52:54.107173 6103 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://34.27.236.186/api/v1/nodes": dial tcp 34.27.236.186:443: connect: connection refused W0129 10:53:04.110867 6103 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://34.27.236.186/api/v1/nodes": dial tcp 34.27.236.186:443: connect: connection refused W0129 10:53:14.113989 6103 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://34.27.236.186/api/v1/nodes": dial tcp 34.27.236.186:443: connect: connection refused W0129 10:53:24.116892 6103 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://34.27.236.186/api/v1/nodes": dial tcp 34.27.236.186:443: connect: connection refused W0129 10:53:34.120656 6103 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://34.27.236.186/api/v1/nodes": dial tcp 34.27.236.186:443: connect: connection refused W0129 10:53:44.123519 6103 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://34.27.236.186/api/v1/nodes": dial tcp 34.27.236.186:443: connect: connection refused W0129 10:54:03.396430 6103 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://34.27.236.186/api/v1/nodes": dial tcp 34.27.236.186:443: connect: connection refused - error from a previous attempt: read tcp 10.60.198.204:42410->34.27.236.186:443: read: connection reset by peer W0129 10:54:35.465602 6103 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://34.27.236.186/api/v1/nodes": net/http: TLS handshake timeout - error from a previous attempt: read tcp 10.60.198.204:44544->34.27.236.186:443: read: connection reset by peer W0129 10:54:45.469436 6103 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://34.27.236.186/api/v1/nodes": dial tcp 34.27.236.186:443: connect: connection refused W0129 10:55:05.473776 6103 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://34.27.236.186/api/v1/nodes": net/http: TLS handshake timeout I0129 10:55:11.062507 5746 boskos.go:86] Sending heartbeat to Boskos W0129 10:55:15.476466 6103 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://34.27.236.186/api/v1/nodes": dial tcp 34.27.236.186:443: connect: connection refused W0129 10:55:25.480264 6103 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://34.27.236.186/api/v1/nodes": dial tcp 34.27.236.186:443: connect: connection refused W0129 10:55:35.483905 6103 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://34.27.236.186/api/v1/nodes": dial tcp 34.27.236.186:443: connect: connection refused W0129 10:55:45.488293 6103 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://34.27.236.186/api/v1/nodes": dial tcp 34.27.236.186:443: connect: connection refused I0129 10:56:02.940746 6103 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-c ControlPlane e2-standard-2 1 1 us-central1 ... skipping 9 lines ... KIND NAME MESSAGE Machine https://www.googleapis.com/compute/v1/projects/k8s-jkns-e2e-bazel/zones/us-central1-a/instances/control-plane-us-central1-a-z33b machine "https://www.googleapis.com/compute/v1/projects/k8s-jkns-e2e-bazel/zones/us-central1-a/instances/control-plane-us-central1-a-z33b" has not yet joined cluster Machine https://www.googleapis.com/compute/v1/projects/k8s-jkns-e2e-bazel/zones/us-central1-a/instances/nodes-us-central1-a-5xlj machine "https://www.googleapis.com/compute/v1/projects/k8s-jkns-e2e-bazel/zones/us-central1-a/instances/nodes-us-central1-a-5xlj" has not yet joined cluster Machine https://www.googleapis.com/compute/v1/projects/k8s-jkns-e2e-bazel/zones/us-central1-c/instances/control-plane-us-central1-c-4nms machine "https://www.googleapis.com/compute/v1/projects/k8s-jkns-e2e-bazel/zones/us-central1-c/instances/control-plane-us-central1-c-4nms" has not yet joined cluster Node control-plane-us-central1-b-ln27 node "control-plane-us-central1-b-ln27" of role "control-plane" is not ready Validation Failed W0129 10:56:03.838757 6103 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 10:56:14.161983 6103 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 12 lines ... Machine https://www.googleapis.com/compute/v1/projects/k8s-jkns-e2e-bazel/zones/us-central1-a/instances/control-plane-us-central1-a-z33b machine "https://www.googleapis.com/compute/v1/projects/k8s-jkns-e2e-bazel/zones/us-central1-a/instances/control-plane-us-central1-a-z33b" has not yet joined cluster Machine https://www.googleapis.com/compute/v1/projects/k8s-jkns-e2e-bazel/zones/us-central1-a/instances/nodes-us-central1-a-5xlj machine "https://www.googleapis.com/compute/v1/projects/k8s-jkns-e2e-bazel/zones/us-central1-a/instances/nodes-us-central1-a-5xlj" has not yet joined cluster Node control-plane-us-central1-b-ln27 node "control-plane-us-central1-b-ln27" of role "control-plane" is not ready Node control-plane-us-central1-c-4nms node "control-plane-us-central1-c-4nms" of role "control-plane" is not ready Pod kube-system/kube-scheduler-control-plane-us-central1-b-ln27 system-cluster-critical pod "kube-scheduler-control-plane-us-central1-b-ln27" is pending Validation Failed W0129 10:56:15.062594 6103 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 10:56:25.357424 6103 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 13 lines ... Machine https://www.googleapis.com/compute/v1/projects/k8s-jkns-e2e-bazel/zones/us-central1-a/instances/nodes-us-central1-a-5xlj machine "https://www.googleapis.com/compute/v1/projects/k8s-jkns-e2e-bazel/zones/us-central1-a/instances/nodes-us-central1-a-5xlj" has not yet joined cluster Node control-plane-us-central1-b-ln27 node "control-plane-us-central1-b-ln27" of role "control-plane" is not ready Node control-plane-us-central1-c-4nms node "control-plane-us-central1-c-4nms" of role "control-plane" is not ready Pod kube-system/kube-controller-manager-control-plane-us-central1-b-ln27 system-cluster-critical pod "kube-controller-manager-control-plane-us-central1-b-ln27" is not ready (kube-controller-manager) Pod kube-system/kube-controller-manager-control-plane-us-central1-c-4nms system-cluster-critical pod "kube-controller-manager-control-plane-us-central1-c-4nms" is not ready (kube-controller-manager) Validation Failed W0129 10:56:26.232669 6103 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 10:56:36.567794 6103 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-ln27 node "control-plane-us-central1-b-ln27" of role "control-plane" is not ready Node control-plane-us-central1-c-4nms node "control-plane-us-central1-c-4nms" of role "control-plane" is not ready Pod kube-system/etcd-manager-main-control-plane-us-central1-b-ln27 system-cluster-critical pod "etcd-manager-main-control-plane-us-central1-b-ln27" is pending Pod kube-system/kube-controller-manager-control-plane-us-central1-b-ln27 system-cluster-critical pod "kube-controller-manager-control-plane-us-central1-b-ln27" is not ready (kube-controller-manager) Pod kube-system/kube-controller-manager-control-plane-us-central1-c-4nms system-cluster-critical pod "kube-controller-manager-control-plane-us-central1-c-4nms" is not ready (kube-controller-manager) Validation Failed W0129 10:56:37.445780 6103 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 10:56:47.807414 6103 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 16 lines ... Node control-plane-us-central1-c-4nms node "control-plane-us-central1-c-4nms" of role "control-plane" is not ready Pod kube-system/etcd-manager-events-control-plane-us-central1-b-ln27 system-cluster-critical pod "etcd-manager-events-control-plane-us-central1-b-ln27" is pending Pod kube-system/kube-controller-manager-control-plane-us-central1-a-z33b system-cluster-critical pod "kube-controller-manager-control-plane-us-central1-a-z33b" is pending Pod kube-system/kube-controller-manager-control-plane-us-central1-b-ln27 system-cluster-critical pod "kube-controller-manager-control-plane-us-central1-b-ln27" is not ready (kube-controller-manager) Pod kube-system/kube-controller-manager-control-plane-us-central1-c-4nms system-cluster-critical pod "kube-controller-manager-control-plane-us-central1-c-4nms" is not ready (kube-controller-manager) Validation Failed W0129 10:56:48.580511 6103 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 10:56:58.923640 6103 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 13 lines ... Machine https://www.googleapis.com/compute/v1/projects/k8s-jkns-e2e-bazel/zones/us-central1-a/instances/nodes-us-central1-a-5xlj machine "https://www.googleapis.com/compute/v1/projects/k8s-jkns-e2e-bazel/zones/us-central1-a/instances/nodes-us-central1-a-5xlj" has not yet joined cluster Node control-plane-us-central1-a-z33b node "control-plane-us-central1-a-z33b" of role "control-plane" is not ready Node control-plane-us-central1-b-ln27 node "control-plane-us-central1-b-ln27" of role "control-plane" is not ready Node control-plane-us-central1-c-4nms node "control-plane-us-central1-c-4nms" of role "control-plane" is not ready Pod kube-system/kube-controller-manager-control-plane-us-central1-a-z33b system-cluster-critical pod "kube-controller-manager-control-plane-us-central1-a-z33b" is not ready (kube-controller-manager) Validation Failed W0129 10:56:59.707722 6103 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 10:57:10.011953 6103 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 14 lines ... Node control-plane-us-central1-a-z33b node "control-plane-us-central1-a-z33b" of role "control-plane" is not ready Node control-plane-us-central1-b-ln27 node "control-plane-us-central1-b-ln27" of role "control-plane" is not ready Node control-plane-us-central1-c-4nms node "control-plane-us-central1-c-4nms" of role "control-plane" is not ready Pod kube-system/kube-apiserver-control-plane-us-central1-b-ln27 system-cluster-critical pod "kube-apiserver-control-plane-us-central1-b-ln27" is pending Pod kube-system/kube-controller-manager-control-plane-us-central1-a-z33b system-cluster-critical pod "kube-controller-manager-control-plane-us-central1-a-z33b" is not ready (kube-controller-manager) Validation Failed W0129 10:57:10.826017 6103 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 10:57:21.151292 6103 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 19 lines ... Pod kube-system/cilium-operator-b4b4c5b48-czvdj system-cluster-critical pod "cilium-operator-b4b4c5b48-czvdj" is pending Pod kube-system/cilium-v5qn8 system-node-critical pod "cilium-v5qn8" is pending Pod kube-system/coredns-5d479c4f64-gsj9x system-cluster-critical pod "coredns-5d479c4f64-gsj9x" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-rkggr system-cluster-critical pod "coredns-autoscaler-557ccb4c66-rkggr" is pending Pod kube-system/dns-controller-6f69f47785-2sqd5 system-cluster-critical pod "dns-controller-6f69f47785-2sqd5" is pending Validation Failed W0129 10:57:21.970100 6103 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 10:57:32.336430 6103 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 20 lines ... Pod kube-system/coredns-5d479c4f64-gsj9x system-cluster-critical pod "coredns-5d479c4f64-gsj9x" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-rkggr system-cluster-critical pod "coredns-autoscaler-557ccb4c66-rkggr" is pending Pod kube-system/dns-controller-6f69f47785-2sqd5 system-cluster-critical pod "dns-controller-6f69f47785-2sqd5" is pending Pod kube-system/etcd-manager-events-control-plane-us-central1-a-z33b system-cluster-critical pod "etcd-manager-events-control-plane-us-central1-a-z33b" is pending Pod kube-system/kube-scheduler-control-plane-us-central1-a-z33b system-cluster-critical pod "kube-scheduler-control-plane-us-central1-a-z33b" is pending Validation Failed W0129 10:57:33.176233 6103 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 10:57:43.458173 6103 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 18 lines ... Pod kube-system/cilium-m6l2s system-node-critical pod "cilium-m6l2s" is not ready (cilium-agent) Pod kube-system/cilium-v5qn8 system-node-critical pod "cilium-v5qn8" is not ready (cilium-agent) Pod kube-system/coredns-5d479c4f64-gsj9x system-cluster-critical pod "coredns-5d479c4f64-gsj9x" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-rkggr system-cluster-critical pod "coredns-autoscaler-557ccb4c66-rkggr" is pending Pod kube-system/dns-controller-6f69f47785-2sqd5 system-cluster-critical pod "dns-controller-6f69f47785-2sqd5" is pending Validation Failed W0129 10:57:44.346156 6103 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 10:57:54.748766 6103 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 17 lines ... Pod kube-system/cilium-v5qn8 system-node-critical pod "cilium-v5qn8" is not ready (cilium-agent) Pod kube-system/coredns-5d479c4f64-gsj9x system-cluster-critical pod "coredns-5d479c4f64-gsj9x" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-rkggr system-cluster-critical pod "coredns-autoscaler-557ccb4c66-rkggr" is pending Pod kube-system/dns-controller-6f69f47785-2sqd5 system-cluster-critical pod "dns-controller-6f69f47785-2sqd5" is pending Pod kube-system/etcd-manager-main-control-plane-us-central1-a-z33b system-cluster-critical pod "etcd-manager-main-control-plane-us-central1-a-z33b" is pending Validation Failed W0129 10:57:55.568013 6103 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 10:58:05.879744 6103 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-ln27 node "control-plane-us-central1-b-ln27" of role "control-plane" is not ready Node control-plane-us-central1-c-4nms node "control-plane-us-central1-c-4nms" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-gsj9x system-cluster-critical pod "coredns-5d479c4f64-gsj9x" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-rkggr system-cluster-critical pod "coredns-autoscaler-557ccb4c66-rkggr" is pending Pod kube-system/dns-controller-6f69f47785-2sqd5 system-cluster-critical pod "dns-controller-6f69f47785-2sqd5" is pending Validation Failed W0129 10:58:06.718539 6103 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 10:58:17.052000 6103 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-ln27 node "control-plane-us-central1-b-ln27" of role "control-plane" is not ready Node control-plane-us-central1-c-4nms node "control-plane-us-central1-c-4nms" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-gsj9x system-cluster-critical pod "coredns-5d479c4f64-gsj9x" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-rkggr system-cluster-critical pod "coredns-autoscaler-557ccb4c66-rkggr" is pending Pod kube-system/dns-controller-6f69f47785-2sqd5 system-cluster-critical pod "dns-controller-6f69f47785-2sqd5" is pending Validation Failed W0129 10:58:17.937145 6103 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 10:58:28.319539 6103 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-ln27 node "control-plane-us-central1-b-ln27" of role "control-plane" is not ready Node control-plane-us-central1-c-4nms node "control-plane-us-central1-c-4nms" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-gsj9x system-cluster-critical pod "coredns-5d479c4f64-gsj9x" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-rkggr system-cluster-critical pod "coredns-autoscaler-557ccb4c66-rkggr" is pending Pod kube-system/dns-controller-6f69f47785-2sqd5 system-cluster-critical pod "dns-controller-6f69f47785-2sqd5" is pending Validation Failed W0129 10:58:29.125183 6103 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 10:58:39.418609 6103 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-ln27 node "control-plane-us-central1-b-ln27" of role "control-plane" is not ready Node control-plane-us-central1-c-4nms node "control-plane-us-central1-c-4nms" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-gsj9x system-cluster-critical pod "coredns-5d479c4f64-gsj9x" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-rkggr system-cluster-critical pod "coredns-autoscaler-557ccb4c66-rkggr" is pending Pod kube-system/dns-controller-6f69f47785-2sqd5 system-cluster-critical pod "dns-controller-6f69f47785-2sqd5" is pending Validation Failed W0129 10:58:40.266345 6103 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 10:58:50.601372 6103 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-ln27 node "control-plane-us-central1-b-ln27" of role "control-plane" is not ready Node control-plane-us-central1-c-4nms node "control-plane-us-central1-c-4nms" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-gsj9x system-cluster-critical pod "coredns-5d479c4f64-gsj9x" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-rkggr system-cluster-critical pod "coredns-autoscaler-557ccb4c66-rkggr" is pending Pod kube-system/dns-controller-6f69f47785-2sqd5 system-cluster-critical pod "dns-controller-6f69f47785-2sqd5" is pending Validation Failed W0129 10:58:51.403085 6103 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 10:59:01.704189 6103 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-ln27 node "control-plane-us-central1-b-ln27" of role "control-plane" is not ready Node control-plane-us-central1-c-4nms node "control-plane-us-central1-c-4nms" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-gsj9x system-cluster-critical pod "coredns-5d479c4f64-gsj9x" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-rkggr system-cluster-critical pod "coredns-autoscaler-557ccb4c66-rkggr" is pending Pod kube-system/dns-controller-6f69f47785-2sqd5 system-cluster-critical pod "dns-controller-6f69f47785-2sqd5" is pending Validation Failed W0129 10:59:02.484165 6103 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 10:59:12.809376 6103 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-ln27 node "control-plane-us-central1-b-ln27" of role "control-plane" is not ready Node control-plane-us-central1-c-4nms node "control-plane-us-central1-c-4nms" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-gsj9x system-cluster-critical pod "coredns-5d479c4f64-gsj9x" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-rkggr system-cluster-critical pod "coredns-autoscaler-557ccb4c66-rkggr" is pending Pod kube-system/dns-controller-6f69f47785-2sqd5 system-cluster-critical pod "dns-controller-6f69f47785-2sqd5" is pending Validation Failed W0129 10:59:13.657865 6103 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 10:59:24.141609 6103 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-ln27 node "control-plane-us-central1-b-ln27" of role "control-plane" is not ready Node control-plane-us-central1-c-4nms node "control-plane-us-central1-c-4nms" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-gsj9x system-cluster-critical pod "coredns-5d479c4f64-gsj9x" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-rkggr system-cluster-critical pod "coredns-autoscaler-557ccb4c66-rkggr" is pending Pod kube-system/dns-controller-6f69f47785-2sqd5 system-cluster-critical pod "dns-controller-6f69f47785-2sqd5" is pending Validation Failed W0129 10:59:25.127944 6103 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 10:59:35.413465 6103 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-ln27 node "control-plane-us-central1-b-ln27" of role "control-plane" is not ready Node control-plane-us-central1-c-4nms node "control-plane-us-central1-c-4nms" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-gsj9x system-cluster-critical pod "coredns-5d479c4f64-gsj9x" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-rkggr system-cluster-critical pod "coredns-autoscaler-557ccb4c66-rkggr" is pending Pod kube-system/dns-controller-6f69f47785-2sqd5 system-cluster-critical pod "dns-controller-6f69f47785-2sqd5" is pending Validation Failed W0129 10:59:36.263872 6103 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 10:59:46.672758 6103 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-ln27 node "control-plane-us-central1-b-ln27" of role "control-plane" is not ready Node control-plane-us-central1-c-4nms node "control-plane-us-central1-c-4nms" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-gsj9x system-cluster-critical pod "coredns-5d479c4f64-gsj9x" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-rkggr system-cluster-critical pod "coredns-autoscaler-557ccb4c66-rkggr" is pending Pod kube-system/dns-controller-6f69f47785-2sqd5 system-cluster-critical pod "dns-controller-6f69f47785-2sqd5" is pending Validation Failed W0129 10:59:47.531697 6103 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 10:59:57.897001 6103 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-ln27 node "control-plane-us-central1-b-ln27" of role "control-plane" is not ready Node control-plane-us-central1-c-4nms node "control-plane-us-central1-c-4nms" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-gsj9x system-cluster-critical pod "coredns-5d479c4f64-gsj9x" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-rkggr system-cluster-critical pod "coredns-autoscaler-557ccb4c66-rkggr" is pending Pod kube-system/dns-controller-6f69f47785-2sqd5 system-cluster-critical pod "dns-controller-6f69f47785-2sqd5" is pending Validation Failed W0129 10:59:58.735336 6103 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 11:00:09.048570 6103 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-ln27 node "control-plane-us-central1-b-ln27" of role "control-plane" is not ready Node control-plane-us-central1-c-4nms node "control-plane-us-central1-c-4nms" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-gsj9x system-cluster-critical pod "coredns-5d479c4f64-gsj9x" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-rkggr system-cluster-critical pod "coredns-autoscaler-557ccb4c66-rkggr" is pending Pod kube-system/dns-controller-6f69f47785-2sqd5 system-cluster-critical pod "dns-controller-6f69f47785-2sqd5" is pending Validation Failed W0129 11:00:09.819609 6103 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 11:00:11.073336 5746 boskos.go:86] Sending heartbeat to Boskos I0129 11:00:20.140486 6103 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 ... skipping 16 lines ... Node control-plane-us-central1-b-ln27 node "control-plane-us-central1-b-ln27" of role "control-plane" is not ready Node control-plane-us-central1-c-4nms node "control-plane-us-central1-c-4nms" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-gsj9x system-cluster-critical pod "coredns-5d479c4f64-gsj9x" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-rkggr system-cluster-critical pod "coredns-autoscaler-557ccb4c66-rkggr" is pending Pod kube-system/dns-controller-6f69f47785-2sqd5 system-cluster-critical pod "dns-controller-6f69f47785-2sqd5" is pending Validation Failed W0129 11:00:20.951943 6103 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 11:00:31.287249 6103 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-ln27 node "control-plane-us-central1-b-ln27" of role "control-plane" is not ready Node control-plane-us-central1-c-4nms node "control-plane-us-central1-c-4nms" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-gsj9x system-cluster-critical pod "coredns-5d479c4f64-gsj9x" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-rkggr system-cluster-critical pod "coredns-autoscaler-557ccb4c66-rkggr" is pending Pod kube-system/dns-controller-6f69f47785-2sqd5 system-cluster-critical pod "dns-controller-6f69f47785-2sqd5" is pending Validation Failed W0129 11:00:32.082423 6103 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 11:00:42.404130 6103 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-ln27 node "control-plane-us-central1-b-ln27" of role "control-plane" is not ready Node control-plane-us-central1-c-4nms node "control-plane-us-central1-c-4nms" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-gsj9x system-cluster-critical pod "coredns-5d479c4f64-gsj9x" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-rkggr system-cluster-critical pod "coredns-autoscaler-557ccb4c66-rkggr" is pending Pod kube-system/dns-controller-6f69f47785-2sqd5 system-cluster-critical pod "dns-controller-6f69f47785-2sqd5" is pending Validation Failed W0129 11:00:43.192417 6103 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 11:00:53.504572 6103 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-ln27 node "control-plane-us-central1-b-ln27" of role "control-plane" is not ready Node control-plane-us-central1-c-4nms node "control-plane-us-central1-c-4nms" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-gsj9x system-cluster-critical pod "coredns-5d479c4f64-gsj9x" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-rkggr system-cluster-critical pod "coredns-autoscaler-557ccb4c66-rkggr" is pending Pod kube-system/dns-controller-6f69f47785-2sqd5 system-cluster-critical pod "dns-controller-6f69f47785-2sqd5" is pending Validation Failed W0129 11:00:54.350764 6103 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 11:01:04.715555 6103 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-ln27 node "control-plane-us-central1-b-ln27" of role "control-plane" is not ready Node control-plane-us-central1-c-4nms node "control-plane-us-central1-c-4nms" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-gsj9x system-cluster-critical pod "coredns-5d479c4f64-gsj9x" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-rkggr system-cluster-critical pod "coredns-autoscaler-557ccb4c66-rkggr" is pending Pod kube-system/dns-controller-6f69f47785-2sqd5 system-cluster-critical pod "dns-controller-6f69f47785-2sqd5" is pending Validation Failed W0129 11:01:05.576389 6103 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 11:01:15.895131 6103 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-ln27 node "control-plane-us-central1-b-ln27" of role "control-plane" is not ready Node control-plane-us-central1-c-4nms node "control-plane-us-central1-c-4nms" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-gsj9x system-cluster-critical pod "coredns-5d479c4f64-gsj9x" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-rkggr system-cluster-critical pod "coredns-autoscaler-557ccb4c66-rkggr" is pending Pod kube-system/dns-controller-6f69f47785-2sqd5 system-cluster-critical pod "dns-controller-6f69f47785-2sqd5" is pending Validation Failed W0129 11:01:16.667221 6103 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 11:01:26.999076 6103 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-ln27 node "control-plane-us-central1-b-ln27" of role "control-plane" is not ready Node control-plane-us-central1-c-4nms node "control-plane-us-central1-c-4nms" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-gsj9x system-cluster-critical pod "coredns-5d479c4f64-gsj9x" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-rkggr system-cluster-critical pod "coredns-autoscaler-557ccb4c66-rkggr" is pending Pod kube-system/dns-controller-6f69f47785-2sqd5 system-cluster-critical pod "dns-controller-6f69f47785-2sqd5" is pending Validation Failed W0129 11:01:27.821755 6103 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 11:01:38.272155 6103 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-ln27 node "control-plane-us-central1-b-ln27" of role "control-plane" is not ready Node control-plane-us-central1-c-4nms node "control-plane-us-central1-c-4nms" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-gsj9x system-cluster-critical pod "coredns-5d479c4f64-gsj9x" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-rkggr system-cluster-critical pod "coredns-autoscaler-557ccb4c66-rkggr" is pending Pod kube-system/dns-controller-6f69f47785-2sqd5 system-cluster-critical pod "dns-controller-6f69f47785-2sqd5" is pending Validation Failed W0129 11:01:39.113402 6103 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 11:01:49.441358 6103 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-ln27 node "control-plane-us-central1-b-ln27" of role "control-plane" is not ready Node control-plane-us-central1-c-4nms node "control-plane-us-central1-c-4nms" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-gsj9x system-cluster-critical pod "coredns-5d479c4f64-gsj9x" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-rkggr system-cluster-critical pod "coredns-autoscaler-557ccb4c66-rkggr" is pending Pod kube-system/dns-controller-6f69f47785-2sqd5 system-cluster-critical pod "dns-controller-6f69f47785-2sqd5" is pending Validation Failed W0129 11:01:50.416672 6103 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 11:02:00.761097 6103 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-ln27 node "control-plane-us-central1-b-ln27" of role "control-plane" is not ready Node control-plane-us-central1-c-4nms node "control-plane-us-central1-c-4nms" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-gsj9x system-cluster-critical pod "coredns-5d479c4f64-gsj9x" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-rkggr system-cluster-critical pod "coredns-autoscaler-557ccb4c66-rkggr" is pending Pod kube-system/dns-controller-6f69f47785-2sqd5 system-cluster-critical pod "dns-controller-6f69f47785-2sqd5" is pending Validation Failed W0129 11:02:01.598017 6103 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 11:02:12.030622 6103 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-ln27 node "control-plane-us-central1-b-ln27" of role "control-plane" is not ready Node control-plane-us-central1-c-4nms node "control-plane-us-central1-c-4nms" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-gsj9x system-cluster-critical pod "coredns-5d479c4f64-gsj9x" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-rkggr system-cluster-critical pod "coredns-autoscaler-557ccb4c66-rkggr" is pending Pod kube-system/dns-controller-6f69f47785-2sqd5 system-cluster-critical pod "dns-controller-6f69f47785-2sqd5" is pending Validation Failed W0129 11:02:12.913345 6103 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 11:02:23.213049 6103 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-ln27 node "control-plane-us-central1-b-ln27" of role "control-plane" is not ready Node control-plane-us-central1-c-4nms node "control-plane-us-central1-c-4nms" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-gsj9x system-cluster-critical pod "coredns-5d479c4f64-gsj9x" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-rkggr system-cluster-critical pod "coredns-autoscaler-557ccb4c66-rkggr" is pending Pod kube-system/dns-controller-6f69f47785-2sqd5 system-cluster-critical pod "dns-controller-6f69f47785-2sqd5" is pending Validation Failed W0129 11:02:24.068292 6103 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 11:02:34.438411 6103 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-ln27 node "control-plane-us-central1-b-ln27" of role "control-plane" is not ready Node control-plane-us-central1-c-4nms node "control-plane-us-central1-c-4nms" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-gsj9x system-cluster-critical pod "coredns-5d479c4f64-gsj9x" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-rkggr system-cluster-critical pod "coredns-autoscaler-557ccb4c66-rkggr" is pending Pod kube-system/dns-controller-6f69f47785-2sqd5 system-cluster-critical pod "dns-controller-6f69f47785-2sqd5" is pending Validation Failed W0129 11:02:35.211065 6103 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 11:02:45.567457 6103 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-ln27 node "control-plane-us-central1-b-ln27" of role "control-plane" is not ready Node control-plane-us-central1-c-4nms node "control-plane-us-central1-c-4nms" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-gsj9x system-cluster-critical pod "coredns-5d479c4f64-gsj9x" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-rkggr system-cluster-critical pod "coredns-autoscaler-557ccb4c66-rkggr" is pending Pod kube-system/dns-controller-6f69f47785-2sqd5 system-cluster-critical pod "dns-controller-6f69f47785-2sqd5" is pending Validation Failed W0129 11:02:46.423017 6103 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 11:02:56.729602 6103 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-ln27 node "control-plane-us-central1-b-ln27" of role "control-plane" is not ready Node control-plane-us-central1-c-4nms node "control-plane-us-central1-c-4nms" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-gsj9x system-cluster-critical pod "coredns-5d479c4f64-gsj9x" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-rkggr system-cluster-critical pod "coredns-autoscaler-557ccb4c66-rkggr" is pending Pod kube-system/dns-controller-6f69f47785-2sqd5 system-cluster-critical pod "dns-controller-6f69f47785-2sqd5" is pending Validation Failed W0129 11:02:57.565486 6103 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 11:03:07.871771 6103 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-ln27 node "control-plane-us-central1-b-ln27" of role "control-plane" is not ready Node control-plane-us-central1-c-4nms node "control-plane-us-central1-c-4nms" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-gsj9x system-cluster-critical pod "coredns-5d479c4f64-gsj9x" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-rkggr system-cluster-critical pod "coredns-autoscaler-557ccb4c66-rkggr" is pending Pod kube-system/dns-controller-6f69f47785-2sqd5 system-cluster-critical pod "dns-controller-6f69f47785-2sqd5" is pending Validation Failed W0129 11:03:08.708649 6103 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 11:03:19.028336 6103 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-ln27 node "control-plane-us-central1-b-ln27" of role "control-plane" is not ready Node control-plane-us-central1-c-4nms node "control-plane-us-central1-c-4nms" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-gsj9x system-cluster-critical pod "coredns-5d479c4f64-gsj9x" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-rkggr system-cluster-critical pod "coredns-autoscaler-557ccb4c66-rkggr" is pending Pod kube-system/dns-controller-6f69f47785-2sqd5 system-cluster-critical pod "dns-controller-6f69f47785-2sqd5" is pending Validation Failed W0129 11:03:19.898623 6103 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 11:03:30.167962 6103 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-ln27 node "control-plane-us-central1-b-ln27" of role "control-plane" is not ready Node control-plane-us-central1-c-4nms node "control-plane-us-central1-c-4nms" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-gsj9x system-cluster-critical pod "coredns-5d479c4f64-gsj9x" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-rkggr system-cluster-critical pod "coredns-autoscaler-557ccb4c66-rkggr" is pending Pod kube-system/dns-controller-6f69f47785-2sqd5 system-cluster-critical pod "dns-controller-6f69f47785-2sqd5" is pending Validation Failed W0129 11:03:31.010807 6103 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 11:03:41.328187 6103 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-ln27 node "control-plane-us-central1-b-ln27" of role "control-plane" is not ready Node control-plane-us-central1-c-4nms node "control-plane-us-central1-c-4nms" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-gsj9x system-cluster-critical pod "coredns-5d479c4f64-gsj9x" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-rkggr system-cluster-critical pod "coredns-autoscaler-557ccb4c66-rkggr" is pending Pod kube-system/dns-controller-6f69f47785-2sqd5 system-cluster-critical pod "dns-controller-6f69f47785-2sqd5" is pending Validation Failed W0129 11:03:42.102651 6103 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 11:03:52.610842 6103 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-ln27 node "control-plane-us-central1-b-ln27" of role "control-plane" is not ready Node control-plane-us-central1-c-4nms node "control-plane-us-central1-c-4nms" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-gsj9x system-cluster-critical pod "coredns-5d479c4f64-gsj9x" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-rkggr system-cluster-critical pod "coredns-autoscaler-557ccb4c66-rkggr" is pending Pod kube-system/dns-controller-6f69f47785-2sqd5 system-cluster-critical pod "dns-controller-6f69f47785-2sqd5" is pending Validation Failed W0129 11:03:53.460301 6103 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 11:04:03.759964 6103 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-ln27 node "control-plane-us-central1-b-ln27" of role "control-plane" is not ready Node control-plane-us-central1-c-4nms node "control-plane-us-central1-c-4nms" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-gsj9x system-cluster-critical pod "coredns-5d479c4f64-gsj9x" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-rkggr system-cluster-critical pod "coredns-autoscaler-557ccb4c66-rkggr" is pending Pod kube-system/dns-controller-6f69f47785-2sqd5 system-cluster-critical pod "dns-controller-6f69f47785-2sqd5" is pending Validation Failed W0129 11:04:04.625661 6103 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 11:04:15.039814 6103 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-ln27 node "control-plane-us-central1-b-ln27" of role "control-plane" is not ready Node control-plane-us-central1-c-4nms node "control-plane-us-central1-c-4nms" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-gsj9x system-cluster-critical pod "coredns-5d479c4f64-gsj9x" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-rkggr system-cluster-critical pod "coredns-autoscaler-557ccb4c66-rkggr" is pending Pod kube-system/dns-controller-6f69f47785-2sqd5 system-cluster-critical pod "dns-controller-6f69f47785-2sqd5" is pending Validation Failed W0129 11:04:15.872302 6103 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 11:04:26.291617 6103 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-ln27 node "control-plane-us-central1-b-ln27" of role "control-plane" is not ready Node control-plane-us-central1-c-4nms node "control-plane-us-central1-c-4nms" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-gsj9x system-cluster-critical pod "coredns-5d479c4f64-gsj9x" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-rkggr system-cluster-critical pod "coredns-autoscaler-557ccb4c66-rkggr" is pending Pod kube-system/dns-controller-6f69f47785-2sqd5 system-cluster-critical pod "dns-controller-6f69f47785-2sqd5" is pending Validation Failed W0129 11:04:27.162579 6103 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 11:04:37.457556 6103 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-ln27 node "control-plane-us-central1-b-ln27" of role "control-plane" is not ready Node control-plane-us-central1-c-4nms node "control-plane-us-central1-c-4nms" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-gsj9x system-cluster-critical pod "coredns-5d479c4f64-gsj9x" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-rkggr system-cluster-critical pod "coredns-autoscaler-557ccb4c66-rkggr" is pending Pod kube-system/dns-controller-6f69f47785-2sqd5 system-cluster-critical pod "dns-controller-6f69f47785-2sqd5" is pending Validation Failed W0129 11:04:38.274324 6103 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 11:04:48.594256 6103 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-ln27 node "control-plane-us-central1-b-ln27" of role "control-plane" is not ready Node control-plane-us-central1-c-4nms node "control-plane-us-central1-c-4nms" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-gsj9x system-cluster-critical pod "coredns-5d479c4f64-gsj9x" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-rkggr system-cluster-critical pod "coredns-autoscaler-557ccb4c66-rkggr" is pending Pod kube-system/dns-controller-6f69f47785-2sqd5 system-cluster-critical pod "dns-controller-6f69f47785-2sqd5" is pending Validation Failed W0129 11:04:49.403041 6103 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 11:04:59.742690 6103 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-ln27 node "control-plane-us-central1-b-ln27" of role "control-plane" is not ready Node control-plane-us-central1-c-4nms node "control-plane-us-central1-c-4nms" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-gsj9x system-cluster-critical pod "coredns-5d479c4f64-gsj9x" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-rkggr system-cluster-critical pod "coredns-autoscaler-557ccb4c66-rkggr" is pending Pod kube-system/dns-controller-6f69f47785-2sqd5 system-cluster-critical pod "dns-controller-6f69f47785-2sqd5" is pending Validation Failed W0129 11:05:00.549628 6103 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 11:05:10.893633 6103 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] I0129 11:05:11.086979 5746 boskos.go:86] Sending heartbeat to Boskos INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 ... skipping 16 lines ... Node control-plane-us-central1-b-ln27 node "control-plane-us-central1-b-ln27" of role "control-plane" is not ready Node control-plane-us-central1-c-4nms node "control-plane-us-central1-c-4nms" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-gsj9x system-cluster-critical pod "coredns-5d479c4f64-gsj9x" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-rkggr system-cluster-critical pod "coredns-autoscaler-557ccb4c66-rkggr" is pending Pod kube-system/dns-controller-6f69f47785-2sqd5 system-cluster-critical pod "dns-controller-6f69f47785-2sqd5" is pending Validation Failed W0129 11:05:11.693163 6103 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 11:05:22.074595 6103 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-ln27 node "control-plane-us-central1-b-ln27" of role "control-plane" is not ready Node control-plane-us-central1-c-4nms node "control-plane-us-central1-c-4nms" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-gsj9x system-cluster-critical pod "coredns-5d479c4f64-gsj9x" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-rkggr system-cluster-critical pod "coredns-autoscaler-557ccb4c66-rkggr" is pending Pod kube-system/dns-controller-6f69f47785-2sqd5 system-cluster-critical pod "dns-controller-6f69f47785-2sqd5" is pending Validation Failed W0129 11:05:22.948862 6103 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 11:05:33.277698 6103 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-ln27 node "control-plane-us-central1-b-ln27" of role "control-plane" is not ready Node control-plane-us-central1-c-4nms node "control-plane-us-central1-c-4nms" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-gsj9x system-cluster-critical pod "coredns-5d479c4f64-gsj9x" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-rkggr system-cluster-critical pod "coredns-autoscaler-557ccb4c66-rkggr" is pending Pod kube-system/dns-controller-6f69f47785-2sqd5 system-cluster-critical pod "dns-controller-6f69f47785-2sqd5" is pending Validation Failed W0129 11:05:34.095590 6103 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 11:05:44.471518 6103 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-ln27 node "control-plane-us-central1-b-ln27" of role "control-plane" is not ready Node control-plane-us-central1-c-4nms node "control-plane-us-central1-c-4nms" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-gsj9x system-cluster-critical pod "coredns-5d479c4f64-gsj9x" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-rkggr system-cluster-critical pod "coredns-autoscaler-557ccb4c66-rkggr" is pending Pod kube-system/dns-controller-6f69f47785-2sqd5 system-cluster-critical pod "dns-controller-6f69f47785-2sqd5" is pending Validation Failed W0129 11:05:45.313937 6103 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 11:05:55.609148 6103 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-ln27 node "control-plane-us-central1-b-ln27" of role "control-plane" is not ready Node control-plane-us-central1-c-4nms node "control-plane-us-central1-c-4nms" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-gsj9x system-cluster-critical pod "coredns-5d479c4f64-gsj9x" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-rkggr system-cluster-critical pod "coredns-autoscaler-557ccb4c66-rkggr" is pending Pod kube-system/dns-controller-6f69f47785-2sqd5 system-cluster-critical pod "dns-controller-6f69f47785-2sqd5" is pending Validation Failed W0129 11:05:56.390563 6103 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 11:06:06.750810 6103 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-ln27 node "control-plane-us-central1-b-ln27" of role "control-plane" is not ready Node control-plane-us-central1-c-4nms node "control-plane-us-central1-c-4nms" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-gsj9x system-cluster-critical pod "coredns-5d479c4f64-gsj9x" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-rkggr system-cluster-critical pod "coredns-autoscaler-557ccb4c66-rkggr" is pending Pod kube-system/dns-controller-6f69f47785-2sqd5 system-cluster-critical pod "dns-controller-6f69f47785-2sqd5" is pending Validation Failed W0129 11:06:07.556983 6103 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 11:06:17.850121 6103 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-ln27 node "control-plane-us-central1-b-ln27" of role "control-plane" is not ready Node control-plane-us-central1-c-4nms node "control-plane-us-central1-c-4nms" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-gsj9x system-cluster-critical pod "coredns-5d479c4f64-gsj9x" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-rkggr system-cluster-critical pod "coredns-autoscaler-557ccb4c66-rkggr" is pending Pod kube-system/dns-controller-6f69f47785-2sqd5 system-cluster-critical pod "dns-controller-6f69f47785-2sqd5" is pending Validation Failed W0129 11:06:18.739463 6103 validate_cluster.go:232] (will retry): cluster not yet healthy I0129 11:06:29.046981 6103 gce_cloud.go:301] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-central1-a ControlPlane e2-standard-2 1 1 us-central1 control-plane-us-central1-b ControlPlane e2-standard-2 1 1 us-central1 ... skipping 15 lines ... Node control-plane-us-central1-b-ln27 node "control-plane-us-central1-b-ln27" of role "control-plane" is not ready Node control-plane-us-central1-c-4nms node "control-plane-us-central1-c-4nms" of role "control-plane" is not ready Pod kube-system/coredns-5d479c4f64-gsj9x system-cluster-critical pod "coredns-5d479c4f64-gsj9x" is pending Pod kube-system/coredns-autoscaler-557ccb4c66-rkggr system-cluster-critical pod "coredns-autoscaler-557ccb4c66-rkggr" is pending Pod kube-system/dns-controller-6f69f47785-2sqd5 system-cluster-critical pod "dns-controller-6f69f47785-2sqd5" is pending Validation Failed W0129 11:06:29.992816 6103 validate_cluster.go:232] (will retry): cluster not yet healthy Error: validation failed: wait time exceeded during validation I0129 11:06:40.005919 5746 dumplogs.go:45] /home/prow/go/src/k8s.io/kops/_rundir/72d7c71b-9fc2-11ed-9a0c-f25a29267a91/kops toolbox dump --name ha-migration.k8s.local --dir /logs/artifacts --private-key /tmp/kops-ssh786086317/key --ssh-user prow I0129 11:06:40.005984 5746 local.go:42] ⚙️ /home/prow/go/src/k8s.io/kops/_rundir/72d7c71b-9fc2-11ed-9a0c-f25a29267a91/kops toolbox dump --name ha-migration.k8s.local --dir /logs/artifacts --private-key /tmp/kops-ssh786086317/key --ssh-user prow I0129 11:06:40.405013 6114 gce.go:99] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b us-central1-d] 2023/01/29 11:06:44 Dumping node control-plane-us-central1-a-z33b 2023/01/29 11:06:48 error dumping node control-plane-us-central1-a-z33b: error executing command "if command -v kubectl &> /dev/null; then kubectl logs -n kube-system --all-containers -l \"k8s-app=external-dns\"; fi": Process exited with status 1 2023/01/29 11:06:48 error dumping node control-plane-us-central1-a-z33b: error executing command "if command -v kubectl &> /dev/null; then kubectl logs -n kube-system --all-containers -l \"k8s-app=dns-controller\"; fi": Process exited with status 1 2023/01/29 11:06:48 Dumping node control-plane-us-central1-b-ln27 2023/01/29 11:06:51 error dumping node control-plane-us-central1-b-ln27: error executing command "if command -v kubectl &> /dev/null; then kubectl logs -n kube-system --all-containers -l \"k8s-app=external-dns\"; fi": Process exited with status 1 2023/01/29 11:06:51 error dumping node control-plane-us-central1-b-ln27: error executing command "if command -v kubectl &> /dev/null; then kubectl logs -n kube-system --all-containers -l \"k8s-app=dns-controller\"; fi": Process exited with status 1 2023/01/29 11:06:51 Dumping node control-plane-us-central1-c-4nms 2023/01/29 11:06:54 error dumping node control-plane-us-central1-c-4nms: error executing command "if command -v kubectl &> /dev/null; then kubectl logs -n kube-system --all-containers -l \"k8s-app=external-dns\"; fi": Process exited with status 1 2023/01/29 11:06:54 error dumping node control-plane-us-central1-c-4nms: error executing command "if command -v kubectl &> /dev/null; then kubectl logs -n kube-system --all-containers -l \"k8s-app=dns-controller\"; fi": Process exited with status 1 2023/01/29 11:06:54 dumping node not registered in kubernetes: 34.173.49.70 2023/01/29 11:06:54 Dumping node 34.173.49.70 I0129 11:06:56.256914 5746 dumplogs.go:79] /home/prow/go/src/k8s.io/kops/_rundir/72d7c71b-9fc2-11ed-9a0c-f25a29267a91/kops get cluster --name ha-migration.k8s.local -o yaml I0129 11:06:56.256962 5746 local.go:42] ⚙️ /home/prow/go/src/k8s.io/kops/_rundir/72d7c71b-9fc2-11ed-9a0c-f25a29267a91/kops get cluster --name ha-migration.k8s.local -o yaml I0129 11:06:56.419381 5746 dumplogs.go:79] /home/prow/go/src/k8s.io/kops/_rundir/72d7c71b-9fc2-11ed-9a0c-f25a29267a91/kops get instancegroups --name ha-migration.k8s.local -o yaml I0129 11:06:56.419428 5746 local.go:42] ⚙️ /home/prow/go/src/k8s.io/kops/_rundir/72d7c71b-9fc2-11ed-9a0c-f25a29267a91/kops get instancegroups --name ha-migration.k8s.local -o yaml ... skipping 166 lines ... Deleted cluster: "ha-migration.k8s.local" I0129 11:09:09.890221 5746 gcs.go:88] gsutil -u k8s-jkns-e2e-bazel rm -r gs://k8s-jkns-e2e-bazel-state-72 I0129 11:09:09.890279 5746 local.go:42] ⚙️ gsutil -u k8s-jkns-e2e-bazel rm -r gs://k8s-jkns-e2e-bazel-state-72 Removing gs://k8s-jkns-e2e-bazel-state-72/... I0129 11:09:12.646229 5746 down.go:62] releasing boskos project I0129 11:09:12.654210 5746 boskos.go:83] Boskos heartbeat func received signal to close Error: exit status 1 + EXIT_VALUE=1 + set +o xtrace