e2e-kops-grid-gce-calico-u2004-k26 #1616261075206934528
Recent runs || View in Spyglass
| Result | FAILURE |
| Tests | 1 failed / 1 succeeded |
| Started | |
| Elapsed | 20m1s |
| Revision | master |
exit status 1
from junit_runner.xml
Filter through log files | View test history on testgrid
kubetest2 Down
... skipping 163 lines ... ..............................Updated [https://www.googleapis.com/compute/v1/projects/k8s-gce-upg-1-5-1-6-up-clu-n]. ...done. WARNING: No host aliases were added to your SSH configs because you do not have any running instances. Try running this command again after running some instances. I0120 02:29:16.677145 5520 up.go:44] Cleaning up any leaked resources from previous cluster I0120 02:29:16.677295 5520 dumplogs.go:45] /home/prow/go/src/k8s.io/kops/_rundir/f9198263-9869-11ed-824d-f64c9135b4ea/kops toolbox dump --name e2e-e2e-kops-grid-gce-calico-u2004-k26.k8s.local --dir /logs/artifacts --private-key /tmp/kops-ssh2580636758/key --ssh-user prow I0120 02:29:16.677324 5520 local.go:42] ⚙️ /home/prow/go/src/k8s.io/kops/_rundir/f9198263-9869-11ed-824d-f64c9135b4ea/kops toolbox dump --name e2e-e2e-kops-grid-gce-calico-u2004-k26.k8s.local --dir /logs/artifacts --private-key /tmp/kops-ssh2580636758/key --ssh-user prow Error: Cluster.kops.k8s.io "e2e-e2e-kops-grid-gce-calico-u2004-k26.k8s.local" not found W0120 02:29:16.884834 5520 down.go:34] Dumping cluster logs at the start of Down() failed: exit status 1 I0120 02:29:16.884880 5520 down.go:48] /home/prow/go/src/k8s.io/kops/_rundir/f9198263-9869-11ed-824d-f64c9135b4ea/kops delete cluster --name e2e-e2e-kops-grid-gce-calico-u2004-k26.k8s.local --yes I0120 02:29:16.884893 5520 local.go:42] ⚙️ /home/prow/go/src/k8s.io/kops/_rundir/f9198263-9869-11ed-824d-f64c9135b4ea/kops delete cluster --name e2e-e2e-kops-grid-gce-calico-u2004-k26.k8s.local --yes Error: error reading cluster configuration: Cluster.kops.k8s.io "e2e-e2e-kops-grid-gce-calico-u2004-k26.k8s.local" not found I0120 02:29:17.048908 5520 gcs.go:51] gsutil ls -b -p k8s-gce-upg-1-5-1-6-up-clu-n gs://k8s-gce-upg-1-5-1-6-up-clu-n-state-f9 I0120 02:29:17.048950 5520 local.go:42] ⚙️ gsutil ls -b -p k8s-gce-upg-1-5-1-6-up-clu-n gs://k8s-gce-upg-1-5-1-6-up-clu-n-state-f9 I0120 02:29:19.019362 5520 gcs.go:70] gsutil mb -p k8s-gce-upg-1-5-1-6-up-clu-n gs://k8s-gce-upg-1-5-1-6-up-clu-n-state-f9 I0120 02:29:19.019401 5520 local.go:42] ⚙️ gsutil mb -p k8s-gce-upg-1-5-1-6-up-clu-n gs://k8s-gce-upg-1-5-1-6-up-clu-n-state-f9 Creating gs://k8s-gce-upg-1-5-1-6-up-clu-n-state-f9/... I0120 02:29:21.524349 5520 http.go:37] curl http://metadata.google.internal/computeMetadata/v1/instance/network-interfaces/0/access-configs/0/external-ip 2023/01/20 02:29:21 failed to get external ip from metadata service: http://metadata.google.internal/computeMetadata/v1/instance/network-interfaces/0/access-configs/0/external-ip returned 404 I0120 02:29:21.538444 5520 http.go:37] curl https://ip.jsb.workers.dev I0120 02:29:21.631669 5520 up.go:167] /home/prow/go/src/k8s.io/kops/_rundir/f9198263-9869-11ed-824d-f64c9135b4ea/kops create cluster --name e2e-e2e-kops-grid-gce-calico-u2004-k26.k8s.local --cloud gce --kubernetes-version https://storage.googleapis.com/kubernetes-release/release/v1.26.1 --ssh-public-key /tmp/kops-ssh2580636758/key.pub --set cluster.spec.nodePortAccess=0.0.0.0/0 --channel=alpha --networking=calico --container-runtime=containerd --gce-service-account=default --admin-access 34.173.25.242/32 --master-count 1 --master-volume-size 48 --node-count 4 --node-volume-size 48 --zones us-east1-b --master-size e2-standard-2 --project k8s-gce-upg-1-5-1-6-up-clu-n I0120 02:29:21.631707 5520 local.go:42] ⚙️ /home/prow/go/src/k8s.io/kops/_rundir/f9198263-9869-11ed-824d-f64c9135b4ea/kops create cluster --name e2e-e2e-kops-grid-gce-calico-u2004-k26.k8s.local --cloud gce --kubernetes-version https://storage.googleapis.com/kubernetes-release/release/v1.26.1 --ssh-public-key /tmp/kops-ssh2580636758/key.pub --set cluster.spec.nodePortAccess=0.0.0.0/0 --channel=alpha --networking=calico --container-runtime=containerd --gce-service-account=default --admin-access 34.173.25.242/32 --master-count 1 --master-volume-size 48 --node-count 4 --node-volume-size 48 --zones us-east1-b --master-size e2-standard-2 --project k8s-gce-upg-1-5-1-6-up-clu-n I0120 02:29:21.679279 5840 create_cluster.go:882] Using SSH public key: /tmp/kops-ssh2580636758/key.pub I0120 02:29:22.026759 5840 new_cluster.go:573] VMs will be configured to use specified Service Account: default I0120 02:29:22.026889 5840 new_cluster.go:1338] Cloud Provider ID: "gce" ... skipping 349 lines ... I0120 02:29:29.984885 5861 keypair.go:226] Issuing new certificate: "apiserver-aggregator-ca" I0120 02:29:29.984917 5861 keypair.go:226] Issuing new certificate: "etcd-peers-ca-main" W0120 02:29:30.173238 5861 vfs_keystorereader.go:143] CA private key was not found I0120 02:29:30.175408 5861 keypair.go:226] Issuing new certificate: "service-account" I0120 02:29:30.277458 5861 keypair.go:226] Issuing new certificate: "kubernetes-ca" I0120 02:29:45.648862 5861 executor.go:111] Tasks: 42 done / 67 total; 19 can run W0120 02:29:55.508082 5861 executor.go:139] error running task "ForwardingRule/api-e2e-e2e-kops-grid-gce-calico-u2004-k26-k8s-local" (9m50s remaining to succeed): error creating ForwardingRule "api-e2e-e2e-kops-grid-gce-calico-u2004-k26-k8s-local": googleapi: Error 400: The resource 'projects/k8s-gce-upg-1-5-1-6-up-clu-n/regions/us-east1/targetPools/api-e2e-e2e-kops-grid-gce-calico-u2004-k26-k8s-local' is not ready, resourceNotReady I0120 02:29:55.508180 5861 executor.go:111] Tasks: 60 done / 67 total; 5 can run I0120 02:30:02.514204 5861 executor.go:111] Tasks: 65 done / 67 total; 2 can run I0120 02:30:16.472587 5861 executor.go:111] Tasks: 67 done / 67 total; 0 can run I0120 02:30:16.534233 5861 update_cluster.go:323] Exporting kubeconfig for cluster kOps has set your kubectl context to e2e-e2e-kops-grid-gce-calico-u2004-k26.k8s.local ... skipping 7 lines ... * read about installing addons at: https://kops.sigs.k8s.io/addons. I0120 02:30:27.163958 5520 up.go:251] /home/prow/go/src/k8s.io/kops/_rundir/f9198263-9869-11ed-824d-f64c9135b4ea/kops validate cluster --name e2e-e2e-kops-grid-gce-calico-u2004-k26.k8s.local --count 10 --wait 15m0s I0120 02:30:27.164026 5520 local.go:42] ⚙️ /home/prow/go/src/k8s.io/kops/_rundir/f9198263-9869-11ed-824d-f64c9135b4ea/kops validate cluster --name e2e-e2e-kops-grid-gce-calico-u2004-k26.k8s.local --count 10 --wait 15m0s Validating cluster e2e-e2e-kops-grid-gce-calico-u2004-k26.k8s.local W0120 02:30:57.539176 5881 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://34.23.62.2/api/v1/nodes": dial tcp 34.23.62.2:443: i/o timeout W0120 02:31:07.573795 5881 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://34.23.62.2/api/v1/nodes": dial tcp 34.23.62.2:443: connect: connection refused W0120 02:31:17.608431 5881 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://34.23.62.2/api/v1/nodes": dial tcp 34.23.62.2:443: connect: connection refused W0120 02:31:27.643732 5881 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://34.23.62.2/api/v1/nodes": dial tcp 34.23.62.2:443: connect: connection refused W0120 02:31:37.678512 5881 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://34.23.62.2/api/v1/nodes": dial tcp 34.23.62.2:443: connect: connection refused W0120 02:31:47.713310 5881 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://34.23.62.2/api/v1/nodes": dial tcp 34.23.62.2:443: connect: connection refused W0120 02:31:57.747266 5881 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://34.23.62.2/api/v1/nodes": dial tcp 34.23.62.2:443: connect: connection refused W0120 02:32:07.780866 5881 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://34.23.62.2/api/v1/nodes": dial tcp 34.23.62.2:443: connect: connection refused W0120 02:32:17.814710 5881 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://34.23.62.2/api/v1/nodes": dial tcp 34.23.62.2:443: connect: connection refused W0120 02:32:37.850984 5881 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://34.23.62.2/api/v1/nodes": net/http: TLS handshake timeout W0120 02:32:47.885206 5881 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://34.23.62.2/api/v1/nodes": dial tcp 34.23.62.2:443: connect: connection refused I0120 02:33:06.616898 5881 gce_cloud.go:301] Scanning zones: [us-east1-b us-east1-c us-east1-d us-east1-a] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-east1-b ControlPlane e2-standard-2 1 1 us-east1 nodes-us-east1-b Node e2-medium 4 4 us-east1 ... skipping 5 lines ... Machine https://www.googleapis.com/compute/v1/projects/k8s-gce-upg-1-5-1-6-up-clu-n/zones/us-east1-b/instances/control-plane-us-east1-b-x01n machine "https://www.googleapis.com/compute/v1/projects/k8s-gce-upg-1-5-1-6-up-clu-n/zones/us-east1-b/instances/control-plane-us-east1-b-x01n" has not yet joined cluster Machine https://www.googleapis.com/compute/v1/projects/k8s-gce-upg-1-5-1-6-up-clu-n/zones/us-east1-b/instances/nodes-us-east1-b-cnqn machine "https://www.googleapis.com/compute/v1/projects/k8s-gce-upg-1-5-1-6-up-clu-n/zones/us-east1-b/instances/nodes-us-east1-b-cnqn" has not yet joined cluster Machine https://www.googleapis.com/compute/v1/projects/k8s-gce-upg-1-5-1-6-up-clu-n/zones/us-east1-b/instances/nodes-us-east1-b-h2c3 machine "https://www.googleapis.com/compute/v1/projects/k8s-gce-upg-1-5-1-6-up-clu-n/zones/us-east1-b/instances/nodes-us-east1-b-h2c3" has not yet joined cluster Machine https://www.googleapis.com/compute/v1/projects/k8s-gce-upg-1-5-1-6-up-clu-n/zones/us-east1-b/instances/nodes-us-east1-b-t5tf machine "https://www.googleapis.com/compute/v1/projects/k8s-gce-upg-1-5-1-6-up-clu-n/zones/us-east1-b/instances/nodes-us-east1-b-t5tf" has not yet joined cluster Machine https://www.googleapis.com/compute/v1/projects/k8s-gce-upg-1-5-1-6-up-clu-n/zones/us-east1-b/instances/nodes-us-east1-b-xqbv machine "https://www.googleapis.com/compute/v1/projects/k8s-gce-upg-1-5-1-6-up-clu-n/zones/us-east1-b/instances/nodes-us-east1-b-xqbv" has not yet joined cluster Validation Failed W0120 02:33:07.575283 5881 validate_cluster.go:232] (will retry): cluster not yet healthy I0120 02:33:18.038001 5881 gce_cloud.go:301] Scanning zones: [us-east1-b us-east1-c us-east1-d us-east1-a] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-east1-b ControlPlane e2-standard-2 1 1 us-east1 nodes-us-east1-b Node e2-medium 4 4 us-east1 ... skipping 6 lines ... Machine https://www.googleapis.com/compute/v1/projects/k8s-gce-upg-1-5-1-6-up-clu-n/zones/us-east1-b/instances/control-plane-us-east1-b-x01n machine "https://www.googleapis.com/compute/v1/projects/k8s-gce-upg-1-5-1-6-up-clu-n/zones/us-east1-b/instances/control-plane-us-east1-b-x01n" has not yet joined cluster Machine https://www.googleapis.com/compute/v1/projects/k8s-gce-upg-1-5-1-6-up-clu-n/zones/us-east1-b/instances/nodes-us-east1-b-cnqn machine "https://www.googleapis.com/compute/v1/projects/k8s-gce-upg-1-5-1-6-up-clu-n/zones/us-east1-b/instances/nodes-us-east1-b-cnqn" has not yet joined cluster Machine https://www.googleapis.com/compute/v1/projects/k8s-gce-upg-1-5-1-6-up-clu-n/zones/us-east1-b/instances/nodes-us-east1-b-h2c3 machine "https://www.googleapis.com/compute/v1/projects/k8s-gce-upg-1-5-1-6-up-clu-n/zones/us-east1-b/instances/nodes-us-east1-b-h2c3" has not yet joined cluster Machine https://www.googleapis.com/compute/v1/projects/k8s-gce-upg-1-5-1-6-up-clu-n/zones/us-east1-b/instances/nodes-us-east1-b-t5tf machine "https://www.googleapis.com/compute/v1/projects/k8s-gce-upg-1-5-1-6-up-clu-n/zones/us-east1-b/instances/nodes-us-east1-b-t5tf" has not yet joined cluster Machine https://www.googleapis.com/compute/v1/projects/k8s-gce-upg-1-5-1-6-up-clu-n/zones/us-east1-b/instances/nodes-us-east1-b-xqbv machine "https://www.googleapis.com/compute/v1/projects/k8s-gce-upg-1-5-1-6-up-clu-n/zones/us-east1-b/instances/nodes-us-east1-b-xqbv" has not yet joined cluster Validation Failed W0120 02:33:18.834873 5881 validate_cluster.go:232] (will retry): cluster not yet healthy I0120 02:33:29.320888 5881 gce_cloud.go:301] Scanning zones: [us-east1-b us-east1-c us-east1-d us-east1-a] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-east1-b ControlPlane e2-standard-2 1 1 us-east1 nodes-us-east1-b Node e2-medium 4 4 us-east1 ... skipping 6 lines ... Machine https://www.googleapis.com/compute/v1/projects/k8s-gce-upg-1-5-1-6-up-clu-n/zones/us-east1-b/instances/control-plane-us-east1-b-x01n machine "https://www.googleapis.com/compute/v1/projects/k8s-gce-upg-1-5-1-6-up-clu-n/zones/us-east1-b/instances/control-plane-us-east1-b-x01n" has not yet joined cluster Machine https://www.googleapis.com/compute/v1/projects/k8s-gce-upg-1-5-1-6-up-clu-n/zones/us-east1-b/instances/nodes-us-east1-b-cnqn machine "https://www.googleapis.com/compute/v1/projects/k8s-gce-upg-1-5-1-6-up-clu-n/zones/us-east1-b/instances/nodes-us-east1-b-cnqn" has not yet joined cluster Machine https://www.googleapis.com/compute/v1/projects/k8s-gce-upg-1-5-1-6-up-clu-n/zones/us-east1-b/instances/nodes-us-east1-b-h2c3 machine "https://www.googleapis.com/compute/v1/projects/k8s-gce-upg-1-5-1-6-up-clu-n/zones/us-east1-b/instances/nodes-us-east1-b-h2c3" has not yet joined cluster Machine https://www.googleapis.com/compute/v1/projects/k8s-gce-upg-1-5-1-6-up-clu-n/zones/us-east1-b/instances/nodes-us-east1-b-t5tf machine "https://www.googleapis.com/compute/v1/projects/k8s-gce-upg-1-5-1-6-up-clu-n/zones/us-east1-b/instances/nodes-us-east1-b-t5tf" has not yet joined cluster Machine https://www.googleapis.com/compute/v1/projects/k8s-gce-upg-1-5-1-6-up-clu-n/zones/us-east1-b/instances/nodes-us-east1-b-xqbv machine "https://www.googleapis.com/compute/v1/projects/k8s-gce-upg-1-5-1-6-up-clu-n/zones/us-east1-b/instances/nodes-us-east1-b-xqbv" has not yet joined cluster Validation Failed W0120 02:33:30.184852 5881 validate_cluster.go:232] (will retry): cluster not yet healthy I0120 02:33:40.651581 5881 gce_cloud.go:301] Scanning zones: [us-east1-b us-east1-c us-east1-d us-east1-a] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-east1-b ControlPlane e2-standard-2 1 1 us-east1 nodes-us-east1-b Node e2-medium 4 4 us-east1 ... skipping 9 lines ... Machine https://www.googleapis.com/compute/v1/projects/k8s-gce-upg-1-5-1-6-up-clu-n/zones/us-east1-b/instances/nodes-us-east1-b-t5tf machine "https://www.googleapis.com/compute/v1/projects/k8s-gce-upg-1-5-1-6-up-clu-n/zones/us-east1-b/instances/nodes-us-east1-b-t5tf" has not yet joined cluster Machine https://www.googleapis.com/compute/v1/projects/k8s-gce-upg-1-5-1-6-up-clu-n/zones/us-east1-b/instances/nodes-us-east1-b-xqbv machine "https://www.googleapis.com/compute/v1/projects/k8s-gce-upg-1-5-1-6-up-clu-n/zones/us-east1-b/instances/nodes-us-east1-b-xqbv" has not yet joined cluster Pod kube-system/calico-kube-controllers-55f488656-9fq6c system-cluster-critical pod "calico-kube-controllers-55f488656-9fq6c" is pending Pod kube-system/calico-node-82rbq system-node-critical pod "calico-node-82rbq" is pending Pod kube-system/kube-scheduler-control-plane-us-east1-b-x01n system-cluster-critical pod "kube-scheduler-control-plane-us-east1-b-x01n" is pending Validation Failed W0120 02:33:41.505558 5881 validate_cluster.go:232] (will retry): cluster not yet healthy I0120 02:33:51.961239 5881 gce_cloud.go:301] Scanning zones: [us-east1-b us-east1-c us-east1-d us-east1-a] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-east1-b ControlPlane e2-standard-2 1 1 us-east1 nodes-us-east1-b Node e2-medium 4 4 us-east1 ... skipping 14 lines ... Pod kube-system/coredns-7bc44848df-72lwv system-cluster-critical pod "coredns-7bc44848df-72lwv" is pending Pod kube-system/coredns-autoscaler-7cb5c5b969-wfl2k system-cluster-critical pod "coredns-autoscaler-7cb5c5b969-wfl2k" is pending Pod kube-system/dns-controller-6d5f4bf5d6-6h5wz system-cluster-critical pod "dns-controller-6d5f4bf5d6-6h5wz" is pending Pod kube-system/etcd-manager-events-control-plane-us-east1-b-x01n system-cluster-critical pod "etcd-manager-events-control-plane-us-east1-b-x01n" is pending Pod kube-system/kops-controller-s4dvr system-cluster-critical pod "kops-controller-s4dvr" is pending Validation Failed W0120 02:33:52.927150 5881 validate_cluster.go:232] (will retry): cluster not yet healthy I0120 02:34:03.348020 5881 gce_cloud.go:301] Scanning zones: [us-east1-b us-east1-c us-east1-d us-east1-a] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-east1-b ControlPlane e2-standard-2 1 1 us-east1 nodes-us-east1-b Node e2-medium 4 4 us-east1 ... skipping 10 lines ... Machine https://www.googleapis.com/compute/v1/projects/k8s-gce-upg-1-5-1-6-up-clu-n/zones/us-east1-b/instances/nodes-us-east1-b-xqbv machine "https://www.googleapis.com/compute/v1/projects/k8s-gce-upg-1-5-1-6-up-clu-n/zones/us-east1-b/instances/nodes-us-east1-b-xqbv" has not yet joined cluster Pod kube-system/calico-kube-controllers-55f488656-9fq6c system-cluster-critical pod "calico-kube-controllers-55f488656-9fq6c" is pending Pod kube-system/calico-node-82rbq system-node-critical pod "calico-node-82rbq" is pending Pod kube-system/coredns-7bc44848df-72lwv system-cluster-critical pod "coredns-7bc44848df-72lwv" is pending Pod kube-system/coredns-autoscaler-7cb5c5b969-wfl2k system-cluster-critical pod "coredns-autoscaler-7cb5c5b969-wfl2k" is pending Validation Failed W0120 02:34:04.199258 5881 validate_cluster.go:232] (will retry): cluster not yet healthy I0120 02:34:06.974062 5520 boskos.go:86] Sending heartbeat to Boskos I0120 02:34:14.622691 5881 gce_cloud.go:301] Scanning zones: [us-east1-b us-east1-c us-east1-d us-east1-a] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-east1-b ControlPlane e2-standard-2 1 1 us-east1 ... skipping 12 lines ... Node control-plane-us-east1-b-x01n control-plane node "control-plane-us-east1-b-x01n" is missing kube-apiserver pod Pod kube-system/calico-node-82rbq system-node-critical pod "calico-node-82rbq" is not ready (calico-node) Pod kube-system/coredns-7bc44848df-72lwv system-cluster-critical pod "coredns-7bc44848df-72lwv" is pending Pod kube-system/coredns-autoscaler-7cb5c5b969-wfl2k system-cluster-critical pod "coredns-autoscaler-7cb5c5b969-wfl2k" is pending Pod kube-system/kube-apiserver-control-plane-us-east1-b-x01n system-cluster-critical pod "kube-apiserver-control-plane-us-east1-b-x01n" is pending Validation Failed W0120 02:34:15.555402 5881 validate_cluster.go:232] (will retry): cluster not yet healthy I0120 02:34:25.972425 5881 gce_cloud.go:301] Scanning zones: [us-east1-b us-east1-c us-east1-d us-east1-a] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-east1-b ControlPlane e2-standard-2 1 1 us-east1 nodes-us-east1-b Node e2-medium 4 4 us-east1 ... skipping 16 lines ... Pod kube-system/calico-node-gkz28 system-node-critical pod "calico-node-gkz28" is pending Pod kube-system/calico-node-kw9vt system-node-critical pod "calico-node-kw9vt" is pending Pod kube-system/calico-node-s8bg6 system-node-critical pod "calico-node-s8bg6" is pending Pod kube-system/coredns-7bc44848df-72lwv system-cluster-critical pod "coredns-7bc44848df-72lwv" is pending Pod kube-system/coredns-autoscaler-7cb5c5b969-wfl2k system-cluster-critical pod "coredns-autoscaler-7cb5c5b969-wfl2k" is pending Validation Failed W0120 02:34:27.125676 5881 validate_cluster.go:232] (will retry): cluster not yet healthy I0120 02:34:37.596633 5881 gce_cloud.go:301] Scanning zones: [us-east1-b us-east1-c us-east1-d us-east1-a] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-east1-b ControlPlane e2-standard-2 1 1 us-east1 nodes-us-east1-b Node e2-medium 4 4 us-east1 ... skipping 16 lines ... Pod kube-system/calico-node-gkz28 system-node-critical pod "calico-node-gkz28" is pending Pod kube-system/calico-node-kw9vt system-node-critical pod "calico-node-kw9vt" is pending Pod kube-system/calico-node-s8bg6 system-node-critical pod "calico-node-s8bg6" is pending Pod kube-system/coredns-7bc44848df-72lwv system-cluster-critical pod "coredns-7bc44848df-72lwv" is pending Pod kube-system/coredns-autoscaler-7cb5c5b969-wfl2k system-cluster-critical pod "coredns-autoscaler-7cb5c5b969-wfl2k" is pending Validation Failed W0120 02:34:38.341153 5881 validate_cluster.go:232] (will retry): cluster not yet healthy I0120 02:34:48.739600 5881 gce_cloud.go:301] Scanning zones: [us-east1-b us-east1-c us-east1-d us-east1-a] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-east1-b ControlPlane e2-standard-2 1 1 us-east1 nodes-us-east1-b Node e2-medium 4 4 us-east1 ... skipping 13 lines ... Pod kube-system/calico-node-gkz28 system-node-critical pod "calico-node-gkz28" is pending Pod kube-system/calico-node-kw9vt system-node-critical pod "calico-node-kw9vt" is not ready (calico-node) Pod kube-system/calico-node-s8bg6 system-node-critical pod "calico-node-s8bg6" is not ready (calico-node) Pod kube-system/coredns-7bc44848df-72lwv system-cluster-critical pod "coredns-7bc44848df-72lwv" is pending Pod kube-system/coredns-autoscaler-7cb5c5b969-wfl2k system-cluster-critical pod "coredns-autoscaler-7cb5c5b969-wfl2k" is pending Validation Failed W0120 02:34:49.498996 5881 validate_cluster.go:232] (will retry): cluster not yet healthy I0120 02:34:59.895182 5881 gce_cloud.go:301] Scanning zones: [us-east1-b us-east1-c us-east1-d us-east1-a] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-east1-b ControlPlane e2-standard-2 1 1 us-east1 nodes-us-east1-b Node e2-medium 4 4 us-east1 ... skipping 12 lines ... Pod kube-system/calico-node-2nndc system-node-critical pod "calico-node-2nndc" is not ready (calico-node) Pod kube-system/calico-node-gkz28 system-node-critical pod "calico-node-gkz28" is pending Pod kube-system/calico-node-kw9vt system-node-critical pod "calico-node-kw9vt" is not ready (calico-node) Pod kube-system/calico-node-s8bg6 system-node-critical pod "calico-node-s8bg6" is not ready (calico-node) Pod kube-system/coredns-7bc44848df-v69h2 system-cluster-critical pod "coredns-7bc44848df-v69h2" is pending Validation Failed W0120 02:35:00.682982 5881 validate_cluster.go:232] (will retry): cluster not yet healthy I0120 02:35:11.091073 5881 gce_cloud.go:301] Scanning zones: [us-east1-b us-east1-c us-east1-d us-east1-a] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-east1-b ControlPlane e2-standard-2 1 1 us-east1 nodes-us-east1-b Node e2-medium 4 4 us-east1 ... skipping 10 lines ... KIND NAME MESSAGE Pod kube-system/calico-node-82rbq system-node-critical pod "calico-node-82rbq" is not ready (calico-node) Pod kube-system/calico-node-gkz28 system-node-critical pod "calico-node-gkz28" is pending Pod kube-system/calico-node-kw9vt system-node-critical pod "calico-node-kw9vt" is not ready (calico-node) Pod kube-system/calico-node-s8bg6 system-node-critical pod "calico-node-s8bg6" is not ready (calico-node) Validation Failed W0120 02:35:11.656792 5881 validate_cluster.go:232] (will retry): cluster not yet healthy I0120 02:35:22.194532 5881 gce_cloud.go:301] Scanning zones: [us-east1-b us-east1-c us-east1-d us-east1-a] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-east1-b ControlPlane e2-standard-2 1 1 us-east1 nodes-us-east1-b Node e2-medium 4 4 us-east1 ... skipping 9 lines ... VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/calico-node-82rbq system-node-critical pod "calico-node-82rbq" is not ready (calico-node) Pod kube-system/calico-node-gkz28 system-node-critical pod "calico-node-gkz28" is not ready (calico-node) Pod kube-system/calico-node-s8bg6 system-node-critical pod "calico-node-s8bg6" is not ready (calico-node) Validation Failed W0120 02:35:23.004970 5881 validate_cluster.go:232] (will retry): cluster not yet healthy I0120 02:35:33.420614 5881 gce_cloud.go:301] Scanning zones: [us-east1-b us-east1-c us-east1-d us-east1-a] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-east1-b ControlPlane e2-standard-2 1 1 us-east1 nodes-us-east1-b Node e2-medium 4 4 us-east1 ... skipping 11 lines ... Pod kube-system/calico-node-82rbq system-node-critical pod "calico-node-82rbq" is not ready (calico-node) Pod kube-system/calico-node-gkz28 system-node-critical pod "calico-node-gkz28" is not ready (calico-node) Pod kube-system/kube-proxy-nodes-us-east1-b-h2c3 system-node-critical pod "kube-proxy-nodes-us-east1-b-h2c3" is pending Pod kube-system/kube-proxy-nodes-us-east1-b-t5tf system-node-critical pod "kube-proxy-nodes-us-east1-b-t5tf" is pending Pod kube-system/kube-proxy-nodes-us-east1-b-xqbv system-node-critical pod "kube-proxy-nodes-us-east1-b-xqbv" is pending Validation Failed W0120 02:35:34.168143 5881 validate_cluster.go:232] (will retry): cluster not yet healthy I0120 02:35:44.594151 5881 gce_cloud.go:301] Scanning zones: [us-east1-b us-east1-c us-east1-d us-east1-a] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-east1-b ControlPlane e2-standard-2 1 1 us-east1 nodes-us-east1-b Node e2-medium 4 4 us-east1 ... skipping 8 lines ... VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/calico-node-82rbq system-node-critical pod "calico-node-82rbq" is not ready (calico-node) Pod kube-system/calico-node-gkz28 system-node-critical pod "calico-node-gkz28" is not ready (calico-node) Validation Failed W0120 02:35:45.374027 5881 validate_cluster.go:232] (will retry): cluster not yet healthy I0120 02:35:55.817219 5881 gce_cloud.go:301] Scanning zones: [us-east1-b us-east1-c us-east1-d us-east1-a] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-east1-b ControlPlane e2-standard-2 1 1 us-east1 nodes-us-east1-b Node e2-medium 4 4 us-east1 ... skipping 7 lines ... nodes-us-east1-b-xqbv node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/calico-node-82rbq system-node-critical pod "calico-node-82rbq" is not ready (calico-node) Validation Failed W0120 02:35:56.560955 5881 validate_cluster.go:232] (will retry): cluster not yet healthy I0120 02:36:07.053914 5881 gce_cloud.go:301] Scanning zones: [us-east1-b us-east1-c us-east1-d us-east1-a] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-east1-b ControlPlane e2-standard-2 1 1 us-east1 nodes-us-east1-b Node e2-medium 4 4 us-east1 ... skipping 7 lines ... nodes-us-east1-b-xqbv node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/calico-node-82rbq system-node-critical pod "calico-node-82rbq" is not ready (calico-node) Validation Failed W0120 02:36:07.814586 5881 validate_cluster.go:232] (will retry): cluster not yet healthy I0120 02:36:18.303885 5881 gce_cloud.go:301] Scanning zones: [us-east1-b us-east1-c us-east1-d us-east1-a] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-east1-b ControlPlane e2-standard-2 1 1 us-east1 nodes-us-east1-b Node e2-medium 4 4 us-east1 ... skipping 7 lines ... nodes-us-east1-b-xqbv node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/calico-node-82rbq system-node-critical pod "calico-node-82rbq" is not ready (calico-node) Validation Failed W0120 02:36:18.944948 5881 validate_cluster.go:232] (will retry): cluster not yet healthy I0120 02:36:29.407260 5881 gce_cloud.go:301] Scanning zones: [us-east1-b us-east1-c us-east1-d us-east1-a] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-east1-b ControlPlane e2-standard-2 1 1 us-east1 nodes-us-east1-b Node e2-medium 4 4 us-east1 ... skipping 7 lines ... nodes-us-east1-b-xqbv node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/calico-node-82rbq system-node-critical pod "calico-node-82rbq" is not ready (calico-node) Validation Failed W0120 02:36:30.000323 5881 validate_cluster.go:232] (will retry): cluster not yet healthy I0120 02:36:40.485531 5881 gce_cloud.go:301] Scanning zones: [us-east1-b us-east1-c us-east1-d us-east1-a] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-east1-b ControlPlane e2-standard-2 1 1 us-east1 nodes-us-east1-b Node e2-medium 4 4 us-east1 ... skipping 7 lines ... nodes-us-east1-b-xqbv node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/calico-node-82rbq system-node-critical pod "calico-node-82rbq" is not ready (calico-node) Validation Failed W0120 02:36:41.230899 5881 validate_cluster.go:232] (will retry): cluster not yet healthy I0120 02:36:51.653682 5881 gce_cloud.go:301] Scanning zones: [us-east1-b us-east1-c us-east1-d us-east1-a] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-east1-b ControlPlane e2-standard-2 1 1 us-east1 nodes-us-east1-b Node e2-medium 4 4 us-east1 ... skipping 7 lines ... nodes-us-east1-b-xqbv node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/calico-node-82rbq system-node-critical pod "calico-node-82rbq" is not ready (calico-node) Validation Failed W0120 02:36:52.572203 5881 validate_cluster.go:232] (will retry): cluster not yet healthy I0120 02:37:03.051080 5881 gce_cloud.go:301] Scanning zones: [us-east1-b us-east1-c us-east1-d us-east1-a] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-east1-b ControlPlane e2-standard-2 1 1 us-east1 nodes-us-east1-b Node e2-medium 4 4 us-east1 ... skipping 7 lines ... nodes-us-east1-b-xqbv node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/calico-node-82rbq system-node-critical pod "calico-node-82rbq" is not ready (calico-node) Validation Failed W0120 02:37:03.618717 5881 validate_cluster.go:232] (will retry): cluster not yet healthy I0120 02:37:14.095246 5881 gce_cloud.go:301] Scanning zones: [us-east1-b us-east1-c us-east1-d us-east1-a] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-east1-b ControlPlane e2-standard-2 1 1 us-east1 nodes-us-east1-b Node e2-medium 4 4 us-east1 ... skipping 7 lines ... nodes-us-east1-b-xqbv node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/calico-node-82rbq system-node-critical pod "calico-node-82rbq" is not ready (calico-node) Validation Failed W0120 02:37:14.832675 5881 validate_cluster.go:232] (will retry): cluster not yet healthy I0120 02:37:25.311282 5881 gce_cloud.go:301] Scanning zones: [us-east1-b us-east1-c us-east1-d us-east1-a] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-east1-b ControlPlane e2-standard-2 1 1 us-east1 nodes-us-east1-b Node e2-medium 4 4 us-east1 ... skipping 7 lines ... nodes-us-east1-b-xqbv node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/calico-node-82rbq system-node-critical pod "calico-node-82rbq" is not ready (calico-node) Validation Failed W0120 02:37:25.889043 5881 validate_cluster.go:232] (will retry): cluster not yet healthy I0120 02:37:36.361465 5881 gce_cloud.go:301] Scanning zones: [us-east1-b us-east1-c us-east1-d us-east1-a] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-east1-b ControlPlane e2-standard-2 1 1 us-east1 nodes-us-east1-b Node e2-medium 4 4 us-east1 ... skipping 7 lines ... nodes-us-east1-b-xqbv node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/calico-node-82rbq system-node-critical pod "calico-node-82rbq" is not ready (calico-node) Validation Failed W0120 02:37:36.978491 5881 validate_cluster.go:232] (will retry): cluster not yet healthy I0120 02:37:47.433791 5881 gce_cloud.go:301] Scanning zones: [us-east1-b us-east1-c us-east1-d us-east1-a] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-east1-b ControlPlane e2-standard-2 1 1 us-east1 nodes-us-east1-b Node e2-medium 4 4 us-east1 ... skipping 7 lines ... nodes-us-east1-b-xqbv node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/calico-node-82rbq system-node-critical pod "calico-node-82rbq" is not ready (calico-node) Validation Failed W0120 02:37:48.013756 5881 validate_cluster.go:232] (will retry): cluster not yet healthy I0120 02:37:58.504392 5881 gce_cloud.go:301] Scanning zones: [us-east1-b us-east1-c us-east1-d us-east1-a] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-east1-b ControlPlane e2-standard-2 1 1 us-east1 nodes-us-east1-b Node e2-medium 4 4 us-east1 ... skipping 7 lines ... nodes-us-east1-b-xqbv node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/calico-node-82rbq system-node-critical pod "calico-node-82rbq" is not ready (calico-node) Validation Failed W0120 02:37:59.092280 5881 validate_cluster.go:232] (will retry): cluster not yet healthy I0120 02:38:09.502904 5881 gce_cloud.go:301] Scanning zones: [us-east1-b us-east1-c us-east1-d us-east1-a] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-east1-b ControlPlane e2-standard-2 1 1 us-east1 nodes-us-east1-b Node e2-medium 4 4 us-east1 ... skipping 7 lines ... nodes-us-east1-b-xqbv node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/calico-node-82rbq system-node-critical pod "calico-node-82rbq" is not ready (calico-node) Validation Failed W0120 02:38:10.084358 5881 validate_cluster.go:232] (will retry): cluster not yet healthy I0120 02:38:20.498720 5881 gce_cloud.go:301] Scanning zones: [us-east1-b us-east1-c us-east1-d us-east1-a] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-east1-b ControlPlane e2-standard-2 1 1 us-east1 nodes-us-east1-b Node e2-medium 4 4 us-east1 ... skipping 7 lines ... nodes-us-east1-b-xqbv node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/calico-node-82rbq system-node-critical pod "calico-node-82rbq" is not ready (calico-node) Validation Failed W0120 02:38:21.242828 5881 validate_cluster.go:232] (will retry): cluster not yet healthy I0120 02:38:31.706349 5881 gce_cloud.go:301] Scanning zones: [us-east1-b us-east1-c us-east1-d us-east1-a] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-east1-b ControlPlane e2-standard-2 1 1 us-east1 nodes-us-east1-b Node e2-medium 4 4 us-east1 ... skipping 7 lines ... nodes-us-east1-b-xqbv node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/calico-node-82rbq system-node-critical pod "calico-node-82rbq" is not ready (calico-node) Validation Failed W0120 02:38:32.293939 5881 validate_cluster.go:232] (will retry): cluster not yet healthy I0120 02:38:42.712245 5881 gce_cloud.go:301] Scanning zones: [us-east1-b us-east1-c us-east1-d us-east1-a] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-east1-b ControlPlane e2-standard-2 1 1 us-east1 nodes-us-east1-b Node e2-medium 4 4 us-east1 ... skipping 7 lines ... nodes-us-east1-b-xqbv node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/calico-node-82rbq system-node-critical pod "calico-node-82rbq" is not ready (calico-node) Validation Failed W0120 02:38:43.519922 5881 validate_cluster.go:232] (will retry): cluster not yet healthy I0120 02:38:54.018633 5881 gce_cloud.go:301] Scanning zones: [us-east1-b us-east1-c us-east1-d us-east1-a] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-east1-b ControlPlane e2-standard-2 1 1 us-east1 nodes-us-east1-b Node e2-medium 4 4 us-east1 ... skipping 7 lines ... nodes-us-east1-b-xqbv node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/calico-node-82rbq system-node-critical pod "calico-node-82rbq" is not ready (calico-node) Validation Failed W0120 02:38:54.594328 5881 validate_cluster.go:232] (will retry): cluster not yet healthy I0120 02:39:04.991482 5881 gce_cloud.go:301] Scanning zones: [us-east1-b us-east1-c us-east1-d us-east1-a] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-east1-b ControlPlane e2-standard-2 1 1 us-east1 nodes-us-east1-b Node e2-medium 4 4 us-east1 ... skipping 7 lines ... nodes-us-east1-b-xqbv node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/calico-node-82rbq system-node-critical pod "calico-node-82rbq" is not ready (calico-node) Validation Failed W0120 02:39:05.630665 5881 validate_cluster.go:232] (will retry): cluster not yet healthy I0120 02:39:06.986491 5520 boskos.go:86] Sending heartbeat to Boskos I0120 02:39:16.073514 5881 gce_cloud.go:301] Scanning zones: [us-east1-b us-east1-c us-east1-d us-east1-a] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-east1-b ControlPlane e2-standard-2 1 1 us-east1 ... skipping 8 lines ... nodes-us-east1-b-xqbv node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/calico-node-82rbq system-node-critical pod "calico-node-82rbq" is not ready (calico-node) Validation Failed W0120 02:39:16.773082 5881 validate_cluster.go:232] (will retry): cluster not yet healthy I0120 02:39:27.235782 5881 gce_cloud.go:301] Scanning zones: [us-east1-b us-east1-c us-east1-d us-east1-a] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-east1-b ControlPlane e2-standard-2 1 1 us-east1 nodes-us-east1-b Node e2-medium 4 4 us-east1 ... skipping 7 lines ... nodes-us-east1-b-xqbv node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/calico-node-82rbq system-node-critical pod "calico-node-82rbq" is not ready (calico-node) Validation Failed W0120 02:39:27.993847 5881 validate_cluster.go:232] (will retry): cluster not yet healthy I0120 02:39:38.428151 5881 gce_cloud.go:301] Scanning zones: [us-east1-b us-east1-c us-east1-d us-east1-a] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-east1-b ControlPlane e2-standard-2 1 1 us-east1 nodes-us-east1-b Node e2-medium 4 4 us-east1 ... skipping 7 lines ... nodes-us-east1-b-xqbv node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/calico-node-82rbq system-node-critical pod "calico-node-82rbq" is not ready (calico-node) Validation Failed W0120 02:39:39.172470 5881 validate_cluster.go:232] (will retry): cluster not yet healthy I0120 02:39:49.664035 5881 gce_cloud.go:301] Scanning zones: [us-east1-b us-east1-c us-east1-d us-east1-a] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-east1-b ControlPlane e2-standard-2 1 1 us-east1 nodes-us-east1-b Node e2-medium 4 4 us-east1 ... skipping 7 lines ... nodes-us-east1-b-xqbv node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/calico-node-82rbq system-node-critical pod "calico-node-82rbq" is not ready (calico-node) Validation Failed W0120 02:39:50.221257 5881 validate_cluster.go:232] (will retry): cluster not yet healthy I0120 02:40:00.656793 5881 gce_cloud.go:301] Scanning zones: [us-east1-b us-east1-c us-east1-d us-east1-a] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-east1-b ControlPlane e2-standard-2 1 1 us-east1 nodes-us-east1-b Node e2-medium 4 4 us-east1 ... skipping 7 lines ... nodes-us-east1-b-xqbv node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/calico-node-82rbq system-node-critical pod "calico-node-82rbq" is not ready (calico-node) Validation Failed W0120 02:40:01.290827 5881 validate_cluster.go:232] (will retry): cluster not yet healthy I0120 02:40:11.848678 5881 gce_cloud.go:301] Scanning zones: [us-east1-b us-east1-c us-east1-d us-east1-a] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-east1-b ControlPlane e2-standard-2 1 1 us-east1 nodes-us-east1-b Node e2-medium 4 4 us-east1 ... skipping 7 lines ... nodes-us-east1-b-xqbv node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/calico-node-82rbq system-node-critical pod "calico-node-82rbq" is not ready (calico-node) Validation Failed W0120 02:40:12.474339 5881 validate_cluster.go:232] (will retry): cluster not yet healthy I0120 02:40:22.951141 5881 gce_cloud.go:301] Scanning zones: [us-east1-b us-east1-c us-east1-d us-east1-a] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-east1-b ControlPlane e2-standard-2 1 1 us-east1 nodes-us-east1-b Node e2-medium 4 4 us-east1 ... skipping 7 lines ... nodes-us-east1-b-xqbv node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/calico-node-82rbq system-node-critical pod "calico-node-82rbq" is not ready (calico-node) Validation Failed W0120 02:40:23.857100 5881 validate_cluster.go:232] (will retry): cluster not yet healthy I0120 02:40:34.296151 5881 gce_cloud.go:301] Scanning zones: [us-east1-b us-east1-c us-east1-d us-east1-a] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-east1-b ControlPlane e2-standard-2 1 1 us-east1 nodes-us-east1-b Node e2-medium 4 4 us-east1 ... skipping 7 lines ... nodes-us-east1-b-xqbv node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/calico-node-82rbq system-node-critical pod "calico-node-82rbq" is not ready (calico-node) Validation Failed W0120 02:40:34.890695 5881 validate_cluster.go:232] (will retry): cluster not yet healthy I0120 02:40:45.413917 5881 gce_cloud.go:301] Scanning zones: [us-east1-b us-east1-c us-east1-d us-east1-a] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-east1-b ControlPlane e2-standard-2 1 1 us-east1 nodes-us-east1-b Node e2-medium 4 4 us-east1 ... skipping 7 lines ... nodes-us-east1-b-xqbv node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/calico-node-82rbq system-node-critical pod "calico-node-82rbq" is not ready (calico-node) Validation Failed W0120 02:40:46.194346 5881 validate_cluster.go:232] (will retry): cluster not yet healthy I0120 02:40:56.625249 5881 gce_cloud.go:301] Scanning zones: [us-east1-b us-east1-c us-east1-d us-east1-a] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-east1-b ControlPlane e2-standard-2 1 1 us-east1 nodes-us-east1-b Node e2-medium 4 4 us-east1 ... skipping 7 lines ... nodes-us-east1-b-xqbv node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/calico-node-82rbq system-node-critical pod "calico-node-82rbq" is not ready (calico-node) Validation Failed W0120 02:40:57.440067 5881 validate_cluster.go:232] (will retry): cluster not yet healthy I0120 02:41:07.872448 5881 gce_cloud.go:301] Scanning zones: [us-east1-b us-east1-c us-east1-d us-east1-a] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-east1-b ControlPlane e2-standard-2 1 1 us-east1 nodes-us-east1-b Node e2-medium 4 4 us-east1 ... skipping 7 lines ... nodes-us-east1-b-xqbv node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/calico-node-82rbq system-node-critical pod "calico-node-82rbq" is not ready (calico-node) Validation Failed W0120 02:41:08.473410 5881 validate_cluster.go:232] (will retry): cluster not yet healthy I0120 02:41:18.934745 5881 gce_cloud.go:301] Scanning zones: [us-east1-b us-east1-c us-east1-d us-east1-a] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-east1-b ControlPlane e2-standard-2 1 1 us-east1 nodes-us-east1-b Node e2-medium 4 4 us-east1 ... skipping 7 lines ... nodes-us-east1-b-xqbv node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/calico-node-82rbq system-node-critical pod "calico-node-82rbq" is not ready (calico-node) Validation Failed W0120 02:41:19.524993 5881 validate_cluster.go:232] (will retry): cluster not yet healthy I0120 02:41:29.928542 5881 gce_cloud.go:301] Scanning zones: [us-east1-b us-east1-c us-east1-d us-east1-a] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-east1-b ControlPlane e2-standard-2 1 1 us-east1 nodes-us-east1-b Node e2-medium 4 4 us-east1 ... skipping 7 lines ... nodes-us-east1-b-xqbv node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/calico-node-82rbq system-node-critical pod "calico-node-82rbq" is not ready (calico-node) Validation Failed W0120 02:41:30.531056 5881 validate_cluster.go:232] (will retry): cluster not yet healthy I0120 02:41:40.959787 5881 gce_cloud.go:301] Scanning zones: [us-east1-b us-east1-c us-east1-d us-east1-a] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-east1-b ControlPlane e2-standard-2 1 1 us-east1 nodes-us-east1-b Node e2-medium 4 4 us-east1 ... skipping 7 lines ... nodes-us-east1-b-xqbv node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/calico-node-82rbq system-node-critical pod "calico-node-82rbq" is not ready (calico-node) Validation Failed W0120 02:41:41.692621 5881 validate_cluster.go:232] (will retry): cluster not yet healthy I0120 02:41:52.184530 5881 gce_cloud.go:301] Scanning zones: [us-east1-b us-east1-c us-east1-d us-east1-a] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-east1-b ControlPlane e2-standard-2 1 1 us-east1 nodes-us-east1-b Node e2-medium 4 4 us-east1 ... skipping 7 lines ... nodes-us-east1-b-xqbv node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/calico-node-82rbq system-node-critical pod "calico-node-82rbq" is not ready (calico-node) Validation Failed W0120 02:41:52.959198 5881 validate_cluster.go:232] (will retry): cluster not yet healthy I0120 02:42:03.433374 5881 gce_cloud.go:301] Scanning zones: [us-east1-b us-east1-c us-east1-d us-east1-a] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-east1-b ControlPlane e2-standard-2 1 1 us-east1 nodes-us-east1-b Node e2-medium 4 4 us-east1 ... skipping 7 lines ... nodes-us-east1-b-xqbv node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/calico-node-82rbq system-node-critical pod "calico-node-82rbq" is not ready (calico-node) Validation Failed W0120 02:42:04.048472 5881 validate_cluster.go:232] (will retry): cluster not yet healthy I0120 02:42:14.417114 5881 gce_cloud.go:301] Scanning zones: [us-east1-b us-east1-c us-east1-d us-east1-a] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-east1-b ControlPlane e2-standard-2 1 1 us-east1 nodes-us-east1-b Node e2-medium 4 4 us-east1 ... skipping 7 lines ... nodes-us-east1-b-xqbv node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/calico-node-82rbq system-node-critical pod "calico-node-82rbq" is not ready (calico-node) Validation Failed W0120 02:42:15.152541 5881 validate_cluster.go:232] (will retry): cluster not yet healthy I0120 02:42:25.594650 5881 gce_cloud.go:301] Scanning zones: [us-east1-b us-east1-c us-east1-d us-east1-a] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-east1-b ControlPlane e2-standard-2 1 1 us-east1 nodes-us-east1-b Node e2-medium 4 4 us-east1 ... skipping 7 lines ... nodes-us-east1-b-xqbv node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/calico-node-82rbq system-node-critical pod "calico-node-82rbq" is not ready (calico-node) Validation Failed W0120 02:42:26.218127 5881 validate_cluster.go:232] (will retry): cluster not yet healthy I0120 02:42:36.665029 5881 gce_cloud.go:301] Scanning zones: [us-east1-b us-east1-c us-east1-d us-east1-a] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-east1-b ControlPlane e2-standard-2 1 1 us-east1 nodes-us-east1-b Node e2-medium 4 4 us-east1 ... skipping 7 lines ... nodes-us-east1-b-xqbv node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/calico-node-82rbq system-node-critical pod "calico-node-82rbq" is not ready (calico-node) Validation Failed W0120 02:42:37.444271 5881 validate_cluster.go:232] (will retry): cluster not yet healthy I0120 02:42:47.882325 5881 gce_cloud.go:301] Scanning zones: [us-east1-b us-east1-c us-east1-d us-east1-a] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-east1-b ControlPlane e2-standard-2 1 1 us-east1 nodes-us-east1-b Node e2-medium 4 4 us-east1 ... skipping 7 lines ... nodes-us-east1-b-xqbv node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/calico-node-82rbq system-node-critical pod "calico-node-82rbq" is not ready (calico-node) Validation Failed W0120 02:42:48.575660 5881 validate_cluster.go:232] (will retry): cluster not yet healthy I0120 02:42:59.088313 5881 gce_cloud.go:301] Scanning zones: [us-east1-b us-east1-c us-east1-d us-east1-a] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-east1-b ControlPlane e2-standard-2 1 1 us-east1 nodes-us-east1-b Node e2-medium 4 4 us-east1 ... skipping 7 lines ... nodes-us-east1-b-xqbv node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/calico-node-82rbq system-node-critical pod "calico-node-82rbq" is not ready (calico-node) Validation Failed W0120 02:42:59.707071 5881 validate_cluster.go:232] (will retry): cluster not yet healthy I0120 02:43:10.183661 5881 gce_cloud.go:301] Scanning zones: [us-east1-b us-east1-c us-east1-d us-east1-a] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-east1-b ControlPlane e2-standard-2 1 1 us-east1 nodes-us-east1-b Node e2-medium 4 4 us-east1 ... skipping 7 lines ... nodes-us-east1-b-xqbv node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/calico-node-82rbq system-node-critical pod "calico-node-82rbq" is not ready (calico-node) Validation Failed W0120 02:43:10.839720 5881 validate_cluster.go:232] (will retry): cluster not yet healthy I0120 02:43:21.447881 5881 gce_cloud.go:301] Scanning zones: [us-east1-b us-east1-c us-east1-d us-east1-a] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-east1-b ControlPlane e2-standard-2 1 1 us-east1 nodes-us-east1-b Node e2-medium 4 4 us-east1 ... skipping 7 lines ... nodes-us-east1-b-xqbv node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/calico-node-82rbq system-node-critical pod "calico-node-82rbq" is not ready (calico-node) Validation Failed W0120 02:43:22.034588 5881 validate_cluster.go:232] (will retry): cluster not yet healthy I0120 02:43:32.515695 5881 gce_cloud.go:301] Scanning zones: [us-east1-b us-east1-c us-east1-d us-east1-a] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-east1-b ControlPlane e2-standard-2 1 1 us-east1 nodes-us-east1-b Node e2-medium 4 4 us-east1 ... skipping 7 lines ... nodes-us-east1-b-xqbv node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/calico-node-82rbq system-node-critical pod "calico-node-82rbq" is not ready (calico-node) Validation Failed W0120 02:43:33.230317 5881 validate_cluster.go:232] (will retry): cluster not yet healthy I0120 02:43:43.613585 5881 gce_cloud.go:301] Scanning zones: [us-east1-b us-east1-c us-east1-d us-east1-a] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-east1-b ControlPlane e2-standard-2 1 1 us-east1 nodes-us-east1-b Node e2-medium 4 4 us-east1 ... skipping 7 lines ... nodes-us-east1-b-xqbv node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/calico-node-82rbq system-node-critical pod "calico-node-82rbq" is not ready (calico-node) Validation Failed W0120 02:43:44.228914 5881 validate_cluster.go:232] (will retry): cluster not yet healthy I0120 02:43:54.705081 5881 gce_cloud.go:301] Scanning zones: [us-east1-b us-east1-c us-east1-d us-east1-a] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-east1-b ControlPlane e2-standard-2 1 1 us-east1 nodes-us-east1-b Node e2-medium 4 4 us-east1 ... skipping 7 lines ... nodes-us-east1-b-xqbv node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/calico-node-82rbq system-node-critical pod "calico-node-82rbq" is not ready (calico-node) Validation Failed W0120 02:43:55.292626 5881 validate_cluster.go:232] (will retry): cluster not yet healthy I0120 02:44:05.705695 5881 gce_cloud.go:301] Scanning zones: [us-east1-b us-east1-c us-east1-d us-east1-a] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-east1-b ControlPlane e2-standard-2 1 1 us-east1 nodes-us-east1-b Node e2-medium 4 4 us-east1 ... skipping 7 lines ... nodes-us-east1-b-xqbv node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/calico-node-82rbq system-node-critical pod "calico-node-82rbq" is not ready (calico-node) Validation Failed W0120 02:44:06.288557 5881 validate_cluster.go:232] (will retry): cluster not yet healthy I0120 02:44:06.998170 5520 boskos.go:86] Sending heartbeat to Boskos I0120 02:44:16.736030 5881 gce_cloud.go:301] Scanning zones: [us-east1-b us-east1-c us-east1-d us-east1-a] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-east1-b ControlPlane e2-standard-2 1 1 us-east1 ... skipping 8 lines ... nodes-us-east1-b-xqbv node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/calico-node-82rbq system-node-critical pod "calico-node-82rbq" is not ready (calico-node) Validation Failed W0120 02:44:17.545485 5881 validate_cluster.go:232] (will retry): cluster not yet healthy I0120 02:44:27.950960 5881 gce_cloud.go:301] Scanning zones: [us-east1-b us-east1-c us-east1-d us-east1-a] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-east1-b ControlPlane e2-standard-2 1 1 us-east1 nodes-us-east1-b Node e2-medium 4 4 us-east1 ... skipping 7 lines ... nodes-us-east1-b-xqbv node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/calico-node-82rbq system-node-critical pod "calico-node-82rbq" is not ready (calico-node) Validation Failed W0120 02:44:28.681328 5881 validate_cluster.go:232] (will retry): cluster not yet healthy I0120 02:44:39.094380 5881 gce_cloud.go:301] Scanning zones: [us-east1-b us-east1-c us-east1-d us-east1-a] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-east1-b ControlPlane e2-standard-2 1 1 us-east1 nodes-us-east1-b Node e2-medium 4 4 us-east1 ... skipping 7 lines ... nodes-us-east1-b-xqbv node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/calico-node-82rbq system-node-critical pod "calico-node-82rbq" is not ready (calico-node) Validation Failed W0120 02:44:39.688551 5881 validate_cluster.go:232] (will retry): cluster not yet healthy I0120 02:44:50.124552 5881 gce_cloud.go:301] Scanning zones: [us-east1-b us-east1-c us-east1-d us-east1-a] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-east1-b ControlPlane e2-standard-2 1 1 us-east1 nodes-us-east1-b Node e2-medium 4 4 us-east1 ... skipping 7 lines ... nodes-us-east1-b-xqbv node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/calico-node-82rbq system-node-critical pod "calico-node-82rbq" is not ready (calico-node) Validation Failed W0120 02:44:50.683407 5881 validate_cluster.go:232] (will retry): cluster not yet healthy I0120 02:45:01.148133 5881 gce_cloud.go:301] Scanning zones: [us-east1-b us-east1-c us-east1-d us-east1-a] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-east1-b ControlPlane e2-standard-2 1 1 us-east1 nodes-us-east1-b Node e2-medium 4 4 us-east1 ... skipping 7 lines ... nodes-us-east1-b-xqbv node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/calico-node-82rbq system-node-critical pod "calico-node-82rbq" is not ready (calico-node) Validation Failed W0120 02:45:01.911823 5881 validate_cluster.go:232] (will retry): cluster not yet healthy I0120 02:45:12.349837 5881 gce_cloud.go:301] Scanning zones: [us-east1-b us-east1-c us-east1-d us-east1-a] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-east1-b ControlPlane e2-standard-2 1 1 us-east1 nodes-us-east1-b Node e2-medium 4 4 us-east1 ... skipping 7 lines ... nodes-us-east1-b-xqbv node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/calico-node-82rbq system-node-critical pod "calico-node-82rbq" is not ready (calico-node) Validation Failed W0120 02:45:13.109918 5881 validate_cluster.go:232] (will retry): cluster not yet healthy I0120 02:45:23.572171 5881 gce_cloud.go:301] Scanning zones: [us-east1-b us-east1-c us-east1-d us-east1-a] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-east1-b ControlPlane e2-standard-2 1 1 us-east1 nodes-us-east1-b Node e2-medium 4 4 us-east1 ... skipping 7 lines ... nodes-us-east1-b-xqbv node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/calico-node-82rbq system-node-critical pod "calico-node-82rbq" is not ready (calico-node) Validation Failed W0120 02:45:24.185287 5881 validate_cluster.go:232] (will retry): cluster not yet healthy Error: validation failed: wait time exceeded during validation I0120 02:45:34.191601 5520 dumplogs.go:45] /home/prow/go/src/k8s.io/kops/_rundir/f9198263-9869-11ed-824d-f64c9135b4ea/kops toolbox dump --name e2e-e2e-kops-grid-gce-calico-u2004-k26.k8s.local --dir /logs/artifacts --private-key /tmp/kops-ssh2580636758/key --ssh-user prow I0120 02:45:34.191654 5520 local.go:42] ⚙️ /home/prow/go/src/k8s.io/kops/_rundir/f9198263-9869-11ed-824d-f64c9135b4ea/kops toolbox dump --name e2e-e2e-kops-grid-gce-calico-u2004-k26.k8s.local --dir /logs/artifacts --private-key /tmp/kops-ssh2580636758/key --ssh-user prow I0120 02:45:34.746149 5891 gce.go:99] Scanning zones: [us-east1-b us-east1-c us-east1-d us-east1-a] 2023/01/20 02:45:39 Dumping node control-plane-us-east1-b-x01n 2023/01/20 02:45:43 error dumping node control-plane-us-east1-b-x01n: error executing command "if command -v kubectl &> /dev/null; then kubectl logs -n kube-system --all-containers -l \"k8s-app=external-dns\"; fi": Process exited with status 1 2023/01/20 02:45:43 error dumping node control-plane-us-east1-b-x01n: error executing command "if command -v kubectl &> /dev/null; then kubectl logs -n kube-system --all-containers -l \"k8s-app=dns-controller\"; fi": Process exited with status 1 2023/01/20 02:45:43 Dumping node nodes-us-east1-b-cnqn 2023/01/20 02:45:46 Dumping node nodes-us-east1-b-h2c3 2023/01/20 02:45:49 Dumping node nodes-us-east1-b-t5tf 2023/01/20 02:45:52 Dumping node nodes-us-east1-b-xqbv I0120 02:45:55.233119 5520 dumplogs.go:79] /home/prow/go/src/k8s.io/kops/_rundir/f9198263-9869-11ed-824d-f64c9135b4ea/kops get cluster --name e2e-e2e-kops-grid-gce-calico-u2004-k26.k8s.local -o yaml I0120 02:45:55.233163 5520 local.go:42] ⚙️ /home/prow/go/src/k8s.io/kops/_rundir/f9198263-9869-11ed-824d-f64c9135b4ea/kops get cluster --name e2e-e2e-kops-grid-gce-calico-u2004-k26.k8s.local -o yaml ... skipping 154 lines ... Deleted cluster: "e2e-e2e-kops-grid-gce-calico-u2004-k26.k8s.local" I0120 02:48:06.833220 5520 gcs.go:88] gsutil -u k8s-gce-upg-1-5-1-6-up-clu-n rm -r gs://k8s-gce-upg-1-5-1-6-up-clu-n-state-f9 I0120 02:48:06.833266 5520 local.go:42] ⚙️ gsutil -u k8s-gce-upg-1-5-1-6-up-clu-n rm -r gs://k8s-gce-upg-1-5-1-6-up-clu-n-state-f9 Removing gs://k8s-gce-upg-1-5-1-6-up-clu-n-state-f9/... I0120 02:48:09.266013 5520 down.go:62] releasing boskos project I0120 02:48:09.275676 5520 boskos.go:83] Boskos heartbeat func received signal to close Error: exit status 1 + EXIT_VALUE=1 + set +o xtrace