e2e-kops-grid-gce-calico-u2004-k26 #1617468996628516864
Recent runs || View in Spyglass
| Result | FAILURE |
| Tests | 1 failed / 1 succeeded |
| Started | |
| Elapsed | 20m35s |
| Revision | master |
exit status 1
from junit_runner.xml
Filter through log files | View test history on testgrid
kubetest2 Down
... skipping 163 lines ... .........................Updated [https://www.googleapis.com/compute/v1/projects/gce-gci-upg-1-4-lat-ctl-skew]. .done. WARNING: No host aliases were added to your SSH configs because you do not have any running instances. Try running this command again after running some instances. I0123 10:29:14.373968 5544 up.go:44] Cleaning up any leaked resources from previous cluster I0123 10:29:14.374130 5544 dumplogs.go:45] /home/prow/go/src/k8s.io/kops/_rundir/810e8320-9b08-11ed-8662-2ea6bdfefe0d/kops toolbox dump --name e2e-e2e-kops-grid-gce-calico-u2004-k26.k8s.local --dir /logs/artifacts --private-key /tmp/kops-ssh1902700557/key --ssh-user prow I0123 10:29:14.374156 5544 local.go:42] ⚙️ /home/prow/go/src/k8s.io/kops/_rundir/810e8320-9b08-11ed-8662-2ea6bdfefe0d/kops toolbox dump --name e2e-e2e-kops-grid-gce-calico-u2004-k26.k8s.local --dir /logs/artifacts --private-key /tmp/kops-ssh1902700557/key --ssh-user prow Error: Cluster.kops.k8s.io "e2e-e2e-kops-grid-gce-calico-u2004-k26.k8s.local" not found W0123 10:29:14.712095 5544 down.go:34] Dumping cluster logs at the start of Down() failed: exit status 1 I0123 10:29:14.712151 5544 down.go:48] /home/prow/go/src/k8s.io/kops/_rundir/810e8320-9b08-11ed-8662-2ea6bdfefe0d/kops delete cluster --name e2e-e2e-kops-grid-gce-calico-u2004-k26.k8s.local --yes I0123 10:29:14.712171 5544 local.go:42] ⚙️ /home/prow/go/src/k8s.io/kops/_rundir/810e8320-9b08-11ed-8662-2ea6bdfefe0d/kops delete cluster --name e2e-e2e-kops-grid-gce-calico-u2004-k26.k8s.local --yes Error: error reading cluster configuration: Cluster.kops.k8s.io "e2e-e2e-kops-grid-gce-calico-u2004-k26.k8s.local" not found I0123 10:29:14.962325 5544 gcs.go:51] gsutil ls -b -p gce-gci-upg-1-4-lat-ctl-skew gs://gce-gci-upg-1-4-lat-ctl-skew-state-81 I0123 10:29:14.962421 5544 local.go:42] ⚙️ gsutil ls -b -p gce-gci-upg-1-4-lat-ctl-skew gs://gce-gci-upg-1-4-lat-ctl-skew-state-81 I0123 10:29:17.074991 5544 gcs.go:70] gsutil mb -p gce-gci-upg-1-4-lat-ctl-skew gs://gce-gci-upg-1-4-lat-ctl-skew-state-81 I0123 10:29:17.075035 5544 local.go:42] ⚙️ gsutil mb -p gce-gci-upg-1-4-lat-ctl-skew gs://gce-gci-upg-1-4-lat-ctl-skew-state-81 Creating gs://gce-gci-upg-1-4-lat-ctl-skew-state-81/... I0123 10:29:19.793505 5544 http.go:37] curl http://metadata.google.internal/computeMetadata/v1/instance/network-interfaces/0/access-configs/0/external-ip 2023/01/23 10:29:19 failed to get external ip from metadata service: http://metadata.google.internal/computeMetadata/v1/instance/network-interfaces/0/access-configs/0/external-ip returned 404 I0123 10:29:19.806081 5544 http.go:37] curl https://ip.jsb.workers.dev I0123 10:29:19.893637 5544 up.go:167] /home/prow/go/src/k8s.io/kops/_rundir/810e8320-9b08-11ed-8662-2ea6bdfefe0d/kops create cluster --name e2e-e2e-kops-grid-gce-calico-u2004-k26.k8s.local --cloud gce --kubernetes-version https://storage.googleapis.com/kubernetes-release/release/v1.26.1 --ssh-public-key /tmp/kops-ssh1902700557/key.pub --set cluster.spec.nodePortAccess=0.0.0.0/0 --channel=alpha --networking=calico --container-runtime=containerd --gce-service-account=default --admin-access 35.224.48.17/32 --master-count 1 --master-volume-size 48 --node-count 4 --node-volume-size 48 --zones us-west4-a --master-size e2-standard-2 --project gce-gci-upg-1-4-lat-ctl-skew I0123 10:29:19.893694 5544 local.go:42] ⚙️ /home/prow/go/src/k8s.io/kops/_rundir/810e8320-9b08-11ed-8662-2ea6bdfefe0d/kops create cluster --name e2e-e2e-kops-grid-gce-calico-u2004-k26.k8s.local --cloud gce --kubernetes-version https://storage.googleapis.com/kubernetes-release/release/v1.26.1 --ssh-public-key /tmp/kops-ssh1902700557/key.pub --set cluster.spec.nodePortAccess=0.0.0.0/0 --channel=alpha --networking=calico --container-runtime=containerd --gce-service-account=default --admin-access 35.224.48.17/32 --master-count 1 --master-volume-size 48 --node-count 4 --node-volume-size 48 --zones us-west4-a --master-size e2-standard-2 --project gce-gci-upg-1-4-lat-ctl-skew I0123 10:29:19.946073 5866 create_cluster.go:882] Using SSH public key: /tmp/kops-ssh1902700557/key.pub I0123 10:29:20.243844 5866 new_cluster.go:573] VMs will be configured to use specified Service Account: default I0123 10:29:20.243974 5866 new_cluster.go:1338] Cloud Provider ID: "gce" ... skipping 349 lines ... I0123 10:29:28.786290 5888 keypair.go:226] Issuing new certificate: "etcd-manager-ca-events" I0123 10:29:28.787573 5888 keypair.go:226] Issuing new certificate: "etcd-clients-ca" W0123 10:29:28.979889 5888 vfs_keystorereader.go:143] CA private key was not found I0123 10:29:29.079630 5888 keypair.go:226] Issuing new certificate: "service-account" I0123 10:29:29.081935 5888 keypair.go:226] Issuing new certificate: "kubernetes-ca" I0123 10:29:41.467302 5888 executor.go:111] Tasks: 42 done / 67 total; 19 can run W0123 10:29:57.415902 5888 executor.go:139] error running task "ForwardingRule/api-e2e-e2e-kops-grid-gce-calico-u2004-k26-k8s-local" (9m44s remaining to succeed): error creating ForwardingRule "api-e2e-e2e-kops-grid-gce-calico-u2004-k26-k8s-local": googleapi: Error 400: The resource 'projects/gce-gci-upg-1-4-lat-ctl-skew/regions/us-west4/targetPools/api-e2e-e2e-kops-grid-gce-calico-u2004-k26-k8s-local' is not ready, resourceNotReady I0123 10:29:57.415985 5888 executor.go:111] Tasks: 60 done / 67 total; 5 can run I0123 10:30:04.898150 5888 executor.go:111] Tasks: 65 done / 67 total; 2 can run I0123 10:30:21.207931 5888 executor.go:111] Tasks: 67 done / 67 total; 0 can run I0123 10:30:21.640393 5888 update_cluster.go:323] Exporting kubeconfig for cluster kOps has set your kubectl context to e2e-e2e-kops-grid-gce-calico-u2004-k26.k8s.local ... skipping 7 lines ... * read about installing addons at: https://kops.sigs.k8s.io/addons. I0123 10:30:31.971153 5544 up.go:251] /home/prow/go/src/k8s.io/kops/_rundir/810e8320-9b08-11ed-8662-2ea6bdfefe0d/kops validate cluster --name e2e-e2e-kops-grid-gce-calico-u2004-k26.k8s.local --count 10 --wait 15m0s I0123 10:30:31.971213 5544 local.go:42] ⚙️ /home/prow/go/src/k8s.io/kops/_rundir/810e8320-9b08-11ed-8662-2ea6bdfefe0d/kops validate cluster --name e2e-e2e-kops-grid-gce-calico-u2004-k26.k8s.local --count 10 --wait 15m0s Validating cluster e2e-e2e-kops-grid-gce-calico-u2004-k26.k8s.local W0123 10:31:02.331332 5908 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://34.125.112.255/api/v1/nodes": dial tcp 34.125.112.255:443: i/o timeout W0123 10:31:12.374342 5908 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://34.125.112.255/api/v1/nodes": dial tcp 34.125.112.255:443: connect: connection refused W0123 10:31:22.417813 5908 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://34.125.112.255/api/v1/nodes": dial tcp 34.125.112.255:443: connect: connection refused W0123 10:31:32.460082 5908 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://34.125.112.255/api/v1/nodes": dial tcp 34.125.112.255:443: connect: connection refused W0123 10:31:42.503995 5908 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://34.125.112.255/api/v1/nodes": dial tcp 34.125.112.255:443: connect: connection refused W0123 10:31:52.546541 5908 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://34.125.112.255/api/v1/nodes": dial tcp 34.125.112.255:443: connect: connection refused W0123 10:32:02.589493 5908 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://34.125.112.255/api/v1/nodes": dial tcp 34.125.112.255:443: connect: connection refused W0123 10:32:12.633465 5908 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://34.125.112.255/api/v1/nodes": dial tcp 34.125.112.255:443: connect: connection refused W0123 10:32:22.677290 5908 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://34.125.112.255/api/v1/nodes": dial tcp 34.125.112.255:443: connect: connection refused W0123 10:32:32.720832 5908 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://34.125.112.255/api/v1/nodes": dial tcp 34.125.112.255:443: connect: connection refused W0123 10:32:42.764688 5908 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://34.125.112.255/api/v1/nodes": dial tcp 34.125.112.255:443: connect: connection refused W0123 10:32:52.808176 5908 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://34.125.112.255/api/v1/nodes": dial tcp 34.125.112.255:443: connect: connection refused W0123 10:33:12.851999 5908 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://34.125.112.255/api/v1/nodes": net/http: TLS handshake timeout W0123 10:33:32.895222 5908 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://34.125.112.255/api/v1/nodes": net/http: TLS handshake timeout W0123 10:33:42.936866 5908 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://34.125.112.255/api/v1/nodes": dial tcp 34.125.112.255:443: connect: connection refused W0123 10:34:02.981940 5908 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://34.125.112.255/api/v1/nodes": net/http: TLS handshake timeout I0123 10:34:05.959640 5544 boskos.go:86] Sending heartbeat to Boskos W0123 10:34:14.710028 5908 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://34.125.112.255/api/v1/nodes": dial tcp 34.125.112.255:443: connect: connection refused - error from a previous attempt: read tcp 10.60.163.170:55084->34.125.112.255:443: read: connection reset by peer W0123 10:34:24.754741 5908 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://34.125.112.255/api/v1/nodes": dial tcp 34.125.112.255:443: connect: connection refused W0123 10:34:34.798295 5908 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://34.125.112.255/api/v1/nodes": dial tcp 34.125.112.255:443: connect: connection refused W0123 10:34:44.845050 5908 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://34.125.112.255/api/v1/nodes": dial tcp 34.125.112.255:443: connect: connection refused I0123 10:34:55.688114 5908 gce_cloud.go:301] Scanning zones: [us-west4-a us-west4-b us-west4-c] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west4-a ControlPlane e2-standard-2 1 1 us-west4 nodes-us-west4-a Node e2-medium 4 4 us-west4 ... skipping 5 lines ... Machine https://www.googleapis.com/compute/v1/projects/gce-gci-upg-1-4-lat-ctl-skew/zones/us-west4-a/instances/control-plane-us-west4-a-s8j2 machine "https://www.googleapis.com/compute/v1/projects/gce-gci-upg-1-4-lat-ctl-skew/zones/us-west4-a/instances/control-plane-us-west4-a-s8j2" has not yet joined cluster Machine https://www.googleapis.com/compute/v1/projects/gce-gci-upg-1-4-lat-ctl-skew/zones/us-west4-a/instances/nodes-us-west4-a-304w machine "https://www.googleapis.com/compute/v1/projects/gce-gci-upg-1-4-lat-ctl-skew/zones/us-west4-a/instances/nodes-us-west4-a-304w" has not yet joined cluster Machine https://www.googleapis.com/compute/v1/projects/gce-gci-upg-1-4-lat-ctl-skew/zones/us-west4-a/instances/nodes-us-west4-a-3n9r machine "https://www.googleapis.com/compute/v1/projects/gce-gci-upg-1-4-lat-ctl-skew/zones/us-west4-a/instances/nodes-us-west4-a-3n9r" has not yet joined cluster Machine https://www.googleapis.com/compute/v1/projects/gce-gci-upg-1-4-lat-ctl-skew/zones/us-west4-a/instances/nodes-us-west4-a-4gcc machine "https://www.googleapis.com/compute/v1/projects/gce-gci-upg-1-4-lat-ctl-skew/zones/us-west4-a/instances/nodes-us-west4-a-4gcc" has not yet joined cluster Machine https://www.googleapis.com/compute/v1/projects/gce-gci-upg-1-4-lat-ctl-skew/zones/us-west4-a/instances/nodes-us-west4-a-l141 machine "https://www.googleapis.com/compute/v1/projects/gce-gci-upg-1-4-lat-ctl-skew/zones/us-west4-a/instances/nodes-us-west4-a-l141" has not yet joined cluster Validation Failed W0123 10:34:56.848346 5908 validate_cluster.go:232] (will retry): cluster not yet healthy I0123 10:35:07.267186 5908 gce_cloud.go:301] Scanning zones: [us-west4-a us-west4-b us-west4-c] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west4-a ControlPlane e2-standard-2 1 1 us-west4 nodes-us-west4-a Node e2-medium 4 4 us-west4 ... skipping 6 lines ... Machine https://www.googleapis.com/compute/v1/projects/gce-gci-upg-1-4-lat-ctl-skew/zones/us-west4-a/instances/control-plane-us-west4-a-s8j2 machine "https://www.googleapis.com/compute/v1/projects/gce-gci-upg-1-4-lat-ctl-skew/zones/us-west4-a/instances/control-plane-us-west4-a-s8j2" has not yet joined cluster Machine https://www.googleapis.com/compute/v1/projects/gce-gci-upg-1-4-lat-ctl-skew/zones/us-west4-a/instances/nodes-us-west4-a-304w machine "https://www.googleapis.com/compute/v1/projects/gce-gci-upg-1-4-lat-ctl-skew/zones/us-west4-a/instances/nodes-us-west4-a-304w" has not yet joined cluster Machine https://www.googleapis.com/compute/v1/projects/gce-gci-upg-1-4-lat-ctl-skew/zones/us-west4-a/instances/nodes-us-west4-a-3n9r machine "https://www.googleapis.com/compute/v1/projects/gce-gci-upg-1-4-lat-ctl-skew/zones/us-west4-a/instances/nodes-us-west4-a-3n9r" has not yet joined cluster Machine https://www.googleapis.com/compute/v1/projects/gce-gci-upg-1-4-lat-ctl-skew/zones/us-west4-a/instances/nodes-us-west4-a-4gcc machine "https://www.googleapis.com/compute/v1/projects/gce-gci-upg-1-4-lat-ctl-skew/zones/us-west4-a/instances/nodes-us-west4-a-4gcc" has not yet joined cluster Machine https://www.googleapis.com/compute/v1/projects/gce-gci-upg-1-4-lat-ctl-skew/zones/us-west4-a/instances/nodes-us-west4-a-l141 machine "https://www.googleapis.com/compute/v1/projects/gce-gci-upg-1-4-lat-ctl-skew/zones/us-west4-a/instances/nodes-us-west4-a-l141" has not yet joined cluster Validation Failed W0123 10:35:07.986010 5908 validate_cluster.go:232] (will retry): cluster not yet healthy I0123 10:35:18.384710 5908 gce_cloud.go:301] Scanning zones: [us-west4-a us-west4-b us-west4-c] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west4-a ControlPlane e2-standard-2 1 1 us-west4 nodes-us-west4-a Node e2-medium 4 4 us-west4 ... skipping 9 lines ... Machine https://www.googleapis.com/compute/v1/projects/gce-gci-upg-1-4-lat-ctl-skew/zones/us-west4-a/instances/nodes-us-west4-a-4gcc machine "https://www.googleapis.com/compute/v1/projects/gce-gci-upg-1-4-lat-ctl-skew/zones/us-west4-a/instances/nodes-us-west4-a-4gcc" has not yet joined cluster Machine https://www.googleapis.com/compute/v1/projects/gce-gci-upg-1-4-lat-ctl-skew/zones/us-west4-a/instances/nodes-us-west4-a-l141 machine "https://www.googleapis.com/compute/v1/projects/gce-gci-upg-1-4-lat-ctl-skew/zones/us-west4-a/instances/nodes-us-west4-a-l141" has not yet joined cluster Pod kube-system/calico-kube-controllers-55f488656-g4tks system-cluster-critical pod "calico-kube-controllers-55f488656-g4tks" is pending Pod kube-system/calico-node-phbgs system-node-critical pod "calico-node-phbgs" is pending Pod kube-system/dns-controller-6d5f4bf5d6-7l7zj system-cluster-critical pod "dns-controller-6d5f4bf5d6-7l7zj" is pending Validation Failed W0123 10:35:19.872099 5908 validate_cluster.go:232] (will retry): cluster not yet healthy I0123 10:35:30.227681 5908 gce_cloud.go:301] Scanning zones: [us-west4-a us-west4-b us-west4-c] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west4-a ControlPlane e2-standard-2 1 1 us-west4 nodes-us-west4-a Node e2-medium 4 4 us-west4 ... skipping 10 lines ... Machine https://www.googleapis.com/compute/v1/projects/gce-gci-upg-1-4-lat-ctl-skew/zones/us-west4-a/instances/nodes-us-west4-a-l141 machine "https://www.googleapis.com/compute/v1/projects/gce-gci-upg-1-4-lat-ctl-skew/zones/us-west4-a/instances/nodes-us-west4-a-l141" has not yet joined cluster Pod kube-system/calico-kube-controllers-55f488656-g4tks system-cluster-critical pod "calico-kube-controllers-55f488656-g4tks" is pending Pod kube-system/calico-node-phbgs system-node-critical pod "calico-node-phbgs" is pending Pod kube-system/coredns-7bc44848df-zckg6 system-cluster-critical pod "coredns-7bc44848df-zckg6" is pending Pod kube-system/dns-controller-6d5f4bf5d6-7l7zj system-cluster-critical pod "dns-controller-6d5f4bf5d6-7l7zj" is pending Validation Failed W0123 10:35:31.294873 5908 validate_cluster.go:232] (will retry): cluster not yet healthy I0123 10:35:41.745157 5908 gce_cloud.go:301] Scanning zones: [us-west4-a us-west4-b us-west4-c] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west4-a ControlPlane e2-standard-2 1 1 us-west4 nodes-us-west4-a Node e2-medium 4 4 us-west4 ... skipping 13 lines ... Pod kube-system/cloud-controller-manager-r99sf system-cluster-critical pod "cloud-controller-manager-r99sf" is pending Pod kube-system/coredns-7bc44848df-zckg6 system-cluster-critical pod "coredns-7bc44848df-zckg6" is pending Pod kube-system/coredns-autoscaler-7cb5c5b969-2hc2j system-cluster-critical pod "coredns-autoscaler-7cb5c5b969-2hc2j" is pending Pod kube-system/dns-controller-6d5f4bf5d6-7l7zj system-cluster-critical pod "dns-controller-6d5f4bf5d6-7l7zj" is pending Pod kube-system/kops-controller-cqcct system-cluster-critical pod "kops-controller-cqcct" is pending Validation Failed W0123 10:35:42.919508 5908 validate_cluster.go:232] (will retry): cluster not yet healthy I0123 10:35:53.230166 5908 gce_cloud.go:301] Scanning zones: [us-west4-a us-west4-b us-west4-c] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west4-a ControlPlane e2-standard-2 1 1 us-west4 nodes-us-west4-a Node e2-medium 4 4 us-west4 ... skipping 12 lines ... Pod kube-system/calico-node-phbgs system-node-critical pod "calico-node-phbgs" is pending Pod kube-system/cloud-controller-manager-r99sf system-cluster-critical pod "cloud-controller-manager-r99sf" is pending Pod kube-system/coredns-7bc44848df-zckg6 system-cluster-critical pod "coredns-7bc44848df-zckg6" is pending Pod kube-system/coredns-autoscaler-7cb5c5b969-2hc2j system-cluster-critical pod "coredns-autoscaler-7cb5c5b969-2hc2j" is pending Pod kube-system/kube-scheduler-control-plane-us-west4-a-s8j2 system-cluster-critical pod "kube-scheduler-control-plane-us-west4-a-s8j2" is pending Validation Failed W0123 10:35:54.053428 5908 validate_cluster.go:232] (will retry): cluster not yet healthy I0123 10:36:04.481945 5908 gce_cloud.go:301] Scanning zones: [us-west4-a us-west4-b us-west4-c] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west4-a ControlPlane e2-standard-2 1 1 us-west4 nodes-us-west4-a Node e2-medium 4 4 us-west4 ... skipping 14 lines ... Pod kube-system/calico-node-fxc5v system-node-critical pod "calico-node-fxc5v" is pending Pod kube-system/calico-node-jkzxz system-node-critical pod "calico-node-jkzxz" is pending Pod kube-system/calico-node-phbgs system-node-critical pod "calico-node-phbgs" is pending Pod kube-system/coredns-7bc44848df-zckg6 system-cluster-critical pod "coredns-7bc44848df-zckg6" is pending Pod kube-system/coredns-autoscaler-7cb5c5b969-2hc2j system-cluster-critical pod "coredns-autoscaler-7cb5c5b969-2hc2j" is pending Validation Failed W0123 10:36:05.246480 5908 validate_cluster.go:232] (will retry): cluster not yet healthy I0123 10:36:15.598700 5908 gce_cloud.go:301] Scanning zones: [us-west4-a us-west4-b us-west4-c] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west4-a ControlPlane e2-standard-2 1 1 us-west4 nodes-us-west4-a Node e2-medium 4 4 us-west4 ... skipping 20 lines ... Pod kube-system/calico-node-jkzxz system-node-critical pod "calico-node-jkzxz" is pending Pod kube-system/calico-node-phbgs system-node-critical pod "calico-node-phbgs" is pending Pod kube-system/coredns-7bc44848df-zckg6 system-cluster-critical pod "coredns-7bc44848df-zckg6" is pending Pod kube-system/coredns-autoscaler-7cb5c5b969-2hc2j system-cluster-critical pod "coredns-autoscaler-7cb5c5b969-2hc2j" is pending Pod kube-system/kube-controller-manager-control-plane-us-west4-a-s8j2 system-cluster-critical pod "kube-controller-manager-control-plane-us-west4-a-s8j2" is pending Validation Failed W0123 10:36:16.470963 5908 validate_cluster.go:232] (will retry): cluster not yet healthy I0123 10:36:26.785694 5908 gce_cloud.go:301] Scanning zones: [us-west4-a us-west4-b us-west4-c] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west4-a ControlPlane e2-standard-2 1 1 us-west4 nodes-us-west4-a Node e2-medium 4 4 us-west4 ... skipping 13 lines ... Pod kube-system/calico-node-fnglk system-node-critical pod "calico-node-fnglk" is pending Pod kube-system/calico-node-fxc5v system-node-critical pod "calico-node-fxc5v" is not ready (calico-node) Pod kube-system/calico-node-jkzxz system-node-critical pod "calico-node-jkzxz" is pending Pod kube-system/coredns-7bc44848df-zckg6 system-cluster-critical pod "coredns-7bc44848df-zckg6" is pending Pod kube-system/coredns-autoscaler-7cb5c5b969-2hc2j system-cluster-critical pod "coredns-autoscaler-7cb5c5b969-2hc2j" is pending Validation Failed W0123 10:36:27.620689 5908 validate_cluster.go:232] (will retry): cluster not yet healthy I0123 10:36:37.986247 5908 gce_cloud.go:301] Scanning zones: [us-west4-a us-west4-b us-west4-c] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west4-a ControlPlane e2-standard-2 1 1 us-west4 nodes-us-west4-a Node e2-medium 4 4 us-west4 ... skipping 12 lines ... Pod kube-system/calico-node-fnglk system-node-critical pod "calico-node-fnglk" is not ready (calico-node) Pod kube-system/calico-node-fxc5v system-node-critical pod "calico-node-fxc5v" is not ready (calico-node) Pod kube-system/calico-node-jkzxz system-node-critical pod "calico-node-jkzxz" is not ready (calico-node) Pod kube-system/coredns-7bc44848df-zckg6 system-cluster-critical pod "coredns-7bc44848df-zckg6" is pending Pod kube-system/coredns-autoscaler-7cb5c5b969-2hc2j system-cluster-critical pod "coredns-autoscaler-7cb5c5b969-2hc2j" is pending Validation Failed W0123 10:36:38.532888 5908 validate_cluster.go:232] (will retry): cluster not yet healthy I0123 10:36:48.919678 5908 gce_cloud.go:301] Scanning zones: [us-west4-a us-west4-b us-west4-c] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west4-a ControlPlane e2-standard-2 1 1 us-west4 nodes-us-west4-a Node e2-medium 4 4 us-west4 ... skipping 10 lines ... KIND NAME MESSAGE Pod kube-system/calico-node-9csb4 system-node-critical pod "calico-node-9csb4" is not ready (calico-node) Pod kube-system/calico-node-fnglk system-node-critical pod "calico-node-fnglk" is not ready (calico-node) Pod kube-system/calico-node-fxc5v system-node-critical pod "calico-node-fxc5v" is not ready (calico-node) Pod kube-system/calico-node-jkzxz system-node-critical pod "calico-node-jkzxz" is not ready (calico-node) Validation Failed W0123 10:36:49.714428 5908 validate_cluster.go:232] (will retry): cluster not yet healthy I0123 10:37:00.155485 5908 gce_cloud.go:301] Scanning zones: [us-west4-a us-west4-b us-west4-c] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west4-a ControlPlane e2-standard-2 1 1 us-west4 nodes-us-west4-a Node e2-medium 4 4 us-west4 ... skipping 11 lines ... Pod kube-system/calico-node-9csb4 system-node-critical pod "calico-node-9csb4" is not ready (calico-node) Pod kube-system/calico-node-fnglk system-node-critical pod "calico-node-fnglk" is not ready (calico-node) Pod kube-system/calico-node-fxc5v system-node-critical pod "calico-node-fxc5v" is not ready (calico-node) Pod kube-system/calico-node-jkzxz system-node-critical pod "calico-node-jkzxz" is not ready (calico-node) Pod kube-system/calico-node-phbgs system-node-critical pod "calico-node-phbgs" is not ready (calico-node) Validation Failed W0123 10:37:00.766239 5908 validate_cluster.go:232] (will retry): cluster not yet healthy I0123 10:37:11.111325 5908 gce_cloud.go:301] Scanning zones: [us-west4-a us-west4-b us-west4-c] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west4-a ControlPlane e2-standard-2 1 1 us-west4 nodes-us-west4-a Node e2-medium 4 4 us-west4 ... skipping 7 lines ... nodes-us-west4-a-l141 node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/calico-node-phbgs system-node-critical pod "calico-node-phbgs" is not ready (calico-node) Validation Failed W0123 10:37:11.957184 5908 validate_cluster.go:232] (will retry): cluster not yet healthy I0123 10:37:22.374467 5908 gce_cloud.go:301] Scanning zones: [us-west4-a us-west4-b us-west4-c] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west4-a ControlPlane e2-standard-2 1 1 us-west4 nodes-us-west4-a Node e2-medium 4 4 us-west4 ... skipping 7 lines ... nodes-us-west4-a-l141 node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/calico-node-phbgs system-node-critical pod "calico-node-phbgs" is not ready (calico-node) Validation Failed W0123 10:37:23.456606 5908 validate_cluster.go:232] (will retry): cluster not yet healthy I0123 10:37:33.806149 5908 gce_cloud.go:301] Scanning zones: [us-west4-a us-west4-b us-west4-c] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west4-a ControlPlane e2-standard-2 1 1 us-west4 nodes-us-west4-a Node e2-medium 4 4 us-west4 ... skipping 7 lines ... nodes-us-west4-a-l141 node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/calico-node-phbgs system-node-critical pod "calico-node-phbgs" is not ready (calico-node) Validation Failed W0123 10:37:34.515519 5908 validate_cluster.go:232] (will retry): cluster not yet healthy I0123 10:37:44.892823 5908 gce_cloud.go:301] Scanning zones: [us-west4-a us-west4-b us-west4-c] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west4-a ControlPlane e2-standard-2 1 1 us-west4 nodes-us-west4-a Node e2-medium 4 4 us-west4 ... skipping 7 lines ... nodes-us-west4-a-l141 node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/calico-node-phbgs system-node-critical pod "calico-node-phbgs" is not ready (calico-node) Validation Failed W0123 10:37:45.680369 5908 validate_cluster.go:232] (will retry): cluster not yet healthy I0123 10:37:56.086302 5908 gce_cloud.go:301] Scanning zones: [us-west4-a us-west4-b us-west4-c] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west4-a ControlPlane e2-standard-2 1 1 us-west4 nodes-us-west4-a Node e2-medium 4 4 us-west4 ... skipping 7 lines ... nodes-us-west4-a-l141 node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/calico-node-phbgs system-node-critical pod "calico-node-phbgs" is not ready (calico-node) Validation Failed W0123 10:37:56.767859 5908 validate_cluster.go:232] (will retry): cluster not yet healthy I0123 10:38:07.115694 5908 gce_cloud.go:301] Scanning zones: [us-west4-a us-west4-b us-west4-c] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west4-a ControlPlane e2-standard-2 1 1 us-west4 nodes-us-west4-a Node e2-medium 4 4 us-west4 ... skipping 7 lines ... nodes-us-west4-a-l141 node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/calico-node-phbgs system-node-critical pod "calico-node-phbgs" is not ready (calico-node) Validation Failed W0123 10:38:08.140863 5908 validate_cluster.go:232] (will retry): cluster not yet healthy I0123 10:38:18.532154 5908 gce_cloud.go:301] Scanning zones: [us-west4-a us-west4-b us-west4-c] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west4-a ControlPlane e2-standard-2 1 1 us-west4 nodes-us-west4-a Node e2-medium 4 4 us-west4 ... skipping 7 lines ... nodes-us-west4-a-l141 node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/calico-node-phbgs system-node-critical pod "calico-node-phbgs" is not ready (calico-node) Validation Failed W0123 10:38:19.341691 5908 validate_cluster.go:232] (will retry): cluster not yet healthy I0123 10:38:29.741931 5908 gce_cloud.go:301] Scanning zones: [us-west4-a us-west4-b us-west4-c] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west4-a ControlPlane e2-standard-2 1 1 us-west4 nodes-us-west4-a Node e2-medium 4 4 us-west4 ... skipping 7 lines ... nodes-us-west4-a-l141 node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/calico-node-phbgs system-node-critical pod "calico-node-phbgs" is not ready (calico-node) Validation Failed W0123 10:38:30.500029 5908 validate_cluster.go:232] (will retry): cluster not yet healthy I0123 10:38:40.883004 5908 gce_cloud.go:301] Scanning zones: [us-west4-a us-west4-b us-west4-c] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west4-a ControlPlane e2-standard-2 1 1 us-west4 nodes-us-west4-a Node e2-medium 4 4 us-west4 ... skipping 7 lines ... nodes-us-west4-a-l141 node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/calico-node-phbgs system-node-critical pod "calico-node-phbgs" is not ready (calico-node) Validation Failed W0123 10:38:41.524137 5908 validate_cluster.go:232] (will retry): cluster not yet healthy I0123 10:38:51.906536 5908 gce_cloud.go:301] Scanning zones: [us-west4-a us-west4-b us-west4-c] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west4-a ControlPlane e2-standard-2 1 1 us-west4 nodes-us-west4-a Node e2-medium 4 4 us-west4 ... skipping 7 lines ... nodes-us-west4-a-l141 node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/calico-node-phbgs system-node-critical pod "calico-node-phbgs" is not ready (calico-node) Validation Failed W0123 10:38:52.769422 5908 validate_cluster.go:232] (will retry): cluster not yet healthy I0123 10:39:03.187220 5908 gce_cloud.go:301] Scanning zones: [us-west4-a us-west4-b us-west4-c] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west4-a ControlPlane e2-standard-2 1 1 us-west4 nodes-us-west4-a Node e2-medium 4 4 us-west4 ... skipping 7 lines ... nodes-us-west4-a-l141 node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/calico-node-phbgs system-node-critical pod "calico-node-phbgs" is not ready (calico-node) Validation Failed W0123 10:39:04.045045 5908 validate_cluster.go:232] (will retry): cluster not yet healthy I0123 10:39:05.972889 5544 boskos.go:86] Sending heartbeat to Boskos I0123 10:39:14.386386 5908 gce_cloud.go:301] Scanning zones: [us-west4-a us-west4-b us-west4-c] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west4-a ControlPlane e2-standard-2 1 1 us-west4 ... skipping 8 lines ... nodes-us-west4-a-l141 node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/calico-node-phbgs system-node-critical pod "calico-node-phbgs" is not ready (calico-node) Validation Failed W0123 10:39:15.318209 5908 validate_cluster.go:232] (will retry): cluster not yet healthy I0123 10:39:25.727441 5908 gce_cloud.go:301] Scanning zones: [us-west4-a us-west4-b us-west4-c] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west4-a ControlPlane e2-standard-2 1 1 us-west4 nodes-us-west4-a Node e2-medium 4 4 us-west4 ... skipping 7 lines ... nodes-us-west4-a-l141 node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/calico-node-phbgs system-node-critical pod "calico-node-phbgs" is not ready (calico-node) Validation Failed W0123 10:39:26.351899 5908 validate_cluster.go:232] (will retry): cluster not yet healthy I0123 10:39:36.693778 5908 gce_cloud.go:301] Scanning zones: [us-west4-a us-west4-b us-west4-c] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west4-a ControlPlane e2-standard-2 1 1 us-west4 nodes-us-west4-a Node e2-medium 4 4 us-west4 ... skipping 7 lines ... nodes-us-west4-a-l141 node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/calico-node-phbgs system-node-critical pod "calico-node-phbgs" is not ready (calico-node) Validation Failed W0123 10:39:37.322865 5908 validate_cluster.go:232] (will retry): cluster not yet healthy I0123 10:39:47.684116 5908 gce_cloud.go:301] Scanning zones: [us-west4-a us-west4-b us-west4-c] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west4-a ControlPlane e2-standard-2 1 1 us-west4 nodes-us-west4-a Node e2-medium 4 4 us-west4 ... skipping 7 lines ... nodes-us-west4-a-l141 node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/calico-node-phbgs system-node-critical pod "calico-node-phbgs" is not ready (calico-node) Validation Failed W0123 10:39:48.522901 5908 validate_cluster.go:232] (will retry): cluster not yet healthy I0123 10:39:58.844543 5908 gce_cloud.go:301] Scanning zones: [us-west4-a us-west4-b us-west4-c] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west4-a ControlPlane e2-standard-2 1 1 us-west4 nodes-us-west4-a Node e2-medium 4 4 us-west4 ... skipping 7 lines ... nodes-us-west4-a-l141 node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/calico-node-phbgs system-node-critical pod "calico-node-phbgs" is not ready (calico-node) Validation Failed W0123 10:39:59.449544 5908 validate_cluster.go:232] (will retry): cluster not yet healthy I0123 10:40:09.819315 5908 gce_cloud.go:301] Scanning zones: [us-west4-a us-west4-b us-west4-c] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west4-a ControlPlane e2-standard-2 1 1 us-west4 nodes-us-west4-a Node e2-medium 4 4 us-west4 ... skipping 7 lines ... nodes-us-west4-a-l141 node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/calico-node-phbgs system-node-critical pod "calico-node-phbgs" is not ready (calico-node) Validation Failed W0123 10:40:10.391000 5908 validate_cluster.go:232] (will retry): cluster not yet healthy I0123 10:40:20.864924 5908 gce_cloud.go:301] Scanning zones: [us-west4-a us-west4-b us-west4-c] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west4-a ControlPlane e2-standard-2 1 1 us-west4 nodes-us-west4-a Node e2-medium 4 4 us-west4 ... skipping 7 lines ... nodes-us-west4-a-l141 node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/calico-node-phbgs system-node-critical pod "calico-node-phbgs" is not ready (calico-node) Validation Failed W0123 10:40:21.514759 5908 validate_cluster.go:232] (will retry): cluster not yet healthy I0123 10:40:31.931994 5908 gce_cloud.go:301] Scanning zones: [us-west4-a us-west4-b us-west4-c] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west4-a ControlPlane e2-standard-2 1 1 us-west4 nodes-us-west4-a Node e2-medium 4 4 us-west4 ... skipping 7 lines ... nodes-us-west4-a-l141 node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/calico-node-phbgs system-node-critical pod "calico-node-phbgs" is not ready (calico-node) Validation Failed W0123 10:40:32.546094 5908 validate_cluster.go:232] (will retry): cluster not yet healthy I0123 10:40:42.927366 5908 gce_cloud.go:301] Scanning zones: [us-west4-a us-west4-b us-west4-c] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west4-a ControlPlane e2-standard-2 1 1 us-west4 nodes-us-west4-a Node e2-medium 4 4 us-west4 ... skipping 7 lines ... nodes-us-west4-a-l141 node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/calico-node-phbgs system-node-critical pod "calico-node-phbgs" is not ready (calico-node) Validation Failed W0123 10:40:43.932383 5908 validate_cluster.go:232] (will retry): cluster not yet healthy I0123 10:40:54.334349 5908 gce_cloud.go:301] Scanning zones: [us-west4-a us-west4-b us-west4-c] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west4-a ControlPlane e2-standard-2 1 1 us-west4 nodes-us-west4-a Node e2-medium 4 4 us-west4 ... skipping 7 lines ... nodes-us-west4-a-l141 node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/calico-node-phbgs system-node-critical pod "calico-node-phbgs" is not ready (calico-node) Validation Failed W0123 10:40:55.132091 5908 validate_cluster.go:232] (will retry): cluster not yet healthy I0123 10:41:05.595973 5908 gce_cloud.go:301] Scanning zones: [us-west4-a us-west4-b us-west4-c] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west4-a ControlPlane e2-standard-2 1 1 us-west4 nodes-us-west4-a Node e2-medium 4 4 us-west4 ... skipping 7 lines ... nodes-us-west4-a-l141 node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/calico-node-phbgs system-node-critical pod "calico-node-phbgs" is not ready (calico-node) Validation Failed W0123 10:41:06.375256 5908 validate_cluster.go:232] (will retry): cluster not yet healthy I0123 10:41:16.793808 5908 gce_cloud.go:301] Scanning zones: [us-west4-a us-west4-b us-west4-c] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west4-a ControlPlane e2-standard-2 1 1 us-west4 nodes-us-west4-a Node e2-medium 4 4 us-west4 ... skipping 7 lines ... nodes-us-west4-a-l141 node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/calico-node-phbgs system-node-critical pod "calico-node-phbgs" is not ready (calico-node) Validation Failed W0123 10:41:17.435225 5908 validate_cluster.go:232] (will retry): cluster not yet healthy I0123 10:41:27.886564 5908 gce_cloud.go:301] Scanning zones: [us-west4-a us-west4-b us-west4-c] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west4-a ControlPlane e2-standard-2 1 1 us-west4 nodes-us-west4-a Node e2-medium 4 4 us-west4 ... skipping 7 lines ... nodes-us-west4-a-l141 node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/calico-node-phbgs system-node-critical pod "calico-node-phbgs" is not ready (calico-node) Validation Failed W0123 10:41:28.763668 5908 validate_cluster.go:232] (will retry): cluster not yet healthy I0123 10:41:39.063604 5908 gce_cloud.go:301] Scanning zones: [us-west4-a us-west4-b us-west4-c] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west4-a ControlPlane e2-standard-2 1 1 us-west4 nodes-us-west4-a Node e2-medium 4 4 us-west4 ... skipping 7 lines ... nodes-us-west4-a-l141 node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/calico-node-phbgs system-node-critical pod "calico-node-phbgs" is not ready (calico-node) Validation Failed W0123 10:41:39.586161 5908 validate_cluster.go:232] (will retry): cluster not yet healthy I0123 10:41:50.023339 5908 gce_cloud.go:301] Scanning zones: [us-west4-a us-west4-b us-west4-c] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west4-a ControlPlane e2-standard-2 1 1 us-west4 nodes-us-west4-a Node e2-medium 4 4 us-west4 ... skipping 7 lines ... nodes-us-west4-a-l141 node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/calico-node-phbgs system-node-critical pod "calico-node-phbgs" is not ready (calico-node) Validation Failed W0123 10:41:50.622072 5908 validate_cluster.go:232] (will retry): cluster not yet healthy I0123 10:42:00.998978 5908 gce_cloud.go:301] Scanning zones: [us-west4-a us-west4-b us-west4-c] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west4-a ControlPlane e2-standard-2 1 1 us-west4 nodes-us-west4-a Node e2-medium 4 4 us-west4 ... skipping 7 lines ... nodes-us-west4-a-l141 node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/calico-node-phbgs system-node-critical pod "calico-node-phbgs" is not ready (calico-node) Validation Failed W0123 10:42:01.706038 5908 validate_cluster.go:232] (will retry): cluster not yet healthy I0123 10:42:12.130360 5908 gce_cloud.go:301] Scanning zones: [us-west4-a us-west4-b us-west4-c] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west4-a ControlPlane e2-standard-2 1 1 us-west4 nodes-us-west4-a Node e2-medium 4 4 us-west4 ... skipping 7 lines ... nodes-us-west4-a-l141 node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/calico-node-phbgs system-node-critical pod "calico-node-phbgs" is not ready (calico-node) Validation Failed W0123 10:42:12.695267 5908 validate_cluster.go:232] (will retry): cluster not yet healthy I0123 10:42:23.046515 5908 gce_cloud.go:301] Scanning zones: [us-west4-a us-west4-b us-west4-c] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west4-a ControlPlane e2-standard-2 1 1 us-west4 nodes-us-west4-a Node e2-medium 4 4 us-west4 ... skipping 7 lines ... nodes-us-west4-a-l141 node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/calico-node-phbgs system-node-critical pod "calico-node-phbgs" is not ready (calico-node) Validation Failed W0123 10:42:23.697914 5908 validate_cluster.go:232] (will retry): cluster not yet healthy I0123 10:42:34.116823 5908 gce_cloud.go:301] Scanning zones: [us-west4-a us-west4-b us-west4-c] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west4-a ControlPlane e2-standard-2 1 1 us-west4 nodes-us-west4-a Node e2-medium 4 4 us-west4 ... skipping 7 lines ... nodes-us-west4-a-l141 node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/calico-node-phbgs system-node-critical pod "calico-node-phbgs" is not ready (calico-node) Validation Failed W0123 10:42:34.786871 5908 validate_cluster.go:232] (will retry): cluster not yet healthy I0123 10:42:45.214789 5908 gce_cloud.go:301] Scanning zones: [us-west4-a us-west4-b us-west4-c] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west4-a ControlPlane e2-standard-2 1 1 us-west4 nodes-us-west4-a Node e2-medium 4 4 us-west4 ... skipping 7 lines ... nodes-us-west4-a-l141 node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/calico-node-phbgs system-node-critical pod "calico-node-phbgs" is not ready (calico-node) Validation Failed W0123 10:42:45.869004 5908 validate_cluster.go:232] (will retry): cluster not yet healthy I0123 10:42:56.208504 5908 gce_cloud.go:301] Scanning zones: [us-west4-a us-west4-b us-west4-c] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west4-a ControlPlane e2-standard-2 1 1 us-west4 nodes-us-west4-a Node e2-medium 4 4 us-west4 ... skipping 7 lines ... nodes-us-west4-a-l141 node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/calico-node-phbgs system-node-critical pod "calico-node-phbgs" is not ready (calico-node) Validation Failed W0123 10:42:56.809869 5908 validate_cluster.go:232] (will retry): cluster not yet healthy I0123 10:43:07.186432 5908 gce_cloud.go:301] Scanning zones: [us-west4-a us-west4-b us-west4-c] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west4-a ControlPlane e2-standard-2 1 1 us-west4 nodes-us-west4-a Node e2-medium 4 4 us-west4 ... skipping 7 lines ... nodes-us-west4-a-l141 node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/calico-node-phbgs system-node-critical pod "calico-node-phbgs" is not ready (calico-node) Validation Failed W0123 10:43:07.710739 5908 validate_cluster.go:232] (will retry): cluster not yet healthy I0123 10:43:18.110455 5908 gce_cloud.go:301] Scanning zones: [us-west4-a us-west4-b us-west4-c] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west4-a ControlPlane e2-standard-2 1 1 us-west4 nodes-us-west4-a Node e2-medium 4 4 us-west4 ... skipping 7 lines ... nodes-us-west4-a-l141 node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/calico-node-phbgs system-node-critical pod "calico-node-phbgs" is not ready (calico-node) Validation Failed W0123 10:43:18.722607 5908 validate_cluster.go:232] (will retry): cluster not yet healthy I0123 10:43:29.038425 5908 gce_cloud.go:301] Scanning zones: [us-west4-a us-west4-b us-west4-c] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west4-a ControlPlane e2-standard-2 1 1 us-west4 nodes-us-west4-a Node e2-medium 4 4 us-west4 ... skipping 7 lines ... nodes-us-west4-a-l141 node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/calico-node-phbgs system-node-critical pod "calico-node-phbgs" is not ready (calico-node) Validation Failed W0123 10:43:29.674773 5908 validate_cluster.go:232] (will retry): cluster not yet healthy I0123 10:43:40.054470 5908 gce_cloud.go:301] Scanning zones: [us-west4-a us-west4-b us-west4-c] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west4-a ControlPlane e2-standard-2 1 1 us-west4 nodes-us-west4-a Node e2-medium 4 4 us-west4 ... skipping 7 lines ... nodes-us-west4-a-l141 node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/calico-node-phbgs system-node-critical pod "calico-node-phbgs" is not ready (calico-node) Validation Failed W0123 10:43:40.740271 5908 validate_cluster.go:232] (will retry): cluster not yet healthy I0123 10:43:51.181581 5908 gce_cloud.go:301] Scanning zones: [us-west4-a us-west4-b us-west4-c] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west4-a ControlPlane e2-standard-2 1 1 us-west4 nodes-us-west4-a Node e2-medium 4 4 us-west4 ... skipping 7 lines ... nodes-us-west4-a-l141 node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/calico-node-phbgs system-node-critical pod "calico-node-phbgs" is not ready (calico-node) Validation Failed W0123 10:43:51.778935 5908 validate_cluster.go:232] (will retry): cluster not yet healthy I0123 10:44:02.215129 5908 gce_cloud.go:301] Scanning zones: [us-west4-a us-west4-b us-west4-c] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west4-a ControlPlane e2-standard-2 1 1 us-west4 nodes-us-west4-a Node e2-medium 4 4 us-west4 ... skipping 7 lines ... nodes-us-west4-a-l141 node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/calico-node-phbgs system-node-critical pod "calico-node-phbgs" is not ready (calico-node) Validation Failed W0123 10:44:02.784619 5908 validate_cluster.go:232] (will retry): cluster not yet healthy I0123 10:44:05.982765 5544 boskos.go:86] Sending heartbeat to Boskos I0123 10:44:13.201581 5908 gce_cloud.go:301] Scanning zones: [us-west4-a us-west4-b us-west4-c] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west4-a ControlPlane e2-standard-2 1 1 us-west4 ... skipping 8 lines ... nodes-us-west4-a-l141 node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/calico-node-phbgs system-node-critical pod "calico-node-phbgs" is not ready (calico-node) Validation Failed W0123 10:44:14.123855 5908 validate_cluster.go:232] (will retry): cluster not yet healthy I0123 10:44:24.522609 5908 gce_cloud.go:301] Scanning zones: [us-west4-a us-west4-b us-west4-c] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west4-a ControlPlane e2-standard-2 1 1 us-west4 nodes-us-west4-a Node e2-medium 4 4 us-west4 ... skipping 7 lines ... nodes-us-west4-a-l141 node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/calico-node-phbgs system-node-critical pod "calico-node-phbgs" is not ready (calico-node) Validation Failed W0123 10:44:25.383304 5908 validate_cluster.go:232] (will retry): cluster not yet healthy I0123 10:44:35.762269 5908 gce_cloud.go:301] Scanning zones: [us-west4-a us-west4-b us-west4-c] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west4-a ControlPlane e2-standard-2 1 1 us-west4 nodes-us-west4-a Node e2-medium 4 4 us-west4 ... skipping 7 lines ... nodes-us-west4-a-l141 node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/calico-node-phbgs system-node-critical pod "calico-node-phbgs" is not ready (calico-node) Validation Failed W0123 10:44:36.325688 5908 validate_cluster.go:232] (will retry): cluster not yet healthy I0123 10:44:46.692913 5908 gce_cloud.go:301] Scanning zones: [us-west4-a us-west4-b us-west4-c] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west4-a ControlPlane e2-standard-2 1 1 us-west4 nodes-us-west4-a Node e2-medium 4 4 us-west4 ... skipping 7 lines ... nodes-us-west4-a-l141 node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/calico-node-phbgs system-node-critical pod "calico-node-phbgs" is not ready (calico-node) Validation Failed W0123 10:44:47.269782 5908 validate_cluster.go:232] (will retry): cluster not yet healthy I0123 10:44:57.614515 5908 gce_cloud.go:301] Scanning zones: [us-west4-a us-west4-b us-west4-c] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west4-a ControlPlane e2-standard-2 1 1 us-west4 nodes-us-west4-a Node e2-medium 4 4 us-west4 ... skipping 7 lines ... nodes-us-west4-a-l141 node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/calico-node-phbgs system-node-critical pod "calico-node-phbgs" is not ready (calico-node) Validation Failed W0123 10:44:58.208361 5908 validate_cluster.go:232] (will retry): cluster not yet healthy I0123 10:45:08.623745 5908 gce_cloud.go:301] Scanning zones: [us-west4-a us-west4-b us-west4-c] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west4-a ControlPlane e2-standard-2 1 1 us-west4 nodes-us-west4-a Node e2-medium 4 4 us-west4 ... skipping 7 lines ... nodes-us-west4-a-l141 node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/calico-node-phbgs system-node-critical pod "calico-node-phbgs" is not ready (calico-node) Validation Failed W0123 10:45:09.212062 5908 validate_cluster.go:232] (will retry): cluster not yet healthy I0123 10:45:19.628623 5908 gce_cloud.go:301] Scanning zones: [us-west4-a us-west4-b us-west4-c] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west4-a ControlPlane e2-standard-2 1 1 us-west4 nodes-us-west4-a Node e2-medium 4 4 us-west4 ... skipping 7 lines ... nodes-us-west4-a-l141 node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/calico-node-phbgs system-node-critical pod "calico-node-phbgs" is not ready (calico-node) Validation Failed W0123 10:45:20.282871 5908 validate_cluster.go:232] (will retry): cluster not yet healthy I0123 10:45:30.601687 5908 gce_cloud.go:301] Scanning zones: [us-west4-a us-west4-b us-west4-c] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west4-a ControlPlane e2-standard-2 1 1 us-west4 nodes-us-west4-a Node e2-medium 4 4 us-west4 ... skipping 7 lines ... nodes-us-west4-a-l141 node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/calico-node-phbgs system-node-critical pod "calico-node-phbgs" is not ready (calico-node) Validation Failed W0123 10:45:31.440978 5908 validate_cluster.go:232] (will retry): cluster not yet healthy Error: validation failed: wait time exceeded during validation I0123 10:45:41.455839 5544 dumplogs.go:45] /home/prow/go/src/k8s.io/kops/_rundir/810e8320-9b08-11ed-8662-2ea6bdfefe0d/kops toolbox dump --name e2e-e2e-kops-grid-gce-calico-u2004-k26.k8s.local --dir /logs/artifacts --private-key /tmp/kops-ssh1902700557/key --ssh-user prow I0123 10:45:41.455896 5544 local.go:42] ⚙️ /home/prow/go/src/k8s.io/kops/_rundir/810e8320-9b08-11ed-8662-2ea6bdfefe0d/kops toolbox dump --name e2e-e2e-kops-grid-gce-calico-u2004-k26.k8s.local --dir /logs/artifacts --private-key /tmp/kops-ssh1902700557/key --ssh-user prow I0123 10:45:41.922824 5918 gce.go:99] Scanning zones: [us-west4-a us-west4-b us-west4-c] 2023/01/23 10:45:48 Dumping node control-plane-us-west4-a-s8j2 2023/01/23 10:45:55 error dumping node control-plane-us-west4-a-s8j2: error executing command "if command -v kubectl &> /dev/null; then kubectl logs -n kube-system --all-containers -l \"k8s-app=external-dns\"; fi": Process exited with status 1 2023/01/23 10:45:55 error dumping node control-plane-us-west4-a-s8j2: error executing command "if command -v kubectl &> /dev/null; then kubectl logs -n kube-system --all-containers -l \"k8s-app=dns-controller\"; fi": Process exited with status 1 2023/01/23 10:45:55 Dumping node nodes-us-west4-a-304w 2023/01/23 10:45:59 Dumping node nodes-us-west4-a-3n9r 2023/01/23 10:46:03 Dumping node nodes-us-west4-a-4gcc 2023/01/23 10:46:06 Dumping node nodes-us-west4-a-l141 I0123 10:46:10.647806 5544 dumplogs.go:79] /home/prow/go/src/k8s.io/kops/_rundir/810e8320-9b08-11ed-8662-2ea6bdfefe0d/kops get cluster --name e2e-e2e-kops-grid-gce-calico-u2004-k26.k8s.local -o yaml I0123 10:46:10.647857 5544 local.go:42] ⚙️ /home/prow/go/src/k8s.io/kops/_rundir/810e8320-9b08-11ed-8662-2ea6bdfefe0d/kops get cluster --name e2e-e2e-kops-grid-gce-calico-u2004-k26.k8s.local -o yaml ... skipping 154 lines ... Deleted cluster: "e2e-e2e-kops-grid-gce-calico-u2004-k26.k8s.local" I0123 10:48:31.818223 5544 gcs.go:88] gsutil -u gce-gci-upg-1-4-lat-ctl-skew rm -r gs://gce-gci-upg-1-4-lat-ctl-skew-state-81 I0123 10:48:31.818282 5544 local.go:42] ⚙️ gsutil -u gce-gci-upg-1-4-lat-ctl-skew rm -r gs://gce-gci-upg-1-4-lat-ctl-skew-state-81 Removing gs://gce-gci-upg-1-4-lat-ctl-skew-state-81/... I0123 10:48:35.360945 5544 down.go:62] releasing boskos project I0123 10:48:35.368699 5544 boskos.go:83] Boskos heartbeat func received signal to close Error: exit status 1 + EXIT_VALUE=1 + set +o xtrace