e2e-kops-grid-gce-kubenet-u2004-k24 #1637635518051127296
Recent runs || View in Spyglass
| Result | FAILURE |
| Tests | 1 failed / 1 succeeded |
| Started | |
| Elapsed | 20m12s |
| Revision | master |
exit status 1
from junit_runner.xml
Filter through log files | View test history on testgrid
kubetest2 Down
... skipping 163 lines ... ..................................Updated [https://www.googleapis.com/compute/v1/projects/k8s-jkns-gce-ubuntu-1-6]. .done. WARNING: No host aliases were added to your SSH configs because you do not have any running instances. Try running this command again after running some instances. I0320 02:03:54.226515 6142 up.go:44] Cleaning up any leaked resources from previous cluster I0320 02:03:54.226684 6142 dumplogs.go:45] /home/prow/go/src/k8s.io/kops/_rundir/2c1e11e8-c6c3-11ed-93d6-92b4ce3fddda/kops toolbox dump --name e2e-e2e-kops-grid-gce-kubenet-u2004-k24.k8s.local --dir /logs/artifacts --private-key /tmp/kops-ssh1868709342/key --ssh-user prow I0320 02:03:54.226717 6142 local.go:42] ⚙️ /home/prow/go/src/k8s.io/kops/_rundir/2c1e11e8-c6c3-11ed-93d6-92b4ce3fddda/kops toolbox dump --name e2e-e2e-kops-grid-gce-kubenet-u2004-k24.k8s.local --dir /logs/artifacts --private-key /tmp/kops-ssh1868709342/key --ssh-user prow Error: Cluster.kops.k8s.io "e2e-e2e-kops-grid-gce-kubenet-u2004-k24.k8s.local" not found W0320 02:03:54.436425 6142 down.go:34] Dumping cluster logs at the start of Down() failed: exit status 1 I0320 02:03:54.436465 6142 down.go:48] /home/prow/go/src/k8s.io/kops/_rundir/2c1e11e8-c6c3-11ed-93d6-92b4ce3fddda/kops delete cluster --name e2e-e2e-kops-grid-gce-kubenet-u2004-k24.k8s.local --yes I0320 02:03:54.436498 6142 local.go:42] ⚙️ /home/prow/go/src/k8s.io/kops/_rundir/2c1e11e8-c6c3-11ed-93d6-92b4ce3fddda/kops delete cluster --name e2e-e2e-kops-grid-gce-kubenet-u2004-k24.k8s.local --yes Error: error reading cluster configuration: Cluster.kops.k8s.io "e2e-e2e-kops-grid-gce-kubenet-u2004-k24.k8s.local" not found I0320 02:03:54.563012 6142 gcs.go:51] gsutil ls -b -p k8s-jkns-gce-ubuntu-1-6 gs://k8s-jkns-gce-ubuntu-1-6-state-2c I0320 02:03:54.563070 6142 local.go:42] ⚙️ gsutil ls -b -p k8s-jkns-gce-ubuntu-1-6 gs://k8s-jkns-gce-ubuntu-1-6-state-2c I0320 02:03:56.491298 6142 gcs.go:70] gsutil mb -p k8s-jkns-gce-ubuntu-1-6 gs://k8s-jkns-gce-ubuntu-1-6-state-2c I0320 02:03:56.491337 6142 local.go:42] ⚙️ gsutil mb -p k8s-jkns-gce-ubuntu-1-6 gs://k8s-jkns-gce-ubuntu-1-6-state-2c Creating gs://k8s-jkns-gce-ubuntu-1-6-state-2c/... I0320 02:03:58.989024 6142 http.go:37] curl http://metadata.google.internal/computeMetadata/v1/instance/network-interfaces/0/access-configs/0/external-ip 2023/03/20 02:03:59 failed to get external ip from metadata service: http://metadata.google.internal/computeMetadata/v1/instance/network-interfaces/0/access-configs/0/external-ip returned 404 I0320 02:03:59.001438 6142 http.go:37] curl https://ip.jsb.workers.dev I0320 02:03:59.103485 6142 up.go:167] /home/prow/go/src/k8s.io/kops/_rundir/2c1e11e8-c6c3-11ed-93d6-92b4ce3fddda/kops create cluster --name e2e-e2e-kops-grid-gce-kubenet-u2004-k24.k8s.local --cloud gce --kubernetes-version https://storage.googleapis.com/kubernetes-release/release/v1.24.12 --ssh-public-key /tmp/kops-ssh1868709342/key.pub --set cluster.spec.nodePortAccess=0.0.0.0/0 --channel=alpha --networking=kubenet --gce-service-account=default --admin-access 34.72.201.195/32 --master-count 1 --master-volume-size 48 --node-count 4 --node-volume-size 48 --zones us-west3-c --master-size e2-standard-2 --project k8s-jkns-gce-ubuntu-1-6 I0320 02:03:59.103525 6142 local.go:42] ⚙️ /home/prow/go/src/k8s.io/kops/_rundir/2c1e11e8-c6c3-11ed-93d6-92b4ce3fddda/kops create cluster --name e2e-e2e-kops-grid-gce-kubenet-u2004-k24.k8s.local --cloud gce --kubernetes-version https://storage.googleapis.com/kubernetes-release/release/v1.24.12 --ssh-public-key /tmp/kops-ssh1868709342/key.pub --set cluster.spec.nodePortAccess=0.0.0.0/0 --channel=alpha --networking=kubenet --gce-service-account=default --admin-access 34.72.201.195/32 --master-count 1 --master-volume-size 48 --node-count 4 --node-volume-size 48 --zones us-west3-c --master-size e2-standard-2 --project k8s-jkns-gce-ubuntu-1-6 I0320 02:03:59.157812 6466 create_cluster.go:882] Using SSH public key: /tmp/kops-ssh1868709342/key.pub I0320 02:03:59.436975 6466 new_cluster.go:574] VMs will be configured to use specified Service Account: default I0320 02:03:59.437070 6466 new_cluster.go:1347] Cloud Provider ID: "gce" ... skipping 379 lines ... I0320 02:04:07.381730 6486 keypair.go:226] Issuing new certificate: "etcd-peers-ca-main" W0320 02:04:07.472754 6486 vfs_keystorereader.go:143] CA private key was not found I0320 02:04:07.563309 6486 keypair.go:226] Issuing new certificate: "kubernetes-ca" I0320 02:04:07.669537 6486 keypair.go:226] Issuing new certificate: "service-account" I0320 02:04:22.970762 6486 executor.go:111] Tasks: 41 done / 68 total; 19 can run I0320 02:04:36.784100 6486 executor.go:111] Tasks: 60 done / 68 total; 6 can run W0320 02:04:41.297910 6486 executor.go:139] error running task "ForwardingRule/api-e2e-e2e-kops-grid-gce-kubenet-u2004-k24-k8s-local" (9m55s remaining to succeed): error creating ForwardingRule "api-e2e-e2e-kops-grid-gce-kubenet-u2004-k24-k8s-local": googleapi: Error 400: The resource 'projects/k8s-jkns-gce-ubuntu-1-6/regions/us-west3/targetPools/api-e2e-e2e-kops-grid-gce-kubenet-u2004-k24-k8s-local' is not ready, resourceNotReady I0320 02:04:41.298002 6486 executor.go:111] Tasks: 65 done / 68 total; 3 can run I0320 02:04:58.172797 6486 executor.go:111] Tasks: 68 done / 68 total; 0 can run I0320 02:04:58.242448 6486 update_cluster.go:323] Exporting kubeconfig for cluster kOps has set your kubectl context to e2e-e2e-kops-grid-gce-kubenet-u2004-k24.k8s.local Cluster is starting. It should be ready in a few minutes. ... skipping 6 lines ... * read about installing addons at: https://kops.sigs.k8s.io/addons. I0320 02:05:08.568642 6142 up.go:251] /home/prow/go/src/k8s.io/kops/_rundir/2c1e11e8-c6c3-11ed-93d6-92b4ce3fddda/kops validate cluster --name e2e-e2e-kops-grid-gce-kubenet-u2004-k24.k8s.local --count 10 --wait 15m0s I0320 02:05:08.568722 6142 local.go:42] ⚙️ /home/prow/go/src/k8s.io/kops/_rundir/2c1e11e8-c6c3-11ed-93d6-92b4ce3fddda/kops validate cluster --name e2e-e2e-kops-grid-gce-kubenet-u2004-k24.k8s.local --count 10 --wait 15m0s Validating cluster e2e-e2e-kops-grid-gce-kubenet-u2004-k24.k8s.local W0320 02:05:38.944856 6506 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://34.106.241.58/api/v1/nodes": dial tcp 34.106.241.58:443: i/o timeout W0320 02:05:48.968548 6506 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://34.106.241.58/api/v1/nodes": dial tcp 34.106.241.58:443: connect: connection refused W0320 02:05:58.990435 6506 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://34.106.241.58/api/v1/nodes": dial tcp 34.106.241.58:443: connect: connection refused W0320 02:06:09.014387 6506 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://34.106.241.58/api/v1/nodes": dial tcp 34.106.241.58:443: connect: connection refused W0320 02:06:19.038039 6506 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://34.106.241.58/api/v1/nodes": dial tcp 34.106.241.58:443: connect: connection refused W0320 02:06:29.061097 6506 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://34.106.241.58/api/v1/nodes": dial tcp 34.106.241.58:443: connect: connection refused W0320 02:06:39.084733 6506 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://34.106.241.58/api/v1/nodes": dial tcp 34.106.241.58:443: connect: connection refused W0320 02:06:49.108902 6506 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://34.106.241.58/api/v1/nodes": dial tcp 34.106.241.58:443: connect: connection refused W0320 02:06:59.131937 6506 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://34.106.241.58/api/v1/nodes": dial tcp 34.106.241.58:443: connect: connection refused W0320 02:07:19.155945 6506 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://34.106.241.58/api/v1/nodes": net/http: TLS handshake timeout W0320 02:07:31.217481 6506 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://34.106.241.58/api/v1/nodes": dial tcp 34.106.241.58:443: connect: connection refused - error from a previous attempt: read tcp 10.60.102.49:60840->34.106.241.58:443: read: connection reset by peer W0320 02:07:51.242511 6506 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://34.106.241.58/api/v1/nodes": net/http: TLS handshake timeout W0320 02:08:01.267108 6506 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://34.106.241.58/api/v1/nodes": dial tcp 34.106.241.58:443: connect: connection refused W0320 02:08:11.291872 6506 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://34.106.241.58/api/v1/nodes": dial tcp 34.106.241.58:443: connect: connection refused W0320 02:08:31.318773 6506 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://34.106.241.58/api/v1/nodes": net/http: TLS handshake timeout W0320 02:08:41.342585 6506 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://34.106.241.58/api/v1/nodes": dial tcp 34.106.241.58:443: connect: connection refused I0320 02:08:44.145557 6142 boskos.go:86] Sending heartbeat to Boskos W0320 02:08:51.367040 6506 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://34.106.241.58/api/v1/nodes": dial tcp 34.106.241.58:443: connect: connection refused W0320 02:09:11.392615 6506 validate_cluster.go:184] (will retry): unexpected error during validation: error listing nodes: Get "https://34.106.241.58/api/v1/nodes": net/http: TLS handshake timeout I0320 02:09:21.865300 6506 gce_cloud.go:301] Scanning zones: [us-west3-a us-west3-b us-west3-c] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west3-c ControlPlane e2-standard-2 1 1 us-west3 nodes-us-west3-c Node e2-medium 4 4 us-west3 ... skipping 5 lines ... Machine https://www.googleapis.com/compute/v1/projects/k8s-jkns-gce-ubuntu-1-6/zones/us-west3-c/instances/control-plane-us-west3-c-11gj machine "https://www.googleapis.com/compute/v1/projects/k8s-jkns-gce-ubuntu-1-6/zones/us-west3-c/instances/control-plane-us-west3-c-11gj" has not yet joined cluster Machine https://www.googleapis.com/compute/v1/projects/k8s-jkns-gce-ubuntu-1-6/zones/us-west3-c/instances/nodes-us-west3-c-0bwp machine "https://www.googleapis.com/compute/v1/projects/k8s-jkns-gce-ubuntu-1-6/zones/us-west3-c/instances/nodes-us-west3-c-0bwp" has not yet joined cluster Machine https://www.googleapis.com/compute/v1/projects/k8s-jkns-gce-ubuntu-1-6/zones/us-west3-c/instances/nodes-us-west3-c-9c95 machine "https://www.googleapis.com/compute/v1/projects/k8s-jkns-gce-ubuntu-1-6/zones/us-west3-c/instances/nodes-us-west3-c-9c95" has not yet joined cluster Machine https://www.googleapis.com/compute/v1/projects/k8s-jkns-gce-ubuntu-1-6/zones/us-west3-c/instances/nodes-us-west3-c-g8k5 machine "https://www.googleapis.com/compute/v1/projects/k8s-jkns-gce-ubuntu-1-6/zones/us-west3-c/instances/nodes-us-west3-c-g8k5" has not yet joined cluster Machine https://www.googleapis.com/compute/v1/projects/k8s-jkns-gce-ubuntu-1-6/zones/us-west3-c/instances/nodes-us-west3-c-mdmr machine "https://www.googleapis.com/compute/v1/projects/k8s-jkns-gce-ubuntu-1-6/zones/us-west3-c/instances/nodes-us-west3-c-mdmr" has not yet joined cluster Validation Failed W0320 02:09:22.489533 6506 validate_cluster.go:232] (will retry): cluster not yet healthy I0320 02:09:32.989538 6506 gce_cloud.go:301] Scanning zones: [us-west3-a us-west3-b us-west3-c] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west3-c ControlPlane e2-standard-2 1 1 us-west3 nodes-us-west3-c Node e2-medium 4 4 us-west3 ... skipping 7 lines ... Machine https://www.googleapis.com/compute/v1/projects/k8s-jkns-gce-ubuntu-1-6/zones/us-west3-c/instances/nodes-us-west3-c-0bwp machine "https://www.googleapis.com/compute/v1/projects/k8s-jkns-gce-ubuntu-1-6/zones/us-west3-c/instances/nodes-us-west3-c-0bwp" has not yet joined cluster Machine https://www.googleapis.com/compute/v1/projects/k8s-jkns-gce-ubuntu-1-6/zones/us-west3-c/instances/nodes-us-west3-c-9c95 machine "https://www.googleapis.com/compute/v1/projects/k8s-jkns-gce-ubuntu-1-6/zones/us-west3-c/instances/nodes-us-west3-c-9c95" has not yet joined cluster Machine https://www.googleapis.com/compute/v1/projects/k8s-jkns-gce-ubuntu-1-6/zones/us-west3-c/instances/nodes-us-west3-c-g8k5 machine "https://www.googleapis.com/compute/v1/projects/k8s-jkns-gce-ubuntu-1-6/zones/us-west3-c/instances/nodes-us-west3-c-g8k5" has not yet joined cluster Machine https://www.googleapis.com/compute/v1/projects/k8s-jkns-gce-ubuntu-1-6/zones/us-west3-c/instances/nodes-us-west3-c-mdmr machine "https://www.googleapis.com/compute/v1/projects/k8s-jkns-gce-ubuntu-1-6/zones/us-west3-c/instances/nodes-us-west3-c-mdmr" has not yet joined cluster Pod kube-system/kube-proxy-control-plane-us-west3-c-11gj system-node-critical pod "kube-proxy-control-plane-us-west3-c-11gj" is pending Validation Failed W0320 02:09:33.627578 6506 validate_cluster.go:232] (will retry): cluster not yet healthy I0320 02:09:44.032975 6506 gce_cloud.go:301] Scanning zones: [us-west3-a us-west3-b us-west3-c] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west3-c ControlPlane e2-standard-2 1 1 us-west3 nodes-us-west3-c Node e2-medium 4 4 us-west3 ... skipping 9 lines ... Machine https://www.googleapis.com/compute/v1/projects/k8s-jkns-gce-ubuntu-1-6/zones/us-west3-c/instances/nodes-us-west3-c-g8k5 machine "https://www.googleapis.com/compute/v1/projects/k8s-jkns-gce-ubuntu-1-6/zones/us-west3-c/instances/nodes-us-west3-c-g8k5" has not yet joined cluster Machine https://www.googleapis.com/compute/v1/projects/k8s-jkns-gce-ubuntu-1-6/zones/us-west3-c/instances/nodes-us-west3-c-mdmr machine "https://www.googleapis.com/compute/v1/projects/k8s-jkns-gce-ubuntu-1-6/zones/us-west3-c/instances/nodes-us-west3-c-mdmr" has not yet joined cluster Pod kube-system/coredns-698bc645cd-qhwpp system-cluster-critical pod "coredns-698bc645cd-qhwpp" is pending Pod kube-system/dns-controller-5ff6f74f8c-bth2v system-cluster-critical pod "dns-controller-5ff6f74f8c-bth2v" is pending Pod kube-system/kube-scheduler-control-plane-us-west3-c-11gj system-cluster-critical pod "kube-scheduler-control-plane-us-west3-c-11gj" is pending Validation Failed W0320 02:09:44.606246 6506 validate_cluster.go:232] (will retry): cluster not yet healthy I0320 02:09:55.087046 6506 gce_cloud.go:301] Scanning zones: [us-west3-a us-west3-b us-west3-c] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west3-c ControlPlane e2-standard-2 1 1 us-west3 nodes-us-west3-c Node e2-medium 4 4 us-west3 ... skipping 11 lines ... Pod kube-system/cloud-controller-manager-9b6sb system-cluster-critical pod "cloud-controller-manager-9b6sb" is pending Pod kube-system/coredns-698bc645cd-qhwpp system-cluster-critical pod "coredns-698bc645cd-qhwpp" is pending Pod kube-system/coredns-autoscaler-5f76bf6c7b-m7cbf system-cluster-critical pod "coredns-autoscaler-5f76bf6c7b-m7cbf" is pending Pod kube-system/dns-controller-5ff6f74f8c-bth2v system-cluster-critical pod "dns-controller-5ff6f74f8c-bth2v" is pending Pod kube-system/kops-controller-kmnts system-cluster-critical pod "kops-controller-kmnts" is pending Validation Failed W0320 02:09:55.658138 6506 validate_cluster.go:232] (will retry): cluster not yet healthy I0320 02:10:06.036248 6506 gce_cloud.go:301] Scanning zones: [us-west3-a us-west3-b us-west3-c] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west3-c ControlPlane e2-standard-2 1 1 us-west3 nodes-us-west3-c Node e2-medium 4 4 us-west3 ... skipping 11 lines ... Pod kube-system/cloud-controller-manager-9b6sb system-cluster-critical pod "cloud-controller-manager-9b6sb" is pending Pod kube-system/coredns-698bc645cd-qhwpp system-cluster-critical pod "coredns-698bc645cd-qhwpp" is pending Pod kube-system/coredns-autoscaler-5f76bf6c7b-m7cbf system-cluster-critical pod "coredns-autoscaler-5f76bf6c7b-m7cbf" is pending Pod kube-system/dns-controller-5ff6f74f8c-bth2v system-cluster-critical pod "dns-controller-5ff6f74f8c-bth2v" is pending Pod kube-system/kops-controller-kmnts system-cluster-critical pod "kops-controller-kmnts" is pending Validation Failed W0320 02:10:06.651017 6506 validate_cluster.go:232] (will retry): cluster not yet healthy I0320 02:10:17.057625 6506 gce_cloud.go:301] Scanning zones: [us-west3-a us-west3-b us-west3-c] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west3-c ControlPlane e2-standard-2 1 1 us-west3 nodes-us-west3-c Node e2-medium 4 4 us-west3 ... skipping 9 lines ... Machine https://www.googleapis.com/compute/v1/projects/k8s-jkns-gce-ubuntu-1-6/zones/us-west3-c/instances/nodes-us-west3-c-g8k5 machine "https://www.googleapis.com/compute/v1/projects/k8s-jkns-gce-ubuntu-1-6/zones/us-west3-c/instances/nodes-us-west3-c-g8k5" has not yet joined cluster Machine https://www.googleapis.com/compute/v1/projects/k8s-jkns-gce-ubuntu-1-6/zones/us-west3-c/instances/nodes-us-west3-c-mdmr machine "https://www.googleapis.com/compute/v1/projects/k8s-jkns-gce-ubuntu-1-6/zones/us-west3-c/instances/nodes-us-west3-c-mdmr" has not yet joined cluster Pod kube-system/cloud-controller-manager-9b6sb system-cluster-critical pod "cloud-controller-manager-9b6sb" is pending Pod kube-system/coredns-698bc645cd-qhwpp system-cluster-critical pod "coredns-698bc645cd-qhwpp" is pending Pod kube-system/coredns-autoscaler-5f76bf6c7b-m7cbf system-cluster-critical pod "coredns-autoscaler-5f76bf6c7b-m7cbf" is pending Validation Failed W0320 02:10:17.538905 6506 validate_cluster.go:232] (will retry): cluster not yet healthy I0320 02:10:27.921822 6506 gce_cloud.go:301] Scanning zones: [us-west3-a us-west3-b us-west3-c] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west3-c ControlPlane e2-standard-2 1 1 us-west3 nodes-us-west3-c Node e2-medium 4 4 us-west3 ... skipping 9 lines ... Machine https://www.googleapis.com/compute/v1/projects/k8s-jkns-gce-ubuntu-1-6/zones/us-west3-c/instances/nodes-us-west3-c-g8k5 machine "https://www.googleapis.com/compute/v1/projects/k8s-jkns-gce-ubuntu-1-6/zones/us-west3-c/instances/nodes-us-west3-c-g8k5" has not yet joined cluster Machine https://www.googleapis.com/compute/v1/projects/k8s-jkns-gce-ubuntu-1-6/zones/us-west3-c/instances/nodes-us-west3-c-mdmr machine "https://www.googleapis.com/compute/v1/projects/k8s-jkns-gce-ubuntu-1-6/zones/us-west3-c/instances/nodes-us-west3-c-mdmr" has not yet joined cluster Node control-plane-us-west3-c-11gj control-plane node "control-plane-us-west3-c-11gj" is missing kube-controller-manager pod Pod kube-system/coredns-698bc645cd-qhwpp system-cluster-critical pod "coredns-698bc645cd-qhwpp" is pending Pod kube-system/coredns-autoscaler-5f76bf6c7b-m7cbf system-cluster-critical pod "coredns-autoscaler-5f76bf6c7b-m7cbf" is pending Validation Failed W0320 02:10:28.527854 6506 validate_cluster.go:232] (will retry): cluster not yet healthy I0320 02:10:38.945780 6506 gce_cloud.go:301] Scanning zones: [us-west3-a us-west3-b us-west3-c] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west3-c ControlPlane e2-standard-2 1 1 us-west3 nodes-us-west3-c Node e2-medium 4 4 us-west3 ... skipping 9 lines ... Machine https://www.googleapis.com/compute/v1/projects/k8s-jkns-gce-ubuntu-1-6/zones/us-west3-c/instances/nodes-us-west3-c-g8k5 machine "https://www.googleapis.com/compute/v1/projects/k8s-jkns-gce-ubuntu-1-6/zones/us-west3-c/instances/nodes-us-west3-c-g8k5" has not yet joined cluster Machine https://www.googleapis.com/compute/v1/projects/k8s-jkns-gce-ubuntu-1-6/zones/us-west3-c/instances/nodes-us-west3-c-mdmr machine "https://www.googleapis.com/compute/v1/projects/k8s-jkns-gce-ubuntu-1-6/zones/us-west3-c/instances/nodes-us-west3-c-mdmr" has not yet joined cluster Node control-plane-us-west3-c-11gj control-plane node "control-plane-us-west3-c-11gj" is missing kube-controller-manager pod Pod kube-system/coredns-698bc645cd-qhwpp system-cluster-critical pod "coredns-698bc645cd-qhwpp" is pending Pod kube-system/coredns-autoscaler-5f76bf6c7b-m7cbf system-cluster-critical pod "coredns-autoscaler-5f76bf6c7b-m7cbf" is pending Validation Failed W0320 02:10:39.433248 6506 validate_cluster.go:232] (will retry): cluster not yet healthy I0320 02:10:49.858071 6506 gce_cloud.go:301] Scanning zones: [us-west3-a us-west3-b us-west3-c] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west3-c ControlPlane e2-standard-2 1 1 us-west3 nodes-us-west3-c Node e2-medium 4 4 us-west3 ... skipping 10 lines ... Machine https://www.googleapis.com/compute/v1/projects/k8s-jkns-gce-ubuntu-1-6/zones/us-west3-c/instances/nodes-us-west3-c-mdmr machine "https://www.googleapis.com/compute/v1/projects/k8s-jkns-gce-ubuntu-1-6/zones/us-west3-c/instances/nodes-us-west3-c-mdmr" has not yet joined cluster Node control-plane-us-west3-c-11gj control-plane node "control-plane-us-west3-c-11gj" is missing kube-controller-manager pod Pod kube-system/coredns-698bc645cd-qhwpp system-cluster-critical pod "coredns-698bc645cd-qhwpp" is pending Pod kube-system/coredns-autoscaler-5f76bf6c7b-m7cbf system-cluster-critical pod "coredns-autoscaler-5f76bf6c7b-m7cbf" is pending Pod kube-system/kube-controller-manager-control-plane-us-west3-c-11gj system-cluster-critical pod "kube-controller-manager-control-plane-us-west3-c-11gj" is pending Validation Failed W0320 02:10:50.295571 6506 validate_cluster.go:232] (will retry): cluster not yet healthy I0320 02:11:00.735192 6506 gce_cloud.go:301] Scanning zones: [us-west3-a us-west3-b us-west3-c] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west3-c ControlPlane e2-standard-2 1 1 us-west3 nodes-us-west3-c Node e2-medium 4 4 us-west3 ... skipping 8 lines ... Machine https://www.googleapis.com/compute/v1/projects/k8s-jkns-gce-ubuntu-1-6/zones/us-west3-c/instances/nodes-us-west3-c-9c95 machine "https://www.googleapis.com/compute/v1/projects/k8s-jkns-gce-ubuntu-1-6/zones/us-west3-c/instances/nodes-us-west3-c-9c95" has not yet joined cluster Machine https://www.googleapis.com/compute/v1/projects/k8s-jkns-gce-ubuntu-1-6/zones/us-west3-c/instances/nodes-us-west3-c-g8k5 machine "https://www.googleapis.com/compute/v1/projects/k8s-jkns-gce-ubuntu-1-6/zones/us-west3-c/instances/nodes-us-west3-c-g8k5" has not yet joined cluster Machine https://www.googleapis.com/compute/v1/projects/k8s-jkns-gce-ubuntu-1-6/zones/us-west3-c/instances/nodes-us-west3-c-mdmr machine "https://www.googleapis.com/compute/v1/projects/k8s-jkns-gce-ubuntu-1-6/zones/us-west3-c/instances/nodes-us-west3-c-mdmr" has not yet joined cluster Pod kube-system/coredns-698bc645cd-qhwpp system-cluster-critical pod "coredns-698bc645cd-qhwpp" is pending Pod kube-system/coredns-autoscaler-5f76bf6c7b-m7cbf system-cluster-critical pod "coredns-autoscaler-5f76bf6c7b-m7cbf" is pending Validation Failed W0320 02:11:01.360960 6506 validate_cluster.go:232] (will retry): cluster not yet healthy I0320 02:11:11.796628 6506 gce_cloud.go:301] Scanning zones: [us-west3-a us-west3-b us-west3-c] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west3-c ControlPlane e2-standard-2 1 1 us-west3 nodes-us-west3-c Node e2-medium 4 4 us-west3 ... skipping 8 lines ... Machine https://www.googleapis.com/compute/v1/projects/k8s-jkns-gce-ubuntu-1-6/zones/us-west3-c/instances/nodes-us-west3-c-9c95 machine "https://www.googleapis.com/compute/v1/projects/k8s-jkns-gce-ubuntu-1-6/zones/us-west3-c/instances/nodes-us-west3-c-9c95" has not yet joined cluster Machine https://www.googleapis.com/compute/v1/projects/k8s-jkns-gce-ubuntu-1-6/zones/us-west3-c/instances/nodes-us-west3-c-g8k5 machine "https://www.googleapis.com/compute/v1/projects/k8s-jkns-gce-ubuntu-1-6/zones/us-west3-c/instances/nodes-us-west3-c-g8k5" has not yet joined cluster Machine https://www.googleapis.com/compute/v1/projects/k8s-jkns-gce-ubuntu-1-6/zones/us-west3-c/instances/nodes-us-west3-c-mdmr machine "https://www.googleapis.com/compute/v1/projects/k8s-jkns-gce-ubuntu-1-6/zones/us-west3-c/instances/nodes-us-west3-c-mdmr" has not yet joined cluster Pod kube-system/coredns-698bc645cd-qhwpp system-cluster-critical pod "coredns-698bc645cd-qhwpp" is pending Pod kube-system/coredns-autoscaler-5f76bf6c7b-m7cbf system-cluster-critical pod "coredns-autoscaler-5f76bf6c7b-m7cbf" is pending Validation Failed W0320 02:11:12.216958 6506 validate_cluster.go:232] (will retry): cluster not yet healthy I0320 02:11:22.705662 6506 gce_cloud.go:301] Scanning zones: [us-west3-a us-west3-b us-west3-c] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west3-c ControlPlane e2-standard-2 1 1 us-west3 nodes-us-west3-c Node e2-medium 4 4 us-west3 ... skipping 8 lines ... Machine https://www.googleapis.com/compute/v1/projects/k8s-jkns-gce-ubuntu-1-6/zones/us-west3-c/instances/nodes-us-west3-c-9c95 machine "https://www.googleapis.com/compute/v1/projects/k8s-jkns-gce-ubuntu-1-6/zones/us-west3-c/instances/nodes-us-west3-c-9c95" has not yet joined cluster Machine https://www.googleapis.com/compute/v1/projects/k8s-jkns-gce-ubuntu-1-6/zones/us-west3-c/instances/nodes-us-west3-c-g8k5 machine "https://www.googleapis.com/compute/v1/projects/k8s-jkns-gce-ubuntu-1-6/zones/us-west3-c/instances/nodes-us-west3-c-g8k5" has not yet joined cluster Machine https://www.googleapis.com/compute/v1/projects/k8s-jkns-gce-ubuntu-1-6/zones/us-west3-c/instances/nodes-us-west3-c-mdmr machine "https://www.googleapis.com/compute/v1/projects/k8s-jkns-gce-ubuntu-1-6/zones/us-west3-c/instances/nodes-us-west3-c-mdmr" has not yet joined cluster Pod kube-system/coredns-698bc645cd-qhwpp system-cluster-critical pod "coredns-698bc645cd-qhwpp" is pending Pod kube-system/coredns-autoscaler-5f76bf6c7b-m7cbf system-cluster-critical pod "coredns-autoscaler-5f76bf6c7b-m7cbf" is pending Validation Failed W0320 02:11:23.173096 6506 validate_cluster.go:232] (will retry): cluster not yet healthy I0320 02:11:33.588520 6506 gce_cloud.go:301] Scanning zones: [us-west3-a us-west3-b us-west3-c] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west3-c ControlPlane e2-standard-2 1 1 us-west3 nodes-us-west3-c Node e2-medium 4 4 us-west3 ... skipping 12 lines ... Node nodes-us-west3-c-g8k5 node "nodes-us-west3-c-g8k5" of role "node" is not ready Pod kube-system/coredns-698bc645cd-qhwpp system-cluster-critical pod "coredns-698bc645cd-qhwpp" is pending Pod kube-system/coredns-autoscaler-5f76bf6c7b-m7cbf system-cluster-critical pod "coredns-autoscaler-5f76bf6c7b-m7cbf" is pending Pod kube-system/kube-proxy-nodes-us-west3-c-g8k5 system-node-critical pod "kube-proxy-nodes-us-west3-c-g8k5" is pending Pod kube-system/metadata-proxy-v0.12-hzzbp system-node-critical pod "metadata-proxy-v0.12-hzzbp" is pending Validation Failed W0320 02:11:34.014956 6506 validate_cluster.go:232] (will retry): cluster not yet healthy I0320 02:11:44.390563 6506 gce_cloud.go:301] Scanning zones: [us-west3-a us-west3-b us-west3-c] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west3-c ControlPlane e2-standard-2 1 1 us-west3 nodes-us-west3-c Node e2-medium 4 4 us-west3 ... skipping 16 lines ... Pod kube-system/coredns-autoscaler-5f76bf6c7b-m7cbf system-cluster-critical pod "coredns-autoscaler-5f76bf6c7b-m7cbf" is pending Pod kube-system/metadata-proxy-v0.12-6n5fz system-node-critical pod "metadata-proxy-v0.12-6n5fz" is pending Pod kube-system/metadata-proxy-v0.12-b9qnr system-node-critical pod "metadata-proxy-v0.12-b9qnr" is pending Pod kube-system/metadata-proxy-v0.12-dtkct system-node-critical pod "metadata-proxy-v0.12-dtkct" is pending Pod kube-system/metadata-proxy-v0.12-hzzbp system-node-critical pod "metadata-proxy-v0.12-hzzbp" is pending Validation Failed W0320 02:11:44.826939 6506 validate_cluster.go:232] (will retry): cluster not yet healthy I0320 02:11:55.285448 6506 gce_cloud.go:301] Scanning zones: [us-west3-a us-west3-b us-west3-c] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west3-c ControlPlane e2-standard-2 1 1 us-west3 nodes-us-west3-c Node e2-medium 4 4 us-west3 ... skipping 8 lines ... VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/coredns-698bc645cd-8m57f system-cluster-critical pod "coredns-698bc645cd-8m57f" is pending Pod kube-system/metadata-proxy-v0.12-dtkct system-node-critical pod "metadata-proxy-v0.12-dtkct" is pending Validation Failed W0320 02:11:55.965586 6506 validate_cluster.go:232] (will retry): cluster not yet healthy I0320 02:12:06.451625 6506 gce_cloud.go:301] Scanning zones: [us-west3-a us-west3-b us-west3-c] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west3-c ControlPlane e2-standard-2 1 1 us-west3 nodes-us-west3-c Node e2-medium 4 4 us-west3 ... skipping 7 lines ... nodes-us-west3-c-mdmr node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/metadata-proxy-v0.12-dtkct system-node-critical pod "metadata-proxy-v0.12-dtkct" is pending Validation Failed W0320 02:12:06.853513 6506 validate_cluster.go:232] (will retry): cluster not yet healthy I0320 02:12:17.269416 6506 gce_cloud.go:301] Scanning zones: [us-west3-a us-west3-b us-west3-c] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west3-c ControlPlane e2-standard-2 1 1 us-west3 nodes-us-west3-c Node e2-medium 4 4 us-west3 ... skipping 7 lines ... nodes-us-west3-c-mdmr node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/metadata-proxy-v0.12-dtkct system-node-critical pod "metadata-proxy-v0.12-dtkct" is pending Validation Failed W0320 02:12:17.643958 6506 validate_cluster.go:232] (will retry): cluster not yet healthy I0320 02:12:28.017968 6506 gce_cloud.go:301] Scanning zones: [us-west3-a us-west3-b us-west3-c] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west3-c ControlPlane e2-standard-2 1 1 us-west3 nodes-us-west3-c Node e2-medium 4 4 us-west3 ... skipping 7 lines ... nodes-us-west3-c-mdmr node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/metadata-proxy-v0.12-dtkct system-node-critical pod "metadata-proxy-v0.12-dtkct" is pending Validation Failed W0320 02:12:28.486965 6506 validate_cluster.go:232] (will retry): cluster not yet healthy I0320 02:12:38.862307 6506 gce_cloud.go:301] Scanning zones: [us-west3-a us-west3-b us-west3-c] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west3-c ControlPlane e2-standard-2 1 1 us-west3 nodes-us-west3-c Node e2-medium 4 4 us-west3 ... skipping 7 lines ... nodes-us-west3-c-mdmr node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/metadata-proxy-v0.12-dtkct system-node-critical pod "metadata-proxy-v0.12-dtkct" is pending Validation Failed W0320 02:12:39.314424 6506 validate_cluster.go:232] (will retry): cluster not yet healthy I0320 02:12:49.771499 6506 gce_cloud.go:301] Scanning zones: [us-west3-a us-west3-b us-west3-c] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west3-c ControlPlane e2-standard-2 1 1 us-west3 nodes-us-west3-c Node e2-medium 4 4 us-west3 ... skipping 8 lines ... VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/kube-proxy-nodes-us-west3-c-0bwp system-node-critical pod "kube-proxy-nodes-us-west3-c-0bwp" is pending Pod kube-system/metadata-proxy-v0.12-dtkct system-node-critical pod "metadata-proxy-v0.12-dtkct" is pending Validation Failed W0320 02:12:50.277334 6506 validate_cluster.go:232] (will retry): cluster not yet healthy I0320 02:13:00.642953 6506 gce_cloud.go:301] Scanning zones: [us-west3-a us-west3-b us-west3-c] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west3-c ControlPlane e2-standard-2 1 1 us-west3 nodes-us-west3-c Node e2-medium 4 4 us-west3 ... skipping 7 lines ... nodes-us-west3-c-mdmr node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/metadata-proxy-v0.12-dtkct system-node-critical pod "metadata-proxy-v0.12-dtkct" is pending Validation Failed W0320 02:13:01.081855 6506 validate_cluster.go:232] (will retry): cluster not yet healthy I0320 02:13:11.531096 6506 gce_cloud.go:301] Scanning zones: [us-west3-a us-west3-b us-west3-c] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west3-c ControlPlane e2-standard-2 1 1 us-west3 nodes-us-west3-c Node e2-medium 4 4 us-west3 ... skipping 7 lines ... nodes-us-west3-c-mdmr node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/metadata-proxy-v0.12-dtkct system-node-critical pod "metadata-proxy-v0.12-dtkct" is pending Validation Failed W0320 02:13:11.917668 6506 validate_cluster.go:232] (will retry): cluster not yet healthy I0320 02:13:22.392023 6506 gce_cloud.go:301] Scanning zones: [us-west3-a us-west3-b us-west3-c] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west3-c ControlPlane e2-standard-2 1 1 us-west3 nodes-us-west3-c Node e2-medium 4 4 us-west3 ... skipping 7 lines ... nodes-us-west3-c-mdmr node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/metadata-proxy-v0.12-dtkct system-node-critical pod "metadata-proxy-v0.12-dtkct" is pending Validation Failed W0320 02:13:22.775631 6506 validate_cluster.go:232] (will retry): cluster not yet healthy I0320 02:13:33.226789 6506 gce_cloud.go:301] Scanning zones: [us-west3-a us-west3-b us-west3-c] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west3-c ControlPlane e2-standard-2 1 1 us-west3 nodes-us-west3-c Node e2-medium 4 4 us-west3 ... skipping 7 lines ... nodes-us-west3-c-mdmr node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/metadata-proxy-v0.12-dtkct system-node-critical pod "metadata-proxy-v0.12-dtkct" is pending Validation Failed W0320 02:13:33.681155 6506 validate_cluster.go:232] (will retry): cluster not yet healthy I0320 02:13:44.100154 6506 gce_cloud.go:301] Scanning zones: [us-west3-a us-west3-b us-west3-c] I0320 02:13:44.158058 6142 boskos.go:86] Sending heartbeat to Boskos INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west3-c ControlPlane e2-standard-2 1 1 us-west3 ... skipping 8 lines ... nodes-us-west3-c-mdmr node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/metadata-proxy-v0.12-dtkct system-node-critical pod "metadata-proxy-v0.12-dtkct" is pending Validation Failed W0320 02:13:44.527058 6506 validate_cluster.go:232] (will retry): cluster not yet healthy I0320 02:13:54.940309 6506 gce_cloud.go:301] Scanning zones: [us-west3-a us-west3-b us-west3-c] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west3-c ControlPlane e2-standard-2 1 1 us-west3 nodes-us-west3-c Node e2-medium 4 4 us-west3 ... skipping 7 lines ... nodes-us-west3-c-mdmr node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/metadata-proxy-v0.12-dtkct system-node-critical pod "metadata-proxy-v0.12-dtkct" is pending Validation Failed W0320 02:13:55.421068 6506 validate_cluster.go:232] (will retry): cluster not yet healthy I0320 02:14:05.891017 6506 gce_cloud.go:301] Scanning zones: [us-west3-a us-west3-b us-west3-c] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west3-c ControlPlane e2-standard-2 1 1 us-west3 nodes-us-west3-c Node e2-medium 4 4 us-west3 ... skipping 7 lines ... nodes-us-west3-c-mdmr node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/metadata-proxy-v0.12-dtkct system-node-critical pod "metadata-proxy-v0.12-dtkct" is pending Validation Failed W0320 02:14:06.322375 6506 validate_cluster.go:232] (will retry): cluster not yet healthy I0320 02:14:16.801579 6506 gce_cloud.go:301] Scanning zones: [us-west3-a us-west3-b us-west3-c] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west3-c ControlPlane e2-standard-2 1 1 us-west3 nodes-us-west3-c Node e2-medium 4 4 us-west3 ... skipping 7 lines ... nodes-us-west3-c-mdmr node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/metadata-proxy-v0.12-dtkct system-node-critical pod "metadata-proxy-v0.12-dtkct" is pending Validation Failed W0320 02:14:17.255654 6506 validate_cluster.go:232] (will retry): cluster not yet healthy I0320 02:14:27.695447 6506 gce_cloud.go:301] Scanning zones: [us-west3-a us-west3-b us-west3-c] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west3-c ControlPlane e2-standard-2 1 1 us-west3 nodes-us-west3-c Node e2-medium 4 4 us-west3 ... skipping 7 lines ... nodes-us-west3-c-mdmr node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/metadata-proxy-v0.12-dtkct system-node-critical pod "metadata-proxy-v0.12-dtkct" is pending Validation Failed W0320 02:14:28.126434 6506 validate_cluster.go:232] (will retry): cluster not yet healthy I0320 02:14:38.569716 6506 gce_cloud.go:301] Scanning zones: [us-west3-a us-west3-b us-west3-c] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west3-c ControlPlane e2-standard-2 1 1 us-west3 nodes-us-west3-c Node e2-medium 4 4 us-west3 ... skipping 7 lines ... nodes-us-west3-c-mdmr node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/metadata-proxy-v0.12-dtkct system-node-critical pod "metadata-proxy-v0.12-dtkct" is pending Validation Failed W0320 02:14:39.007689 6506 validate_cluster.go:232] (will retry): cluster not yet healthy I0320 02:14:49.363780 6506 gce_cloud.go:301] Scanning zones: [us-west3-a us-west3-b us-west3-c] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west3-c ControlPlane e2-standard-2 1 1 us-west3 nodes-us-west3-c Node e2-medium 4 4 us-west3 ... skipping 7 lines ... nodes-us-west3-c-mdmr node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/metadata-proxy-v0.12-dtkct system-node-critical pod "metadata-proxy-v0.12-dtkct" is pending Validation Failed W0320 02:14:49.834641 6506 validate_cluster.go:232] (will retry): cluster not yet healthy I0320 02:15:00.267496 6506 gce_cloud.go:301] Scanning zones: [us-west3-a us-west3-b us-west3-c] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west3-c ControlPlane e2-standard-2 1 1 us-west3 nodes-us-west3-c Node e2-medium 4 4 us-west3 ... skipping 7 lines ... nodes-us-west3-c-mdmr node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/metadata-proxy-v0.12-dtkct system-node-critical pod "metadata-proxy-v0.12-dtkct" is pending Validation Failed W0320 02:15:00.804563 6506 validate_cluster.go:232] (will retry): cluster not yet healthy I0320 02:15:11.260693 6506 gce_cloud.go:301] Scanning zones: [us-west3-a us-west3-b us-west3-c] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west3-c ControlPlane e2-standard-2 1 1 us-west3 nodes-us-west3-c Node e2-medium 4 4 us-west3 ... skipping 7 lines ... nodes-us-west3-c-mdmr node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/metadata-proxy-v0.12-dtkct system-node-critical pod "metadata-proxy-v0.12-dtkct" is pending Validation Failed W0320 02:15:11.804988 6506 validate_cluster.go:232] (will retry): cluster not yet healthy I0320 02:15:22.208167 6506 gce_cloud.go:301] Scanning zones: [us-west3-a us-west3-b us-west3-c] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west3-c ControlPlane e2-standard-2 1 1 us-west3 nodes-us-west3-c Node e2-medium 4 4 us-west3 ... skipping 7 lines ... nodes-us-west3-c-mdmr node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/metadata-proxy-v0.12-dtkct system-node-critical pod "metadata-proxy-v0.12-dtkct" is pending Validation Failed W0320 02:15:22.633240 6506 validate_cluster.go:232] (will retry): cluster not yet healthy I0320 02:15:33.057393 6506 gce_cloud.go:301] Scanning zones: [us-west3-a us-west3-b us-west3-c] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west3-c ControlPlane e2-standard-2 1 1 us-west3 nodes-us-west3-c Node e2-medium 4 4 us-west3 ... skipping 7 lines ... nodes-us-west3-c-mdmr node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/metadata-proxy-v0.12-dtkct system-node-critical pod "metadata-proxy-v0.12-dtkct" is pending Validation Failed W0320 02:15:33.486986 6506 validate_cluster.go:232] (will retry): cluster not yet healthy I0320 02:15:43.899726 6506 gce_cloud.go:301] Scanning zones: [us-west3-a us-west3-b us-west3-c] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west3-c ControlPlane e2-standard-2 1 1 us-west3 nodes-us-west3-c Node e2-medium 4 4 us-west3 ... skipping 7 lines ... nodes-us-west3-c-mdmr node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/metadata-proxy-v0.12-dtkct system-node-critical pod "metadata-proxy-v0.12-dtkct" is pending Validation Failed W0320 02:15:44.381384 6506 validate_cluster.go:232] (will retry): cluster not yet healthy I0320 02:15:54.801956 6506 gce_cloud.go:301] Scanning zones: [us-west3-a us-west3-b us-west3-c] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west3-c ControlPlane e2-standard-2 1 1 us-west3 nodes-us-west3-c Node e2-medium 4 4 us-west3 ... skipping 7 lines ... nodes-us-west3-c-mdmr node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/metadata-proxy-v0.12-dtkct system-node-critical pod "metadata-proxy-v0.12-dtkct" is pending Validation Failed W0320 02:15:55.366822 6506 validate_cluster.go:232] (will retry): cluster not yet healthy I0320 02:16:05.744873 6506 gce_cloud.go:301] Scanning zones: [us-west3-a us-west3-b us-west3-c] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west3-c ControlPlane e2-standard-2 1 1 us-west3 nodes-us-west3-c Node e2-medium 4 4 us-west3 ... skipping 7 lines ... nodes-us-west3-c-mdmr node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/metadata-proxy-v0.12-dtkct system-node-critical pod "metadata-proxy-v0.12-dtkct" is pending Validation Failed W0320 02:16:06.206906 6506 validate_cluster.go:232] (will retry): cluster not yet healthy I0320 02:16:16.629576 6506 gce_cloud.go:301] Scanning zones: [us-west3-a us-west3-b us-west3-c] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west3-c ControlPlane e2-standard-2 1 1 us-west3 nodes-us-west3-c Node e2-medium 4 4 us-west3 ... skipping 7 lines ... nodes-us-west3-c-mdmr node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/metadata-proxy-v0.12-dtkct system-node-critical pod "metadata-proxy-v0.12-dtkct" is pending Validation Failed W0320 02:16:17.094348 6506 validate_cluster.go:232] (will retry): cluster not yet healthy I0320 02:16:27.491955 6506 gce_cloud.go:301] Scanning zones: [us-west3-a us-west3-b us-west3-c] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west3-c ControlPlane e2-standard-2 1 1 us-west3 nodes-us-west3-c Node e2-medium 4 4 us-west3 ... skipping 7 lines ... nodes-us-west3-c-mdmr node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/metadata-proxy-v0.12-dtkct system-node-critical pod "metadata-proxy-v0.12-dtkct" is pending Validation Failed W0320 02:16:27.907241 6506 validate_cluster.go:232] (will retry): cluster not yet healthy I0320 02:16:38.313646 6506 gce_cloud.go:301] Scanning zones: [us-west3-a us-west3-b us-west3-c] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west3-c ControlPlane e2-standard-2 1 1 us-west3 nodes-us-west3-c Node e2-medium 4 4 us-west3 ... skipping 7 lines ... nodes-us-west3-c-mdmr node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/metadata-proxy-v0.12-dtkct system-node-critical pod "metadata-proxy-v0.12-dtkct" is pending Validation Failed W0320 02:16:38.691393 6506 validate_cluster.go:232] (will retry): cluster not yet healthy I0320 02:16:49.103483 6506 gce_cloud.go:301] Scanning zones: [us-west3-a us-west3-b us-west3-c] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west3-c ControlPlane e2-standard-2 1 1 us-west3 nodes-us-west3-c Node e2-medium 4 4 us-west3 ... skipping 7 lines ... nodes-us-west3-c-mdmr node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/metadata-proxy-v0.12-dtkct system-node-critical pod "metadata-proxy-v0.12-dtkct" is pending Validation Failed W0320 02:16:49.559903 6506 validate_cluster.go:232] (will retry): cluster not yet healthy I0320 02:16:59.945136 6506 gce_cloud.go:301] Scanning zones: [us-west3-a us-west3-b us-west3-c] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west3-c ControlPlane e2-standard-2 1 1 us-west3 nodes-us-west3-c Node e2-medium 4 4 us-west3 ... skipping 7 lines ... nodes-us-west3-c-mdmr node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/metadata-proxy-v0.12-dtkct system-node-critical pod "metadata-proxy-v0.12-dtkct" is pending Validation Failed W0320 02:17:00.370363 6506 validate_cluster.go:232] (will retry): cluster not yet healthy I0320 02:17:10.795884 6506 gce_cloud.go:301] Scanning zones: [us-west3-a us-west3-b us-west3-c] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west3-c ControlPlane e2-standard-2 1 1 us-west3 nodes-us-west3-c Node e2-medium 4 4 us-west3 ... skipping 7 lines ... nodes-us-west3-c-mdmr node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/metadata-proxy-v0.12-dtkct system-node-critical pod "metadata-proxy-v0.12-dtkct" is pending Validation Failed W0320 02:17:11.255217 6506 validate_cluster.go:232] (will retry): cluster not yet healthy I0320 02:17:21.758145 6506 gce_cloud.go:301] Scanning zones: [us-west3-a us-west3-b us-west3-c] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west3-c ControlPlane e2-standard-2 1 1 us-west3 nodes-us-west3-c Node e2-medium 4 4 us-west3 ... skipping 7 lines ... nodes-us-west3-c-mdmr node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/metadata-proxy-v0.12-dtkct system-node-critical pod "metadata-proxy-v0.12-dtkct" is pending Validation Failed W0320 02:17:22.180753 6506 validate_cluster.go:232] (will retry): cluster not yet healthy I0320 02:17:32.594584 6506 gce_cloud.go:301] Scanning zones: [us-west3-a us-west3-b us-west3-c] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west3-c ControlPlane e2-standard-2 1 1 us-west3 nodes-us-west3-c Node e2-medium 4 4 us-west3 ... skipping 7 lines ... nodes-us-west3-c-mdmr node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/metadata-proxy-v0.12-dtkct system-node-critical pod "metadata-proxy-v0.12-dtkct" is pending Validation Failed W0320 02:17:33.032713 6506 validate_cluster.go:232] (will retry): cluster not yet healthy I0320 02:17:43.414434 6506 gce_cloud.go:301] Scanning zones: [us-west3-a us-west3-b us-west3-c] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west3-c ControlPlane e2-standard-2 1 1 us-west3 nodes-us-west3-c Node e2-medium 4 4 us-west3 ... skipping 7 lines ... nodes-us-west3-c-mdmr node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/metadata-proxy-v0.12-dtkct system-node-critical pod "metadata-proxy-v0.12-dtkct" is pending Validation Failed W0320 02:17:43.830350 6506 validate_cluster.go:232] (will retry): cluster not yet healthy I0320 02:17:54.303086 6506 gce_cloud.go:301] Scanning zones: [us-west3-a us-west3-b us-west3-c] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west3-c ControlPlane e2-standard-2 1 1 us-west3 nodes-us-west3-c Node e2-medium 4 4 us-west3 ... skipping 7 lines ... nodes-us-west3-c-mdmr node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/metadata-proxy-v0.12-dtkct system-node-critical pod "metadata-proxy-v0.12-dtkct" is pending Validation Failed W0320 02:17:54.757255 6506 validate_cluster.go:232] (will retry): cluster not yet healthy I0320 02:18:05.171701 6506 gce_cloud.go:301] Scanning zones: [us-west3-a us-west3-b us-west3-c] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west3-c ControlPlane e2-standard-2 1 1 us-west3 nodes-us-west3-c Node e2-medium 4 4 us-west3 ... skipping 7 lines ... nodes-us-west3-c-mdmr node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/metadata-proxy-v0.12-dtkct system-node-critical pod "metadata-proxy-v0.12-dtkct" is pending Validation Failed W0320 02:18:05.622925 6506 validate_cluster.go:232] (will retry): cluster not yet healthy I0320 02:18:16.048612 6506 gce_cloud.go:301] Scanning zones: [us-west3-a us-west3-b us-west3-c] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west3-c ControlPlane e2-standard-2 1 1 us-west3 nodes-us-west3-c Node e2-medium 4 4 us-west3 ... skipping 7 lines ... nodes-us-west3-c-mdmr node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/metadata-proxy-v0.12-dtkct system-node-critical pod "metadata-proxy-v0.12-dtkct" is pending Validation Failed W0320 02:18:16.465457 6506 validate_cluster.go:232] (will retry): cluster not yet healthy I0320 02:18:26.869118 6506 gce_cloud.go:301] Scanning zones: [us-west3-a us-west3-b us-west3-c] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west3-c ControlPlane e2-standard-2 1 1 us-west3 nodes-us-west3-c Node e2-medium 4 4 us-west3 ... skipping 7 lines ... nodes-us-west3-c-mdmr node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/metadata-proxy-v0.12-dtkct system-node-critical pod "metadata-proxy-v0.12-dtkct" is pending Validation Failed W0320 02:18:27.357162 6506 validate_cluster.go:232] (will retry): cluster not yet healthy I0320 02:18:37.742865 6506 gce_cloud.go:301] Scanning zones: [us-west3-a us-west3-b us-west3-c] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west3-c ControlPlane e2-standard-2 1 1 us-west3 nodes-us-west3-c Node e2-medium 4 4 us-west3 ... skipping 7 lines ... nodes-us-west3-c-mdmr node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/metadata-proxy-v0.12-dtkct system-node-critical pod "metadata-proxy-v0.12-dtkct" is pending Validation Failed W0320 02:18:38.183201 6506 validate_cluster.go:232] (will retry): cluster not yet healthy I0320 02:18:44.170839 6142 boskos.go:86] Sending heartbeat to Boskos I0320 02:18:48.681775 6506 gce_cloud.go:301] Scanning zones: [us-west3-a us-west3-b us-west3-c] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west3-c ControlPlane e2-standard-2 1 1 us-west3 ... skipping 8 lines ... nodes-us-west3-c-mdmr node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/metadata-proxy-v0.12-dtkct system-node-critical pod "metadata-proxy-v0.12-dtkct" is pending Validation Failed W0320 02:18:49.115526 6506 validate_cluster.go:232] (will retry): cluster not yet healthy I0320 02:18:59.497527 6506 gce_cloud.go:301] Scanning zones: [us-west3-a us-west3-b us-west3-c] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west3-c ControlPlane e2-standard-2 1 1 us-west3 nodes-us-west3-c Node e2-medium 4 4 us-west3 ... skipping 7 lines ... nodes-us-west3-c-mdmr node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/metadata-proxy-v0.12-dtkct system-node-critical pod "metadata-proxy-v0.12-dtkct" is pending Validation Failed W0320 02:18:59.906259 6506 validate_cluster.go:232] (will retry): cluster not yet healthy I0320 02:19:10.333178 6506 gce_cloud.go:301] Scanning zones: [us-west3-a us-west3-b us-west3-c] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west3-c ControlPlane e2-standard-2 1 1 us-west3 nodes-us-west3-c Node e2-medium 4 4 us-west3 ... skipping 7 lines ... nodes-us-west3-c-mdmr node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/metadata-proxy-v0.12-dtkct system-node-critical pod "metadata-proxy-v0.12-dtkct" is pending Validation Failed W0320 02:19:10.846905 6506 validate_cluster.go:232] (will retry): cluster not yet healthy I0320 02:19:21.293038 6506 gce_cloud.go:301] Scanning zones: [us-west3-a us-west3-b us-west3-c] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west3-c ControlPlane e2-standard-2 1 1 us-west3 nodes-us-west3-c Node e2-medium 4 4 us-west3 ... skipping 7 lines ... nodes-us-west3-c-mdmr node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/metadata-proxy-v0.12-dtkct system-node-critical pod "metadata-proxy-v0.12-dtkct" is pending Validation Failed W0320 02:19:21.728754 6506 validate_cluster.go:232] (will retry): cluster not yet healthy I0320 02:19:32.135564 6506 gce_cloud.go:301] Scanning zones: [us-west3-a us-west3-b us-west3-c] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west3-c ControlPlane e2-standard-2 1 1 us-west3 nodes-us-west3-c Node e2-medium 4 4 us-west3 ... skipping 7 lines ... nodes-us-west3-c-mdmr node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/metadata-proxy-v0.12-dtkct system-node-critical pod "metadata-proxy-v0.12-dtkct" is pending Validation Failed W0320 02:19:32.593754 6506 validate_cluster.go:232] (will retry): cluster not yet healthy I0320 02:19:43.051021 6506 gce_cloud.go:301] Scanning zones: [us-west3-a us-west3-b us-west3-c] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west3-c ControlPlane e2-standard-2 1 1 us-west3 nodes-us-west3-c Node e2-medium 4 4 us-west3 ... skipping 7 lines ... nodes-us-west3-c-mdmr node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/metadata-proxy-v0.12-dtkct system-node-critical pod "metadata-proxy-v0.12-dtkct" is pending Validation Failed W0320 02:19:43.481667 6506 validate_cluster.go:232] (will retry): cluster not yet healthy I0320 02:19:53.919556 6506 gce_cloud.go:301] Scanning zones: [us-west3-a us-west3-b us-west3-c] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west3-c ControlPlane e2-standard-2 1 1 us-west3 nodes-us-west3-c Node e2-medium 4 4 us-west3 ... skipping 7 lines ... nodes-us-west3-c-mdmr node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/metadata-proxy-v0.12-dtkct system-node-critical pod "metadata-proxy-v0.12-dtkct" is pending Validation Failed W0320 02:19:54.410842 6506 validate_cluster.go:232] (will retry): cluster not yet healthy I0320 02:20:04.813268 6506 gce_cloud.go:301] Scanning zones: [us-west3-a us-west3-b us-west3-c] INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-west3-c ControlPlane e2-standard-2 1 1 us-west3 nodes-us-west3-c Node e2-medium 4 4 us-west3 ... skipping 7 lines ... nodes-us-west3-c-mdmr node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/metadata-proxy-v0.12-dtkct system-node-critical pod "metadata-proxy-v0.12-dtkct" is pending Validation Failed W0320 02:20:05.223876 6506 validate_cluster.go:232] (will retry): cluster not yet healthy Error: validation failed: wait time exceeded during validation I0320 02:20:15.233952 6142 dumplogs.go:45] /home/prow/go/src/k8s.io/kops/_rundir/2c1e11e8-c6c3-11ed-93d6-92b4ce3fddda/kops toolbox dump --name e2e-e2e-kops-grid-gce-kubenet-u2004-k24.k8s.local --dir /logs/artifacts --private-key /tmp/kops-ssh1868709342/key --ssh-user prow I0320 02:20:15.234040 6142 local.go:42] ⚙️ /home/prow/go/src/k8s.io/kops/_rundir/2c1e11e8-c6c3-11ed-93d6-92b4ce3fddda/kops toolbox dump --name e2e-e2e-kops-grid-gce-kubenet-u2004-k24.k8s.local --dir /logs/artifacts --private-key /tmp/kops-ssh1868709342/key --ssh-user prow I0320 02:20:15.696812 6516 gce.go:99] Scanning zones: [us-west3-a us-west3-b us-west3-c] 2023/03/20 02:20:19 Dumping node control-plane-us-west3-c-11gj 2023/03/20 02:20:23 error dumping node control-plane-us-west3-c-11gj: error executing command "if command -v kubectl &> /dev/null; then kubectl logs -n kube-system --all-containers -l \"k8s-app=external-dns\"; fi": Process exited with status 1 2023/03/20 02:20:23 error dumping node control-plane-us-west3-c-11gj: error executing command "if command -v kubectl &> /dev/null; then kubectl logs -n kube-system --all-containers -l \"k8s-app=dns-controller\"; fi": Process exited with status 1 2023/03/20 02:20:23 Dumping node nodes-us-west3-c-0bwp 2023/03/20 02:20:25 Dumping node nodes-us-west3-c-9c95 2023/03/20 02:20:27 Dumping node nodes-us-west3-c-g8k5 2023/03/20 02:20:30 Dumping node nodes-us-west3-c-mdmr I0320 02:20:32.931049 6142 dumplogs.go:79] /home/prow/go/src/k8s.io/kops/_rundir/2c1e11e8-c6c3-11ed-93d6-92b4ce3fddda/kops get cluster --name e2e-e2e-kops-grid-gce-kubenet-u2004-k24.k8s.local -o yaml I0320 02:20:32.931101 6142 local.go:42] ⚙️ /home/prow/go/src/k8s.io/kops/_rundir/2c1e11e8-c6c3-11ed-93d6-92b4ce3fddda/kops get cluster --name e2e-e2e-kops-grid-gce-kubenet-u2004-k24.k8s.local -o yaml ... skipping 156 lines ... Deleted cluster: "e2e-e2e-kops-grid-gce-kubenet-u2004-k24.k8s.local" I0320 02:22:44.739314 6142 gcs.go:88] gsutil -u k8s-jkns-gce-ubuntu-1-6 rm -r gs://k8s-jkns-gce-ubuntu-1-6-state-2c I0320 02:22:44.739365 6142 local.go:42] ⚙️ gsutil -u k8s-jkns-gce-ubuntu-1-6 rm -r gs://k8s-jkns-gce-ubuntu-1-6-state-2c Removing gs://k8s-jkns-gce-ubuntu-1-6-state-2c/... I0320 02:22:47.420450 6142 down.go:62] releasing boskos project I0320 02:22:47.430677 6142 boskos.go:83] Boskos heartbeat func received signal to close Error: exit status 1 + EXIT_VALUE=1 + set +o xtrace