e2e-kops-aws-upgrade-k122-kolatest-to-k123-kolatest-many-addons #1617917839031668736
Recent runs || View in Spyglass
Recent runs || View in Spyglass
| Result | FAILURE |
| Tests | 0 failed / 1 succeeded |
| Started | |
| Elapsed | 1h41m |
| Revision | master |
kubetest2 Down
... skipping 192 lines ... I0124 16:13:17.457758 5821 app.go:64] RunDir for this run: "/home/prow/go/src/k8s.io/kops/_rundir/a965ec2f-9c01-11ed-b10d-8a774b882db2" I0124 16:13:17.461529 5821 app.go:128] ID for this run: "a965ec2f-9c01-11ed-b10d-8a774b882db2" I0124 16:13:17.461929 5821 local.go:42] ⚙️ ssh-keygen -t ed25519 -N -q -f /tmp/kops/e2e-4b12d141c5-a5f97.test-cncf-aws.k8s.io/id_ed25519 I0124 16:13:17.479226 5821 dumplogs.go:45] /tmp/kops.PgHvUTiQF toolbox dump --name e2e-4b12d141c5-a5f97.test-cncf-aws.k8s.io --dir /logs/artifacts --private-key /tmp/kops/e2e-4b12d141c5-a5f97.test-cncf-aws.k8s.io/id_ed25519 --ssh-user ubuntu I0124 16:13:17.479291 5821 local.go:42] ⚙️ /tmp/kops.PgHvUTiQF toolbox dump --name e2e-4b12d141c5-a5f97.test-cncf-aws.k8s.io --dir /logs/artifacts --private-key /tmp/kops/e2e-4b12d141c5-a5f97.test-cncf-aws.k8s.io/id_ed25519 --ssh-user ubuntu I0124 16:13:17.555755 5832 featureflag.go:167] Unknown FeatureFlag "SpecOverrideFlag" Error: Cluster.kops.k8s.io "e2e-4b12d141c5-a5f97.test-cncf-aws.k8s.io" not found W0124 16:13:18.077571 5821 down.go:34] Dumping cluster logs at the start of Down() failed: exit status 1 I0124 16:13:18.077715 5821 down.go:48] /tmp/kops.PgHvUTiQF delete cluster --name e2e-4b12d141c5-a5f97.test-cncf-aws.k8s.io --yes I0124 16:13:18.077743 5821 local.go:42] ⚙️ /tmp/kops.PgHvUTiQF delete cluster --name e2e-4b12d141c5-a5f97.test-cncf-aws.k8s.io --yes I0124 16:13:18.149127 5842 featureflag.go:167] Unknown FeatureFlag "SpecOverrideFlag" Error: error reading cluster configuration: Cluster.kops.k8s.io "e2e-4b12d141c5-a5f97.test-cncf-aws.k8s.io" not found Error: exit status 1 + echo 'kubetest2 down failed' kubetest2 down failed + [[ l == \v ]] ++ kops-base-from-marker latest ++ [[ latest =~ ^https: ]] ++ [[ latest == \l\a\t\e\s\t ]] ++ curl -fs https://storage.googleapis.com/kops-ci/bin/latest-ci-updown-green.txt + KOPS_BASE_URL=https://storage.googleapis.com/kops-ci/bin/1.27.0-alpha.2+v1.27.0-alpha.1-260-g68705f39c9 ... skipping 15 lines ... I0124 16:13:21.052382 5879 app.go:64] RunDir for this run: "/home/prow/go/src/k8s.io/kops/_rundir/a965ec2f-9c01-11ed-b10d-8a774b882db2" I0124 16:13:21.058300 5879 app.go:128] ID for this run: "a965ec2f-9c01-11ed-b10d-8a774b882db2" I0124 16:13:21.058446 5879 up.go:44] Cleaning up any leaked resources from previous cluster I0124 16:13:21.058536 5879 dumplogs.go:45] /tmp/kops.w7L0fjueq toolbox dump --name e2e-4b12d141c5-a5f97.test-cncf-aws.k8s.io --dir /logs/artifacts --private-key /tmp/kops/e2e-4b12d141c5-a5f97.test-cncf-aws.k8s.io/id_ed25519 --ssh-user ubuntu I0124 16:13:21.058603 5879 local.go:42] ⚙️ /tmp/kops.w7L0fjueq toolbox dump --name e2e-4b12d141c5-a5f97.test-cncf-aws.k8s.io --dir /logs/artifacts --private-key /tmp/kops/e2e-4b12d141c5-a5f97.test-cncf-aws.k8s.io/id_ed25519 --ssh-user ubuntu I0124 16:13:21.149031 5886 featureflag.go:167] Unknown FeatureFlag "SpecOverrideFlag" Error: Cluster.kops.k8s.io "e2e-4b12d141c5-a5f97.test-cncf-aws.k8s.io" not found W0124 16:13:21.679067 5879 down.go:34] Dumping cluster logs at the start of Down() failed: exit status 1 I0124 16:13:21.679116 5879 down.go:48] /tmp/kops.w7L0fjueq delete cluster --name e2e-4b12d141c5-a5f97.test-cncf-aws.k8s.io --yes I0124 16:13:21.679127 5879 local.go:42] ⚙️ /tmp/kops.w7L0fjueq delete cluster --name e2e-4b12d141c5-a5f97.test-cncf-aws.k8s.io --yes I0124 16:13:21.743361 5898 featureflag.go:167] Unknown FeatureFlag "SpecOverrideFlag" Error: error reading cluster configuration: Cluster.kops.k8s.io "e2e-4b12d141c5-a5f97.test-cncf-aws.k8s.io" not found I0124 16:13:22.250745 5879 http.go:37] curl http://metadata.google.internal/computeMetadata/v1/instance/network-interfaces/0/access-configs/0/external-ip 2023/01/24 16:13:22 failed to get external ip from metadata service: http://metadata.google.internal/computeMetadata/v1/instance/network-interfaces/0/access-configs/0/external-ip returned 404 I0124 16:13:22.269471 5879 http.go:37] curl https://ip.jsb.workers.dev I0124 16:13:22.384509 5879 template.go:58] /tmp/kops.w7L0fjueq toolbox template --template tests/e2e/templates/many-addons.yaml.tmpl --output /tmp/kops-template2548074036/manifest.yaml --values /tmp/kops-template2548074036/values.yaml --name e2e-4b12d141c5-a5f97.test-cncf-aws.k8s.io I0124 16:13:22.384678 5879 local.go:42] ⚙️ /tmp/kops.w7L0fjueq toolbox template --template tests/e2e/templates/many-addons.yaml.tmpl --output /tmp/kops-template2548074036/manifest.yaml --values /tmp/kops-template2548074036/values.yaml --name e2e-4b12d141c5-a5f97.test-cncf-aws.k8s.io I0124 16:13:22.460991 5911 featureflag.go:167] Unknown FeatureFlag "SpecOverrideFlag" I0124 16:13:22.596523 5879 create.go:33] /tmp/kops.w7L0fjueq create --filename /tmp/kops-template2548074036/manifest.yaml --name e2e-4b12d141c5-a5f97.test-cncf-aws.k8s.io I0124 16:13:22.596749 5879 local.go:42] ⚙️ /tmp/kops.w7L0fjueq create --filename /tmp/kops-template2548074036/manifest.yaml --name e2e-4b12d141c5-a5f97.test-cncf-aws.k8s.io ... skipping 80 lines ... NODE STATUS NAME ROLE READY VALIDATION ERRORS KIND NAME MESSAGE dns apiserver Validation Failed The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address. The API DNS IP address is the placeholder address that kops creates: 203.0.113.123. Please wait about 5-10 minutes for a control plane node to start, dns-controller to launch, and DNS to propagate. The protokube container and dns-controller deployment logs may contain more diagnostic information. Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start. Validation Failed W0124 16:14:00.477554 5951 validate_cluster.go:232] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS master-ap-northeast-2a ControlPlane c5.large 1 1 ap-northeast-2a master-ap-northeast-2c ControlPlane c5.large 1 1 ap-northeast-2c master-ap-northeast-2d ControlPlane c5.large 1 1 ap-northeast-2d ... skipping 3 lines ... NODE STATUS NAME ROLE READY VALIDATION ERRORS KIND NAME MESSAGE dns apiserver Validation Failed The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address. The API DNS IP address is the placeholder address that kops creates: 203.0.113.123. Please wait about 5-10 minutes for a control plane node to start, dns-controller to launch, and DNS to propagate. The protokube container and dns-controller deployment logs may contain more diagnostic information. Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start. Validation Failed W0124 16:14:10.515344 5951 validate_cluster.go:232] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS master-ap-northeast-2a ControlPlane c5.large 1 1 ap-northeast-2a master-ap-northeast-2c ControlPlane c5.large 1 1 ap-northeast-2c master-ap-northeast-2d ControlPlane c5.large 1 1 ap-northeast-2d ... skipping 3 lines ... NODE STATUS NAME ROLE READY VALIDATION ERRORS KIND NAME MESSAGE dns apiserver Validation Failed The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address. The API DNS IP address is the placeholder address that kops creates: 203.0.113.123. Please wait about 5-10 minutes for a control plane node to start, dns-controller to launch, and DNS to propagate. The protokube container and dns-controller deployment logs may contain more diagnostic information. Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start. Validation Failed W0124 16:14:20.557185 5951 validate_cluster.go:232] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS master-ap-northeast-2a ControlPlane c5.large 1 1 ap-northeast-2a master-ap-northeast-2c ControlPlane c5.large 1 1 ap-northeast-2c master-ap-northeast-2d ControlPlane c5.large 1 1 ap-northeast-2d ... skipping 3 lines ... NODE STATUS NAME ROLE READY VALIDATION ERRORS KIND NAME MESSAGE dns apiserver Validation Failed The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address. The API DNS IP address is the placeholder address that kops creates: 203.0.113.123. Please wait about 5-10 minutes for a control plane node to start, dns-controller to launch, and DNS to propagate. The protokube container and dns-controller deployment logs may contain more diagnostic information. Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start. Validation Failed W0124 16:14:30.597459 5951 validate_cluster.go:232] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS master-ap-northeast-2a ControlPlane c5.large 1 1 ap-northeast-2a master-ap-northeast-2c ControlPlane c5.large 1 1 ap-northeast-2c master-ap-northeast-2d ControlPlane c5.large 1 1 ap-northeast-2d ... skipping 3 lines ... NODE STATUS NAME ROLE READY VALIDATION ERRORS KIND NAME MESSAGE dns apiserver Validation Failed The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address. The API DNS IP address is the placeholder address that kops creates: 203.0.113.123. Please wait about 5-10 minutes for a control plane node to start, dns-controller to launch, and DNS to propagate. The protokube container and dns-controller deployment logs may contain more diagnostic information. Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start. Validation Failed W0124 16:14:40.636073 5951 validate_cluster.go:232] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS master-ap-northeast-2a ControlPlane c5.large 1 1 ap-northeast-2a master-ap-northeast-2c ControlPlane c5.large 1 1 ap-northeast-2c master-ap-northeast-2d ControlPlane c5.large 1 1 ap-northeast-2d ... skipping 3 lines ... NODE STATUS NAME ROLE READY VALIDATION ERRORS KIND NAME MESSAGE dns apiserver Validation Failed The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address. The API DNS IP address is the placeholder address that kops creates: 203.0.113.123. Please wait about 5-10 minutes for a control plane node to start, dns-controller to launch, and DNS to propagate. The protokube container and dns-controller deployment logs may contain more diagnostic information. Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start. Validation Failed W0124 16:14:50.689058 5951 validate_cluster.go:232] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS master-ap-northeast-2a ControlPlane c5.large 1 1 ap-northeast-2a master-ap-northeast-2c ControlPlane c5.large 1 1 ap-northeast-2c master-ap-northeast-2d ControlPlane c5.large 1 1 ap-northeast-2d ... skipping 3 lines ... NODE STATUS NAME ROLE READY VALIDATION ERRORS KIND NAME MESSAGE dns apiserver Validation Failed The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address. The API DNS IP address is the placeholder address that kops creates: 203.0.113.123. Please wait about 5-10 minutes for a control plane node to start, dns-controller to launch, and DNS to propagate. The protokube container and dns-controller deployment logs may contain more diagnostic information. Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start. Validation Failed W0124 16:15:00.728887 5951 validate_cluster.go:232] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS master-ap-northeast-2a ControlPlane c5.large 1 1 ap-northeast-2a master-ap-northeast-2c ControlPlane c5.large 1 1 ap-northeast-2c master-ap-northeast-2d ControlPlane c5.large 1 1 ap-northeast-2d ... skipping 3 lines ... NODE STATUS NAME ROLE READY VALIDATION ERRORS KIND NAME MESSAGE dns apiserver Validation Failed The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address. The API DNS IP address is the placeholder address that kops creates: 203.0.113.123. Please wait about 5-10 minutes for a control plane node to start, dns-controller to launch, and DNS to propagate. The protokube container and dns-controller deployment logs may contain more diagnostic information. Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start. Validation Failed W0124 16:15:10.767182 5951 validate_cluster.go:232] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS master-ap-northeast-2a ControlPlane c5.large 1 1 ap-northeast-2a master-ap-northeast-2c ControlPlane c5.large 1 1 ap-northeast-2c master-ap-northeast-2d ControlPlane c5.large 1 1 ap-northeast-2d ... skipping 3 lines ... NODE STATUS NAME ROLE READY VALIDATION ERRORS KIND NAME MESSAGE dns apiserver Validation Failed The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address. The API DNS IP address is the placeholder address that kops creates: 203.0.113.123. Please wait about 5-10 minutes for a control plane node to start, dns-controller to launch, and DNS to propagate. The protokube container and dns-controller deployment logs may contain more diagnostic information. Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start. Validation Failed W0124 16:15:20.821576 5951 validate_cluster.go:232] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS master-ap-northeast-2a ControlPlane c5.large 1 1 ap-northeast-2a master-ap-northeast-2c ControlPlane c5.large 1 1 ap-northeast-2c master-ap-northeast-2d ControlPlane c5.large 1 1 ap-northeast-2d ... skipping 3 lines ... NODE STATUS NAME ROLE READY VALIDATION ERRORS KIND NAME MESSAGE dns apiserver Validation Failed The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address. The API DNS IP address is the placeholder address that kops creates: 203.0.113.123. Please wait about 5-10 minutes for a control plane node to start, dns-controller to launch, and DNS to propagate. The protokube container and dns-controller deployment logs may contain more diagnostic information. Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start. Validation Failed W0124 16:15:30.868640 5951 validate_cluster.go:232] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS master-ap-northeast-2a ControlPlane c5.large 1 1 ap-northeast-2a master-ap-northeast-2c ControlPlane c5.large 1 1 ap-northeast-2c master-ap-northeast-2d ControlPlane c5.large 1 1 ap-northeast-2d ... skipping 3 lines ... NODE STATUS NAME ROLE READY VALIDATION ERRORS KIND NAME MESSAGE dns apiserver Validation Failed The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address. The API DNS IP address is the placeholder address that kops creates: 203.0.113.123. Please wait about 5-10 minutes for a control plane node to start, dns-controller to launch, and DNS to propagate. The protokube container and dns-controller deployment logs may contain more diagnostic information. Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start. Validation Failed W0124 16:15:40.907022 5951 validate_cluster.go:232] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS master-ap-northeast-2a ControlPlane c5.large 1 1 ap-northeast-2a master-ap-northeast-2c ControlPlane c5.large 1 1 ap-northeast-2c master-ap-northeast-2d ControlPlane c5.large 1 1 ap-northeast-2d ... skipping 3 lines ... NODE STATUS NAME ROLE READY VALIDATION ERRORS KIND NAME MESSAGE dns apiserver Validation Failed The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address. The API DNS IP address is the placeholder address that kops creates: 203.0.113.123. Please wait about 5-10 minutes for a control plane node to start, dns-controller to launch, and DNS to propagate. The protokube container and dns-controller deployment logs may contain more diagnostic information. Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start. Validation Failed W0124 16:15:50.944849 5951 validate_cluster.go:232] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS master-ap-northeast-2a ControlPlane c5.large 1 1 ap-northeast-2a master-ap-northeast-2c ControlPlane c5.large 1 1 ap-northeast-2c master-ap-northeast-2d ControlPlane c5.large 1 1 ap-northeast-2d ... skipping 3 lines ... NODE STATUS NAME ROLE READY VALIDATION ERRORS KIND NAME MESSAGE dns apiserver Validation Failed The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address. The API DNS IP address is the placeholder address that kops creates: 203.0.113.123. Please wait about 5-10 minutes for a control plane node to start, dns-controller to launch, and DNS to propagate. The protokube container and dns-controller deployment logs may contain more diagnostic information. Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start. Validation Failed W0124 16:16:00.997794 5951 validate_cluster.go:232] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS master-ap-northeast-2a ControlPlane c5.large 1 1 ap-northeast-2a master-ap-northeast-2c ControlPlane c5.large 1 1 ap-northeast-2c master-ap-northeast-2d ControlPlane c5.large 1 1 ap-northeast-2d ... skipping 3 lines ... NODE STATUS NAME ROLE READY VALIDATION ERRORS KIND NAME MESSAGE dns apiserver Validation Failed The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address. The API DNS IP address is the placeholder address that kops creates: 203.0.113.123. Please wait about 5-10 minutes for a control plane node to start, dns-controller to launch, and DNS to propagate. The protokube container and dns-controller deployment logs may contain more diagnostic information. Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start. Validation Failed W0124 16:16:11.040427 5951 validate_cluster.go:232] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS master-ap-northeast-2a ControlPlane c5.large 1 1 ap-northeast-2a master-ap-northeast-2c ControlPlane c5.large 1 1 ap-northeast-2c master-ap-northeast-2d ControlPlane c5.large 1 1 ap-northeast-2d ... skipping 3 lines ... NODE STATUS NAME ROLE READY VALIDATION ERRORS KIND NAME MESSAGE dns apiserver Validation Failed The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address. The API DNS IP address is the placeholder address that kops creates: 203.0.113.123. Please wait about 5-10 minutes for a control plane node to start, dns-controller to launch, and DNS to propagate. The protokube container and dns-controller deployment logs may contain more diagnostic information. Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start. Validation Failed W0124 16:16:21.082747 5951 validate_cluster.go:232] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS master-ap-northeast-2a ControlPlane c5.large 1 1 ap-northeast-2a master-ap-northeast-2c ControlPlane c5.large 1 1 ap-northeast-2c master-ap-northeast-2d ControlPlane c5.large 1 1 ap-northeast-2d ... skipping 3 lines ... NODE STATUS NAME ROLE READY VALIDATION ERRORS KIND NAME MESSAGE dns apiserver Validation Failed The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address. The API DNS IP address is the placeholder address that kops creates: 203.0.113.123. Please wait about 5-10 minutes for a control plane node to start, dns-controller to launch, and DNS to propagate. The protokube container and dns-controller deployment logs may contain more diagnostic information. Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start. Validation Failed W0124 16:16:31.130978 5951 validate_cluster.go:232] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS master-ap-northeast-2a ControlPlane c5.large 1 1 ap-northeast-2a master-ap-northeast-2c ControlPlane c5.large 1 1 ap-northeast-2c master-ap-northeast-2d ControlPlane c5.large 1 1 ap-northeast-2d ... skipping 3 lines ... NODE STATUS NAME ROLE READY VALIDATION ERRORS KIND NAME MESSAGE dns apiserver Validation Failed The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address. The API DNS IP address is the placeholder address that kops creates: 203.0.113.123. Please wait about 5-10 minutes for a control plane node to start, dns-controller to launch, and DNS to propagate. The protokube container and dns-controller deployment logs may contain more diagnostic information. Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start. Validation Failed W0124 16:16:41.172777 5951 validate_cluster.go:232] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS master-ap-northeast-2a ControlPlane c5.large 1 1 ap-northeast-2a master-ap-northeast-2c ControlPlane c5.large 1 1 ap-northeast-2c master-ap-northeast-2d ControlPlane c5.large 1 1 ap-northeast-2d ... skipping 3 lines ... NODE STATUS NAME ROLE READY VALIDATION ERRORS KIND NAME MESSAGE dns apiserver Validation Failed The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address. The API DNS IP address is the placeholder address that kops creates: 203.0.113.123. Please wait about 5-10 minutes for a control plane node to start, dns-controller to launch, and DNS to propagate. The protokube container and dns-controller deployment logs may contain more diagnostic information. Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start. Validation Failed W0124 16:16:51.230718 5951 validate_cluster.go:232] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS master-ap-northeast-2a ControlPlane c5.large 1 1 ap-northeast-2a master-ap-northeast-2c ControlPlane c5.large 1 1 ap-northeast-2c master-ap-northeast-2d ControlPlane c5.large 1 1 ap-northeast-2d ... skipping 3 lines ... NODE STATUS NAME ROLE READY VALIDATION ERRORS KIND NAME MESSAGE dns apiserver Validation Failed The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address. The API DNS IP address is the placeholder address that kops creates: 203.0.113.123. Please wait about 5-10 minutes for a control plane node to start, dns-controller to launch, and DNS to propagate. The protokube container and dns-controller deployment logs may contain more diagnostic information. Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start. Validation Failed W0124 16:17:01.269295 5951 validate_cluster.go:232] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS master-ap-northeast-2a ControlPlane c5.large 1 1 ap-northeast-2a master-ap-northeast-2c ControlPlane c5.large 1 1 ap-northeast-2c master-ap-northeast-2d ControlPlane c5.large 1 1 ap-northeast-2d ... skipping 3 lines ... NODE STATUS NAME ROLE READY VALIDATION ERRORS KIND NAME MESSAGE dns apiserver Validation Failed The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address. The API DNS IP address is the placeholder address that kops creates: 203.0.113.123. Please wait about 5-10 minutes for a control plane node to start, dns-controller to launch, and DNS to propagate. The protokube container and dns-controller deployment logs may contain more diagnostic information. Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start. Validation Failed W0124 16:17:11.301494 5951 validate_cluster.go:232] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS master-ap-northeast-2a ControlPlane c5.large 1 1 ap-northeast-2a master-ap-northeast-2c ControlPlane c5.large 1 1 ap-northeast-2c master-ap-northeast-2d ControlPlane c5.large 1 1 ap-northeast-2d ... skipping 3 lines ... NODE STATUS NAME ROLE READY VALIDATION ERRORS KIND NAME MESSAGE dns apiserver Validation Failed The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address. The API DNS IP address is the placeholder address that kops creates: 203.0.113.123. Please wait about 5-10 minutes for a control plane node to start, dns-controller to launch, and DNS to propagate. The protokube container and dns-controller deployment logs may contain more diagnostic information. Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start. Validation Failed W0124 16:17:21.343373 5951 validate_cluster.go:232] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS master-ap-northeast-2a ControlPlane c5.large 1 1 ap-northeast-2a master-ap-northeast-2c ControlPlane c5.large 1 1 ap-northeast-2c master-ap-northeast-2d ControlPlane c5.large 1 1 ap-northeast-2d ... skipping 3 lines ... NODE STATUS NAME ROLE READY VALIDATION ERRORS KIND NAME MESSAGE dns apiserver Validation Failed The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address. The API DNS IP address is the placeholder address that kops creates: 203.0.113.123. Please wait about 5-10 minutes for a control plane node to start, dns-controller to launch, and DNS to propagate. The protokube container and dns-controller deployment logs may contain more diagnostic information. Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start. Validation Failed W0124 16:17:31.384780 5951 validate_cluster.go:232] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS master-ap-northeast-2a ControlPlane c5.large 1 1 ap-northeast-2a master-ap-northeast-2c ControlPlane c5.large 1 1 ap-northeast-2c master-ap-northeast-2d ControlPlane c5.large 1 1 ap-northeast-2d ... skipping 3 lines ... NODE STATUS NAME ROLE READY VALIDATION ERRORS KIND NAME MESSAGE dns apiserver Validation Failed The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address. The API DNS IP address is the placeholder address that kops creates: 203.0.113.123. Please wait about 5-10 minutes for a control plane node to start, dns-controller to launch, and DNS to propagate. The protokube container and dns-controller deployment logs may contain more diagnostic information. Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start. Validation Failed W0124 16:17:41.432339 5951 validate_cluster.go:232] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS master-ap-northeast-2a ControlPlane c5.large 1 1 ap-northeast-2a master-ap-northeast-2c ControlPlane c5.large 1 1 ap-northeast-2c master-ap-northeast-2d ControlPlane c5.large 1 1 ap-northeast-2d ... skipping 3 lines ... NODE STATUS NAME ROLE READY VALIDATION ERRORS KIND NAME MESSAGE dns apiserver Validation Failed The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address. The API DNS IP address is the placeholder address that kops creates: 203.0.113.123. Please wait about 5-10 minutes for a control plane node to start, dns-controller to launch, and DNS to propagate. The protokube container and dns-controller deployment logs may contain more diagnostic information. Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start. Validation Failed W0124 16:17:51.537868 5951 validate_cluster.go:232] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS master-ap-northeast-2a ControlPlane c5.large 1 1 ap-northeast-2a master-ap-northeast-2c ControlPlane c5.large 1 1 ap-northeast-2c master-ap-northeast-2d ControlPlane c5.large 1 1 ap-northeast-2d ... skipping 3 lines ... NODE STATUS NAME ROLE READY VALIDATION ERRORS KIND NAME MESSAGE dns apiserver Validation Failed The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address. The API DNS IP address is the placeholder address that kops creates: 203.0.113.123. Please wait about 5-10 minutes for a control plane node to start, dns-controller to launch, and DNS to propagate. The protokube container and dns-controller deployment logs may contain more diagnostic information. Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start. Validation Failed W0124 16:18:01.590463 5951 validate_cluster.go:232] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS master-ap-northeast-2a ControlPlane c5.large 1 1 ap-northeast-2a master-ap-northeast-2c ControlPlane c5.large 1 1 ap-northeast-2c master-ap-northeast-2d ControlPlane c5.large 1 1 ap-northeast-2d ... skipping 3 lines ... NODE STATUS NAME ROLE READY VALIDATION ERRORS KIND NAME MESSAGE dns apiserver Validation Failed The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address. The API DNS IP address is the placeholder address that kops creates: 203.0.113.123. Please wait about 5-10 minutes for a control plane node to start, dns-controller to launch, and DNS to propagate. The protokube container and dns-controller deployment logs may contain more diagnostic information. Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start. Validation Failed W0124 16:18:11.630213 5951 validate_cluster.go:232] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS master-ap-northeast-2a ControlPlane c5.large 1 1 ap-northeast-2a master-ap-northeast-2c ControlPlane c5.large 1 1 ap-northeast-2c master-ap-northeast-2d ControlPlane c5.large 1 1 ap-northeast-2d ... skipping 3 lines ... NODE STATUS NAME ROLE READY VALIDATION ERRORS KIND NAME MESSAGE dns apiserver Validation Failed The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address. The API DNS IP address is the placeholder address that kops creates: 203.0.113.123. Please wait about 5-10 minutes for a control plane node to start, dns-controller to launch, and DNS to propagate. The protokube container and dns-controller deployment logs may contain more diagnostic information. Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start. Validation Failed W0124 16:18:21.669851 5951 validate_cluster.go:232] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS master-ap-northeast-2a ControlPlane c5.large 1 1 ap-northeast-2a master-ap-northeast-2c ControlPlane c5.large 1 1 ap-northeast-2c master-ap-northeast-2d ControlPlane c5.large 1 1 ap-northeast-2d ... skipping 3 lines ... NODE STATUS NAME ROLE READY VALIDATION ERRORS KIND NAME MESSAGE dns apiserver Validation Failed The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address. The API DNS IP address is the placeholder address that kops creates: 203.0.113.123. Please wait about 5-10 minutes for a control plane node to start, dns-controller to launch, and DNS to propagate. The protokube container and dns-controller deployment logs may contain more diagnostic information. Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start. Validation Failed W0124 16:18:31.703622 5951 validate_cluster.go:232] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS master-ap-northeast-2a ControlPlane c5.large 1 1 ap-northeast-2a master-ap-northeast-2c ControlPlane c5.large 1 1 ap-northeast-2c master-ap-northeast-2d ControlPlane c5.large 1 1 ap-northeast-2d ... skipping 3 lines ... NODE STATUS NAME ROLE READY VALIDATION ERRORS KIND NAME MESSAGE dns apiserver Validation Failed The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address. The API DNS IP address is the placeholder address that kops creates: 203.0.113.123. Please wait about 5-10 minutes for a control plane node to start, dns-controller to launch, and DNS to propagate. The protokube container and dns-controller deployment logs may contain more diagnostic information. Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start. Validation Failed W0124 16:18:41.766782 5951 validate_cluster.go:232] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS master-ap-northeast-2a ControlPlane c5.large 1 1 ap-northeast-2a master-ap-northeast-2c ControlPlane c5.large 1 1 ap-northeast-2c master-ap-northeast-2d ControlPlane c5.large 1 1 ap-northeast-2d ... skipping 3 lines ... NODE STATUS NAME ROLE READY VALIDATION ERRORS KIND NAME MESSAGE dns apiserver Validation Failed The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address. The API DNS IP address is the placeholder address that kops creates: 203.0.113.123. Please wait about 5-10 minutes for a control plane node to start, dns-controller to launch, and DNS to propagate. The protokube container and dns-controller deployment logs may contain more diagnostic information. Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start. Validation Failed W0124 16:18:51.807157 5951 validate_cluster.go:232] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS master-ap-northeast-2a ControlPlane c5.large 1 1 ap-northeast-2a master-ap-northeast-2c ControlPlane c5.large 1 1 ap-northeast-2c master-ap-northeast-2d ControlPlane c5.large 1 1 ap-northeast-2d ... skipping 3 lines ... NODE STATUS NAME ROLE READY VALIDATION ERRORS KIND NAME MESSAGE dns apiserver Validation Failed The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address. The API DNS IP address is the placeholder address that kops creates: 203.0.113.123. Please wait about 5-10 minutes for a control plane node to start, dns-controller to launch, and DNS to propagate. The protokube container and dns-controller deployment logs may contain more diagnostic information. Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start. Validation Failed W0124 16:19:01.857111 5951 validate_cluster.go:232] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS master-ap-northeast-2a ControlPlane c5.large 1 1 ap-northeast-2a master-ap-northeast-2c ControlPlane c5.large 1 1 ap-northeast-2c master-ap-northeast-2d ControlPlane c5.large 1 1 ap-northeast-2d ... skipping 42 lines ... Pod kube-system/ebs-csi-node-l5lqm system-node-critical pod "ebs-csi-node-l5lqm" is pending Pod kube-system/metrics-server-7fc9b7ff64-4b28b system-cluster-critical pod "metrics-server-7fc9b7ff64-4b28b" is pending Pod kube-system/metrics-server-7fc9b7ff64-cxzkt system-cluster-critical pod "metrics-server-7fc9b7ff64-cxzkt" is pending Pod kube-system/pod-identity-webhook-5896dc97fb-qxjsp system-cluster-critical pod "pod-identity-webhook-5896dc97fb-qxjsp" is pending Pod kube-system/pod-identity-webhook-5896dc97fb-zchtn system-cluster-critical pod "pod-identity-webhook-5896dc97fb-zchtn" is pending Validation Failed W0124 16:19:18.532120 5951 validate_cluster.go:232] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS master-ap-northeast-2a ControlPlane c5.large 1 1 ap-northeast-2a master-ap-northeast-2c ControlPlane c5.large 1 1 ap-northeast-2c master-ap-northeast-2d ControlPlane c5.large 1 1 ap-northeast-2d ... skipping 40 lines ... Pod kube-system/ebs-csi-node-drj78 system-node-critical pod "ebs-csi-node-drj78" is pending Pod kube-system/metrics-server-7fc9b7ff64-4b28b system-cluster-critical pod "metrics-server-7fc9b7ff64-4b28b" is pending Pod kube-system/metrics-server-7fc9b7ff64-cxzkt system-cluster-critical pod "metrics-server-7fc9b7ff64-cxzkt" is pending Pod kube-system/pod-identity-webhook-5896dc97fb-qxjsp system-cluster-critical pod "pod-identity-webhook-5896dc97fb-qxjsp" is pending Pod kube-system/pod-identity-webhook-5896dc97fb-zchtn system-cluster-critical pod "pod-identity-webhook-5896dc97fb-zchtn" is pending Validation Failed W0124 16:19:33.706572 5951 validate_cluster.go:232] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS master-ap-northeast-2a ControlPlane c5.large 1 1 ap-northeast-2a master-ap-northeast-2c ControlPlane c5.large 1 1 ap-northeast-2c master-ap-northeast-2d ControlPlane c5.large 1 1 ap-northeast-2d ... skipping 39 lines ... Pod kube-system/coredns-autoscaler-fd4f775cb-rgdx7 system-cluster-critical pod "coredns-autoscaler-fd4f775cb-rgdx7" is pending Pod kube-system/metrics-server-7fc9b7ff64-4b28b system-cluster-critical pod "metrics-server-7fc9b7ff64-4b28b" is pending Pod kube-system/metrics-server-7fc9b7ff64-cxzkt system-cluster-critical pod "metrics-server-7fc9b7ff64-cxzkt" is pending Pod kube-system/pod-identity-webhook-5896dc97fb-qxjsp system-cluster-critical pod "pod-identity-webhook-5896dc97fb-qxjsp" is pending Pod kube-system/pod-identity-webhook-5896dc97fb-zchtn system-cluster-critical pod "pod-identity-webhook-5896dc97fb-zchtn" is pending Validation Failed W0124 16:19:49.223539 5951 validate_cluster.go:232] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS master-ap-northeast-2a ControlPlane c5.large 1 1 ap-northeast-2a master-ap-northeast-2c ControlPlane c5.large 1 1 ap-northeast-2c master-ap-northeast-2d ControlPlane c5.large 1 1 ap-northeast-2d ... skipping 43 lines ... Pod kube-system/metrics-server-7fc9b7ff64-4b28b system-cluster-critical pod "metrics-server-7fc9b7ff64-4b28b" is pending Pod kube-system/metrics-server-7fc9b7ff64-cxzkt system-cluster-critical pod "metrics-server-7fc9b7ff64-cxzkt" is pending Pod kube-system/node-local-dns-pwc8q system-node-critical pod "node-local-dns-pwc8q" is pending Pod kube-system/pod-identity-webhook-5896dc97fb-qxjsp system-cluster-critical pod "pod-identity-webhook-5896dc97fb-qxjsp" is pending Pod kube-system/pod-identity-webhook-5896dc97fb-zchtn system-cluster-critical pod "pod-identity-webhook-5896dc97fb-zchtn" is pending Validation Failed W0124 16:20:04.860699 5951 validate_cluster.go:232] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS master-ap-northeast-2a ControlPlane c5.large 1 1 ap-northeast-2a master-ap-northeast-2c ControlPlane c5.large 1 1 ap-northeast-2c master-ap-northeast-2d ControlPlane c5.large 1 1 ap-northeast-2d ... skipping 63 lines ... Pod kube-system/node-local-dns-tv592 system-node-critical pod "node-local-dns-tv592" is pending Pod kube-system/node-local-dns-vd7lc system-node-critical pod "node-local-dns-vd7lc" is pending Pod kube-system/node-local-dns-zwdnp system-node-critical pod "node-local-dns-zwdnp" is pending Pod kube-system/pod-identity-webhook-5896dc97fb-qxjsp system-cluster-critical pod "pod-identity-webhook-5896dc97fb-qxjsp" is pending Pod kube-system/pod-identity-webhook-5896dc97fb-zchtn system-cluster-critical pod "pod-identity-webhook-5896dc97fb-zchtn" is pending Validation Failed W0124 16:20:20.224817 5951 validate_cluster.go:232] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS master-ap-northeast-2a ControlPlane c5.large 1 1 ap-northeast-2a master-ap-northeast-2c ControlPlane c5.large 1 1 ap-northeast-2c master-ap-northeast-2d ControlPlane c5.large 1 1 ap-northeast-2d ... skipping 64 lines ... Pod kube-system/node-local-dns-j6pwl system-node-critical pod "node-local-dns-j6pwl" is pending Pod kube-system/node-local-dns-z8wth system-node-critical pod "node-local-dns-z8wth" is pending Pod kube-system/node-local-dns-zwdnp system-node-critical pod "node-local-dns-zwdnp" is pending Pod kube-system/pod-identity-webhook-5896dc97fb-qxjsp system-cluster-critical pod "pod-identity-webhook-5896dc97fb-qxjsp" is pending Pod kube-system/pod-identity-webhook-5896dc97fb-zchtn system-cluster-critical pod "pod-identity-webhook-5896dc97fb-zchtn" is pending Validation Failed W0124 16:20:35.560376 5951 validate_cluster.go:232] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS master-ap-northeast-2a ControlPlane c5.large 1 1 ap-northeast-2a master-ap-northeast-2c ControlPlane c5.large 1 1 ap-northeast-2c master-ap-northeast-2d ControlPlane c5.large 1 1 ap-northeast-2d ... skipping 46 lines ... Pod kube-system/ebs-csi-node-xmjfs system-node-critical pod "ebs-csi-node-xmjfs" is pending Pod kube-system/metrics-server-7fc9b7ff64-4b28b system-cluster-critical pod "metrics-server-7fc9b7ff64-4b28b" is not ready (metrics-server) Pod kube-system/metrics-server-7fc9b7ff64-cxzkt system-cluster-critical pod "metrics-server-7fc9b7ff64-cxzkt" is not ready (metrics-server) Pod kube-system/pod-identity-webhook-5896dc97fb-qxjsp system-cluster-critical pod "pod-identity-webhook-5896dc97fb-qxjsp" is pending Pod kube-system/pod-identity-webhook-5896dc97fb-zchtn system-cluster-critical pod "pod-identity-webhook-5896dc97fb-zchtn" is pending Validation Failed W0124 16:20:50.657955 5951 validate_cluster.go:232] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS master-ap-northeast-2a ControlPlane c5.large 1 1 ap-northeast-2a master-ap-northeast-2c ControlPlane c5.large 1 1 ap-northeast-2c master-ap-northeast-2d ControlPlane c5.large 1 1 ap-northeast-2d ... skipping 28 lines ... Node ip-172-20-2-78.ap-northeast-2.compute.internal control-plane node "ip-172-20-2-78.ap-northeast-2.compute.internal" is missing kube-scheduler pod Pod kube-system/aws-load-balancer-controller-56d878f7d4-79wt5 system-cluster-critical pod "aws-load-balancer-controller-56d878f7d4-79wt5" is pending Pod kube-system/aws-load-balancer-controller-56d878f7d4-lldbj system-cluster-critical pod "aws-load-balancer-controller-56d878f7d4-lldbj" is pending Pod kube-system/pod-identity-webhook-5896dc97fb-qxjsp system-cluster-critical pod "pod-identity-webhook-5896dc97fb-qxjsp" is not ready (pod-identity-webhook) Pod kube-system/pod-identity-webhook-5896dc97fb-zchtn system-cluster-critical pod "pod-identity-webhook-5896dc97fb-zchtn" is pending Validation Failed W0124 16:21:06.000244 5951 validate_cluster.go:232] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS master-ap-northeast-2a ControlPlane c5.large 1 1 ap-northeast-2a master-ap-northeast-2c ControlPlane c5.large 1 1 ap-northeast-2c master-ap-northeast-2d ControlPlane c5.large 1 1 ap-northeast-2d ... skipping 27 lines ... Node ip-172-20-2-78.ap-northeast-2.compute.internal control-plane node "ip-172-20-2-78.ap-northeast-2.compute.internal" is missing kube-controller-manager pod Node ip-172-20-2-78.ap-northeast-2.compute.internal control-plane node "ip-172-20-2-78.ap-northeast-2.compute.internal" is missing kube-scheduler pod Pod kube-system/aws-load-balancer-controller-56d878f7d4-79wt5 system-cluster-critical pod "aws-load-balancer-controller-56d878f7d4-79wt5" is pending Pod kube-system/aws-load-balancer-controller-56d878f7d4-lldbj system-cluster-critical pod "aws-load-balancer-controller-56d878f7d4-lldbj" is pending Pod kube-system/pod-identity-webhook-5896dc97fb-zchtn system-cluster-critical pod "pod-identity-webhook-5896dc97fb-zchtn" is pending Validation Failed W0124 16:21:21.683599 5951 validate_cluster.go:232] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS master-ap-northeast-2a ControlPlane c5.large 1 1 ap-northeast-2a master-ap-northeast-2c ControlPlane c5.large 1 1 ap-northeast-2c master-ap-northeast-2d ControlPlane c5.large 1 1 ap-northeast-2d ... skipping 33 lines ... Pod kube-system/kube-proxy-ip-172-20-0-196.ap-northeast-2.compute.internal system-node-critical pod "kube-proxy-ip-172-20-0-196.ap-northeast-2.compute.internal" is pending Pod kube-system/kube-proxy-ip-172-20-1-112.ap-northeast-2.compute.internal system-node-critical pod "kube-proxy-ip-172-20-1-112.ap-northeast-2.compute.internal" is pending Pod kube-system/kube-proxy-ip-172-20-1-158.ap-northeast-2.compute.internal system-node-critical pod "kube-proxy-ip-172-20-1-158.ap-northeast-2.compute.internal" is pending Pod kube-system/kube-proxy-ip-172-20-1-66.ap-northeast-2.compute.internal system-node-critical pod "kube-proxy-ip-172-20-1-66.ap-northeast-2.compute.internal" is pending Pod kube-system/pod-identity-webhook-5896dc97fb-zchtn system-cluster-critical pod "pod-identity-webhook-5896dc97fb-zchtn" is not ready (pod-identity-webhook) Validation Failed W0124 16:21:36.875956 5951 validate_cluster.go:232] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS master-ap-northeast-2a ControlPlane c5.large 1 1 ap-northeast-2a master-ap-northeast-2c ControlPlane c5.large 1 1 ap-northeast-2c master-ap-northeast-2d ControlPlane c5.large 1 1 ap-northeast-2d ... skipping 25 lines ... Node ip-172-20-1-158.ap-northeast-2.compute.internal control-plane node "ip-172-20-1-158.ap-northeast-2.compute.internal" is missing kube-controller-manager pod Node ip-172-20-2-78.ap-northeast-2.compute.internal control-plane node "ip-172-20-2-78.ap-northeast-2.compute.internal" is missing kube-scheduler pod Pod kube-system/aws-load-balancer-controller-56d878f7d4-79wt5 system-cluster-critical pod "aws-load-balancer-controller-56d878f7d4-79wt5" is pending Pod kube-system/aws-load-balancer-controller-56d878f7d4-lldbj system-cluster-critical pod "aws-load-balancer-controller-56d878f7d4-lldbj" is pending Pod kube-system/kube-proxy-ip-172-20-0-251.ap-northeast-2.compute.internal system-node-critical pod "kube-proxy-ip-172-20-0-251.ap-northeast-2.compute.internal" is pending Validation Failed W0124 16:21:52.473189 5951 validate_cluster.go:232] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS master-ap-northeast-2a ControlPlane c5.large 1 1 ap-northeast-2a master-ap-northeast-2c ControlPlane c5.large 1 1 ap-northeast-2c master-ap-northeast-2d ControlPlane c5.large 1 1 ap-northeast-2d ... skipping 24 lines ... Node ip-172-20-0-251.ap-northeast-2.compute.internal control-plane node "ip-172-20-0-251.ap-northeast-2.compute.internal" is missing kube-scheduler pod Node ip-172-20-1-158.ap-northeast-2.compute.internal control-plane node "ip-172-20-1-158.ap-northeast-2.compute.internal" is missing kube-controller-manager pod Pod kube-system/aws-load-balancer-controller-56d878f7d4-79wt5 system-cluster-critical pod "aws-load-balancer-controller-56d878f7d4-79wt5" is pending Pod kube-system/aws-load-balancer-controller-56d878f7d4-lldbj system-cluster-critical pod "aws-load-balancer-controller-56d878f7d4-lldbj" is pending Pod kube-system/etcd-manager-main-ip-172-20-2-78.ap-northeast-2.compute.internal system-cluster-critical pod "etcd-manager-main-ip-172-20-2-78.ap-northeast-2.compute.internal" is pending Validation Failed W0124 16:22:07.648531 5951 validate_cluster.go:232] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS master-ap-northeast-2a ControlPlane c5.large 1 1 ap-northeast-2a master-ap-northeast-2c ControlPlane c5.large 1 1 ap-northeast-2c master-ap-northeast-2d ControlPlane c5.large 1 1 ap-northeast-2d ... skipping 1220 lines ... WARNING: ignoring DaemonSet-managed Pods: kube-system/cilium-bwzcz, kube-system/ebs-csi-node-l5lqm, kube-system/kops-controller-7knl7, kube-system/node-local-dns-m7fgs evicting pod kube-system/ebs-csi-controller-64bc565d5d-6vxts evicting pod kube-system/cilium-operator-85cf7bd9b9-75nmq evicting pod kube-system/cluster-autoscaler-554f75867b-4hbmd evicting pod kube-system/cilium-operator-85cf7bd9b9-p5vbd evicting pod kube-system/dns-controller-fdc8bc69c-nwvr2 error when evicting pods/"cilium-operator-85cf7bd9b9-p5vbd" -n "kube-system" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. evicting pod kube-system/cilium-operator-85cf7bd9b9-p5vbd error when evicting pods/"cilium-operator-85cf7bd9b9-p5vbd" -n "kube-system" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. evicting pod kube-system/cilium-operator-85cf7bd9b9-p5vbd I0124 16:28:12.053602 6068 instancegroups.go:708] Waiting for 5s for pods to stabilize after draining. I0124 16:28:17.053871 6068 instancegroups.go:625] Stopping instance "i-02f65faebd75d599a", node "ip-172-20-0-251.ap-northeast-2.compute.internal", in group "master-ap-northeast-2a.masters.e2e-4b12d141c5-a5f97.test-cncf-aws.k8s.io" (this may take a while). I0124 16:28:17.470511 6068 instancegroups.go:467] waiting for 15s after terminating instance I0124 16:28:32.470932 6068 instancegroups.go:501] Validating the cluster. I0124 16:28:37.802065 6068 instancegroups.go:560] Cluster did not pass validation, will retry in "30s": InstanceGroup "master-ap-northeast-2a" did not have enough nodes 0 vs 1, system-cluster-critical pod "cilium-operator-85cf7bd9b9-jh2hb" is pending. ... skipping 239 lines ... I0124 17:42:17.212755 6068 instancegroups.go:560] Cluster did not pass validation, will retry in "30s": machine "i-02b15e001d6654af7" has not yet joined cluster, machine "i-042a8f59a751c9c46" has not yet joined cluster, machine "i-0c52bfb8b98c59a9c" has not yet joined cluster. I0124 17:42:52.923794 6068 instancegroups.go:560] Cluster did not pass validation, will retry in "30s": machine "i-02b15e001d6654af7" has not yet joined cluster, machine "i-042a8f59a751c9c46" has not yet joined cluster, machine "i-0c52bfb8b98c59a9c" has not yet joined cluster. I0124 17:43:28.064989 6068 instancegroups.go:560] Cluster did not pass validation, will retry in "30s": machine "i-02b15e001d6654af7" has not yet joined cluster, machine "i-042a8f59a751c9c46" has not yet joined cluster, machine "i-0c52bfb8b98c59a9c" has not yet joined cluster. I0124 17:44:03.263503 6068 instancegroups.go:560] Cluster did not pass validation, will retry in "30s": machine "i-02b15e001d6654af7" has not yet joined cluster, machine "i-042a8f59a751c9c46" has not yet joined cluster, machine "i-0c52bfb8b98c59a9c" has not yet joined cluster. I0124 17:44:38.556252 6068 instancegroups.go:557] Cluster did not pass validation within deadline: machine "i-02b15e001d6654af7" has not yet joined cluster, machine "i-042a8f59a751c9c46" has not yet joined cluster, machine "i-0c52bfb8b98c59a9c" has not yet joined cluster. E0124 17:44:38.556309 6068 instancegroups.go:506] Cluster did not validate within 30m0s E0124 17:44:38.556321 6068 rollingupdate.go:218] failed to roll InstanceGroup "nodes-ap-northeast-2d": error validating cluster after detaching instance: cluster did not validate within a duration of "30m0s" Error: error validating cluster after detaching instance: cluster did not validate within a duration of "30m0s" + kops-finish + kubetest2 kops -v=2 --cloud-provider=aws --cluster-name=e2e-4b12d141c5-a5f97.test-cncf-aws.k8s.io --kops-root=/home/prow/go/src/k8s.io/kops --admin-access= --env=KOPS_FEATURE_FLAGS=SpecOverrideFlag --kops-binary-path=/tmp/kops.PgHvUTiQF --down I0124 17:44:38.608604 6087 featureflag.go:167] Unknown FeatureFlag "SpecOverrideFlag" I0124 17:44:38.609581 6087 app.go:61] The files in RunDir shall not be part of Artifacts I0124 17:44:38.609606 6087 app.go:62] pass rundir-in-artifacts flag True for RunDir to be part of Artifacts I0124 17:44:38.609631 6087 app.go:64] RunDir for this run: "/home/prow/go/src/k8s.io/kops/_rundir/a965ec2f-9c01-11ed-b10d-8a774b882db2" ... skipping 847 lines ...