This job view page is being replaced by Spyglass soon. Check out the new job view.
PRjustinsb: Prune addons via labels
ResultFAILURE
Tests 0 failed / 0 succeeded
Started2021-10-14 12:29
Elapsed35m10s
Revisionb037db120a3e44859952db81397bb76ee363dfb7
Refs 12156

No Test Failures!


Error lines from build-log.txt

... skipping 167 lines ...
I1014 12:30:11.974537    4903 dumplogs.go:40] /tmp/kops.AZKChVvET toolbox dump --name e2e-8f6cf1d295-e5e21.test-cncf-aws.k8s.io --dir /logs/artifacts --private-key /etc/aws-ssh/aws-ssh-private --ssh-user ubuntu
I1014 12:30:11.989672    4910 featureflag.go:165] FeatureFlag "SpecOverrideFlag"=true
I1014 12:30:11.989751    4910 featureflag.go:165] FeatureFlag "AlphaAllowGCE"=true
I1014 12:30:11.989756    4910 featureflag.go:165] FeatureFlag "SpecOverrideFlag"=true

Cluster.kops.k8s.io "e2e-8f6cf1d295-e5e21.test-cncf-aws.k8s.io" not found
W1014 12:30:12.492039    4903 down.go:34] Dumping cluster logs at the start of Down() failed: exit status 1
I1014 12:30:12.492109    4903 down.go:48] /tmp/kops.AZKChVvET delete cluster --name e2e-8f6cf1d295-e5e21.test-cncf-aws.k8s.io --yes
I1014 12:30:12.513578    4921 featureflag.go:165] FeatureFlag "SpecOverrideFlag"=true
I1014 12:30:12.513693    4921 featureflag.go:165] FeatureFlag "AlphaAllowGCE"=true
I1014 12:30:12.513699    4921 featureflag.go:165] FeatureFlag "SpecOverrideFlag"=true

error reading cluster configuration: Cluster.kops.k8s.io "e2e-8f6cf1d295-e5e21.test-cncf-aws.k8s.io" not found
I1014 12:30:12.996396    4903 http.go:37] curl http://metadata.google.internal/computeMetadata/v1/instance/network-interfaces/0/access-configs/0/external-ip
2021/10/14 12:30:13 failed to get external ip from metadata service: http://metadata.google.internal/computeMetadata/v1/instance/network-interfaces/0/access-configs/0/external-ip returned 404
I1014 12:30:13.004554    4903 http.go:37] curl https://ip.jsb.workers.dev
I1014 12:30:13.099171    4903 up.go:144] /tmp/kops.AZKChVvET create cluster --name e2e-8f6cf1d295-e5e21.test-cncf-aws.k8s.io --cloud aws --kubernetes-version 1.21.0 --ssh-public-key /etc/aws-ssh/aws-ssh-public --override cluster.spec.nodePortAccess=0.0.0.0/0 --yes --override=cluster.spec.nodeTerminationHandler.enabled=true --admin-access 34.72.114.179/32 --master-count 1 --master-volume-size 48 --node-count 4 --node-volume-size 48 --zones us-west-1a --master-size c5.large
I1014 12:30:13.114547    4935 featureflag.go:165] FeatureFlag "SpecOverrideFlag"=true
I1014 12:30:13.114642    4935 featureflag.go:165] FeatureFlag "AlphaAllowGCE"=true
I1014 12:30:13.114647    4935 featureflag.go:165] FeatureFlag "SpecOverrideFlag"=true
I1014 12:30:13.157912    4935 create_cluster.go:728] Using SSH public key: /etc/aws-ssh/aws-ssh-public
... skipping 44 lines ...
I1014 12:30:38.241004    4903 up.go:181] /tmp/kops.AZKChVvET validate cluster --name e2e-8f6cf1d295-e5e21.test-cncf-aws.k8s.io --count 10 --wait 15m0s
I1014 12:30:38.257060    4955 featureflag.go:165] FeatureFlag "SpecOverrideFlag"=true
I1014 12:30:38.257136    4955 featureflag.go:165] FeatureFlag "AlphaAllowGCE"=true
I1014 12:30:38.257141    4955 featureflag.go:165] FeatureFlag "SpecOverrideFlag"=true
Validating cluster e2e-8f6cf1d295-e5e21.test-cncf-aws.k8s.io

W1014 12:30:39.316991    4955 validate_cluster.go:173] (will retry): unexpected error during validation: unable to resolve Kubernetes cluster API URL dns: lookup api.e2e-8f6cf1d295-e5e21.test-cncf-aws.k8s.io on 10.63.240.10:53: no such host
W1014 12:30:49.346211    4955 validate_cluster.go:173] (will retry): unexpected error during validation: unable to resolve Kubernetes cluster API URL dns: lookup api.e2e-8f6cf1d295-e5e21.test-cncf-aws.k8s.io on 10.63.240.10:53: no such host
INSTANCE GROUPS
NAME			ROLE	MACHINETYPE	MIN	MAX	SUBNETS
master-us-west-1a	Master	c5.large	1	1	us-west-1a
nodes-us-west-1a	Node	t3.medium	4	4	us-west-1a

NODE STATUS
NAME	ROLE	READY

VALIDATION ERRORS
KIND	NAME		MESSAGE
dns	apiserver	Validation Failed

The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address.  The API DNS IP address is the placeholder address that kops creates: 203.0.113.123.  Please wait about 5-10 minutes for a master to start, dns-controller to launch, and DNS to propagate.  The protokube container and dns-controller deployment logs may contain more diagnostic information.  Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start.

Validation Failed
W1014 12:30:59.375058    4955 validate_cluster.go:221] (will retry): cluster not yet healthy
INSTANCE GROUPS
NAME			ROLE	MACHINETYPE	MIN	MAX	SUBNETS
master-us-west-1a	Master	c5.large	1	1	us-west-1a
nodes-us-west-1a	Node	t3.medium	4	4	us-west-1a

NODE STATUS
NAME	ROLE	READY

VALIDATION ERRORS
KIND	NAME		MESSAGE
dns	apiserver	Validation Failed

The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address.  The API DNS IP address is the placeholder address that kops creates: 203.0.113.123.  Please wait about 5-10 minutes for a master to start, dns-controller to launch, and DNS to propagate.  The protokube container and dns-controller deployment logs may contain more diagnostic information.  Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start.

Validation Failed
W1014 12:31:09.411460    4955 validate_cluster.go:221] (will retry): cluster not yet healthy
INSTANCE GROUPS
NAME			ROLE	MACHINETYPE	MIN	MAX	SUBNETS
master-us-west-1a	Master	c5.large	1	1	us-west-1a
nodes-us-west-1a	Node	t3.medium	4	4	us-west-1a

NODE STATUS
NAME	ROLE	READY

VALIDATION ERRORS
KIND	NAME		MESSAGE
dns	apiserver	Validation Failed

The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address.  The API DNS IP address is the placeholder address that kops creates: 203.0.113.123.  Please wait about 5-10 minutes for a master to start, dns-controller to launch, and DNS to propagate.  The protokube container and dns-controller deployment logs may contain more diagnostic information.  Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start.

Validation Failed
W1014 12:31:19.446944    4955 validate_cluster.go:221] (will retry): cluster not yet healthy
INSTANCE GROUPS
NAME			ROLE	MACHINETYPE	MIN	MAX	SUBNETS
master-us-west-1a	Master	c5.large	1	1	us-west-1a
nodes-us-west-1a	Node	t3.medium	4	4	us-west-1a

NODE STATUS
NAME	ROLE	READY

VALIDATION ERRORS
KIND	NAME		MESSAGE
dns	apiserver	Validation Failed

The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address.  The API DNS IP address is the placeholder address that kops creates: 203.0.113.123.  Please wait about 5-10 minutes for a master to start, dns-controller to launch, and DNS to propagate.  The protokube container and dns-controller deployment logs may contain more diagnostic information.  Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start.

Validation Failed
W1014 12:31:29.515214    4955 validate_cluster.go:221] (will retry): cluster not yet healthy
INSTANCE GROUPS
NAME			ROLE	MACHINETYPE	MIN	MAX	SUBNETS
master-us-west-1a	Master	c5.large	1	1	us-west-1a
nodes-us-west-1a	Node	t3.medium	4	4	us-west-1a

NODE STATUS
NAME	ROLE	READY

VALIDATION ERRORS
KIND	NAME		MESSAGE
dns	apiserver	Validation Failed

The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address.  The API DNS IP address is the placeholder address that kops creates: 203.0.113.123.  Please wait about 5-10 minutes for a master to start, dns-controller to launch, and DNS to propagate.  The protokube container and dns-controller deployment logs may contain more diagnostic information.  Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start.

Validation Failed
W1014 12:31:39.546882    4955 validate_cluster.go:221] (will retry): cluster not yet healthy
INSTANCE GROUPS
NAME			ROLE	MACHINETYPE	MIN	MAX	SUBNETS
master-us-west-1a	Master	c5.large	1	1	us-west-1a
nodes-us-west-1a	Node	t3.medium	4	4	us-west-1a

NODE STATUS
NAME	ROLE	READY

VALIDATION ERRORS
KIND	NAME		MESSAGE
dns	apiserver	Validation Failed

The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address.  The API DNS IP address is the placeholder address that kops creates: 203.0.113.123.  Please wait about 5-10 minutes for a master to start, dns-controller to launch, and DNS to propagate.  The protokube container and dns-controller deployment logs may contain more diagnostic information.  Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start.

Validation Failed
W1014 12:31:49.593717    4955 validate_cluster.go:221] (will retry): cluster not yet healthy
INSTANCE GROUPS
NAME			ROLE	MACHINETYPE	MIN	MAX	SUBNETS
master-us-west-1a	Master	c5.large	1	1	us-west-1a
nodes-us-west-1a	Node	t3.medium	4	4	us-west-1a

NODE STATUS
NAME	ROLE	READY

VALIDATION ERRORS
KIND	NAME		MESSAGE
dns	apiserver	Validation Failed

The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address.  The API DNS IP address is the placeholder address that kops creates: 203.0.113.123.  Please wait about 5-10 minutes for a master to start, dns-controller to launch, and DNS to propagate.  The protokube container and dns-controller deployment logs may contain more diagnostic information.  Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start.

Validation Failed
W1014 12:31:59.626649    4955 validate_cluster.go:221] (will retry): cluster not yet healthy
INSTANCE GROUPS
NAME			ROLE	MACHINETYPE	MIN	MAX	SUBNETS
master-us-west-1a	Master	c5.large	1	1	us-west-1a
nodes-us-west-1a	Node	t3.medium	4	4	us-west-1a

NODE STATUS
NAME	ROLE	READY

VALIDATION ERRORS
KIND	NAME		MESSAGE
dns	apiserver	Validation Failed

The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address.  The API DNS IP address is the placeholder address that kops creates: 203.0.113.123.  Please wait about 5-10 minutes for a master to start, dns-controller to launch, and DNS to propagate.  The protokube container and dns-controller deployment logs may contain more diagnostic information.  Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start.

Validation Failed
W1014 12:32:09.657818    4955 validate_cluster.go:221] (will retry): cluster not yet healthy
INSTANCE GROUPS
NAME			ROLE	MACHINETYPE	MIN	MAX	SUBNETS
master-us-west-1a	Master	c5.large	1	1	us-west-1a
nodes-us-west-1a	Node	t3.medium	4	4	us-west-1a

NODE STATUS
NAME	ROLE	READY

VALIDATION ERRORS
KIND	NAME		MESSAGE
dns	apiserver	Validation Failed

The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address.  The API DNS IP address is the placeholder address that kops creates: 203.0.113.123.  Please wait about 5-10 minutes for a master to start, dns-controller to launch, and DNS to propagate.  The protokube container and dns-controller deployment logs may contain more diagnostic information.  Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start.

Validation Failed
W1014 12:32:19.685642    4955 validate_cluster.go:221] (will retry): cluster not yet healthy
INSTANCE GROUPS
NAME			ROLE	MACHINETYPE	MIN	MAX	SUBNETS
master-us-west-1a	Master	c5.large	1	1	us-west-1a
nodes-us-west-1a	Node	t3.medium	4	4	us-west-1a

NODE STATUS
NAME	ROLE	READY

VALIDATION ERRORS
KIND	NAME		MESSAGE
dns	apiserver	Validation Failed

The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address.  The API DNS IP address is the placeholder address that kops creates: 203.0.113.123.  Please wait about 5-10 minutes for a master to start, dns-controller to launch, and DNS to propagate.  The protokube container and dns-controller deployment logs may contain more diagnostic information.  Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start.

Validation Failed
W1014 12:32:29.716931    4955 validate_cluster.go:221] (will retry): cluster not yet healthy
INSTANCE GROUPS
NAME			ROLE	MACHINETYPE	MIN	MAX	SUBNETS
master-us-west-1a	Master	c5.large	1	1	us-west-1a
nodes-us-west-1a	Node	t3.medium	4	4	us-west-1a

NODE STATUS
NAME	ROLE	READY

VALIDATION ERRORS
KIND	NAME		MESSAGE
dns	apiserver	Validation Failed

The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address.  The API DNS IP address is the placeholder address that kops creates: 203.0.113.123.  Please wait about 5-10 minutes for a master to start, dns-controller to launch, and DNS to propagate.  The protokube container and dns-controller deployment logs may contain more diagnostic information.  Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start.

Validation Failed
W1014 12:32:39.762015    4955 validate_cluster.go:221] (will retry): cluster not yet healthy
INSTANCE GROUPS
NAME			ROLE	MACHINETYPE	MIN	MAX	SUBNETS
master-us-west-1a	Master	c5.large	1	1	us-west-1a
nodes-us-west-1a	Node	t3.medium	4	4	us-west-1a

NODE STATUS
NAME	ROLE	READY

VALIDATION ERRORS
KIND	NAME		MESSAGE
dns	apiserver	Validation Failed

The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address.  The API DNS IP address is the placeholder address that kops creates: 203.0.113.123.  Please wait about 5-10 minutes for a master to start, dns-controller to launch, and DNS to propagate.  The protokube container and dns-controller deployment logs may contain more diagnostic information.  Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start.

Validation Failed
W1014 12:32:49.791472    4955 validate_cluster.go:221] (will retry): cluster not yet healthy
INSTANCE GROUPS
NAME			ROLE	MACHINETYPE	MIN	MAX	SUBNETS
master-us-west-1a	Master	c5.large	1	1	us-west-1a
nodes-us-west-1a	Node	t3.medium	4	4	us-west-1a

NODE STATUS
NAME	ROLE	READY

VALIDATION ERRORS
KIND	NAME		MESSAGE
dns	apiserver	Validation Failed

The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address.  The API DNS IP address is the placeholder address that kops creates: 203.0.113.123.  Please wait about 5-10 minutes for a master to start, dns-controller to launch, and DNS to propagate.  The protokube container and dns-controller deployment logs may contain more diagnostic information.  Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start.

Validation Failed
W1014 12:32:59.821574    4955 validate_cluster.go:221] (will retry): cluster not yet healthy
INSTANCE GROUPS
NAME			ROLE	MACHINETYPE	MIN	MAX	SUBNETS
master-us-west-1a	Master	c5.large	1	1	us-west-1a
nodes-us-west-1a	Node	t3.medium	4	4	us-west-1a

NODE STATUS
NAME	ROLE	READY

VALIDATION ERRORS
KIND	NAME		MESSAGE
dns	apiserver	Validation Failed

The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address.  The API DNS IP address is the placeholder address that kops creates: 203.0.113.123.  Please wait about 5-10 minutes for a master to start, dns-controller to launch, and DNS to propagate.  The protokube container and dns-controller deployment logs may contain more diagnostic information.  Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start.

Validation Failed
W1014 12:33:09.854045    4955 validate_cluster.go:221] (will retry): cluster not yet healthy
INSTANCE GROUPS
NAME			ROLE	MACHINETYPE	MIN	MAX	SUBNETS
master-us-west-1a	Master	c5.large	1	1	us-west-1a
nodes-us-west-1a	Node	t3.medium	4	4	us-west-1a

NODE STATUS
NAME	ROLE	READY

VALIDATION ERRORS
KIND	NAME		MESSAGE
dns	apiserver	Validation Failed

The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address.  The API DNS IP address is the placeholder address that kops creates: 203.0.113.123.  Please wait about 5-10 minutes for a master to start, dns-controller to launch, and DNS to propagate.  The protokube container and dns-controller deployment logs may contain more diagnostic information.  Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start.

Validation Failed
W1014 12:33:19.920282    4955 validate_cluster.go:221] (will retry): cluster not yet healthy
INSTANCE GROUPS
NAME			ROLE	MACHINETYPE	MIN	MAX	SUBNETS
master-us-west-1a	Master	c5.large	1	1	us-west-1a
nodes-us-west-1a	Node	t3.medium	4	4	us-west-1a

NODE STATUS
NAME	ROLE	READY

VALIDATION ERRORS
KIND	NAME		MESSAGE
dns	apiserver	Validation Failed

The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address.  The API DNS IP address is the placeholder address that kops creates: 203.0.113.123.  Please wait about 5-10 minutes for a master to start, dns-controller to launch, and DNS to propagate.  The protokube container and dns-controller deployment logs may contain more diagnostic information.  Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start.

Validation Failed
W1014 12:33:29.954775    4955 validate_cluster.go:221] (will retry): cluster not yet healthy
INSTANCE GROUPS
NAME			ROLE	MACHINETYPE	MIN	MAX	SUBNETS
master-us-west-1a	Master	c5.large	1	1	us-west-1a
nodes-us-west-1a	Node	t3.medium	4	4	us-west-1a

NODE STATUS
NAME	ROLE	READY

VALIDATION ERRORS
KIND	NAME		MESSAGE
dns	apiserver	Validation Failed

The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address.  The API DNS IP address is the placeholder address that kops creates: 203.0.113.123.  Please wait about 5-10 minutes for a master to start, dns-controller to launch, and DNS to propagate.  The protokube container and dns-controller deployment logs may contain more diagnostic information.  Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start.

Validation Failed
W1014 12:33:40.020464    4955 validate_cluster.go:221] (will retry): cluster not yet healthy
INSTANCE GROUPS
NAME			ROLE	MACHINETYPE	MIN	MAX	SUBNETS
master-us-west-1a	Master	c5.large	1	1	us-west-1a
nodes-us-west-1a	Node	t3.medium	4	4	us-west-1a

NODE STATUS
NAME	ROLE	READY

VALIDATION ERRORS
KIND	NAME		MESSAGE
dns	apiserver	Validation Failed

The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address.  The API DNS IP address is the placeholder address that kops creates: 203.0.113.123.  Please wait about 5-10 minutes for a master to start, dns-controller to launch, and DNS to propagate.  The protokube container and dns-controller deployment logs may contain more diagnostic information.  Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start.

Validation Failed
W1014 12:33:50.049750    4955 validate_cluster.go:221] (will retry): cluster not yet healthy
INSTANCE GROUPS
NAME			ROLE	MACHINETYPE	MIN	MAX	SUBNETS
master-us-west-1a	Master	c5.large	1	1	us-west-1a
nodes-us-west-1a	Node	t3.medium	4	4	us-west-1a

NODE STATUS
NAME	ROLE	READY

VALIDATION ERRORS
KIND	NAME		MESSAGE
dns	apiserver	Validation Failed

The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address.  The API DNS IP address is the placeholder address that kops creates: 203.0.113.123.  Please wait about 5-10 minutes for a master to start, dns-controller to launch, and DNS to propagate.  The protokube container and dns-controller deployment logs may contain more diagnostic information.  Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start.

Validation Failed
W1014 12:34:00.079632    4955 validate_cluster.go:221] (will retry): cluster not yet healthy
INSTANCE GROUPS
NAME			ROLE	MACHINETYPE	MIN	MAX	SUBNETS
master-us-west-1a	Master	c5.large	1	1	us-west-1a
nodes-us-west-1a	Node	t3.medium	4	4	us-west-1a

NODE STATUS
NAME	ROLE	READY

VALIDATION ERRORS
KIND	NAME		MESSAGE
dns	apiserver	Validation Failed

The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address.  The API DNS IP address is the placeholder address that kops creates: 203.0.113.123.  Please wait about 5-10 minutes for a master to start, dns-controller to launch, and DNS to propagate.  The protokube container and dns-controller deployment logs may contain more diagnostic information.  Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start.

Validation Failed
W1014 12:34:10.109949    4955 validate_cluster.go:221] (will retry): cluster not yet healthy
INSTANCE GROUPS
NAME			ROLE	MACHINETYPE	MIN	MAX	SUBNETS
master-us-west-1a	Master	c5.large	1	1	us-west-1a
nodes-us-west-1a	Node	t3.medium	4	4	us-west-1a

NODE STATUS
NAME	ROLE	READY

VALIDATION ERRORS
KIND	NAME		MESSAGE
dns	apiserver	Validation Failed

The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address.  The API DNS IP address is the placeholder address that kops creates: 203.0.113.123.  Please wait about 5-10 minutes for a master to start, dns-controller to launch, and DNS to propagate.  The protokube container and dns-controller deployment logs may contain more diagnostic information.  Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start.

Validation Failed
W1014 12:34:20.139406    4955 validate_cluster.go:221] (will retry): cluster not yet healthy
INSTANCE GROUPS
NAME			ROLE	MACHINETYPE	MIN	MAX	SUBNETS
master-us-west-1a	Master	c5.large	1	1	us-west-1a
nodes-us-west-1a	Node	t3.medium	4	4	us-west-1a

NODE STATUS
NAME	ROLE	READY

VALIDATION ERRORS
KIND	NAME		MESSAGE
dns	apiserver	Validation Failed

The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address.  The API DNS IP address is the placeholder address that kops creates: 203.0.113.123.  Please wait about 5-10 minutes for a master to start, dns-controller to launch, and DNS to propagate.  The protokube container and dns-controller deployment logs may contain more diagnostic information.  Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start.

Validation Failed
W1014 12:34:30.169508    4955 validate_cluster.go:221] (will retry): cluster not yet healthy
INSTANCE GROUPS
NAME			ROLE	MACHINETYPE	MIN	MAX	SUBNETS
master-us-west-1a	Master	c5.large	1	1	us-west-1a
nodes-us-west-1a	Node	t3.medium	4	4	us-west-1a

NODE STATUS
NAME	ROLE	READY

VALIDATION ERRORS
KIND	NAME		MESSAGE
dns	apiserver	Validation Failed

The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address.  The API DNS IP address is the placeholder address that kops creates: 203.0.113.123.  Please wait about 5-10 minutes for a master to start, dns-controller to launch, and DNS to propagate.  The protokube container and dns-controller deployment logs may contain more diagnostic information.  Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start.

Validation Failed
W1014 12:34:40.205297    4955 validate_cluster.go:221] (will retry): cluster not yet healthy
INSTANCE GROUPS
NAME			ROLE	MACHINETYPE	MIN	MAX	SUBNETS
master-us-west-1a	Master	c5.large	1	1	us-west-1a
nodes-us-west-1a	Node	t3.medium	4	4	us-west-1a

NODE STATUS
NAME	ROLE	READY

VALIDATION ERRORS
KIND	NAME		MESSAGE
dns	apiserver	Validation Failed

The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address.  The API DNS IP address is the placeholder address that kops creates: 203.0.113.123.  Please wait about 5-10 minutes for a master to start, dns-controller to launch, and DNS to propagate.  The protokube container and dns-controller deployment logs may contain more diagnostic information.  Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start.

Validation Failed
W1014 12:34:50.235670    4955 validate_cluster.go:221] (will retry): cluster not yet healthy
INSTANCE GROUPS
NAME			ROLE	MACHINETYPE	MIN	MAX	SUBNETS
master-us-west-1a	Master	c5.large	1	1	us-west-1a
nodes-us-west-1a	Node	t3.medium	4	4	us-west-1a

NODE STATUS
NAME	ROLE	READY

VALIDATION ERRORS
KIND	NAME		MESSAGE
dns	apiserver	Validation Failed

The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address.  The API DNS IP address is the placeholder address that kops creates: 203.0.113.123.  Please wait about 5-10 minutes for a master to start, dns-controller to launch, and DNS to propagate.  The protokube container and dns-controller deployment logs may contain more diagnostic information.  Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start.

Validation Failed
W1014 12:35:00.264637    4955 validate_cluster.go:221] (will retry): cluster not yet healthy
INSTANCE GROUPS
NAME			ROLE	MACHINETYPE	MIN	MAX	SUBNETS
master-us-west-1a	Master	c5.large	1	1	us-west-1a
nodes-us-west-1a	Node	t3.medium	4	4	us-west-1a

NODE STATUS
NAME	ROLE	READY

VALIDATION ERRORS
KIND	NAME		MESSAGE
dns	apiserver	Validation Failed

The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address.  The API DNS IP address is the placeholder address that kops creates: 203.0.113.123.  Please wait about 5-10 minutes for a master to start, dns-controller to launch, and DNS to propagate.  The protokube container and dns-controller deployment logs may contain more diagnostic information.  Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start.

Validation Failed
W1014 12:35:10.297119    4955 validate_cluster.go:221] (will retry): cluster not yet healthy
INSTANCE GROUPS
NAME			ROLE	MACHINETYPE	MIN	MAX	SUBNETS
master-us-west-1a	Master	c5.large	1	1	us-west-1a
nodes-us-west-1a	Node	t3.medium	4	4	us-west-1a

NODE STATUS
NAME	ROLE	READY

VALIDATION ERRORS
KIND	NAME		MESSAGE
dns	apiserver	Validation Failed

The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address.  The API DNS IP address is the placeholder address that kops creates: 203.0.113.123.  Please wait about 5-10 minutes for a master to start, dns-controller to launch, and DNS to propagate.  The protokube container and dns-controller deployment logs may contain more diagnostic information.  Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start.

Validation Failed
W1014 12:35:20.332338    4955 validate_cluster.go:221] (will retry): cluster not yet healthy
INSTANCE GROUPS
NAME			ROLE	MACHINETYPE	MIN	MAX	SUBNETS
master-us-west-1a	Master	c5.large	1	1	us-west-1a
nodes-us-west-1a	Node	t3.medium	4	4	us-west-1a

NODE STATUS
NAME	ROLE	READY

VALIDATION ERRORS
KIND	NAME		MESSAGE
dns	apiserver	Validation Failed

The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address.  The API DNS IP address is the placeholder address that kops creates: 203.0.113.123.  Please wait about 5-10 minutes for a master to start, dns-controller to launch, and DNS to propagate.  The protokube container and dns-controller deployment logs may contain more diagnostic information.  Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start.

Validation Failed
W1014 12:35:30.374896    4955 validate_cluster.go:221] (will retry): cluster not yet healthy
INSTANCE GROUPS
NAME			ROLE	MACHINETYPE	MIN	MAX	SUBNETS
master-us-west-1a	Master	c5.large	1	1	us-west-1a
nodes-us-west-1a	Node	t3.medium	4	4	us-west-1a

... skipping 7 lines ...
Machine	i-091daa22206556d7f				machine "i-091daa22206556d7f" has not yet joined cluster
Machine	i-0a399fbd239122c7c				machine "i-0a399fbd239122c7c" has not yet joined cluster
Machine	i-0c0a0860ee2260e0f				machine "i-0c0a0860ee2260e0f" has not yet joined cluster
Pod	kube-system/coredns-5dc785954d-jjhzn		system-cluster-critical pod "coredns-5dc785954d-jjhzn" is pending
Pod	kube-system/coredns-autoscaler-84d4cfd89c-2mbhk	system-cluster-critical pod "coredns-autoscaler-84d4cfd89c-2mbhk" is pending

Validation Failed
W1014 12:35:42.096676    4955 validate_cluster.go:221] (will retry): cluster not yet healthy
INSTANCE GROUPS
NAME			ROLE	MACHINETYPE	MIN	MAX	SUBNETS
master-us-west-1a	Master	c5.large	1	1	us-west-1a
nodes-us-west-1a	Node	t3.medium	4	4	us-west-1a

... skipping 13 lines ...
Pod	kube-system/coredns-5dc785954d-jjhzn					system-cluster-critical pod "coredns-5dc785954d-jjhzn" is pending
Pod	kube-system/coredns-autoscaler-84d4cfd89c-2mbhk				system-cluster-critical pod "coredns-autoscaler-84d4cfd89c-2mbhk" is pending
Pod	kube-system/kube-proxy-ip-172-20-37-217.us-west-1.compute.internal	system-node-critical pod "kube-proxy-ip-172-20-37-217.us-west-1.compute.internal" is pending
Pod	kube-system/kube-proxy-ip-172-20-42-53.us-west-1.compute.internal	system-node-critical pod "kube-proxy-ip-172-20-42-53.us-west-1.compute.internal" is pending
Pod	kube-system/kube-proxy-ip-172-20-45-235.us-west-1.compute.internal	system-node-critical pod "kube-proxy-ip-172-20-45-235.us-west-1.compute.internal" is pending

Validation Failed
W1014 12:35:53.417628    4955 validate_cluster.go:221] (will retry): cluster not yet healthy
INSTANCE GROUPS
NAME			ROLE	MACHINETYPE	MIN	MAX	SUBNETS
master-us-west-1a	Master	c5.large	1	1	us-west-1a
nodes-us-west-1a	Node	t3.medium	4	4	us-west-1a

... skipping 8 lines ...
VALIDATION ERRORS
KIND	NAME									MESSAGE
Node	ip-172-20-48-115.us-west-1.compute.internal				node "ip-172-20-48-115.us-west-1.compute.internal" of role "node" is not ready
Pod	kube-system/coredns-5dc785954d-jjhzn					system-cluster-critical pod "coredns-5dc785954d-jjhzn" is pending
Pod	kube-system/kube-proxy-ip-172-20-48-115.us-west-1.compute.internal	system-node-critical pod "kube-proxy-ip-172-20-48-115.us-west-1.compute.internal" is pending

Validation Failed
W1014 12:36:04.744118    4955 validate_cluster.go:221] (will retry): cluster not yet healthy
INSTANCE GROUPS
NAME			ROLE	MACHINETYPE	MIN	MAX	SUBNETS
master-us-west-1a	Master	c5.large	1	1	us-west-1a
nodes-us-west-1a	Node	t3.medium	4	4	us-west-1a

... skipping 6 lines ...
ip-172-20-57-220.us-west-1.compute.internal	master	True

VALIDATION ERRORS
KIND	NAME					MESSAGE
Pod	kube-system/coredns-5dc785954d-jjhzn	system-cluster-critical pod "coredns-5dc785954d-jjhzn" is not ready (coredns)

Validation Failed
W1014 12:36:15.961090    4955 validate_cluster.go:221] (will retry): cluster not yet healthy
INSTANCE GROUPS
NAME			ROLE	MACHINETYPE	MIN	MAX	SUBNETS
master-us-west-1a	Master	c5.large	1	1	us-west-1a
nodes-us-west-1a	Node	t3.medium	4	4	us-west-1a

... skipping 6 lines ...
ip-172-20-57-220.us-west-1.compute.internal	master	True

VALIDATION ERRORS
KIND	NAME					MESSAGE
Pod	kube-system/coredns-5dc785954d-jjhzn	system-cluster-critical pod "coredns-5dc785954d-jjhzn" is not ready (coredns)

Validation Failed
W1014 12:36:27.150780    4955 validate_cluster.go:221] (will retry): cluster not yet healthy
INSTANCE GROUPS
NAME			ROLE	MACHINETYPE	MIN	MAX	SUBNETS
master-us-west-1a	Master	c5.large	1	1	us-west-1a
nodes-us-west-1a	Node	t3.medium	4	4	us-west-1a

... skipping 6 lines ...
ip-172-20-57-220.us-west-1.compute.internal	master	True

VALIDATION ERRORS
KIND	NAME					MESSAGE
Pod	kube-system/coredns-5dc785954d-jjhzn	system-cluster-critical pod "coredns-5dc785954d-jjhzn" is not ready (coredns)

Validation Failed
W1014 12:36:38.387921    4955 validate_cluster.go:221] (will retry): cluster not yet healthy
INSTANCE GROUPS
NAME			ROLE	MACHINETYPE	MIN	MAX	SUBNETS
master-us-west-1a	Master	c5.large	1	1	us-west-1a
nodes-us-west-1a	Node	t3.medium	4	4	us-west-1a

... skipping 1293 lines ...
  	                    	        for cmd in "${commands[@]}"; do
  	                    	...
  	                    	            continue
  	                    	          fi
  	                    	+         if ! validate-hash "${file}" "${hash}"; then
  	                    	-         if [[ -n "${hash}" ]] && ! validate-hash "${file}" "${hash}"; then
  	                    	            echo "== Hash validation of ${url} failed. Retrying. =="
  	                    	            rm -f "${file}"
  	                    	          else
  	                    	-           if [[ -n "${hash}" ]]; then
  	                    	+           echo "== Downloaded ${url} (SHA256 = ${hash}) =="
  	                    	-             echo "== Downloaded ${url} (SHA1 = ${hash}) =="
  	                    	-           else
... skipping 229 lines ...
  	                    	        for cmd in "${commands[@]}"; do
  	                    	...
  	                    	            continue
  	                    	          fi
  	                    	+         if ! validate-hash "${file}" "${hash}"; then
  	                    	-         if [[ -n "${hash}" ]] && ! validate-hash "${file}" "${hash}"; then
  	                    	            echo "== Hash validation of ${url} failed. Retrying. =="
  	                    	            rm -f "${file}"
  	                    	          else
  	                    	-           if [[ -n "${hash}" ]]; then
  	                    	+           echo "== Downloaded ${url} (SHA256 = ${hash}) =="
  	                    	-             echo "== Downloaded ${url} (SHA1 = ${hash}) =="
  	                    	-           else
... skipping 905 lines ...
evicting pod kube-system/dns-controller-55bbcbb54b-7t26s
I1014 12:43:14.863645   12670 instancegroups.go:658] Waiting for 5s for pods to stabilize after draining.
I1014 12:43:19.866562   12670 instancegroups.go:417] deleting node "ip-172-20-57-220.us-west-1.compute.internal" from kubernetes
I1014 12:43:19.917559   12670 instancegroups.go:591] Stopping instance "i-074cb72adcd4e0300", node "ip-172-20-57-220.us-west-1.compute.internal", in group "master-us-west-1a.masters.e2e-8f6cf1d295-e5e21.test-cncf-aws.k8s.io" (this may take a while).
I1014 12:43:20.072975   12670 instancegroups.go:435] waiting for 15s after terminating instance
I1014 12:43:35.073237   12670 instancegroups.go:470] Validating the cluster.
I1014 12:44:05.115989   12670 instancegroups.go:516] Cluster did not validate, will retry in "30s": error listing nodes: Get "https://api.e2e-8f6cf1d295-e5e21.test-cncf-aws.k8s.io/api/v1/nodes": dial tcp 13.52.78.92:443: i/o timeout.
I1014 12:45:05.178322   12670 instancegroups.go:516] Cluster did not validate, will retry in "30s": error listing nodes: Get "https://api.e2e-8f6cf1d295-e5e21.test-cncf-aws.k8s.io/api/v1/nodes": dial tcp 13.52.78.92:443: i/o timeout.
I1014 12:46:05.215692   12670 instancegroups.go:516] Cluster did not validate, will retry in "30s": error listing nodes: Get "https://api.e2e-8f6cf1d295-e5e21.test-cncf-aws.k8s.io/api/v1/nodes": dial tcp 13.52.78.92:443: i/o timeout.
I1014 12:47:05.265227   12670 instancegroups.go:516] Cluster did not validate, will retry in "30s": error listing nodes: Get "https://api.e2e-8f6cf1d295-e5e21.test-cncf-aws.k8s.io/api/v1/nodes": dial tcp 13.52.78.92:443: i/o timeout.
I1014 12:48:05.297611   12670 instancegroups.go:516] Cluster did not validate, will retry in "30s": error listing nodes: Get "https://api.e2e-8f6cf1d295-e5e21.test-cncf-aws.k8s.io/api/v1/nodes": dial tcp 13.52.78.92:443: i/o timeout.
I1014 12:49:05.331520   12670 instancegroups.go:516] Cluster did not validate, will retry in "30s": error listing nodes: Get "https://api.e2e-8f6cf1d295-e5e21.test-cncf-aws.k8s.io/api/v1/nodes": dial tcp 13.52.78.92:443: i/o timeout.
I1014 12:50:05.374768   12670 instancegroups.go:516] Cluster did not validate, will retry in "30s": error listing nodes: Get "https://api.e2e-8f6cf1d295-e5e21.test-cncf-aws.k8s.io/api/v1/nodes": dial tcp 13.52.78.92:443: i/o timeout.
I1014 12:51:05.422163   12670 instancegroups.go:516] Cluster did not validate, will retry in "30s": error listing nodes: Get "https://api.e2e-8f6cf1d295-e5e21.test-cncf-aws.k8s.io/api/v1/nodes": dial tcp 13.52.78.92:443: i/o timeout.
I1014 12:52:05.460179   12670 instancegroups.go:516] Cluster did not validate, will retry in "30s": error listing nodes: Get "https://api.e2e-8f6cf1d295-e5e21.test-cncf-aws.k8s.io/api/v1/nodes": dial tcp 13.52.78.92:443: i/o timeout.
I1014 12:53:05.506749   12670 instancegroups.go:516] Cluster did not validate, will retry in "30s": error listing nodes: Get "https://api.e2e-8f6cf1d295-e5e21.test-cncf-aws.k8s.io/api/v1/nodes": dial tcp 13.52.78.92:443: i/o timeout.
I1014 12:54:05.541324   12670 instancegroups.go:516] Cluster did not validate, will retry in "30s": error listing nodes: Get "https://api.e2e-8f6cf1d295-e5e21.test-cncf-aws.k8s.io/api/v1/nodes": dial tcp 13.52.78.92:443: i/o timeout.
I1014 12:55:05.585115   12670 instancegroups.go:516] Cluster did not validate, will retry in "30s": error listing nodes: Get "https://api.e2e-8f6cf1d295-e5e21.test-cncf-aws.k8s.io/api/v1/nodes": dial tcp 13.52.78.92:443: i/o timeout.
I1014 12:56:05.624480   12670 instancegroups.go:516] Cluster did not validate, will retry in "30s": error listing nodes: Get "https://api.e2e-8f6cf1d295-e5e21.test-cncf-aws.k8s.io/api/v1/nodes": dial tcp 13.52.78.92:443: i/o timeout.
I1014 12:57:05.655338   12670 instancegroups.go:516] Cluster did not validate, will retry in "30s": error listing nodes: Get "https://api.e2e-8f6cf1d295-e5e21.test-cncf-aws.k8s.io/api/v1/nodes": dial tcp 13.52.78.92:443: i/o timeout.
I1014 12:58:05.719710   12670 instancegroups.go:516] Cluster did not validate, will retry in "30s": error listing nodes: Get "https://api.e2e-8f6cf1d295-e5e21.test-cncf-aws.k8s.io/api/v1/nodes": dial tcp 13.52.78.92:443: i/o timeout.
I1014 12:59:05.766002   12670 instancegroups.go:513] Cluster did not validate within deadline: error listing nodes: Get "https://api.e2e-8f6cf1d295-e5e21.test-cncf-aws.k8s.io/api/v1/nodes": dial tcp 13.52.78.92:443: i/o timeout.
E1014 12:59:05.766052   12670 instancegroups.go:475] Cluster did not validate within 15m0s
Error: master not healthy after update, stopping rolling-update: "error validating cluster after terminating instance: cluster did not validate within a duration of \"15m0s\""
+ kops-finish
+ kubetest2 kops -v=2 --cloud-provider=aws --cluster-name=e2e-8f6cf1d295-e5e21.test-cncf-aws.k8s.io --kops-root=/home/prow/go/src/k8s.io/kops --admin-access= --env=KOPS_FEATURE_FLAGS=SpecOverrideFlag --kops-binary-path=/home/prow/go/src/k8s.io/kops/.bazelbuild/dist/linux/amd64/kops --down
I1014 12:59:05.791557   12689 app.go:59] RunDir for this run: "/logs/artifacts/4616d072-2cea-11ec-8d4f-ca63ca797e20"
I1014 12:59:05.791709   12689 app.go:90] ID for this run: "4616d072-2cea-11ec-8d4f-ca63ca797e20"
I1014 12:59:05.814147   12689 dumplogs.go:40] /home/prow/go/src/k8s.io/kops/.bazelbuild/dist/linux/amd64/kops toolbox dump --name e2e-8f6cf1d295-e5e21.test-cncf-aws.k8s.io --dir /logs/artifacts --private-key /etc/aws-ssh/aws-ssh-private --ssh-user ubuntu
I1014 12:59:05.832377   12698 featureflag.go:166] FeatureFlag "SpecOverrideFlag"=true
I1014 12:59:05.832451   12698 featureflag.go:166] FeatureFlag "AlphaAllowGCE"=true
I1014 12:59:05.832455   12698 featureflag.go:166] FeatureFlag "SpecOverrideFlag"=true
W1014 12:59:43.069212   12698 toolbox_dump.go:168] error listing nodes in cluster: Get "https://api.e2e-8f6cf1d295-e5e21.test-cncf-aws.k8s.io/api/v1/nodes": dial tcp 13.52.78.92:443: i/o timeout
2021/10/14 12:59:43 dumping node not registered in kubernetes: 54.177.180.237
2021/10/14 12:59:43 Dumping node 54.177.180.237
2021/10/14 12:59:46 dumping node not registered in kubernetes: 13.52.181.85
2021/10/14 12:59:46 Dumping node 13.52.181.85
2021/10/14 12:59:49 dumping node not registered in kubernetes: 13.57.225.28
2021/10/14 12:59:49 Dumping node 13.57.225.28
2021/10/14 12:59:52 dumping node not registered in kubernetes: 54.241.195.2
2021/10/14 12:59:52 Dumping node 54.241.195.2
2021/10/14 12:59:55 dumping node not registered in kubernetes: 54.215.249.47
2021/10/14 12:59:55 Dumping node 54.215.249.47
2021/10/14 12:59:59 error dumping node 54.215.249.47: error executing command "if command -v kubectl &> /dev/null; then kubectl logs -n kube-system --all-containers -l \"k8s-app=external-dns\"; fi": Process exited with status 1
2021/10/14 12:59:59 error dumping node 54.215.249.47: error executing command "if command -v kubectl &> /dev/null; then kubectl logs -n kube-system --all-containers -l \"k8s-app=dns-controller\"; fi": Process exited with status 1
instances:
- name: i-07e4e7ca5a7243d7d
  publicAddresses:
  - 54.177.180.237
  roles:
  - node
... skipping 1087 lines ...
I1014 12:59:59.691130   12689 dumplogs.go:72] /home/prow/go/src/k8s.io/kops/.bazelbuild/dist/linux/amd64/kops get instancegroups --name e2e-8f6cf1d295-e5e21.test-cncf-aws.k8s.io -o yaml
I1014 13:00:00.420866   12689 dumplogs.go:91] kubectl cluster-info dump --all-namespaces -o yaml --output-directory /logs/artifacts/cluster-info
I1014 13:00:30.470609   12689 dumplogs.go:114] /home/prow/go/src/k8s.io/kops/.bazelbuild/dist/linux/amd64/kops toolbox dump --name e2e-8f6cf1d295-e5e21.test-cncf-aws.k8s.io --private-key /etc/aws-ssh/aws-ssh-private --ssh-user ubuntu -o yaml
I1014 13:00:37.167088   12689 dumplogs.go:143] ssh -i /etc/aws-ssh/aws-ssh-private -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null ubuntu@54.215.249.47 -- kubectl cluster-info dump --all-namespaces -o yaml --output-directory /tmp/cluster-info
Warning: Permanently added '54.215.249.47' (ECDSA) to the list of known hosts.
The connection to the server 127.0.0.1 was refused - did you specify the right host or port?
W1014 13:00:38.165311   12689 down.go:34] Dumping cluster logs at the start of Down() failed: exit status 1
I1014 13:00:38.165376   12689 down.go:48] /home/prow/go/src/k8s.io/kops/.bazelbuild/dist/linux/amd64/kops delete cluster --name e2e-8f6cf1d295-e5e21.test-cncf-aws.k8s.io --yes
I1014 13:00:38.182282   12751 featureflag.go:166] FeatureFlag "SpecOverrideFlag"=true
I1014 13:00:38.182808   12751 featureflag.go:166] FeatureFlag "AlphaAllowGCE"=true
I1014 13:00:38.182839   12751 featureflag.go:166] FeatureFlag "SpecOverrideFlag"=true
TYPE			NAME													ID
autoscaling-config	master-us-west-1a.masters.e2e-8f6cf1d295-e5e21.test-cncf-aws.k8s.io					lt-0c5be28256a07d1d7
... skipping 512 lines ...