This job view page is being replaced by Spyglass soon. Check out the new job view.
ResultFAILURE
Tests 0 failed / 0 succeeded
Started2021-10-18 04:25
Elapsed44m59s
Revisionmaster

No Test Failures!


Error lines from build-log.txt

... skipping 176 lines ...
I1018 04:25:55.662070    4960 dumplogs.go:40] /tmp/kops.UtWEdJrBY toolbox dump --name e2e-89d41a7532-e6156.test-cncf-aws.k8s.io --dir /logs/artifacts --private-key /etc/aws-ssh/aws-ssh-private --ssh-user ubuntu
I1018 04:25:55.674087    4970 featureflag.go:165] FeatureFlag "SpecOverrideFlag"=true
I1018 04:25:55.674187    4970 featureflag.go:165] FeatureFlag "AlphaAllowGCE"=true
I1018 04:25:55.674191    4970 featureflag.go:165] FeatureFlag "SpecOverrideFlag"=true

Cluster.kops.k8s.io "e2e-89d41a7532-e6156.test-cncf-aws.k8s.io" not found
W1018 04:25:56.163561    4960 down.go:34] Dumping cluster logs at the start of Down() failed: exit status 1
I1018 04:25:56.163641    4960 down.go:48] /tmp/kops.UtWEdJrBY delete cluster --name e2e-89d41a7532-e6156.test-cncf-aws.k8s.io --yes
I1018 04:25:56.177795    4980 featureflag.go:165] FeatureFlag "SpecOverrideFlag"=true
I1018 04:25:56.177932    4980 featureflag.go:165] FeatureFlag "AlphaAllowGCE"=true
I1018 04:25:56.177937    4980 featureflag.go:165] FeatureFlag "SpecOverrideFlag"=true

error reading cluster configuration: Cluster.kops.k8s.io "e2e-89d41a7532-e6156.test-cncf-aws.k8s.io" not found
I1018 04:25:56.659920    4960 http.go:37] curl http://metadata.google.internal/computeMetadata/v1/instance/network-interfaces/0/access-configs/0/external-ip
2021/10/18 04:25:56 failed to get external ip from metadata service: http://metadata.google.internal/computeMetadata/v1/instance/network-interfaces/0/access-configs/0/external-ip returned 404
I1018 04:25:56.671496    4960 http.go:37] curl https://ip.jsb.workers.dev
I1018 04:25:56.774733    4960 up.go:144] /tmp/kops.UtWEdJrBY create cluster --name e2e-89d41a7532-e6156.test-cncf-aws.k8s.io --cloud aws --kubernetes-version v1.21.0 --ssh-public-key /etc/aws-ssh/aws-ssh-public --override cluster.spec.nodePortAccess=0.0.0.0/0 --yes --networking calico --admin-access 35.232.146.29/32 --master-count 1 --master-volume-size 48 --node-count 4 --node-volume-size 48 --zones us-east-1a --master-size c5.large
I1018 04:25:56.787188    4990 featureflag.go:165] FeatureFlag "SpecOverrideFlag"=true
I1018 04:25:56.787290    4990 featureflag.go:165] FeatureFlag "AlphaAllowGCE"=true
I1018 04:25:56.787293    4990 featureflag.go:165] FeatureFlag "SpecOverrideFlag"=true
I1018 04:25:56.831921    4990 create_cluster.go:728] Using SSH public key: /etc/aws-ssh/aws-ssh-public
... skipping 44 lines ...
I1018 04:26:23.016519    4960 up.go:181] /tmp/kops.UtWEdJrBY validate cluster --name e2e-89d41a7532-e6156.test-cncf-aws.k8s.io --count 10 --wait 15m0s
I1018 04:26:23.028855    5010 featureflag.go:165] FeatureFlag "SpecOverrideFlag"=true
I1018 04:26:23.028959    5010 featureflag.go:165] FeatureFlag "AlphaAllowGCE"=true
I1018 04:26:23.028963    5010 featureflag.go:165] FeatureFlag "SpecOverrideFlag"=true
Validating cluster e2e-89d41a7532-e6156.test-cncf-aws.k8s.io

W1018 04:26:24.143244    5010 validate_cluster.go:173] (will retry): unexpected error during validation: unable to resolve Kubernetes cluster API URL dns: lookup api.e2e-89d41a7532-e6156.test-cncf-aws.k8s.io on 10.63.240.10:53: no such host
INSTANCE GROUPS
NAME			ROLE	MACHINETYPE	MIN	MAX	SUBNETS
master-us-east-1a	Master	c5.large	1	1	us-east-1a
nodes-us-east-1a	Node	t3.medium	4	4	us-east-1a

NODE STATUS
NAME	ROLE	READY

VALIDATION ERRORS
KIND	NAME		MESSAGE
dns	apiserver	Validation Failed

The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address.  The API DNS IP address is the placeholder address that kops creates: 203.0.113.123.  Please wait about 5-10 minutes for a master to start, dns-controller to launch, and DNS to propagate.  The protokube container and dns-controller deployment logs may contain more diagnostic information.  Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start.

Validation Failed
W1018 04:26:34.186494    5010 validate_cluster.go:221] (will retry): cluster not yet healthy
INSTANCE GROUPS
NAME			ROLE	MACHINETYPE	MIN	MAX	SUBNETS
master-us-east-1a	Master	c5.large	1	1	us-east-1a
nodes-us-east-1a	Node	t3.medium	4	4	us-east-1a

NODE STATUS
NAME	ROLE	READY

VALIDATION ERRORS
KIND	NAME		MESSAGE
dns	apiserver	Validation Failed

The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address.  The API DNS IP address is the placeholder address that kops creates: 203.0.113.123.  Please wait about 5-10 minutes for a master to start, dns-controller to launch, and DNS to propagate.  The protokube container and dns-controller deployment logs may contain more diagnostic information.  Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start.

Validation Failed
W1018 04:26:44.223213    5010 validate_cluster.go:221] (will retry): cluster not yet healthy
INSTANCE GROUPS
NAME			ROLE	MACHINETYPE	MIN	MAX	SUBNETS
master-us-east-1a	Master	c5.large	1	1	us-east-1a
nodes-us-east-1a	Node	t3.medium	4	4	us-east-1a

NODE STATUS
NAME	ROLE	READY

VALIDATION ERRORS
KIND	NAME		MESSAGE
dns	apiserver	Validation Failed

The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address.  The API DNS IP address is the placeholder address that kops creates: 203.0.113.123.  Please wait about 5-10 minutes for a master to start, dns-controller to launch, and DNS to propagate.  The protokube container and dns-controller deployment logs may contain more diagnostic information.  Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start.

Validation Failed
W1018 04:26:54.262102    5010 validate_cluster.go:221] (will retry): cluster not yet healthy
INSTANCE GROUPS
NAME			ROLE	MACHINETYPE	MIN	MAX	SUBNETS
master-us-east-1a	Master	c5.large	1	1	us-east-1a
nodes-us-east-1a	Node	t3.medium	4	4	us-east-1a

NODE STATUS
NAME	ROLE	READY

VALIDATION ERRORS
KIND	NAME		MESSAGE
dns	apiserver	Validation Failed

The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address.  The API DNS IP address is the placeholder address that kops creates: 203.0.113.123.  Please wait about 5-10 minutes for a master to start, dns-controller to launch, and DNS to propagate.  The protokube container and dns-controller deployment logs may contain more diagnostic information.  Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start.

Validation Failed
W1018 04:27:04.317735    5010 validate_cluster.go:221] (will retry): cluster not yet healthy
INSTANCE GROUPS
NAME			ROLE	MACHINETYPE	MIN	MAX	SUBNETS
master-us-east-1a	Master	c5.large	1	1	us-east-1a
nodes-us-east-1a	Node	t3.medium	4	4	us-east-1a

NODE STATUS
NAME	ROLE	READY

VALIDATION ERRORS
KIND	NAME		MESSAGE
dns	apiserver	Validation Failed

The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address.  The API DNS IP address is the placeholder address that kops creates: 203.0.113.123.  Please wait about 5-10 minutes for a master to start, dns-controller to launch, and DNS to propagate.  The protokube container and dns-controller deployment logs may contain more diagnostic information.  Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start.

Validation Failed
W1018 04:27:14.352040    5010 validate_cluster.go:221] (will retry): cluster not yet healthy
INSTANCE GROUPS
NAME			ROLE	MACHINETYPE	MIN	MAX	SUBNETS
master-us-east-1a	Master	c5.large	1	1	us-east-1a
nodes-us-east-1a	Node	t3.medium	4	4	us-east-1a

NODE STATUS
NAME	ROLE	READY

VALIDATION ERRORS
KIND	NAME		MESSAGE
dns	apiserver	Validation Failed

The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address.  The API DNS IP address is the placeholder address that kops creates: 203.0.113.123.  Please wait about 5-10 minutes for a master to start, dns-controller to launch, and DNS to propagate.  The protokube container and dns-controller deployment logs may contain more diagnostic information.  Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start.

Validation Failed
W1018 04:27:24.393538    5010 validate_cluster.go:221] (will retry): cluster not yet healthy
INSTANCE GROUPS
NAME			ROLE	MACHINETYPE	MIN	MAX	SUBNETS
master-us-east-1a	Master	c5.large	1	1	us-east-1a
nodes-us-east-1a	Node	t3.medium	4	4	us-east-1a

NODE STATUS
NAME	ROLE	READY

VALIDATION ERRORS
KIND	NAME		MESSAGE
dns	apiserver	Validation Failed

The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address.  The API DNS IP address is the placeholder address that kops creates: 203.0.113.123.  Please wait about 5-10 minutes for a master to start, dns-controller to launch, and DNS to propagate.  The protokube container and dns-controller deployment logs may contain more diagnostic information.  Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start.

Validation Failed
W1018 04:27:34.431128    5010 validate_cluster.go:221] (will retry): cluster not yet healthy
INSTANCE GROUPS
NAME			ROLE	MACHINETYPE	MIN	MAX	SUBNETS
master-us-east-1a	Master	c5.large	1	1	us-east-1a
nodes-us-east-1a	Node	t3.medium	4	4	us-east-1a

NODE STATUS
NAME	ROLE	READY

VALIDATION ERRORS
KIND	NAME		MESSAGE
dns	apiserver	Validation Failed

The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address.  The API DNS IP address is the placeholder address that kops creates: 203.0.113.123.  Please wait about 5-10 minutes for a master to start, dns-controller to launch, and DNS to propagate.  The protokube container and dns-controller deployment logs may contain more diagnostic information.  Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start.

Validation Failed
W1018 04:27:44.475392    5010 validate_cluster.go:221] (will retry): cluster not yet healthy
INSTANCE GROUPS
NAME			ROLE	MACHINETYPE	MIN	MAX	SUBNETS
master-us-east-1a	Master	c5.large	1	1	us-east-1a
nodes-us-east-1a	Node	t3.medium	4	4	us-east-1a

NODE STATUS
NAME	ROLE	READY

VALIDATION ERRORS
KIND	NAME		MESSAGE
dns	apiserver	Validation Failed

The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address.  The API DNS IP address is the placeholder address that kops creates: 203.0.113.123.  Please wait about 5-10 minutes for a master to start, dns-controller to launch, and DNS to propagate.  The protokube container and dns-controller deployment logs may contain more diagnostic information.  Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start.

Validation Failed
W1018 04:27:54.511122    5010 validate_cluster.go:221] (will retry): cluster not yet healthy
W1018 04:28:04.578584    5010 validate_cluster.go:173] (will retry): unexpected error during validation: unable to resolve Kubernetes cluster API URL dns: lookup api.e2e-89d41a7532-e6156.test-cncf-aws.k8s.io on 10.63.240.10:53: no such host
INSTANCE GROUPS
NAME			ROLE	MACHINETYPE	MIN	MAX	SUBNETS
master-us-east-1a	Master	c5.large	1	1	us-east-1a
nodes-us-east-1a	Node	t3.medium	4	4	us-east-1a

NODE STATUS
NAME	ROLE	READY

VALIDATION ERRORS
KIND	NAME		MESSAGE
dns	apiserver	Validation Failed

The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address.  The API DNS IP address is the placeholder address that kops creates: 203.0.113.123.  Please wait about 5-10 minutes for a master to start, dns-controller to launch, and DNS to propagate.  The protokube container and dns-controller deployment logs may contain more diagnostic information.  Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start.

Validation Failed
W1018 04:28:14.628817    5010 validate_cluster.go:221] (will retry): cluster not yet healthy
INSTANCE GROUPS
NAME			ROLE	MACHINETYPE	MIN	MAX	SUBNETS
master-us-east-1a	Master	c5.large	1	1	us-east-1a
nodes-us-east-1a	Node	t3.medium	4	4	us-east-1a

NODE STATUS
NAME	ROLE	READY

VALIDATION ERRORS
KIND	NAME		MESSAGE
dns	apiserver	Validation Failed

The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address.  The API DNS IP address is the placeholder address that kops creates: 203.0.113.123.  Please wait about 5-10 minutes for a master to start, dns-controller to launch, and DNS to propagate.  The protokube container and dns-controller deployment logs may contain more diagnostic information.  Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start.

Validation Failed
W1018 04:28:24.668419    5010 validate_cluster.go:221] (will retry): cluster not yet healthy
INSTANCE GROUPS
NAME			ROLE	MACHINETYPE	MIN	MAX	SUBNETS
master-us-east-1a	Master	c5.large	1	1	us-east-1a
nodes-us-east-1a	Node	t3.medium	4	4	us-east-1a

NODE STATUS
NAME	ROLE	READY

VALIDATION ERRORS
KIND	NAME		MESSAGE
dns	apiserver	Validation Failed

The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address.  The API DNS IP address is the placeholder address that kops creates: 203.0.113.123.  Please wait about 5-10 minutes for a master to start, dns-controller to launch, and DNS to propagate.  The protokube container and dns-controller deployment logs may contain more diagnostic information.  Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start.

Validation Failed
W1018 04:28:34.709106    5010 validate_cluster.go:221] (will retry): cluster not yet healthy
INSTANCE GROUPS
NAME			ROLE	MACHINETYPE	MIN	MAX	SUBNETS
master-us-east-1a	Master	c5.large	1	1	us-east-1a
nodes-us-east-1a	Node	t3.medium	4	4	us-east-1a

NODE STATUS
NAME	ROLE	READY

VALIDATION ERRORS
KIND	NAME		MESSAGE
dns	apiserver	Validation Failed

The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address.  The API DNS IP address is the placeholder address that kops creates: 203.0.113.123.  Please wait about 5-10 minutes for a master to start, dns-controller to launch, and DNS to propagate.  The protokube container and dns-controller deployment logs may contain more diagnostic information.  Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start.

Validation Failed
W1018 04:28:44.748466    5010 validate_cluster.go:221] (will retry): cluster not yet healthy
INSTANCE GROUPS
NAME			ROLE	MACHINETYPE	MIN	MAX	SUBNETS
master-us-east-1a	Master	c5.large	1	1	us-east-1a
nodes-us-east-1a	Node	t3.medium	4	4	us-east-1a

NODE STATUS
NAME	ROLE	READY

VALIDATION ERRORS
KIND	NAME		MESSAGE
dns	apiserver	Validation Failed

The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address.  The API DNS IP address is the placeholder address that kops creates: 203.0.113.123.  Please wait about 5-10 minutes for a master to start, dns-controller to launch, and DNS to propagate.  The protokube container and dns-controller deployment logs may contain more diagnostic information.  Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start.

Validation Failed
W1018 04:28:54.792889    5010 validate_cluster.go:221] (will retry): cluster not yet healthy
INSTANCE GROUPS
NAME			ROLE	MACHINETYPE	MIN	MAX	SUBNETS
master-us-east-1a	Master	c5.large	1	1	us-east-1a
nodes-us-east-1a	Node	t3.medium	4	4	us-east-1a

NODE STATUS
NAME	ROLE	READY

VALIDATION ERRORS
KIND	NAME		MESSAGE
dns	apiserver	Validation Failed

The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address.  The API DNS IP address is the placeholder address that kops creates: 203.0.113.123.  Please wait about 5-10 minutes for a master to start, dns-controller to launch, and DNS to propagate.  The protokube container and dns-controller deployment logs may contain more diagnostic information.  Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start.

Validation Failed
W1018 04:29:04.833320    5010 validate_cluster.go:221] (will retry): cluster not yet healthy
INSTANCE GROUPS
NAME			ROLE	MACHINETYPE	MIN	MAX	SUBNETS
master-us-east-1a	Master	c5.large	1	1	us-east-1a
nodes-us-east-1a	Node	t3.medium	4	4	us-east-1a

NODE STATUS
NAME	ROLE	READY

VALIDATION ERRORS
KIND	NAME		MESSAGE
dns	apiserver	Validation Failed

The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address.  The API DNS IP address is the placeholder address that kops creates: 203.0.113.123.  Please wait about 5-10 minutes for a master to start, dns-controller to launch, and DNS to propagate.  The protokube container and dns-controller deployment logs may contain more diagnostic information.  Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start.

Validation Failed
W1018 04:29:14.870795    5010 validate_cluster.go:221] (will retry): cluster not yet healthy
INSTANCE GROUPS
NAME			ROLE	MACHINETYPE	MIN	MAX	SUBNETS
master-us-east-1a	Master	c5.large	1	1	us-east-1a
nodes-us-east-1a	Node	t3.medium	4	4	us-east-1a

NODE STATUS
NAME	ROLE	READY

VALIDATION ERRORS
KIND	NAME		MESSAGE
dns	apiserver	Validation Failed

The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address.  The API DNS IP address is the placeholder address that kops creates: 203.0.113.123.  Please wait about 5-10 minutes for a master to start, dns-controller to launch, and DNS to propagate.  The protokube container and dns-controller deployment logs may contain more diagnostic information.  Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start.

Validation Failed
W1018 04:29:24.915332    5010 validate_cluster.go:221] (will retry): cluster not yet healthy
INSTANCE GROUPS
NAME			ROLE	MACHINETYPE	MIN	MAX	SUBNETS
master-us-east-1a	Master	c5.large	1	1	us-east-1a
nodes-us-east-1a	Node	t3.medium	4	4	us-east-1a

NODE STATUS
NAME	ROLE	READY

VALIDATION ERRORS
KIND	NAME		MESSAGE
dns	apiserver	Validation Failed

The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address.  The API DNS IP address is the placeholder address that kops creates: 203.0.113.123.  Please wait about 5-10 minutes for a master to start, dns-controller to launch, and DNS to propagate.  The protokube container and dns-controller deployment logs may contain more diagnostic information.  Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start.

Validation Failed
W1018 04:29:34.957596    5010 validate_cluster.go:221] (will retry): cluster not yet healthy
INSTANCE GROUPS
NAME			ROLE	MACHINETYPE	MIN	MAX	SUBNETS
master-us-east-1a	Master	c5.large	1	1	us-east-1a
nodes-us-east-1a	Node	t3.medium	4	4	us-east-1a

NODE STATUS
NAME	ROLE	READY

VALIDATION ERRORS
KIND	NAME		MESSAGE
dns	apiserver	Validation Failed

The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address.  The API DNS IP address is the placeholder address that kops creates: 203.0.113.123.  Please wait about 5-10 minutes for a master to start, dns-controller to launch, and DNS to propagate.  The protokube container and dns-controller deployment logs may contain more diagnostic information.  Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start.

Validation Failed
W1018 04:29:44.995093    5010 validate_cluster.go:221] (will retry): cluster not yet healthy
INSTANCE GROUPS
NAME			ROLE	MACHINETYPE	MIN	MAX	SUBNETS
master-us-east-1a	Master	c5.large	1	1	us-east-1a
nodes-us-east-1a	Node	t3.medium	4	4	us-east-1a

NODE STATUS
NAME	ROLE	READY

VALIDATION ERRORS
KIND	NAME		MESSAGE
dns	apiserver	Validation Failed

The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address.  The API DNS IP address is the placeholder address that kops creates: 203.0.113.123.  Please wait about 5-10 minutes for a master to start, dns-controller to launch, and DNS to propagate.  The protokube container and dns-controller deployment logs may contain more diagnostic information.  Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start.

Validation Failed
W1018 04:29:55.027752    5010 validate_cluster.go:221] (will retry): cluster not yet healthy
INSTANCE GROUPS
NAME			ROLE	MACHINETYPE	MIN	MAX	SUBNETS
master-us-east-1a	Master	c5.large	1	1	us-east-1a
nodes-us-east-1a	Node	t3.medium	4	4	us-east-1a

... skipping 7 lines ...
Machine	i-0536a4c38d6280c6e				machine "i-0536a4c38d6280c6e" has not yet joined cluster
Machine	i-06c9c19da3837eeac				machine "i-06c9c19da3837eeac" has not yet joined cluster
Machine	i-09952bcfbd8739387				machine "i-09952bcfbd8739387" has not yet joined cluster
Pod	kube-system/coredns-5dc785954d-j8hfv		system-cluster-critical pod "coredns-5dc785954d-j8hfv" is pending
Pod	kube-system/coredns-autoscaler-84d4cfd89c-956bj	system-cluster-critical pod "coredns-autoscaler-84d4cfd89c-956bj" is pending

Validation Failed
W1018 04:30:06.827579    5010 validate_cluster.go:221] (will retry): cluster not yet healthy
W1018 04:30:16.862430    5010 validate_cluster.go:173] (will retry): unexpected error during validation: unable to resolve Kubernetes cluster API URL dns: lookup api.e2e-89d41a7532-e6156.test-cncf-aws.k8s.io on 10.63.240.10:53: no such host
INSTANCE GROUPS
NAME			ROLE	MACHINETYPE	MIN	MAX	SUBNETS
master-us-east-1a	Master	c5.large	1	1	us-east-1a
nodes-us-east-1a	Node	t3.medium	4	4	us-east-1a

NODE STATUS
... skipping 14 lines ...
Pod	kube-system/calico-node-m58dw			system-node-critical pod "calico-node-m58dw" is pending
Pod	kube-system/calico-node-mjn7h			system-node-critical pod "calico-node-mjn7h" is pending
Pod	kube-system/calico-node-nc4sb			system-node-critical pod "calico-node-nc4sb" is pending
Pod	kube-system/coredns-5dc785954d-j8hfv		system-cluster-critical pod "coredns-5dc785954d-j8hfv" is pending
Pod	kube-system/coredns-autoscaler-84d4cfd89c-956bj	system-cluster-critical pod "coredns-autoscaler-84d4cfd89c-956bj" is pending

Validation Failed
W1018 04:30:28.265858    5010 validate_cluster.go:221] (will retry): cluster not yet healthy
INSTANCE GROUPS
NAME			ROLE	MACHINETYPE	MIN	MAX	SUBNETS
master-us-east-1a	Master	c5.large	1	1	us-east-1a
nodes-us-east-1a	Node	t3.medium	4	4	us-east-1a

... skipping 11 lines ...
Pod	kube-system/calico-node-7qhm5		system-node-critical pod "calico-node-7qhm5" is not ready (calico-node)
Pod	kube-system/calico-node-m58dw		system-node-critical pod "calico-node-m58dw" is not ready (calico-node)
Pod	kube-system/calico-node-mjn7h		system-node-critical pod "calico-node-mjn7h" is not ready (calico-node)
Pod	kube-system/calico-node-nc4sb		system-node-critical pod "calico-node-nc4sb" is not ready (calico-node)
Pod	kube-system/coredns-5dc785954d-d24qb	system-cluster-critical pod "coredns-5dc785954d-d24qb" is pending

Validation Failed
W1018 04:30:39.305786    5010 validate_cluster.go:221] (will retry): cluster not yet healthy
INSTANCE GROUPS
NAME			ROLE	MACHINETYPE	MIN	MAX	SUBNETS
master-us-east-1a	Master	c5.large	1	1	us-east-1a
nodes-us-east-1a	Node	t3.medium	4	4	us-east-1a

... skipping 7 lines ...

VALIDATION ERRORS
KIND	NAME				MESSAGE
Pod	kube-system/calico-node-m58dw	system-node-critical pod "calico-node-m58dw" is not ready (calico-node)
Pod	kube-system/calico-node-mjn7h	system-node-critical pod "calico-node-mjn7h" is not ready (calico-node)

Validation Failed
W1018 04:30:50.351370    5010 validate_cluster.go:221] (will retry): cluster not yet healthy
INSTANCE GROUPS
NAME			ROLE	MACHINETYPE	MIN	MAX	SUBNETS
master-us-east-1a	Master	c5.large	1	1	us-east-1a
nodes-us-east-1a	Node	t3.medium	4	4	us-east-1a

... skipping 806 lines ...
  	                    	        for cmd in "${commands[@]}"; do
  	                    	...
  	                    	            continue
  	                    	          fi
  	                    	+         if ! validate-hash "${file}" "${hash}"; then
  	                    	-         if [[ -n "${hash}" ]] && ! validate-hash "${file}" "${hash}"; then
  	                    	            echo "== Hash validation of ${url} failed. Retrying. =="
  	                    	            rm -f "${file}"
  	                    	          else
  	                    	-           if [[ -n "${hash}" ]]; then
  	                    	+           echo "== Downloaded ${url} (SHA256 = ${hash}) =="
  	                    	-             echo "== Downloaded ${url} (SHA1 = ${hash}) =="
  	                    	-           else
... skipping 301 lines ...
  	                    	        for cmd in "${commands[@]}"; do
  	                    	...
  	                    	            continue
  	                    	          fi
  	                    	+         if ! validate-hash "${file}" "${hash}"; then
  	                    	-         if [[ -n "${hash}" ]] && ! validate-hash "${file}" "${hash}"; then
  	                    	            echo "== Hash validation of ${url} failed. Retrying. =="
  	                    	            rm -f "${file}"
  	                    	          else
  	                    	-           if [[ -n "${hash}" ]]; then
  	                    	+           echo "== Downloaded ${url} (SHA256 = ${hash}) =="
  	                    	-             echo "== Downloaded ${url} (SHA1 = ${hash}) =="
  	                    	-           else
... skipping 4382 lines ...
evicting pod kube-system/dns-controller-7cf9d66d6d-vs6h6
I1018 04:34:27.913403    5134 instancegroups.go:658] Waiting for 5s for pods to stabilize after draining.
I1018 04:34:32.914962    5134 instancegroups.go:417] deleting node "ip-172-20-32-94.ec2.internal" from kubernetes
I1018 04:34:32.948649    5134 instancegroups.go:591] Stopping instance "i-046455eea5654933c", node "ip-172-20-32-94.ec2.internal", in group "master-us-east-1a.masters.e2e-89d41a7532-e6156.test-cncf-aws.k8s.io" (this may take a while).
I1018 04:34:33.121236    5134 instancegroups.go:435] waiting for 15s after terminating instance
I1018 04:34:48.123901    5134 instancegroups.go:470] Validating the cluster.
I1018 04:35:18.169356    5134 instancegroups.go:516] Cluster did not validate, will retry in "30s": error listing nodes: Get "https://api.e2e-89d41a7532-e6156.test-cncf-aws.k8s.io/api/v1/nodes": dial tcp 34.235.136.220:443: i/o timeout.
I1018 04:36:18.209128    5134 instancegroups.go:516] Cluster did not validate, will retry in "30s": error listing nodes: Get "https://api.e2e-89d41a7532-e6156.test-cncf-aws.k8s.io/api/v1/nodes": dial tcp 34.235.136.220:443: i/o timeout.
I1018 04:37:18.275822    5134 instancegroups.go:516] Cluster did not validate, will retry in "30s": error listing nodes: Get "https://api.e2e-89d41a7532-e6156.test-cncf-aws.k8s.io/api/v1/nodes": dial tcp 34.235.136.220:443: i/o timeout.
I1018 04:38:18.318943    5134 instancegroups.go:516] Cluster did not validate, will retry in "30s": error listing nodes: Get "https://api.e2e-89d41a7532-e6156.test-cncf-aws.k8s.io/api/v1/nodes": dial tcp 34.235.136.220:443: i/o timeout.
I1018 04:39:18.358209    5134 instancegroups.go:516] Cluster did not validate, will retry in "30s": error listing nodes: Get "https://api.e2e-89d41a7532-e6156.test-cncf-aws.k8s.io/api/v1/nodes": dial tcp 34.235.136.220:443: i/o timeout.
I1018 04:40:18.392441    5134 instancegroups.go:516] Cluster did not validate, will retry in "30s": error listing nodes: Get "https://api.e2e-89d41a7532-e6156.test-cncf-aws.k8s.io/api/v1/nodes": dial tcp 34.235.136.220:443: i/o timeout.
I1018 04:41:18.438289    5134 instancegroups.go:516] Cluster did not validate, will retry in "30s": error listing nodes: Get "https://api.e2e-89d41a7532-e6156.test-cncf-aws.k8s.io/api/v1/nodes": dial tcp 34.235.136.220:443: i/o timeout.
I1018 04:42:18.482656    5134 instancegroups.go:516] Cluster did not validate, will retry in "30s": error listing nodes: Get "https://api.e2e-89d41a7532-e6156.test-cncf-aws.k8s.io/api/v1/nodes": dial tcp 34.235.136.220:443: i/o timeout.
I1018 04:43:18.524255    5134 instancegroups.go:516] Cluster did not validate, will retry in "30s": error listing nodes: Get "https://api.e2e-89d41a7532-e6156.test-cncf-aws.k8s.io/api/v1/nodes": dial tcp 34.235.136.220:443: i/o timeout.
I1018 04:44:18.572788    5134 instancegroups.go:516] Cluster did not validate, will retry in "30s": error listing nodes: Get "https://api.e2e-89d41a7532-e6156.test-cncf-aws.k8s.io/api/v1/nodes": dial tcp 34.235.136.220:443: i/o timeout.
I1018 04:45:18.612589    5134 instancegroups.go:516] Cluster did not validate, will retry in "30s": error listing nodes: Get "https://api.e2e-89d41a7532-e6156.test-cncf-aws.k8s.io/api/v1/nodes": dial tcp 34.235.136.220:443: i/o timeout.
I1018 04:46:18.650560    5134 instancegroups.go:516] Cluster did not validate, will retry in "30s": error listing nodes: Get "https://api.e2e-89d41a7532-e6156.test-cncf-aws.k8s.io/api/v1/nodes": dial tcp 34.235.136.220:443: i/o timeout.
I1018 04:47:18.685363    5134 instancegroups.go:516] Cluster did not validate, will retry in "30s": error listing nodes: Get "https://api.e2e-89d41a7532-e6156.test-cncf-aws.k8s.io/api/v1/nodes": dial tcp 34.235.136.220:443: i/o timeout.
I1018 04:48:18.726508    5134 instancegroups.go:516] Cluster did not validate, will retry in "30s": error listing nodes: Get "https://api.e2e-89d41a7532-e6156.test-cncf-aws.k8s.io/api/v1/nodes": dial tcp 34.235.136.220:443: i/o timeout.
I1018 04:49:18.781521    5134 instancegroups.go:516] Cluster did not validate, will retry in "30s": error listing nodes: Get "https://api.e2e-89d41a7532-e6156.test-cncf-aws.k8s.io/api/v1/nodes": dial tcp 34.235.136.220:443: i/o timeout.
I1018 04:50:18.821622    5134 instancegroups.go:516] Cluster did not validate, will retry in "30s": error listing nodes: Get "https://api.e2e-89d41a7532-e6156.test-cncf-aws.k8s.io/api/v1/nodes": dial tcp 34.235.136.220:443: i/o timeout.
I1018 04:51:18.857519    5134 instancegroups.go:516] Cluster did not validate, will retry in "30s": error listing nodes: Get "https://api.e2e-89d41a7532-e6156.test-cncf-aws.k8s.io/api/v1/nodes": dial tcp 34.235.136.220:443: i/o timeout.
I1018 04:52:18.897062    5134 instancegroups.go:516] Cluster did not validate, will retry in "30s": error listing nodes: Get "https://api.e2e-89d41a7532-e6156.test-cncf-aws.k8s.io/api/v1/nodes": dial tcp 34.235.136.220:443: i/o timeout.
I1018 04:53:18.935227    5134 instancegroups.go:516] Cluster did not validate, will retry in "30s": error listing nodes: Get "https://api.e2e-89d41a7532-e6156.test-cncf-aws.k8s.io/api/v1/nodes": dial tcp 34.235.136.220:443: i/o timeout.
I1018 04:54:18.992449    5134 instancegroups.go:516] Cluster did not validate, will retry in "30s": error listing nodes: Get "https://api.e2e-89d41a7532-e6156.test-cncf-aws.k8s.io/api/v1/nodes": dial tcp 34.235.136.220:443: i/o timeout.
I1018 04:55:19.029983    5134 instancegroups.go:516] Cluster did not validate, will retry in "30s": error listing nodes: Get "https://api.e2e-89d41a7532-e6156.test-cncf-aws.k8s.io/api/v1/nodes": dial tcp 34.235.136.220:443: i/o timeout.
I1018 04:56:19.075693    5134 instancegroups.go:516] Cluster did not validate, will retry in "30s": error listing nodes: Get "https://api.e2e-89d41a7532-e6156.test-cncf-aws.k8s.io/api/v1/nodes": dial tcp 34.235.136.220:443: i/o timeout.
I1018 04:57:19.117400    5134 instancegroups.go:516] Cluster did not validate, will retry in "30s": error listing nodes: Get "https://api.e2e-89d41a7532-e6156.test-cncf-aws.k8s.io/api/v1/nodes": dial tcp 34.235.136.220:443: i/o timeout.
I1018 04:58:19.150940    5134 instancegroups.go:516] Cluster did not validate, will retry in "30s": error listing nodes: Get "https://api.e2e-89d41a7532-e6156.test-cncf-aws.k8s.io/api/v1/nodes": dial tcp 34.235.136.220:443: i/o timeout.
I1018 04:59:19.189721    5134 instancegroups.go:516] Cluster did not validate, will retry in "30s": error listing nodes: Get "https://api.e2e-89d41a7532-e6156.test-cncf-aws.k8s.io/api/v1/nodes": dial tcp 34.235.136.220:443: i/o timeout.
I1018 05:00:19.229100    5134 instancegroups.go:516] Cluster did not validate, will retry in "30s": error listing nodes: Get "https://api.e2e-89d41a7532-e6156.test-cncf-aws.k8s.io/api/v1/nodes": dial tcp 34.235.136.220:443: i/o timeout.
I1018 05:01:19.273419    5134 instancegroups.go:516] Cluster did not validate, will retry in "30s": error listing nodes: Get "https://api.e2e-89d41a7532-e6156.test-cncf-aws.k8s.io/api/v1/nodes": dial tcp 34.235.136.220:443: i/o timeout.
I1018 05:02:19.314590    5134 instancegroups.go:516] Cluster did not validate, will retry in "30s": error listing nodes: Get "https://api.e2e-89d41a7532-e6156.test-cncf-aws.k8s.io/api/v1/nodes": dial tcp 34.235.136.220:443: i/o timeout.
I1018 05:03:19.355319    5134 instancegroups.go:516] Cluster did not validate, will retry in "30s": error listing nodes: Get "https://api.e2e-89d41a7532-e6156.test-cncf-aws.k8s.io/api/v1/nodes": dial tcp 34.235.136.220:443: i/o timeout.
I1018 05:04:19.394744    5134 instancegroups.go:516] Cluster did not validate, will retry in "30s": error listing nodes: Get "https://api.e2e-89d41a7532-e6156.test-cncf-aws.k8s.io/api/v1/nodes": dial tcp 34.235.136.220:443: i/o timeout.
I1018 05:05:19.448300    5134 instancegroups.go:513] Cluster did not validate within deadline: error listing nodes: Get "https://api.e2e-89d41a7532-e6156.test-cncf-aws.k8s.io/api/v1/nodes": dial tcp 34.235.136.220:443: i/o timeout.
E1018 05:05:19.448370    5134 instancegroups.go:475] Cluster did not validate within 30m0s
Error: master not healthy after update, stopping rolling-update: "error validating cluster after terminating instance: cluster did not validate within a duration of \"30m0s\""
+ kops-finish
+ kubetest2 kops -v=2 --cloud-provider=aws --cluster-name=e2e-89d41a7532-e6156.test-cncf-aws.k8s.io --kops-root=/home/prow/go/src/k8s.io/kops --admin-access= --env=KOPS_FEATURE_FLAGS=SpecOverrideFlag --kops-binary-path=/tmp/kops.BbX2IzPtP --down
I1018 05:05:19.474803    5152 app.go:59] RunDir for this run: "/logs/artifacts/4f648395-2fcb-11ec-8a05-6acfde499713"
I1018 05:05:19.474964    5152 app.go:90] ID for this run: "4f648395-2fcb-11ec-8a05-6acfde499713"
I1018 05:05:19.474990    5152 dumplogs.go:40] /tmp/kops.BbX2IzPtP toolbox dump --name e2e-89d41a7532-e6156.test-cncf-aws.k8s.io --dir /logs/artifacts --private-key /etc/aws-ssh/aws-ssh-private --ssh-user ubuntu
I1018 05:05:19.492153    5163 featureflag.go:166] FeatureFlag "SpecOverrideFlag"=true
I1018 05:05:19.492245    5163 featureflag.go:166] FeatureFlag "AlphaAllowGCE"=true
I1018 05:05:19.492250    5163 featureflag.go:166] FeatureFlag "SpecOverrideFlag"=true
W1018 05:05:55.969640    5163 toolbox_dump.go:168] error listing nodes in cluster: Get "https://api.e2e-89d41a7532-e6156.test-cncf-aws.k8s.io/api/v1/nodes": dial tcp 34.235.136.220:443: i/o timeout
2021/10/18 05:05:55 dumping node not registered in kubernetes: 3.80.99.145
2021/10/18 05:05:55 Dumping node 3.80.99.145
2021/10/18 05:05:58 dumping node not registered in kubernetes: 18.232.150.56
2021/10/18 05:05:58 Dumping node 18.232.150.56
2021/10/18 05:06:00 dumping node not registered in kubernetes: 3.87.89.100
2021/10/18 05:06:00 Dumping node 3.87.89.100
... skipping 1049 lines ...
I1018 05:06:08.226431    5152 dumplogs.go:72] /tmp/kops.BbX2IzPtP get instancegroups --name e2e-89d41a7532-e6156.test-cncf-aws.k8s.io -o yaml
I1018 05:06:09.012175    5152 dumplogs.go:91] kubectl cluster-info dump --all-namespaces -o yaml --output-directory /logs/artifacts/cluster-info
I1018 05:06:39.062715    5152 dumplogs.go:114] /tmp/kops.BbX2IzPtP toolbox dump --name e2e-89d41a7532-e6156.test-cncf-aws.k8s.io --private-key /etc/aws-ssh/aws-ssh-private --ssh-user ubuntu -o yaml
I1018 05:06:45.146176    5152 dumplogs.go:143] ssh -i /etc/aws-ssh/aws-ssh-private -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null ubuntu@3.87.89.100 -- kubectl cluster-info dump --all-namespaces -o yaml --output-directory /tmp/cluster-info
Warning: Permanently added '3.87.89.100' (ECDSA) to the list of known hosts.
The connection to the server 127.0.0.1 was refused - did you specify the right host or port?
W1018 05:06:45.961186    5152 down.go:34] Dumping cluster logs at the start of Down() failed: exit status 1
I1018 05:06:45.961289    5152 down.go:48] /tmp/kops.BbX2IzPtP delete cluster --name e2e-89d41a7532-e6156.test-cncf-aws.k8s.io --yes
I1018 05:06:45.976370    5216 featureflag.go:166] FeatureFlag "SpecOverrideFlag"=true
I1018 05:06:45.976471    5216 featureflag.go:166] FeatureFlag "AlphaAllowGCE"=true
I1018 05:06:45.976475    5216 featureflag.go:166] FeatureFlag "SpecOverrideFlag"=true
TYPE			NAME													ID
autoscaling-config	master-us-east-1a.masters.e2e-89d41a7532-e6156.test-cncf-aws.k8s.io					lt-00d5df69563ae8657
... skipping 444 lines ...