This job view page is being replaced by Spyglass soon. Check out the new job view.
ResultFAILURE
Tests 0 failed / 0 succeeded
Started2021-10-16 06:25
Elapsed47m15s
Revisionmaster

No Test Failures!


Error lines from build-log.txt

... skipping 176 lines ...
I1016 06:26:10.935526    4982 dumplogs.go:40] /tmp/kops.OfqBDEZRF toolbox dump --name e2e-89d41a7532-e6156.test-cncf-aws.k8s.io --dir /logs/artifacts --private-key /etc/aws-ssh/aws-ssh-private --ssh-user ubuntu
I1016 06:26:10.956369    4992 featureflag.go:165] FeatureFlag "SpecOverrideFlag"=true
I1016 06:26:10.956465    4992 featureflag.go:165] FeatureFlag "AlphaAllowGCE"=true
I1016 06:26:10.956471    4992 featureflag.go:165] FeatureFlag "SpecOverrideFlag"=true

Cluster.kops.k8s.io "e2e-89d41a7532-e6156.test-cncf-aws.k8s.io" not found
W1016 06:26:11.490110    4982 down.go:34] Dumping cluster logs at the start of Down() failed: exit status 1
I1016 06:26:11.490151    4982 down.go:48] /tmp/kops.OfqBDEZRF delete cluster --name e2e-89d41a7532-e6156.test-cncf-aws.k8s.io --yes
I1016 06:26:11.506901    5002 featureflag.go:165] FeatureFlag "SpecOverrideFlag"=true
I1016 06:26:11.506998    5002 featureflag.go:165] FeatureFlag "AlphaAllowGCE"=true
I1016 06:26:11.507003    5002 featureflag.go:165] FeatureFlag "SpecOverrideFlag"=true

error reading cluster configuration: Cluster.kops.k8s.io "e2e-89d41a7532-e6156.test-cncf-aws.k8s.io" not found
I1016 06:26:12.009290    4982 http.go:37] curl http://metadata.google.internal/computeMetadata/v1/instance/network-interfaces/0/access-configs/0/external-ip
2021/10/16 06:26:12 failed to get external ip from metadata service: http://metadata.google.internal/computeMetadata/v1/instance/network-interfaces/0/access-configs/0/external-ip returned 404
I1016 06:26:12.016490    4982 http.go:37] curl https://ip.jsb.workers.dev
I1016 06:26:12.123950    4982 up.go:144] /tmp/kops.OfqBDEZRF create cluster --name e2e-89d41a7532-e6156.test-cncf-aws.k8s.io --cloud aws --kubernetes-version v1.21.0 --ssh-public-key /etc/aws-ssh/aws-ssh-public --override cluster.spec.nodePortAccess=0.0.0.0/0 --yes --networking calico --admin-access 35.184.84.142/32 --master-count 1 --master-volume-size 48 --node-count 4 --node-volume-size 48 --zones ap-southeast-2a --master-size c5.large
I1016 06:26:12.138826    5012 featureflag.go:165] FeatureFlag "SpecOverrideFlag"=true
I1016 06:26:12.138900    5012 featureflag.go:165] FeatureFlag "AlphaAllowGCE"=true
I1016 06:26:12.138904    5012 featureflag.go:165] FeatureFlag "SpecOverrideFlag"=true
I1016 06:26:12.189112    5012 create_cluster.go:728] Using SSH public key: /etc/aws-ssh/aws-ssh-public
... skipping 43 lines ...
I1016 06:26:33.048722    4982 up.go:181] /tmp/kops.OfqBDEZRF validate cluster --name e2e-89d41a7532-e6156.test-cncf-aws.k8s.io --count 10 --wait 15m0s
I1016 06:26:33.065883    5032 featureflag.go:165] FeatureFlag "SpecOverrideFlag"=true
I1016 06:26:33.066143    5032 featureflag.go:165] FeatureFlag "AlphaAllowGCE"=true
I1016 06:26:33.066221    5032 featureflag.go:165] FeatureFlag "SpecOverrideFlag"=true
Validating cluster e2e-89d41a7532-e6156.test-cncf-aws.k8s.io

W1016 06:26:34.813112    5032 validate_cluster.go:173] (will retry): unexpected error during validation: unable to resolve Kubernetes cluster API URL dns: lookup api.e2e-89d41a7532-e6156.test-cncf-aws.k8s.io on 10.63.240.10:53: no such host
INSTANCE GROUPS
NAME			ROLE	MACHINETYPE	MIN	MAX	SUBNETS
master-ap-southeast-2a	Master	c5.large	1	1	ap-southeast-2a
nodes-ap-southeast-2a	Node	t3.medium	4	4	ap-southeast-2a

NODE STATUS
NAME	ROLE	READY

VALIDATION ERRORS
KIND	NAME		MESSAGE
dns	apiserver	Validation Failed

The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address.  The API DNS IP address is the placeholder address that kops creates: 203.0.113.123.  Please wait about 5-10 minutes for a master to start, dns-controller to launch, and DNS to propagate.  The protokube container and dns-controller deployment logs may contain more diagnostic information.  Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start.

Validation Failed
W1016 06:26:44.844640    5032 validate_cluster.go:221] (will retry): cluster not yet healthy
INSTANCE GROUPS
NAME			ROLE	MACHINETYPE	MIN	MAX	SUBNETS
master-ap-southeast-2a	Master	c5.large	1	1	ap-southeast-2a
nodes-ap-southeast-2a	Node	t3.medium	4	4	ap-southeast-2a

NODE STATUS
NAME	ROLE	READY

VALIDATION ERRORS
KIND	NAME		MESSAGE
dns	apiserver	Validation Failed

The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address.  The API DNS IP address is the placeholder address that kops creates: 203.0.113.123.  Please wait about 5-10 minutes for a master to start, dns-controller to launch, and DNS to propagate.  The protokube container and dns-controller deployment logs may contain more diagnostic information.  Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start.

Validation Failed
W1016 06:26:54.874202    5032 validate_cluster.go:221] (will retry): cluster not yet healthy
INSTANCE GROUPS
NAME			ROLE	MACHINETYPE	MIN	MAX	SUBNETS
master-ap-southeast-2a	Master	c5.large	1	1	ap-southeast-2a
nodes-ap-southeast-2a	Node	t3.medium	4	4	ap-southeast-2a

NODE STATUS
NAME	ROLE	READY

VALIDATION ERRORS
KIND	NAME		MESSAGE
dns	apiserver	Validation Failed

The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address.  The API DNS IP address is the placeholder address that kops creates: 203.0.113.123.  Please wait about 5-10 minutes for a master to start, dns-controller to launch, and DNS to propagate.  The protokube container and dns-controller deployment logs may contain more diagnostic information.  Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start.

Validation Failed
W1016 06:27:04.921981    5032 validate_cluster.go:221] (will retry): cluster not yet healthy
INSTANCE GROUPS
NAME			ROLE	MACHINETYPE	MIN	MAX	SUBNETS
master-ap-southeast-2a	Master	c5.large	1	1	ap-southeast-2a
nodes-ap-southeast-2a	Node	t3.medium	4	4	ap-southeast-2a

NODE STATUS
NAME	ROLE	READY

VALIDATION ERRORS
KIND	NAME		MESSAGE
dns	apiserver	Validation Failed

The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address.  The API DNS IP address is the placeholder address that kops creates: 203.0.113.123.  Please wait about 5-10 minutes for a master to start, dns-controller to launch, and DNS to propagate.  The protokube container and dns-controller deployment logs may contain more diagnostic information.  Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start.

Validation Failed
W1016 06:27:14.953476    5032 validate_cluster.go:221] (will retry): cluster not yet healthy
INSTANCE GROUPS
NAME			ROLE	MACHINETYPE	MIN	MAX	SUBNETS
master-ap-southeast-2a	Master	c5.large	1	1	ap-southeast-2a
nodes-ap-southeast-2a	Node	t3.medium	4	4	ap-southeast-2a

NODE STATUS
NAME	ROLE	READY

VALIDATION ERRORS
KIND	NAME		MESSAGE
dns	apiserver	Validation Failed

The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address.  The API DNS IP address is the placeholder address that kops creates: 203.0.113.123.  Please wait about 5-10 minutes for a master to start, dns-controller to launch, and DNS to propagate.  The protokube container and dns-controller deployment logs may contain more diagnostic information.  Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start.

Validation Failed
W1016 06:27:24.986915    5032 validate_cluster.go:221] (will retry): cluster not yet healthy
INSTANCE GROUPS
NAME			ROLE	MACHINETYPE	MIN	MAX	SUBNETS
master-ap-southeast-2a	Master	c5.large	1	1	ap-southeast-2a
nodes-ap-southeast-2a	Node	t3.medium	4	4	ap-southeast-2a

NODE STATUS
NAME	ROLE	READY

VALIDATION ERRORS
KIND	NAME		MESSAGE
dns	apiserver	Validation Failed

The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address.  The API DNS IP address is the placeholder address that kops creates: 203.0.113.123.  Please wait about 5-10 minutes for a master to start, dns-controller to launch, and DNS to propagate.  The protokube container and dns-controller deployment logs may contain more diagnostic information.  Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start.

Validation Failed
W1016 06:27:35.019522    5032 validate_cluster.go:221] (will retry): cluster not yet healthy
INSTANCE GROUPS
NAME			ROLE	MACHINETYPE	MIN	MAX	SUBNETS
master-ap-southeast-2a	Master	c5.large	1	1	ap-southeast-2a
nodes-ap-southeast-2a	Node	t3.medium	4	4	ap-southeast-2a

NODE STATUS
NAME	ROLE	READY

VALIDATION ERRORS
KIND	NAME		MESSAGE
dns	apiserver	Validation Failed

The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address.  The API DNS IP address is the placeholder address that kops creates: 203.0.113.123.  Please wait about 5-10 minutes for a master to start, dns-controller to launch, and DNS to propagate.  The protokube container and dns-controller deployment logs may contain more diagnostic information.  Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start.

Validation Failed
W1016 06:27:45.055958    5032 validate_cluster.go:221] (will retry): cluster not yet healthy
INSTANCE GROUPS
NAME			ROLE	MACHINETYPE	MIN	MAX	SUBNETS
master-ap-southeast-2a	Master	c5.large	1	1	ap-southeast-2a
nodes-ap-southeast-2a	Node	t3.medium	4	4	ap-southeast-2a

NODE STATUS
NAME	ROLE	READY

VALIDATION ERRORS
KIND	NAME		MESSAGE
dns	apiserver	Validation Failed

The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address.  The API DNS IP address is the placeholder address that kops creates: 203.0.113.123.  Please wait about 5-10 minutes for a master to start, dns-controller to launch, and DNS to propagate.  The protokube container and dns-controller deployment logs may contain more diagnostic information.  Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start.

Validation Failed
W1016 06:27:55.090458    5032 validate_cluster.go:221] (will retry): cluster not yet healthy
INSTANCE GROUPS
NAME			ROLE	MACHINETYPE	MIN	MAX	SUBNETS
master-ap-southeast-2a	Master	c5.large	1	1	ap-southeast-2a
nodes-ap-southeast-2a	Node	t3.medium	4	4	ap-southeast-2a

NODE STATUS
NAME	ROLE	READY

VALIDATION ERRORS
KIND	NAME		MESSAGE
dns	apiserver	Validation Failed

The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address.  The API DNS IP address is the placeholder address that kops creates: 203.0.113.123.  Please wait about 5-10 minutes for a master to start, dns-controller to launch, and DNS to propagate.  The protokube container and dns-controller deployment logs may contain more diagnostic information.  Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start.

Validation Failed
W1016 06:28:05.121277    5032 validate_cluster.go:221] (will retry): cluster not yet healthy
INSTANCE GROUPS
NAME			ROLE	MACHINETYPE	MIN	MAX	SUBNETS
master-ap-southeast-2a	Master	c5.large	1	1	ap-southeast-2a
nodes-ap-southeast-2a	Node	t3.medium	4	4	ap-southeast-2a

NODE STATUS
NAME	ROLE	READY

VALIDATION ERRORS
KIND	NAME		MESSAGE
dns	apiserver	Validation Failed

The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address.  The API DNS IP address is the placeholder address that kops creates: 203.0.113.123.  Please wait about 5-10 minutes for a master to start, dns-controller to launch, and DNS to propagate.  The protokube container and dns-controller deployment logs may contain more diagnostic information.  Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start.

Validation Failed
W1016 06:28:15.156358    5032 validate_cluster.go:221] (will retry): cluster not yet healthy
INSTANCE GROUPS
NAME			ROLE	MACHINETYPE	MIN	MAX	SUBNETS
master-ap-southeast-2a	Master	c5.large	1	1	ap-southeast-2a
nodes-ap-southeast-2a	Node	t3.medium	4	4	ap-southeast-2a

NODE STATUS
NAME	ROLE	READY

VALIDATION ERRORS
KIND	NAME		MESSAGE
dns	apiserver	Validation Failed

The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address.  The API DNS IP address is the placeholder address that kops creates: 203.0.113.123.  Please wait about 5-10 minutes for a master to start, dns-controller to launch, and DNS to propagate.  The protokube container and dns-controller deployment logs may contain more diagnostic information.  Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start.

Validation Failed
W1016 06:28:25.184657    5032 validate_cluster.go:221] (will retry): cluster not yet healthy
INSTANCE GROUPS
NAME			ROLE	MACHINETYPE	MIN	MAX	SUBNETS
master-ap-southeast-2a	Master	c5.large	1	1	ap-southeast-2a
nodes-ap-southeast-2a	Node	t3.medium	4	4	ap-southeast-2a

NODE STATUS
NAME	ROLE	READY

VALIDATION ERRORS
KIND	NAME		MESSAGE
dns	apiserver	Validation Failed

The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address.  The API DNS IP address is the placeholder address that kops creates: 203.0.113.123.  Please wait about 5-10 minutes for a master to start, dns-controller to launch, and DNS to propagate.  The protokube container and dns-controller deployment logs may contain more diagnostic information.  Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start.

Validation Failed
W1016 06:28:35.214518    5032 validate_cluster.go:221] (will retry): cluster not yet healthy
INSTANCE GROUPS
NAME			ROLE	MACHINETYPE	MIN	MAX	SUBNETS
master-ap-southeast-2a	Master	c5.large	1	1	ap-southeast-2a
nodes-ap-southeast-2a	Node	t3.medium	4	4	ap-southeast-2a

NODE STATUS
NAME	ROLE	READY

VALIDATION ERRORS
KIND	NAME		MESSAGE
dns	apiserver	Validation Failed

The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address.  The API DNS IP address is the placeholder address that kops creates: 203.0.113.123.  Please wait about 5-10 minutes for a master to start, dns-controller to launch, and DNS to propagate.  The protokube container and dns-controller deployment logs may contain more diagnostic information.  Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start.

Validation Failed
W1016 06:28:45.248911    5032 validate_cluster.go:221] (will retry): cluster not yet healthy
INSTANCE GROUPS
NAME			ROLE	MACHINETYPE	MIN	MAX	SUBNETS
master-ap-southeast-2a	Master	c5.large	1	1	ap-southeast-2a
nodes-ap-southeast-2a	Node	t3.medium	4	4	ap-southeast-2a

NODE STATUS
NAME	ROLE	READY

VALIDATION ERRORS
KIND	NAME		MESSAGE
dns	apiserver	Validation Failed

The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address.  The API DNS IP address is the placeholder address that kops creates: 203.0.113.123.  Please wait about 5-10 minutes for a master to start, dns-controller to launch, and DNS to propagate.  The protokube container and dns-controller deployment logs may contain more diagnostic information.  Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start.

Validation Failed
W1016 06:28:55.291324    5032 validate_cluster.go:221] (will retry): cluster not yet healthy
INSTANCE GROUPS
NAME			ROLE	MACHINETYPE	MIN	MAX	SUBNETS
master-ap-southeast-2a	Master	c5.large	1	1	ap-southeast-2a
nodes-ap-southeast-2a	Node	t3.medium	4	4	ap-southeast-2a

NODE STATUS
NAME	ROLE	READY

VALIDATION ERRORS
KIND	NAME		MESSAGE
dns	apiserver	Validation Failed

The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address.  The API DNS IP address is the placeholder address that kops creates: 203.0.113.123.  Please wait about 5-10 minutes for a master to start, dns-controller to launch, and DNS to propagate.  The protokube container and dns-controller deployment logs may contain more diagnostic information.  Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start.

Validation Failed
W1016 06:29:05.322149    5032 validate_cluster.go:221] (will retry): cluster not yet healthy
INSTANCE GROUPS
NAME			ROLE	MACHINETYPE	MIN	MAX	SUBNETS
master-ap-southeast-2a	Master	c5.large	1	1	ap-southeast-2a
nodes-ap-southeast-2a	Node	t3.medium	4	4	ap-southeast-2a

NODE STATUS
NAME	ROLE	READY

VALIDATION ERRORS
KIND	NAME		MESSAGE
dns	apiserver	Validation Failed

The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address.  The API DNS IP address is the placeholder address that kops creates: 203.0.113.123.  Please wait about 5-10 minutes for a master to start, dns-controller to launch, and DNS to propagate.  The protokube container and dns-controller deployment logs may contain more diagnostic information.  Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start.

Validation Failed
W1016 06:29:15.351317    5032 validate_cluster.go:221] (will retry): cluster not yet healthy
INSTANCE GROUPS
NAME			ROLE	MACHINETYPE	MIN	MAX	SUBNETS
master-ap-southeast-2a	Master	c5.large	1	1	ap-southeast-2a
nodes-ap-southeast-2a	Node	t3.medium	4	4	ap-southeast-2a

NODE STATUS
NAME	ROLE	READY

VALIDATION ERRORS
KIND	NAME		MESSAGE
dns	apiserver	Validation Failed

The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address.  The API DNS IP address is the placeholder address that kops creates: 203.0.113.123.  Please wait about 5-10 minutes for a master to start, dns-controller to launch, and DNS to propagate.  The protokube container and dns-controller deployment logs may contain more diagnostic information.  Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start.

Validation Failed
W1016 06:29:25.380904    5032 validate_cluster.go:221] (will retry): cluster not yet healthy
INSTANCE GROUPS
NAME			ROLE	MACHINETYPE	MIN	MAX	SUBNETS
master-ap-southeast-2a	Master	c5.large	1	1	ap-southeast-2a
nodes-ap-southeast-2a	Node	t3.medium	4	4	ap-southeast-2a

NODE STATUS
NAME	ROLE	READY

VALIDATION ERRORS
KIND	NAME		MESSAGE
dns	apiserver	Validation Failed

The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address.  The API DNS IP address is the placeholder address that kops creates: 203.0.113.123.  Please wait about 5-10 minutes for a master to start, dns-controller to launch, and DNS to propagate.  The protokube container and dns-controller deployment logs may contain more diagnostic information.  Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start.

Validation Failed
W1016 06:29:35.411803    5032 validate_cluster.go:221] (will retry): cluster not yet healthy
INSTANCE GROUPS
NAME			ROLE	MACHINETYPE	MIN	MAX	SUBNETS
master-ap-southeast-2a	Master	c5.large	1	1	ap-southeast-2a
nodes-ap-southeast-2a	Node	t3.medium	4	4	ap-southeast-2a

NODE STATUS
NAME	ROLE	READY

VALIDATION ERRORS
KIND	NAME		MESSAGE
dns	apiserver	Validation Failed

The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address.  The API DNS IP address is the placeholder address that kops creates: 203.0.113.123.  Please wait about 5-10 minutes for a master to start, dns-controller to launch, and DNS to propagate.  The protokube container and dns-controller deployment logs may contain more diagnostic information.  Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start.

Validation Failed
W1016 06:29:45.446687    5032 validate_cluster.go:221] (will retry): cluster not yet healthy
INSTANCE GROUPS
NAME			ROLE	MACHINETYPE	MIN	MAX	SUBNETS
master-ap-southeast-2a	Master	c5.large	1	1	ap-southeast-2a
nodes-ap-southeast-2a	Node	t3.medium	4	4	ap-southeast-2a

NODE STATUS
NAME	ROLE	READY

VALIDATION ERRORS
KIND	NAME		MESSAGE
dns	apiserver	Validation Failed

The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address.  The API DNS IP address is the placeholder address that kops creates: 203.0.113.123.  Please wait about 5-10 minutes for a master to start, dns-controller to launch, and DNS to propagate.  The protokube container and dns-controller deployment logs may contain more diagnostic information.  Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start.

Validation Failed
W1016 06:29:55.476766    5032 validate_cluster.go:221] (will retry): cluster not yet healthy
INSTANCE GROUPS
NAME			ROLE	MACHINETYPE	MIN	MAX	SUBNETS
master-ap-southeast-2a	Master	c5.large	1	1	ap-southeast-2a
nodes-ap-southeast-2a	Node	t3.medium	4	4	ap-southeast-2a

... skipping 9 lines ...
Machine	i-0e5ecba1185318ef0					machine "i-0e5ecba1185318ef0" has not yet joined cluster
Pod	kube-system/calico-kube-controllers-6b59cd85f8-lnhcv	system-cluster-critical pod "calico-kube-controllers-6b59cd85f8-lnhcv" is pending
Pod	kube-system/calico-node-c9rxc				system-node-critical pod "calico-node-c9rxc" is pending
Pod	kube-system/coredns-5dc785954d-qs6xf			system-cluster-critical pod "coredns-5dc785954d-qs6xf" is pending
Pod	kube-system/coredns-autoscaler-84d4cfd89c-hvndz		system-cluster-critical pod "coredns-autoscaler-84d4cfd89c-hvndz" is pending

Validation Failed
W1016 06:30:10.218587    5032 validate_cluster.go:221] (will retry): cluster not yet healthy
INSTANCE GROUPS
NAME			ROLE	MACHINETYPE	MIN	MAX	SUBNETS
master-ap-southeast-2a	Master	c5.large	1	1	ap-southeast-2a
nodes-ap-southeast-2a	Node	t3.medium	4	4	ap-southeast-2a

... skipping 11 lines ...
Pod	kube-system/calico-kube-controllers-6b59cd85f8-lnhcv			system-cluster-critical pod "calico-kube-controllers-6b59cd85f8-lnhcv" is pending
Pod	kube-system/calico-node-b6sdn						system-node-critical pod "calico-node-b6sdn" is pending
Pod	kube-system/coredns-5dc785954d-qs6xf					system-cluster-critical pod "coredns-5dc785954d-qs6xf" is pending
Pod	kube-system/coredns-autoscaler-84d4cfd89c-hvndz				system-cluster-critical pod "coredns-autoscaler-84d4cfd89c-hvndz" is pending
Pod	kube-system/kube-proxy-ip-172-20-61-28.ap-southeast-2.compute.internal	system-node-critical pod "kube-proxy-ip-172-20-61-28.ap-southeast-2.compute.internal" is pending

Validation Failed
W1016 06:30:23.476183    5032 validate_cluster.go:221] (will retry): cluster not yet healthy
INSTANCE GROUPS
NAME			ROLE	MACHINETYPE	MIN	MAX	SUBNETS
master-ap-southeast-2a	Master	c5.large	1	1	ap-southeast-2a
nodes-ap-southeast-2a	Node	t3.medium	4	4	ap-southeast-2a

... skipping 16 lines ...
Pod	kube-system/coredns-5dc785954d-qs6xf					system-cluster-critical pod "coredns-5dc785954d-qs6xf" is pending
Pod	kube-system/coredns-autoscaler-84d4cfd89c-hvndz				system-cluster-critical pod "coredns-autoscaler-84d4cfd89c-hvndz" is pending
Pod	kube-system/kube-proxy-ip-172-20-38-252.ap-southeast-2.compute.internal	system-node-critical pod "kube-proxy-ip-172-20-38-252.ap-southeast-2.compute.internal" is pending
Pod	kube-system/kube-proxy-ip-172-20-39-5.ap-southeast-2.compute.internal	system-node-critical pod "kube-proxy-ip-172-20-39-5.ap-southeast-2.compute.internal" is pending
Pod	kube-system/kube-proxy-ip-172-20-61-28.ap-southeast-2.compute.internal	system-node-critical pod "kube-proxy-ip-172-20-61-28.ap-southeast-2.compute.internal" is pending

Validation Failed
W1016 06:30:36.779625    5032 validate_cluster.go:221] (will retry): cluster not yet healthy
INSTANCE GROUPS
NAME			ROLE	MACHINETYPE	MIN	MAX	SUBNETS
master-ap-southeast-2a	Master	c5.large	1	1	ap-southeast-2a
nodes-ap-southeast-2a	Node	t3.medium	4	4	ap-southeast-2a

... skipping 17 lines ...
Pod	kube-system/coredns-5dc785954d-qs6xf					system-cluster-critical pod "coredns-5dc785954d-qs6xf" is pending
Pod	kube-system/coredns-autoscaler-84d4cfd89c-hvndz				system-cluster-critical pod "coredns-autoscaler-84d4cfd89c-hvndz" is pending
Pod	kube-system/kube-proxy-ip-172-20-38-252.ap-southeast-2.compute.internal	system-node-critical pod "kube-proxy-ip-172-20-38-252.ap-southeast-2.compute.internal" is pending
Pod	kube-system/kube-proxy-ip-172-20-39-5.ap-southeast-2.compute.internal	system-node-critical pod "kube-proxy-ip-172-20-39-5.ap-southeast-2.compute.internal" is pending
Pod	kube-system/kube-proxy-ip-172-20-55-180.ap-southeast-2.compute.internal	system-node-critical pod "kube-proxy-ip-172-20-55-180.ap-southeast-2.compute.internal" is pending

Validation Failed
W1016 06:30:49.763986    5032 validate_cluster.go:221] (will retry): cluster not yet healthy
INSTANCE GROUPS
NAME			ROLE	MACHINETYPE	MIN	MAX	SUBNETS
master-ap-southeast-2a	Master	c5.large	1	1	ap-southeast-2a
nodes-ap-southeast-2a	Node	t3.medium	4	4	ap-southeast-2a

... skipping 10 lines ...
Pod	kube-system/calico-node-962b2		system-node-critical pod "calico-node-962b2" is pending
Pod	kube-system/calico-node-h4hbh		system-node-critical pod "calico-node-h4hbh" is pending
Pod	kube-system/calico-node-wwv9v		system-node-critical pod "calico-node-wwv9v" is pending
Pod	kube-system/coredns-5dc785954d-ljbsw	system-cluster-critical pod "coredns-5dc785954d-ljbsw" is pending
Pod	kube-system/coredns-5dc785954d-qs6xf	system-cluster-critical pod "coredns-5dc785954d-qs6xf" is not ready (coredns)

Validation Failed
W1016 06:31:03.085515    5032 validate_cluster.go:221] (will retry): cluster not yet healthy
INSTANCE GROUPS
NAME			ROLE	MACHINETYPE	MIN	MAX	SUBNETS
master-ap-southeast-2a	Master	c5.large	1	1	ap-southeast-2a
nodes-ap-southeast-2a	Node	t3.medium	4	4	ap-southeast-2a

... skipping 8 lines ...
VALIDATION ERRORS
KIND	NAME					MESSAGE
Pod	kube-system/calico-node-962b2		system-node-critical pod "calico-node-962b2" is not ready (calico-node)
Pod	kube-system/calico-node-h4hbh		system-node-critical pod "calico-node-h4hbh" is not ready (calico-node)
Pod	kube-system/coredns-5dc785954d-ljbsw	system-cluster-critical pod "coredns-5dc785954d-ljbsw" is pending

Validation Failed
W1016 06:31:16.343073    5032 validate_cluster.go:221] (will retry): cluster not yet healthy
INSTANCE GROUPS
NAME			ROLE	MACHINETYPE	MIN	MAX	SUBNETS
master-ap-southeast-2a	Master	c5.large	1	1	ap-southeast-2a
nodes-ap-southeast-2a	Node	t3.medium	4	4	ap-southeast-2a

... skipping 799 lines ...
  	                    	        for cmd in "${commands[@]}"; do
  	                    	...
  	                    	            continue
  	                    	          fi
  	                    	+         if ! validate-hash "${file}" "${hash}"; then
  	                    	-         if [[ -n "${hash}" ]] && ! validate-hash "${file}" "${hash}"; then
  	                    	            echo "== Hash validation of ${url} failed. Retrying. =="
  	                    	            rm -f "${file}"
  	                    	          else
  	                    	-           if [[ -n "${hash}" ]]; then
  	                    	+           echo "== Downloaded ${url} (SHA256 = ${hash}) =="
  	                    	-             echo "== Downloaded ${url} (SHA1 = ${hash}) =="
  	                    	-           else
... skipping 294 lines ...
  	                    	        for cmd in "${commands[@]}"; do
  	                    	...
  	                    	            continue
  	                    	          fi
  	                    	+         if ! validate-hash "${file}" "${hash}"; then
  	                    	-         if [[ -n "${hash}" ]] && ! validate-hash "${file}" "${hash}"; then
  	                    	            echo "== Hash validation of ${url} failed. Retrying. =="
  	                    	            rm -f "${file}"
  	                    	          else
  	                    	-           if [[ -n "${hash}" ]]; then
  	                    	+           echo "== Downloaded ${url} (SHA256 = ${hash}) =="
  	                    	-             echo "== Downloaded ${url} (SHA1 = ${hash}) =="
  	                    	-           else
... skipping 6193 lines ...
evicting pod kube-system/dns-controller-7cf9d66d6d-z9fct
I1016 06:36:01.271899    5162 instancegroups.go:658] Waiting for 5s for pods to stabilize after draining.
I1016 06:36:06.274056    5162 instancegroups.go:417] deleting node "ip-172-20-44-64.ap-southeast-2.compute.internal" from kubernetes
I1016 06:36:06.464639    5162 instancegroups.go:591] Stopping instance "i-09310d7b4796c6465", node "ip-172-20-44-64.ap-southeast-2.compute.internal", in group "master-ap-southeast-2a.masters.e2e-89d41a7532-e6156.test-cncf-aws.k8s.io" (this may take a while).
I1016 06:36:06.781948    5162 instancegroups.go:435] waiting for 15s after terminating instance
I1016 06:36:21.782937    5162 instancegroups.go:470] Validating the cluster.
I1016 06:36:51.812225    5162 instancegroups.go:516] Cluster did not validate, will retry in "30s": error listing nodes: Get "https://api.e2e-89d41a7532-e6156.test-cncf-aws.k8s.io/api/v1/nodes": dial tcp 3.25.119.95:443: i/o timeout.
I1016 06:37:51.843955    5162 instancegroups.go:516] Cluster did not validate, will retry in "30s": error listing nodes: Get "https://api.e2e-89d41a7532-e6156.test-cncf-aws.k8s.io/api/v1/nodes": dial tcp 3.25.119.95:443: i/o timeout.
I1016 06:38:51.878194    5162 instancegroups.go:516] Cluster did not validate, will retry in "30s": error listing nodes: Get "https://api.e2e-89d41a7532-e6156.test-cncf-aws.k8s.io/api/v1/nodes": dial tcp 3.25.119.95:443: i/o timeout.
I1016 06:39:51.910948    5162 instancegroups.go:516] Cluster did not validate, will retry in "30s": error listing nodes: Get "https://api.e2e-89d41a7532-e6156.test-cncf-aws.k8s.io/api/v1/nodes": dial tcp 3.25.119.95:443: i/o timeout.
I1016 06:40:51.951004    5162 instancegroups.go:516] Cluster did not validate, will retry in "30s": error listing nodes: Get "https://api.e2e-89d41a7532-e6156.test-cncf-aws.k8s.io/api/v1/nodes": dial tcp 3.25.119.95:443: i/o timeout.
I1016 06:41:51.991123    5162 instancegroups.go:516] Cluster did not validate, will retry in "30s": error listing nodes: Get "https://api.e2e-89d41a7532-e6156.test-cncf-aws.k8s.io/api/v1/nodes": dial tcp 3.25.119.95:443: i/o timeout.
I1016 06:42:52.022791    5162 instancegroups.go:516] Cluster did not validate, will retry in "30s": error listing nodes: Get "https://api.e2e-89d41a7532-e6156.test-cncf-aws.k8s.io/api/v1/nodes": dial tcp 3.25.119.95:443: i/o timeout.
I1016 06:43:52.062045    5162 instancegroups.go:516] Cluster did not validate, will retry in "30s": error listing nodes: Get "https://api.e2e-89d41a7532-e6156.test-cncf-aws.k8s.io/api/v1/nodes": dial tcp 3.25.119.95:443: i/o timeout.
I1016 06:44:52.095971    5162 instancegroups.go:516] Cluster did not validate, will retry in "30s": error listing nodes: Get "https://api.e2e-89d41a7532-e6156.test-cncf-aws.k8s.io/api/v1/nodes": dial tcp 3.25.119.95:443: i/o timeout.
I1016 06:45:52.145936    5162 instancegroups.go:516] Cluster did not validate, will retry in "30s": error listing nodes: Get "https://api.e2e-89d41a7532-e6156.test-cncf-aws.k8s.io/api/v1/nodes": dial tcp 3.25.119.95:443: i/o timeout.
I1016 06:46:52.181252    5162 instancegroups.go:516] Cluster did not validate, will retry in "30s": error listing nodes: Get "https://api.e2e-89d41a7532-e6156.test-cncf-aws.k8s.io/api/v1/nodes": dial tcp 3.25.119.95:443: i/o timeout.
I1016 06:47:52.220648    5162 instancegroups.go:516] Cluster did not validate, will retry in "30s": error listing nodes: Get "https://api.e2e-89d41a7532-e6156.test-cncf-aws.k8s.io/api/v1/nodes": dial tcp 3.25.119.95:443: i/o timeout.
I1016 06:48:52.258075    5162 instancegroups.go:516] Cluster did not validate, will retry in "30s": error listing nodes: Get "https://api.e2e-89d41a7532-e6156.test-cncf-aws.k8s.io/api/v1/nodes": dial tcp 3.25.119.95:443: i/o timeout.
I1016 06:49:52.306531    5162 instancegroups.go:516] Cluster did not validate, will retry in "30s": error listing nodes: Get "https://api.e2e-89d41a7532-e6156.test-cncf-aws.k8s.io/api/v1/nodes": dial tcp 3.25.119.95:443: i/o timeout.
I1016 06:50:52.346978    5162 instancegroups.go:516] Cluster did not validate, will retry in "30s": error listing nodes: Get "https://api.e2e-89d41a7532-e6156.test-cncf-aws.k8s.io/api/v1/nodes": dial tcp 3.25.119.95:443: i/o timeout.
I1016 06:51:52.399828    5162 instancegroups.go:516] Cluster did not validate, will retry in "30s": error listing nodes: Get "https://api.e2e-89d41a7532-e6156.test-cncf-aws.k8s.io/api/v1/nodes": dial tcp 3.25.119.95:443: i/o timeout.
I1016 06:52:52.432495    5162 instancegroups.go:516] Cluster did not validate, will retry in "30s": error listing nodes: Get "https://api.e2e-89d41a7532-e6156.test-cncf-aws.k8s.io/api/v1/nodes": dial tcp 3.25.119.95:443: i/o timeout.
I1016 06:53:52.473679    5162 instancegroups.go:516] Cluster did not validate, will retry in "30s": error listing nodes: Get "https://api.e2e-89d41a7532-e6156.test-cncf-aws.k8s.io/api/v1/nodes": dial tcp 3.25.119.95:443: i/o timeout.
I1016 06:54:52.507479    5162 instancegroups.go:516] Cluster did not validate, will retry in "30s": error listing nodes: Get "https://api.e2e-89d41a7532-e6156.test-cncf-aws.k8s.io/api/v1/nodes": dial tcp 3.25.119.95:443: i/o timeout.
I1016 06:55:52.552496    5162 instancegroups.go:516] Cluster did not validate, will retry in "30s": error listing nodes: Get "https://api.e2e-89d41a7532-e6156.test-cncf-aws.k8s.io/api/v1/nodes": dial tcp 3.25.119.95:443: i/o timeout.
I1016 06:56:52.593543    5162 instancegroups.go:516] Cluster did not validate, will retry in "30s": error listing nodes: Get "https://api.e2e-89d41a7532-e6156.test-cncf-aws.k8s.io/api/v1/nodes": dial tcp 3.25.119.95:443: i/o timeout.
I1016 06:57:52.630151    5162 instancegroups.go:516] Cluster did not validate, will retry in "30s": error listing nodes: Get "https://api.e2e-89d41a7532-e6156.test-cncf-aws.k8s.io/api/v1/nodes": dial tcp 3.25.119.95:443: i/o timeout.
I1016 06:58:52.660032    5162 instancegroups.go:516] Cluster did not validate, will retry in "30s": error listing nodes: Get "https://api.e2e-89d41a7532-e6156.test-cncf-aws.k8s.io/api/v1/nodes": dial tcp 3.25.119.95:443: i/o timeout.
I1016 06:59:52.712058    5162 instancegroups.go:516] Cluster did not validate, will retry in "30s": error listing nodes: Get "https://api.e2e-89d41a7532-e6156.test-cncf-aws.k8s.io/api/v1/nodes": dial tcp 3.25.119.95:443: i/o timeout.
I1016 07:00:52.745895    5162 instancegroups.go:516] Cluster did not validate, will retry in "30s": error listing nodes: Get "https://api.e2e-89d41a7532-e6156.test-cncf-aws.k8s.io/api/v1/nodes": dial tcp 3.25.119.95:443: i/o timeout.
I1016 07:01:52.775836    5162 instancegroups.go:516] Cluster did not validate, will retry in "30s": error listing nodes: Get "https://api.e2e-89d41a7532-e6156.test-cncf-aws.k8s.io/api/v1/nodes": dial tcp 3.25.119.95:443: i/o timeout.
I1016 07:02:52.823621    5162 instancegroups.go:516] Cluster did not validate, will retry in "30s": error listing nodes: Get "https://api.e2e-89d41a7532-e6156.test-cncf-aws.k8s.io/api/v1/nodes": dial tcp 3.25.119.95:443: i/o timeout.
I1016 07:03:52.862239    5162 instancegroups.go:516] Cluster did not validate, will retry in "30s": error listing nodes: Get "https://api.e2e-89d41a7532-e6156.test-cncf-aws.k8s.io/api/v1/nodes": dial tcp 3.25.119.95:443: i/o timeout.
I1016 07:04:52.903847    5162 instancegroups.go:516] Cluster did not validate, will retry in "30s": error listing nodes: Get "https://api.e2e-89d41a7532-e6156.test-cncf-aws.k8s.io/api/v1/nodes": dial tcp 3.25.119.95:443: i/o timeout.
I1016 07:05:52.950116    5162 instancegroups.go:516] Cluster did not validate, will retry in "30s": error listing nodes: Get "https://api.e2e-89d41a7532-e6156.test-cncf-aws.k8s.io/api/v1/nodes": dial tcp 3.25.119.95:443: i/o timeout.
I1016 07:06:52.985978    5162 instancegroups.go:513] Cluster did not validate within deadline: error listing nodes: Get "https://api.e2e-89d41a7532-e6156.test-cncf-aws.k8s.io/api/v1/nodes": dial tcp 3.25.119.95:443: i/o timeout.
E1016 07:06:52.986016    5162 instancegroups.go:475] Cluster did not validate within 30m0s
Error: master not healthy after update, stopping rolling-update: "error validating cluster after terminating instance: cluster did not validate within a duration of \"30m0s\""
+ kops-finish
+ kubetest2 kops -v=2 --cloud-provider=aws --cluster-name=e2e-89d41a7532-e6156.test-cncf-aws.k8s.io --kops-root=/home/prow/go/src/k8s.io/kops --admin-access= --env=KOPS_FEATURE_FLAGS=SpecOverrideFlag --kops-binary-path=/tmp/kops.Di7horyee --down
I1016 07:06:53.016103    5182 app.go:59] RunDir for this run: "/logs/artifacts/bd1d11ae-2e49-11ec-8a05-6acfde499713"
I1016 07:06:53.016297    5182 app.go:90] ID for this run: "bd1d11ae-2e49-11ec-8a05-6acfde499713"
I1016 07:06:53.016329    5182 dumplogs.go:40] /tmp/kops.Di7horyee toolbox dump --name e2e-89d41a7532-e6156.test-cncf-aws.k8s.io --dir /logs/artifacts --private-key /etc/aws-ssh/aws-ssh-private --ssh-user ubuntu
I1016 07:06:53.037101    5191 featureflag.go:166] FeatureFlag "SpecOverrideFlag"=true
I1016 07:06:53.037539    5191 featureflag.go:166] FeatureFlag "AlphaAllowGCE"=true
I1016 07:06:53.037547    5191 featureflag.go:166] FeatureFlag "SpecOverrideFlag"=true
W1016 07:07:35.779911    5191 toolbox_dump.go:168] error listing nodes in cluster: Get "https://api.e2e-89d41a7532-e6156.test-cncf-aws.k8s.io/api/v1/nodes": dial tcp 3.25.119.95:443: i/o timeout
2021/10/16 07:07:35 dumping node not registered in kubernetes: 13.239.136.220
2021/10/16 07:07:35 Dumping node 13.239.136.220
2021/10/16 07:07:45 dumping node not registered in kubernetes: 13.55.22.9
2021/10/16 07:07:45 Dumping node 13.55.22.9
2021/10/16 07:07:53 dumping node not registered in kubernetes: 13.236.67.225
2021/10/16 07:07:53 Dumping node 13.236.67.225
... skipping 1049 lines ...
I1016 07:08:18.844137    5182 dumplogs.go:72] /tmp/kops.Di7horyee get instancegroups --name e2e-89d41a7532-e6156.test-cncf-aws.k8s.io -o yaml
I1016 07:08:19.649052    5182 dumplogs.go:91] kubectl cluster-info dump --all-namespaces -o yaml --output-directory /logs/artifacts/cluster-info
I1016 07:08:49.702190    5182 dumplogs.go:114] /tmp/kops.Di7horyee toolbox dump --name e2e-89d41a7532-e6156.test-cncf-aws.k8s.io --private-key /etc/aws-ssh/aws-ssh-private --ssh-user ubuntu -o yaml
I1016 07:09:02.237834    5182 dumplogs.go:143] ssh -i /etc/aws-ssh/aws-ssh-private -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null ubuntu@13.239.136.220 -- kubectl cluster-info dump --all-namespaces -o yaml --output-directory /tmp/cluster-info
Warning: Permanently added '13.239.136.220' (ECDSA) to the list of known hosts.
The connection to the server 127.0.0.1 was refused - did you specify the right host or port?
W1016 07:09:04.619496    5182 down.go:34] Dumping cluster logs at the start of Down() failed: exit status 1
I1016 07:09:04.619606    5182 down.go:48] /tmp/kops.Di7horyee delete cluster --name e2e-89d41a7532-e6156.test-cncf-aws.k8s.io --yes
I1016 07:09:04.636801    5241 featureflag.go:166] FeatureFlag "SpecOverrideFlag"=true
I1016 07:09:04.636878    5241 featureflag.go:166] FeatureFlag "AlphaAllowGCE"=true
I1016 07:09:04.636883    5241 featureflag.go:166] FeatureFlag "SpecOverrideFlag"=true
TYPE			NAME													ID
autoscaling-config	master-ap-southeast-2a.masters.e2e-89d41a7532-e6156.test-cncf-aws.k8s.io				lt-0375416c4d8316b50
... skipping 410 lines ...