ci-kubernetes-e2e-gce-gpu-master-stable1-cluster-downgrade #1192940014003032066
Recent runs || View in Spyglass
Recent runs || View in Spyglass
| Result | FAILURE |
| Tests | 1 failed / 6 succeeded |
| Started | |
| Elapsed | 19m0s |
| Revision | |
| Builder | gke-prow-ssd-pool-1a225945-ldhb |
| links | {u'resultstore': {u'url': u'https://source.cloud.google.com/results/invocations/02150240-cbea-4ffa-8016-1329a53881bf/targets/test'}} |
| pod | 7ed1b3a0-027b-11ea-a7ca-12348a448b6a |
| resultstore | https://source.cloud.google.com/results/invocations/02150240-cbea-4ffa-8016-1329a53881bf/targets/test |
| infra-commit | b4a6cf72f |
| job-version | v1.18.0-alpha.0.555+6569bc736fac76 |
| pod | 7ed1b3a0-027b-11ea-a7ca-12348a448b6a |
| revision | v1.18.0-alpha.0.555+6569bc736fac76 |
error during ./hack/e2e-internal/e2e-up.sh: exit status 2
from junit_runner.xml
Filter through log files | View test history on testgrid
Deferred TearDown
DumpClusterLogs (--up failed)
Extract
TearDown Previous
Timeout
listResources Before
... skipping 15 lines ... I1108 23:00:59.690] process 45 exited with code 0 after 0.0m I1108 23:00:59.690] Will upload results to gs://kubernetes-jenkins/logs using pr-kubekins@kubernetes-jenkins-pull.iam.gserviceaccount.com I1108 23:00:59.690] Root: /workspace I1108 23:00:59.690] cd to /workspace I1108 23:00:59.690] Configure environment... I1108 23:00:59.691] Call: git show -s --format=format:%ct HEAD W1108 23:00:59.694] fatal: not a git repository (or any of the parent directories): .git I1108 23:00:59.695] process 57 exited with code 128 after 0.0m W1108 23:00:59.695] Unable to print commit date for HEAD I1108 23:00:59.695] Call: gcloud auth activate-service-account --key-file=/etc/service-account/service-account.json W1108 23:01:00.246] Activated service account credentials for: [pr-kubekins@kubernetes-jenkins-pull.iam.gserviceaccount.com] I1108 23:01:00.614] process 58 exited with code 0 after 0.0m I1108 23:01:00.614] Call: gcloud config get-value account ... skipping 390 lines ... W1108 23:05:40.419] Trying to find master named 'bootstrap-e2e-master' W1108 23:05:40.419] Looking for address 'bootstrap-e2e-master-ip' W1108 23:05:41.221] Using master: bootstrap-e2e-master (external IP: 35.233.255.42; internal IP: (not set)) I1108 23:05:41.321] Waiting up to 300 seconds for cluster initialization. I1108 23:05:41.322] I1108 23:05:41.322] This will continually check to see if the API for kubernetes is reachable. I1108 23:05:41.322] This may time out if there was some uncaught error during start up. I1108 23:05:41.322] I1108 23:10:45.763] .........................................................................................................Checking for custom logdump instances, if any I1108 23:10:45.768] Sourcing kube-util.sh I1108 23:10:45.817] Detecting project I1108 23:10:45.817] Project: k8s-gke-gpu-boskos-04 I1108 23:10:45.818] Network Project: k8s-gke-gpu-boskos-04 I1108 23:10:45.818] Zone: us-west1-b I1108 23:10:45.818] Dumping logs from master locally to '/workspace/_artifacts' W1108 23:10:45.919] [0;31mCluster failed to initialize within 300 seconds.[0m W1108 23:10:45.919] Last output from querying API server follows: W1108 23:10:45.920] ----------------------------------------------------- W1108 23:10:45.920] % Total % Received % Xferd Average Speed Time Time Time Current W1108 23:10:45.920] Dload Upload Total Spent Left Speed W1108 23:10:45.921] 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:02 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:03 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:04 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:05 --:--:-- 0 W1108 23:10:45.921] curl: (28) Operation timed out after 5001 milliseconds with 0 out of 0 bytes received ... skipping 11 lines ... W1108 23:11:23.555] scp: /var/log/glbc.log*: No such file or directory W1108 23:11:23.555] scp: /var/log/cluster-autoscaler.log*: No such file or directory W1108 23:11:23.555] scp: /var/log/kube-addon-manager.log*: No such file or directory W1108 23:11:23.556] scp: /var/log/fluentd.log*: No such file or directory W1108 23:11:23.556] scp: /var/log/kubelet.cov*: No such file or directory W1108 23:11:23.556] scp: /var/log/startupscript.log*: No such file or directory W1108 23:11:23.558] ERROR: (gcloud.compute.scp) [/usr/bin/scp] exited with return code [1]. I1108 23:11:23.659] Dumping logs from nodes locally to '/workspace/_artifacts' I1108 23:11:23.659] Detecting nodes in the cluster I1108 23:12:00.258] Changing logfiles to be world-readable for download I1108 23:12:00.870] Changing logfiles to be world-readable for download I1108 23:12:01.155] Changing logfiles to be world-readable for download I1108 23:12:03.437] Copying 'kube-proxy.log fluentd.log node-problem-detector.log kubelet.cov startupscript.log' from bootstrap-e2e-minion-group-c0mz ... skipping 6 lines ... W1108 23:12:05.427] W1108 23:12:05.427] Specify --start=40320 in the next get-serial-port-output invocation to get only the new output starting from here. W1108 23:12:06.253] scp: /var/log/fluentd.log*: No such file or directory W1108 23:12:06.253] scp: /var/log/node-problem-detector.log*: No such file or directory W1108 23:12:06.253] scp: /var/log/kubelet.cov*: No such file or directory W1108 23:12:06.254] scp: /var/log/startupscript.log*: No such file or directory W1108 23:12:06.256] ERROR: (gcloud.compute.scp) [/usr/bin/scp] exited with return code [1]. W1108 23:12:06.840] scp: /var/log/fluentd.log*: No such file or directory W1108 23:12:06.840] scp: /var/log/node-problem-detector.log*: No such file or directory W1108 23:12:06.841] scp: /var/log/kubelet.cov*: No such file or directory W1108 23:12:06.841] scp: /var/log/startupscript.log*: No such file or directory W1108 23:12:06.844] ERROR: (gcloud.compute.scp) [/usr/bin/scp] exited with return code [1]. W1108 23:12:07.114] scp: /var/log/fluentd.log*: No such file or directory W1108 23:12:07.114] scp: /var/log/node-problem-detector.log*: No such file or directory W1108 23:12:07.114] scp: /var/log/kubelet.cov*: No such file or directory W1108 23:12:07.115] scp: /var/log/startupscript.log*: No such file or directory W1108 23:12:07.118] ERROR: (gcloud.compute.scp) [/usr/bin/scp] exited with return code [1]. W1108 23:12:10.241] INSTANCE_GROUPS=bootstrap-e2e-minion-group W1108 23:12:10.241] NODE_NAMES=bootstrap-e2e-minion-group-1zkt bootstrap-e2e-minion-group-c0mz bootstrap-e2e-minion-group-s118 I1108 23:12:11.192] Failures for bootstrap-e2e-minion-group (if any): W1108 23:12:12.176] 2019/11/08 23:12:12 process.go:155: Step './cluster/log-dump/log-dump.sh /workspace/_artifacts' finished in 1m26.416270874s W1108 23:12:12.177] 2019/11/08 23:12:12 process.go:153: Running: ./hack/e2e-internal/e2e-down.sh W1108 23:12:12.228] Project: k8s-gke-gpu-boskos-04 ... skipping 13 lines ... W1108 23:12:19.888] Deleting Managed Instance Group... W1108 23:14:17.468] ..........................Deleted [https://www.googleapis.com/compute/v1/projects/k8s-gke-gpu-boskos-04/zones/us-west1-b/instanceGroupManagers/bootstrap-e2e-minion-group]. W1108 23:14:17.468] done. W1108 23:14:21.953] Deleted [https://www.googleapis.com/compute/v1/projects/k8s-gke-gpu-boskos-04/global/instanceTemplates/bootstrap-e2e-minion-template]. W1108 23:14:22.796] Deleted [https://www.googleapis.com/compute/v1/projects/k8s-gke-gpu-boskos-04/global/instanceTemplates/bootstrap-e2e-windows-node-template]. I1108 23:15:25.598] Removing etcd replica, name: bootstrap-e2e-master, port: 2379, result: 1 W1108 23:15:25.699] Failed to execute 'curl -s --cacert /etc/srv/kubernetes/pki/etcd-apiserver-ca.crt --cert /etc/srv/kubernetes/pki/etcd-apiserver-client.crt --key /etc/srv/kubernetes/pki/etcd-apiserver-client.key https://127.0.0.1:2379/v2/members/$(curl -s --cacert /etc/srv/kubernetes/pki/etcd-apiserver-ca.crt --cert /etc/srv/kubernetes/pki/etcd-apiserver-client.crt --key /etc/srv/kubernetes/pki/etcd-apiserver-client.key https://127.0.0.1:2379/v2/members -XGET | sed 's/{\"id/\n/g' | grep bootstrap-e2e-master\" | cut -f 3 -d \") -XDELETE -L 2>/dev/null' on bootstrap-e2e-master despite 5 attempts W1108 23:15:25.701] Last attempt failed with: I1108 23:15:27.620] Successfully executed 'curl -s http://127.0.0.1:4002/v2/members/$(curl -s http://127.0.0.1:4002/v2/members -XGET | sed 's/{\"id/\n/g' | grep bootstrap-e2e-master\" | cut -f 3 -d \") -XDELETE -L 2>/dev/null' on bootstrap-e2e-master I1108 23:15:27.621] Removing etcd replica, name: bootstrap-e2e-master, port: 4002, result: 0 W1108 23:15:33.375] Updated [https://www.googleapis.com/compute/v1/projects/k8s-gke-gpu-boskos-04/zones/us-west1-b/instances/bootstrap-e2e-master]. W1108 23:18:18.880] Deleted [https://www.googleapis.com/compute/v1/projects/k8s-gke-gpu-boskos-04/zones/us-west1-b/instances/bootstrap-e2e-master]. W1108 23:18:29.192] Deleted [https://www.googleapis.com/compute/v1/projects/k8s-gke-gpu-boskos-04/global/firewalls/bootstrap-e2e-master-https]. W1108 23:18:31.379] Deleted [https://www.googleapis.com/compute/v1/projects/k8s-gke-gpu-boskos-04/global/firewalls/bootstrap-e2e-minion-all]. ... skipping 18 lines ... I1108 23:19:47.700] Cleared config for k8s-gke-gpu-boskos-04_bootstrap-e2e from /workspace/.kube/config I1108 23:19:47.701] Done W1108 23:19:47.801] W1108 23:19:47.696587 10051 loader.go:223] Config not found: /workspace/.kube/config W1108 23:19:47.801] W1108 23:19:47.696781 10051 loader.go:223] Config not found: /workspace/.kube/config W1108 23:19:47.801] 2019/11/08 23:19:47 process.go:155: Step './hack/e2e-internal/e2e-down.sh' finished in 7m35.524990725s W1108 23:19:47.802] 2019/11/08 23:19:47 process.go:96: Saved XML output to /workspace/_artifacts/junit_runner.xml. W1108 23:19:51.582] 2019/11/08 23:19:51 main.go:319: Something went wrong: starting e2e cluster: error during ./hack/e2e-internal/e2e-up.sh: exit status 2 W1108 23:19:51.586] Traceback (most recent call last): W1108 23:19:51.586] File "/workspace/./test-infra/jenkins/../scenarios/kubernetes_e2e.py", line 778, in <module> W1108 23:19:51.586] main(parse_args()) W1108 23:19:51.586] File "/workspace/./test-infra/jenkins/../scenarios/kubernetes_e2e.py", line 626, in main W1108 23:19:51.586] mode.start(runner_args) W1108 23:19:51.587] File "/workspace/./test-infra/jenkins/../scenarios/kubernetes_e2e.py", line 262, in start W1108 23:19:51.587] check_env(env, self.command, *args) W1108 23:19:51.587] File "/workspace/./test-infra/jenkins/../scenarios/kubernetes_e2e.py", line 111, in check_env W1108 23:19:51.587] subprocess.check_call(cmd, env=env) W1108 23:19:51.587] File "/usr/lib/python2.7/subprocess.py", line 190, in check_call W1108 23:19:51.587] raise CalledProcessError(retcode, cmd) W1108 23:19:51.588] subprocess.CalledProcessError: Command '('kubetest', '--dump=/workspace/_artifacts', '--gcp-service-account=/etc/service-account/service-account.json', '--up', '--down', '--test', '--provider=gce', '--cluster=bootstrap-e2e', '--gcp-network=bootstrap-e2e', '--check-leaked-resources', '--check-version-skew=false', '--extract=ci/k8s-stable1', '--extract=ci/latest', '--gcp-node-image=gci', '--gcp-project-type=gpu-project', '--gcp-zone=us-west1-b', '--test_args=--kubectl-path=../../../../kubernetes_skew/cluster/kubectl.sh --minStartupPods=8 --ginkgo.skip=\\[.+\\]|Initializers|Dashboard', '--timeout=150m', '--upgrade_args=--ginkgo.focus=\\[Feature:GPUClusterDowngrade\\] --upgrade-target=ci/k8s-stable1 --upgrade-image=gci')' returned non-zero exit status 1 E1108 23:19:51.594] Command failed I1108 23:19:51.594] process 262 exited with code 1 after 18.8m E1108 23:19:51.594] FAIL: ci-kubernetes-e2e-gce-gpu-master-stable1-cluster-downgrade I1108 23:19:51.594] Call: gcloud auth activate-service-account --key-file=/etc/service-account/service-account.json W1108 23:19:52.034] Activated service account credentials for: [pr-kubekins@kubernetes-jenkins-pull.iam.gserviceaccount.com] I1108 23:19:52.076] process 10061 exited with code 0 after 0.0m I1108 23:19:52.076] Call: gcloud config get-value account I1108 23:19:52.351] process 10073 exited with code 0 after 0.0m I1108 23:19:52.352] Will upload results to gs://kubernetes-jenkins/logs using pr-kubekins@kubernetes-jenkins-pull.iam.gserviceaccount.com I1108 23:19:52.352] Upload result and artifacts... I1108 23:19:52.352] Gubernator results at https://gubernator.k8s.io/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-gpu-master-stable1-cluster-downgrade/1192940014003032066 I1108 23:19:52.353] Call: gsutil ls gs://kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-gpu-master-stable1-cluster-downgrade/1192940014003032066/artifacts W1108 23:19:53.093] CommandException: One or more URLs matched no objects. E1108 23:19:53.185] Command failed I1108 23:19:53.186] process 10085 exited with code 1 after 0.0m W1108 23:19:53.186] Remote dir gs://kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-gpu-master-stable1-cluster-downgrade/1192940014003032066/artifacts not exist yet I1108 23:19:53.186] Call: gsutil -m -q -o GSUtil:use_magicfile=True cp -r -c -z log,txt,xml /workspace/_artifacts gs://kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-gpu-master-stable1-cluster-downgrade/1192940014003032066/artifacts I1108 23:19:55.561] process 10229 exited with code 0 after 0.0m I1108 23:19:55.562] Call: git rev-parse HEAD W1108 23:19:55.566] fatal: not a git repository (or any of the parent directories): .git E1108 23:19:55.566] Command failed I1108 23:19:55.566] process 10875 exited with code 128 after 0.0m I1108 23:19:55.567] Call: git rev-parse HEAD I1108 23:19:55.570] process 10876 exited with code 0 after 0.0m I1108 23:19:55.571] Call: gsutil stat gs://kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-gpu-master-stable1-cluster-downgrade/jobResultsCache.json I1108 23:19:56.395] process 10877 exited with code 0 after 0.0m I1108 23:19:56.396] Call: gsutil -q cat 'gs://kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-gpu-master-stable1-cluster-downgrade/jobResultsCache.json#1573211975434277' ... skipping 8 lines ...