This job view page is being replaced by Spyglass soon. Check out the new job view.
ResultFAILURE
Tests 1 failed / 10 succeeded
Started2021-12-22 17:04
Elapsed1h34m
Revisionmaster
job-versionv1.24.0-alpha.1.216+c75d254beb662e
kubetest-version
revisionv1.24.0-alpha.1.216+c75d254beb662e

Test Failures


kubetest ClusterLoaderV2 16m20s

error during /home/prow/go/src/k8s.io/perf-tests/run-e2e.sh cluster-loader2 --experimental-gcp-snapshot-prometheus-disk=true --experimental-prometheus-disk-snapshot-name=ci-kubernetes-e2e-gce-scale-performance-1473700239201275904 --nodes=5000 --prometheus-scrape-node-exporter --provider=gce --report-dir=/logs/artifacts --testconfig=testing/load/config.yaml --testconfig=testing/access-tokens/config.yaml --testoverrides=./testing/experiments/enable_restart_count_check.yaml --testoverrides=./testing/experiments/ignore_known_gce_container_restarts.yaml --testoverrides=./testing/overrides/5000_nodes.yaml: exit status 1
				from junit_runner.xml

Filter through log files | View test history on testgrid


Show 10 Passed Tests

Error lines from build-log.txt

... skipping 224 lines ...
k8s-fw-a426df123669b438bba23894a9f91b0c
k8s-fw-aa8ed6ae3f46a43d088781b4565c68de
k8s-fw-aef654066089f4eacaa374e24715964f
k8s-fw-aefba1a25bc074bb691bf789bba9851d
k8s-fw-afc0152fc1b984828b78e10acb7fe254
Deleting custom subnet...
ERROR: (gcloud.compute.networks.subnets.delete) Could not fetch resource:
 - The subnetwork resource 'projects/k8s-infra-e2e-scale-5k-project/regions/us-east1/subnetworks/gce-scale-cluster-custom-subnet' is already being used by 'projects/k8s-infra-e2e-scale-5k-project/regions/us-east1/forwardingRules/a19a5758d149d4c1489ee7f47f7a0366'

ERROR: (gcloud.compute.networks.delete) Could not fetch resource:
 - The network resource 'projects/k8s-infra-e2e-scale-5k-project/global/networks/gce-scale-cluster' is already being used by 'projects/k8s-infra-e2e-scale-5k-project/global/firewalls/k8s-3ba1a35adff30235-node-http-hc'

Failed to delete network 'gce-scale-cluster'. Listing firewall-rules:
NAME                                          NETWORK            DIRECTION  PRIORITY  ALLOW      DENY  DISABLED
k8s-3ba1a35adff30235-node-http-hc             gce-scale-cluster  INGRESS    1000      tcp:10256        False
k8s-4431a7fffae18eb8-node-http-hc             gce-scale-cluster  INGRESS    1000      tcp:10256        False
k8s-7f9b0a4b6de355e2-node-http-hc             gce-scale-cluster  INGRESS    1000      tcp:10256        False
k8s-8fc79f930c33c662-node-hc                  gce-scale-cluster  INGRESS    1000      tcp:10256        False
k8s-8fc79f930c33c662-node-http-hc             gce-scale-cluster  INGRESS    1000      tcp:10256        False
... skipping 393 lines ...
Looking for address 'gce-scale-cluster-master-ip'
Looking for address 'gce-scale-cluster-master-internal-ip'
Using master: gce-scale-cluster-master (external IP: 34.139.33.86; internal IP: 10.40.0.2)
Waiting up to 300 seconds for cluster initialization.

  This will continually check to see if the API for kubernetes is reachable.
  This may time out if there was some uncaught error during start up.

Kubernetes cluster created.
Cluster "k8s-infra-e2e-scale-5k-project_gce-scale-cluster" set.
User "k8s-infra-e2e-scale-5k-project_gce-scale-cluster" set.
Context "k8s-infra-e2e-scale-5k-project_gce-scale-cluster" created.
Switched to context "k8s-infra-e2e-scale-5k-project_gce-scale-cluster".
... skipping 10032 lines ...
gce-scale-cluster-minion-group-zz5n     Ready                         <none>   8m25s   v1.24.0-alpha.1.216+c75d254beb662e
gce-scale-cluster-minion-group-zzmd     Ready                         <none>   6m38s   v1.24.0-alpha.1.216+c75d254beb662e
gce-scale-cluster-minion-heapster       Ready                         <none>   11m     v1.24.0-alpha.1.216+c75d254beb662e
Warning: v1 ComponentStatus is deprecated in v1.19+
Validate output:
Warning: v1 ComponentStatus is deprecated in v1.19+
NAME                 STATUS    MESSAGE                         ERROR
etcd-1               Healthy   {"health":"true","reason":""}   
etcd-0               Healthy   {"health":"true","reason":""}   
controller-manager   Healthy   ok                              
scheduler            Healthy   ok                              
Cluster validation encountered some problems, but cluster should be in working order
...ignoring non-fatal errors in validate-cluster
Done, listing cluster services:

Kubernetes control plane is running at https://34.139.33.86
GLBCDefaultBackend is running at https://34.139.33.86/api/v1/namespaces/kube-system/services/default-http-backend:http/proxy
CoreDNS is running at https://34.139.33.86/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
Metrics-server is running at https://34.139.33.86/api/v1/namespaces/kube-system/services/https:metrics-server:/proxy
... skipping 5237 lines ...
I1222 17:23:58.300628   15487 framework.go:274] Applying /home/prow/go/src/k8s.io/perf-tests/clusterloader2/pkg/prometheus/manifests/default/prometheus-serviceMonitorLegacyKubeDNS.yaml
I1222 17:23:58.339261   15487 prometheus.go:304] Exposing kube-apiserver metrics in the cluster
I1222 17:23:58.493114   15487 framework.go:274] Applying /home/prow/go/src/k8s.io/perf-tests/clusterloader2/pkg/prometheus/manifests/master-ip/master-endpoints.yaml
I1222 17:23:58.531758   15487 framework.go:274] Applying /home/prow/go/src/k8s.io/perf-tests/clusterloader2/pkg/prometheus/manifests/master-ip/master-service.yaml
I1222 17:23:58.571514   15487 framework.go:274] Applying /home/prow/go/src/k8s.io/perf-tests/clusterloader2/pkg/prometheus/manifests/master-ip/master-serviceMonitor.yaml
I1222 17:23:58.610881   15487 prometheus.go:383] Waiting for Prometheus stack to become healthy...
W1222 17:24:28.648291   15487 util.go:72] error while calling prometheus api: the server is currently unable to handle the request (get services http:prometheus-k8s:9090), response: "k8s\x00\n\f\n\x02v1\x12\x06Status\x12]\n\x06\n\x00\x12\x00\x1a\x00\x12\aFailure\x1a3no endpoints available for service \"prometheus-k8s\"\"\x12ServiceUnavailable0\xf7\x03\x1a\x00\"\x00"
I1222 17:24:59.321569   15487 util.go:101] 578/1319 targets are ready, example not ready target: {map[endpoint:node-exporter instance:10.40.0.3:9100 job:master namespace:monitoring service:master] down}
I1222 17:25:29.339343   15487 util.go:101] 1131/1319 targets are ready, example not ready target: {map[endpoint:http-metrics instance:10.40.11.195:10249 job:kube-proxy namespace:kube-system pod:kube-proxy-gce-scale-cluster-minion-group-2-1cvj service:kube-proxy] unknown}
I1222 17:25:59.337723   15487 util.go:101] 1318/1319 targets are ready, example not ready target: {map[container:coredns endpoint:metrics instance:10.69.125.2:9153 job:kube-dns namespace:kube-system pod:coredns-6fb999fd84-ft77k service:kube-dns] down}
I1222 17:26:29.225668   15487 util.go:101] 1318/1319 targets are ready, example not ready target: {map[container:coredns endpoint:metrics instance:10.69.125.2:9153 job:kube-dns namespace:kube-system pod:coredns-6fb999fd84-ft77k service:kube-dns] down}
I1222 17:26:59.213610   15487 util.go:101] 1318/1319 targets are ready, example not ready target: {map[container:coredns endpoint:metrics instance:10.69.125.2:9153 job:kube-dns namespace:kube-system pod:coredns-6fb999fd84-ft77k service:kube-dns] down}
I1222 17:27:29.235635   15487 util.go:101] 1318/1319 targets are ready, example not ready target: {map[container:coredns endpoint:metrics instance:10.69.125.2:9153 job:kube-dns namespace:kube-system pod:coredns-6fb999fd84-ft77k service:kube-dns] down}
... skipping 126 lines ...
      "eventTime": null,
      "reportingComponent": "",
      "reportingInstance": ""
    }
  ]
}
F1222 17:38:59.889513   15487 clusterloader.go:297] Error while setting up prometheus stack: timed out waiting for the condition
2021/12/22 17:38:59 process.go:155: Step '/home/prow/go/src/k8s.io/perf-tests/run-e2e.sh cluster-loader2 --experimental-gcp-snapshot-prometheus-disk=true --experimental-prometheus-disk-snapshot-name=ci-kubernetes-e2e-gce-scale-performance-1473700239201275904 --nodes=5000 --prometheus-scrape-node-exporter --provider=gce --report-dir=/logs/artifacts --testconfig=testing/load/config.yaml --testconfig=testing/access-tokens/config.yaml --testoverrides=./testing/experiments/enable_restart_count_check.yaml --testoverrides=./testing/experiments/ignore_known_gce_container_restarts.yaml --testoverrides=./testing/overrides/5000_nodes.yaml' finished in 16m20.750025777s
2021/12/22 17:38:59 e2e.go:562: Dumping logs from nodes to GCS directly at path: gs://k8s-infra-scalability-tests-logs/ci-kubernetes-e2e-gce-scale-performance/1473700239201275904
2021/12/22 17:38:59 process.go:153: Running: /workspace/log-dump.sh /logs/artifacts gs://k8s-infra-scalability-tests-logs/ci-kubernetes-e2e-gce-scale-performance/1473700239201275904
Checking for custom logdump instances, if any
Using gce provider, skipping check for LOG_DUMP_SSH_KEY and LOG_DUMP_SSH_USER
Project: k8s-infra-e2e-scale-5k-project
... skipping 11 lines ...
Specify --start=104017 in the next get-serial-port-output invocation to get only the new output starting from here.
scp: /var/log/cluster-autoscaler.log*: No such file or directory
scp: /var/log/fluentd.log*: No such file or directory
scp: /var/log/kubelet.cov*: No such file or directory
scp: /var/log/cl2-**: No such file or directory
scp: /var/log/startupscript.log*: No such file or directory
ERROR: (gcloud.compute.scp) [/usr/bin/scp] exited with return code [1].
Dumping logs from nodes to GCS directly at 'gs://k8s-infra-scalability-tests-logs/ci-kubernetes-e2e-gce-scale-performance/1473700239201275904' using logexporter
Detecting nodes in the cluster
namespace/logexporter created
secret/google-service-account created
daemonset.apps/logexporter created
Listing marker files (gs://k8s-infra-scalability-tests-logs/ci-kubernetes-e2e-gce-scale-performance/1473700239201275904/logexported-nodes-registry) for successful nodes...
... skipping 10064 lines ...
namespace "logexporter" deleted
Dumping logs through SSH for the following nodes:
gce-scale-cluster-minion-group-43r3
Dumping logs for nodes provided as args to dump_nodes() function
Changing logfiles to be world-readable for download
External IP address was not found; defaulting to using IAP tunneling.
ERROR: (gcloud.compute.start-iap-tunnel) Error while connecting [4033: 'not authorized'].
ERROR: (gcloud.compute.ssh) [/usr/bin/ssh] exited with return code [255].
External IP address was not found; defaulting to using IAP tunneling.
ERROR: (gcloud.compute.start-iap-tunnel) Error while connecting [4033: 'not authorized'].
ERROR: (gcloud.compute.ssh) [/usr/bin/ssh] exited with return code [255].
External IP address was not found; defaulting to using IAP tunneling.
ERROR: (gcloud.compute.start-iap-tunnel) Error while connecting [4033: 'not authorized'].
ERROR: (gcloud.compute.ssh) [/usr/bin/ssh] exited with return code [255].
External IP address was not found; defaulting to using IAP tunneling.
ERROR: (gcloud.compute.start-iap-tunnel) Error while connecting [4033: 'not authorized'].
ERROR: (gcloud.compute.ssh) [/usr/bin/ssh] exited with return code [255].
External IP address was not found; defaulting to using IAP tunneling.
ERROR: (gcloud.compute.start-iap-tunnel) Error while connecting [4033: 'not authorized'].
ERROR: (gcloud.compute.ssh) [/usr/bin/ssh] exited with return code [255].
External IP address was not found; defaulting to using IAP tunneling.
ERROR: (gcloud.compute.start-iap-tunnel) Error while connecting [4033: 'not authorized'].
ERROR: (gcloud.compute.ssh) [/usr/bin/ssh] exited with return code [255].
Copying 'kube-proxy.log containers/konnectivity-agent-*.log fluentd.log node-problem-detector.log kubelet.cov cl2-* startupscript.log kern.log docker/log kubelet.log supervisor/supervisord.log supervisor/kubelet-stdout.log supervisor/kubelet-stderr.log supervisor/docker-stdout.log supervisor/docker-stderr.log' from gce-scale-cluster-minion-group-43r3

Specify --start=104283 in the next get-serial-port-output invocation to get only the new output starting from here.
External IP address was not found; defaulting to using IAP tunneling.
ERROR: (gcloud.compute.start-iap-tunnel) Error while connecting [4033: 'not authorized'].
ssh_exchange_identification: Connection closed by remote host
ERROR: (gcloud.compute.scp) [/usr/bin/scp] exited with return code [1].
Uploading '/tmp/tmp.jsFGQksSpf/logs' to 'gs://k8s-infra-scalability-tests-logs/ci-kubernetes-e2e-gce-scale-performance/1473700239201275904'
Copying file:///tmp/tmp.jsFGQksSpf/1473700239201275904/gce-scale-cluster-minion-group-3-2d83/logexporter-b42lj.log [Content-Type=application/octet-stream]...
Copying file:///tmp/tmp.jsFGQksSpf/1473700239201275904/gce-scale-cluster-minion-group-3-z088/logexporter-8czct.log [Content-Type=application/octet-stream]...
Copying file:///tmp/tmp.jsFGQksSpf/1473700239201275904/gce-scale-cluster-minion-group-1-5hpd/logexporter-v8x9g.log [Content-Type=application/octet-stream]...
Copying file:///tmp/tmp.jsFGQksSpf/1473700239201275904/gce-scale-cluster-minion-group-3-qgw4/logexporter-shdsk.log [Content-Type=application/octet-stream]...
Copying file:///tmp/tmp.jsFGQksSpf/1473700239201275904/gce-scale-cluster-minion-group-1-h95l/logexporter-48v8d.log [Content-Type=application/octet-stream]...
... skipping 5104 lines ...
k8s-fw-a426df123669b438bba23894a9f91b0c
k8s-fw-aa8ed6ae3f46a43d088781b4565c68de
k8s-fw-aef654066089f4eacaa374e24715964f
k8s-fw-aefba1a25bc074bb691bf789bba9851d
k8s-fw-afc0152fc1b984828b78e10acb7fe254
Deleting custom subnet...
ERROR: (gcloud.compute.networks.subnets.delete) Could not fetch resource:
 - The subnetwork resource 'projects/k8s-infra-e2e-scale-5k-project/regions/us-east1/subnetworks/gce-scale-cluster-custom-subnet' is already being used by 'projects/k8s-infra-e2e-scale-5k-project/regions/us-east1/forwardingRules/a19a5758d149d4c1489ee7f47f7a0366'

ERROR: (gcloud.compute.networks.delete) Could not fetch resource:
 - The network resource 'projects/k8s-infra-e2e-scale-5k-project/global/networks/gce-scale-cluster' is already being used by 'projects/k8s-infra-e2e-scale-5k-project/global/firewalls/k8s-7f9b0a4b6de355e2-node-http-hc'

Failed to delete network 'gce-scale-cluster'. Listing firewall-rules:
NAME                                          NETWORK            DIRECTION  PRIORITY  ALLOW      DENY  DISABLED
k8s-3ba1a35adff30235-node-http-hc             gce-scale-cluster  INGRESS    1000      tcp:10256        False
k8s-4431a7fffae18eb8-node-http-hc             gce-scale-cluster  INGRESS    1000      tcp:10256        False
k8s-7f9b0a4b6de355e2-node-http-hc             gce-scale-cluster  INGRESS    1000      tcp:10256        False
k8s-8fc79f930c33c662-node-hc                  gce-scale-cluster  INGRESS    1000      tcp:10256        False
k8s-8fc79f930c33c662-node-http-hc             gce-scale-cluster  INGRESS    1000      tcp:10256        False
... skipping 16 lines ...
Property "users.k8s-infra-e2e-scale-5k-project_gce-scale-cluster-basic-auth" unset.
Property "contexts.k8s-infra-e2e-scale-5k-project_gce-scale-cluster" unset.
Cleared config for k8s-infra-e2e-scale-5k-project_gce-scale-cluster from /workspace/.kube/config
Done
2021/12/22 18:39:29 process.go:155: Step './hack/e2e-internal/e2e-down.sh' finished in 18m16.552954361s
2021/12/22 18:39:29 process.go:96: Saved XML output to /logs/artifacts/junit_runner.xml.
2021/12/22 18:39:29 main.go:331: Something went wrong: encountered 1 errors: [error during /home/prow/go/src/k8s.io/perf-tests/run-e2e.sh cluster-loader2 --experimental-gcp-snapshot-prometheus-disk=true --experimental-prometheus-disk-snapshot-name=ci-kubernetes-e2e-gce-scale-performance-1473700239201275904 --nodes=5000 --prometheus-scrape-node-exporter --provider=gce --report-dir=/logs/artifacts --testconfig=testing/load/config.yaml --testconfig=testing/access-tokens/config.yaml --testoverrides=./testing/experiments/enable_restart_count_check.yaml --testoverrides=./testing/experiments/ignore_known_gce_container_restarts.yaml --testoverrides=./testing/overrides/5000_nodes.yaml: exit status 1]
Traceback (most recent call last):
  File "/workspace/scenarios/kubernetes_e2e.py", line 723, in <module>
    main(parse_args())
  File "/workspace/scenarios/kubernetes_e2e.py", line 569, in main
    mode.start(runner_args)
  File "/workspace/scenarios/kubernetes_e2e.py", line 228, in start
... skipping 9 lines ...