ci-kubernetes-e2e-gce-scale-performance #1652357519764361216
Recent runs || View in Spyglass
| Result | FAILURE |
| Tests | 6 failed / 91 succeeded |
| Started | |
| Elapsed | 5h0m |
| Revision | master |
| job-version | v1.28.0-alpha.0.534+d94c733ee2bfae |
| kubetest-version | v20230406-23cb1879e3 |
| masterInstanceIDs | 5326202703685152834 |
| revision | v1.28.0-alpha.0.534+d94c733ee2bfae |
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=ClusterLoaderV2\sload\soverall\s\(testing\/load\/config\.yaml\)$'
:0
[measurement call WaitForControlledPodsRunning - WaitForRunningDaemonSets error: 1 objects timed out: DaemonSets: test-u6ku0u-1/daemonset-0
measurement call WaitForControlledPodsRunning - WaitForRunningJobs error: 1 objects timed out: Jobs: test-u6ku0u-22/big-job-0
measurement call WaitForControlledPodsRunning - WaitForRunningDeployments error: 28 objects timed out: Deployments: test-u6ku0u-46/medium-deployment-21, test-u6ku0u-48/medium-deployment-1, test-u6ku0u-40/small-deployment-64, test-u6ku0u-27/big-deployment-1, test-u6ku0u-27/small-deployment-178, test-u6ku0u-42/big-deployment-1, test-u6ku0u-41/small-deployment-261, test-u6ku0u-12/big-deployment-1, test-u6ku0u-7/medium-deployment-7, test-u6ku0u-5/small-deployment-184, test-u6ku0u-13/small-deployment-48, test-u6ku0u-3/small-deployment-22, test-u6ku0u-11/small-deployment-159, test-u6ku0u-44/small-deployment-83, test-u6ku0u-21/small-deployment-75, test-u6ku0u-24/small-deployment-282, test-u6ku0u-20/small-deployment-157, test-u6ku0u-37/big-deployment-0, test-u6ku0u-5/small-deployment-57, test-u6ku0u-30/medium-deployment-10, test-u6ku0u-24/medium-deployment-9, test-u6ku0u-25/big-deployment-1, test-u6ku0u-18/medium-deployment-3, test-u6ku0u-14/big-deployment-1, test-u6ku0u-33/medium-deployment-14, test-u6ku0u-2/medium-deployment-8, test-u6ku0u-18/medium-deployment-2, test-u6ku0u-30/small-deployment-122
measurement call WaitForControlledPodsRunning - WaitForRunningDaemonSets error: 1 objects timed out: DaemonSets: test-u6ku0u-1/daemonset-0
measurement call WaitForControlledPodsRunning - WaitForRunningStatefulSets error: 1 objects timed out: StatefulSets: test-u6ku0u-14/small-statefulset-0
measurement call WaitForControlledPodsRunning - WaitForRunningJobs error: 9 objects timed out: Jobs: test-u6ku0u-49/big-job-0, test-u6ku0u-25/medium-job-0, test-u6ku0u-32/big-job-0, test-u6ku0u-13/big-job-0, test-u6ku0u-20/big-job-0, test-u6ku0u-25/big-job-0, test-u6ku0u-1/big-job-0, test-u6ku0u-29/big-job-0, test-u6ku0u-22/big-job-0
measurement call WaitForControlledPodsRunning - WaitForRunningDeployments error: 77 objects timed out: Deployments: test-u6ku0u-42/big-deployment-1, test-u6ku0u-41/small-deployment-261, test-u6ku0u-13/small-deployment-72, test-u6ku0u-34/medium-deployment-17, test-u6ku0u-16/medium-deployment-13, test-u6ku0u-46/big-deployment-0, test-u6ku0u-38/small-deployment-37, test-u6ku0u-11/big-deployment-0, test-u6ku0u-29/small-deployment-274, test-u6ku0u-25/small-deployment-283, test-u6ku0u-25/medium-deployment-13, test-u6ku0u-12/big-deployment-1, test-u6ku0u-34/small-deployment-1, test-u6ku0u-13/small-deployment-34, test-u6ku0u-3/small-deployment-166, test-u6ku0u-22/big-deployment-1, test-u6ku0u-27/medium-deployment-7, test-u6ku0u-31/small-deployment-156, test-u6ku0u-3/medium-deployment-9, test-u6ku0u-40/medium-deployment-12, test-u6ku0u-34/medium-deployment-15, test-u6ku0u-3/big-deployment-0, test-u6ku0u-40/medium-deployment-14, test-u6ku0u-7/medium-deployment-7, test-u6ku0u-10/small-deployment-213, test-u6ku0u-7/medium-deployment-12, test-u6ku0u-5/small-deployment-184, test-u6ku0u-13/small-deployment-48, test-u6ku0u-10/big-deployment-0, test-u6ku0u-3/small-deployment-22, test-u6ku0u-11/small-deployment-159, test-u6ku0u-44/small-deployment-83, test-u6ku0u-21/small-deployment-75, test-u6ku0u-30/small-deployment-211, test-u6ku0u-24/small-deployment-282, test-u6ku0u-29/big-deployment-1, test-u6ku0u-32/big-deployment-1, test-u6ku0u-20/small-deployment-157, test-u6ku0u-23/medium-deployment-6, test-u6ku0u-14/small-deployment-146, test-u6ku0u-37/big-deployment-0, test-u6ku0u-25/small-deployment-34, test-u6ku0u-5/small-deployment-57, test-u6ku0u-30/medium-deployment-10, test-u6ku0u-8/small-deployment-132, test-u6ku0u-24/medium-deployment-9, test-u6ku0u-44/small-deployment-101, test-u6ku0u-10/small-deployment-253, test-u6ku0u-13/small-deployment-216, test-u6ku0u-5/big-deployment-0, test-u6ku0u-25/big-deployment-1, test-u6ku0u-18/medium-deployment-3, test-u6ku0u-44/small-deployment-10, test-u6ku0u-50/big-deployment-0, test-u6ku0u-14/big-deployment-1, test-u6ku0u-33/medium-deployment-14, test-u6ku0u-2/medium-deployment-8, test-u6ku0u-18/medium-deployment-2, test-u6ku0u-48/big-deployment-0, test-u6ku0u-30/small-deployment-122, test-u6ku0u-9/medium-deployment-20, test-u6ku0u-41/medium-deployment-3, test-u6ku0u-22/medium-deployment-13, test-u6ku0u-43/big-deployment-0, test-u6ku0u-17/big-deployment-1, test-u6ku0u-26/small-deployment-141, test-u6ku0u-30/small-deployment-69, test-u6ku0u-46/medium-deployment-21, test-u6ku0u-25/small-deployment-227, test-u6ku0u-46/small-deployment-216, test-u6ku0u-48/medium-deployment-1, test-u6ku0u-44/small-deployment-296, test-u6ku0u-40/small-deployment-64, test-u6ku0u-49/medium-deployment-13, test-u6ku0u-27/big-deployment-1, test-u6ku0u-27/small-deployment-178, test-u6ku0u-8/medium-deployment-12
measurement call WaitForBoundPVCs - WaitForPVCsToBeDeleted error: timeout while waiting for 0 PVCs to be running in namespace '' with labels 'group = load' and fields '' - only 1 found bound
measurement call TestMetrics - TestMetrics error: [action gather failed for SystemPodMetrics measurement: restart counts violation: RestartCount(metrics-server-v0.6.3-6d58b7896b-tv24s, metrics-server)=3, want <= 0; RestartCount(metrics-server-v0.6.3-6d58b7896b-xlrcl, metrics-server)=6, want <= 0
action gather failed for ClusterOOMsTracker measurement: OOMs recorded: [{Node:gce-scale-cluster-minion-group-3-xkq3 Process:metrics-server ProcessMemory:3659772kB ProcessID:2981 Time:2023-04-29 17:48:16 +0000 UTC} {Node:gce-scale-cluster-minion-group-3-xkq3 Process:metrics-server ProcessMemory:3659324kB ProcessID:6317 Time:2023-04-29 17:51:14 +0000 UTC} {Node:gce-scale-cluster-minion-group-gtws Process:metrics-server ProcessMemory:3436732kB ProcessID:11475 Time:2023-04-29 18:09:17 +0000 UTC} {Node:gce-scale-cluster-minion-group-gtws Process:metrics-server ProcessMemory:3455024kB ProcessID:11817 Time:2023-04-29 18:10:50 +0000 UTC} {Node:gce-scale-cluster-minion-group-gtws Process:metrics-server ProcessMemory:3443440kB ProcessID:12420 Time:2023-04-29 18:11:45 +0000 UTC} {Node:gce-scale-cluster-minion-group-gtws Process:metrics-server ProcessMemory:3590584kB ProcessID:12522 Time:2023-04-29 18:13:00 +0000 UTC} {Node:gce-scale-cluster-minion-group-gtws Process:metrics-server ProcessMemory:3520436kB ProcessID:12724 Time:2023-04-29 18:15:51 +0000 UTC} {Node:gce-scale-cluster-minion-group-gtws Process:metrics-server ProcessMemory:3454896kB ProcessID:13521 Time:2023-04-29 18:17:48 +0000 UTC} {Node:gce-scale-cluster-minion-group-3-xkq3 Process:metrics-server ProcessMemory:3522356kB ProcessID:6573 Time:2023-04-29 18:21:16 +0000 UTC} {Node:gce-scale-cluster-minion-group-2-h3lf Process:metrics-server ProcessMemory:3591480kB ProcessID:14838 Time:2023-04-29 18:41:22 +0000 UTC} {Node:gce-scale-cluster-minion-group-2-h3lf Process:metrics-server ProcessMemory:3589048kB ProcessID:15237 Time:2023-04-29 18:46:10 +0000 UTC}]]]
:0
from junit.xml
Filter through log files | View test history on testgrid
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=ClusterLoaderV2\sload\:\s\[step\:\s22\]\sWaiting\sfor\s\'scale\sand\supdate\sobjects\'\sto\sbe\scompleted\s\[00\]\s\-\s$'
:0
[measurement call WaitForControlledPodsRunning - WaitForRunningDaemonSets error: 1 objects timed out: DaemonSets: test-u6ku0u-1/daemonset-0
measurement call WaitForControlledPodsRunning - WaitForRunningJobs error: 1 objects timed out: Jobs: test-u6ku0u-22/big-job-0
measurement call WaitForControlledPodsRunning - WaitForRunningDeployments error: 28 objects timed out: Deployments: test-u6ku0u-46/medium-deployment-21, test-u6ku0u-48/medium-deployment-1, test-u6ku0u-40/small-deployment-64, test-u6ku0u-27/big-deployment-1, test-u6ku0u-27/small-deployment-178, test-u6ku0u-42/big-deployment-1, test-u6ku0u-41/small-deployment-261, test-u6ku0u-12/big-deployment-1, test-u6ku0u-7/medium-deployment-7, test-u6ku0u-5/small-deployment-184, test-u6ku0u-13/small-deployment-48, test-u6ku0u-3/small-deployment-22, test-u6ku0u-11/small-deployment-159, test-u6ku0u-44/small-deployment-83, test-u6ku0u-21/small-deployment-75, test-u6ku0u-24/small-deployment-282, test-u6ku0u-20/small-deployment-157, test-u6ku0u-37/big-deployment-0, test-u6ku0u-5/small-deployment-57, test-u6ku0u-30/medium-deployment-10, test-u6ku0u-24/medium-deployment-9, test-u6ku0u-25/big-deployment-1, test-u6ku0u-18/medium-deployment-3, test-u6ku0u-14/big-deployment-1, test-u6ku0u-33/medium-deployment-14, test-u6ku0u-2/medium-deployment-8, test-u6ku0u-18/medium-deployment-2, test-u6ku0u-30/small-deployment-122]
:0
from junit.xml
Filter through log files | View test history on testgrid
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=ClusterLoaderV2\sload\:\s\[step\:\s25\]\sWaiting\sfor\s\'delete\sobjects\'\sto\sbe\scompleted\s\[00\]\s\-\s$'
:0
[measurement call WaitForControlledPodsRunning - WaitForRunningDaemonSets error: 1 objects timed out: DaemonSets: test-u6ku0u-1/daemonset-0
measurement call WaitForControlledPodsRunning - WaitForRunningStatefulSets error: 1 objects timed out: StatefulSets: test-u6ku0u-14/small-statefulset-0
measurement call WaitForControlledPodsRunning - WaitForRunningJobs error: 9 objects timed out: Jobs: test-u6ku0u-49/big-job-0, test-u6ku0u-25/medium-job-0, test-u6ku0u-32/big-job-0, test-u6ku0u-13/big-job-0, test-u6ku0u-20/big-job-0, test-u6ku0u-25/big-job-0, test-u6ku0u-1/big-job-0, test-u6ku0u-29/big-job-0, test-u6ku0u-22/big-job-0
measurement call WaitForControlledPodsRunning - WaitForRunningDeployments error: 77 objects timed out: Deployments: test-u6ku0u-42/big-deployment-1, test-u6ku0u-41/small-deployment-261, test-u6ku0u-13/small-deployment-72, test-u6ku0u-34/medium-deployment-17, test-u6ku0u-16/medium-deployment-13, test-u6ku0u-46/big-deployment-0, test-u6ku0u-38/small-deployment-37, test-u6ku0u-11/big-deployment-0, test-u6ku0u-29/small-deployment-274, test-u6ku0u-25/small-deployment-283, test-u6ku0u-25/medium-deployment-13, test-u6ku0u-12/big-deployment-1, test-u6ku0u-34/small-deployment-1, test-u6ku0u-13/small-deployment-34, test-u6ku0u-3/small-deployment-166, test-u6ku0u-22/big-deployment-1, test-u6ku0u-27/medium-deployment-7, test-u6ku0u-31/small-deployment-156, test-u6ku0u-3/medium-deployment-9, test-u6ku0u-40/medium-deployment-12, test-u6ku0u-34/medium-deployment-15, test-u6ku0u-3/big-deployment-0, test-u6ku0u-40/medium-deployment-14, test-u6ku0u-7/medium-deployment-7, test-u6ku0u-10/small-deployment-213, test-u6ku0u-7/medium-deployment-12, test-u6ku0u-5/small-deployment-184, test-u6ku0u-13/small-deployment-48, test-u6ku0u-10/big-deployment-0, test-u6ku0u-3/small-deployment-22, test-u6ku0u-11/small-deployment-159, test-u6ku0u-44/small-deployment-83, test-u6ku0u-21/small-deployment-75, test-u6ku0u-30/small-deployment-211, test-u6ku0u-24/small-deployment-282, test-u6ku0u-29/big-deployment-1, test-u6ku0u-32/big-deployment-1, test-u6ku0u-20/small-deployment-157, test-u6ku0u-23/medium-deployment-6, test-u6ku0u-14/small-deployment-146, test-u6ku0u-37/big-deployment-0, test-u6ku0u-25/small-deployment-34, test-u6ku0u-5/small-deployment-57, test-u6ku0u-30/medium-deployment-10, test-u6ku0u-8/small-deployment-132, test-u6ku0u-24/medium-deployment-9, test-u6ku0u-44/small-deployment-101, test-u6ku0u-10/small-deployment-253, test-u6ku0u-13/small-deployment-216, test-u6ku0u-5/big-deployment-0, test-u6ku0u-25/big-deployment-1, test-u6ku0u-18/medium-deployment-3, test-u6ku0u-44/small-deployment-10, test-u6ku0u-50/big-deployment-0, test-u6ku0u-14/big-deployment-1, test-u6ku0u-33/medium-deployment-14, test-u6ku0u-2/medium-deployment-8, test-u6ku0u-18/medium-deployment-2, test-u6ku0u-48/big-deployment-0, test-u6ku0u-30/small-deployment-122, test-u6ku0u-9/medium-deployment-20, test-u6ku0u-41/medium-deployment-3, test-u6ku0u-22/medium-deployment-13, test-u6ku0u-43/big-deployment-0, test-u6ku0u-17/big-deployment-1, test-u6ku0u-26/small-deployment-141, test-u6ku0u-30/small-deployment-69, test-u6ku0u-46/medium-deployment-21, test-u6ku0u-25/small-deployment-227, test-u6ku0u-46/small-deployment-216, test-u6ku0u-48/medium-deployment-1, test-u6ku0u-44/small-deployment-296, test-u6ku0u-40/small-deployment-64, test-u6ku0u-49/medium-deployment-13, test-u6ku0u-27/big-deployment-1, test-u6ku0u-27/small-deployment-178, test-u6ku0u-8/medium-deployment-12]
:0
from junit.xml
Filter through log files | View test history on testgrid
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=ClusterLoaderV2\sload\:\s\[step\:\s25\]\sWaiting\sfor\s\'delete\sobjects\'\sto\sbe\scompleted\s\[01\]\s\-\sWaitForPVCsToBeDeleted$'
:0
[measurement call WaitForBoundPVCs - WaitForPVCsToBeDeleted error: timeout while waiting for 0 PVCs to be running in namespace '' with labels 'group = load' and fields '' - only 1 found bound]
:0
from junit.xml
Filter through log files | View test history on testgrid
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=ClusterLoaderV2\sload\:\s\[step\:\s29\]\sgathering\smeasurements\s\[09\]\s\-\sTestMetrics$'
:0
[measurement call TestMetrics - TestMetrics error: [action gather failed for SystemPodMetrics measurement: restart counts violation: RestartCount(metrics-server-v0.6.3-6d58b7896b-tv24s, metrics-server)=3, want <= 0; RestartCount(metrics-server-v0.6.3-6d58b7896b-xlrcl, metrics-server)=6, want <= 0
action gather failed for ClusterOOMsTracker measurement: OOMs recorded: [{Node:gce-scale-cluster-minion-group-3-xkq3 Process:metrics-server ProcessMemory:3659772kB ProcessID:2981 Time:2023-04-29 17:48:16 +0000 UTC} {Node:gce-scale-cluster-minion-group-3-xkq3 Process:metrics-server ProcessMemory:3659324kB ProcessID:6317 Time:2023-04-29 17:51:14 +0000 UTC} {Node:gce-scale-cluster-minion-group-gtws Process:metrics-server ProcessMemory:3436732kB ProcessID:11475 Time:2023-04-29 18:09:17 +0000 UTC} {Node:gce-scale-cluster-minion-group-gtws Process:metrics-server ProcessMemory:3455024kB ProcessID:11817 Time:2023-04-29 18:10:50 +0000 UTC} {Node:gce-scale-cluster-minion-group-gtws Process:metrics-server ProcessMemory:3443440kB ProcessID:12420 Time:2023-04-29 18:11:45 +0000 UTC} {Node:gce-scale-cluster-minion-group-gtws Process:metrics-server ProcessMemory:3590584kB ProcessID:12522 Time:2023-04-29 18:13:00 +0000 UTC} {Node:gce-scale-cluster-minion-group-gtws Process:metrics-server ProcessMemory:3520436kB ProcessID:12724 Time:2023-04-29 18:15:51 +0000 UTC} {Node:gce-scale-cluster-minion-group-gtws Process:metrics-server ProcessMemory:3454896kB ProcessID:13521 Time:2023-04-29 18:17:48 +0000 UTC} {Node:gce-scale-cluster-minion-group-3-xkq3 Process:metrics-server ProcessMemory:3522356kB ProcessID:6573 Time:2023-04-29 18:21:16 +0000 UTC} {Node:gce-scale-cluster-minion-group-2-h3lf Process:metrics-server ProcessMemory:3591480kB ProcessID:14838 Time:2023-04-29 18:41:22 +0000 UTC} {Node:gce-scale-cluster-minion-group-2-h3lf Process:metrics-server ProcessMemory:3589048kB ProcessID:15237 Time:2023-04-29 18:46:10 +0000 UTC}]]]
:0
from junit.xml
Filter through log files | View test history on testgrid
error during /home/prow/go/src/k8s.io/perf-tests/run-e2e.sh cluster-loader2 --experimental-gcp-snapshot-prometheus-disk=true --experimental-prometheus-disk-snapshot-name=ci-kubernetes-e2e-gce-scale-performance-1652357519764361216 --experimental-prometheus-snapshot-to-report-dir=true --nodes=5000 --prometheus-scrape-node-exporter --provider=gce --report-dir=/logs/artifacts --testconfig=testing/load/config.yaml --testconfig=testing/huge-service/config.yaml --testconfig=testing/access-tokens/config.yaml --testoverrides=./testing/experiments/enable_restart_count_check.yaml --testoverrides=./testing/experiments/ignore_known_gce_container_restarts.yaml --testoverrides=./testing/overrides/5000_nodes.yaml: exit status 1
from junit_runner.xml
Filter through log files | View test history on testgrid
ClusterLoaderV2 access-tokens overall (testing/access-tokens/config.yaml)
ClusterLoaderV2 access-tokens: [step: 01] Starting measurements [00] - APIResponsivenessPrometheus
ClusterLoaderV2 access-tokens: [step: 01] Starting measurements [01] - TestMetrics
ClusterLoaderV2 access-tokens: [step: 02] Creating ServiceAccounts
ClusterLoaderV2 access-tokens: [step: 03] Creating Tokens
ClusterLoaderV2 access-tokens: [step: 04] Starting measurement for waiting for pods [00] - WaitForRunningPods
ClusterLoaderV2 access-tokens: [step: 05] Creating pods
ClusterLoaderV2 access-tokens: [step: 06] Waiting for pods to be running [00] - WaitForRunningPods
ClusterLoaderV2 access-tokens: [step: 07] Wait 5min [00] - Wait
ClusterLoaderV2 access-tokens: [step: 08] Deleting pods
ClusterLoaderV2 access-tokens: [step: 09] Waiting for pods to be deleted [00] - WaitForRunningPods
ClusterLoaderV2 access-tokens: [step: 10] Collecting measurements [00] - APIResponsivenessPrometheus
ClusterLoaderV2 access-tokens: [step: 10] Collecting measurements [01] - TestMetrics
ClusterLoaderV2 huge-service overall (testing/huge-service/config.yaml)
ClusterLoaderV2 huge-service: [step: 01] starting measurements [00] - APIResponsivenessPrometheus
ClusterLoaderV2 huge-service: [step: 01] starting measurements [01] - APIResponsivenessPrometheusSimple
ClusterLoaderV2 huge-service: [step: 01] starting measurements [02] - TestMetrics
ClusterLoaderV2 huge-service: [step: 01] starting measurements [03] - InClusterNetworkLatency
ClusterLoaderV2 huge-service: [step: 02] Create huge-service
ClusterLoaderV2 huge-service: [step: 03] Creating huge-service measurements [00] - WaitForHugeServiceDeployments
ClusterLoaderV2 huge-service: [step: 04] Creating huge-service pods
ClusterLoaderV2 huge-service: [step: 05] Waiting for huge-service pods to be created [00] - WaitForHugeServiceDeployments
ClusterLoaderV2 huge-service: [step: 06] Updating huge-service pods
ClusterLoaderV2 huge-service: [step: 07] Waiting for huge-service pods to be updated [00] - WaitForHugeServiceDeployments
ClusterLoaderV2 huge-service: [step: 08] Deleting huge-service pods
ClusterLoaderV2 huge-service: [step: 09] Waiting for huge-service pods to be deleted [00] - WaitForHugeServiceDeployments
ClusterLoaderV2 huge-service: [step: 10] Delete huge-service
ClusterLoaderV2 huge-service: [step: 11] gathering measurements [00] - APIResponsivenessPrometheus
ClusterLoaderV2 huge-service: [step: 11] gathering measurements [01] - APIResponsivenessPrometheusSimple
ClusterLoaderV2 huge-service: [step: 11] gathering measurements [02] - TestMetrics
ClusterLoaderV2 huge-service: [step: 11] gathering measurements [03] - InClusterNetworkLatency
ClusterLoaderV2 load: [step: 01] starting measurements [00] - APIResponsivenessPrometheus
ClusterLoaderV2 load: [step: 01] starting measurements [01] - APIResponsivenessPrometheusSimple
ClusterLoaderV2 load: [step: 01] starting measurements [02] - CreatePhasePodStartupLatency
ClusterLoaderV2 load: [step: 01] starting measurements [03] - InClusterNetworkLatency
ClusterLoaderV2 load: [step: 01] starting measurements [04] - SLOMeasurement
ClusterLoaderV2 load: [step: 01] starting measurements [05] - NetworkProgrammingLatency
ClusterLoaderV2 load: [step: 01] starting measurements [06] - Kube-proxy partial iptables restore failures
ClusterLoaderV2 load: [step: 01] starting measurements [07] - APIAvailability
ClusterLoaderV2 load: [step: 01] starting measurements [08] - Quotas total usage
ClusterLoaderV2 load: [step: 01] starting measurements [09] - TestMetrics
ClusterLoaderV2 load: [step: 02] Creating k8s services
ClusterLoaderV2 load: [step: 03] Creating PriorityClass for DaemonSets
ClusterLoaderV2 load: [step: 04] create objects configmaps and secrets
ClusterLoaderV2 load: [step: 05] Starting measurement for 'create objects' [00] -
ClusterLoaderV2 load: [step: 06] create objects
ClusterLoaderV2 load: [step: 07] Waiting for 'create objects' to be completed [00] -
ClusterLoaderV2 load: [step: 08] Creating scheduler throughput measurements [00] - HighThroughputPodStartupLatency
ClusterLoaderV2 load: [step: 08] Creating scheduler throughput measurements [01] - WaitForSchedulerThroughputDeployments
ClusterLoaderV2 load: [step: 08] Creating scheduler throughput measurements [02] - SchedulingThroughput
ClusterLoaderV2 load: [step: 09] create scheduler throughput pods
ClusterLoaderV2 load: [step: 10] Waiting for scheduler throughput pods to be created [00] - WaitForSchedulerThroughputDeployments
ClusterLoaderV2 load: [step: 11] Collecting scheduler throughput measurements [00] - HighThroughputPodStartupLatency
ClusterLoaderV2 load: [step: 11] Collecting scheduler throughput measurements [01] - SchedulingThroughput
ClusterLoaderV2 load: [step: 12] delete scheduler throughput pods
ClusterLoaderV2 load: [step: 13] Waiting for scheduler throughput pods to be deleted [00] - WaitForSchedulerThroughputDeployments
ClusterLoaderV2 load: [step: 14] Starting latency pod measurements [00] - PodStartupLatency
ClusterLoaderV2 load: [step: 14] Starting latency pod measurements [01] - WaitForRunningLatencyDeployments
ClusterLoaderV2 load: [step: 15] Creating latency pods
ClusterLoaderV2 load: [step: 16] Waiting for latency pods to be running [00] - WaitForRunningLatencyDeployments
ClusterLoaderV2 load: [step: 17] Deleting latency pods
ClusterLoaderV2 load: [step: 18] Waiting for latency pods to be deleted [00] - WaitForRunningLatencyDeployments
ClusterLoaderV2 load: [step: 19] Collecting pod startup latency [00] - PodStartupLatency
ClusterLoaderV2 load: [step: 20] Starting measurement for 'scale and update objects' [00] -
ClusterLoaderV2 load: [step: 21] scale and update objects
ClusterLoaderV2 load: [step: 23] Starting measurement for 'delete objects' [00] -
ClusterLoaderV2 load: [step: 24] delete objects
ClusterLoaderV2 load: [step: 26] delete objects configmaps and secrets
ClusterLoaderV2 load: [step: 27] Deleting PriorityClass for DaemonSets
ClusterLoaderV2 load: [step: 28] Deleting k8s services
ClusterLoaderV2 load: [step: 29] gathering measurements [00] - APIResponsivenessPrometheus
ClusterLoaderV2 load: [step: 29] gathering measurements [01] - APIResponsivenessPrometheusSimple
ClusterLoaderV2 load: [step: 29] gathering measurements [02] - CreatePhasePodStartupLatency
ClusterLoaderV2 load: [step: 29] gathering measurements [03] - InClusterNetworkLatency
ClusterLoaderV2 load: [step: 29] gathering measurements [04] - SLOMeasurement
ClusterLoaderV2 load: [step: 29] gathering measurements [05] - NetworkProgrammingLatency
ClusterLoaderV2 load: [step: 29] gathering measurements [06] - Kube-proxy partial iptables restore failures
ClusterLoaderV2 load: [step: 29] gathering measurements [07] - APIAvailability
ClusterLoaderV2 load: [step: 29] gathering measurements [08] - Quotas total usage
kubetest Check APIReachability
kubetest Deferred TearDown
kubetest DumpClusterLogs
kubetest Extract
kubetest GetDeployer
kubetest Prepare
kubetest TearDown
kubetest TearDown Previous
kubetest Timeout
kubetest Up
kubetest list nodes
kubetest test setup