PR #115331 pull-kubernetes-e2e-gce-100-performance #1631316720402567168
Recent runs || View in Spyglass
Recent runs || View in Spyglass
| PR | mimowo: Give terminal phase correctly to all pods that will not be restarted |
| Result | ABORTED |
| Tests | 0 failed / 86 succeeded |
| Started | |
| Elapsed | 1h0m |
| Revision | 875d1676f6959292721af3866e3081b8cf3d871a |
| Refs |
115331 |
| job-version | v1.27.0-alpha.2.457+f96b55413a4756 |
| kubetest-version | v20230222-b5208facd4 |
| revision | v1.27.0-alpha.2.457+f96b55413a4756 |
ClusterLoaderV2 huge-service overall (testing/huge-service/config.yaml)
ClusterLoaderV2 huge-service: [step: 01] starting measurements [00] - APIResponsivenessPrometheus
ClusterLoaderV2 huge-service: [step: 01] starting measurements [01] - APIResponsivenessPrometheusSimple
ClusterLoaderV2 huge-service: [step: 01] starting measurements [02] - TestMetrics
ClusterLoaderV2 huge-service: [step: 01] starting measurements [03] - InClusterNetworkLatency
ClusterLoaderV2 huge-service: [step: 02] Create huge-service
ClusterLoaderV2 huge-service: [step: 03] Creating huge-service measurements [00] - WaitForHugeServiceDeployments
ClusterLoaderV2 huge-service: [step: 04] Creating huge-service pods
ClusterLoaderV2 huge-service: [step: 05] Waiting for huge-service pods to be created [00] - WaitForHugeServiceDeployments
ClusterLoaderV2 huge-service: [step: 06] Updating huge-service pods
ClusterLoaderV2 huge-service: [step: 07] Waiting for huge-service pods to be updated [00] - WaitForHugeServiceDeployments
ClusterLoaderV2 huge-service: [step: 08] Deleting huge-service pods
ClusterLoaderV2 huge-service: [step: 09] Waiting for huge-service pods to be deleted [00] - WaitForHugeServiceDeployments
ClusterLoaderV2 huge-service: [step: 10] Delete huge-service
ClusterLoaderV2 huge-service: [step: 11] gathering measurements [00] - APIResponsivenessPrometheus
ClusterLoaderV2 huge-service: [step: 11] gathering measurements [01] - APIResponsivenessPrometheusSimple
ClusterLoaderV2 huge-service: [step: 11] gathering measurements [02] - TestMetrics
ClusterLoaderV2 huge-service: [step: 11] gathering measurements [03] - InClusterNetworkLatency
ClusterLoaderV2 load overall (testing/load/config.yaml)
ClusterLoaderV2 load: [step: 01] starting measurements [00] - APIResponsivenessPrometheus
ClusterLoaderV2 load: [step: 01] starting measurements [01] - APIResponsivenessPrometheusSimple
ClusterLoaderV2 load: [step: 01] starting measurements [02] - CreatePhasePodStartupLatency
ClusterLoaderV2 load: [step: 01] starting measurements [03] - InClusterNetworkLatency
ClusterLoaderV2 load: [step: 01] starting measurements [04] - SLOMeasurement
ClusterLoaderV2 load: [step: 01] starting measurements [05] - NetworkProgrammingLatency
ClusterLoaderV2 load: [step: 01] starting measurements [06] - Kube-proxy partial iptables restore failures
ClusterLoaderV2 load: [step: 01] starting measurements [07] - APIAvailability
ClusterLoaderV2 load: [step: 01] starting measurements [08] - Quotas total usage
ClusterLoaderV2 load: [step: 01] starting measurements [09] - TestMetrics
ClusterLoaderV2 load: [step: 02] Creating k8s services
ClusterLoaderV2 load: [step: 03] Creating PriorityClass for DaemonSets
ClusterLoaderV2 load: [step: 04] create objects configmaps and secrets
ClusterLoaderV2 load: [step: 05] Starting measurement for 'create objects' [00] -
ClusterLoaderV2 load: [step: 06] create objects
ClusterLoaderV2 load: [step: 07] Waiting for 'create objects' to be completed [00] -
ClusterLoaderV2 load: [step: 08] Creating scheduler throughput measurements [00] - HighThroughputPodStartupLatency
ClusterLoaderV2 load: [step: 08] Creating scheduler throughput measurements [01] - WaitForSchedulerThroughputDeployments
ClusterLoaderV2 load: [step: 08] Creating scheduler throughput measurements [02] - SchedulingThroughput
ClusterLoaderV2 load: [step: 09] create scheduler throughput pods
ClusterLoaderV2 load: [step: 10] Waiting for scheduler throughput pods to be created [00] - WaitForSchedulerThroughputDeployments
ClusterLoaderV2 load: [step: 11] Collecting scheduler throughput measurements [00] - HighThroughputPodStartupLatency
ClusterLoaderV2 load: [step: 11] Collecting scheduler throughput measurements [01] - SchedulingThroughput
ClusterLoaderV2 load: [step: 12] delete scheduler throughput pods
ClusterLoaderV2 load: [step: 13] Waiting for scheduler throughput pods to be deleted [00] - WaitForSchedulerThroughputDeployments
ClusterLoaderV2 load: [step: 14] Starting latency pod measurements [00] - PodStartupLatency
ClusterLoaderV2 load: [step: 14] Starting latency pod measurements [01] - WaitForRunningLatencyDeployments
ClusterLoaderV2 load: [step: 15] Creating latency pods
ClusterLoaderV2 load: [step: 16] Waiting for latency pods to be running [00] - WaitForRunningLatencyDeployments
ClusterLoaderV2 load: [step: 17] Deleting latency pods
ClusterLoaderV2 load: [step: 18] Waiting for latency pods to be deleted [00] - WaitForRunningLatencyDeployments
ClusterLoaderV2 load: [step: 19] Collecting pod startup latency [00] - PodStartupLatency
ClusterLoaderV2 load: [step: 20] Starting measurement for 'scale and update objects' [00] -
ClusterLoaderV2 load: [step: 21] scale and update objects
ClusterLoaderV2 load: [step: 22] Waiting for 'scale and update objects' to be completed [00] -
ClusterLoaderV2 load: [step: 23] Starting measurement for 'delete objects' [00] -
ClusterLoaderV2 load: [step: 24] delete objects
ClusterLoaderV2 load: [step: 25] Waiting for 'delete objects' to be completed [00] -
ClusterLoaderV2 load: [step: 25] Waiting for 'delete objects' to be completed [01] - WaitForPVCsToBeDeleted
ClusterLoaderV2 load: [step: 26] delete objects configmaps and secrets
ClusterLoaderV2 load: [step: 27] Deleting PriorityClass for DaemonSets
ClusterLoaderV2 load: [step: 28] Deleting k8s services
ClusterLoaderV2 load: [step: 29] gathering measurements [00] - APIResponsivenessPrometheus
ClusterLoaderV2 load: [step: 29] gathering measurements [01] - APIResponsivenessPrometheusSimple
ClusterLoaderV2 load: [step: 29] gathering measurements [02] - CreatePhasePodStartupLatency
ClusterLoaderV2 load: [step: 29] gathering measurements [03] - InClusterNetworkLatency
ClusterLoaderV2 load: [step: 29] gathering measurements [04] - SLOMeasurement
ClusterLoaderV2 load: [step: 29] gathering measurements [05] - NetworkProgrammingLatency
ClusterLoaderV2 load: [step: 29] gathering measurements [06] - Kube-proxy partial iptables restore failures
ClusterLoaderV2 load: [step: 29] gathering measurements [07] - APIAvailability
ClusterLoaderV2 load: [step: 29] gathering measurements [08] - Quotas total usage
ClusterLoaderV2 load: [step: 29] gathering measurements [09] - TestMetrics
kubetest Build
kubetest Check APIReachability
kubetest ClusterLoaderV2
kubetest Deferred TearDown
kubetest DumpClusterLogs
kubetest Extract
kubetest GetDeployer
kubetest Prepare
kubetest Stage
kubetest TearDown
kubetest TearDown Previous
kubetest Timeout
kubetest Up
kubetest list nodes
kubetest test setup
... skipping 67 lines ... HOSTNAME=845d9d67-b90f-11ed-8e16-e65bc73ce4db IMAGE=gcr.io/k8s-staging-test-infra/kubekins-e2e:v20230222-b5208facd4-master INSTANCE_PREFIX=e2e-115331-95a39 JENKINS_GCE_SSH_PRIVATE_KEY_FILE=/workspace/.ssh/google_compute_engine JENKINS_GCE_SSH_PUBLIC_KEY_FILE=/workspace/.ssh/google_compute_engine.pub JOB_NAME=pull-kubernetes-e2e-gce-100-performance JOB_SPEC={"type":"presubmit","job":"pull-kubernetes-e2e-gce-100-performance","buildid":"1631316720402567168","prowjobid":"845d9d67-b90f-11ed-8e16-e65bc73ce4db","refs":{"org":"kubernetes","repo":"kubernetes","repo_link":"https://github.com/kubernetes/kubernetes","base_ref":"master","base_sha":"b6d102d634d357600bc0fcb076395c88caef6acb","base_link":"https://github.com/kubernetes/kubernetes/commit/b6d102d634d357600bc0fcb076395c88caef6acb","pulls":[{"number":115331,"author":"mimowo","sha":"875d1676f6959292721af3866e3081b8cf3d871a","title":"Mark Pending Terminating pods as Failed by Kubelet","link":"https://github.com/kubernetes/kubernetes/pull/115331","commit_link":"https://github.com/kubernetes/kubernetes/pull/115331/commits/875d1676f6959292721af3866e3081b8cf3d871a","author_link":"https://github.com/mimowo"}],"path_alias":"k8s.io/kubernetes"},"extra_refs":[{"org":"kubernetes","repo":"perf-tests","base_ref":"master","path_alias":"k8s.io/perf-tests"},{"org":"kubernetes","repo":"release","base_ref":"master","path_alias":"k8s.io/release"}],"decoration_config":{"timeout":"2h0m0s","grace_period":"15m0s","utility_images":{"clonerefs":"gcr.io/k8s-prow/clonerefs:v20230301-6893d98ee8","initupload":"gcr.io/k8s-prow/initupload:v20230301-6893d98ee8","entrypoint":"gcr.io/k8s-prow/entrypoint:v20230301-6893d98ee8","sidecar":"gcr.io/k8s-prow/sidecar:v20230301-6893d98ee8"},"resources":{"clonerefs":{"requests":{"cpu":"100m"}},"initupload":{"requests":{"cpu":"100m"}},"place_entrypoint":{"requests":{"cpu":"100m"}},"sidecar":{"requests":{"cpu":"100m"}}},"gcs_configuration":{"bucket":"kubernetes-jenkins","path_strategy":"legacy","default_org":"kubernetes","default_repo":"kubernetes"},"gcs_credentials_secret":"service-account"}} JOB_TYPE=presubmit KUBECTL_PRUNE_WHITELIST_OVERRIDE=core/v1/ConfigMap core/v1/Endpoints core/v1/Namespace core/v1/PersistentVolumeClaim core/v1/PersistentVolume core/v1/ReplicationController core/v1/Secret core/v1/Service batch/v1/Job batch/v1/CronJob apps/v1/DaemonSet apps/v1/Deployment apps/v1/ReplicaSet apps/v1/StatefulSet networking.k8s.io/v1/Ingress KUBELET_TEST_ARGS=--enable-debugging-handlers --kube-api-qps=100 --kube-api-burst=100 KUBEMARK_APISERVER_TEST_ARGS=--max-requests-inflight=80 --max-mutating-requests-inflight=0 --profiling --contention-profiling KUBEPROXY_TEST_ARGS=--profiling --metrics-bind-address=0.0.0.0 --feature-gates=MinimizeIPTablesRestore=true KUBERNETES_PORT=tcp://10.35.240.1:443 ... skipping 568 lines ... NAME ZONE MACHINE_TYPE PREEMPTIBLE INTERNAL_IP EXTERNAL_IP STATUS e2e-115331-95a39-master us-east1-b n1-standard-8 10.40.0.3 34.148.208.126 RUNNING Setting e2e-115331-95a39-master's aliases to 'pods-default:10.64.0.0/24;10.40.0.2/32' (added 10.40.0.2) Updating network interface [nic0] of instance [e2e-115331-95a39-master]... ..........done. Updated [https://www.googleapis.com/compute/v1/projects/k8s-infra-e2e-boskos-scale-08/zones/us-east1-b/instances/e2e-115331-95a39-master]. Failed to execute 'sudo /bin/bash /home/kubernetes/bin/kube-master-internal-route.sh' on e2e-115331-95a39-master despite 5 attempts Last attempt failed with: /bin/bash: /home/kubernetes/bin/kube-master-internal-route.sh: No such file or directory Creating firewall... ..Created [https://www.googleapis.com/compute/v1/projects/k8s-infra-e2e-boskos-scale-08/global/firewalls/e2e-115331-95a39-minion-all]. done. NAME NETWORK DIRECTION PRIORITY ALLOW DENY DISABLED e2e-115331-95a39-minion-all e2e-115331-95a39 INGRESS 1000 tcp,udp,icmp,esp,ah,sctp False Creating nodes. ... skipping 34 lines ... Looking for address 'e2e-115331-95a39-master-ip' Looking for address 'e2e-115331-95a39-master-internal-ip' Using master: e2e-115331-95a39-master (external IP: 34.148.208.126; internal IP: 10.40.0.2) Waiting up to 300 seconds for cluster initialization. This will continually check to see if the API for kubernetes is reachable. This may time out if there was some uncaught error during start up. Kubernetes cluster created. Cluster "k8s-infra-e2e-boskos-scale-08_e2e-115331-95a39" set. User "k8s-infra-e2e-boskos-scale-08_e2e-115331-95a39" set. Context "k8s-infra-e2e-boskos-scale-08_e2e-115331-95a39" created. Switched to context "k8s-infra-e2e-boskos-scale-08_e2e-115331-95a39". ... skipping 238 lines ... e2e-115331-95a39-minion-group-zrlj Ready <none> 50s v1.27.0-alpha.2.457+f96b55413a4756 e2e-115331-95a39-minion-group-zsr9 Ready <none> 53s v1.27.0-alpha.2.457+f96b55413a4756 e2e-115331-95a39-minion-heapster Ready <none> 67s v1.27.0-alpha.2.457+f96b55413a4756 Warning: v1 ComponentStatus is deprecated in v1.19+ Validate output: Warning: v1 ComponentStatus is deprecated in v1.19+ NAME STATUS MESSAGE ERROR etcd-1 Healthy {"health":"true","reason":""} etcd-0 Healthy {"health":"true","reason":""} controller-manager Healthy ok scheduler Healthy ok [0;33mCluster validation encountered some problems, but cluster should be in working order[0m ...ignoring non-fatal errors in validate-cluster Done, listing cluster services: [0;32mKubernetes control plane[0m is running at [0;33mhttps://34.148.208.126[0m [0;32mGLBCDefaultBackend[0m is running at [0;33mhttps://34.148.208.126/api/v1/namespaces/kube-system/services/default-http-backend:http/proxy[0m [0;32mCoreDNS[0m is running at [0;33mhttps://34.148.208.126/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy[0m [0;32mMetrics-server[0m is running at [0;33mhttps://34.148.208.126/api/v1/namespaces/kube-system/services/https:metrics-server:/proxy[0m ... skipping 1133 lines ... I0302 16:11:49.424861 91183 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-napc01-1), controlledBy(small-deployment-4): Pods: 5 out of 5 created, 5 running (5 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0302 16:11:49.479033 91183 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-napc01-1), controlledBy(small-deployment-291): Pods: 5 out of 5 created, 5 running (5 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0302 16:11:49.716515 91183 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-napc01-1), controlledBy(small-deployment-268): Pods: 5 out of 5 created, 5 running (5 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0302 16:11:50.249877 91183 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-napc01-1), controlledBy(small-deployment-150): Pods: 5 out of 5 created, 5 running (5 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0302 16:11:50.725802 91183 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-napc01-1), controlledBy(small-deployment-229): Pods: 5 out of 5 created, 5 running (5 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0302 16:11:50.771040 91183 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-napc01-1), controlledBy(small-deployment-182): Pods: 5 out of 5 created, 5 running (5 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0302 16:11:50.812963 91183 wait_for_controlled_pods.go:365] WaitForControlledPodsRunning: running 3, deleted 0, timeout: 0, failed: 0 I0302 16:11:50.813003 91183 wait_for_controlled_pods.go:370] WaitForControlledPodsRunning: maxDuration=5.004480621s, operationTimeout=15m0s, ratio=0.01 I0302 16:11:50.813023 91183 wait_for_controlled_pods.go:384] WaitForControlledPodsRunning: 3/3 Jobs are running with all pods I0302 16:11:50.819217 91183 wait_for_controlled_pods.go:365] WaitForControlledPodsRunning: running 1, deleted 0, timeout: 0, failed: 0 I0302 16:11:50.819264 91183 wait_for_controlled_pods.go:370] WaitForControlledPodsRunning: maxDuration=5.002485071s, operationTimeout=15m0s, ratio=0.01 I0302 16:11:50.819290 91183 wait_for_controlled_pods.go:384] WaitForControlledPodsRunning: 1/1 DaemonSets are running with all pods I0302 16:11:50.998075 91183 wait_for_controlled_pods.go:365] WaitForControlledPodsRunning: running 323, deleted 0, timeout: 0, failed: 0 I0302 16:11:50.998141 91183 wait_for_controlled_pods.go:370] WaitForControlledPodsRunning: maxDuration=10.017243259s, operationTimeout=15m0s, ratio=0.01 I0302 16:11:50.998158 91183 wait_for_controlled_pods.go:384] WaitForControlledPodsRunning: 323/323 Deployments are running with all pods I0302 16:11:51.381427 91183 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-napc01-1), controlledBy(medium-statefulset-0): Pods: 30 out of 30 created, 18 running (18 updated), 12 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0302 16:11:56.382886 91183 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-napc01-1), controlledBy(medium-statefulset-0): Pods: 30 out of 30 created, 22 running (22 updated), 8 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0302 16:12:01.384430 91183 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-napc01-1), controlledBy(medium-statefulset-0): Pods: 30 out of 30 created, 28 running (28 updated), 2 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0302 16:12:06.386051 91183 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-napc01-1), controlledBy(medium-statefulset-0): Pods: 30 out of 30 created, 30 running (30 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0302 16:12:06.386142 91183 wait_for_controlled_pods.go:365] WaitForControlledPodsRunning: running 2, deleted 0, timeout: 0, failed: 0 I0302 16:12:06.386163 91183 wait_for_controlled_pods.go:370] WaitForControlledPodsRunning: maxDuration=45.008421783s, operationTimeout=15m0s, ratio=0.05 I0302 16:12:06.386185 91183 wait_for_controlled_pods.go:384] WaitForControlledPodsRunning: 2/2 StatefulSets are running with all pods I0302 16:12:06.386214 91183 simple_test_executor.go:171] Step "[step: 07] Waiting for 'create objects' to be completed" ended I0302 16:12:06.386239 91183 simple_test_executor.go:149] Step "[step: 08] Creating scheduler throughput measurements" started I0302 16:12:06.386371 91183 pod_startup_latency.go:132] PodStartupLatency: labelSelector(group = scheduler-throughput): starting pod startup latency measurement... I0302 16:12:06.386410 91183 wait_for_controlled_pods.go:257] WaitForControlledPodsRunning: starting wait for controlled pods measurement... ... skipping 4 lines ... I0302 16:12:06.528534 91183 simple_test_executor.go:149] Step "[step: 10] Waiting for scheduler throughput pods to be created" started I0302 16:12:06.528554 91183 wait_for_pods.go:64] WaitForControlledPodsRunning: namespace(test-napc01-2), controlledBy(scheduler-throughput-deployment-0): starting with timeout: 19m59.99997623s I0302 16:12:06.528561 91183 wait_for_controlled_pods.go:288] WaitForControlledPodsRunning: waiting for controlled pods measurement... I0302 16:12:11.537503 91183 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-napc01-2), controlledBy(scheduler-throughput-deployment-0): Pods: 575 out of 1000 created, 405 running (405 updated), 168 pending scheduled, 2 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0302 16:12:16.554927 91183 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-napc01-2), controlledBy(scheduler-throughput-deployment-0): Pods: 1000 out of 1000 created, 906 running (906 updated), 94 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0302 16:12:21.572619 91183 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-napc01-2), controlledBy(scheduler-throughput-deployment-0): Pods: 1000 out of 1000 created, 1000 running (1000 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0302 16:12:21.572687 91183 wait_for_controlled_pods.go:365] WaitForControlledPodsRunning: running 1, deleted 0, timeout: 0, failed: 0 I0302 16:12:21.572701 91183 wait_for_controlled_pods.go:370] WaitForControlledPodsRunning: maxDuration=15.044110878s, operationTimeout=20m0s, ratio=0.01 I0302 16:12:21.572717 91183 wait_for_controlled_pods.go:384] WaitForControlledPodsRunning: 1/1 Deployments are running with all pods I0302 16:12:21.572742 91183 simple_test_executor.go:171] Step "[step: 10] Waiting for scheduler throughput pods to be created" ended I0302 16:12:21.572761 91183 simple_test_executor.go:149] Step "[step: 11] Collecting scheduler throughput measurements" started I0302 16:12:21.572799 91183 scheduling_throughput.go:154] SchedulingThroughput: gathering data I0302 16:12:21.572850 91183 pod_startup_latency.go:226] PodStartupLatency: labelSelector(group = scheduler-throughput): gathering pod startup latency measurement... ... skipping 33 lines ... I0302 16:12:22.002308 91183 simple_test_executor.go:149] Step "[step: 13] Waiting for scheduler throughput pods to be deleted" started I0302 16:12:22.002341 91183 wait_for_controlled_pods.go:288] WaitForControlledPodsRunning: waiting for controlled pods measurement... I0302 16:12:22.002524 91183 wait_for_pods.go:64] WaitForControlledPodsRunning: namespace(test-napc01-2), controlledBy(scheduler-throughput-deployment-0): starting with timeout: 19m59.999977458s I0302 16:12:27.011847 91183 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-napc01-2), controlledBy(scheduler-throughput-deployment-0): Pods: 464 out of 0 created, 464 running (464 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 111 terminating, 0 unknown, 0 runningButNotReady I0302 16:12:32.013378 91183 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-napc01-2), controlledBy(scheduler-throughput-deployment-0): Pods: 0 out of 0 created, 0 running (0 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 43 terminating, 0 unknown, 0 runningButNotReady I0302 16:12:37.014103 91183 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-napc01-2), controlledBy(scheduler-throughput-deployment-0): Pods: 0 out of 0 created, 0 running (0 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0302 16:12:37.014171 91183 wait_for_controlled_pods.go:365] WaitForControlledPodsRunning: running 0, deleted 1, timeout: 0, failed: 0 I0302 16:12:37.014191 91183 wait_for_controlled_pods.go:370] WaitForControlledPodsRunning: maxDuration=15.011621734s, operationTimeout=20m0s, ratio=0.01 I0302 16:12:37.014213 91183 wait_for_controlled_pods.go:384] WaitForControlledPodsRunning: 0/0 Deployments are running with all pods I0302 16:12:37.014239 91183 simple_test_executor.go:171] Step "[step: 13] Waiting for scheduler throughput pods to be deleted" ended I0302 16:12:37.014265 91183 simple_test_executor.go:149] Step "[step: 14] Starting latency pod measurements" started I0302 16:12:37.014328 91183 wait_for_controlled_pods.go:257] WaitForControlledPodsRunning: starting wait for controlled pods measurement... I0302 16:12:37.014376 91183 pod_startup_latency.go:132] PodStartupLatency: labelSelector(group = latency): starting pod startup latency measurement... ... skipping 999 lines ... I0302 16:14:21.250148 91183 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-napc01-1), controlledBy(latency-deployment-494): Pods: 1 out of 1 created, 1 running (1 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0302 16:14:21.447835 91183 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-napc01-1), controlledBy(latency-deployment-495): Pods: 1 out of 1 created, 1 running (1 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0302 16:14:21.654286 91183 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-napc01-1), controlledBy(latency-deployment-496): Pods: 1 out of 1 created, 1 running (1 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0302 16:14:21.849332 91183 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-napc01-1), controlledBy(latency-deployment-497): Pods: 1 out of 1 created, 1 running (1 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0302 16:14:22.051475 91183 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-napc01-1), controlledBy(latency-deployment-498): Pods: 1 out of 1 created, 1 running (1 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0302 16:14:22.247845 91183 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-napc01-1), controlledBy(latency-deployment-499): Pods: 1 out of 1 created, 1 running (1 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0302 16:14:22.653135 91183 wait_for_controlled_pods.go:365] WaitForControlledPodsRunning: running 500, deleted 0, timeout: 0, failed: 0 I0302 16:14:22.653177 91183 wait_for_controlled_pods.go:370] WaitForControlledPodsRunning: maxDuration=5.001604947s, operationTimeout=15m0s, ratio=0.01 I0302 16:14:22.653199 91183 wait_for_controlled_pods.go:384] WaitForControlledPodsRunning: 500/500 Deployments are running with all pods I0302 16:14:22.653233 91183 simple_test_executor.go:171] Step "[step: 16] Waiting for latency pods to be running" ended I0302 16:14:22.653252 91183 simple_test_executor.go:149] Step "[step: 17] Deleting latency pods" started I0302 16:14:22.691855 91183 wait_for_pods.go:64] WaitForControlledPodsRunning: namespace(test-napc01-1), controlledBy(latency-deployment-0): starting with timeout: 14m59.999978237s I0302 16:14:22.792731 91183 wait_for_pods.go:64] WaitForControlledPodsRunning: namespace(test-napc01-1), controlledBy(latency-deployment-1): starting with timeout: 14m59.99997798s ... skipping 495 lines ... I0302 16:15:12.392029 91183 wait_for_pods.go:64] WaitForControlledPodsRunning: namespace(test-napc01-1), controlledBy(latency-deployment-497): starting with timeout: 14m59.999983295s I0302 16:15:12.491906 91183 wait_for_pods.go:64] WaitForControlledPodsRunning: namespace(test-napc01-1), controlledBy(latency-deployment-498): starting with timeout: 14m59.999984931s I0302 16:15:12.596540 91183 simple_test_executor.go:171] Step "[step: 17] Deleting latency pods" ended I0302 16:15:12.596580 91183 simple_test_executor.go:149] Step "[step: 18] Waiting for latency pods to be deleted" started I0302 16:15:12.596609 91183 wait_for_controlled_pods.go:288] WaitForControlledPodsRunning: waiting for controlled pods measurement... I0302 16:15:12.596764 91183 wait_for_pods.go:64] WaitForControlledPodsRunning: namespace(test-napc01-1), controlledBy(latency-deployment-499): starting with timeout: 14m59.999975784s I0302 16:15:17.647589 91183 wait_for_controlled_pods.go:365] WaitForControlledPodsRunning: running 0, deleted 500, timeout: 0, failed: 0 I0302 16:15:17.647632 91183 wait_for_controlled_pods.go:370] WaitForControlledPodsRunning: maxDuration=5.001502637s, operationTimeout=15m0s, ratio=0.01 I0302 16:15:17.647650 91183 wait_for_controlled_pods.go:384] WaitForControlledPodsRunning: 0/0 Deployments are running with all pods I0302 16:15:17.647682 91183 simple_test_executor.go:171] Step "[step: 18] Waiting for latency pods to be deleted" ended I0302 16:15:17.647713 91183 simple_test_executor.go:149] Step "[step: 19] Collecting pod startup latency" started I0302 16:15:17.647760 91183 pod_startup_latency.go:226] PodStartupLatency: labelSelector(group = latency): gathering pod startup latency measurement... I0302 16:15:18.153343 91183 phase_latency.go:141] PodStartupLatency: 100 worst pod_startup latencies: [{test-napc01-1/latency-deployment-361-85b9759bc4-tllqp 2.444450488s} {test-napc01-1/latency-deployment-118-6fbdb79966-rk7pl 2.445660837s} {test-napc01-1/latency-deployment-373-85cbc95d6d-hmmm6 2.447797065s} {test-napc01-1/latency-deployment-267-54c9f755f6-zz9lt 2.460596104s} {test-napc01-1/latency-deployment-231-59c868fcdb-25nmd 2.462326641s} {test-napc01-1/latency-deployment-167-76b49f48b6-fllw5 2.485862766s} {test-napc01-1/latency-deployment-152-7d94dd687d-j6h4t 2.490776688s} {test-napc01-1/latency-deployment-148-7f7bfd64d-vxnj9 2.492924932s} {test-napc01-1/latency-deployment-162-56f8854584-mv4s7 2.495545459s} {test-napc01-1/latency-deployment-388-7b868f8c44-44bcp 2.502770239s} {test-napc01-1/latency-deployment-22-5c6697c9f4-9j7pq 2.505132166s} {test-napc01-1/latency-deployment-32-5f9fc4cbfd-9nx8c 2.505161265s} {test-napc01-1/latency-deployment-336-5b5cfcf8b6-58fdd 2.521195869s} {test-napc01-1/latency-deployment-12-55d55c7946-rbgkz 2.525936823s} {test-napc01-1/latency-deployment-83-68c877b4cd-t9q89 2.528317865s} {test-napc01-1/latency-deployment-146-695b497bb4-vdd28 2.528419523s} {test-napc01-1/latency-deployment-278-66d6c86bb4-ffz25 2.533083151s} {test-napc01-1/latency-deployment-253-56b84944fd-wkf5s 2.533896739s} {test-napc01-1/latency-deployment-446-65b969544b-sljkb 2.534148602s} {test-napc01-1/latency-deployment-297-6797cbbcdb-mq7g4 2.538586924s} {test-napc01-1/latency-deployment-461-5dcf9bb844-4phbd 2.544759483s} {test-napc01-1/latency-deployment-211-75c469b5d6-9827g 2.547636318s} {test-napc01-1/latency-deployment-352-77db46585d-zxlnt 2.548494043s} {test-napc01-1/latency-deployment-34-969d7fcf8-z5tjt 2.556373423s} {test-napc01-1/latency-deployment-397-5ccc9dd9b-h7m8b 2.55702035s} {test-napc01-1/latency-deployment-72-6475d69bd6-srfjg 2.561295603s} {test-napc01-1/latency-deployment-262-647dff4546-wxsm2 2.571551811s} {test-napc01-1/latency-deployment-147-5dd9787498-vlzsl 2.578029584s} {test-napc01-1/latency-deployment-328-65587857c6-lhhfk 2.579338099s} {test-napc01-1/latency-deployment-4-7d6bcd7474-qbtbw 2.58580931s} {test-napc01-1/latency-deployment-293-6947c7b94b-px9m9 2.594036402s} {test-napc01-1/latency-deployment-9-77cd7fd7bd-mfwpw 2.595593866s} {test-napc01-1/latency-deployment-123-78fd649d7d-s75qb 2.597554338s} {test-napc01-1/latency-deployment-416-8cc956996-xrfb7 2.602318758s} {test-napc01-1/latency-deployment-417-785d8d659d-q9n9g 2.603678734s} {test-napc01-1/latency-deployment-222-65bd6695fd-trxx6 2.604608634s} {test-napc01-1/latency-deployment-73-559996b474-bk99p 2.60635284s} {test-napc01-1/latency-deployment-291-7b6b6b7f86-fh9rj 2.607796762s} {test-napc01-1/latency-deployment-168-5db67d8948-bzh7d 2.61607419s} {test-napc01-1/latency-deployment-302-7df57756dd-9vkwm 2.61643545s} {test-napc01-1/latency-deployment-223-86d8db68d-smswk 2.618349032s} {test-napc01-1/latency-deployment-67-59bbd9d874-pcztl 2.632974109s} {test-napc01-1/latency-deployment-37-55cc74746-2m4mg 2.633346387s} {test-napc01-1/latency-deployment-382-5bbb557fcb-r7c7w 2.633413032s} {test-napc01-1/latency-deployment-107-75d89f764b-g4xr2 2.636860824s} {test-napc01-1/latency-deployment-327-587974b6-58wgc 2.638537952s} {test-napc01-1/latency-deployment-368-6cfc7bfc6-5kz6g 2.640282876s} {test-napc01-1/latency-deployment-177-bd87cc5b8-kmrh8 2.642126957s} {test-napc01-1/latency-deployment-393-6f769b8596-tczjs 2.6596888s} {test-napc01-1/latency-deployment-227-6785f8b69b-hn76p 2.660115662s} {test-napc01-1/latency-deployment-39-85ff5f64db-zsdt8 2.660553881s} {test-napc01-1/latency-deployment-128-68cc4b9cf6-x4dmd 2.667835785s} {test-napc01-1/latency-deployment-283-555ffb857b-l69jw 2.676709086s} {test-napc01-1/latency-deployment-378-7fd5d9749d-7gwvr 2.678670847s} {test-napc01-1/latency-deployment-228-bfb654c86-hgfcn 2.68066706s} {test-napc01-1/latency-deployment-64-78b758949d-th4cs 2.686220908s} {test-napc01-1/latency-deployment-298-577c46ffcb-rtmxs 2.686223831s} {test-napc01-1/latency-deployment-193-78d75dbd84-d476k 2.689297015s} {test-napc01-1/latency-deployment-457-589dbcf75d-76fh4 2.697006691s} {test-napc01-1/latency-deployment-338-d5f979f7d-k6jt2 2.700346175s} {test-napc01-1/latency-deployment-341-844845d79d-knt6s 2.708135286s} {test-napc01-1/latency-deployment-143-76778ccfc8-4kmfh 2.709309521s} {test-napc01-1/latency-deployment-486-68bf6f68b6-c2qgz 2.712389428s} {test-napc01-1/latency-deployment-52-6569fbbc9d-cr4cj 2.713432179s} {test-napc01-1/latency-deployment-158-8466665d9d-q6rxv 2.724072831s} {test-napc01-1/latency-deployment-403-6446dc74cb-wmctz 2.725948596s} {test-napc01-1/latency-deployment-208-796965867d-splsq 2.735794472s} {test-napc01-1/latency-deployment-318-5884cbcfbb-t9jpn 2.738052405s} {test-napc01-1/latency-deployment-348-64ddd8798d-ntghx 2.75471882s} {test-napc01-1/latency-deployment-292-6db98f58b8-jrh4p 2.75857613s} {test-napc01-1/latency-deployment-413-d7b8d5cd6-kls6p 2.761646996s} {test-napc01-1/latency-deployment-19-5985c44c78-7vd5m 2.77136772s} {test-napc01-1/latency-deployment-173-dcb9f799b-zjg7d 2.782976611s} {test-napc01-1/latency-deployment-43-6497c74c4-w2dwx 2.784715021s} {test-napc01-1/latency-deployment-372-64b474946-xm978 2.785419692s} {test-napc01-1/latency-deployment-98-6d74f9bc46-fh8kz 2.797297975s} {test-napc01-1/latency-deployment-427-5744bf8576-872rk 2.80326736s} {test-napc01-1/latency-deployment-497-f8b6495d6-hm4dt 2.804699483s} {test-napc01-1/latency-deployment-322-bcfc7fff6-255c7 2.818280053s} {test-napc01-1/latency-deployment-263-c87d4f964-gvk2d 2.818679549s} {test-napc01-1/latency-deployment-343-56c98bd9fd-z62cl 2.825606974s} {test-napc01-1/latency-deployment-49-9c6bfbbf4-qh4qh 2.828910611s} {test-napc01-1/latency-deployment-18-857db5b6db-pwx2l 2.831653068s} {test-napc01-1/latency-deployment-103-7cdd466f86-cv4js 2.836261087s} {test-napc01-1/latency-deployment-153-6bcc9b8964-hggj5 2.83652846s} {test-napc01-1/latency-deployment-48-8b9f7fdf8-cbw87 2.838537476s} {test-napc01-1/latency-deployment-387-5b56b46cd-wr58q 2.840549286s} {test-napc01-1/latency-deployment-218-8648b79fcb-gm7p9 2.843713906s} {test-napc01-1/latency-deployment-29-bd5c8cf58-c22tg 2.843802022s} {test-napc01-1/latency-deployment-203-8559fbb84d-8mpml 2.860387696s} {test-napc01-1/latency-deployment-363-666c6c4bf8-rknrg 2.863549031s} {test-napc01-1/latency-deployment-308-d599c6c44-8rmb5 2.867279699s} {test-napc01-1/latency-deployment-288-6fdf979946-nlshm 2.886478928s} {test-napc01-1/latency-deployment-358-5c55b5647d-jlgcl 2.905011586s} {test-napc01-1/latency-deployment-487-6f4f987c4d-h97kt 2.90595812s} {test-napc01-1/latency-deployment-398-678dddfd66-tz7v8 2.934323397s} {test-napc01-1/latency-deployment-353-6d4658b996-tcwb7 2.943385092s} {test-napc01-1/latency-deployment-337-7f56cd7d4-mvss2 3.025909912s} {test-napc01-1/latency-deployment-268-6c58b69db6-2w765 3.037619879s} {test-napc01-1/latency-deployment-24-55484ddf7d-sc85m 3.046314605s}] ... skipping 591 lines ... I0302 16:15:59.759919 91183 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-napc01-1), controlledBy(medium-deployment-10): Pods: 44 out of 44 created, 44 running (44 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0302 16:15:59.868702 91183 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-napc01-1), controlledBy(small-deployment-26): Pods: 2 out of 2 created, 2 running (2 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0302 16:16:00.082049 91183 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-napc01-1), controlledBy(small-deployment-233): Pods: 7 out of 7 created, 7 running (7 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0302 16:16:00.097053 91183 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-napc01-1), controlledBy(small-deployment-277): Pods: 4 out of 4 created, 4 running (4 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0302 16:16:00.208104 91183 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-napc01-1), controlledBy(small-deployment-179): Pods: 7 out of 7 created, 7 running (7 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0302 16:16:00.240942 91183 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-napc01-1), controlledBy(small-deployment-112): Pods: 6 out of 6 created, 6 running (6 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0302 16:16:00.291941 91183 wait_for_controlled_pods.go:365] WaitForControlledPodsRunning: running 3, deleted 0, timeout: 0, failed: 0 I0302 16:16:00.291986 91183 wait_for_controlled_pods.go:365] WaitForControlledPodsRunning: running 2, deleted 0, timeout: 0, failed: 0 I0302 16:16:00.292019 91183 wait_for_controlled_pods.go:370] WaitForControlledPodsRunning: maxDuration=20.004331206s, operationTimeout=15m0s, ratio=0.02 I0302 16:16:00.291989 91183 wait_for_controlled_pods.go:370] WaitForControlledPodsRunning: maxDuration=5.003965022s, operationTimeout=15m0s, ratio=0.01 I0302 16:16:00.292074 91183 wait_for_controlled_pods.go:384] WaitForControlledPodsRunning: 3/3 Jobs are running with all pods I0302 16:16:00.292081 91183 wait_for_controlled_pods.go:384] WaitForControlledPodsRunning: 2/2 StatefulSets are running with all pods I0302 16:16:00.457987 91183 wait_for_controlled_pods.go:365] WaitForControlledPodsRunning: running 323, deleted 0, timeout: 0, failed: 0 I0302 16:16:00.458038 91183 wait_for_controlled_pods.go:370] WaitForControlledPodsRunning: maxDuration=10.002440038s, operationTimeout=15m0s, ratio=0.01 I0302 16:16:00.458092 91183 wait_for_controlled_pods.go:384] WaitForControlledPodsRunning: 323/323 Deployments are running with all pods I0302 16:16:03.680583 91183 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-napc01-1), controlledBy(daemonset-0): Pods: 98 out of 100 created, 90 running (62 updated), 8 pending scheduled, 0 not scheduled, 0 inactive, 2 terminating, 0 unknown, 0 runningButNotReady I0302 16:16:08.687072 91183 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-napc01-1), controlledBy(daemonset-0): Pods: 96 out of 100 created, 90 running (82 updated), 6 pending scheduled, 0 not scheduled, 0 inactive, 4 terminating, 0 unknown, 0 runningButNotReady I0302 16:16:13.689329 91183 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-napc01-1), controlledBy(daemonset-0): Pods: 100 out of 100 created, 99 running (99 updated), 1 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0302 16:16:18.691689 91183 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-napc01-1), controlledBy(daemonset-0): Pods: 100 out of 100 created, 100 running (100 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0302 16:16:18.691759 91183 wait_for_controlled_pods.go:365] WaitForControlledPodsRunning: running 1, deleted 0, timeout: 0, failed: 0 I0302 16:16:18.691772 91183 wait_for_controlled_pods.go:370] WaitForControlledPodsRunning: maxDuration=35.115432397s, operationTimeout=15m0s, ratio=0.04 I0302 16:16:18.691788 91183 wait_for_controlled_pods.go:384] WaitForControlledPodsRunning: 1/1 DaemonSets are running with all pods I0302 16:16:18.691805 91183 simple_test_executor.go:171] Step "[step: 22] Waiting for 'scale and update objects' to be completed" ended I0302 16:16:18.691823 91183 simple_test_executor.go:149] Step "[step: 23] Starting measurement for 'delete objects'" started I0302 16:16:18.691868 91183 wait_for_controlled_pods.go:254] WaitForControlledPodsRunning: wait for controlled pods measurement already running I0302 16:16:18.691959 91183 wait_for_controlled_pods.go:254] WaitForControlledPodsRunning: wait for controlled pods measurement already running ... skipping 337 lines ... I0302 16:19:51.943365 91183 simple_test_executor.go:149] Step "[step: 25] Waiting for 'delete objects' to be completed" started I0302 16:19:51.943448 91183 wait_for_pods.go:64] WaitForControlledPodsRunning: namespace(test-napc01-1), controlledBy(small-deployment-190): starting with timeout: 14m59.999984387s I0302 16:19:51.943442 91183 wait_for_controlled_pods.go:288] WaitForControlledPodsRunning: waiting for controlled pods measurement... I0302 16:19:51.943466 91183 wait_for_controlled_pods.go:288] WaitForControlledPodsRunning: waiting for controlled pods measurement... I0302 16:19:51.943483 91183 wait_for_controlled_pods.go:288] WaitForControlledPodsRunning: waiting for controlled pods measurement... I0302 16:19:51.943561 91183 wait_for_controlled_pods.go:288] WaitForControlledPodsRunning: waiting for controlled pods measurement... I0302 16:19:56.980335 91183 wait_for_controlled_pods.go:365] WaitForControlledPodsRunning: running 0, deleted 1, timeout: 0, failed: 0 I0302 16:19:56.980373 91183 wait_for_controlled_pods.go:365] WaitForControlledPodsRunning: running 0, deleted 3, timeout: 0, failed: 0 I0302 16:19:56.980398 91183 wait_for_controlled_pods.go:370] WaitForControlledPodsRunning: maxDuration=5.000515679s, operationTimeout=15m0s, ratio=0.01 I0302 16:19:56.980376 91183 wait_for_controlled_pods.go:370] WaitForControlledPodsRunning: maxDuration=5.000398833s, operationTimeout=15m0s, ratio=0.01 I0302 16:19:56.980417 91183 wait_for_controlled_pods.go:384] WaitForControlledPodsRunning: 0/0 Jobs are running with all pods I0302 16:19:56.980418 91183 wait_for_controlled_pods.go:384] WaitForControlledPodsRunning: 0/0 DaemonSets are running with all pods I0302 16:19:56.980348 91183 wait_for_controlled_pods.go:365] WaitForControlledPodsRunning: running 0, deleted 2, timeout: 0, failed: 0 I0302 16:19:56.980437 91183 wait_for_controlled_pods.go:370] WaitForControlledPodsRunning: maxDuration=5.001025434s, operationTimeout=15m0s, ratio=0.01 I0302 16:19:56.980448 91183 wait_for_controlled_pods.go:384] WaitForControlledPodsRunning: 0/0 StatefulSets are running with all pods I0302 16:19:56.981663 91183 wait_for_controlled_pods.go:365] WaitForControlledPodsRunning: running 0, deleted 323, timeout: 0, failed: 0 I0302 16:19:56.981683 91183 wait_for_controlled_pods.go:370] WaitForControlledPodsRunning: maxDuration=10.002239749s, operationTimeout=15m0s, ratio=0.01 I0302 16:19:56.981695 91183 wait_for_controlled_pods.go:384] WaitForControlledPodsRunning: 0/0 Deployments are running with all pods I0302 16:19:56.993905 91183 wait_for_pvcs.go:78] WaitForBoundPVCs: labelSelector(group = load): PVCs: 0 out of 0 created, 0 bound, 0 pending, 0 lost I0302 16:19:56.993968 91183 simple_test_executor.go:171] Step "[step: 25] Waiting for 'delete objects' to be completed" ended I0302 16:19:56.993989 91183 simple_test_executor.go:149] Step "[step: 26] delete objects configmaps and secrets" started I0302 16:20:29.268920 91183 simple_test_executor.go:171] Step "[step: 26] delete objects configmaps and secrets" ended ... skipping 251 lines ... I0302 16:21:44.853252 91183 simple_test_executor.go:171] Step "[step: 04] Creating huge-service pods" ended I0302 16:21:44.853268 91183 simple_test_executor.go:149] Step "[step: 05] Waiting for huge-service pods to be created" started I0302 16:21:44.853290 91183 wait_for_controlled_pods.go:288] WaitForControlledPodsRunning: waiting for controlled pods measurement... I0302 16:21:49.860595 91183 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-w4hje8-1), controlledBy(huge-service-deployment-0): Pods: 538 out of 1000 created, 385 running (385 updated), 132 pending scheduled, 21 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0302 16:21:54.875695 91183 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-w4hje8-1), controlledBy(huge-service-deployment-0): Pods: 1000 out of 1000 created, 889 running (889 updated), 111 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0302 16:21:59.891628 91183 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-w4hje8-1), controlledBy(huge-service-deployment-0): Pods: 1000 out of 1000 created, 1000 running (1000 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0302 16:21:59.891725 91183 wait_for_controlled_pods.go:365] WaitForControlledPodsRunning: running 1, deleted 0, timeout: 0, failed: 0 I0302 16:21:59.891744 91183 wait_for_controlled_pods.go:370] WaitForControlledPodsRunning: maxDuration=15.038538579s, operationTimeout=30m0s, ratio=0.01 I0302 16:21:59.891763 91183 wait_for_controlled_pods.go:384] WaitForControlledPodsRunning: 1/1 Deployments are running with all pods I0302 16:21:59.891793 91183 simple_test_executor.go:171] Step "[step: 05] Waiting for huge-service pods to be created" ended I0302 16:21:59.891816 91183 simple_test_executor.go:149] Step "[step: 06] Updating huge-service pods" started I0302 16:21:59.968378 91183 simple_test_executor.go:171] Step "[step: 06] Updating huge-service pods" ended I0302 16:21:59.968388 91183 wait_for_pods.go:64] WaitForControlledPodsRunning: namespace(test-w4hje8-1), controlledBy(huge-service-deployment-0): starting with timeout: 29m59.999985498s ... skipping 4 lines ... I0302 16:22:10.533174 91183 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-w4hje8-1), controlledBy(huge-service-deployment-0): Pods: 1051 out of 1000 created, 1040 running (558 updated), 11 pending scheduled, 0 not scheduled, 0 inactive, 156 terminating, 0 unknown, 0 runningButNotReady W0302 16:22:15.666962 91183 wait_for_pods.go:119] WaitForControlledPodsRunning: namespace(test-w4hje8-1), controlledBy(huge-service-deployment-0): 291 pods appeared: huge-service-deployment-0-7945dfc44-plp6x, huge-service-deployment-0-7945dfc44-cbhc6, huge-service-deployment-0-7945dfc44-mxltm, huge-service-deployment-0-7945dfc44-9p66w, huge-service-deployment-0-7945dfc44-r584j, huge-service-deployment-0-7945dfc44-2d9sg, huge-service-deployment-0-7945dfc44-5fjbf, huge-service-deployment-0-7945dfc44-98j65, huge-service-deployment-0-7945dfc44-r9986, huge-service-deployment-0-7945dfc44-8xtbm, huge-service-deployment-0-7945dfc44-p9qfc, huge-service-deployment-0-7945dfc44-2rzbw, huge-service-deployment-0-7945dfc44-gft6t, huge-service-deployment-0-7945dfc44-bk4p8, huge-service-deployment-0-7945dfc44-rdd8z, huge-service-deployment-0-7945dfc44-2tr58, huge-service-deployment-0-7945dfc44-697j9, huge-service-deployment-0-7945dfc44-fvmfp, huge-service-deployment-0-7945dfc44-vcc77, huge-service-deployment-0-7945dfc44-2jtx5, huge-service-deployment-0-7945dfc44-t759t, huge-service-deployment-0-7945dfc44-p8vhj, huge-service-deployment-0-7945dfc44-hh6l7, huge-service-deployment-0-7945dfc44-mvsn9, huge-service-deployment-0-7945dfc44-d62dz, huge-service-deployment-0-7945dfc44-dvnkf, huge-service-deployment-0-7945dfc44-bcjln, huge-service-deployment-0-7945dfc44-45rsb, huge-service-deployment-0-7945dfc44-8grjc, huge-service-deployment-0-7945dfc44-fl28x, huge-service-deployment-0-7945dfc44-mm7bt, huge-service-deployment-0-7945dfc44-z8nv6, huge-service-deployment-0-7945dfc44-chl9w, huge-service-deployment-0-7945dfc44-2sdmf, huge-service-deployment-0-7945dfc44-vd22x, huge-service-deployment-0-7945dfc44-xkwzw, huge-service-deployment-0-7945dfc44-qjk54, huge-service-deployment-0-7945dfc44-2dwrp, huge-service-deployment-0-7945dfc44-nwq4g, huge-service-deployment-0-7945dfc44-t97lk, huge-service-deployment-0-7945dfc44-f88fg, huge-service-deployment-0-7945dfc44-jk4vp, huge-service-deployment-0-7945dfc44-n4frj, huge-service-deployment-0-7945dfc44-72qjg, huge-service-deployment-0-7945dfc44-bbqnq, huge-service-deployment-0-7945dfc44-qrfrr, huge-service-deployment-0-7945dfc44-wndwv, huge-service-deployment-0-7945dfc44-n5lqq, huge-service-deployment-0-7945dfc44-6kgwt, huge-service-deployment-0-7945dfc44-jbw2b, huge-service-deployment-0-7945dfc44-pdpd5, huge-service-deployment-0-7945dfc44-zgq8z, huge-service-deployment-0-7945dfc44-8jhlk, huge-service-deployment-0-7945dfc44-5wszb, huge-service-deployment-0-7945dfc44-vtwhk, huge-service-deployment-0-7945dfc44-nqjbz, huge-service-deployment-0-7945dfc44-99kwc, huge-service-deployment-0-7945dfc44-77rbq, huge-service-deployment-0-7945dfc44-zbljq, huge-service-deployment-0-7945dfc44-7gxh4, huge-service-deployment-0-7945dfc44-xtfvp, huge-service-deployment-0-7945dfc44-cl7sh, huge-service-deployment-0-7945dfc44-lggp8, huge-service-deployment-0-7945dfc44-hncff, huge-service-deployment-0-7945dfc44-zjzlj, huge-service-deployment-0-7945dfc44-brxbr, huge-service-deployment-0-7945dfc44-xmzd7, huge-service-deployment-0-7945dfc44-ckch6, huge-service-deployment-0-7945dfc44-dxxx7, huge-service-deployment-0-7945dfc44-kvvqd, huge-service-deployment-0-7945dfc44-pwkx6, huge-service-deployment-0-7945dfc44-6hnbq, huge-service-deployment-0-7945dfc44-h4fbk, huge-service-deployment-0-7945dfc44-8q2lf, huge-service-deployment-0-7945dfc44-dtgw7, huge-service-deployment-0-7945dfc44-nphtl, huge-service-deployment-0-7945dfc44-t8h7v, huge-service-deployment-0-7945dfc44-rz5w9, huge-service-deployment-0-7945dfc44-slm6q, huge-service-deployment-0-7945dfc44-wldsw, huge-service-deployment-0-7945dfc44-jz4d7, huge-service-deployment-0-7945dfc44-nwsx5, huge-service-deployment-0-7945dfc44-kdrz6, huge-service-deployment-0-7945dfc44-9gc64, huge-service-deployment-0-7945dfc44-7qvvl, huge-service-deployment-0-7945dfc44-t9sc5, huge-service-deployment-0-7945dfc44-mxqj5, huge-service-deployment-0-7945dfc44-bmzbr, huge-service-deployment-0-7945dfc44-cdx45, huge-service-deployment-0-7945dfc44-frg4h, huge-service-deployment-0-7945dfc44-vdmvn, huge-service-deployment-0-7945dfc44-qvdmk, huge-service-deployment-0-7945dfc44-xr682, huge-service-deployment-0-7945dfc44-cksrs, huge-service-deployment-0-7945dfc44-m66hn, huge-service-deployment-0-7945dfc44-lclc8, huge-service-deployment-0-7945dfc44-xtfhm, huge-service-deployment-0-7945dfc44-fs6n4, huge-service-deployment-0-7945dfc44-szsx2, huge-service-deployment-0-7945dfc44-gfvlv, huge-service-deployment-0-7945dfc44-64zq2, huge-service-deployment-0-7945dfc44-lwqdt, huge-service-deployment-0-7945dfc44-4xqxh, huge-service-deployment-0-7945dfc44-wjgq6, huge-service-deployment-0-7945dfc44-b2l9z, huge-service-deployment-0-7945dfc44-25xd9, huge-service-deployment-0-7945dfc44-dv4ks, huge-service-deployment-0-7945dfc44-s2bvp, huge-service-deployment-0-7945dfc44-jtg5c, huge-service-deployment-0-7945dfc44-h2mrr, huge-service-deployment-0-7945dfc44-frbt7, huge-service-deployment-0-7945dfc44-6pjhf, huge-service-deployment-0-7945dfc44-gtvbv, huge-service-deployment-0-7945dfc44-kkrnb, huge-service-deployment-0-7945dfc44-p5vsn, huge-service-deployment-0-7945dfc44-f99k7, huge-service-deployment-0-7945dfc44-68fzc, huge-service-deployment-0-7945dfc44-mckxt, huge-service-deployment-0-7945dfc44-zhf6j, huge-service-deployment-0-7945dfc44-l6kd2, huge-service-deployment-0-7945dfc44-8mg5w, huge-service-deployment-0-7945dfc44-v2npw, huge-service-deployment-0-7945dfc44-rqmv2, huge-service-deployment-0-7945dfc44-q6nxk, huge-service-deployment-0-7945dfc44-xqs6v, huge-service-deployment-0-7945dfc44-7fjs4, huge-service-deployment-0-7945dfc44-p4tbg, huge-service-deployment-0-7945dfc44-dmrpc, huge-service-deployment-0-7945dfc44-qb9kl, huge-service-deployment-0-7945dfc44-6txvw, huge-service-deployment-0-7945dfc44-rc6rt, huge-service-deployment-0-7945dfc44-m64cz, huge-service-deployment-0-7945dfc44-fcwmr, huge-service-deployment-0-7945dfc44-vjlch, huge-service-deployment-0-7945dfc44-g42m6, huge-service-deployment-0-7945dfc44-kwlbk, huge-service-deployment-0-7945dfc44-xqwmt, huge-service-deployment-0-7945dfc44-4wcpg, huge-service-deployment-0-7945dfc44-268g2, huge-service-deployment-0-7945dfc44-ksxb6, huge-service-deployment-0-7945dfc44-x79vs, huge-service-deployment-0-7945dfc44-n7lx6, huge-service-deployment-0-7945dfc44-2qhh8, huge-service-deployment-0-7945dfc44-t75nx, huge-service-deployment-0-7945dfc44-qhdts, huge-service-deployment-0-7945dfc44-pklv9, huge-service-deployment-0-7945dfc44-x5kc2, huge-service-deployment-0-7945dfc44-nn2ml, huge-service-deployment-0-7945dfc44-58sv5, huge-service-deployment-0-7945dfc44-tj4bk, huge-service-deployment-0-7945dfc44-pt52q, huge-service-deployment-0-7945dfc44-9pmk4, huge-service-deployment-0-7945dfc44-rkkxz, huge-service-deployment-0-7945dfc44-7qp6c, huge-service-deployment-0-7945dfc44-wtx5l, huge-service-deployment-0-7945dfc44-lb9hr, huge-service-deployment-0-7945dfc44-v7q9s, huge-service-deployment-0-7945dfc44-9k7r8, huge-service-deployment-0-7945dfc44-vkk2n, huge-service-deployment-0-7945dfc44-znpc6, huge-service-deployment-0-7945dfc44-jx2xs, huge-service-deployment-0-7945dfc44-4wqwh, huge-service-deployment-0-7945dfc44-dtw22, huge-service-deployment-0-7945dfc44-rb4np, huge-service-deployment-0-7945dfc44-mv4ts, huge-service-deployment-0-7945dfc44-mfzks, huge-service-deployment-0-7945dfc44-vx6g7, huge-service-deployment-0-7945dfc44-zjhs4, huge-service-deployment-0-7945dfc44-tkrgz, huge-service-deployment-0-7945dfc44-ktz2c, huge-service-deployment-0-7945dfc44-hg9gz, huge-service-deployment-0-7945dfc44-94h4z, huge-service-deployment-0-7945dfc44-g2chm, huge-service-deployment-0-7945dfc44-t87nt, huge-service-deployment-0-7945dfc44-7z9cn, huge-service-deployment-0-7945dfc44-2h9md, huge-service-deployment-0-7945dfc44-sd298, huge-service-deployment-0-7945dfc44-n58dr, huge-service-deployment-0-7945dfc44-hmcsq, huge-service-deployment-0-7945dfc44-xwx6q, huge-service-deployment-0-7945dfc44-2dm2t, huge-service-deployment-0-7945dfc44-rtjlh, huge-service-deployment-0-7945dfc44-ngwzk, huge-service-deployment-0-7945dfc44-xppt2, huge-service-deployment-0-7945dfc44-wmn4b, huge-service-deployment-0-7945dfc44-vkwsw, huge-service-deployment-0-7945dfc44-s42n6, huge-service-deployment-0-7945dfc44-mpckr, huge-service-deployment-0-7945dfc44-tqb56, huge-service-deployment-0-7945dfc44-wkwxj, huge-service-deployment-0-7945dfc44-ncbw9, huge-service-deployment-0-7945dfc44-psxlc, huge-service-deployment-0-7945dfc44-bgfxd, huge-service-deployment-0-7945dfc44-75mhg, huge-service-deployment-0-7945dfc44-hb94n, huge-service-deployment-0-7945dfc44-2bsqh, huge-service-deployment-0-7945dfc44-dllxw, huge-service-deployment-0-7945dfc44-xmlv4, huge-service-deployment-0-7945dfc44-94mp7, huge-service-deployment-0-7945dfc44-7kph7, huge-service-deployment-0-7945dfc44-fxsmq, huge-service-deployment-0-7945dfc44-8ztm9, huge-service-deployment-0-7945dfc44-xqkhg, huge-service-deployment-0-7945dfc44-jzndm, huge-service-deployment-0-7945dfc44-jbclr, huge-service-deployment-0-7945dfc44-5h5k8, huge-service-deployment-0-7945dfc44-zb5kb, huge-service-deployment-0-7945dfc44-s8kvc, huge-service-deployment-0-7945dfc44-c5v9c, huge-service-deployment-0-7945dfc44-ztnzk, huge-service-deployment-0-7945dfc44-d7c9t, huge-service-deployment-0-7945dfc44-j8zbg, huge-service-deployment-0-7945dfc44-knwcg, huge-service-deployment-0-7945dfc44-86bcp, huge-service-deployment-0-7945dfc44-h2mh9, huge-service-deployment-0-7945dfc44-2jznx, huge-service-deployment-0-7945dfc44-76mw4, huge-service-deployment-0-7945dfc44-k5fhb, huge-service-deployment-0-7945dfc44-74wlj, huge-service-deployment-0-7945dfc44-dvs95, huge-service-deployment-0-7945dfc44-bbhlh, huge-service-deployment-0-7945dfc44-9n7vk, huge-service-deployment-0-7945dfc44-7xm5m, huge-service-deployment-0-7945dfc44-kprr9, huge-service-deployment-0-7945dfc44-zm5rt, huge-service-deployment-0-7945dfc44-p8jgz, huge-service-deployment-0-7945dfc44-k68m2, huge-service-deployment-0-7945dfc44-zwd92, huge-service-deployment-0-7945dfc44-9gjxb, huge-service-deployment-0-7945dfc44-2z6tw, huge-service-deployment-0-7945dfc44-wwrb8, huge-service-deployment-0-7945dfc44-xtpzq, huge-service-deployment-0-7945dfc44-t5sn9, huge-service-deployment-0-7945dfc44-rrqx6, huge-service-deployment-0-7945dfc44-jpvcj, huge-service-deployment-0-7945dfc44-zlh5f, huge-service-deployment-0-7945dfc44-znpmk, huge-service-deployment-0-7945dfc44-9fcfz, huge-service-deployment-0-7945dfc44-57f7q, huge-service-deployment-0-7945dfc44-b8j4b, huge-service-deployment-0-7945dfc44-c25c5, huge-service-deployment-0-7945dfc44-v748q, huge-service-deployment-0-7945dfc44-ht8ng, huge-service-deployment-0-7945dfc44-cvvbt, huge-service-deployment-0-7945dfc44-vj2zb, huge-service-deployment-0-7945dfc44-kcq8h, huge-service-deployment-0-7945dfc44-ldqw2, huge-service-deployment-0-7945dfc44-w8dtt, huge-service-deployment-0-7945dfc44-lf75p, huge-service-deployment-0-7945dfc44-qcnpm, huge-service-deployment-0-7945dfc44-s94k9, huge-service-deployment-0-7945dfc44-5x6s6, huge-service-deployment-0-7945dfc44-5lldx, huge-service-deployment-0-7945dfc44-fvc4w, huge-service-deployment-0-7945dfc44-5bplr, huge-service-deployment-0-7945dfc44-wn6ld, huge-service-deployment-0-7945dfc44-x4ctb, huge-service-deployment-0-7945dfc44-xm7m4, huge-service-deployment-0-7945dfc44-r5g4s, huge-service-deployment-0-7945dfc44-qwv5w, huge-service-deployment-0-7945dfc44-8f2wr, huge-service-deployment-0-7945dfc44-gh6cs, huge-service-deployment-0-7945dfc44-zbpg7, huge-service-deployment-0-7945dfc44-8vbj7, huge-service-deployment-0-7945dfc44-sgtzd, huge-service-deployment-0-7945dfc44-flhzj, huge-service-deployment-0-7945dfc44-glb6w, huge-service-deployment-0-7945dfc44-5jxhh, huge-service-deployment-0-7945dfc44-rxp8x, huge-service-deployment-0-7945dfc44-tgz4c, huge-service-deployment-0-7945dfc44-sdhbz, huge-service-deployment-0-7945dfc44-msdww, huge-service-deployment-0-7945dfc44-n854j, huge-service-deployment-0-7945dfc44-259hz, huge-service-deployment-0-7945dfc44-kwbw4, huge-service-deployment-0-7945dfc44-v2wt7, huge-service-deployment-0-7945dfc44-8tml5, huge-service-deployment-0-7945dfc44-c9n6b, huge-service-deployment-0-7945dfc44-fxscm, huge-service-deployment-0-7945dfc44-r6gch, huge-service-deployment-0-7945dfc44-mmnlv, huge-service-deployment-0-7945dfc44-t6bfs, huge-service-deployment-0-7945dfc44-zn6bc, huge-service-deployment-0-7945dfc44-9pmfr, huge-service-deployment-0-7945dfc44-hrtj5, huge-service-deployment-0-7945dfc44-hnsh6, huge-service-deployment-0-7945dfc44-9prnk, huge-service-deployment-0-7945dfc44-mrfds, huge-service-deployment-0-7945dfc44-pzwh6, huge-service-deployment-0-7945dfc44-77zh9, huge-service-deployment-0-7945dfc44-5sh2j I0302 16:22:15.667027 91183 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-w4hje8-1), controlledBy(huge-service-deployment-0): Pods: 1120 out of 1000 created, 991 running (731 updated), 129 pending scheduled, 0 not scheduled, 0 inactive, 78 terminating, 0 unknown, 0 runningButNotReady W0302 16:22:20.684867 91183 wait_for_pods.go:119] WaitForControlledPodsRunning: namespace(test-w4hje8-1), controlledBy(huge-service-deployment-0): 140 pods appeared: huge-service-deployment-0-7945dfc44-mv2jb, huge-service-deployment-0-7945dfc44-mjcbc, huge-service-deployment-0-7945dfc44-6h29k, huge-service-deployment-0-7945dfc44-x68qr, huge-service-deployment-0-7945dfc44-zvdmt, huge-service-deployment-0-7945dfc44-kfxtr, huge-service-deployment-0-7945dfc44-82dn5, huge-service-deployment-0-7945dfc44-np7dz, huge-service-deployment-0-7945dfc44-sl2rb, huge-service-deployment-0-7945dfc44-rj9cl, huge-service-deployment-0-7945dfc44-qmfkz, huge-service-deployment-0-7945dfc44-7688n, huge-service-deployment-0-7945dfc44-lnzn4, huge-service-deployment-0-7945dfc44-fdrvb, huge-service-deployment-0-7945dfc44-5269v, huge-service-deployment-0-7945dfc44-w6bbn, huge-service-deployment-0-7945dfc44-q4zzh, huge-service-deployment-0-7945dfc44-h2c8l, huge-service-deployment-0-7945dfc44-jjbcx, huge-service-deployment-0-7945dfc44-d7wgp, huge-service-deployment-0-7945dfc44-6dzxs, huge-service-deployment-0-7945dfc44-lx9bl, huge-service-deployment-0-7945dfc44-hxnt7, huge-service-deployment-0-7945dfc44-xzm62, huge-service-deployment-0-7945dfc44-khwhb, huge-service-deployment-0-7945dfc44-ds78f, huge-service-deployment-0-7945dfc44-lkf8m, huge-service-deployment-0-7945dfc44-4cvlz, huge-service-deployment-0-7945dfc44-bflcj, huge-service-deployment-0-7945dfc44-nqmz9, huge-service-deployment-0-7945dfc44-8wcms, huge-service-deployment-0-7945dfc44-qjrxk, huge-service-deployment-0-7945dfc44-v9jnz, huge-service-deployment-0-7945dfc44-hdr2q, huge-service-deployment-0-7945dfc44-wvrgb, huge-service-deployment-0-7945dfc44-xfz7r, huge-service-deployment-0-7945dfc44-9lz6t, huge-service-deployment-0-7945dfc44-w2xrx, huge-service-deployment-0-7945dfc44-k8r2k, huge-service-deployment-0-7945dfc44-k9trn, huge-service-deployment-0-7945dfc44-bfxbx, huge-service-deployment-0-7945dfc44-d4xsh, huge-service-deployment-0-7945dfc44-kxqhc, huge-service-deployment-0-7945dfc44-mg8hg, huge-service-deployment-0-7945dfc44-5nxpq, huge-service-deployment-0-7945dfc44-4gwl9, huge-service-deployment-0-7945dfc44-fwnww, huge-service-deployment-0-7945dfc44-wfzmx, huge-service-deployment-0-7945dfc44-mcwt2, huge-service-deployment-0-7945dfc44-n7hdn, huge-service-deployment-0-7945dfc44-qhvtl, huge-service-deployment-0-7945dfc44-d6mds, huge-service-deployment-0-7945dfc44-wdvq5, huge-service-deployment-0-7945dfc44-dvvgl, huge-service-deployment-0-7945dfc44-d82g9, huge-service-deployment-0-7945dfc44-l5kls, huge-service-deployment-0-7945dfc44-rlk25, huge-service-deployment-0-7945dfc44-wdk2w, huge-service-deployment-0-7945dfc44-2wr2w, huge-service-deployment-0-7945dfc44-2xkz4, huge-service-deployment-0-7945dfc44-xl7tf, huge-service-deployment-0-7945dfc44-ltkcg, huge-service-deployment-0-7945dfc44-z76t9, huge-service-deployment-0-7945dfc44-h8mss, huge-service-deployment-0-7945dfc44-9gvv9, huge-service-deployment-0-7945dfc44-vhlh2, huge-service-deployment-0-7945dfc44-x4jhz, huge-service-deployment-0-7945dfc44-r8bmk, huge-service-deployment-0-7945dfc44-xjg5p, huge-service-deployment-0-7945dfc44-rb87k, huge-service-deployment-0-7945dfc44-6bpms, huge-service-deployment-0-7945dfc44-gcrxr, huge-service-deployment-0-7945dfc44-7tl6c, huge-service-deployment-0-7945dfc44-74hhd, huge-service-deployment-0-7945dfc44-7dtsd, huge-service-deployment-0-7945dfc44-fw8wx, huge-service-deployment-0-7945dfc44-q8clk, huge-service-deployment-0-7945dfc44-x6v5z, huge-service-deployment-0-7945dfc44-h9cfg, huge-service-deployment-0-7945dfc44-7vtqj, huge-service-deployment-0-7945dfc44-7cjfn, huge-service-deployment-0-7945dfc44-576l7, huge-service-deployment-0-7945dfc44-fxs5p, huge-service-deployment-0-7945dfc44-4f7r2, huge-service-deployment-0-7945dfc44-zhssc, huge-service-deployment-0-7945dfc44-8r6jf, huge-service-deployment-0-7945dfc44-phnrh, huge-service-deployment-0-7945dfc44-t4zqz, huge-service-deployment-0-7945dfc44-jzd5f, huge-service-deployment-0-7945dfc44-d8tzf, huge-service-deployment-0-7945dfc44-zfm4n, huge-service-deployment-0-7945dfc44-4bqvq, huge-service-deployment-0-7945dfc44-q2xn4, huge-service-deployment-0-7945dfc44-x9fl6, huge-service-deployment-0-7945dfc44-gt44n, huge-service-deployment-0-7945dfc44-tmg4z, huge-service-deployment-0-7945dfc44-ngz56, huge-service-deployment-0-7945dfc44-vqhlt, huge-service-deployment-0-7945dfc44-n7g44, huge-service-deployment-0-7945dfc44-bvwrr, huge-service-deployment-0-7945dfc44-cdqns, huge-service-deployment-0-7945dfc44-95wzk, huge-service-deployment-0-7945dfc44-jpg2l, huge-service-deployment-0-7945dfc44-95p8t, huge-service-deployment-0-7945dfc44-72cbg, huge-service-deployment-0-7945dfc44-57gpc, huge-service-deployment-0-7945dfc44-88fzn, huge-service-deployment-0-7945dfc44-sm5t4, huge-service-deployment-0-7945dfc44-852kv, huge-service-deployment-0-7945dfc44-dc9qv, huge-service-deployment-0-7945dfc44-jpfl4, huge-service-deployment-0-7945dfc44-g88cf, huge-service-deployment-0-7945dfc44-vdbxj, huge-service-deployment-0-7945dfc44-xs8bd, huge-service-deployment-0-7945dfc44-rr5kx, huge-service-deployment-0-7945dfc44-cdgv8, huge-service-deployment-0-7945dfc44-httc4, huge-service-deployment-0-7945dfc44-6ckkx, huge-service-deployment-0-7945dfc44-w27dc, huge-service-deployment-0-7945dfc44-cpxvc, huge-service-deployment-0-7945dfc44-r5wqc, huge-service-deployment-0-7945dfc44-k7jnt, huge-service-deployment-0-7945dfc44-ngrtx, huge-service-deployment-0-7945dfc44-ld6g8, huge-service-deployment-0-7945dfc44-tjdfh, huge-service-deployment-0-7945dfc44-dt5hl, huge-service-deployment-0-7945dfc44-gntjz, huge-service-deployment-0-7945dfc44-92v76, huge-service-deployment-0-7945dfc44-5lmbf, huge-service-deployment-0-7945dfc44-cp9vf, huge-service-deployment-0-7945dfc44-hnfxm, huge-service-deployment-0-7945dfc44-np495, huge-service-deployment-0-7945dfc44-sh42t, huge-service-deployment-0-7945dfc44-8nxbg, huge-service-deployment-0-7945dfc44-vmd8q, huge-service-deployment-0-7945dfc44-kmztl, huge-service-deployment-0-7945dfc44-krnw4, huge-service-deployment-0-7945dfc44-pczt4, huge-service-deployment-0-7945dfc44-vjmkw, huge-service-deployment-0-7945dfc44-j924b I0302 16:22:20.684941 91183 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-w4hje8-1), controlledBy(huge-service-deployment-0): Pods: 1000 out of 1000 created, 977 running (977 updated), 23 pending scheduled, 0 not scheduled, 0 inactive, 28 terminating, 0 unknown, 0 runningButNotReady I0302 16:22:25.701248 91183 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-w4hje8-1), controlledBy(huge-service-deployment-0): Pods: 1000 out of 1000 created, 1000 running (1000 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0302 16:22:25.701322 91183 wait_for_controlled_pods.go:365] WaitForControlledPodsRunning: running 1, deleted 0, timeout: 0, failed: 0 I0302 16:22:25.701337 91183 wait_for_controlled_pods.go:370] WaitForControlledPodsRunning: maxDuration=25.732902877s, operationTimeout=30m0s, ratio=0.01 I0302 16:22:25.701352 91183 wait_for_controlled_pods.go:384] WaitForControlledPodsRunning: 1/1 Deployments are running with all pods I0302 16:22:25.701374 91183 simple_test_executor.go:171] Step "[step: 07] Waiting for huge-service pods to be updated" ended I0302 16:22:25.701393 91183 simple_test_executor.go:149] Step "[step: 08] Deleting huge-service pods" started I0302 16:22:25.740144 91183 simple_test_executor.go:171] Step "[step: 08] Deleting huge-service pods" ended I0302 16:22:25.740165 91183 wait_for_pods.go:64] WaitForControlledPodsRunning: namespace(test-w4hje8-1), controlledBy(huge-service-deployment-0): starting with timeout: 29m59.999981086s I0302 16:22:25.740171 91183 simple_test_executor.go:149] Step "[step: 09] Waiting for huge-service pods to be deleted" started I0302 16:22:25.740208 91183 wait_for_controlled_pods.go:288] WaitForControlledPodsRunning: waiting for controlled pods measurement... I0302 16:22:30.749085 91183 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-w4hje8-1), controlledBy(huge-service-deployment-0): Pods: 463 out of 0 created, 463 running (463 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 95 terminating, 0 unknown, 0 runningButNotReady I0302 16:22:35.750479 91183 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-w4hje8-1), controlledBy(huge-service-deployment-0): Pods: 0 out of 0 created, 0 running (0 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 55 terminating, 0 unknown, 0 runningButNotReady I0302 16:22:40.751034 91183 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-w4hje8-1), controlledBy(huge-service-deployment-0): Pods: 0 out of 0 created, 0 running (0 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0302 16:22:40.751105 91183 wait_for_controlled_pods.go:365] WaitForControlledPodsRunning: running 0, deleted 1, timeout: 0, failed: 0 I0302 16:22:40.751121 91183 wait_for_controlled_pods.go:370] WaitForControlledPodsRunning: maxDuration=15.010908897s, operationTimeout=30m0s, ratio=0.01 I0302 16:22:40.751140 91183 wait_for_controlled_pods.go:384] WaitForControlledPodsRunning: 0/0 Deployments are running with all pods I0302 16:22:40.751157 91183 simple_test_executor.go:171] Step "[step: 09] Waiting for huge-service pods to be deleted" ended I0302 16:22:40.751197 91183 simple_test_executor.go:149] Step "[step: 10] Delete huge-service" started I0302 16:22:40.798606 91183 simple_test_executor.go:171] Step "[step: 10] Delete huge-service" ended I0302 16:22:40.798644 91183 simple_test_executor.go:149] Step "[step: 11] gathering measurements" started ... skipping 183 lines ...