PR #115331 pull-kubernetes-e2e-gce-100-performance #1631330241970442240
Recent runs || View in Spyglass
Recent runs || View in Spyglass
| PR | mimowo: Give terminal phase correctly to all pods that will not be restarted |
| Result | ABORTED |
| Tests | 0 failed / 86 succeeded |
| Started | |
| Elapsed | 57m42s |
| Revision | 3953c70faeeb6617e9ce7b863a55face027d1921 |
| Refs |
115331 |
| job-version | v1.27.0-alpha.2.457+03df2cd22451cb |
| kubetest-version | v20230222-b5208facd4 |
| revision | v1.27.0-alpha.2.457+03df2cd22451cb |
ClusterLoaderV2 huge-service overall (testing/huge-service/config.yaml)
ClusterLoaderV2 huge-service: [step: 01] starting measurements [00] - APIResponsivenessPrometheus
ClusterLoaderV2 huge-service: [step: 01] starting measurements [01] - APIResponsivenessPrometheusSimple
ClusterLoaderV2 huge-service: [step: 01] starting measurements [02] - TestMetrics
ClusterLoaderV2 huge-service: [step: 01] starting measurements [03] - InClusterNetworkLatency
ClusterLoaderV2 huge-service: [step: 02] Create huge-service
ClusterLoaderV2 huge-service: [step: 03] Creating huge-service measurements [00] - WaitForHugeServiceDeployments
ClusterLoaderV2 huge-service: [step: 04] Creating huge-service pods
ClusterLoaderV2 huge-service: [step: 05] Waiting for huge-service pods to be created [00] - WaitForHugeServiceDeployments
ClusterLoaderV2 huge-service: [step: 06] Updating huge-service pods
ClusterLoaderV2 huge-service: [step: 07] Waiting for huge-service pods to be updated [00] - WaitForHugeServiceDeployments
ClusterLoaderV2 huge-service: [step: 08] Deleting huge-service pods
ClusterLoaderV2 huge-service: [step: 09] Waiting for huge-service pods to be deleted [00] - WaitForHugeServiceDeployments
ClusterLoaderV2 huge-service: [step: 10] Delete huge-service
ClusterLoaderV2 huge-service: [step: 11] gathering measurements [00] - APIResponsivenessPrometheus
ClusterLoaderV2 huge-service: [step: 11] gathering measurements [01] - APIResponsivenessPrometheusSimple
ClusterLoaderV2 huge-service: [step: 11] gathering measurements [02] - TestMetrics
ClusterLoaderV2 huge-service: [step: 11] gathering measurements [03] - InClusterNetworkLatency
ClusterLoaderV2 load overall (testing/load/config.yaml)
ClusterLoaderV2 load: [step: 01] starting measurements [00] - APIResponsivenessPrometheus
ClusterLoaderV2 load: [step: 01] starting measurements [01] - APIResponsivenessPrometheusSimple
ClusterLoaderV2 load: [step: 01] starting measurements [02] - CreatePhasePodStartupLatency
ClusterLoaderV2 load: [step: 01] starting measurements [03] - InClusterNetworkLatency
ClusterLoaderV2 load: [step: 01] starting measurements [04] - SLOMeasurement
ClusterLoaderV2 load: [step: 01] starting measurements [05] - NetworkProgrammingLatency
ClusterLoaderV2 load: [step: 01] starting measurements [06] - Kube-proxy partial iptables restore failures
ClusterLoaderV2 load: [step: 01] starting measurements [07] - APIAvailability
ClusterLoaderV2 load: [step: 01] starting measurements [08] - Quotas total usage
ClusterLoaderV2 load: [step: 01] starting measurements [09] - TestMetrics
ClusterLoaderV2 load: [step: 02] Creating k8s services
ClusterLoaderV2 load: [step: 03] Creating PriorityClass for DaemonSets
ClusterLoaderV2 load: [step: 04] create objects configmaps and secrets
ClusterLoaderV2 load: [step: 05] Starting measurement for 'create objects' [00] -
ClusterLoaderV2 load: [step: 06] create objects
ClusterLoaderV2 load: [step: 07] Waiting for 'create objects' to be completed [00] -
ClusterLoaderV2 load: [step: 08] Creating scheduler throughput measurements [00] - HighThroughputPodStartupLatency
ClusterLoaderV2 load: [step: 08] Creating scheduler throughput measurements [01] - WaitForSchedulerThroughputDeployments
ClusterLoaderV2 load: [step: 08] Creating scheduler throughput measurements [02] - SchedulingThroughput
ClusterLoaderV2 load: [step: 09] create scheduler throughput pods
ClusterLoaderV2 load: [step: 10] Waiting for scheduler throughput pods to be created [00] - WaitForSchedulerThroughputDeployments
ClusterLoaderV2 load: [step: 11] Collecting scheduler throughput measurements [00] - HighThroughputPodStartupLatency
ClusterLoaderV2 load: [step: 11] Collecting scheduler throughput measurements [01] - SchedulingThroughput
ClusterLoaderV2 load: [step: 12] delete scheduler throughput pods
ClusterLoaderV2 load: [step: 13] Waiting for scheduler throughput pods to be deleted [00] - WaitForSchedulerThroughputDeployments
ClusterLoaderV2 load: [step: 14] Starting latency pod measurements [00] - PodStartupLatency
ClusterLoaderV2 load: [step: 14] Starting latency pod measurements [01] - WaitForRunningLatencyDeployments
ClusterLoaderV2 load: [step: 15] Creating latency pods
ClusterLoaderV2 load: [step: 16] Waiting for latency pods to be running [00] - WaitForRunningLatencyDeployments
ClusterLoaderV2 load: [step: 17] Deleting latency pods
ClusterLoaderV2 load: [step: 18] Waiting for latency pods to be deleted [00] - WaitForRunningLatencyDeployments
ClusterLoaderV2 load: [step: 19] Collecting pod startup latency [00] - PodStartupLatency
ClusterLoaderV2 load: [step: 20] Starting measurement for 'scale and update objects' [00] -
ClusterLoaderV2 load: [step: 21] scale and update objects
ClusterLoaderV2 load: [step: 22] Waiting for 'scale and update objects' to be completed [00] -
ClusterLoaderV2 load: [step: 23] Starting measurement for 'delete objects' [00] -
ClusterLoaderV2 load: [step: 24] delete objects
ClusterLoaderV2 load: [step: 25] Waiting for 'delete objects' to be completed [00] -
ClusterLoaderV2 load: [step: 25] Waiting for 'delete objects' to be completed [01] - WaitForPVCsToBeDeleted
ClusterLoaderV2 load: [step: 26] delete objects configmaps and secrets
ClusterLoaderV2 load: [step: 27] Deleting PriorityClass for DaemonSets
ClusterLoaderV2 load: [step: 28] Deleting k8s services
ClusterLoaderV2 load: [step: 29] gathering measurements [00] - APIResponsivenessPrometheus
ClusterLoaderV2 load: [step: 29] gathering measurements [01] - APIResponsivenessPrometheusSimple
ClusterLoaderV2 load: [step: 29] gathering measurements [02] - CreatePhasePodStartupLatency
ClusterLoaderV2 load: [step: 29] gathering measurements [03] - InClusterNetworkLatency
ClusterLoaderV2 load: [step: 29] gathering measurements [04] - SLOMeasurement
ClusterLoaderV2 load: [step: 29] gathering measurements [05] - NetworkProgrammingLatency
ClusterLoaderV2 load: [step: 29] gathering measurements [06] - Kube-proxy partial iptables restore failures
ClusterLoaderV2 load: [step: 29] gathering measurements [07] - APIAvailability
ClusterLoaderV2 load: [step: 29] gathering measurements [08] - Quotas total usage
ClusterLoaderV2 load: [step: 29] gathering measurements [09] - TestMetrics
kubetest Build
kubetest Check APIReachability
kubetest ClusterLoaderV2
kubetest Deferred TearDown
kubetest DumpClusterLogs
kubetest Extract
kubetest GetDeployer
kubetest Prepare
kubetest Stage
kubetest TearDown
kubetest TearDown Previous
kubetest Timeout
kubetest Up
kubetest list nodes
kubetest test setup
... skipping 67 lines ... HOSTNAME=08fa6551-b917-11ed-955f-1adabc486015 IMAGE=gcr.io/k8s-staging-test-infra/kubekins-e2e:v20230222-b5208facd4-master INSTANCE_PREFIX=e2e-115331-95a39 JENKINS_GCE_SSH_PRIVATE_KEY_FILE=/workspace/.ssh/google_compute_engine JENKINS_GCE_SSH_PUBLIC_KEY_FILE=/workspace/.ssh/google_compute_engine.pub JOB_NAME=pull-kubernetes-e2e-gce-100-performance JOB_SPEC={"type":"presubmit","job":"pull-kubernetes-e2e-gce-100-performance","buildid":"1631330241970442240","prowjobid":"08fa6551-b917-11ed-955f-1adabc486015","refs":{"org":"kubernetes","repo":"kubernetes","repo_link":"https://github.com/kubernetes/kubernetes","base_ref":"master","base_sha":"b6d102d634d357600bc0fcb076395c88caef6acb","base_link":"https://github.com/kubernetes/kubernetes/commit/b6d102d634d357600bc0fcb076395c88caef6acb","pulls":[{"number":115331,"author":"mimowo","sha":"3953c70faeeb6617e9ce7b863a55face027d1921","title":"Mark Pending Terminating pods as Failed by Kubelet","link":"https://github.com/kubernetes/kubernetes/pull/115331","commit_link":"https://github.com/kubernetes/kubernetes/pull/115331/commits/3953c70faeeb6617e9ce7b863a55face027d1921","author_link":"https://github.com/mimowo"}],"path_alias":"k8s.io/kubernetes"},"extra_refs":[{"org":"kubernetes","repo":"perf-tests","base_ref":"master","path_alias":"k8s.io/perf-tests"},{"org":"kubernetes","repo":"release","base_ref":"master","path_alias":"k8s.io/release"}],"decoration_config":{"timeout":"2h0m0s","grace_period":"15m0s","utility_images":{"clonerefs":"gcr.io/k8s-prow/clonerefs:v20230301-6893d98ee8","initupload":"gcr.io/k8s-prow/initupload:v20230301-6893d98ee8","entrypoint":"gcr.io/k8s-prow/entrypoint:v20230301-6893d98ee8","sidecar":"gcr.io/k8s-prow/sidecar:v20230301-6893d98ee8"},"resources":{"clonerefs":{"requests":{"cpu":"100m"}},"initupload":{"requests":{"cpu":"100m"}},"place_entrypoint":{"requests":{"cpu":"100m"}},"sidecar":{"requests":{"cpu":"100m"}}},"gcs_configuration":{"bucket":"kubernetes-jenkins","path_strategy":"legacy","default_org":"kubernetes","default_repo":"kubernetes"},"gcs_credentials_secret":"service-account"}} JOB_TYPE=presubmit KUBECTL_PRUNE_WHITELIST_OVERRIDE=core/v1/ConfigMap core/v1/Endpoints core/v1/Namespace core/v1/PersistentVolumeClaim core/v1/PersistentVolume core/v1/ReplicationController core/v1/Secret core/v1/Service batch/v1/Job batch/v1/CronJob apps/v1/DaemonSet apps/v1/Deployment apps/v1/ReplicaSet apps/v1/StatefulSet networking.k8s.io/v1/Ingress KUBELET_TEST_ARGS=--enable-debugging-handlers --kube-api-qps=100 --kube-api-burst=100 KUBEMARK_APISERVER_TEST_ARGS=--max-requests-inflight=80 --max-mutating-requests-inflight=0 --profiling --contention-profiling KUBEPROXY_TEST_ARGS=--profiling --metrics-bind-address=0.0.0.0 --feature-gates=MinimizeIPTablesRestore=true KUBERNETES_PORT=tcp://10.35.240.1:443 ... skipping 568 lines ... NAME ZONE MACHINE_TYPE PREEMPTIBLE INTERNAL_IP EXTERNAL_IP STATUS e2e-115331-95a39-master us-east1-b n1-standard-8 10.40.0.3 35.196.27.45 RUNNING Setting e2e-115331-95a39-master's aliases to 'pods-default:10.64.0.0/24;10.40.0.2/32' (added 10.40.0.2) Updating network interface [nic0] of instance [e2e-115331-95a39-master]... ...........done. Updated [https://www.googleapis.com/compute/v1/projects/k8s-infra-e2e-boskos-scale-02/zones/us-east1-b/instances/e2e-115331-95a39-master]. Failed to execute 'sudo /bin/bash /home/kubernetes/bin/kube-master-internal-route.sh' on e2e-115331-95a39-master despite 5 attempts Last attempt failed with: /bin/bash: /home/kubernetes/bin/kube-master-internal-route.sh: No such file or directory Creating firewall... ..Created [https://www.googleapis.com/compute/v1/projects/k8s-infra-e2e-boskos-scale-02/global/firewalls/e2e-115331-95a39-minion-all]. done. NAME NETWORK DIRECTION PRIORITY ALLOW DENY DISABLED e2e-115331-95a39-minion-all e2e-115331-95a39 INGRESS 1000 tcp,udp,icmp,esp,ah,sctp False Creating nodes. ... skipping 33 lines ... Looking for address 'e2e-115331-95a39-master-ip' Looking for address 'e2e-115331-95a39-master-internal-ip' Using master: e2e-115331-95a39-master (external IP: 35.196.27.45; internal IP: 10.40.0.2) Waiting up to 300 seconds for cluster initialization. This will continually check to see if the API for kubernetes is reachable. This may time out if there was some uncaught error during start up. Kubernetes cluster created. Cluster "k8s-infra-e2e-boskos-scale-02_e2e-115331-95a39" set. User "k8s-infra-e2e-boskos-scale-02_e2e-115331-95a39" set. Context "k8s-infra-e2e-boskos-scale-02_e2e-115331-95a39" created. Switched to context "k8s-infra-e2e-boskos-scale-02_e2e-115331-95a39". ... skipping 242 lines ... e2e-115331-95a39-minion-group-zb6l Ready <none> 41s v1.27.0-alpha.2.457+03df2cd22451cb e2e-115331-95a39-minion-group-zbmt Ready <none> 43s v1.27.0-alpha.2.457+03df2cd22451cb e2e-115331-95a39-minion-heapster Ready <none> 50s v1.27.0-alpha.2.457+03df2cd22451cb Warning: v1 ComponentStatus is deprecated in v1.19+ Validate output: Warning: v1 ComponentStatus is deprecated in v1.19+ NAME STATUS MESSAGE ERROR etcd-1 Healthy {"health":"true","reason":""} etcd-0 Healthy {"health":"true","reason":""} scheduler Healthy ok controller-manager Healthy ok [0;33mCluster validation encountered some problems, but cluster should be in working order[0m ...ignoring non-fatal errors in validate-cluster Done, listing cluster services: [0;32mKubernetes control plane[0m is running at [0;33mhttps://35.196.27.45[0m [0;32mGLBCDefaultBackend[0m is running at [0;33mhttps://35.196.27.45/api/v1/namespaces/kube-system/services/default-http-backend:http/proxy[0m [0;32mCoreDNS[0m is running at [0;33mhttps://35.196.27.45/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy[0m [0;32mMetrics-server[0m is running at [0;33mhttps://35.196.27.45/api/v1/namespaces/kube-system/services/https:metrics-server:/proxy[0m ... skipping 1121 lines ... I0302 17:02:04.815137 90965 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-pq3du4-1), controlledBy(small-deployment-261): Pods: 5 out of 5 created, 5 running (5 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0302 17:02:04.921412 90965 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-pq3du4-1), controlledBy(small-deployment-151): Pods: 5 out of 5 created, 5 running (5 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0302 17:02:05.232993 90965 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-pq3du4-1), controlledBy(small-deployment-175): Pods: 5 out of 5 created, 5 running (5 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0302 17:02:05.401223 90965 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-pq3du4-1), controlledBy(small-deployment-121): Pods: 5 out of 5 created, 5 running (5 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0302 17:02:05.462508 90965 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-pq3du4-1), controlledBy(small-deployment-36): Pods: 5 out of 5 created, 5 running (5 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0302 17:02:05.964472 90965 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-pq3du4-1), controlledBy(small-deployment-45): Pods: 5 out of 5 created, 5 running (5 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0302 17:02:06.006551 90965 wait_for_controlled_pods.go:365] WaitForControlledPodsRunning: running 2, deleted 0, timeout: 0, failed: 0 I0302 17:02:06.006592 90965 wait_for_controlled_pods.go:370] WaitForControlledPodsRunning: maxDuration=50.008360471s, operationTimeout=15m0s, ratio=0.06 I0302 17:02:06.006629 90965 wait_for_controlled_pods.go:384] WaitForControlledPodsRunning: 2/2 StatefulSets are running with all pods I0302 17:02:06.007756 90965 wait_for_controlled_pods.go:365] WaitForControlledPodsRunning: running 1, deleted 0, timeout: 0, failed: 0 I0302 17:02:06.007790 90965 wait_for_controlled_pods.go:370] WaitForControlledPodsRunning: maxDuration=5.001908264s, operationTimeout=15m0s, ratio=0.01 I0302 17:02:06.007790 90965 wait_for_controlled_pods.go:365] WaitForControlledPodsRunning: running 3, deleted 0, timeout: 0, failed: 0 I0302 17:02:06.007807 90965 wait_for_controlled_pods.go:384] WaitForControlledPodsRunning: 1/1 DaemonSets are running with all pods I0302 17:02:06.007816 90965 wait_for_controlled_pods.go:370] WaitForControlledPodsRunning: maxDuration=5.004888221s, operationTimeout=15m0s, ratio=0.01 I0302 17:02:06.007834 90965 wait_for_controlled_pods.go:384] WaitForControlledPodsRunning: 3/3 Jobs are running with all pods I0302 17:02:06.211581 90965 wait_for_controlled_pods.go:365] WaitForControlledPodsRunning: running 323, deleted 0, timeout: 0, failed: 0 I0302 17:02:06.211619 90965 wait_for_controlled_pods.go:370] WaitForControlledPodsRunning: maxDuration=5.008153123s, operationTimeout=15m0s, ratio=0.01 I0302 17:02:06.211635 90965 wait_for_controlled_pods.go:384] WaitForControlledPodsRunning: 323/323 Deployments are running with all pods I0302 17:02:06.211660 90965 simple_test_executor.go:171] Step "[step: 07] Waiting for 'create objects' to be completed" ended I0302 17:02:06.211677 90965 simple_test_executor.go:149] Step "[step: 08] Creating scheduler throughput measurements" started I0302 17:02:06.211787 90965 wait_for_controlled_pods.go:257] WaitForControlledPodsRunning: starting wait for controlled pods measurement... I0302 17:02:06.211773 90965 pod_startup_latency.go:132] PodStartupLatency: labelSelector(group = scheduler-throughput): starting pod startup latency measurement... ... skipping 4 lines ... I0302 17:02:06.355086 90965 simple_test_executor.go:171] Step "[step: 09] create scheduler throughput pods" ended I0302 17:02:06.355111 90965 simple_test_executor.go:149] Step "[step: 10] Waiting for scheduler throughput pods to be created" started I0302 17:02:06.355158 90965 wait_for_controlled_pods.go:288] WaitForControlledPodsRunning: waiting for controlled pods measurement... I0302 17:02:11.363437 90965 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-pq3du4-2), controlledBy(scheduler-throughput-deployment-0): Pods: 577 out of 1000 created, 384 running (384 updated), 191 pending scheduled, 2 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0302 17:02:16.379652 90965 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-pq3du4-2), controlledBy(scheduler-throughput-deployment-0): Pods: 1000 out of 1000 created, 909 running (909 updated), 91 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0302 17:02:21.397471 90965 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-pq3du4-2), controlledBy(scheduler-throughput-deployment-0): Pods: 1000 out of 1000 created, 1000 running (1000 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0302 17:02:21.397553 90965 wait_for_controlled_pods.go:365] WaitForControlledPodsRunning: running 1, deleted 0, timeout: 0, failed: 0 I0302 17:02:21.397569 90965 wait_for_controlled_pods.go:370] WaitForControlledPodsRunning: maxDuration=15.042529058s, operationTimeout=20m0s, ratio=0.01 I0302 17:02:21.397590 90965 wait_for_controlled_pods.go:384] WaitForControlledPodsRunning: 1/1 Deployments are running with all pods I0302 17:02:21.397621 90965 simple_test_executor.go:171] Step "[step: 10] Waiting for scheduler throughput pods to be created" ended I0302 17:02:21.397642 90965 simple_test_executor.go:149] Step "[step: 11] Collecting scheduler throughput measurements" started I0302 17:02:21.397682 90965 scheduling_throughput.go:154] SchedulingThroughput: gathering data I0302 17:02:21.397732 90965 pod_startup_latency.go:226] PodStartupLatency: labelSelector(group = scheduler-throughput): gathering pod startup latency measurement... ... skipping 33 lines ... I0302 17:02:21.802673 90965 simple_test_executor.go:149] Step "[step: 13] Waiting for scheduler throughput pods to be deleted" started I0302 17:02:21.802722 90965 wait_for_pods.go:64] WaitForControlledPodsRunning: namespace(test-pq3du4-2), controlledBy(scheduler-throughput-deployment-0): starting with timeout: 19m59.999975117s I0302 17:02:21.802712 90965 wait_for_controlled_pods.go:288] WaitForControlledPodsRunning: waiting for controlled pods measurement... I0302 17:02:26.813736 90965 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-pq3du4-2), controlledBy(scheduler-throughput-deployment-0): Pods: 460 out of 0 created, 460 running (460 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 86 terminating, 0 unknown, 0 runningButNotReady I0302 17:02:31.814633 90965 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-pq3du4-2), controlledBy(scheduler-throughput-deployment-0): Pods: 0 out of 0 created, 0 running (0 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 42 terminating, 0 unknown, 0 runningButNotReady I0302 17:02:36.815450 90965 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-pq3du4-2), controlledBy(scheduler-throughput-deployment-0): Pods: 0 out of 0 created, 0 running (0 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0302 17:02:36.815498 90965 wait_for_controlled_pods.go:365] WaitForControlledPodsRunning: running 0, deleted 1, timeout: 0, failed: 0 I0302 17:02:36.815512 90965 wait_for_controlled_pods.go:370] WaitForControlledPodsRunning: maxDuration=15.01275996s, operationTimeout=20m0s, ratio=0.01 I0302 17:02:36.815530 90965 wait_for_controlled_pods.go:384] WaitForControlledPodsRunning: 0/0 Deployments are running with all pods I0302 17:02:36.815555 90965 simple_test_executor.go:171] Step "[step: 13] Waiting for scheduler throughput pods to be deleted" ended I0302 17:02:36.815579 90965 simple_test_executor.go:149] Step "[step: 14] Starting latency pod measurements" started I0302 17:02:36.815645 90965 wait_for_controlled_pods.go:257] WaitForControlledPodsRunning: starting wait for controlled pods measurement... I0302 17:02:36.815682 90965 pod_startup_latency.go:132] PodStartupLatency: labelSelector(group = latency): starting pod startup latency measurement... ... skipping 999 lines ... I0302 17:04:21.059138 90965 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-pq3du4-1), controlledBy(latency-deployment-494): Pods: 1 out of 1 created, 1 running (1 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0302 17:04:21.256601 90965 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-pq3du4-1), controlledBy(latency-deployment-495): Pods: 1 out of 1 created, 1 running (1 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0302 17:04:21.456570 90965 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-pq3du4-1), controlledBy(latency-deployment-496): Pods: 1 out of 1 created, 1 running (1 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0302 17:04:21.659471 90965 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-pq3du4-1), controlledBy(latency-deployment-497): Pods: 1 out of 1 created, 1 running (1 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0302 17:04:21.860862 90965 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-pq3du4-1), controlledBy(latency-deployment-498): Pods: 1 out of 1 created, 1 running (1 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0302 17:04:22.058605 90965 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-pq3du4-1), controlledBy(latency-deployment-499): Pods: 1 out of 1 created, 1 running (1 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0302 17:04:22.467333 90965 wait_for_controlled_pods.go:365] WaitForControlledPodsRunning: running 500, deleted 0, timeout: 0, failed: 0 I0302 17:04:22.467383 90965 wait_for_controlled_pods.go:370] WaitForControlledPodsRunning: maxDuration=5.001414429s, operationTimeout=15m0s, ratio=0.01 I0302 17:04:22.467411 90965 wait_for_controlled_pods.go:384] WaitForControlledPodsRunning: 500/500 Deployments are running with all pods I0302 17:04:22.467466 90965 simple_test_executor.go:171] Step "[step: 16] Waiting for latency pods to be running" ended I0302 17:04:22.467497 90965 simple_test_executor.go:149] Step "[step: 17] Deleting latency pods" started I0302 17:04:22.506609 90965 wait_for_pods.go:64] WaitForControlledPodsRunning: namespace(test-pq3du4-1), controlledBy(latency-deployment-0): starting with timeout: 14m59.999976858s I0302 17:04:22.607086 90965 wait_for_pods.go:64] WaitForControlledPodsRunning: namespace(test-pq3du4-1), controlledBy(latency-deployment-1): starting with timeout: 14m59.999974293s ... skipping 495 lines ... I0302 17:05:12.208980 90965 wait_for_pods.go:64] WaitForControlledPodsRunning: namespace(test-pq3du4-1), controlledBy(latency-deployment-497): starting with timeout: 14m59.999976585s I0302 17:05:12.306976 90965 wait_for_pods.go:64] WaitForControlledPodsRunning: namespace(test-pq3du4-1), controlledBy(latency-deployment-498): starting with timeout: 14m59.999973413s I0302 17:05:12.406357 90965 simple_test_executor.go:171] Step "[step: 17] Deleting latency pods" ended I0302 17:05:12.406396 90965 simple_test_executor.go:149] Step "[step: 18] Waiting for latency pods to be deleted" started I0302 17:05:12.406433 90965 wait_for_controlled_pods.go:288] WaitForControlledPodsRunning: waiting for controlled pods measurement... I0302 17:05:12.406740 90965 wait_for_pods.go:64] WaitForControlledPodsRunning: namespace(test-pq3du4-1), controlledBy(latency-deployment-499): starting with timeout: 14m59.99997626s I0302 17:05:17.454527 90965 wait_for_controlled_pods.go:365] WaitForControlledPodsRunning: running 0, deleted 500, timeout: 0, failed: 0 I0302 17:05:17.454569 90965 wait_for_controlled_pods.go:370] WaitForControlledPodsRunning: maxDuration=5.001218044s, operationTimeout=15m0s, ratio=0.01 I0302 17:05:17.454586 90965 wait_for_controlled_pods.go:384] WaitForControlledPodsRunning: 0/0 Deployments are running with all pods I0302 17:05:17.454613 90965 simple_test_executor.go:171] Step "[step: 18] Waiting for latency pods to be deleted" ended I0302 17:05:17.454637 90965 simple_test_executor.go:149] Step "[step: 19] Collecting pod startup latency" started I0302 17:05:17.454682 90965 pod_startup_latency.go:226] PodStartupLatency: labelSelector(group = latency): gathering pod startup latency measurement... I0302 17:05:17.895725 90965 phase_latency.go:141] PodStartupLatency: 100 worst schedule_to_watch latencies: [{test-pq3du4-1/latency-deployment-281-6dbb644456-z7rrn 2.345558176s} {test-pq3du4-1/latency-deployment-198-68f894869d-6wp6l 2.348474915s} {test-pq3du4-1/latency-deployment-79-5c9d8c8c54-59dq7 2.348509194s} {test-pq3du4-1/latency-deployment-338-d5f979f7d-w7mx8 2.349475336s} {test-pq3du4-1/latency-deployment-417-785d8d659d-vvrgd 2.350920219s} {test-pq3du4-1/latency-deployment-9-77cd7fd7bd-j5tx4 2.351643397s} {test-pq3du4-1/latency-deployment-458-8df8f9dc6-vjlwp 2.354993908s} {test-pq3du4-1/latency-deployment-193-78d75dbd84-s8mp2 2.358659s} {test-pq3du4-1/latency-deployment-348-64ddd8798d-n2c28 2.363340672s} {test-pq3du4-1/latency-deployment-362-bb98ff556-ptnkm 2.366267387s} {test-pq3du4-1/latency-deployment-244-6479699d74-lrmmt 2.371280117s} {test-pq3du4-1/latency-deployment-303-5ccdfbf7f4-lvb2g 2.374245483s} {test-pq3du4-1/latency-deployment-108-6b4b78d756-v2rmb 2.374853619s} {test-pq3du4-1/latency-deployment-124-645f7b96d4-s6j54 2.375863386s} {test-pq3du4-1/latency-deployment-268-6c58b69db6-fspcr 2.382815483s} {test-pq3du4-1/latency-deployment-433-578b66d974-hhmqt 2.386016624s} {test-pq3du4-1/latency-deployment-403-6446dc74cb-8gh46 2.389583943s} {test-pq3du4-1/latency-deployment-278-66d6c86bb4-57gn6 2.389954083s} {test-pq3du4-1/latency-deployment-149-596b6bfc78-gds7t 2.391555456s} {test-pq3du4-1/latency-deployment-82-869755fb78-wbxhf 2.393821114s} {test-pq3du4-1/latency-deployment-203-8559fbb84d-2qhfq 2.400064945s} {test-pq3du4-1/latency-deployment-47-7d84d9459d-scqpd 2.400392252s} {test-pq3du4-1/latency-deployment-188-7957bb6884-rp2xk 2.414140076s} {test-pq3du4-1/latency-deployment-313-b76b64f48-nvwz8 2.41446356s} {test-pq3du4-1/latency-deployment-374-6599c6c9cd-p86t8 2.418534182s} {test-pq3du4-1/latency-deployment-332-6f45f5588b-tksl2 2.418737077s} {test-pq3du4-1/latency-deployment-329-66487b9994-c9rvf 2.422120034s} {test-pq3du4-1/latency-deployment-428-65584d4954-xj6jv 2.423413035s} {test-pq3du4-1/latency-deployment-483-f55c458f6-xt69p 2.426391001s} {test-pq3du4-1/latency-deployment-423-c4d557f68-h5xmb 2.428464661s} {test-pq3du4-1/latency-deployment-328-65587857c6-dtx9q 2.430461842s} {test-pq3du4-1/latency-deployment-477-8cffc5ccd-qrr6b 2.432754088s} {test-pq3du4-1/latency-deployment-64-78b758949d-qjv62 2.434524857s} {test-pq3du4-1/latency-deployment-219-6fcdf9b9d6-httw6 2.435143924s} {test-pq3du4-1/latency-deployment-0-7855574f6b-pc56k 2.436971684s} {test-pq3du4-1/latency-deployment-204-5f7fd69d6-fvkvd 2.441115011s} {test-pq3du4-1/latency-deployment-228-bfb654c86-5w824 2.442748129s} {test-pq3du4-1/latency-deployment-45-f655c6bfd-5t2dp 2.447199946s} {test-pq3du4-1/latency-deployment-304-866c5c8f8d-78g8l 2.448774705s} {test-pq3du4-1/latency-deployment-349-86d6ffbcc8-lmmtf 2.453273237s} {test-pq3du4-1/latency-deployment-258-789775f476-58hn4 2.463670118s} {test-pq3du4-1/latency-deployment-24-55484ddf7d-7zwt2 2.471475885s} {test-pq3du4-1/latency-deployment-408-6b659d4db-pkjrz 2.474495085s} {test-pq3du4-1/latency-deployment-202-655b9fd794-grj6v 2.478650208s} {test-pq3du4-1/latency-deployment-297-6797cbbcdb-qcs78 2.482639596s} {test-pq3du4-1/latency-deployment-352-77db46585d-p4crt 2.482695217s} {test-pq3du4-1/latency-deployment-308-d599c6c44-j6kq8 2.49788355s} {test-pq3du4-1/latency-deployment-427-5744bf8576-4vzxw 2.498197046s} {test-pq3du4-1/latency-deployment-457-589dbcf75d-q8bb2 2.498748962s} {test-pq3du4-1/latency-deployment-10-6594c66c46-pbp2h 2.502273165s} {test-pq3du4-1/latency-deployment-343-56c98bd9fd-b99cx 2.508124806s} {test-pq3du4-1/latency-deployment-89-74dc7cc654-xpg8x 2.509834507s} {test-pq3du4-1/latency-deployment-263-c87d4f964-5d4w7 2.51457865s} {test-pq3du4-1/latency-deployment-368-6cfc7bfc6-rzkxf 2.520769271s} {test-pq3du4-1/latency-deployment-209-784499d9dd-sdsnk 2.527460397s} {test-pq3du4-1/latency-deployment-463-876878c5b-8mkpd 2.528345885s} {test-pq3du4-1/latency-deployment-373-85cbc95d6d-c8c6k 2.535955196s} {test-pq3du4-1/latency-deployment-363-666c6c4bf8-vhtwv 2.537040814s} {test-pq3du4-1/latency-deployment-432-69c89c66f6-hr6gd 2.53802971s} {test-pq3du4-1/latency-deployment-442-9c7db6786-q2wtp 2.539446745s} {test-pq3du4-1/latency-deployment-40-859749fc94-877vj 2.551175795s} {test-pq3du4-1/latency-deployment-284-7c94c6fcfd-xfwsv 2.572985965s} {test-pq3du4-1/latency-deployment-83-68c877b4cd-q5sbd 2.577497089s} {test-pq3du4-1/latency-deployment-248-79d649c594-kt2bs 2.57838634s} {test-pq3du4-1/latency-deployment-274-7cbb5dd4d8-7v7kn 2.579364743s} {test-pq3du4-1/latency-deployment-443-67bb5c76f6-pv55q 2.582013332s} {test-pq3du4-1/latency-deployment-453-648859f44b-xpvzf 2.582672293s} {test-pq3du4-1/latency-deployment-78-669b97f6fd-jhnx6 2.590135696s} {test-pq3du4-1/latency-deployment-234-5d747fbc5b-vrzdf 2.59159847s} {test-pq3du4-1/latency-deployment-173-dcb9f799b-5c78g 2.598819679s} {test-pq3du4-1/latency-deployment-393-6f769b8596-86cmg 2.600714771s} {test-pq3du4-1/latency-deployment-199-5879d88cbd-fqj4w 2.621240903s} {test-pq3du4-1/latency-deployment-354-6784479898-vkbrl 2.626089097s} {test-pq3du4-1/latency-deployment-422-85ddf8ff54-zg8rq 2.63094347s} {test-pq3du4-1/latency-deployment-447-76f9dcc67b-b7zwl 2.635430695s} {test-pq3du4-1/latency-deployment-158-8466665d9d-8gbfl 2.640480541s} {test-pq3du4-1/latency-deployment-104-d76b76dc6-wcrrk 2.641489806s} {test-pq3du4-1/latency-deployment-189-5745c8d7d6-c8t2w 2.643196122s} {test-pq3du4-1/latency-deployment-74-65cb994774-g6z2j 2.659516337s} {test-pq3du4-1/latency-deployment-289-77ccf55464-c75wc 2.670816651s} {test-pq3du4-1/latency-deployment-344-55c8778976-vzzpn 2.673917795s} {test-pq3du4-1/latency-deployment-49-9c6bfbbf4-kq5nv 2.686272424s} {test-pq3du4-1/latency-deployment-224-86f6d7cd74-9n9l8 2.688617335s} {test-pq3du4-1/latency-deployment-238-7ff8856cfb-ms6s8 2.690514415s} {test-pq3du4-1/latency-deployment-194-7f8c688b94-gl2rw 2.697052879s} {test-pq3du4-1/latency-deployment-109-7cb5979bb-48565 2.728149937s} {test-pq3du4-1/latency-deployment-478-65d5db7976-4ztk8 2.730622699s} {test-pq3du4-1/latency-deployment-388-7b868f8c44-jjmfd 2.752312807s} {test-pq3du4-1/latency-deployment-468-955b9b4c4-l6xmz 2.760617556s} {test-pq3du4-1/latency-deployment-44-67f76cc48b-4kft5 2.763896991s} {test-pq3du4-1/latency-deployment-339-5cd775f864-hd95c 2.765602455s} {test-pq3du4-1/latency-deployment-309-76b684d586-k7jwm 2.786152s} {test-pq3du4-1/latency-deployment-418-6885d967f6-cfhfl 2.81326464s} {test-pq3du4-1/latency-deployment-239-6774dd9558-6bh8c 2.832527041s} {test-pq3du4-1/latency-deployment-488-58b6f787f4-8qflt 2.83965752s} {test-pq3du4-1/latency-deployment-159-69bc55b668-qjqbr 2.855988445s} {test-pq3du4-1/latency-deployment-169-854b8d4866-lpcjb 2.902800367s} {test-pq3du4-1/latency-deployment-249-65c5444fdd-tfl7q 2.909550039s} {test-pq3du4-1/latency-deployment-55-85856845c8-mfqt5 2.94241835s} {test-pq3du4-1/latency-deployment-35-577c9595c8-m5hbt 2.949791741s}] ... skipping 596 lines ... I0302 17:05:58.810827 90965 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-pq3du4-1), controlledBy(small-deployment-281): Pods: 2 out of 2 created, 2 running (2 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0302 17:05:58.850304 90965 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-pq3du4-1), controlledBy(small-deployment-12): Pods: 7 out of 7 created, 7 running (7 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0302 17:05:59.380281 90965 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-pq3du4-1), controlledBy(small-deployment-6): Pods: 3 out of 3 created, 3 running (3 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0302 17:05:59.486109 90965 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-pq3du4-1), controlledBy(small-deployment-163): Pods: 3 out of 3 created, 3 running (3 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0302 17:05:59.515260 90965 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-pq3du4-1), controlledBy(small-deployment-117): Pods: 3 out of 3 created, 3 running (3 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0302 17:05:59.678740 90965 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-pq3du4-1), controlledBy(small-deployment-146): Pods: 4 out of 4 created, 4 running (4 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0302 17:05:59.813331 90965 wait_for_controlled_pods.go:365] WaitForControlledPodsRunning: running 3, deleted 0, timeout: 0, failed: 0 I0302 17:05:59.813384 90965 wait_for_controlled_pods.go:370] WaitForControlledPodsRunning: maxDuration=5.005329884s, operationTimeout=15m0s, ratio=0.01 I0302 17:05:59.813404 90965 wait_for_controlled_pods.go:384] WaitForControlledPodsRunning: 3/3 Jobs are running with all pods I0302 17:05:59.833254 90965 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-pq3du4-1), controlledBy(daemonset-0): Pods: 100 out of 100 created, 94 running (94 updated), 6 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0302 17:05:59.971497 90965 wait_for_controlled_pods.go:365] WaitForControlledPodsRunning: running 323, deleted 0, timeout: 0, failed: 0 I0302 17:05:59.971533 90965 wait_for_controlled_pods.go:370] WaitForControlledPodsRunning: maxDuration=5.011049275s, operationTimeout=15m0s, ratio=0.01 I0302 17:05:59.971553 90965 wait_for_controlled_pods.go:384] WaitForControlledPodsRunning: 323/323 Deployments are running with all pods I0302 17:06:00.665950 90965 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-pq3du4-1), controlledBy(medium-statefulset-0): Pods: 40 out of 40 created, 40 running (40 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0302 17:06:00.666005 90965 wait_for_controlled_pods.go:365] WaitForControlledPodsRunning: running 2, deleted 0, timeout: 0, failed: 0 I0302 17:06:00.666018 90965 wait_for_controlled_pods.go:370] WaitForControlledPodsRunning: maxDuration=20.005417122s, operationTimeout=15m0s, ratio=0.02 I0302 17:06:00.666033 90965 wait_for_controlled_pods.go:384] WaitForControlledPodsRunning: 2/2 StatefulSets are running with all pods I0302 17:06:04.835662 90965 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-pq3du4-1), controlledBy(daemonset-0): Pods: 100 out of 100 created, 100 running (100 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0302 17:06:04.835753 90965 wait_for_controlled_pods.go:365] WaitForControlledPodsRunning: running 1, deleted 0, timeout: 0, failed: 0 I0302 17:06:04.835771 90965 wait_for_controlled_pods.go:370] WaitForControlledPodsRunning: maxDuration=30.100600912s, operationTimeout=15m0s, ratio=0.03 I0302 17:06:04.835792 90965 wait_for_controlled_pods.go:384] WaitForControlledPodsRunning: 1/1 DaemonSets are running with all pods I0302 17:06:04.835811 90965 simple_test_executor.go:171] Step "[step: 22] Waiting for 'scale and update objects' to be completed" ended I0302 17:06:04.835832 90965 simple_test_executor.go:149] Step "[step: 23] Starting measurement for 'delete objects'" started I0302 17:06:04.835880 90965 wait_for_controlled_pods.go:254] WaitForControlledPodsRunning: wait for controlled pods measurement already running I0302 17:06:04.835949 90965 wait_for_controlled_pods.go:254] WaitForControlledPodsRunning: wait for controlled pods measurement already running ... skipping 333 lines ... I0302 17:09:38.293351 90965 simple_test_executor.go:149] Step "[step: 25] Waiting for 'delete objects' to be completed" started I0302 17:09:38.293361 90965 wait_for_pods.go:64] WaitForControlledPodsRunning: namespace(test-pq3du4-1), controlledBy(small-deployment-135): starting with timeout: 14m59.999973706s I0302 17:09:38.293426 90965 wait_for_controlled_pods.go:288] WaitForControlledPodsRunning: waiting for controlled pods measurement... I0302 17:09:38.293453 90965 wait_for_controlled_pods.go:288] WaitForControlledPodsRunning: waiting for controlled pods measurement... I0302 17:09:38.293480 90965 wait_for_controlled_pods.go:288] WaitForControlledPodsRunning: waiting for controlled pods measurement... I0302 17:09:38.293429 90965 wait_for_controlled_pods.go:288] WaitForControlledPodsRunning: waiting for controlled pods measurement... I0302 17:09:43.328320 90965 wait_for_controlled_pods.go:365] WaitForControlledPodsRunning: running 0, deleted 2, timeout: 0, failed: 0 I0302 17:09:43.328381 90965 wait_for_controlled_pods.go:370] WaitForControlledPodsRunning: maxDuration=5.001021013s, operationTimeout=15m0s, ratio=0.01 I0302 17:09:43.328418 90965 wait_for_controlled_pods.go:384] WaitForControlledPodsRunning: 0/0 StatefulSets are running with all pods I0302 17:09:43.329643 90965 wait_for_controlled_pods.go:365] WaitForControlledPodsRunning: running 0, deleted 323, timeout: 0, failed: 0 I0302 17:09:43.329666 90965 wait_for_controlled_pods.go:365] WaitForControlledPodsRunning: running 0, deleted 3, timeout: 0, failed: 0 I0302 17:09:43.329670 90965 wait_for_controlled_pods.go:370] WaitForControlledPodsRunning: maxDuration=5.001400545s, operationTimeout=15m0s, ratio=0.01 I0302 17:09:43.329686 90965 wait_for_controlled_pods.go:370] WaitForControlledPodsRunning: maxDuration=5.000632982s, operationTimeout=15m0s, ratio=0.01 I0302 17:09:43.329691 90965 wait_for_controlled_pods.go:384] WaitForControlledPodsRunning: 0/0 Deployments are running with all pods I0302 17:09:43.329701 90965 wait_for_controlled_pods.go:384] WaitForControlledPodsRunning: 0/0 Jobs are running with all pods I0302 17:09:43.330766 90965 wait_for_controlled_pods.go:365] WaitForControlledPodsRunning: running 0, deleted 1, timeout: 0, failed: 0 I0302 17:09:43.330792 90965 wait_for_controlled_pods.go:370] WaitForControlledPodsRunning: maxDuration=5.000393157s, operationTimeout=15m0s, ratio=0.01 I0302 17:09:43.330828 90965 wait_for_controlled_pods.go:384] WaitForControlledPodsRunning: 0/0 DaemonSets are running with all pods I0302 17:09:43.344023 90965 wait_for_pvcs.go:78] WaitForBoundPVCs: labelSelector(group = load): PVCs: 0 out of 0 created, 0 bound, 0 pending, 0 lost I0302 17:09:43.344061 90965 simple_test_executor.go:171] Step "[step: 25] Waiting for 'delete objects' to be completed" ended I0302 17:09:43.344085 90965 simple_test_executor.go:149] Step "[step: 26] delete objects configmaps and secrets" started I0302 17:10:15.620021 90965 simple_test_executor.go:171] Step "[step: 26] delete objects configmaps and secrets" ended ... skipping 252 lines ... I0302 17:11:46.030755 90965 simple_test_executor.go:149] Step "[step: 05] Waiting for huge-service pods to be created" started I0302 17:11:46.030778 90965 wait_for_pods.go:64] WaitForControlledPodsRunning: namespace(test-xxmml4-1), controlledBy(huge-service-deployment-0): starting with timeout: 29m59.99998229s I0302 17:11:46.030783 90965 wait_for_controlled_pods.go:288] WaitForControlledPodsRunning: waiting for controlled pods measurement... I0302 17:11:51.039879 90965 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-xxmml4-1), controlledBy(huge-service-deployment-0): Pods: 570 out of 1000 created, 408 running (408 updated), 158 pending scheduled, 4 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0302 17:11:56.055778 90965 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-xxmml4-1), controlledBy(huge-service-deployment-0): Pods: 1000 out of 1000 created, 883 running (883 updated), 117 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0302 17:12:01.073330 90965 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-xxmml4-1), controlledBy(huge-service-deployment-0): Pods: 1000 out of 1000 created, 1000 running (1000 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0302 17:12:01.073423 90965 wait_for_controlled_pods.go:365] WaitForControlledPodsRunning: running 1, deleted 0, timeout: 0, failed: 0 I0302 17:12:01.073449 90965 wait_for_controlled_pods.go:370] WaitForControlledPodsRunning: maxDuration=15.042604328s, operationTimeout=30m0s, ratio=0.01 I0302 17:12:01.073471 90965 wait_for_controlled_pods.go:384] WaitForControlledPodsRunning: 1/1 Deployments are running with all pods I0302 17:12:01.073489 90965 simple_test_executor.go:171] Step "[step: 05] Waiting for huge-service pods to be created" ended I0302 17:12:01.073512 90965 simple_test_executor.go:149] Step "[step: 06] Updating huge-service pods" started I0302 17:12:01.150875 90965 simple_test_executor.go:171] Step "[step: 06] Updating huge-service pods" ended I0302 17:12:01.150909 90965 simple_test_executor.go:149] Step "[step: 07] Waiting for huge-service pods to be updated" started ... skipping 4 lines ... I0302 17:12:11.709159 90965 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-xxmml4-1), controlledBy(huge-service-deployment-0): Pods: 974 out of 1000 created, 954 running (511 updated), 16 pending scheduled, 4 not scheduled, 0 inactive, 145 terminating, 0 unknown, 0 runningButNotReady W0302 17:12:16.819552 90965 wait_for_pods.go:119] WaitForControlledPodsRunning: namespace(test-xxmml4-1), controlledBy(huge-service-deployment-0): 299 pods appeared: huge-service-deployment-0-7945dfc44-684rq, huge-service-deployment-0-7945dfc44-5vzs7, huge-service-deployment-0-7945dfc44-bx87f, huge-service-deployment-0-7945dfc44-ndb6w, huge-service-deployment-0-7945dfc44-qdp7t, huge-service-deployment-0-7945dfc44-9mftk, huge-service-deployment-0-7945dfc44-52bgd, huge-service-deployment-0-7945dfc44-8zsw4, huge-service-deployment-0-7945dfc44-c67fm, huge-service-deployment-0-7945dfc44-7x2pw, huge-service-deployment-0-7945dfc44-xnqbh, huge-service-deployment-0-7945dfc44-7vqz5, huge-service-deployment-0-7945dfc44-k7hnc, huge-service-deployment-0-7945dfc44-qwq8d, huge-service-deployment-0-7945dfc44-kgmzz, huge-service-deployment-0-7945dfc44-w26ld, huge-service-deployment-0-7945dfc44-5tqtc, huge-service-deployment-0-7945dfc44-m8x28, huge-service-deployment-0-7945dfc44-qrz27, huge-service-deployment-0-7945dfc44-csgfr, huge-service-deployment-0-7945dfc44-qfl6m, huge-service-deployment-0-7945dfc44-wft4f, huge-service-deployment-0-7945dfc44-gr5r4, huge-service-deployment-0-7945dfc44-zltpm, huge-service-deployment-0-7945dfc44-8vxxt, huge-service-deployment-0-7945dfc44-j7pbs, huge-service-deployment-0-7945dfc44-5stm4, huge-service-deployment-0-7945dfc44-t574m, huge-service-deployment-0-7945dfc44-x7v7b, huge-service-deployment-0-7945dfc44-45nqv, huge-service-deployment-0-7945dfc44-nm98s, huge-service-deployment-0-7945dfc44-56gbr, huge-service-deployment-0-7945dfc44-htmc8, huge-service-deployment-0-7945dfc44-xm4sh, huge-service-deployment-0-7945dfc44-z7sdp, huge-service-deployment-0-7945dfc44-xtbk9, huge-service-deployment-0-7945dfc44-bkw44, huge-service-deployment-0-7945dfc44-7kgbv, huge-service-deployment-0-7945dfc44-mmv8k, huge-service-deployment-0-7945dfc44-rj96v, huge-service-deployment-0-7945dfc44-hxhkk, huge-service-deployment-0-7945dfc44-ppwst, huge-service-deployment-0-7945dfc44-ngn9m, huge-service-deployment-0-7945dfc44-8fmt4, huge-service-deployment-0-7945dfc44-xjn7j, huge-service-deployment-0-7945dfc44-ml876, huge-service-deployment-0-7945dfc44-2rt8m, huge-service-deployment-0-7945dfc44-jvb99, huge-service-deployment-0-7945dfc44-bzz8b, huge-service-deployment-0-7945dfc44-7pwlk, huge-service-deployment-0-7945dfc44-c8x7m, huge-service-deployment-0-7945dfc44-dt97s, huge-service-deployment-0-7945dfc44-nq552, huge-service-deployment-0-7945dfc44-xzxg7, huge-service-deployment-0-7945dfc44-9b7pn, huge-service-deployment-0-7945dfc44-mrk2l, huge-service-deployment-0-7945dfc44-lmxw2, huge-service-deployment-0-7945dfc44-gc7xm, huge-service-deployment-0-7945dfc44-h5jzn, huge-service-deployment-0-7945dfc44-2jtz6, huge-service-deployment-0-7945dfc44-sjvmw, huge-service-deployment-0-7945dfc44-tgxb6, huge-service-deployment-0-7945dfc44-gknhg, huge-service-deployment-0-7945dfc44-csmdk, huge-service-deployment-0-7945dfc44-hh7t6, huge-service-deployment-0-7945dfc44-5q52r, huge-service-deployment-0-7945dfc44-5hxvn, huge-service-deployment-0-7945dfc44-j79js, huge-service-deployment-0-7945dfc44-zvwpz, huge-service-deployment-0-7945dfc44-64nww, huge-service-deployment-0-7945dfc44-llmwp, huge-service-deployment-0-7945dfc44-s2vks, huge-service-deployment-0-7945dfc44-xwbvr, huge-service-deployment-0-7945dfc44-zn7qp, huge-service-deployment-0-7945dfc44-798rj, huge-service-deployment-0-7945dfc44-5lkqg, huge-service-deployment-0-7945dfc44-fmbpr, huge-service-deployment-0-7945dfc44-bjhs4, huge-service-deployment-0-7945dfc44-p25sq, huge-service-deployment-0-7945dfc44-gwx92, huge-service-deployment-0-7945dfc44-nw6d2, huge-service-deployment-0-7945dfc44-n6sxd, huge-service-deployment-0-7945dfc44-dmhnl, huge-service-deployment-0-7945dfc44-c57bl, huge-service-deployment-0-7945dfc44-qqfgv, huge-service-deployment-0-7945dfc44-kqwnf, huge-service-deployment-0-7945dfc44-kzj88, huge-service-deployment-0-7945dfc44-9vx79, huge-service-deployment-0-7945dfc44-tl9nb, huge-service-deployment-0-7945dfc44-lgkh2, huge-service-deployment-0-7945dfc44-h5szd, huge-service-deployment-0-7945dfc44-bqb86, huge-service-deployment-0-7945dfc44-c2dx6, huge-service-deployment-0-7945dfc44-626gm, huge-service-deployment-0-7945dfc44-s2z5p, huge-service-deployment-0-7945dfc44-4pjdg, huge-service-deployment-0-7945dfc44-dz8h9, huge-service-deployment-0-7945dfc44-zfxzw, huge-service-deployment-0-7945dfc44-tqbh8, huge-service-deployment-0-7945dfc44-prgch, huge-service-deployment-0-7945dfc44-vvg7g, huge-service-deployment-0-7945dfc44-xp2pn, huge-service-deployment-0-7945dfc44-ktwg8, huge-service-deployment-0-7945dfc44-vfwmx, huge-service-deployment-0-7945dfc44-2b87p, huge-service-deployment-0-7945dfc44-tb56g, huge-service-deployment-0-7945dfc44-vw9vp, huge-service-deployment-0-7945dfc44-z7ds9, huge-service-deployment-0-7945dfc44-n8hkb, huge-service-deployment-0-7945dfc44-g2hwc, huge-service-deployment-0-7945dfc44-fvr7v, huge-service-deployment-0-7945dfc44-fmb4w, huge-service-deployment-0-7945dfc44-qwdp6, huge-service-deployment-0-7945dfc44-f568h, huge-service-deployment-0-7945dfc44-btz7c, huge-service-deployment-0-7945dfc44-6nbd7, huge-service-deployment-0-7945dfc44-b29kc, huge-service-deployment-0-7945dfc44-95rwx, huge-service-deployment-0-7945dfc44-rwc26, huge-service-deployment-0-7945dfc44-2bjzv, huge-service-deployment-0-7945dfc44-rk4nj, huge-service-deployment-0-7945dfc44-sdfsk, huge-service-deployment-0-7945dfc44-mjjg6, huge-service-deployment-0-7945dfc44-n7bkq, huge-service-deployment-0-7945dfc44-hcz7w, huge-service-deployment-0-7945dfc44-9zbqq, huge-service-deployment-0-7945dfc44-zlcms, huge-service-deployment-0-7945dfc44-49mzm, huge-service-deployment-0-7945dfc44-sxcm8, huge-service-deployment-0-7945dfc44-6xpjt, huge-service-deployment-0-7945dfc44-w7m84, huge-service-deployment-0-7945dfc44-jj594, huge-service-deployment-0-7945dfc44-9txwp, huge-service-deployment-0-7945dfc44-mfmlj, huge-service-deployment-0-7945dfc44-kvzz4, huge-service-deployment-0-7945dfc44-82nrw, huge-service-deployment-0-7945dfc44-kmtbs, huge-service-deployment-0-7945dfc44-zkf2w, huge-service-deployment-0-7945dfc44-mgfls, huge-service-deployment-0-7945dfc44-9r77j, huge-service-deployment-0-7945dfc44-kzzss, huge-service-deployment-0-7945dfc44-mz7ln, huge-service-deployment-0-7945dfc44-vxjc4, huge-service-deployment-0-7945dfc44-jtq7n, huge-service-deployment-0-7945dfc44-rfjzn, huge-service-deployment-0-7945dfc44-n55g5, huge-service-deployment-0-7945dfc44-v4kc5, huge-service-deployment-0-7945dfc44-zqks9, huge-service-deployment-0-7945dfc44-25trx, huge-service-deployment-0-7945dfc44-flf5g, huge-service-deployment-0-7945dfc44-54vmg, huge-service-deployment-0-7945dfc44-c6dkr, huge-service-deployment-0-7945dfc44-fnxpl, huge-service-deployment-0-7945dfc44-vlpxx, huge-service-deployment-0-7945dfc44-gkqbp, huge-service-deployment-0-7945dfc44-gpgwm, huge-service-deployment-0-7945dfc44-9fkw7, huge-service-deployment-0-7945dfc44-l2rkf, huge-service-deployment-0-7945dfc44-gfcmf, huge-service-deployment-0-7945dfc44-598gz, huge-service-deployment-0-7945dfc44-5lz5q, huge-service-deployment-0-7945dfc44-g66bp, huge-service-deployment-0-7945dfc44-nr6wh, huge-service-deployment-0-7945dfc44-8xgxx, huge-service-deployment-0-7945dfc44-qbzjc, huge-service-deployment-0-7945dfc44-4wh2k, huge-service-deployment-0-7945dfc44-brstn, huge-service-deployment-0-7945dfc44-bshqq, huge-service-deployment-0-7945dfc44-d5w8v, huge-service-deployment-0-7945dfc44-2vnc7, huge-service-deployment-0-7945dfc44-mclsj, huge-service-deployment-0-7945dfc44-zccbg, huge-service-deployment-0-7945dfc44-9lrxs, huge-service-deployment-0-7945dfc44-wb9bc, huge-service-deployment-0-7945dfc44-ftmhr, huge-service-deployment-0-7945dfc44-5lg7l, huge-service-deployment-0-7945dfc44-mhl5f, huge-service-deployment-0-7945dfc44-dzrx5, huge-service-deployment-0-7945dfc44-fpq7q, huge-service-deployment-0-7945dfc44-s4pvh, huge-service-deployment-0-7945dfc44-gbch4, huge-service-deployment-0-7945dfc44-g5mtd, huge-service-deployment-0-7945dfc44-h9n2x, huge-service-deployment-0-7945dfc44-d4sdc, huge-service-deployment-0-7945dfc44-zwwq7, huge-service-deployment-0-7945dfc44-ccj2q, huge-service-deployment-0-7945dfc44-shcxz, huge-service-deployment-0-7945dfc44-bw2ss, huge-service-deployment-0-7945dfc44-fbf4k, huge-service-deployment-0-7945dfc44-wjqd4, huge-service-deployment-0-7945dfc44-2s87l, huge-service-deployment-0-7945dfc44-x5fs7, huge-service-deployment-0-7945dfc44-9wbg9, huge-service-deployment-0-7945dfc44-l7hbw, huge-service-deployment-0-7945dfc44-lzg98, huge-service-deployment-0-7945dfc44-c9tp4, huge-service-deployment-0-7945dfc44-mskns, huge-service-deployment-0-7945dfc44-lg2qb, huge-service-deployment-0-7945dfc44-r7lgx, huge-service-deployment-0-7945dfc44-8jcdt, huge-service-deployment-0-7945dfc44-lfltf, huge-service-deployment-0-7945dfc44-jv2h2, huge-service-deployment-0-7945dfc44-jt8pp, huge-service-deployment-0-7945dfc44-7nmpb, huge-service-deployment-0-7945dfc44-jvjsq, huge-service-deployment-0-7945dfc44-288kf, huge-service-deployment-0-7945dfc44-jf8lj, huge-service-deployment-0-7945dfc44-cw9wf, huge-service-deployment-0-7945dfc44-dxkh2, huge-service-deployment-0-7945dfc44-c62hr, huge-service-deployment-0-7945dfc44-kfgqx, huge-service-deployment-0-7945dfc44-tmm78, huge-service-deployment-0-7945dfc44-dhvst, huge-service-deployment-0-7945dfc44-4mzk5, huge-service-deployment-0-7945dfc44-74cxq, huge-service-deployment-0-7945dfc44-5mwrc, huge-service-deployment-0-7945dfc44-7pqz4, huge-service-deployment-0-7945dfc44-t8g6q, huge-service-deployment-0-7945dfc44-mw4hp, huge-service-deployment-0-7945dfc44-hllpz, huge-service-deployment-0-7945dfc44-tpqcc, huge-service-deployment-0-7945dfc44-zfbdl, huge-service-deployment-0-7945dfc44-2kq4v, huge-service-deployment-0-7945dfc44-fjz68, huge-service-deployment-0-7945dfc44-zt4f2, huge-service-deployment-0-7945dfc44-t7sbh, huge-service-deployment-0-7945dfc44-vmdt8, huge-service-deployment-0-7945dfc44-xbnlw, huge-service-deployment-0-7945dfc44-v784b, huge-service-deployment-0-7945dfc44-d6m5m, huge-service-deployment-0-7945dfc44-h5nw9, huge-service-deployment-0-7945dfc44-h2vb5, huge-service-deployment-0-7945dfc44-jvntd, huge-service-deployment-0-7945dfc44-6zzr9, huge-service-deployment-0-7945dfc44-m5t7q, huge-service-deployment-0-7945dfc44-9w98p, huge-service-deployment-0-7945dfc44-pnwgw, huge-service-deployment-0-7945dfc44-ldl4p, huge-service-deployment-0-7945dfc44-bbpg5, huge-service-deployment-0-7945dfc44-lp5h7, huge-service-deployment-0-7945dfc44-px8ww, huge-service-deployment-0-7945dfc44-zd7hf, huge-service-deployment-0-7945dfc44-xhkmm, huge-service-deployment-0-7945dfc44-bvq58, huge-service-deployment-0-7945dfc44-s4m42, huge-service-deployment-0-7945dfc44-c54cg, huge-service-deployment-0-7945dfc44-d9q9k, huge-service-deployment-0-7945dfc44-xbvbw, huge-service-deployment-0-7945dfc44-v2qgp, huge-service-deployment-0-7945dfc44-tq84p, huge-service-deployment-0-7945dfc44-bvvwj, huge-service-deployment-0-7945dfc44-64wqj, huge-service-deployment-0-7945dfc44-crw7c, huge-service-deployment-0-7945dfc44-r7fkh, huge-service-deployment-0-7945dfc44-8jj5s, huge-service-deployment-0-7945dfc44-86qtj, huge-service-deployment-0-7945dfc44-s98pt, huge-service-deployment-0-7945dfc44-mbr4v, huge-service-deployment-0-7945dfc44-7c8sd, huge-service-deployment-0-7945dfc44-kvffs, huge-service-deployment-0-7945dfc44-shlnh, huge-service-deployment-0-7945dfc44-lmfsh, huge-service-deployment-0-7945dfc44-rf7mn, huge-service-deployment-0-7945dfc44-trmnv, huge-service-deployment-0-7945dfc44-n2n4q, huge-service-deployment-0-7945dfc44-hd6rf, huge-service-deployment-0-7945dfc44-z7w5l, huge-service-deployment-0-7945dfc44-fsnf8, huge-service-deployment-0-7945dfc44-skfq7, huge-service-deployment-0-7945dfc44-jwv2b, huge-service-deployment-0-7945dfc44-vx9l2, huge-service-deployment-0-7945dfc44-kbwnw, huge-service-deployment-0-7945dfc44-hwntl, huge-service-deployment-0-7945dfc44-5vlsh, huge-service-deployment-0-7945dfc44-2r2hl, huge-service-deployment-0-7945dfc44-hzkmf, huge-service-deployment-0-7945dfc44-xjrz4, huge-service-deployment-0-7945dfc44-s2h48, huge-service-deployment-0-7945dfc44-w8jzt, huge-service-deployment-0-7945dfc44-zx4dg, huge-service-deployment-0-7945dfc44-v2cxt, huge-service-deployment-0-7945dfc44-tnwbb, huge-service-deployment-0-7945dfc44-h4vbd, huge-service-deployment-0-7945dfc44-wxxss, huge-service-deployment-0-7945dfc44-9f2fh, huge-service-deployment-0-7945dfc44-kt8hl, huge-service-deployment-0-7945dfc44-vlnp2, huge-service-deployment-0-7945dfc44-qkhps, huge-service-deployment-0-7945dfc44-8wn8z, huge-service-deployment-0-7945dfc44-gk8lq, huge-service-deployment-0-7945dfc44-ls87j, huge-service-deployment-0-7945dfc44-726w9, huge-service-deployment-0-7945dfc44-kvfpd, huge-service-deployment-0-7945dfc44-d6h7n, huge-service-deployment-0-7945dfc44-h74hg, huge-service-deployment-0-7945dfc44-5255r, huge-service-deployment-0-7945dfc44-lgflh, huge-service-deployment-0-7945dfc44-pbvgn, huge-service-deployment-0-7945dfc44-smfvv I0302 17:12:16.819606 90965 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-xxmml4-1), controlledBy(huge-service-deployment-0): Pods: 1067 out of 1000 created, 988 running (751 updated), 76 pending scheduled, 3 not scheduled, 0 inactive, 108 terminating, 0 unknown, 0 runningButNotReady W0302 17:12:21.837092 90965 wait_for_pods.go:119] WaitForControlledPodsRunning: namespace(test-xxmml4-1), controlledBy(huge-service-deployment-0): 170 pods appeared: huge-service-deployment-0-7945dfc44-t2xc2, huge-service-deployment-0-7945dfc44-g44rm, huge-service-deployment-0-7945dfc44-cl5ks, huge-service-deployment-0-7945dfc44-9tmg2, huge-service-deployment-0-7945dfc44-jv6v7, huge-service-deployment-0-7945dfc44-t9bvf, huge-service-deployment-0-7945dfc44-crnp2, huge-service-deployment-0-7945dfc44-tfh6z, huge-service-deployment-0-7945dfc44-qqjfj, huge-service-deployment-0-7945dfc44-gsgrk, huge-service-deployment-0-7945dfc44-5nrfc, huge-service-deployment-0-7945dfc44-kn6z2, huge-service-deployment-0-7945dfc44-zw4bm, huge-service-deployment-0-7945dfc44-fwfnq, huge-service-deployment-0-7945dfc44-c9fd5, huge-service-deployment-0-7945dfc44-ssppd, huge-service-deployment-0-7945dfc44-kzp2l, huge-service-deployment-0-7945dfc44-56qfz, huge-service-deployment-0-7945dfc44-sd6z6, huge-service-deployment-0-7945dfc44-52lx9, huge-service-deployment-0-7945dfc44-6lrx2, huge-service-deployment-0-7945dfc44-qzjs5, huge-service-deployment-0-7945dfc44-hrqjw, huge-service-deployment-0-7945dfc44-llr4c, huge-service-deployment-0-7945dfc44-hxtxt, huge-service-deployment-0-7945dfc44-szq5x, huge-service-deployment-0-7945dfc44-w4khc, huge-service-deployment-0-7945dfc44-zgrj6, huge-service-deployment-0-7945dfc44-f4pq8, huge-service-deployment-0-7945dfc44-8q9l6, huge-service-deployment-0-7945dfc44-465lj, huge-service-deployment-0-7945dfc44-jc52m, huge-service-deployment-0-7945dfc44-hbdqp, huge-service-deployment-0-7945dfc44-s5p54, huge-service-deployment-0-7945dfc44-zkjwq, huge-service-deployment-0-7945dfc44-j788h, huge-service-deployment-0-7945dfc44-f88ch, huge-service-deployment-0-7945dfc44-ll82d, huge-service-deployment-0-7945dfc44-lmq9b, huge-service-deployment-0-7945dfc44-zz5dc, huge-service-deployment-0-7945dfc44-n58s8, huge-service-deployment-0-7945dfc44-djk5p, huge-service-deployment-0-7945dfc44-tqnqj, huge-service-deployment-0-7945dfc44-kch74, huge-service-deployment-0-7945dfc44-qmrdj, huge-service-deployment-0-7945dfc44-j4r6l, huge-service-deployment-0-7945dfc44-6b5kd, huge-service-deployment-0-7945dfc44-6jqzr, huge-service-deployment-0-7945dfc44-7r86x, huge-service-deployment-0-7945dfc44-l97ph, huge-service-deployment-0-7945dfc44-tv8zd, huge-service-deployment-0-7945dfc44-k9l4p, huge-service-deployment-0-7945dfc44-lv66g, huge-service-deployment-0-7945dfc44-6ddv5, huge-service-deployment-0-7945dfc44-rxn5f, huge-service-deployment-0-7945dfc44-4wrsl, huge-service-deployment-0-7945dfc44-kjqtd, huge-service-deployment-0-7945dfc44-49nvz, huge-service-deployment-0-7945dfc44-dw72j, huge-service-deployment-0-7945dfc44-6dmhf, huge-service-deployment-0-7945dfc44-xxrnz, huge-service-deployment-0-7945dfc44-678xb, huge-service-deployment-0-7945dfc44-x2w8c, huge-service-deployment-0-7945dfc44-dnq2w, huge-service-deployment-0-7945dfc44-zbshr, huge-service-deployment-0-7945dfc44-5fg6z, huge-service-deployment-0-7945dfc44-bp6zj, huge-service-deployment-0-7945dfc44-z29xd, huge-service-deployment-0-7945dfc44-hlgdc, huge-service-deployment-0-7945dfc44-m7dcx, huge-service-deployment-0-7945dfc44-7gw2p, huge-service-deployment-0-7945dfc44-dpz46, huge-service-deployment-0-7945dfc44-clkzr, huge-service-deployment-0-7945dfc44-btw7b, huge-service-deployment-0-7945dfc44-95fhh, huge-service-deployment-0-7945dfc44-tn9fp, huge-service-deployment-0-7945dfc44-7jsjj, huge-service-deployment-0-7945dfc44-2s7m4, huge-service-deployment-0-7945dfc44-x6rpl, huge-service-deployment-0-7945dfc44-hqnt6, huge-service-deployment-0-7945dfc44-wkpms, huge-service-deployment-0-7945dfc44-cqbkb, huge-service-deployment-0-7945dfc44-t8h6h, huge-service-deployment-0-7945dfc44-tkh76, huge-service-deployment-0-7945dfc44-mzpqm, huge-service-deployment-0-7945dfc44-b5bhj, huge-service-deployment-0-7945dfc44-t82ff, huge-service-deployment-0-7945dfc44-45mzf, huge-service-deployment-0-7945dfc44-kgss8, huge-service-deployment-0-7945dfc44-nr8jp, huge-service-deployment-0-7945dfc44-45rcr, huge-service-deployment-0-7945dfc44-69vvc, huge-service-deployment-0-7945dfc44-6vfxz, huge-service-deployment-0-7945dfc44-jgvl5, huge-service-deployment-0-7945dfc44-mvzrd, huge-service-deployment-0-7945dfc44-qf7cj, huge-service-deployment-0-7945dfc44-6pdpp, huge-service-deployment-0-7945dfc44-4fw54, huge-service-deployment-0-7945dfc44-sltnn, huge-service-deployment-0-7945dfc44-lrkwt, huge-service-deployment-0-7945dfc44-4ll6d, huge-service-deployment-0-7945dfc44-r7knb, huge-service-deployment-0-7945dfc44-rqqdj, huge-service-deployment-0-7945dfc44-hx2w7, huge-service-deployment-0-7945dfc44-7zwgj, huge-service-deployment-0-7945dfc44-vh6sm, huge-service-deployment-0-7945dfc44-r8rjc, huge-service-deployment-0-7945dfc44-vhlch, huge-service-deployment-0-7945dfc44-pchfj, huge-service-deployment-0-7945dfc44-68w8k, huge-service-deployment-0-7945dfc44-g844h, huge-service-deployment-0-7945dfc44-x8b8c, huge-service-deployment-0-7945dfc44-vhvsh, huge-service-deployment-0-7945dfc44-tghjc, huge-service-deployment-0-7945dfc44-h5tht, huge-service-deployment-0-7945dfc44-6htq4, huge-service-deployment-0-7945dfc44-v6n2d, huge-service-deployment-0-7945dfc44-f7brj, huge-service-deployment-0-7945dfc44-nbpdf, huge-service-deployment-0-7945dfc44-ncsz7, huge-service-deployment-0-7945dfc44-2pk5n, huge-service-deployment-0-7945dfc44-krkb9, huge-service-deployment-0-7945dfc44-d5svw, huge-service-deployment-0-7945dfc44-j24k8, huge-service-deployment-0-7945dfc44-fjl7p, huge-service-deployment-0-7945dfc44-9nclh, huge-service-deployment-0-7945dfc44-rhc6t, huge-service-deployment-0-7945dfc44-xrkzr, huge-service-deployment-0-7945dfc44-kg5j7, huge-service-deployment-0-7945dfc44-qmngl, huge-service-deployment-0-7945dfc44-6759p, huge-service-deployment-0-7945dfc44-9xzqn, huge-service-deployment-0-7945dfc44-2ljcm, huge-service-deployment-0-7945dfc44-mzwlq, huge-service-deployment-0-7945dfc44-lsq52, huge-service-deployment-0-7945dfc44-w8sgh, huge-service-deployment-0-7945dfc44-bh9p6, huge-service-deployment-0-7945dfc44-p9nj9, huge-service-deployment-0-7945dfc44-qfxdv, huge-service-deployment-0-7945dfc44-bjgcg, huge-service-deployment-0-7945dfc44-tk6sl, huge-service-deployment-0-7945dfc44-sjw9v, huge-service-deployment-0-7945dfc44-cz667, huge-service-deployment-0-7945dfc44-cql2t, huge-service-deployment-0-7945dfc44-ljrhc, huge-service-deployment-0-7945dfc44-rt9g2, huge-service-deployment-0-7945dfc44-rmw8h, huge-service-deployment-0-7945dfc44-wkcfw, huge-service-deployment-0-7945dfc44-tqkct, huge-service-deployment-0-7945dfc44-4fqwc, huge-service-deployment-0-7945dfc44-l46lb, huge-service-deployment-0-7945dfc44-dqnzc, huge-service-deployment-0-7945dfc44-pmbzt, huge-service-deployment-0-7945dfc44-6m6tm, huge-service-deployment-0-7945dfc44-wz28x, huge-service-deployment-0-7945dfc44-qxk9t, huge-service-deployment-0-7945dfc44-j67jx, huge-service-deployment-0-7945dfc44-4kqdw, huge-service-deployment-0-7945dfc44-2vvpx, huge-service-deployment-0-7945dfc44-zp5rd, huge-service-deployment-0-7945dfc44-sc842, huge-service-deployment-0-7945dfc44-28gxw, huge-service-deployment-0-7945dfc44-qnx5k, huge-service-deployment-0-7945dfc44-d7zmj, huge-service-deployment-0-7945dfc44-wzh52, huge-service-deployment-0-7945dfc44-fr7dg, huge-service-deployment-0-7945dfc44-8b9nq, huge-service-deployment-0-7945dfc44-g9xpp, huge-service-deployment-0-7945dfc44-cz2mk, huge-service-deployment-0-7945dfc44-27w55 I0302 17:12:21.837147 90965 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-xxmml4-1), controlledBy(huge-service-deployment-0): Pods: 1000 out of 1000 created, 978 running (978 updated), 22 pending scheduled, 0 not scheduled, 0 inactive, 45 terminating, 0 unknown, 0 runningButNotReady I0302 17:12:26.854599 90965 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-xxmml4-1), controlledBy(huge-service-deployment-0): Pods: 1000 out of 1000 created, 1000 running (1000 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0302 17:12:26.854672 90965 wait_for_controlled_pods.go:365] WaitForControlledPodsRunning: running 1, deleted 0, timeout: 0, failed: 0 I0302 17:12:26.854686 90965 wait_for_controlled_pods.go:370] WaitForControlledPodsRunning: maxDuration=25.703709788s, operationTimeout=30m0s, ratio=0.01 I0302 17:12:26.854702 90965 wait_for_controlled_pods.go:384] WaitForControlledPodsRunning: 1/1 Deployments are running with all pods I0302 17:12:26.854736 90965 simple_test_executor.go:171] Step "[step: 07] Waiting for huge-service pods to be updated" ended I0302 17:12:26.854756 90965 simple_test_executor.go:149] Step "[step: 08] Deleting huge-service pods" started I0302 17:12:26.893649 90965 simple_test_executor.go:171] Step "[step: 08] Deleting huge-service pods" ended I0302 17:12:26.893701 90965 simple_test_executor.go:149] Step "[step: 09] Waiting for huge-service pods to be deleted" started I0302 17:12:26.893760 90965 wait_for_controlled_pods.go:288] WaitForControlledPodsRunning: waiting for controlled pods measurement... I0302 17:12:26.893772 90965 wait_for_pods.go:64] WaitForControlledPodsRunning: namespace(test-xxmml4-1), controlledBy(huge-service-deployment-0): starting with timeout: 29m59.999932702s I0302 17:12:31.903227 90965 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-xxmml4-1), controlledBy(huge-service-deployment-0): Pods: 462 out of 0 created, 462 running (462 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 91 terminating, 0 unknown, 0 runningButNotReady I0302 17:12:36.904404 90965 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-xxmml4-1), controlledBy(huge-service-deployment-0): Pods: 0 out of 0 created, 0 running (0 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 47 terminating, 0 unknown, 0 runningButNotReady I0302 17:12:41.905505 90965 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-xxmml4-1), controlledBy(huge-service-deployment-0): Pods: 0 out of 0 created, 0 running (0 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0302 17:12:41.905575 90965 wait_for_controlled_pods.go:365] WaitForControlledPodsRunning: running 0, deleted 1, timeout: 0, failed: 0 I0302 17:12:41.905589 90965 wait_for_controlled_pods.go:370] WaitForControlledPodsRunning: maxDuration=15.011775892s, operationTimeout=30m0s, ratio=0.01 I0302 17:12:41.905605 90965 wait_for_controlled_pods.go:384] WaitForControlledPodsRunning: 0/0 Deployments are running with all pods I0302 17:12:41.905627 90965 simple_test_executor.go:171] Step "[step: 09] Waiting for huge-service pods to be deleted" ended I0302 17:12:41.905647 90965 simple_test_executor.go:149] Step "[step: 10] Delete huge-service" started I0302 17:12:41.952251 90965 simple_test_executor.go:171] Step "[step: 10] Delete huge-service" ended I0302 17:12:41.952292 90965 simple_test_executor.go:149] Step "[step: 11] gathering measurements" started ... skipping 135 lines ...