PR #115145 pull-kubernetes-e2e-gce-100-performance #1617975695894384640
Recent runs || View in Spyglass
Recent runs || View in Spyglass
| PR | alexzielenski: ValidatingAdmissionPolicy: use natively-typed informer for params |
| Result | ABORTED |
| Tests | 0 failed / 86 succeeded |
| Started | |
| Elapsed | 58m48s |
| Revision | 5f179e4aa8b5b8773f6e00c5f3454f5132eb51ab |
| Refs |
115145 |
| job-version | v1.27.0-alpha.0.1232+92d467223d6cf0 |
| kubetest-version | v20230124-61b36115b7 |
| revision | v1.27.0-alpha.0.1232+92d467223d6cf0 |
ClusterLoaderV2 huge-service overall (testing/huge-service/config.yaml)
ClusterLoaderV2 huge-service: [step: 01] starting measurements [00] - APIResponsivenessPrometheus
ClusterLoaderV2 huge-service: [step: 01] starting measurements [01] - APIResponsivenessPrometheusSimple
ClusterLoaderV2 huge-service: [step: 01] starting measurements [02] - TestMetrics
ClusterLoaderV2 huge-service: [step: 01] starting measurements [03] - InClusterNetworkLatency
ClusterLoaderV2 huge-service: [step: 02] Create huge-service
ClusterLoaderV2 huge-service: [step: 03] Creating huge-service measurements [00] - WaitForHugeServiceDeployments
ClusterLoaderV2 huge-service: [step: 04] Creating huge-service pods
ClusterLoaderV2 huge-service: [step: 05] Waiting for huge-service pods to be created [00] - WaitForHugeServiceDeployments
ClusterLoaderV2 huge-service: [step: 06] Updating huge-service pods
ClusterLoaderV2 huge-service: [step: 07] Waiting for huge-service pods to be updated [00] - WaitForHugeServiceDeployments
ClusterLoaderV2 huge-service: [step: 08] Deleting huge-service pods
ClusterLoaderV2 huge-service: [step: 09] Waiting for huge-service pods to be deleted [00] - WaitForHugeServiceDeployments
ClusterLoaderV2 huge-service: [step: 10] Delete huge-service
ClusterLoaderV2 huge-service: [step: 11] gathering measurements [00] - APIResponsivenessPrometheus
ClusterLoaderV2 huge-service: [step: 11] gathering measurements [01] - APIResponsivenessPrometheusSimple
ClusterLoaderV2 huge-service: [step: 11] gathering measurements [02] - TestMetrics
ClusterLoaderV2 huge-service: [step: 11] gathering measurements [03] - InClusterNetworkLatency
ClusterLoaderV2 load overall (testing/load/config.yaml)
ClusterLoaderV2 load: [step: 01] starting measurements [00] - APIResponsivenessPrometheus
ClusterLoaderV2 load: [step: 01] starting measurements [01] - APIResponsivenessPrometheusSimple
ClusterLoaderV2 load: [step: 01] starting measurements [02] - CreatePhasePodStartupLatency
ClusterLoaderV2 load: [step: 01] starting measurements [03] - InClusterNetworkLatency
ClusterLoaderV2 load: [step: 01] starting measurements [04] - SLOMeasurement
ClusterLoaderV2 load: [step: 01] starting measurements [05] - NetworkProgrammingLatency
ClusterLoaderV2 load: [step: 01] starting measurements [06] - Kube-proxy partial iptables restore failures
ClusterLoaderV2 load: [step: 01] starting measurements [07] - APIAvailability
ClusterLoaderV2 load: [step: 01] starting measurements [08] - Quotas total usage
ClusterLoaderV2 load: [step: 01] starting measurements [09] - TestMetrics
ClusterLoaderV2 load: [step: 02] Creating k8s services
ClusterLoaderV2 load: [step: 03] Creating PriorityClass for DaemonSets
ClusterLoaderV2 load: [step: 04] create objects configmaps and secrets
ClusterLoaderV2 load: [step: 05] Starting measurement for 'create objects' [00] -
ClusterLoaderV2 load: [step: 06] create objects
ClusterLoaderV2 load: [step: 07] Waiting for 'create objects' to be completed [00] -
ClusterLoaderV2 load: [step: 08] Creating scheduler throughput measurements [00] - HighThroughputPodStartupLatency
ClusterLoaderV2 load: [step: 08] Creating scheduler throughput measurements [01] - WaitForSchedulerThroughputDeployments
ClusterLoaderV2 load: [step: 08] Creating scheduler throughput measurements [02] - SchedulingThroughput
ClusterLoaderV2 load: [step: 09] create scheduler throughput pods
ClusterLoaderV2 load: [step: 10] Waiting for scheduler throughput pods to be created [00] - WaitForSchedulerThroughputDeployments
ClusterLoaderV2 load: [step: 11] Collecting scheduler throughput measurements [00] - HighThroughputPodStartupLatency
ClusterLoaderV2 load: [step: 11] Collecting scheduler throughput measurements [01] - SchedulingThroughput
ClusterLoaderV2 load: [step: 12] delete scheduler throughput pods
ClusterLoaderV2 load: [step: 13] Waiting for scheduler throughput pods to be deleted [00] - WaitForSchedulerThroughputDeployments
ClusterLoaderV2 load: [step: 14] Starting latency pod measurements [00] - PodStartupLatency
ClusterLoaderV2 load: [step: 14] Starting latency pod measurements [01] - WaitForRunningLatencyDeployments
ClusterLoaderV2 load: [step: 15] Creating latency pods
ClusterLoaderV2 load: [step: 16] Waiting for latency pods to be running [00] - WaitForRunningLatencyDeployments
ClusterLoaderV2 load: [step: 17] Deleting latency pods
ClusterLoaderV2 load: [step: 18] Waiting for latency pods to be deleted [00] - WaitForRunningLatencyDeployments
ClusterLoaderV2 load: [step: 19] Collecting pod startup latency [00] - PodStartupLatency
ClusterLoaderV2 load: [step: 20] Starting measurement for 'scale and update objects' [00] -
ClusterLoaderV2 load: [step: 21] scale and update objects
ClusterLoaderV2 load: [step: 22] Waiting for 'scale and update objects' to be completed [00] -
ClusterLoaderV2 load: [step: 23] Starting measurement for 'delete objects' [00] -
ClusterLoaderV2 load: [step: 24] delete objects
ClusterLoaderV2 load: [step: 25] Waiting for 'delete objects' to be completed [00] -
ClusterLoaderV2 load: [step: 25] Waiting for 'delete objects' to be completed [01] - WaitForPVCsToBeDeleted
ClusterLoaderV2 load: [step: 26] delete objects configmaps and secrets
ClusterLoaderV2 load: [step: 27] Deleting PriorityClass for DaemonSets
ClusterLoaderV2 load: [step: 28] Deleting k8s services
ClusterLoaderV2 load: [step: 29] gathering measurements [00] - APIResponsivenessPrometheus
ClusterLoaderV2 load: [step: 29] gathering measurements [01] - APIResponsivenessPrometheusSimple
ClusterLoaderV2 load: [step: 29] gathering measurements [02] - CreatePhasePodStartupLatency
ClusterLoaderV2 load: [step: 29] gathering measurements [03] - InClusterNetworkLatency
ClusterLoaderV2 load: [step: 29] gathering measurements [04] - SLOMeasurement
ClusterLoaderV2 load: [step: 29] gathering measurements [05] - NetworkProgrammingLatency
ClusterLoaderV2 load: [step: 29] gathering measurements [06] - Kube-proxy partial iptables restore failures
ClusterLoaderV2 load: [step: 29] gathering measurements [07] - APIAvailability
ClusterLoaderV2 load: [step: 29] gathering measurements [08] - Quotas total usage
ClusterLoaderV2 load: [step: 29] gathering measurements [09] - TestMetrics
kubetest Build
kubetest Check APIReachability
kubetest ClusterLoaderV2
kubetest Deferred TearDown
kubetest DumpClusterLogs
kubetest Extract
kubetest GetDeployer
kubetest Prepare
kubetest Stage
kubetest TearDown
kubetest TearDown Previous
kubetest Timeout
kubetest Up
kubetest list nodes
kubetest test setup
... skipping 691 lines ... Looking for address 'e2e-115145-95a39-master-ip' Looking for address 'e2e-115145-95a39-master-internal-ip' Using master: e2e-115145-95a39-master (external IP: 34.75.116.203; internal IP: 10.40.0.2) Waiting up to 300 seconds for cluster initialization. This will continually check to see if the API for kubernetes is reachable. This may time out if there was some uncaught error during start up. Kubernetes cluster created. Cluster "k8s-infra-e2e-boskos-scale-18_e2e-115145-95a39" set. User "k8s-infra-e2e-boskos-scale-18_e2e-115145-95a39" set. Context "k8s-infra-e2e-boskos-scale-18_e2e-115145-95a39" created. Switched to context "k8s-infra-e2e-boskos-scale-18_e2e-115145-95a39". ... skipping 243 lines ... e2e-115145-95a39-minion-group-zccl Ready <none> 62s v1.27.0-alpha.0.1232+92d467223d6cf0 e2e-115145-95a39-minion-group-zl3d Ready <none> 59s v1.27.0-alpha.0.1232+92d467223d6cf0 e2e-115145-95a39-minion-heapster Ready <none> 76s v1.27.0-alpha.0.1232+92d467223d6cf0 Warning: v1 ComponentStatus is deprecated in v1.19+ Validate output: Warning: v1 ComponentStatus is deprecated in v1.19+ NAME STATUS MESSAGE ERROR etcd-1 Healthy {"health":"true","reason":""} etcd-0 Healthy {"health":"true","reason":""} controller-manager Healthy ok scheduler Healthy ok [0;33mCluster validation encountered some problems, but cluster should be in working order[0m ...ignoring non-fatal errors in validate-cluster Done, listing cluster services: [0;32mKubernetes control plane[0m is running at [0;33mhttps://34.75.116.203[0m [0;32mGLBCDefaultBackend[0m is running at [0;33mhttps://34.75.116.203/api/v1/namespaces/kube-system/services/default-http-backend:http/proxy[0m [0;32mCoreDNS[0m is running at [0;33mhttps://34.75.116.203/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy[0m [0;32mMetrics-server[0m is running at [0;33mhttps://34.75.116.203/api/v1/namespaces/kube-system/services/https:metrics-server:/proxy[0m ... skipping 1118 lines ... I0124 20:37:04.416972 89857 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-lie7op-1), controlledBy(small-deployment-198): Pods: 5 out of 5 created, 5 running (5 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0124 20:37:04.527809 89857 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-lie7op-1), controlledBy(small-deployment-249): Pods: 5 out of 5 created, 5 running (5 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0124 20:37:04.564793 89857 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-lie7op-1), controlledBy(medium-deployment-0): Pods: 30 out of 30 created, 25 running (25 updated), 5 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0124 20:37:05.203637 89857 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-lie7op-1), controlledBy(small-deployment-292): Pods: 5 out of 5 created, 5 running (5 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0124 20:37:05.228462 89857 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-lie7op-1), controlledBy(small-deployment-215): Pods: 5 out of 5 created, 5 running (5 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0124 20:37:06.157362 89857 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-lie7op-1), controlledBy(big-deployment-0): Pods: 250 out of 250 created, 172 running (172 updated), 78 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0124 20:37:06.189123 89857 wait_for_controlled_pods.go:365] WaitForControlledPodsRunning: running 1, deleted 0, timeout: 0, failed: 0 I0124 20:37:06.189157 89857 wait_for_controlled_pods.go:370] WaitForControlledPodsRunning: maxDuration=5.002257899s, operationTimeout=15m0s, ratio=0.01 I0124 20:37:06.189174 89857 wait_for_controlled_pods.go:384] WaitForControlledPodsRunning: 1/1 DaemonSets are running with all pods I0124 20:37:06.193464 89857 wait_for_controlled_pods.go:365] WaitForControlledPodsRunning: running 3, deleted 0, timeout: 0, failed: 0 I0124 20:37:06.193490 89857 wait_for_controlled_pods.go:370] WaitForControlledPodsRunning: maxDuration=5.006182079s, operationTimeout=15m0s, ratio=0.01 I0124 20:37:06.193512 89857 wait_for_controlled_pods.go:384] WaitForControlledPodsRunning: 3/3 Jobs are running with all pods I0124 20:37:06.193513 89857 wait_for_controlled_pods.go:365] WaitForControlledPodsRunning: running 2, deleted 0, timeout: 0, failed: 0 I0124 20:37:06.193560 89857 wait_for_controlled_pods.go:370] WaitForControlledPodsRunning: maxDuration=40.009060896s, operationTimeout=15m0s, ratio=0.04 I0124 20:37:06.193582 89857 wait_for_controlled_pods.go:384] WaitForControlledPodsRunning: 2/2 StatefulSets are running with all pods I0124 20:37:09.390412 89857 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-lie7op-1), controlledBy(big-deployment-1): Pods: 250 out of 250 created, 250 running (250 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0124 20:37:09.566752 89857 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-lie7op-1), controlledBy(medium-deployment-0): Pods: 30 out of 30 created, 30 running (30 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0124 20:37:11.164802 89857 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-lie7op-1), controlledBy(big-deployment-0): Pods: 250 out of 250 created, 250 running (250 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0124 20:37:11.164959 89857 wait_for_controlled_pods.go:365] WaitForControlledPodsRunning: running 323, deleted 0, timeout: 0, failed: 0 I0124 20:37:11.164983 89857 wait_for_controlled_pods.go:370] WaitForControlledPodsRunning: maxDuration=10.016438878s, operationTimeout=15m0s, ratio=0.01 I0124 20:37:11.165004 89857 wait_for_controlled_pods.go:384] WaitForControlledPodsRunning: 323/323 Deployments are running with all pods I0124 20:37:11.165030 89857 simple_test_executor.go:171] Step "[step: 07] Waiting for 'create objects' to be completed" ended I0124 20:37:11.165074 89857 simple_test_executor.go:149] Step "[step: 08] Creating scheduler throughput measurements" started I0124 20:37:11.165183 89857 pod_startup_latency.go:132] PodStartupLatency: labelSelector(group = scheduler-throughput): starting pod startup latency measurement... I0124 20:37:11.165311 89857 wait_for_controlled_pods.go:257] WaitForControlledPodsRunning: starting wait for controlled pods measurement... ... skipping 4 lines ... I0124 20:37:11.308839 89857 simple_test_executor.go:171] Step "[step: 09] create scheduler throughput pods" ended I0124 20:37:11.308866 89857 simple_test_executor.go:149] Step "[step: 10] Waiting for scheduler throughput pods to be created" started I0124 20:37:11.308905 89857 wait_for_controlled_pods.go:288] WaitForControlledPodsRunning: waiting for controlled pods measurement... I0124 20:37:16.317502 89857 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-lie7op-2), controlledBy(scheduler-throughput-deployment-0): Pods: 573 out of 1000 created, 428 running (428 updated), 144 pending scheduled, 1 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0124 20:37:21.338109 89857 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-lie7op-2), controlledBy(scheduler-throughput-deployment-0): Pods: 1000 out of 1000 created, 910 running (910 updated), 90 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0124 20:37:26.360576 89857 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-lie7op-2), controlledBy(scheduler-throughput-deployment-0): Pods: 1000 out of 1000 created, 1000 running (1000 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0124 20:37:26.360640 89857 wait_for_controlled_pods.go:365] WaitForControlledPodsRunning: running 1, deleted 0, timeout: 0, failed: 0 I0124 20:37:26.360652 89857 wait_for_controlled_pods.go:370] WaitForControlledPodsRunning: maxDuration=15.051794775s, operationTimeout=20m0s, ratio=0.01 I0124 20:37:26.360668 89857 wait_for_controlled_pods.go:384] WaitForControlledPodsRunning: 1/1 Deployments are running with all pods I0124 20:37:26.360681 89857 simple_test_executor.go:171] Step "[step: 10] Waiting for scheduler throughput pods to be created" ended I0124 20:37:26.360703 89857 simple_test_executor.go:149] Step "[step: 11] Collecting scheduler throughput measurements" started I0124 20:37:26.360735 89857 scheduling_throughput.go:154] SchedulingThroughput: gathering data I0124 20:37:26.360792 89857 pod_startup_latency.go:226] PodStartupLatency: labelSelector(group = scheduler-throughput): gathering pod startup latency measurement... ... skipping 33 lines ... I0124 20:37:26.804933 89857 simple_test_executor.go:149] Step "[step: 13] Waiting for scheduler throughput pods to be deleted" started I0124 20:37:26.804979 89857 wait_for_controlled_pods.go:288] WaitForControlledPodsRunning: waiting for controlled pods measurement... I0124 20:37:26.805048 89857 wait_for_pods.go:64] WaitForControlledPodsRunning: namespace(test-lie7op-2), controlledBy(scheduler-throughput-deployment-0): starting with timeout: 19m59.999982558s I0124 20:37:31.820997 89857 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-lie7op-2), controlledBy(scheduler-throughput-deployment-0): Pods: 458 out of 0 created, 458 running (458 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 97 terminating, 0 unknown, 0 runningButNotReady I0124 20:37:36.822237 89857 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-lie7op-2), controlledBy(scheduler-throughput-deployment-0): Pods: 0 out of 0 created, 0 running (0 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 36 terminating, 0 unknown, 0 runningButNotReady I0124 20:37:41.823162 89857 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-lie7op-2), controlledBy(scheduler-throughput-deployment-0): Pods: 0 out of 0 created, 0 running (0 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0124 20:37:41.823224 89857 wait_for_controlled_pods.go:365] WaitForControlledPodsRunning: running 0, deleted 1, timeout: 0, failed: 0 I0124 20:37:41.823237 89857 wait_for_controlled_pods.go:370] WaitForControlledPodsRunning: maxDuration=15.018151697s, operationTimeout=20m0s, ratio=0.01 I0124 20:37:41.823252 89857 wait_for_controlled_pods.go:384] WaitForControlledPodsRunning: 0/0 Deployments are running with all pods I0124 20:37:41.823272 89857 simple_test_executor.go:171] Step "[step: 13] Waiting for scheduler throughput pods to be deleted" ended I0124 20:37:41.823301 89857 simple_test_executor.go:149] Step "[step: 14] Starting latency pod measurements" started I0124 20:37:41.823369 89857 wait_for_controlled_pods.go:257] WaitForControlledPodsRunning: starting wait for controlled pods measurement... I0124 20:37:41.823411 89857 pod_startup_latency.go:132] PodStartupLatency: labelSelector(group = latency): starting pod startup latency measurement... ... skipping 999 lines ... I0124 20:39:26.047665 89857 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-lie7op-1), controlledBy(latency-deployment-494): Pods: 1 out of 1 created, 1 running (1 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0124 20:39:26.248029 89857 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-lie7op-1), controlledBy(latency-deployment-495): Pods: 1 out of 1 created, 1 running (1 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0124 20:39:26.449322 89857 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-lie7op-1), controlledBy(latency-deployment-496): Pods: 1 out of 1 created, 1 running (1 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0124 20:39:26.646945 89857 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-lie7op-1), controlledBy(latency-deployment-497): Pods: 1 out of 1 created, 1 running (1 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0124 20:39:26.847770 89857 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-lie7op-1), controlledBy(latency-deployment-498): Pods: 1 out of 1 created, 1 running (1 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0124 20:39:27.049194 89857 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-lie7op-1), controlledBy(latency-deployment-499): Pods: 1 out of 1 created, 1 running (1 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0124 20:39:27.475110 89857 wait_for_controlled_pods.go:365] WaitForControlledPodsRunning: running 500, deleted 0, timeout: 0, failed: 0 I0124 20:39:27.475156 89857 wait_for_controlled_pods.go:370] WaitForControlledPodsRunning: maxDuration=5.00140491s, operationTimeout=15m0s, ratio=0.01 I0124 20:39:27.475182 89857 wait_for_controlled_pods.go:384] WaitForControlledPodsRunning: 500/500 Deployments are running with all pods I0124 20:39:27.475208 89857 simple_test_executor.go:171] Step "[step: 16] Waiting for latency pods to be running" ended I0124 20:39:27.475227 89857 simple_test_executor.go:149] Step "[step: 17] Deleting latency pods" started I0124 20:39:27.513299 89857 wait_for_pods.go:64] WaitForControlledPodsRunning: namespace(test-lie7op-1), controlledBy(latency-deployment-0): starting with timeout: 14m59.999977804s I0124 20:39:27.612956 89857 wait_for_pods.go:64] WaitForControlledPodsRunning: namespace(test-lie7op-1), controlledBy(latency-deployment-1): starting with timeout: 14m59.999973949s ... skipping 495 lines ... I0124 20:40:17.212001 89857 wait_for_pods.go:64] WaitForControlledPodsRunning: namespace(test-lie7op-1), controlledBy(latency-deployment-497): starting with timeout: 14m59.999978734s I0124 20:40:17.313150 89857 wait_for_pods.go:64] WaitForControlledPodsRunning: namespace(test-lie7op-1), controlledBy(latency-deployment-498): starting with timeout: 14m59.999980413s I0124 20:40:17.412503 89857 simple_test_executor.go:171] Step "[step: 17] Deleting latency pods" ended I0124 20:40:17.412563 89857 simple_test_executor.go:149] Step "[step: 18] Waiting for latency pods to be deleted" started I0124 20:40:17.412601 89857 wait_for_controlled_pods.go:288] WaitForControlledPodsRunning: waiting for controlled pods measurement... I0124 20:40:17.412836 89857 wait_for_pods.go:64] WaitForControlledPodsRunning: namespace(test-lie7op-1), controlledBy(latency-deployment-499): starting with timeout: 14m59.999976591s I0124 20:40:22.460603 89857 wait_for_controlled_pods.go:365] WaitForControlledPodsRunning: running 0, deleted 500, timeout: 0, failed: 0 I0124 20:40:22.460647 89857 wait_for_controlled_pods.go:370] WaitForControlledPodsRunning: maxDuration=5.013296464s, operationTimeout=15m0s, ratio=0.01 I0124 20:40:22.460670 89857 wait_for_controlled_pods.go:384] WaitForControlledPodsRunning: 0/0 Deployments are running with all pods I0124 20:40:22.460714 89857 simple_test_executor.go:171] Step "[step: 18] Waiting for latency pods to be deleted" ended I0124 20:40:22.460738 89857 simple_test_executor.go:149] Step "[step: 19] Collecting pod startup latency" started I0124 20:40:22.460781 89857 pod_startup_latency.go:226] PodStartupLatency: labelSelector(group = latency): gathering pod startup latency measurement... I0124 20:40:22.890801 89857 phase_latency.go:141] PodStartupLatency: 100 worst pod_startup latencies: [{test-lie7op-1/latency-deployment-209-788cbb757-85s8c 2.415432548s} {test-lie7op-1/latency-deployment-468-7897c9f679-qz6pv 2.418967922s} {test-lie7op-1/latency-deployment-5-7d7575db7c-j8r68 2.430496473s} {test-lie7op-1/latency-deployment-497-77d8b78cf9-8s9br 2.434033457s} {test-lie7op-1/latency-deployment-64-5d5c9d7bcf-c65vw 2.436102856s} {test-lie7op-1/latency-deployment-23-7bbb55f647-xmg8k 2.437224826s} {test-lie7op-1/latency-deployment-69-68995f7c9f-67nsg 2.441608241s} {test-lie7op-1/latency-deployment-269-799879448f-mj7xz 2.447015604s} {test-lie7op-1/latency-deployment-298-59449f5d57-mqgpc 2.447180903s} {test-lie7op-1/latency-deployment-359-6979df78f7-mm86k 2.449410401s} {test-lie7op-1/latency-deployment-423-5b5b4b685-cvhlr 2.454033958s} {test-lie7op-1/latency-deployment-78-5cc65ff47f-vhpfq 2.455654435s} {test-lie7op-1/latency-deployment-238-7fb5c5d7cc-kd5l9 2.462077061s} {test-lie7op-1/latency-deployment-73-68476b846c-pfcgr 2.463993565s} {test-lie7op-1/latency-deployment-344-6f4c5d5fd9-dvr6d 2.464730554s} {test-lie7op-1/latency-deployment-417-7b67bf6fdc-c8xfn 2.469764526s} {test-lie7op-1/latency-deployment-374-695ffb49d7-l84pg 2.47429003s} {test-lie7op-1/latency-deployment-373-567b6fb64f-6f6sf 2.475752889s} {test-lie7op-1/latency-deployment-248-5c855fd5dc-87gtl 2.482485388s} {test-lie7op-1/latency-deployment-172-69cc877955-rn5s9 2.484756s} {test-lie7op-1/latency-deployment-408-59bcbccd6c-k4fhf 2.487501934s} {test-lie7op-1/latency-deployment-332-5bdcffc54c-tjlsz 2.493804169s} {test-lie7op-1/latency-deployment-94-64d789f575-gdkfz 2.501272906s} {test-lie7op-1/latency-deployment-49-cf4597b8c-b27bk 2.504675957s} {test-lie7op-1/latency-deployment-53-6694d96cc5-qplrk 2.526409911s} {test-lie7op-1/latency-deployment-193-567666d79c-9wmc2 2.527310522s} {test-lie7op-1/latency-deployment-399-6594b9cfdf-kklc9 2.532381457s} {test-lie7op-1/latency-deployment-307-78d67b4c99-s5ddw 2.532423595s} {test-lie7op-1/latency-deployment-363-6f4cc7445f-xhlbg 2.533242598s} {test-lie7op-1/latency-deployment-229-5b7fc8fccc-t8fq8 2.533660441s} {test-lie7op-1/latency-deployment-159-565b684d6f-dqpfd 2.535972885s} {test-lie7op-1/latency-deployment-14-8577448d87-ttl8r 2.541716175s} {test-lie7op-1/latency-deployment-309-5b7bc584c9-m2czt 2.542635415s} {test-lie7op-1/latency-deployment-50-5ccf8994ff-q459t 2.543008852s} {test-lie7op-1/latency-deployment-287-7974bfd769-4dj4v 2.544760758s} {test-lie7op-1/latency-deployment-264-78594c98c9-knz4x 2.544838266s} {test-lie7op-1/latency-deployment-30-cdfb9895f-4n7mq 2.545975491s} {test-lie7op-1/latency-deployment-89-78997566f9-gzxwv 2.546686276s} {test-lie7op-1/latency-deployment-297-5b47fd7cc9-rghns 2.549698404s} {test-lie7op-1/latency-deployment-55-6ff496f9f9-s9ccc 2.556839218s} {test-lie7op-1/latency-deployment-168-695cc8d95c-s8w4v 2.55862308s} {test-lie7op-1/latency-deployment-173-6f74865fbf-xx8lj 2.559632018s} {test-lie7op-1/latency-deployment-437-78fbcdd699-bbcnz 2.561789062s} {test-lie7op-1/latency-deployment-174-79dbdb7f59-d9gh7 2.565770194s} {test-lie7op-1/latency-deployment-40-67f4dfb4ff-hn8dl 2.579877563s} {test-lie7op-1/latency-deployment-103-64d96c87cf-vwqz6 2.590203906s} {test-lie7op-1/latency-deployment-8-6f5594d46c-zcnfm 2.590241157s} {test-lie7op-1/latency-deployment-148-6c79776cc-5lxdz 2.591131916s} {test-lie7op-1/latency-deployment-409-7b59577585-424fm 2.593427599s} {test-lie7op-1/latency-deployment-48-77b7478dff-snhkj 2.599687968s} {test-lie7op-1/latency-deployment-28-59ccd86595-lp7l5 2.602429981s} {test-lie7op-1/latency-deployment-334-6c5897cf6c-tl5h5 2.604941363s} {test-lie7op-1/latency-deployment-119-84fcf57945-2sw58 2.608866664s} {test-lie7op-1/latency-deployment-254-57f84497c7-t6dnb 2.611433825s} {test-lie7op-1/latency-deployment-473-ff97b46dc-q59l6 2.612592702s} {test-lie7op-1/latency-deployment-402-5c7b8b5b7-ls9mz 2.614302429s} {test-lie7op-1/latency-deployment-314-7b495757cf-2k4gh 2.617774754s} {test-lie7op-1/latency-deployment-492-75958df4b5-7fl5x 2.632658634s} {test-lie7op-1/latency-deployment-318-6858fc6679-mhqrm 2.637830072s} {test-lie7op-1/latency-deployment-10-cddb4d6d7-smh6l 2.642677023s} {test-lie7op-1/latency-deployment-283-5ffd9cd79c-gd4m9 2.645564679s} {test-lie7op-1/latency-deployment-98-59c486dbd7-rh9mj 2.647283079s} {test-lie7op-1/latency-deployment-403-7c7d978957-p7xm2 2.653151383s} {test-lie7op-1/latency-deployment-368-6f987d8df-7bnrz 2.662139517s} {test-lie7op-1/latency-deployment-113-549874bf7-g5554 2.666233548s} {test-lie7op-1/latency-deployment-348-65c6bd4f49-p4stx 2.669940693s} {test-lie7op-1/latency-deployment-448-944ffff9-kj6bx 2.681354159s} {test-lie7op-1/latency-deployment-398-985d766bc-nbs9n 2.686524476s} {test-lie7op-1/latency-deployment-74-7f6ff5884c-tjhf5 2.715616388s} {test-lie7op-1/latency-deployment-389-74d587bdc9-mr6bx 2.717442212s} {test-lie7op-1/latency-deployment-39-c99b6b6f5-7w2lv 2.732709785s} {test-lie7op-1/latency-deployment-224-6f664957f9-krnvk 2.733489964s} {test-lie7op-1/latency-deployment-163-687d84d77f-vhggl 2.74055985s} {test-lie7op-1/latency-deployment-154-d6648d467-sc5mz 2.747882459s} {test-lie7op-1/latency-deployment-104-874b5d545-4mtqq 2.74962821s} {test-lie7op-1/latency-deployment-428-58fccc8cb7-f2q42 2.780189293s} {test-lie7op-1/latency-deployment-353-67c7d8dbf-br7bw 2.785860944s} {test-lie7op-1/latency-deployment-15-6f64b4777f-ljzcc 2.791737245s} {test-lie7op-1/latency-deployment-343-56774cf887-qtnfm 2.795137363s} {test-lie7op-1/latency-deployment-404-57fff9dd87-9gh7f 2.80371659s} {test-lie7op-1/latency-deployment-54-78ffb869dc-xlxgv 2.804986088s} {test-lie7op-1/latency-deployment-354-85749f8b47-wm6kp 2.80521983s} {test-lie7op-1/latency-deployment-45-558c7b579-z7h7j 2.805243865s} {test-lie7op-1/latency-deployment-289-7578bf5445-4j298 2.821040875s} {test-lie7op-1/latency-deployment-339-7cf5d75cd9-c6pgv 2.824171079s} {test-lie7op-1/latency-deployment-379-67f66d7d9c-qzr98 2.82651353s} {test-lie7op-1/latency-deployment-288-7f9788c767-z8twm 2.830689343s} {test-lie7op-1/latency-deployment-29-7989cbb547-d4xxw 2.842462206s} {test-lie7op-1/latency-deployment-319-6c78bcb66f-q4gtv 2.867560424s} {test-lie7op-1/latency-deployment-299-577c557999-s9rzt 2.87388626s} {test-lie7op-1/latency-deployment-294-ff686499c-89zfb 2.880977509s} {test-lie7op-1/latency-deployment-279-5d65cb4cff-jbgx7 2.882712673s} {test-lie7op-1/latency-deployment-349-5bc8959999-95b4q 2.887121646s} {test-lie7op-1/latency-deployment-364-775cf57b8f-kzcmq 2.890320825s} {test-lie7op-1/latency-deployment-394-595c5448ff-jx4dv 2.893901208s} {test-lie7op-1/latency-deployment-84-74dc457967-gn6cv 2.913521011s} {test-lie7op-1/latency-deployment-244-6bbf48dd7-zclkg 2.926278137s} {test-lie7op-1/latency-deployment-234-77dc9d96c7-vf4bh 2.926420183s} {test-lie7op-1/latency-deployment-20-857fcc5d8c-7j7xr 2.933721283s} {test-lie7op-1/latency-deployment-384-846b98d5c-2z9ts 2.966194354s}] ... skipping 591 lines ... I0124 20:41:03.999633 89857 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-lie7op-1), controlledBy(small-deployment-7): Pods: 4 out of 4 created, 4 running (4 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0124 20:41:04.192485 89857 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-lie7op-1), controlledBy(small-deployment-55): Pods: 7 out of 7 created, 7 running (7 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0124 20:41:04.391274 89857 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-lie7op-1), controlledBy(small-deployment-179): Pods: 2 out of 2 created, 2 running (2 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0124 20:41:04.609463 89857 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-lie7op-1), controlledBy(small-deployment-297): Pods: 6 out of 6 created, 6 running (6 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0124 20:41:04.631686 89857 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-lie7op-1), controlledBy(small-deployment-228): Pods: 3 out of 3 created, 3 running (3 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0124 20:41:04.732995 89857 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-lie7op-1), controlledBy(small-deployment-155): Pods: 2 out of 2 created, 2 running (2 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0124 20:41:04.769234 89857 wait_for_controlled_pods.go:365] WaitForControlledPodsRunning: running 3, deleted 0, timeout: 0, failed: 0 I0124 20:41:04.769291 89857 wait_for_controlled_pods.go:370] WaitForControlledPodsRunning: maxDuration=5.004096003s, operationTimeout=15m0s, ratio=0.01 I0124 20:41:04.769308 89857 wait_for_controlled_pods.go:384] WaitForControlledPodsRunning: 3/3 Jobs are running with all pods I0124 20:41:04.976927 89857 wait_for_controlled_pods.go:365] WaitForControlledPodsRunning: running 323, deleted 0, timeout: 0, failed: 0 I0124 20:41:04.976965 89857 wait_for_controlled_pods.go:370] WaitForControlledPodsRunning: maxDuration=5.00696829s, operationTimeout=15m0s, ratio=0.01 I0124 20:41:04.976985 89857 wait_for_controlled_pods.go:384] WaitForControlledPodsRunning: 323/323 Deployments are running with all pods I0124 20:41:07.378538 89857 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-lie7op-1), controlledBy(daemonset-0): Pods: 98 out of 100 created, 90 running (69 updated), 8 pending scheduled, 0 not scheduled, 0 inactive, 2 terminating, 0 unknown, 0 runningButNotReady I0124 20:41:08.646253 89857 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-lie7op-1), controlledBy(medium-statefulset-0): Pods: 44 out of 44 created, 40 running (40 updated), 4 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0124 20:41:12.382130 89857 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-lie7op-1), controlledBy(daemonset-0): Pods: 99 out of 100 created, 91 running (91 updated), 8 pending scheduled, 0 not scheduled, 0 inactive, 1 terminating, 0 unknown, 0 runningButNotReady I0124 20:41:13.647816 89857 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-lie7op-1), controlledBy(medium-statefulset-0): Pods: 44 out of 44 created, 42 running (42 updated), 2 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0124 20:41:17.384443 89857 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-lie7op-1), controlledBy(daemonset-0): Pods: 100 out of 100 created, 100 running (100 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0124 20:41:17.384510 89857 wait_for_controlled_pods.go:365] WaitForControlledPodsRunning: running 1, deleted 0, timeout: 0, failed: 0 I0124 20:41:17.384540 89857 wait_for_controlled_pods.go:370] WaitForControlledPodsRunning: maxDuration=30.113279916s, operationTimeout=15m0s, ratio=0.03 I0124 20:41:17.384556 89857 wait_for_controlled_pods.go:384] WaitForControlledPodsRunning: 1/1 DaemonSets are running with all pods I0124 20:41:18.648996 89857 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-lie7op-1), controlledBy(medium-statefulset-0): Pods: 44 out of 44 created, 44 running (44 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0124 20:41:18.649061 89857 wait_for_controlled_pods.go:365] WaitForControlledPodsRunning: running 2, deleted 0, timeout: 0, failed: 0 I0124 20:41:18.649075 89857 wait_for_controlled_pods.go:370] WaitForControlledPodsRunning: maxDuration=30.010106555s, operationTimeout=15m0s, ratio=0.03 I0124 20:41:18.649089 89857 wait_for_controlled_pods.go:384] WaitForControlledPodsRunning: 2/2 StatefulSets are running with all pods I0124 20:41:18.649116 89857 simple_test_executor.go:171] Step "[step: 22] Waiting for 'scale and update objects' to be completed" ended I0124 20:41:18.649138 89857 simple_test_executor.go:149] Step "[step: 23] Starting measurement for 'delete objects'" started I0124 20:41:18.649198 89857 wait_for_controlled_pods.go:254] WaitForControlledPodsRunning: wait for controlled pods measurement already running I0124 20:41:18.649229 89857 wait_for_controlled_pods.go:254] WaitForControlledPodsRunning: wait for controlled pods measurement already running ... skipping 333 lines ... I0124 20:44:52.583733 89857 simple_test_executor.go:149] Step "[step: 25] Waiting for 'delete objects' to be completed" started I0124 20:44:52.583789 89857 wait_for_pods.go:64] WaitForControlledPodsRunning: namespace(test-lie7op-1), controlledBy(small-deployment-33): starting with timeout: 14m59.999976951s I0124 20:44:52.583838 89857 wait_for_controlled_pods.go:288] WaitForControlledPodsRunning: waiting for controlled pods measurement... I0124 20:44:52.583841 89857 wait_for_controlled_pods.go:288] WaitForControlledPodsRunning: waiting for controlled pods measurement... I0124 20:44:52.583856 89857 wait_for_controlled_pods.go:288] WaitForControlledPodsRunning: waiting for controlled pods measurement... I0124 20:44:52.583901 89857 wait_for_controlled_pods.go:288] WaitForControlledPodsRunning: waiting for controlled pods measurement... I0124 20:44:57.617871 89857 wait_for_controlled_pods.go:365] WaitForControlledPodsRunning: running 0, deleted 2, timeout: 0, failed: 0 I0124 20:44:57.617912 89857 wait_for_controlled_pods.go:370] WaitForControlledPodsRunning: maxDuration=5.000322753s, operationTimeout=15m0s, ratio=0.01 I0124 20:44:57.617932 89857 wait_for_controlled_pods.go:384] WaitForControlledPodsRunning: 0/0 StatefulSets are running with all pods I0124 20:44:57.619052 89857 wait_for_controlled_pods.go:365] WaitForControlledPodsRunning: running 0, deleted 1, timeout: 0, failed: 0 I0124 20:44:57.619075 89857 wait_for_controlled_pods.go:370] WaitForControlledPodsRunning: maxDuration=5.000909062s, operationTimeout=15m0s, ratio=0.01 I0124 20:44:57.619089 89857 wait_for_controlled_pods.go:384] WaitForControlledPodsRunning: 0/0 DaemonSets are running with all pods I0124 20:44:57.619290 89857 wait_for_controlled_pods.go:365] WaitForControlledPodsRunning: running 0, deleted 323, timeout: 0, failed: 0 I0124 20:44:57.619327 89857 wait_for_controlled_pods.go:370] WaitForControlledPodsRunning: maxDuration=5.00132992s, operationTimeout=15m0s, ratio=0.01 I0124 20:44:57.619346 89857 wait_for_controlled_pods.go:384] WaitForControlledPodsRunning: 0/0 Deployments are running with all pods I0124 20:44:57.619343 89857 wait_for_controlled_pods.go:365] WaitForControlledPodsRunning: running 0, deleted 3, timeout: 0, failed: 0 I0124 20:44:57.619372 89857 wait_for_controlled_pods.go:370] WaitForControlledPodsRunning: maxDuration=5.001236851s, operationTimeout=15m0s, ratio=0.01 I0124 20:44:57.619389 89857 wait_for_controlled_pods.go:384] WaitForControlledPodsRunning: 0/0 Jobs are running with all pods I0124 20:44:57.635500 89857 wait_for_pvcs.go:78] WaitForBoundPVCs: labelSelector(group = load): PVCs: 0 out of 0 created, 0 bound, 0 pending, 0 lost I0124 20:44:57.635533 89857 simple_test_executor.go:171] Step "[step: 25] Waiting for 'delete objects' to be completed" ended I0124 20:44:57.635563 89857 simple_test_executor.go:149] Step "[step: 26] delete objects configmaps and secrets" started I0124 20:45:29.906910 89857 simple_test_executor.go:171] Step "[step: 26] delete objects configmaps and secrets" ended ... skipping 250 lines ... I0124 20:46:59.974049 89857 simple_test_executor.go:171] Step "[step: 04] Creating huge-service pods" ended I0124 20:46:59.974071 89857 simple_test_executor.go:149] Step "[step: 05] Waiting for huge-service pods to be created" started I0124 20:46:59.974107 89857 wait_for_controlled_pods.go:288] WaitForControlledPodsRunning: waiting for controlled pods measurement... I0124 20:47:04.981976 89857 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-h11ayl-1), controlledBy(huge-service-deployment-0): Pods: 569 out of 1000 created, 374 running (374 updated), 189 pending scheduled, 6 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0124 20:47:09.998835 89857 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-h11ayl-1), controlledBy(huge-service-deployment-0): Pods: 1000 out of 1000 created, 891 running (891 updated), 109 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0124 20:47:15.018907 89857 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-h11ayl-1), controlledBy(huge-service-deployment-0): Pods: 1000 out of 1000 created, 1000 running (1000 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0124 20:47:15.018983 89857 wait_for_controlled_pods.go:365] WaitForControlledPodsRunning: running 1, deleted 0, timeout: 0, failed: 0 I0124 20:47:15.018997 89857 wait_for_controlled_pods.go:370] WaitForControlledPodsRunning: maxDuration=15.044974047s, operationTimeout=30m0s, ratio=0.01 I0124 20:47:15.019011 89857 wait_for_controlled_pods.go:384] WaitForControlledPodsRunning: 1/1 Deployments are running with all pods I0124 20:47:15.019028 89857 simple_test_executor.go:171] Step "[step: 05] Waiting for huge-service pods to be created" ended I0124 20:47:15.019065 89857 simple_test_executor.go:149] Step "[step: 06] Updating huge-service pods" started I0124 20:47:15.094406 89857 wait_for_pods.go:64] WaitForControlledPodsRunning: namespace(test-h11ayl-1), controlledBy(huge-service-deployment-0): starting with timeout: 29m59.999968663s I0124 20:47:15.094628 89857 simple_test_executor.go:171] Step "[step: 06] Updating huge-service pods" ended ... skipping 4 lines ... I0124 20:47:25.672860 89857 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-h11ayl-1), controlledBy(huge-service-deployment-0): Pods: 976 out of 1000 created, 962 running (517 updated), 14 pending scheduled, 0 not scheduled, 0 inactive, 156 terminating, 0 unknown, 0 runningButNotReady W0124 20:47:30.808772 89857 wait_for_pods.go:119] WaitForControlledPodsRunning: namespace(test-h11ayl-1), controlledBy(huge-service-deployment-0): 324 pods appeared: huge-service-deployment-0-5f48d94767-tpblp, huge-service-deployment-0-5f48d94767-vf5gv, huge-service-deployment-0-5f48d94767-rjqdp, huge-service-deployment-0-5f48d94767-2pjwn, huge-service-deployment-0-5f48d94767-nn6cw, huge-service-deployment-0-5f48d94767-prwsv, huge-service-deployment-0-5f48d94767-d46b6, huge-service-deployment-0-5f48d94767-bp46c, huge-service-deployment-0-5f48d94767-85j9l, huge-service-deployment-0-5f48d94767-8khx4, huge-service-deployment-0-5f48d94767-w958f, huge-service-deployment-0-5f48d94767-4jlmj, huge-service-deployment-0-5f48d94767-sj4bt, huge-service-deployment-0-5f48d94767-gxvfc, huge-service-deployment-0-5f48d94767-j627z, huge-service-deployment-0-5f48d94767-5jt5c, huge-service-deployment-0-5f48d94767-shxxf, huge-service-deployment-0-5f48d94767-6275p, huge-service-deployment-0-5f48d94767-dt49x, huge-service-deployment-0-5f48d94767-6tdlm, huge-service-deployment-0-5f48d94767-kkcgs, huge-service-deployment-0-5f48d94767-rcbdj, huge-service-deployment-0-5f48d94767-dnssl, huge-service-deployment-0-5f48d94767-6vwpm, huge-service-deployment-0-5f48d94767-txdz6, huge-service-deployment-0-5f48d94767-2kxk7, huge-service-deployment-0-5f48d94767-cvpvv, huge-service-deployment-0-5f48d94767-tjsvc, huge-service-deployment-0-5f48d94767-lv54m, huge-service-deployment-0-5f48d94767-dskzm, huge-service-deployment-0-5f48d94767-7d99f, huge-service-deployment-0-5f48d94767-x4g94, huge-service-deployment-0-5f48d94767-zg7dh, huge-service-deployment-0-5f48d94767-qjstx, huge-service-deployment-0-5f48d94767-8wpss, huge-service-deployment-0-5f48d94767-4cnm2, huge-service-deployment-0-5f48d94767-l9bds, huge-service-deployment-0-5f48d94767-d6h9w, huge-service-deployment-0-5f48d94767-cfznn, huge-service-deployment-0-5f48d94767-jkmqg, huge-service-deployment-0-5f48d94767-xn4wh, huge-service-deployment-0-5f48d94767-hkw77, huge-service-deployment-0-5f48d94767-mtsrg, huge-service-deployment-0-5f48d94767-kgsps, huge-service-deployment-0-5f48d94767-zctrv, huge-service-deployment-0-5f48d94767-v79gl, huge-service-deployment-0-5f48d94767-r4xr8, huge-service-deployment-0-5f48d94767-9rb7d, huge-service-deployment-0-5f48d94767-scrls, huge-service-deployment-0-5f48d94767-2tlvn, huge-service-deployment-0-5f48d94767-twfj8, huge-service-deployment-0-5f48d94767-fn84l, huge-service-deployment-0-5f48d94767-2rvxb, huge-service-deployment-0-5f48d94767-b8qf6, huge-service-deployment-0-5f48d94767-gdlkr, huge-service-deployment-0-5f48d94767-96gxm, huge-service-deployment-0-5f48d94767-574mf, huge-service-deployment-0-5f48d94767-cs6pg, huge-service-deployment-0-5f48d94767-68jkb, huge-service-deployment-0-5f48d94767-lkpp4, huge-service-deployment-0-5f48d94767-wkvjd, huge-service-deployment-0-5f48d94767-p25s7, huge-service-deployment-0-5f48d94767-54fqz, huge-service-deployment-0-5f48d94767-2qzhp, huge-service-deployment-0-5f48d94767-jd87b, huge-service-deployment-0-5f48d94767-d68m5, huge-service-deployment-0-5f48d94767-j5pll, huge-service-deployment-0-5f48d94767-vp2mx, huge-service-deployment-0-5f48d94767-npnwp, huge-service-deployment-0-5f48d94767-crh4k, huge-service-deployment-0-5f48d94767-lsvtx, huge-service-deployment-0-5f48d94767-2km4c, huge-service-deployment-0-5f48d94767-rtx4x, huge-service-deployment-0-5f48d94767-jrxhm, huge-service-deployment-0-5f48d94767-p86v4, huge-service-deployment-0-5f48d94767-8x4vv, huge-service-deployment-0-5f48d94767-dcgfb, huge-service-deployment-0-5f48d94767-mw9xk, huge-service-deployment-0-5f48d94767-8lkx9, huge-service-deployment-0-5f48d94767-5c6lz, huge-service-deployment-0-5f48d94767-ghcwp, huge-service-deployment-0-5f48d94767-v46n4, huge-service-deployment-0-5f48d94767-pxwv4, huge-service-deployment-0-5f48d94767-7c5vw, huge-service-deployment-0-5f48d94767-dp5kq, huge-service-deployment-0-5f48d94767-wcvgw, huge-service-deployment-0-5f48d94767-kks55, huge-service-deployment-0-5f48d94767-z494s, huge-service-deployment-0-5f48d94767-t87bx, huge-service-deployment-0-5f48d94767-vxm4q, huge-service-deployment-0-5f48d94767-65pqr, huge-service-deployment-0-5f48d94767-p97qg, huge-service-deployment-0-5f48d94767-j5v5w, huge-service-deployment-0-5f48d94767-gpb2d, huge-service-deployment-0-5f48d94767-m6qfx, huge-service-deployment-0-5f48d94767-qznqd, huge-service-deployment-0-5f48d94767-zkl4j, huge-service-deployment-0-5f48d94767-649fn, huge-service-deployment-0-5f48d94767-sjnxd, huge-service-deployment-0-5f48d94767-sgkwk, huge-service-deployment-0-5f48d94767-wqrgj, huge-service-deployment-0-5f48d94767-dlrpx, huge-service-deployment-0-5f48d94767-sbwcg, huge-service-deployment-0-5f48d94767-s5p2p, huge-service-deployment-0-5f48d94767-bfkm4, huge-service-deployment-0-5f48d94767-vs9ff, huge-service-deployment-0-5f48d94767-gpghk, huge-service-deployment-0-5f48d94767-drvcp, huge-service-deployment-0-5f48d94767-drnlk, huge-service-deployment-0-5f48d94767-pgl2c, huge-service-deployment-0-5f48d94767-r8mpc, huge-service-deployment-0-5f48d94767-7s652, huge-service-deployment-0-5f48d94767-rjsh5, huge-service-deployment-0-5f48d94767-jz5jp, huge-service-deployment-0-5f48d94767-9pk67, huge-service-deployment-0-5f48d94767-4gdxw, huge-service-deployment-0-5f48d94767-rd44v, huge-service-deployment-0-5f48d94767-hqgcs, huge-service-deployment-0-5f48d94767-lgwgk, huge-service-deployment-0-5f48d94767-k5tr6, huge-service-deployment-0-5f48d94767-q5tc4, huge-service-deployment-0-5f48d94767-dqwzd, huge-service-deployment-0-5f48d94767-24tcl, huge-service-deployment-0-5f48d94767-88thr, huge-service-deployment-0-5f48d94767-vdwsz, huge-service-deployment-0-5f48d94767-rzwfp, huge-service-deployment-0-5f48d94767-bwlmj, huge-service-deployment-0-5f48d94767-qldzs, huge-service-deployment-0-5f48d94767-vz5d5, huge-service-deployment-0-5f48d94767-7d6bk, huge-service-deployment-0-5f48d94767-67727, huge-service-deployment-0-5f48d94767-g4hnn, huge-service-deployment-0-5f48d94767-phxg6, huge-service-deployment-0-5f48d94767-krvxl, huge-service-deployment-0-5f48d94767-trxgh, huge-service-deployment-0-5f48d94767-bwdwm, huge-service-deployment-0-5f48d94767-vkjjk, huge-service-deployment-0-5f48d94767-cqcs5, huge-service-deployment-0-5f48d94767-plkhw, huge-service-deployment-0-5f48d94767-fs67m, huge-service-deployment-0-5f48d94767-cmz7j, huge-service-deployment-0-5f48d94767-pwjkq, huge-service-deployment-0-5f48d94767-z6fwx, huge-service-deployment-0-5f48d94767-hsrqn, huge-service-deployment-0-5f48d94767-bdlsj, huge-service-deployment-0-5f48d94767-qp9j6, huge-service-deployment-0-5f48d94767-pmbj8, huge-service-deployment-0-5f48d94767-rzdch, huge-service-deployment-0-5f48d94767-5s2kc, huge-service-deployment-0-5f48d94767-gxrvl, huge-service-deployment-0-5f48d94767-b2qxd, huge-service-deployment-0-5f48d94767-svshj, huge-service-deployment-0-5f48d94767-4c6sx, huge-service-deployment-0-5f48d94767-mcpb2, huge-service-deployment-0-5f48d94767-kmgcc, huge-service-deployment-0-5f48d94767-dplfc, huge-service-deployment-0-5f48d94767-rk26m, huge-service-deployment-0-5f48d94767-mqrxk, huge-service-deployment-0-5f48d94767-9hdnq, huge-service-deployment-0-5f48d94767-ft66p, huge-service-deployment-0-5f48d94767-xd6k4, huge-service-deployment-0-5f48d94767-xlwph, huge-service-deployment-0-5f48d94767-qpvnq, huge-service-deployment-0-5f48d94767-klct4, huge-service-deployment-0-5f48d94767-xvkk8, huge-service-deployment-0-5f48d94767-b7cm9, huge-service-deployment-0-5f48d94767-d244c, huge-service-deployment-0-5f48d94767-xb4gd, huge-service-deployment-0-5f48d94767-hfrmg, huge-service-deployment-0-5f48d94767-nz85s, huge-service-deployment-0-5f48d94767-68m89, huge-service-deployment-0-5f48d94767-7wjcv, huge-service-deployment-0-5f48d94767-64h69, huge-service-deployment-0-5f48d94767-6djm7, huge-service-deployment-0-5f48d94767-59brz, huge-service-deployment-0-5f48d94767-nlmvb, huge-service-deployment-0-5f48d94767-g495r, huge-service-deployment-0-5f48d94767-qksqp, huge-service-deployment-0-5f48d94767-ht2nn, huge-service-deployment-0-5f48d94767-qvcws, huge-service-deployment-0-5f48d94767-pzzps, huge-service-deployment-0-5f48d94767-k6fdt, huge-service-deployment-0-5f48d94767-gr9ch, huge-service-deployment-0-5f48d94767-nqbqf, huge-service-deployment-0-5f48d94767-bghmg, huge-service-deployment-0-5f48d94767-g9m4s, huge-service-deployment-0-5f48d94767-6cv95, huge-service-deployment-0-5f48d94767-r4wfx, huge-service-deployment-0-5f48d94767-qcm4c, huge-service-deployment-0-5f48d94767-kd7ll, huge-service-deployment-0-5f48d94767-qqfx8, huge-service-deployment-0-5f48d94767-gfc7f, huge-service-deployment-0-5f48d94767-ppxgc, huge-service-deployment-0-5f48d94767-f9vst, huge-service-deployment-0-5f48d94767-qqlpd, huge-service-deployment-0-5f48d94767-swwng, huge-service-deployment-0-5f48d94767-9d8rj, huge-service-deployment-0-5f48d94767-wwdrj, huge-service-deployment-0-5f48d94767-6k5pn, huge-service-deployment-0-5f48d94767-5pfqb, huge-service-deployment-0-5f48d94767-667mn, huge-service-deployment-0-5f48d94767-mf254, huge-service-deployment-0-5f48d94767-9nwjk, huge-service-deployment-0-5f48d94767-n9w87, huge-service-deployment-0-5f48d94767-bx7x2, huge-service-deployment-0-5f48d94767-hv7ht, huge-service-deployment-0-5f48d94767-dnz5m, huge-service-deployment-0-5f48d94767-lq8qh, huge-service-deployment-0-5f48d94767-q4hsp, huge-service-deployment-0-5f48d94767-sz26q, huge-service-deployment-0-5f48d94767-v2d8j, huge-service-deployment-0-5f48d94767-hpk2h, huge-service-deployment-0-5f48d94767-mxbdl, huge-service-deployment-0-5f48d94767-4j7pn, huge-service-deployment-0-5f48d94767-mfn84, huge-service-deployment-0-5f48d94767-hsrp8, huge-service-deployment-0-5f48d94767-cbfbl, huge-service-deployment-0-5f48d94767-sp6zn, huge-service-deployment-0-5f48d94767-fqxqw, huge-service-deployment-0-5f48d94767-nrr6b, huge-service-deployment-0-5f48d94767-b27c8, huge-service-deployment-0-5f48d94767-ksjj9, huge-service-deployment-0-5f48d94767-d8cz8, huge-service-deployment-0-5f48d94767-2xqv9, huge-service-deployment-0-5f48d94767-j4lbd, huge-service-deployment-0-5f48d94767-nm9p8, huge-service-deployment-0-5f48d94767-bvrjc, huge-service-deployment-0-5f48d94767-rpns8, huge-service-deployment-0-5f48d94767-8f8lk, huge-service-deployment-0-5f48d94767-4fqcv, huge-service-deployment-0-5f48d94767-rvnnc, huge-service-deployment-0-5f48d94767-z89tn, huge-service-deployment-0-5f48d94767-s7z4g, huge-service-deployment-0-5f48d94767-2nbjx, huge-service-deployment-0-5f48d94767-v8tgp, huge-service-deployment-0-5f48d94767-mts7v, huge-service-deployment-0-5f48d94767-48vwn, huge-service-deployment-0-5f48d94767-h58zw, huge-service-deployment-0-5f48d94767-dp4b9, huge-service-deployment-0-5f48d94767-88xbc, huge-service-deployment-0-5f48d94767-tb2nq, huge-service-deployment-0-5f48d94767-chbl9, huge-service-deployment-0-5f48d94767-t2mzs, huge-service-deployment-0-5f48d94767-6tjzt, huge-service-deployment-0-5f48d94767-rc9bl, huge-service-deployment-0-5f48d94767-mtc9c, huge-service-deployment-0-5f48d94767-s4gsq, huge-service-deployment-0-5f48d94767-fbr5n, huge-service-deployment-0-5f48d94767-ql84m, huge-service-deployment-0-5f48d94767-bx4c9, huge-service-deployment-0-5f48d94767-b7v72, huge-service-deployment-0-5f48d94767-gf2sc, huge-service-deployment-0-5f48d94767-j6cwh, huge-service-deployment-0-5f48d94767-wfpfk, huge-service-deployment-0-5f48d94767-lsr4d, huge-service-deployment-0-5f48d94767-nxf4g, huge-service-deployment-0-5f48d94767-z4vxj, huge-service-deployment-0-5f48d94767-pf8dh, huge-service-deployment-0-5f48d94767-w4nqq, huge-service-deployment-0-5f48d94767-pr2f5, huge-service-deployment-0-5f48d94767-bnx6k, huge-service-deployment-0-5f48d94767-2f7cq, huge-service-deployment-0-5f48d94767-2grfz, huge-service-deployment-0-5f48d94767-4rprl, huge-service-deployment-0-5f48d94767-c4h99, huge-service-deployment-0-5f48d94767-w5w8k, huge-service-deployment-0-5f48d94767-tg4mn, huge-service-deployment-0-5f48d94767-knkdm, huge-service-deployment-0-5f48d94767-vp4mc, huge-service-deployment-0-5f48d94767-tz5nq, huge-service-deployment-0-5f48d94767-rmps6, huge-service-deployment-0-5f48d94767-gnlxp, huge-service-deployment-0-5f48d94767-kgblh, huge-service-deployment-0-5f48d94767-cx86m, huge-service-deployment-0-5f48d94767-tnh8l, huge-service-deployment-0-5f48d94767-pqffh, huge-service-deployment-0-5f48d94767-n5nhc, huge-service-deployment-0-5f48d94767-2bfl6, huge-service-deployment-0-5f48d94767-xjfkd, huge-service-deployment-0-5f48d94767-2z7gq, huge-service-deployment-0-5f48d94767-sjqqf, huge-service-deployment-0-5f48d94767-t2m9b, huge-service-deployment-0-5f48d94767-nfp9h, huge-service-deployment-0-5f48d94767-pnslr, huge-service-deployment-0-5f48d94767-4rj9h, huge-service-deployment-0-5f48d94767-9g7v4, huge-service-deployment-0-5f48d94767-tks65, huge-service-deployment-0-5f48d94767-xxl22, huge-service-deployment-0-5f48d94767-vcdch, huge-service-deployment-0-5f48d94767-dc8xs, huge-service-deployment-0-5f48d94767-n5vfr, huge-service-deployment-0-5f48d94767-knqc8, huge-service-deployment-0-5f48d94767-9f925, huge-service-deployment-0-5f48d94767-2zmxp, huge-service-deployment-0-5f48d94767-hr8cw, huge-service-deployment-0-5f48d94767-zb2dl, huge-service-deployment-0-5f48d94767-f7nbl, huge-service-deployment-0-5f48d94767-x76c6, huge-service-deployment-0-5f48d94767-v54fc, huge-service-deployment-0-5f48d94767-kr65q, huge-service-deployment-0-5f48d94767-hcz4f, huge-service-deployment-0-5f48d94767-7rtpg, huge-service-deployment-0-5f48d94767-vsn98, huge-service-deployment-0-5f48d94767-9fp5l, huge-service-deployment-0-5f48d94767-qtbtk, huge-service-deployment-0-5f48d94767-gp57m, huge-service-deployment-0-5f48d94767-jmgfm, huge-service-deployment-0-5f48d94767-f8mvn, huge-service-deployment-0-5f48d94767-9m4lt, huge-service-deployment-0-5f48d94767-rmw55, huge-service-deployment-0-5f48d94767-mj8zc, huge-service-deployment-0-5f48d94767-cwvxc, huge-service-deployment-0-5f48d94767-fwvw2, huge-service-deployment-0-5f48d94767-xlrjl, huge-service-deployment-0-5f48d94767-7wjz5, huge-service-deployment-0-5f48d94767-zll8w, huge-service-deployment-0-5f48d94767-j4nnv, huge-service-deployment-0-5f48d94767-9b2qg, huge-service-deployment-0-5f48d94767-pmv4r, huge-service-deployment-0-5f48d94767-zgspc, huge-service-deployment-0-5f48d94767-nrbpm, huge-service-deployment-0-5f48d94767-dvbv8, huge-service-deployment-0-5f48d94767-j8j8s, huge-service-deployment-0-5f48d94767-5fkff I0124 20:47:30.808836 89857 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-h11ayl-1), controlledBy(huge-service-deployment-0): Pods: 1115 out of 1000 created, 1003 running (743 updated), 112 pending scheduled, 0 not scheduled, 0 inactive, 62 terminating, 0 unknown, 0 runningButNotReady W0124 20:47:35.828483 89857 wait_for_pods.go:119] WaitForControlledPodsRunning: namespace(test-h11ayl-1), controlledBy(huge-service-deployment-0): 145 pods appeared: huge-service-deployment-0-5f48d94767-fg976, huge-service-deployment-0-5f48d94767-ns62n, huge-service-deployment-0-5f48d94767-px2gr, huge-service-deployment-0-5f48d94767-wxfgr, huge-service-deployment-0-5f48d94767-5bt9b, huge-service-deployment-0-5f48d94767-fznpr, huge-service-deployment-0-5f48d94767-hm4cz, huge-service-deployment-0-5f48d94767-kbxzf, huge-service-deployment-0-5f48d94767-l687t, huge-service-deployment-0-5f48d94767-kgh9l, huge-service-deployment-0-5f48d94767-df7xm, huge-service-deployment-0-5f48d94767-w6b8n, huge-service-deployment-0-5f48d94767-7xlrn, huge-service-deployment-0-5f48d94767-hnhhr, huge-service-deployment-0-5f48d94767-rklvf, huge-service-deployment-0-5f48d94767-r8v5f, huge-service-deployment-0-5f48d94767-fvsmv, huge-service-deployment-0-5f48d94767-v28qh, huge-service-deployment-0-5f48d94767-526pn, huge-service-deployment-0-5f48d94767-qc57n, huge-service-deployment-0-5f48d94767-66chj, huge-service-deployment-0-5f48d94767-7jzjb, huge-service-deployment-0-5f48d94767-xrcbp, huge-service-deployment-0-5f48d94767-mc96b, huge-service-deployment-0-5f48d94767-blpbk, huge-service-deployment-0-5f48d94767-7q6hp, huge-service-deployment-0-5f48d94767-kddk2, huge-service-deployment-0-5f48d94767-2c5jd, huge-service-deployment-0-5f48d94767-kls8k, huge-service-deployment-0-5f48d94767-hxpr4, huge-service-deployment-0-5f48d94767-tqrh5, huge-service-deployment-0-5f48d94767-n2ccs, huge-service-deployment-0-5f48d94767-fhcl4, huge-service-deployment-0-5f48d94767-srd25, huge-service-deployment-0-5f48d94767-gzssx, huge-service-deployment-0-5f48d94767-2mv8k, huge-service-deployment-0-5f48d94767-r8pwh, huge-service-deployment-0-5f48d94767-mn445, huge-service-deployment-0-5f48d94767-r6hls, huge-service-deployment-0-5f48d94767-svkjc, huge-service-deployment-0-5f48d94767-prs47, huge-service-deployment-0-5f48d94767-p8brh, huge-service-deployment-0-5f48d94767-xn8bc, huge-service-deployment-0-5f48d94767-h7n52, huge-service-deployment-0-5f48d94767-9dq55, huge-service-deployment-0-5f48d94767-sngd4, huge-service-deployment-0-5f48d94767-k8s8g, huge-service-deployment-0-5f48d94767-x54wq, huge-service-deployment-0-5f48d94767-hwp26, huge-service-deployment-0-5f48d94767-7b7mw, huge-service-deployment-0-5f48d94767-c776d, huge-service-deployment-0-5f48d94767-77f88, huge-service-deployment-0-5f48d94767-dlrpt, huge-service-deployment-0-5f48d94767-ng5vl, huge-service-deployment-0-5f48d94767-xwqcg, huge-service-deployment-0-5f48d94767-p9xzq, huge-service-deployment-0-5f48d94767-r7lqj, huge-service-deployment-0-5f48d94767-g99st, huge-service-deployment-0-5f48d94767-7q4nt, huge-service-deployment-0-5f48d94767-qqpn2, huge-service-deployment-0-5f48d94767-p4jhn, huge-service-deployment-0-5f48d94767-rjmzl, huge-service-deployment-0-5f48d94767-j7j64, huge-service-deployment-0-5f48d94767-7lgsq, huge-service-deployment-0-5f48d94767-zsf2r, huge-service-deployment-0-5f48d94767-cpbjv, huge-service-deployment-0-5f48d94767-hlwrn, huge-service-deployment-0-5f48d94767-d7f84, huge-service-deployment-0-5f48d94767-r22rb, huge-service-deployment-0-5f48d94767-vvhz9, huge-service-deployment-0-5f48d94767-tktvm, huge-service-deployment-0-5f48d94767-k24vx, huge-service-deployment-0-5f48d94767-4dxcc, huge-service-deployment-0-5f48d94767-fft6w, huge-service-deployment-0-5f48d94767-mhhht, huge-service-deployment-0-5f48d94767-bf5r7, huge-service-deployment-0-5f48d94767-8jzrd, huge-service-deployment-0-5f48d94767-xrq7r, huge-service-deployment-0-5f48d94767-26tzg, huge-service-deployment-0-5f48d94767-wrtd5, huge-service-deployment-0-5f48d94767-rhgqq, huge-service-deployment-0-5f48d94767-qwdzt, huge-service-deployment-0-5f48d94767-lp86m, huge-service-deployment-0-5f48d94767-zn8mg, huge-service-deployment-0-5f48d94767-vjvwp, huge-service-deployment-0-5f48d94767-5pfj2, huge-service-deployment-0-5f48d94767-qgt7l, huge-service-deployment-0-5f48d94767-qtbcp, huge-service-deployment-0-5f48d94767-5nmqg, huge-service-deployment-0-5f48d94767-dprfh, huge-service-deployment-0-5f48d94767-zmlrp, huge-service-deployment-0-5f48d94767-h2kkl, huge-service-deployment-0-5f48d94767-sxn55, huge-service-deployment-0-5f48d94767-rbjld, huge-service-deployment-0-5f48d94767-5cfhl, huge-service-deployment-0-5f48d94767-qnh9z, huge-service-deployment-0-5f48d94767-sg856, huge-service-deployment-0-5f48d94767-45xwj, huge-service-deployment-0-5f48d94767-vfn4z, huge-service-deployment-0-5f48d94767-vfcv9, huge-service-deployment-0-5f48d94767-722mz, huge-service-deployment-0-5f48d94767-zz9lw, huge-service-deployment-0-5f48d94767-vj2nj, huge-service-deployment-0-5f48d94767-k82ll, huge-service-deployment-0-5f48d94767-hbmxj, huge-service-deployment-0-5f48d94767-zhd5p, huge-service-deployment-0-5f48d94767-2jt8v, huge-service-deployment-0-5f48d94767-8gnj2, huge-service-deployment-0-5f48d94767-5rw4d, huge-service-deployment-0-5f48d94767-n2bfs, huge-service-deployment-0-5f48d94767-rqvqd, huge-service-deployment-0-5f48d94767-7mc8h, huge-service-deployment-0-5f48d94767-dq5gs, huge-service-deployment-0-5f48d94767-zzzwf, huge-service-deployment-0-5f48d94767-x7l5q, huge-service-deployment-0-5f48d94767-9l6f2, huge-service-deployment-0-5f48d94767-5tn2l, huge-service-deployment-0-5f48d94767-rslgx, huge-service-deployment-0-5f48d94767-sf9r9, huge-service-deployment-0-5f48d94767-tkmfb, huge-service-deployment-0-5f48d94767-rstkm, huge-service-deployment-0-5f48d94767-jhzvw, huge-service-deployment-0-5f48d94767-8bn6h, huge-service-deployment-0-5f48d94767-78dh9, huge-service-deployment-0-5f48d94767-d7pl2, huge-service-deployment-0-5f48d94767-dg2lb, huge-service-deployment-0-5f48d94767-p8b9g, huge-service-deployment-0-5f48d94767-hgn9l, huge-service-deployment-0-5f48d94767-w7c92, huge-service-deployment-0-5f48d94767-q4hwc, huge-service-deployment-0-5f48d94767-2bthm, huge-service-deployment-0-5f48d94767-wnmb8, huge-service-deployment-0-5f48d94767-vw874, huge-service-deployment-0-5f48d94767-sw9ks, huge-service-deployment-0-5f48d94767-jmdxv, huge-service-deployment-0-5f48d94767-kgcpt, huge-service-deployment-0-5f48d94767-bsh4s, huge-service-deployment-0-5f48d94767-d2hpt, huge-service-deployment-0-5f48d94767-h9msd, huge-service-deployment-0-5f48d94767-tgrhj, huge-service-deployment-0-5f48d94767-qkz6l, huge-service-deployment-0-5f48d94767-9jvcp, huge-service-deployment-0-5f48d94767-smsnp, huge-service-deployment-0-5f48d94767-ldpvs, huge-service-deployment-0-5f48d94767-dbh58 I0124 20:47:35.828540 89857 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-h11ayl-1), controlledBy(huge-service-deployment-0): Pods: 1000 out of 1000 created, 974 running (974 updated), 26 pending scheduled, 0 not scheduled, 0 inactive, 19 terminating, 0 unknown, 0 runningButNotReady I0124 20:47:40.851662 89857 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-h11ayl-1), controlledBy(huge-service-deployment-0): Pods: 1000 out of 1000 created, 1000 running (1000 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0124 20:47:40.851724 89857 wait_for_controlled_pods.go:365] WaitForControlledPodsRunning: running 1, deleted 0, timeout: 0, failed: 0 I0124 20:47:40.851738 89857 wait_for_controlled_pods.go:370] WaitForControlledPodsRunning: maxDuration=25.757299609s, operationTimeout=30m0s, ratio=0.01 I0124 20:47:40.851752 89857 wait_for_controlled_pods.go:384] WaitForControlledPodsRunning: 1/1 Deployments are running with all pods I0124 20:47:40.851773 89857 simple_test_executor.go:171] Step "[step: 07] Waiting for huge-service pods to be updated" ended I0124 20:47:40.851795 89857 simple_test_executor.go:149] Step "[step: 08] Deleting huge-service pods" started I0124 20:47:40.889717 89857 wait_for_pods.go:64] WaitForControlledPodsRunning: namespace(test-h11ayl-1), controlledBy(huge-service-deployment-0): starting with timeout: 29m59.999976269s I0124 20:47:40.889783 89857 simple_test_executor.go:171] Step "[step: 08] Deleting huge-service pods" ended I0124 20:47:40.889830 89857 simple_test_executor.go:149] Step "[step: 09] Waiting for huge-service pods to be deleted" started I0124 20:47:40.889887 89857 wait_for_controlled_pods.go:288] WaitForControlledPodsRunning: waiting for controlled pods measurement... I0124 20:47:45.899536 89857 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-h11ayl-1), controlledBy(huge-service-deployment-0): Pods: 465 out of 0 created, 465 running (465 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 96 terminating, 0 unknown, 0 runningButNotReady I0124 20:47:50.900420 89857 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-h11ayl-1), controlledBy(huge-service-deployment-0): Pods: 0 out of 0 created, 0 running (0 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 49 terminating, 0 unknown, 0 runningButNotReady I0124 20:47:55.901493 89857 wait_for_pods.go:122] WaitForControlledPodsRunning: namespace(test-h11ayl-1), controlledBy(huge-service-deployment-0): Pods: 0 out of 0 created, 0 running (0 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0124 20:47:55.901565 89857 wait_for_controlled_pods.go:365] WaitForControlledPodsRunning: running 0, deleted 1, timeout: 0, failed: 0 I0124 20:47:55.901578 89857 wait_for_controlled_pods.go:370] WaitForControlledPodsRunning: maxDuration=15.011822937s, operationTimeout=30m0s, ratio=0.01 I0124 20:47:55.901613 89857 wait_for_controlled_pods.go:384] WaitForControlledPodsRunning: 0/0 Deployments are running with all pods I0124 20:47:55.901635 89857 simple_test_executor.go:171] Step "[step: 09] Waiting for huge-service pods to be deleted" ended I0124 20:47:55.901656 89857 simple_test_executor.go:149] Step "[step: 10] Delete huge-service" started I0124 20:47:55.946699 89857 simple_test_executor.go:171] Step "[step: 10] Delete huge-service" ended I0124 20:47:55.946732 89857 simple_test_executor.go:149] Step "[step: 11] gathering measurements" started ... skipping 181 lines ... Specify --start=76749 in the next get-serial-port-output invocation to get only the new output starting from here. scp: /var/log/cluster-autoscaler.log*: No such file or directory scp: /var/log/fluentd.log*: No such file or directory scp: /var/log/kubelet.cov*: No such file or directory scp: /var/log/cl2-**: No such file or directory scp: /var/log/startupscript.log*: No such file or directory ERROR: (gcloud.compute.scp) [/usr/bin/scp] exited with return code [1]. Dumping logs from nodes to GCS directly at 'gs://sig-scalability-logs/pull-kubernetes-e2e-gce-100-performance/1617975695894384640' using logexporter namespace/logexporter created secret/google-service-account created daemonset.apps/logexporter created Listing marker files (gs://sig-scalability-logs/pull-kubernetes-e2e-gce-100-performance/1617975695894384640/logexported-nodes-registry) for successful nodes... CommandException: One or more URLs matched no objects. ... skipping 217 lines ...