This job view page is being replaced by Spyglass soon. Check out the new job view.
PRfromanirh: node: cpumgr: stricter pre-check for the policy option full-pcpus-only
ResultABORTED
Tests 0 failed / 86 succeeded
Started2022-11-24 12:57
Elapsed57m20s
Revision923baa7d01fca2aa1c5bc80e911a5221f003fd6c
Refs 114114
job-versionv1.27.0-alpha.0.48+c9884f3ae68bbb
kubetest-versionv20221116-7c85504268
revisionv1.27.0-alpha.0.48+c9884f3ae68bbb

No Test Failures!


Show 86 Passed Tests

Error lines from build-log.txt

... skipping 691 lines ...
Looking for address 'e2e-114114-95a39-master-ip'
Looking for address 'e2e-114114-95a39-master-internal-ip'
Using master: e2e-114114-95a39-master (external IP: 34.23.221.178; internal IP: 10.40.0.2)
Waiting up to 300 seconds for cluster initialization.

  This will continually check to see if the API for kubernetes is reachable.
  This may time out if there was some uncaught error during start up.

Kubernetes cluster created.
Cluster "k8s-infra-e2e-boskos-scale-06_e2e-114114-95a39" set.
User "k8s-infra-e2e-boskos-scale-06_e2e-114114-95a39" set.
Context "k8s-infra-e2e-boskos-scale-06_e2e-114114-95a39" created.
Switched to context "k8s-infra-e2e-boskos-scale-06_e2e-114114-95a39".
... skipping 234 lines ...
e2e-114114-95a39-minion-group-x6db   Ready                         <none>   58s    v1.27.0-alpha.0.48+c9884f3ae68bbb
e2e-114114-95a39-minion-group-zb51   Ready                         <none>   54s    v1.27.0-alpha.0.48+c9884f3ae68bbb
e2e-114114-95a39-minion-heapster     Ready                         <none>   75s    v1.27.0-alpha.0.48+c9884f3ae68bbb
Warning: v1 ComponentStatus is deprecated in v1.19+
Validate output:
Warning: v1 ComponentStatus is deprecated in v1.19+
NAME                 STATUS    MESSAGE                         ERROR
etcd-1               Healthy   {"health":"true","reason":""}   
etcd-0               Healthy   {"health":"true","reason":""}   
controller-manager   Healthy   ok                              
scheduler            Healthy   ok                              
Cluster validation encountered some problems, but cluster should be in working order
...ignoring non-fatal errors in validate-cluster
Done, listing cluster services:

Kubernetes control plane is running at https://34.23.221.178
GLBCDefaultBackend is running at https://34.23.221.178/api/v1/namespaces/kube-system/services/default-http-backend:http/proxy
CoreDNS is running at https://34.23.221.178/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
Metrics-server is running at https://34.23.221.178/api/v1/namespaces/kube-system/services/https:metrics-server:/proxy
... skipping 1112 lines ...
I1124 13:29:29.226573   89755 wait_for_pods.go:61] WaitForControlledPodsRunning: namespace(test-ex7ks4-1), controlledBy(daemonset-0): starting with timeout: 14m59.899913607s
I1124 13:29:30.089165   89755 wait_for_pods.go:111] WaitForControlledPodsRunning: namespace(test-ex7ks4-1), controlledBy(small-deployment-262): Pods: 5 out of 5 created, 5 running (5 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady 
I1124 13:29:30.171738   89755 wait_for_pods.go:111] WaitForControlledPodsRunning: namespace(test-ex7ks4-1), controlledBy(small-deployment-84): Pods: 5 out of 5 created, 5 running (5 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady 
I1124 13:29:30.281007   89755 wait_for_pods.go:111] WaitForControlledPodsRunning: namespace(test-ex7ks4-1), controlledBy(small-deployment-70): Pods: 5 out of 5 created, 5 running (5 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady 
I1124 13:29:32.266578   89755 wait_for_pods.go:111] WaitForControlledPodsRunning: namespace(test-ex7ks4-1), controlledBy(small-deployment-36): Pods: 5 out of 5 created, 5 running (5 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady 
I1124 13:29:33.068293   89755 wait_for_pods.go:111] WaitForControlledPodsRunning: namespace(test-ex7ks4-1), controlledBy(small-deployment-271): Pods: 5 out of 5 created, 5 running (5 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady 
I1124 13:29:34.169128   89755 wait_for_controlled_pods.go:319] WaitForControlledPodsRunning: running 2, deleted 0, timeout: 0, failed: 0
I1124 13:29:34.169177   89755 wait_for_controlled_pods.go:324] WaitForControlledPodsRunning: maxDuration=40.007354699s, operationTimeout=15m0s, ratio=0.04
I1124 13:29:34.169206   89755 wait_for_controlled_pods.go:338] WaitForControlledPodsRunning: 2/2 StatefulSets are running with all pods
I1124 13:29:34.169219   89755 wait_for_controlled_pods.go:319] WaitForControlledPodsRunning: running 3, deleted 0, timeout: 0, failed: 0
I1124 13:29:34.169269   89755 wait_for_controlled_pods.go:324] WaitForControlledPodsRunning: maxDuration=5.006072769s, operationTimeout=15m0s, ratio=0.01
I1124 13:29:34.169296   89755 wait_for_controlled_pods.go:338] WaitForControlledPodsRunning: 3/3 Jobs are running with all pods
I1124 13:29:34.229313   89755 wait_for_pods.go:111] WaitForControlledPodsRunning: namespace(test-ex7ks4-1), controlledBy(daemonset-0): Pods: 100 out of 100 created, 100 running (100 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady 
I1124 13:29:34.229367   89755 wait_for_controlled_pods.go:319] WaitForControlledPodsRunning: running 1, deleted 0, timeout: 0, failed: 0
I1124 13:29:34.229379   89755 wait_for_controlled_pods.go:324] WaitForControlledPodsRunning: maxDuration=5.002775519s, operationTimeout=15m0s, ratio=0.01
I1124 13:29:34.229395   89755 wait_for_controlled_pods.go:338] WaitForControlledPodsRunning: 1/1 DaemonSets are running with all pods
I1124 13:29:34.345043   89755 wait_for_controlled_pods.go:319] WaitForControlledPodsRunning: running 323, deleted 0, timeout: 0, failed: 0
I1124 13:29:34.345073   89755 wait_for_controlled_pods.go:324] WaitForControlledPodsRunning: maxDuration=5.008122661s, operationTimeout=15m0s, ratio=0.01
I1124 13:29:34.345087   89755 wait_for_controlled_pods.go:338] WaitForControlledPodsRunning: 323/323 Deployments are running with all pods
I1124 13:29:34.345111   89755 simple_test_executor.go:171] Step "[step: 07] Waiting for 'create objects' to be completed" ended
I1124 13:29:34.345129   89755 simple_test_executor.go:149] Step "[step: 08] Creating scheduler throughput measurements" started
I1124 13:29:34.345212   89755 pod_startup_latency.go:132] PodStartupLatency: labelSelector(group = scheduler-throughput): starting pod startup latency measurement...
I1124 13:29:34.345247   89755 wait_for_controlled_pods.go:215] WaitForControlledPodsRunning: starting wait for controlled pods measurement...
... skipping 2 lines ...
I1124 13:29:34.445625   89755 simple_test_executor.go:149] Step "[step: 09] create scheduler throughput pods" started
I1124 13:29:34.484380   89755 wait_for_pods.go:61] WaitForControlledPodsRunning: namespace(test-ex7ks4-2), controlledBy(scheduler-throughput-deployment-0): starting with timeout: 19m59.999983322s
I1124 13:29:34.484731   89755 simple_test_executor.go:171] Step "[step: 09] create scheduler throughput pods" ended
I1124 13:29:34.484758   89755 simple_test_executor.go:149] Step "[step: 10] Waiting for scheduler throughput pods to be created" started
I1124 13:29:34.484796   89755 wait_for_controlled_pods.go:246] WaitForControlledPodsRunning: waiting for controlled pods measurement...
I1124 13:29:39.492771   89755 wait_for_pods.go:111] WaitForControlledPodsRunning: namespace(test-ex7ks4-2), controlledBy(scheduler-throughput-deployment-0): Pods: 576 out of 1000 created, 420 running (420 updated), 154 pending scheduled, 2 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady 
error dialing prow@34.23.221.178:22: 'ssh: handshake failed: read tcp 10.35.170.16:54196->34.23.221.178:22: read: connection reset by peer', retrying
I1124 13:29:44.511448   89755 wait_for_pods.go:111] WaitForControlledPodsRunning: namespace(test-ex7ks4-2), controlledBy(scheduler-throughput-deployment-0): Pods: 1000 out of 1000 created, 904 running (904 updated), 96 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady 
I1124 13:29:49.545751   89755 wait_for_pods.go:111] WaitForControlledPodsRunning: namespace(test-ex7ks4-2), controlledBy(scheduler-throughput-deployment-0): Pods: 1000 out of 1000 created, 1000 running (1000 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady 
I1124 13:29:49.545814   89755 wait_for_controlled_pods.go:319] WaitForControlledPodsRunning: running 1, deleted 0, timeout: 0, failed: 0
I1124 13:29:49.545834   89755 wait_for_controlled_pods.go:324] WaitForControlledPodsRunning: maxDuration=15.061408561s, operationTimeout=20m0s, ratio=0.01
I1124 13:29:49.545851   89755 wait_for_controlled_pods.go:338] WaitForControlledPodsRunning: 1/1 Deployments are running with all pods
I1124 13:29:49.545872   89755 simple_test_executor.go:171] Step "[step: 10] Waiting for scheduler throughput pods to be created" ended
I1124 13:29:49.545895   89755 simple_test_executor.go:149] Step "[step: 11] Collecting scheduler throughput measurements" started
I1124 13:29:49.545942   89755 scheduling_throughput.go:154] SchedulingThroughput: gathering data
I1124 13:29:49.546072   89755 pod_startup_latency.go:226] PodStartupLatency: labelSelector(group = scheduler-throughput): gathering pod startup latency measurement...
... skipping 33 lines ...
I1124 13:29:50.009111   89755 simple_test_executor.go:149] Step "[step: 13] Waiting for scheduler throughput pods to be deleted" started
I1124 13:29:50.009141   89755 wait_for_controlled_pods.go:246] WaitForControlledPodsRunning: waiting for controlled pods measurement...
I1124 13:29:50.009274   89755 wait_for_pods.go:61] WaitForControlledPodsRunning: namespace(test-ex7ks4-2), controlledBy(scheduler-throughput-deployment-0): starting with timeout: 19m59.999978348s
I1124 13:29:55.019858   89755 wait_for_pods.go:111] WaitForControlledPodsRunning: namespace(test-ex7ks4-2), controlledBy(scheduler-throughput-deployment-0): Pods: 458 out of 0 created, 458 running (458 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 90 terminating, 0 unknown, 0 runningButNotReady 
I1124 13:30:00.021284   89755 wait_for_pods.go:111] WaitForControlledPodsRunning: namespace(test-ex7ks4-2), controlledBy(scheduler-throughput-deployment-0): Pods: 0 out of 0 created, 0 running (0 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 35 terminating, 0 unknown, 0 runningButNotReady 
I1124 13:30:05.022082   89755 wait_for_pods.go:111] WaitForControlledPodsRunning: namespace(test-ex7ks4-2), controlledBy(scheduler-throughput-deployment-0): Pods: 0 out of 0 created, 0 running (0 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady 
I1124 13:30:05.022161   89755 wait_for_controlled_pods.go:319] WaitForControlledPodsRunning: running 0, deleted 1, timeout: 0, failed: 0
I1124 13:30:05.022208   89755 wait_for_controlled_pods.go:324] WaitForControlledPodsRunning: maxDuration=15.012850816s, operationTimeout=20m0s, ratio=0.01
I1124 13:30:05.022230   89755 wait_for_controlled_pods.go:338] WaitForControlledPodsRunning: 0/0 Deployments are running with all pods
I1124 13:30:05.022260   89755 simple_test_executor.go:171] Step "[step: 13] Waiting for scheduler throughput pods to be deleted" ended
I1124 13:30:05.022290   89755 simple_test_executor.go:149] Step "[step: 14] Starting latency pod measurements" started
I1124 13:30:05.022358   89755 wait_for_controlled_pods.go:215] WaitForControlledPodsRunning: starting wait for controlled pods measurement...
I1124 13:30:05.022396   89755 pod_startup_latency.go:132] PodStartupLatency: labelSelector(group = latency): starting pod startup latency measurement...
... skipping 999 lines ...
I1124 13:31:49.256795   89755 wait_for_pods.go:111] WaitForControlledPodsRunning: namespace(test-ex7ks4-1), controlledBy(latency-deployment-494): Pods: 1 out of 1 created, 1 running (1 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady 
I1124 13:31:49.455305   89755 wait_for_pods.go:111] WaitForControlledPodsRunning: namespace(test-ex7ks4-1), controlledBy(latency-deployment-495): Pods: 1 out of 1 created, 1 running (1 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady 
I1124 13:31:49.658261   89755 wait_for_pods.go:111] WaitForControlledPodsRunning: namespace(test-ex7ks4-1), controlledBy(latency-deployment-496): Pods: 1 out of 1 created, 1 running (1 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady 
I1124 13:31:49.856643   89755 wait_for_pods.go:111] WaitForControlledPodsRunning: namespace(test-ex7ks4-1), controlledBy(latency-deployment-497): Pods: 1 out of 1 created, 1 running (1 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady 
I1124 13:31:50.058667   89755 wait_for_pods.go:111] WaitForControlledPodsRunning: namespace(test-ex7ks4-1), controlledBy(latency-deployment-498): Pods: 1 out of 1 created, 1 running (1 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady 
I1124 13:31:50.259016   89755 wait_for_pods.go:111] WaitForControlledPodsRunning: namespace(test-ex7ks4-1), controlledBy(latency-deployment-499): Pods: 1 out of 1 created, 1 running (1 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady 
I1124 13:31:50.650672   89755 wait_for_controlled_pods.go:319] WaitForControlledPodsRunning: running 500, deleted 0, timeout: 0, failed: 0
I1124 13:31:50.650714   89755 wait_for_controlled_pods.go:324] WaitForControlledPodsRunning: maxDuration=5.001332149s, operationTimeout=15m0s, ratio=0.01
I1124 13:31:50.650744   89755 wait_for_controlled_pods.go:338] WaitForControlledPodsRunning: 500/500 Deployments are running with all pods
I1124 13:31:50.650774   89755 simple_test_executor.go:171] Step "[step: 16] Waiting for latency pods to be running" ended
I1124 13:31:50.650802   89755 simple_test_executor.go:149] Step "[step: 17] Deleting latency pods" started
I1124 13:31:50.687960   89755 wait_for_pods.go:61] WaitForControlledPodsRunning: namespace(test-ex7ks4-1), controlledBy(latency-deployment-0): starting with timeout: 14m59.999977962s
I1124 13:31:50.788141   89755 wait_for_pods.go:61] WaitForControlledPodsRunning: namespace(test-ex7ks4-1), controlledBy(latency-deployment-1): starting with timeout: 14m59.999979553s
... skipping 495 lines ...
I1124 13:32:40.387548   89755 wait_for_pods.go:61] WaitForControlledPodsRunning: namespace(test-ex7ks4-1), controlledBy(latency-deployment-497): starting with timeout: 14m59.999988142s
I1124 13:32:40.487137   89755 wait_for_pods.go:61] WaitForControlledPodsRunning: namespace(test-ex7ks4-1), controlledBy(latency-deployment-498): starting with timeout: 14m59.999985304s
I1124 13:32:40.588603   89755 simple_test_executor.go:171] Step "[step: 17] Deleting latency pods" ended
I1124 13:32:40.588638   89755 simple_test_executor.go:149] Step "[step: 18] Waiting for latency pods to be deleted" started
I1124 13:32:40.588662   89755 wait_for_controlled_pods.go:246] WaitForControlledPodsRunning: waiting for controlled pods measurement...
I1124 13:32:40.588752   89755 wait_for_pods.go:61] WaitForControlledPodsRunning: namespace(test-ex7ks4-1), controlledBy(latency-deployment-499): starting with timeout: 14m59.999983657s
I1124 13:32:45.635932   89755 wait_for_controlled_pods.go:319] WaitForControlledPodsRunning: running 0, deleted 500, timeout: 0, failed: 0
I1124 13:32:45.635963   89755 wait_for_controlled_pods.go:324] WaitForControlledPodsRunning: maxDuration=5.001167692s, operationTimeout=15m0s, ratio=0.01
I1124 13:32:45.635977   89755 wait_for_controlled_pods.go:338] WaitForControlledPodsRunning: 0/0 Deployments are running with all pods
I1124 13:32:45.636001   89755 simple_test_executor.go:171] Step "[step: 18] Waiting for latency pods to be deleted" ended
I1124 13:32:45.636021   89755 simple_test_executor.go:149] Step "[step: 19] Collecting pod startup latency" started
I1124 13:32:45.636056   89755 pod_startup_latency.go:226] PodStartupLatency: labelSelector(group = latency): gathering pod startup latency measurement...
I1124 13:32:46.118450   89755 phase_latency.go:141] PodStartupLatency: 100 worst pod_startup latencies: [{test-ex7ks4-1/latency-deployment-123-78f57d84f5-rwk6w 2.401664866s} {test-ex7ks4-1/latency-deployment-301-5c75dbfc4c-ns9w2 2.406874157s} {test-ex7ks4-1/latency-deployment-481-5dd979c4b5-w8j67 2.410816706s} {test-ex7ks4-1/latency-deployment-332-5bdcffc54c-bj8tm 2.411420644s} {test-ex7ks4-1/latency-deployment-391-576d8bcc5c-76srj 2.419916782s} {test-ex7ks4-1/latency-deployment-291-74c4fbdd79-5h9ds 2.421756397s} {test-ex7ks4-1/latency-deployment-300-5b84bb9fcc-xfwnr 2.422541713s} {test-ex7ks4-1/latency-deployment-306-5f865b4689-6hb5f 2.434835169s} {test-ex7ks4-1/latency-deployment-158-78d55f944c-rq59z 2.441732384s} {test-ex7ks4-1/latency-deployment-402-5c7b8b5b7-xml58 2.444854991s} {test-ex7ks4-1/latency-deployment-156-55f7c9b7c5-vmn56 2.447745034s} {test-ex7ks4-1/latency-deployment-116-b85f7f6ff-5jgjm 2.452087107s} {test-ex7ks4-1/latency-deployment-345-759995649f-hstfc 2.452464531s} {test-ex7ks4-1/latency-deployment-471-9cd6784f5-vlp52 2.457395454s} {test-ex7ks4-1/latency-deployment-482-5584d8b66c-jrsd4 2.457760878s} {test-ex7ks4-1/latency-deployment-243-8cf7495b9-hdj4w 2.460466664s} {test-ex7ks4-1/latency-deployment-462-77c59c5759-cfj5h 2.470021424s} {test-ex7ks4-1/latency-deployment-408-59bcbccd6c-r9nf9 2.474684248s} {test-ex7ks4-1/latency-deployment-111-787cdb5497-ww5tq 2.475501168s} {test-ex7ks4-1/latency-deployment-202-796d4b5b9f-q6xnz 2.479968905s} {test-ex7ks4-1/latency-deployment-370-5b6c96c947-jwlvm 2.48130159s} {test-ex7ks4-1/latency-deployment-457-7f6dd96cdc-hk4wn 2.482055071s} {test-ex7ks4-1/latency-deployment-353-67c7d8dbf-mg94w 2.484173479s} {test-ex7ks4-1/latency-deployment-72-7bbc49f8c5-hbdfx 2.490530505s} {test-ex7ks4-1/latency-deployment-68-7496dcd4b7-hqz98 2.49353782s} {test-ex7ks4-1/latency-deployment-366-776b7b97-6d4gg 2.508050973s} {test-ex7ks4-1/latency-deployment-422-799b988bf5-mt2t5 2.517065585s} {test-ex7ks4-1/latency-deployment-192-586f894dd7-wxlb9 2.5177646s} {test-ex7ks4-1/latency-deployment-451-7668f97995-t2mgv 2.52175893s} {test-ex7ks4-1/latency-deployment-327-5c4458879-rrcvz 2.52687874s} {test-ex7ks4-1/latency-deployment-406-988f8fbc9-5n8bj 2.528882129s} {test-ex7ks4-1/latency-deployment-213-6f88f976c5-s4r4v 2.531609156s} {test-ex7ks4-1/latency-deployment-120-7888c566f-cq2rx 2.53577392s} {test-ex7ks4-1/latency-deployment-53-6694d96cc5-ztdpk 2.549342387s} {test-ex7ks4-1/latency-deployment-103-64d96c87cf-mgc7t 2.552938527s} {test-ex7ks4-1/latency-deployment-121-597846c575-kk9bc 2.554904892s} {test-ex7ks4-1/latency-deployment-296-5696f9c5fc-c9z4z 2.558175285s} {test-ex7ks4-1/latency-deployment-197-65f7bb4585-znz77 2.564198231s} {test-ex7ks4-1/latency-deployment-63-77bc4dc967-jkkqb 2.581109771s} {test-ex7ks4-1/latency-deployment-287-7974bfd769-5fxqq 2.584919108s} {test-ex7ks4-1/latency-deployment-492-75958df4b5-4fvrx 2.59815608s} {test-ex7ks4-1/latency-deployment-426-5df9df6fcf-9r8n9 2.598937234s} {test-ex7ks4-1/latency-deployment-367-564cbcc98c-kkzsk 2.60339607s} {test-ex7ks4-1/latency-deployment-98-59c486dbd7-49ll6 2.609143756s} {test-ex7ks4-1/latency-deployment-57-67d5b8fbd7-b8phc 2.611442533s} {test-ex7ks4-1/latency-deployment-3-5c5744bf89-888l4 2.616829874s} {test-ex7ks4-1/latency-deployment-292-6fb7dc6cdf-l728j 2.630568494s} {test-ex7ks4-1/latency-deployment-293-64789fdbf9-kgcrn 2.630570492s} {test-ex7ks4-1/latency-deployment-22-5bbddd7f67-czqc2 2.637004145s} {test-ex7ks4-1/latency-deployment-357-b8cb7f69-42q54 2.639710226s} {test-ex7ks4-1/latency-deployment-102-5f59bb5b69-xc9dd 2.640180066s} {test-ex7ks4-1/latency-deployment-187-6d84979967-kpw6f 2.645991755s} {test-ex7ks4-1/latency-deployment-69-68995f7c9f-rwdjh 2.650629916s} {test-ex7ks4-1/latency-deployment-416-76f8b4ffc7-bqhln 2.655006155s} {test-ex7ks4-1/latency-deployment-368-6f987d8df-mjc84 2.658785391s} {test-ex7ks4-1/latency-deployment-487-7764fcf589-9s7h5 2.667156509s} {test-ex7ks4-1/latency-deployment-343-56774cf887-7pczk 2.669263115s} {test-ex7ks4-1/latency-deployment-218-7b7bb6c985-xv7fw 2.674249672s} {test-ex7ks4-1/latency-deployment-32-84fbf59499-4jr56 2.676299974s} {test-ex7ks4-1/latency-deployment-441-689c74b6f7-jccgz 2.681551758s} {test-ex7ks4-1/latency-deployment-248-5c855fd5dc-ptx9g 2.69073946s} {test-ex7ks4-1/latency-deployment-212-648556c585-lmwrp 2.713069908s} {test-ex7ks4-1/latency-deployment-337-66d7b78869-rrjmx 2.714737748s} {test-ex7ks4-1/latency-deployment-19-78b8fd69c5-fwnkb 2.719160775s} {test-ex7ks4-1/latency-deployment-108-544c6b64d9-smsxg 2.729732156s} {test-ex7ks4-1/latency-deployment-59-65b74fc4d7-7sq85 2.732461633s} {test-ex7ks4-1/latency-deployment-178-7dd5cbb6f5-ns2cx 2.732568863s} {test-ex7ks4-1/latency-deployment-382-c6f7444bf-xzmvd 2.736246517s} {test-ex7ks4-1/latency-deployment-74-7f6ff5884c-f7lxw 2.738436742s} {test-ex7ks4-1/latency-deployment-348-65c6bd4f49-mhch5 2.743108825s} {test-ex7ks4-1/latency-deployment-442-574d85ff4c-bgdjp 2.74490432s} {test-ex7ks4-1/latency-deployment-39-c99b6b6f5-tg7qv 2.74717761s} {test-ex7ks4-1/latency-deployment-338-f44c9d7bc-7hngw 2.758601302s} {test-ex7ks4-1/latency-deployment-14-8577448d87-rp8x8 2.760654359s} {test-ex7ks4-1/latency-deployment-333-cc8565957-vwfhn 2.761884599s} {test-ex7ks4-1/latency-deployment-162-6fd598d69-ltljv 2.76955643s} {test-ex7ks4-1/latency-deployment-242-d89584495-jvqgl 2.77697653s} {test-ex7ks4-1/latency-deployment-113-549874bf7-6gztr 2.797024994s} {test-ex7ks4-1/latency-deployment-34-56b69b56b7-k9j2g 2.824604669s} {test-ex7ks4-1/latency-deployment-403-7c7d978957-nqwsh 2.844546099s} {test-ex7ks4-1/latency-deployment-447-54d959f84f-nmpmv 2.846213394s} {test-ex7ks4-1/latency-deployment-4-84b74cfdc-p8bl4 2.854085059s} {test-ex7ks4-1/latency-deployment-303-64b9dcc8ff-kvxnv 2.868415693s} {test-ex7ks4-1/latency-deployment-138-6c67dd8695-bdkl5 2.8714798s} {test-ex7ks4-1/latency-deployment-383-566df769c5-d5f6r 2.873756937s} {test-ex7ks4-1/latency-deployment-33-fbc646dc9-5drsx 2.877869333s} {test-ex7ks4-1/latency-deployment-273-6c9d4d6c4c-ltlwb 2.880003918s} {test-ex7ks4-1/latency-deployment-51-6bc4b99b7-l4z54 2.881936265s} {test-ex7ks4-1/latency-deployment-363-6f4cc7445f-jffpb 2.892918038s} {test-ex7ks4-1/latency-deployment-449-ccb887857-x57lf 2.899897678s} {test-ex7ks4-1/latency-deployment-160-9b7677f69-chk6m 2.985284469s} {test-ex7ks4-1/latency-deployment-193-567666d79c-24khj 2.997034501s} {test-ex7ks4-1/latency-deployment-64-5d5c9d7bcf-fbnpr 3.007498855s} {test-ex7ks4-1/latency-deployment-49-cf4597b8c-mffv6 3.013070136s} {test-ex7ks4-1/latency-deployment-233-64bc4c68bc-w9zxx 3.018630937s} {test-ex7ks4-1/latency-deployment-61-658fd58d75-xnkst 3.030808261s} {test-ex7ks4-1/latency-deployment-29-7989cbb547-9kwpv 3.081179723s} {test-ex7ks4-1/latency-deployment-263-76c7f9b67c-khr5w 3.090755021s} {test-ex7ks4-1/latency-deployment-23-7bbb55f647-jfp8g 3.308085303s} {test-ex7ks4-1/latency-deployment-207-f6b7b7d57-7jnxv 3.62069449s}]
... skipping 610 lines ...
I1124 13:33:27.374114   89755 wait_for_pods.go:111] WaitForControlledPodsRunning: namespace(test-ex7ks4-1), controlledBy(small-deployment-23): Pods: 4 out of 4 created, 4 running (4 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady 
I1124 13:33:27.615553   89755 wait_for_pods.go:111] WaitForControlledPodsRunning: namespace(test-ex7ks4-1), controlledBy(small-deployment-102): Pods: 6 out of 6 created, 6 running (6 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady 
I1124 13:33:27.624683   89755 wait_for_pods.go:111] WaitForControlledPodsRunning: namespace(test-ex7ks4-1), controlledBy(small-deployment-266): Pods: 6 out of 6 created, 6 running (6 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady 
I1124 13:33:27.875224   89755 wait_for_pods.go:111] WaitForControlledPodsRunning: namespace(test-ex7ks4-1), controlledBy(small-deployment-32): Pods: 2 out of 2 created, 2 running (2 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady 
I1124 13:33:28.020770   89755 wait_for_pods.go:111] WaitForControlledPodsRunning: namespace(test-ex7ks4-1), controlledBy(small-deployment-87): Pods: 6 out of 6 created, 6 running (6 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady 
I1124 13:33:28.161042   89755 wait_for_pods.go:111] WaitForControlledPodsRunning: namespace(test-ex7ks4-1), controlledBy(small-deployment-83): Pods: 3 out of 3 created, 3 running (3 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady 
I1124 13:33:28.197079   89755 wait_for_controlled_pods.go:319] WaitForControlledPodsRunning: running 2, deleted 0, timeout: 0, failed: 0
I1124 13:33:28.197131   89755 wait_for_controlled_pods.go:324] WaitForControlledPodsRunning: maxDuration=20.00595452s, operationTimeout=15m0s, ratio=0.02
I1124 13:33:28.197148   89755 wait_for_controlled_pods.go:338] WaitForControlledPodsRunning: 2/2 StatefulSets are running with all pods
I1124 13:33:28.197139   89755 wait_for_controlled_pods.go:319] WaitForControlledPodsRunning: running 3, deleted 0, timeout: 0, failed: 0
I1124 13:33:28.197170   89755 wait_for_controlled_pods.go:324] WaitForControlledPodsRunning: maxDuration=5.004682645s, operationTimeout=15m0s, ratio=0.01
I1124 13:33:28.197184   89755 wait_for_controlled_pods.go:338] WaitForControlledPodsRunning: 3/3 Jobs are running with all pods
I1124 13:33:28.355856   89755 wait_for_controlled_pods.go:319] WaitForControlledPodsRunning: running 323, deleted 0, timeout: 0, failed: 0
I1124 13:33:28.355907   89755 wait_for_controlled_pods.go:324] WaitForControlledPodsRunning: maxDuration=5.008923129s, operationTimeout=15m0s, ratio=0.01
I1124 13:33:28.355930   89755 wait_for_controlled_pods.go:338] WaitForControlledPodsRunning: 323/323 Deployments are running with all pods
I1124 13:33:28.804978   89755 wait_for_pods.go:111] WaitForControlledPodsRunning: namespace(test-ex7ks4-1), controlledBy(daemonset-0): Pods: 100 out of 100 created, 100 running (100 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady 
I1124 13:33:28.805036   89755 wait_for_controlled_pods.go:319] WaitForControlledPodsRunning: running 1, deleted 0, timeout: 0, failed: 0
I1124 13:33:28.805047   89755 wait_for_controlled_pods.go:324] WaitForControlledPodsRunning: maxDuration=30.109393062s, operationTimeout=15m0s, ratio=0.03
I1124 13:33:28.805060   89755 wait_for_controlled_pods.go:338] WaitForControlledPodsRunning: 1/1 DaemonSets are running with all pods
I1124 13:33:28.805074   89755 simple_test_executor.go:171] Step "[step: 22] Waiting for 'scale and update objects' to be completed" ended
I1124 13:33:28.805090   89755 simple_test_executor.go:149] Step "[step: 23] Starting measurement for 'delete objects'" started
I1124 13:33:28.805189   89755 wait_for_controlled_pods.go:212] WaitForControlledPodsRunning: wait for controlled pods measurement already running
I1124 13:33:28.805201   89755 wait_for_controlled_pods.go:212] WaitForControlledPodsRunning: wait for controlled pods measurement already running
... skipping 333 lines ...
I1124 13:37:01.967674   89755 simple_test_executor.go:149] Step "[step: 25] Waiting for 'delete objects' to be completed" started
I1124 13:37:01.967757   89755 wait_for_controlled_pods.go:246] WaitForControlledPodsRunning: waiting for controlled pods measurement...
I1124 13:37:01.967778   89755 wait_for_controlled_pods.go:246] WaitForControlledPodsRunning: waiting for controlled pods measurement...
I1124 13:37:01.967816   89755 wait_for_controlled_pods.go:246] WaitForControlledPodsRunning: waiting for controlled pods measurement...
I1124 13:37:01.967764   89755 wait_for_controlled_pods.go:246] WaitForControlledPodsRunning: waiting for controlled pods measurement...
I1124 13:37:01.968888   89755 wait_for_pods.go:61] WaitForControlledPodsRunning: namespace(test-ex7ks4-1), controlledBy(small-deployment-3): starting with timeout: 14m59.999981067s
I1124 13:37:07.000736   89755 wait_for_controlled_pods.go:319] WaitForControlledPodsRunning: running 0, deleted 3, timeout: 0, failed: 0
I1124 13:37:07.000773   89755 wait_for_controlled_pods.go:324] WaitForControlledPodsRunning: maxDuration=5.001327865s, operationTimeout=15m0s, ratio=0.01
I1124 13:37:07.000798   89755 wait_for_controlled_pods.go:338] WaitForControlledPodsRunning: 0/0 Jobs are running with all pods
I1124 13:37:07.001959   89755 wait_for_controlled_pods.go:319] WaitForControlledPodsRunning: running 0, deleted 2, timeout: 0, failed: 0
I1124 13:37:07.001983   89755 wait_for_controlled_pods.go:324] WaitForControlledPodsRunning: maxDuration=5.000711497s, operationTimeout=15m0s, ratio=0.01
I1124 13:37:07.001987   89755 wait_for_controlled_pods.go:319] WaitForControlledPodsRunning: running 0, deleted 1, timeout: 0, failed: 0
I1124 13:37:07.002009   89755 wait_for_controlled_pods.go:324] WaitForControlledPodsRunning: maxDuration=5.000889861s, operationTimeout=15m0s, ratio=0.01
I1124 13:37:07.002022   89755 wait_for_controlled_pods.go:338] WaitForControlledPodsRunning: 0/0 DaemonSets are running with all pods
I1124 13:37:07.001999   89755 wait_for_controlled_pods.go:338] WaitForControlledPodsRunning: 0/0 StatefulSets are running with all pods
I1124 13:37:07.003326   89755 wait_for_controlled_pods.go:319] WaitForControlledPodsRunning: running 0, deleted 323, timeout: 0, failed: 0
I1124 13:37:07.003356   89755 wait_for_controlled_pods.go:324] WaitForControlledPodsRunning: maxDuration=5.001429045s, operationTimeout=15m0s, ratio=0.01
I1124 13:37:07.003375   89755 wait_for_controlled_pods.go:338] WaitForControlledPodsRunning: 0/0 Deployments are running with all pods
I1124 13:37:07.019572   89755 wait_for_pvcs.go:78] WaitForBoundPVCs: labelSelector(group = load): PVCs: 0 out of 0 created, 0 bound, 0 pending, 0 lost
I1124 13:37:07.019612   89755 simple_test_executor.go:171] Step "[step: 25] Waiting for 'delete objects' to be completed" ended
I1124 13:37:07.019645   89755 simple_test_executor.go:149] Step "[step: 26] delete objects configmaps and secrets" started
I1124 13:37:39.291125   89755 simple_test_executor.go:171] Step "[step: 26] delete objects configmaps and secrets" ended
... skipping 254 lines ...
I1124 13:40:08.763277   89755 wait_for_pods.go:61] WaitForControlledPodsRunning: namespace(test-mbfsuq-1), controlledBy(huge-service-deployment-0): starting with timeout: 29m59.999981808s
I1124 13:40:08.763293   89755 simple_test_executor.go:149] Step "[step: 05] Waiting for huge-service pods to be created" started
I1124 13:40:08.763329   89755 wait_for_controlled_pods.go:246] WaitForControlledPodsRunning: waiting for controlled pods measurement...
I1124 13:40:13.773307   89755 wait_for_pods.go:111] WaitForControlledPodsRunning: namespace(test-mbfsuq-1), controlledBy(huge-service-deployment-0): Pods: 576 out of 1000 created, 423 running (423 updated), 153 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady 
I1124 13:40:18.794126   89755 wait_for_pods.go:111] WaitForControlledPodsRunning: namespace(test-mbfsuq-1), controlledBy(huge-service-deployment-0): Pods: 1000 out of 1000 created, 911 running (911 updated), 89 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady 
I1124 13:40:23.811512   89755 wait_for_pods.go:111] WaitForControlledPodsRunning: namespace(test-mbfsuq-1), controlledBy(huge-service-deployment-0): Pods: 1000 out of 1000 created, 1000 running (1000 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady 
I1124 13:40:23.811567   89755 wait_for_controlled_pods.go:319] WaitForControlledPodsRunning: running 1, deleted 0, timeout: 0, failed: 0
I1124 13:40:23.811576   89755 wait_for_controlled_pods.go:324] WaitForControlledPodsRunning: maxDuration=15.048273094s, operationTimeout=30m0s, ratio=0.01
I1124 13:40:23.811589   89755 wait_for_controlled_pods.go:338] WaitForControlledPodsRunning: 1/1 Deployments are running with all pods
I1124 13:40:23.811609   89755 simple_test_executor.go:171] Step "[step: 05] Waiting for huge-service pods to be created" ended
I1124 13:40:23.811628   89755 simple_test_executor.go:149] Step "[step: 06] Updating huge-service pods" started
I1124 13:40:23.886183   89755 wait_for_pods.go:61] WaitForControlledPodsRunning: namespace(test-mbfsuq-1), controlledBy(huge-service-deployment-0): starting with timeout: 29m59.999979919s
I1124 13:40:23.886264   89755 simple_test_executor.go:171] Step "[step: 06] Updating huge-service pods" ended
... skipping 4 lines ...
I1124 13:40:34.426530   89755 wait_for_pods.go:111] WaitForControlledPodsRunning: namespace(test-mbfsuq-1), controlledBy(huge-service-deployment-0): Pods: 1095 out of 1000 created, 1058 running (555 updated), 37 pending scheduled, 0 not scheduled, 0 inactive, 122 terminating, 0 unknown, 0 runningButNotReady 
W1124 13:40:39.549655   89755 wait_for_pods.go:108] WaitForControlledPodsRunning: namespace(test-mbfsuq-1), controlledBy(huge-service-deployment-0): 261 pods appeared: huge-service-deployment-0-5f48d94767-ljr6k, huge-service-deployment-0-5f48d94767-vkzkw, huge-service-deployment-0-5f48d94767-kkwtw, huge-service-deployment-0-5f48d94767-5h7z4, huge-service-deployment-0-5f48d94767-bqr5r, huge-service-deployment-0-5f48d94767-tmtvt, huge-service-deployment-0-5f48d94767-gvzfx, huge-service-deployment-0-5f48d94767-4gl2j, huge-service-deployment-0-5f48d94767-x2bjt, huge-service-deployment-0-5f48d94767-z92h4, huge-service-deployment-0-5f48d94767-m4d6d, huge-service-deployment-0-5f48d94767-twqlv, huge-service-deployment-0-5f48d94767-svrwq, huge-service-deployment-0-5f48d94767-rtlxh, huge-service-deployment-0-5f48d94767-sw4f9, huge-service-deployment-0-5f48d94767-558hr, huge-service-deployment-0-5f48d94767-h8xpz, huge-service-deployment-0-5f48d94767-sxf8t, huge-service-deployment-0-5f48d94767-vmqb5, huge-service-deployment-0-5f48d94767-r92f8, huge-service-deployment-0-5f48d94767-nkzhc, huge-service-deployment-0-5f48d94767-6q5jg, huge-service-deployment-0-5f48d94767-58tcg, huge-service-deployment-0-5f48d94767-tz958, huge-service-deployment-0-5f48d94767-pv5c6, huge-service-deployment-0-5f48d94767-pf654, huge-service-deployment-0-5f48d94767-lfzhv, huge-service-deployment-0-5f48d94767-qwltp, huge-service-deployment-0-5f48d94767-5889v, huge-service-deployment-0-5f48d94767-56wp6, huge-service-deployment-0-5f48d94767-txr9g, huge-service-deployment-0-5f48d94767-fwssz, huge-service-deployment-0-5f48d94767-q7fc8, huge-service-deployment-0-5f48d94767-dl5ls, huge-service-deployment-0-5f48d94767-sjvdh, huge-service-deployment-0-5f48d94767-6bhck, huge-service-deployment-0-5f48d94767-n6stf, huge-service-deployment-0-5f48d94767-fgm4t, huge-service-deployment-0-5f48d94767-nmqzp, huge-service-deployment-0-5f48d94767-vvqpw, huge-service-deployment-0-5f48d94767-2t7ns, huge-service-deployment-0-5f48d94767-5glmg, huge-service-deployment-0-5f48d94767-v2lqt, huge-service-deployment-0-5f48d94767-qjkm7, huge-service-deployment-0-5f48d94767-f6wf5, huge-service-deployment-0-5f48d94767-99bnw, huge-service-deployment-0-5f48d94767-whrbm, huge-service-deployment-0-5f48d94767-hdj9t, huge-service-deployment-0-5f48d94767-jbrpk, huge-service-deployment-0-5f48d94767-ft65h, huge-service-deployment-0-5f48d94767-nnhkp, huge-service-deployment-0-5f48d94767-knsdx, huge-service-deployment-0-5f48d94767-8dkgl, huge-service-deployment-0-5f48d94767-jffl8, huge-service-deployment-0-5f48d94767-wczt7, huge-service-deployment-0-5f48d94767-q8chq, huge-service-deployment-0-5f48d94767-hnkjd, huge-service-deployment-0-5f48d94767-xllzn, huge-service-deployment-0-5f48d94767-ds78c, huge-service-deployment-0-5f48d94767-rclwm, huge-service-deployment-0-5f48d94767-knn27, huge-service-deployment-0-5f48d94767-6mxjc, huge-service-deployment-0-5f48d94767-fj8wb, huge-service-deployment-0-5f48d94767-k77rk, huge-service-deployment-0-5f48d94767-9gkld, huge-service-deployment-0-5f48d94767-c5wck, huge-service-deployment-0-5f48d94767-bwk24, huge-service-deployment-0-5f48d94767-n9sdk, huge-service-deployment-0-5f48d94767-7ltr2, huge-service-deployment-0-5f48d94767-5lckw, huge-service-deployment-0-5f48d94767-nnxkm, huge-service-deployment-0-5f48d94767-hdgb2, huge-service-deployment-0-5f48d94767-p88fz, huge-service-deployment-0-5f48d94767-fcxws, huge-service-deployment-0-5f48d94767-pdpqj, huge-service-deployment-0-5f48d94767-hz8fv, huge-service-deployment-0-5f48d94767-kn9bz, huge-service-deployment-0-5f48d94767-mrbgv, huge-service-deployment-0-5f48d94767-n7999, huge-service-deployment-0-5f48d94767-gg5vn, huge-service-deployment-0-5f48d94767-4dsl2, huge-service-deployment-0-5f48d94767-mkk9t, huge-service-deployment-0-5f48d94767-6x69b, huge-service-deployment-0-5f48d94767-lp28p, huge-service-deployment-0-5f48d94767-g2bkm, huge-service-deployment-0-5f48d94767-7gvrp, huge-service-deployment-0-5f48d94767-gh7h4, huge-service-deployment-0-5f48d94767-cnbtl, huge-service-deployment-0-5f48d94767-t65mq, huge-service-deployment-0-5f48d94767-p9jw8, huge-service-deployment-0-5f48d94767-hbnds, huge-service-deployment-0-5f48d94767-hmpwk, huge-service-deployment-0-5f48d94767-5h82b, huge-service-deployment-0-5f48d94767-xz7xq, huge-service-deployment-0-5f48d94767-ws9rq, huge-service-deployment-0-5f48d94767-9knjn, huge-service-deployment-0-5f48d94767-mkhfm, huge-service-deployment-0-5f48d94767-2smcq, huge-service-deployment-0-5f48d94767-6kh8d, huge-service-deployment-0-5f48d94767-lgl5p, huge-service-deployment-0-5f48d94767-hjzmz, huge-service-deployment-0-5f48d94767-lgkbt, huge-service-deployment-0-5f48d94767-tnls4, huge-service-deployment-0-5f48d94767-465sw, huge-service-deployment-0-5f48d94767-hbhvh, huge-service-deployment-0-5f48d94767-fpjds, huge-service-deployment-0-5f48d94767-4xsnq, huge-service-deployment-0-5f48d94767-nw7g6, huge-service-deployment-0-5f48d94767-rf7wg, huge-service-deployment-0-5f48d94767-qfxcz, huge-service-deployment-0-5f48d94767-wg9jn, huge-service-deployment-0-5f48d94767-xc2wm, huge-service-deployment-0-5f48d94767-wkjck, huge-service-deployment-0-5f48d94767-cvlpl, huge-service-deployment-0-5f48d94767-4d7cg, huge-service-deployment-0-5f48d94767-29t5p, huge-service-deployment-0-5f48d94767-r6kz6, huge-service-deployment-0-5f48d94767-4vqzh, huge-service-deployment-0-5f48d94767-nvhnf, huge-service-deployment-0-5f48d94767-rxv6z, huge-service-deployment-0-5f48d94767-pmdvp, huge-service-deployment-0-5f48d94767-rf7r4, huge-service-deployment-0-5f48d94767-zrxxc, huge-service-deployment-0-5f48d94767-nfkfm, huge-service-deployment-0-5f48d94767-d8ptm, huge-service-deployment-0-5f48d94767-z2nlt, huge-service-deployment-0-5f48d94767-zz9dt, huge-service-deployment-0-5f48d94767-4gtnc, huge-service-deployment-0-5f48d94767-qhdvk, huge-service-deployment-0-5f48d94767-p69m2, huge-service-deployment-0-5f48d94767-7p97m, huge-service-deployment-0-5f48d94767-s99s5, huge-service-deployment-0-5f48d94767-dd585, huge-service-deployment-0-5f48d94767-rnffr, huge-service-deployment-0-5f48d94767-lv9xm, huge-service-deployment-0-5f48d94767-gcsrv, huge-service-deployment-0-5f48d94767-sswjf, huge-service-deployment-0-5f48d94767-9tv6g, huge-service-deployment-0-5f48d94767-mmcwg, huge-service-deployment-0-5f48d94767-w46gz, huge-service-deployment-0-5f48d94767-9gkbk, huge-service-deployment-0-5f48d94767-gtlsh, huge-service-deployment-0-5f48d94767-t5728, huge-service-deployment-0-5f48d94767-l4m64, huge-service-deployment-0-5f48d94767-ts94s, huge-service-deployment-0-5f48d94767-5hw2m, huge-service-deployment-0-5f48d94767-9pnrn, huge-service-deployment-0-5f48d94767-tv74l, huge-service-deployment-0-5f48d94767-47528, huge-service-deployment-0-5f48d94767-dfb8c, huge-service-deployment-0-5f48d94767-rzxxn, huge-service-deployment-0-5f48d94767-rhbzf, huge-service-deployment-0-5f48d94767-hjc52, huge-service-deployment-0-5f48d94767-jllvw, huge-service-deployment-0-5f48d94767-fbwqv, huge-service-deployment-0-5f48d94767-vlj4b, huge-service-deployment-0-5f48d94767-npx9c, huge-service-deployment-0-5f48d94767-v7m4c, huge-service-deployment-0-5f48d94767-dl827, huge-service-deployment-0-5f48d94767-5t6pr, huge-service-deployment-0-5f48d94767-4wlxj, huge-service-deployment-0-5f48d94767-h4z7g, huge-service-deployment-0-5f48d94767-kbh4k, huge-service-deployment-0-5f48d94767-d5wkt, huge-service-deployment-0-5f48d94767-c5sd8, huge-service-deployment-0-5f48d94767-g5g8s, huge-service-deployment-0-5f48d94767-gwnjr, huge-service-deployment-0-5f48d94767-6x6c4, huge-service-deployment-0-5f48d94767-9bqpd, huge-service-deployment-0-5f48d94767-cspzl, huge-service-deployment-0-5f48d94767-mszcf, huge-service-deployment-0-5f48d94767-89t2s, huge-service-deployment-0-5f48d94767-g7s9s, huge-service-deployment-0-5f48d94767-8vs5p, huge-service-deployment-0-5f48d94767-jqg4h, huge-service-deployment-0-5f48d94767-jkhfr, huge-service-deployment-0-5f48d94767-qthsm, huge-service-deployment-0-5f48d94767-pmm6f, huge-service-deployment-0-5f48d94767-z5dfn, huge-service-deployment-0-5f48d94767-7kqxn, huge-service-deployment-0-5f48d94767-bhps9, huge-service-deployment-0-5f48d94767-sjqln, huge-service-deployment-0-5f48d94767-p8jmw, huge-service-deployment-0-5f48d94767-bphpj, huge-service-deployment-0-5f48d94767-qgkkw, huge-service-deployment-0-5f48d94767-thb8b, huge-service-deployment-0-5f48d94767-6smnr, huge-service-deployment-0-5f48d94767-4gpm8, huge-service-deployment-0-5f48d94767-6xtzn, huge-service-deployment-0-5f48d94767-hwk5h, huge-service-deployment-0-5f48d94767-qm9c7, huge-service-deployment-0-5f48d94767-nsj52, huge-service-deployment-0-5f48d94767-98f67, huge-service-deployment-0-5f48d94767-nc799, huge-service-deployment-0-5f48d94767-qmqkl, huge-service-deployment-0-5f48d94767-9b26j, huge-service-deployment-0-5f48d94767-s828z, huge-service-deployment-0-5f48d94767-vr5dx, huge-service-deployment-0-5f48d94767-8jlw8, huge-service-deployment-0-5f48d94767-8vvcp, huge-service-deployment-0-5f48d94767-7v7wf, huge-service-deployment-0-5f48d94767-vhwx9, huge-service-deployment-0-5f48d94767-nc8fb, huge-service-deployment-0-5f48d94767-b669t, huge-service-deployment-0-5f48d94767-d25hw, huge-service-deployment-0-5f48d94767-sxqtp, huge-service-deployment-0-5f48d94767-94ctf, huge-service-deployment-0-5f48d94767-dnpk9, huge-service-deployment-0-5f48d94767-vl79m, huge-service-deployment-0-5f48d94767-zd428, huge-service-deployment-0-5f48d94767-kgfhf, huge-service-deployment-0-5f48d94767-lvkgb, huge-service-deployment-0-5f48d94767-qkmwt, huge-service-deployment-0-5f48d94767-khbws, huge-service-deployment-0-5f48d94767-l8j2w, huge-service-deployment-0-5f48d94767-h586k, huge-service-deployment-0-5f48d94767-5brvd, huge-service-deployment-0-5f48d94767-zdq9r, huge-service-deployment-0-5f48d94767-7mhvn, huge-service-deployment-0-5f48d94767-lfdjp, huge-service-deployment-0-5f48d94767-bjm65, huge-service-deployment-0-5f48d94767-ff6q8, huge-service-deployment-0-5f48d94767-xwfbt, huge-service-deployment-0-5f48d94767-jxf5g, huge-service-deployment-0-5f48d94767-9pxt2, huge-service-deployment-0-5f48d94767-6f427, huge-service-deployment-0-5f48d94767-92g9d, huge-service-deployment-0-5f48d94767-j7mgq, huge-service-deployment-0-5f48d94767-vk5mh, huge-service-deployment-0-5f48d94767-p4hjz, huge-service-deployment-0-5f48d94767-h4kfd, huge-service-deployment-0-5f48d94767-nxp5r, huge-service-deployment-0-5f48d94767-kz945, huge-service-deployment-0-5f48d94767-8fb44, huge-service-deployment-0-5f48d94767-7whdk, huge-service-deployment-0-5f48d94767-8tgjf, huge-service-deployment-0-5f48d94767-qvfrq, huge-service-deployment-0-5f48d94767-bvdp5, huge-service-deployment-0-5f48d94767-vf8l7, huge-service-deployment-0-5f48d94767-r6k8k, huge-service-deployment-0-5f48d94767-cg6sg, huge-service-deployment-0-5f48d94767-6tpx8, huge-service-deployment-0-5f48d94767-k8qbf, huge-service-deployment-0-5f48d94767-zdj29, huge-service-deployment-0-5f48d94767-fq4sn, huge-service-deployment-0-5f48d94767-7d56f, huge-service-deployment-0-5f48d94767-9mcll, huge-service-deployment-0-5f48d94767-jdp27, huge-service-deployment-0-5f48d94767-ld9nm, huge-service-deployment-0-5f48d94767-9kc4z, huge-service-deployment-0-5f48d94767-pclmt, huge-service-deployment-0-5f48d94767-tq2d2, huge-service-deployment-0-5f48d94767-rscgc, huge-service-deployment-0-5f48d94767-hc7nw, huge-service-deployment-0-5f48d94767-j4hb4, huge-service-deployment-0-5f48d94767-d7n6t, huge-service-deployment-0-5f48d94767-bprsc, huge-service-deployment-0-5f48d94767-v54rn, huge-service-deployment-0-5f48d94767-gc82l, huge-service-deployment-0-5f48d94767-jntw5, huge-service-deployment-0-5f48d94767-vhgr8
I1124 13:40:39.549701   89755 wait_for_pods.go:111] WaitForControlledPodsRunning: namespace(test-mbfsuq-1), controlledBy(huge-service-deployment-0): Pods: 1115 out of 1000 created, 954 running (692 updated), 159 pending scheduled, 2 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady 
W1124 13:40:44.567656   89755 wait_for_pods.go:108] WaitForControlledPodsRunning: namespace(test-mbfsuq-1), controlledBy(huge-service-deployment-0): 147 pods appeared: huge-service-deployment-0-5f48d94767-rwgtz, huge-service-deployment-0-5f48d94767-hgzcs, huge-service-deployment-0-5f48d94767-bws9r, huge-service-deployment-0-5f48d94767-p6dc9, huge-service-deployment-0-5f48d94767-r8dpl, huge-service-deployment-0-5f48d94767-rk9ms, huge-service-deployment-0-5f48d94767-5wb7f, huge-service-deployment-0-5f48d94767-p4k9n, huge-service-deployment-0-5f48d94767-bxwpk, huge-service-deployment-0-5f48d94767-xl24k, huge-service-deployment-0-5f48d94767-598b9, huge-service-deployment-0-5f48d94767-7qgrj, huge-service-deployment-0-5f48d94767-wbkqp, huge-service-deployment-0-5f48d94767-c5l8h, huge-service-deployment-0-5f48d94767-7jwng, huge-service-deployment-0-5f48d94767-b46rc, huge-service-deployment-0-5f48d94767-rz94q, huge-service-deployment-0-5f48d94767-xhnf4, huge-service-deployment-0-5f48d94767-t84dw, huge-service-deployment-0-5f48d94767-2x4dq, huge-service-deployment-0-5f48d94767-tkrkc, huge-service-deployment-0-5f48d94767-ld562, huge-service-deployment-0-5f48d94767-fj6xj, huge-service-deployment-0-5f48d94767-sv4mc, huge-service-deployment-0-5f48d94767-bsjdt, huge-service-deployment-0-5f48d94767-2nzvw, huge-service-deployment-0-5f48d94767-jn2zl, huge-service-deployment-0-5f48d94767-gwvlq, huge-service-deployment-0-5f48d94767-rw2sg, huge-service-deployment-0-5f48d94767-68wwt, huge-service-deployment-0-5f48d94767-4trt5, huge-service-deployment-0-5f48d94767-qrws8, huge-service-deployment-0-5f48d94767-vzzrs, huge-service-deployment-0-5f48d94767-95hpk, huge-service-deployment-0-5f48d94767-kctnb, huge-service-deployment-0-5f48d94767-q6bsg, huge-service-deployment-0-5f48d94767-jzxvc, huge-service-deployment-0-5f48d94767-c9vc7, huge-service-deployment-0-5f48d94767-6v8wh, huge-service-deployment-0-5f48d94767-9w7qc, huge-service-deployment-0-5f48d94767-269q4, huge-service-deployment-0-5f48d94767-jdn9t, huge-service-deployment-0-5f48d94767-sknkm, huge-service-deployment-0-5f48d94767-5hftj, huge-service-deployment-0-5f48d94767-7xhc4, huge-service-deployment-0-5f48d94767-s6dl8, huge-service-deployment-0-5f48d94767-rnv87, huge-service-deployment-0-5f48d94767-5rptc, huge-service-deployment-0-5f48d94767-h7dzp, huge-service-deployment-0-5f48d94767-782wl, huge-service-deployment-0-5f48d94767-hrcdk, huge-service-deployment-0-5f48d94767-ggthn, huge-service-deployment-0-5f48d94767-vpjkj, huge-service-deployment-0-5f48d94767-d6tfw, huge-service-deployment-0-5f48d94767-ks55q, huge-service-deployment-0-5f48d94767-htkzm, huge-service-deployment-0-5f48d94767-xwtmw, huge-service-deployment-0-5f48d94767-ch7rh, huge-service-deployment-0-5f48d94767-4fb5f, huge-service-deployment-0-5f48d94767-q4vwl, huge-service-deployment-0-5f48d94767-zl4zb, huge-service-deployment-0-5f48d94767-qlkbw, huge-service-deployment-0-5f48d94767-fwrgj, huge-service-deployment-0-5f48d94767-mfqkk, huge-service-deployment-0-5f48d94767-bgln7, huge-service-deployment-0-5f48d94767-dcb4p, huge-service-deployment-0-5f48d94767-phfkk, huge-service-deployment-0-5f48d94767-lkk6t, huge-service-deployment-0-5f48d94767-7w6qg, huge-service-deployment-0-5f48d94767-fjxhr, huge-service-deployment-0-5f48d94767-ms7jm, huge-service-deployment-0-5f48d94767-99972, huge-service-deployment-0-5f48d94767-94hdp, huge-service-deployment-0-5f48d94767-5tnws, huge-service-deployment-0-5f48d94767-l8z7l, huge-service-deployment-0-5f48d94767-bl6f8, huge-service-deployment-0-5f48d94767-9pk9j, huge-service-deployment-0-5f48d94767-h7mrw, huge-service-deployment-0-5f48d94767-d8zmg, huge-service-deployment-0-5f48d94767-vn6wc, huge-service-deployment-0-5f48d94767-59vvs, huge-service-deployment-0-5f48d94767-x7dh9, huge-service-deployment-0-5f48d94767-44dbf, huge-service-deployment-0-5f48d94767-jnnqn, huge-service-deployment-0-5f48d94767-nvrc8, huge-service-deployment-0-5f48d94767-7lbj5, huge-service-deployment-0-5f48d94767-tvhjr, huge-service-deployment-0-5f48d94767-zwfr6, huge-service-deployment-0-5f48d94767-h4ct5, huge-service-deployment-0-5f48d94767-qcnkx, huge-service-deployment-0-5f48d94767-qfdmh, huge-service-deployment-0-5f48d94767-9rnbz, huge-service-deployment-0-5f48d94767-tbmzx, huge-service-deployment-0-5f48d94767-svbjv, huge-service-deployment-0-5f48d94767-5wjqb, huge-service-deployment-0-5f48d94767-g5nkn, huge-service-deployment-0-5f48d94767-862rs, huge-service-deployment-0-5f48d94767-kg5cl, huge-service-deployment-0-5f48d94767-kqvg2, huge-service-deployment-0-5f48d94767-sgg9s, huge-service-deployment-0-5f48d94767-tpz7h, huge-service-deployment-0-5f48d94767-sws9s, huge-service-deployment-0-5f48d94767-jrmmm, huge-service-deployment-0-5f48d94767-p6vj8, huge-service-deployment-0-5f48d94767-dnvws, huge-service-deployment-0-5f48d94767-kc49k, huge-service-deployment-0-5f48d94767-fw9t8, huge-service-deployment-0-5f48d94767-4k45v, huge-service-deployment-0-5f48d94767-mzqcz, huge-service-deployment-0-5f48d94767-xpkp9, huge-service-deployment-0-5f48d94767-8spln, huge-service-deployment-0-5f48d94767-wp5jr, huge-service-deployment-0-5f48d94767-p6g4g, huge-service-deployment-0-5f48d94767-w9wvp, huge-service-deployment-0-5f48d94767-pwzcq, huge-service-deployment-0-5f48d94767-7fths, huge-service-deployment-0-5f48d94767-jgf64, huge-service-deployment-0-5f48d94767-r56kx, huge-service-deployment-0-5f48d94767-jbxlh, huge-service-deployment-0-5f48d94767-srzsd, huge-service-deployment-0-5f48d94767-7lr6j, huge-service-deployment-0-5f48d94767-9srhx, huge-service-deployment-0-5f48d94767-5l8qd, huge-service-deployment-0-5f48d94767-78g9g, huge-service-deployment-0-5f48d94767-926wl, huge-service-deployment-0-5f48d94767-pdtdw, huge-service-deployment-0-5f48d94767-4rww4, huge-service-deployment-0-5f48d94767-hkhnb, huge-service-deployment-0-5f48d94767-4mdm7, huge-service-deployment-0-5f48d94767-wrf4v, huge-service-deployment-0-5f48d94767-972zw, huge-service-deployment-0-5f48d94767-ndq87, huge-service-deployment-0-5f48d94767-m5wgh, huge-service-deployment-0-5f48d94767-6hx24, huge-service-deployment-0-5f48d94767-kmbrh, huge-service-deployment-0-5f48d94767-ffq2h, huge-service-deployment-0-5f48d94767-xphzb, huge-service-deployment-0-5f48d94767-6hvv7, huge-service-deployment-0-5f48d94767-glxqr, huge-service-deployment-0-5f48d94767-87stv, huge-service-deployment-0-5f48d94767-fzsx6, huge-service-deployment-0-5f48d94767-9flz2, huge-service-deployment-0-5f48d94767-7k66n, huge-service-deployment-0-5f48d94767-zfrl6, huge-service-deployment-0-5f48d94767-vjxwm, huge-service-deployment-0-5f48d94767-k825b, huge-service-deployment-0-5f48d94767-gbjp7
I1124 13:40:44.567704   89755 wait_for_pods.go:111] WaitForControlledPodsRunning: namespace(test-mbfsuq-1), controlledBy(huge-service-deployment-0): Pods: 1000 out of 1000 created, 998 running (998 updated), 2 pending scheduled, 0 not scheduled, 0 inactive, 1 terminating, 0 unknown, 0 runningButNotReady 
I1124 13:40:49.589817   89755 wait_for_pods.go:111] WaitForControlledPodsRunning: namespace(test-mbfsuq-1), controlledBy(huge-service-deployment-0): Pods: 1000 out of 1000 created, 1000 running (1000 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady 
I1124 13:40:49.589869   89755 wait_for_controlled_pods.go:319] WaitForControlledPodsRunning: running 1, deleted 0, timeout: 0, failed: 0
I1124 13:40:49.589879   89755 wait_for_controlled_pods.go:324] WaitForControlledPodsRunning: maxDuration=25.703669313s, operationTimeout=30m0s, ratio=0.01
I1124 13:40:49.589891   89755 wait_for_controlled_pods.go:338] WaitForControlledPodsRunning: 1/1 Deployments are running with all pods
I1124 13:40:49.589909   89755 simple_test_executor.go:171] Step "[step: 07] Waiting for huge-service pods to be updated" ended
I1124 13:40:49.589924   89755 simple_test_executor.go:149] Step "[step: 08] Deleting huge-service pods" started
I1124 13:40:49.626770   89755 simple_test_executor.go:171] Step "[step: 08] Deleting huge-service pods" ended
I1124 13:40:49.626799   89755 simple_test_executor.go:149] Step "[step: 09] Waiting for huge-service pods to be deleted" started
I1124 13:40:49.626831   89755 wait_for_controlled_pods.go:246] WaitForControlledPodsRunning: waiting for controlled pods measurement...
I1124 13:40:49.626889   89755 wait_for_pods.go:61] WaitForControlledPodsRunning: namespace(test-mbfsuq-1), controlledBy(huge-service-deployment-0): starting with timeout: 29m59.999985393s
I1124 13:40:54.636430   89755 wait_for_pods.go:111] WaitForControlledPodsRunning: namespace(test-mbfsuq-1), controlledBy(huge-service-deployment-0): Pods: 459 out of 0 created, 459 running (459 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 79 terminating, 0 unknown, 0 runningButNotReady 
I1124 13:40:59.637478   89755 wait_for_pods.go:111] WaitForControlledPodsRunning: namespace(test-mbfsuq-1), controlledBy(huge-service-deployment-0): Pods: 0 out of 0 created, 0 running (0 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 36 terminating, 0 unknown, 0 runningButNotReady 
I1124 13:41:04.638429   89755 wait_for_pods.go:111] WaitForControlledPodsRunning: namespace(test-mbfsuq-1), controlledBy(huge-service-deployment-0): Pods: 0 out of 0 created, 0 running (0 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady 
I1124 13:41:04.638484   89755 wait_for_controlled_pods.go:319] WaitForControlledPodsRunning: running 0, deleted 1, timeout: 0, failed: 0
I1124 13:41:04.638495   89755 wait_for_controlled_pods.go:324] WaitForControlledPodsRunning: maxDuration=15.011572863s, operationTimeout=30m0s, ratio=0.01
I1124 13:41:04.638512   89755 wait_for_controlled_pods.go:338] WaitForControlledPodsRunning: 0/0 Deployments are running with all pods
I1124 13:41:04.638547   89755 simple_test_executor.go:171] Step "[step: 09] Waiting for huge-service pods to be deleted" ended
I1124 13:41:04.638575   89755 simple_test_executor.go:149] Step "[step: 10] Delete huge-service" started
I1124 13:41:04.682419   89755 simple_test_executor.go:171] Step "[step: 10] Delete huge-service" ended
I1124 13:41:04.682447   89755 simple_test_executor.go:149] Step "[step: 11] gathering measurements" started
... skipping 181 lines ...
Specify --start=76680 in the next get-serial-port-output invocation to get only the new output starting from here.
scp: /var/log/cluster-autoscaler.log*: No such file or directory
scp: /var/log/fluentd.log*: No such file or directory
scp: /var/log/kubelet.cov*: No such file or directory
scp: /var/log/cl2-**: No such file or directory
scp: /var/log/startupscript.log*: No such file or directory
ERROR: (gcloud.compute.scp) [/usr/bin/scp] exited with return code [1].
Dumping logs from nodes to GCS directly at 'gs://sig-scalability-logs/pull-kubernetes-e2e-gce-100-performance/1595763177910439936' using logexporter
namespace/logexporter created
secret/google-service-account created
daemonset.apps/logexporter created
Listing marker files (gs://sig-scalability-logs/pull-kubernetes-e2e-gce-100-performance/1595763177910439936/logexported-nodes-registry) for successful nodes...
CommandException: One or more URLs matched no objects.
... skipping 366 lines ...