This job view page is being replaced by Spyglass soon. Check out the new job view.
PRdimakow: Add missing locks for device-manager in 1.19
ResultFAILURE
Tests 1 failed / 956 succeeded
Started2021-07-14 14:33
Elapsed32m12s
Revision6e340d89497bb60251fd46f1be5f89b5165f5bb0
Refs 103262

Test Failures


//pkg/kubelet/cm/devicemanager:go_default_test 0.00s

bazel test //pkg/kubelet/cm/devicemanager:go_default_test
exec ${PAGER:-/usr/bin/less} "$0" || exit 1
Executing tests from //pkg/kubelet/cm/devicemanager:go_default_test
-----------------------------------------------------------------------------
I0714 14:59:59.787499  241323 device_plugin_stub.go:131] Starting to serve on /tmp/mock.sock
I0714 14:59:59.790840  241323 device_plugin_stub.go:131] Starting to serve on /tmp/mock.sock
I0714 14:59:59.794305  241323 device_plugin_stub.go:227] ListAndWatch
E0714 14:59:59.795569  241323 endpoint.go:107] listAndWatch ended unexpectedly for device plugin mock with error rpc error: code = Unavailable desc = transport is closing
I0714 14:59:59.797227  241323 device_plugin_stub.go:131] Starting to serve on /tmp/mock.sock
I0714 14:59:59.801028  241323 device_plugin_stub.go:227] ListAndWatch
I0714 14:59:59.802076  241323 device_plugin_stub.go:261] Allocate, &AllocateRequest{ContainerRequests:[]*ContainerAllocateRequest{&ContainerAllocateRequest{DevicesIDs:[ADeviceId],},},}
E0714 14:59:59.802903  241323 endpoint.go:107] listAndWatch ended unexpectedly for device plugin mock with error rpc error: code = Unavailable desc = transport is closing
I0714 14:59:59.804966  241323 device_plugin_stub.go:131] Starting to serve on /tmp/mock.sock
I0714 14:59:59.807112  241323 device_plugin_stub.go:227] ListAndWatch
I0714 14:59:59.808115  241323 device_plugin_stub.go:248] GetPreferredAllocation, &PreferredAllocationRequest{ContainerRequests:[]*ContainerPreferredAllocationRequest{&ContainerPreferredAllocationRequest{AvailableDeviceIDs:[],MustIncludeDeviceIDs:[],AllocationSize:-1,},},}
E0714 14:59:59.809118  241323 endpoint.go:107] listAndWatch ended unexpectedly for device plugin mock with error rpc error: code = Unavailable desc = transport is closing
I0714 14:59:59.809419  241323 fake_topology_manager.go:29] [fake topologymanager] NewFakeManager
I0714 14:59:59.809881  241323 fake_topology_manager.go:29] [fake topologymanager] NewFakeManager
W0714 14:59:59.810275  241323 manager.go:596] Failed to retrieve checkpoint for "kubelet_internal_checkpoint": checkpoint is not found
I0714 14:59:59.812126  241323 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin761044901/device-plugin.sock
I0714 14:59:59.812922  241323 fake_topology_manager.go:29] [fake topologymanager] NewFakeManager
W0714 14:59:59.813096  241323 manager.go:596] Failed to retrieve checkpoint for "kubelet_internal_checkpoint": checkpoint is not found
I0714 14:59:59.813915  241323 plugin_manager.go:114] Starting Kubelet Plugin Manager
I0714 14:59:59.816707  241323 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin983123136/device-plugin.sock
I0714 14:59:59.818982  241323 fake_topology_manager.go:29] [fake topologymanager] NewFakeManager
W0714 14:59:59.819219  241323 manager.go:596] Failed to retrieve checkpoint for "kubelet_internal_checkpoint": checkpoint is not found
I0714 14:59:59.820785  241323 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin806971679/device-plugin.sock
I0714 14:59:59.820859  241323 device_plugin_stub.go:180] Deprecation file not found. Invoke registration
I0714 14:59:59.822604  241323 manager.go:410] Got registration request from device plugin with resource name "fake-domain/resource"
I0714 14:59:59.825278  241323 device_plugin_stub.go:227] ListAndWatch
I0714 14:59:59.832540  241323 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin806971679/device-plugin.sock.new
I0714 14:59:59.832595  241323 device_plugin_stub.go:180] Deprecation file not found. Invoke registration
I0714 14:59:59.834412  241323 manager.go:410] Got registration request from device plugin with resource name "fake-domain/resource"
I0714 14:59:59.838329  241323 device_plugin_stub.go:227] ListAndWatch
I0714 14:59:59.843272  241323 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin806971679/device-plugin.sock.third
I0714 14:59:59.843333  241323 device_plugin_stub.go:180] Deprecation file not found. Invoke registration
I0714 14:59:59.845269  241323 manager.go:410] Got registration request from device plugin with resource name "fake-domain/resource"
I0714 14:59:59.847579  241323 device_plugin_stub.go:227] ListAndWatch
E0714 14:59:59.851372  241323 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Unavailable desc = transport is closing
E0714 14:59:59.851642  241323 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Unavailable desc = transport is closing
E0714 14:59:59.851964  241323 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Unavailable desc = transport is closing
I0714 14:59:59.852153  241323 fake_topology_manager.go:29] [fake topologymanager] NewFakeManager
I0714 14:59:59.854342  241323 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin806971679/device-plugin.sock
I0714 14:59:59.854446  241323 device_plugin_stub.go:180] Deprecation file not found. Invoke registration
I0714 14:59:59.856755  241323 manager.go:410] Got registration request from device plugin with resource name "fake-domain/resource"
I0714 14:59:59.860332  241323 device_plugin_stub.go:227] ListAndWatch
I0714 14:59:59.865455  241323 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin806971679/device-plugin.sock.new
I0714 14:59:59.865526  241323 device_plugin_stub.go:180] Deprecation file not found. Invoke registration
I0714 14:59:59.867068  241323 manager.go:410] Got registration request from device plugin with resource name "fake-domain/resource"
I0714 14:59:59.869569  241323 device_plugin_stub.go:227] ListAndWatch
I0714 14:59:59.874384  241323 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin806971679/device-plugin.sock.third
I0714 14:59:59.874453  241323 device_plugin_stub.go:180] Deprecation file not found. Invoke registration
I0714 14:59:59.876162  241323 manager.go:410] Got registration request from device plugin with resource name "fake-domain/resource"
I0714 14:59:59.878339  241323 device_plugin_stub.go:227] ListAndWatch
E0714 14:59:59.882387  241323 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Unavailable desc = transport is closing
E0714 14:59:59.882589  241323 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Unavailable desc = transport is closing
E0714 14:59:59.883187  241323 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Unavailable desc = transport is closing
I0714 14:59:59.883900  241323 fake_topology_manager.go:29] [fake topologymanager] NewFakeManager
I0714 14:59:59.886619  241323 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin806971679/device-plugin.sock
I0714 14:59:59.886669  241323 device_plugin_stub.go:180] Deprecation file not found. Invoke registration
I0714 14:59:59.888256  241323 manager.go:410] Got registration request from device plugin with resource name "fake-domain/resource"
I0714 14:59:59.890569  241323 device_plugin_stub.go:227] ListAndWatch
I0714 14:59:59.895278  241323 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin806971679/device-plugin.sock.new
I0714 14:59:59.895352  241323 device_plugin_stub.go:180] Deprecation file not found. Invoke registration
I0714 14:59:59.897149  241323 manager.go:410] Got registration request from device plugin with resource name "fake-domain/resource"
I0714 14:59:59.900658  241323 device_plugin_stub.go:227] ListAndWatch
I0714 14:59:59.905471  241323 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin806971679/device-plugin.sock.third
I0714 14:59:59.905538  241323 device_plugin_stub.go:180] Deprecation file not found. Invoke registration
I0714 14:59:59.906984  241323 manager.go:410] Got registration request from device plugin with resource name "fake-domain/resource"
I0714 14:59:59.909197  241323 device_plugin_stub.go:227] ListAndWatch
E0714 14:59:59.912887  241323 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Unavailable desc = transport is closing
E0714 14:59:59.913203  241323 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Unavailable desc = transport is closing
E0714 14:59:59.913628  241323 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Unavailable desc = transport is closing
I0714 14:59:59.913675  241323 fake_topology_manager.go:29] [fake topologymanager] NewFakeManager
I0714 14:59:59.915535  241323 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin806971679/device-plugin.sock
I0714 14:59:59.915609  241323 device_plugin_stub.go:180] Deprecation file not found. Invoke registration
I0714 14:59:59.917108  241323 manager.go:410] Got registration request from device plugin with resource name "fake-domain/resource"
I0714 14:59:59.919380  241323 device_plugin_stub.go:227] ListAndWatch
I0714 14:59:59.924406  241323 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin806971679/device-plugin.sock.new
I0714 14:59:59.924515  241323 device_plugin_stub.go:180] Deprecation file not found. Invoke registration
I0714 14:59:59.926831  241323 manager.go:410] Got registration request from device plugin with resource name "fake-domain/resource"
I0714 14:59:59.929021  241323 device_plugin_stub.go:227] ListAndWatch
I0714 14:59:59.933505  241323 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin806971679/device-plugin.sock.third
I0714 14:59:59.933560  241323 device_plugin_stub.go:180] Deprecation file not found. Invoke registration
I0714 14:59:59.935116  241323 manager.go:410] Got registration request from device plugin with resource name "fake-domain/resource"
I0714 14:59:59.937574  241323 device_plugin_stub.go:227] ListAndWatch
E0714 14:59:59.941112  241323 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Unavailable desc = transport is closing
E0714 14:59:59.941535  241323 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Unavailable desc = transport is closing
E0714 14:59:59.941615  241323 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Unavailable desc = transport is closing
I0714 14:59:59.942327  241323 fake_topology_manager.go:29] [fake topologymanager] NewFakeManager
W0714 14:59:59.942459  241323 manager.go:596] Failed to retrieve checkpoint for "kubelet_internal_checkpoint": checkpoint is not found
I0714 14:59:59.943228  241323 plugin_manager.go:114] Starting Kubelet Plugin Manager
I0714 14:59:59.944827  241323 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin564962222/device-plugin.sock
I0714 14:59:59.947507  241323 device_plugin_stub.go:153] GetInfo
E0714 14:59:59.948053  241323 goroutinemap.go:150] Operation for "/tmp/device_plugin564962222/server.sock" failed. No retries permitted until 2021-07-14 15:00:00.447902405 +0000 UTC m=+0.740331954 (durationBeforeRetry 500ms). Error: "RegisterPlugin error -- failed to get plugin info using RPC GetInfo at socket /tmp/device_plugin564962222/server.sock, err: rpc error: code = Unimplemented desc = unknown service pluginregistration.Registration"
I0714 14:59:59.950775  241323 device_plugin_stub.go:227] ListAndWatch
I0714 14:59:59.955483  241323 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin564962222/device-plugin.sock.new
I0714 15:00:00.947288  241323 device_plugin_stub.go:153] GetInfo
E0714 15:00:00.948187  241323 goroutinemap.go:150] Operation for "/tmp/device_plugin564962222/server.sock" failed. No retries permitted until 2021-07-14 15:00:01.948008509 +0000 UTC m=+2.240438078 (durationBeforeRetry 1s). Error: "RegisterPlugin error -- failed to get plugin info using RPC GetInfo at socket /tmp/device_plugin564962222/server.sock, err: rpc error: code = Unimplemented desc = unknown service pluginregistration.Registration"
I0714 15:00:00.952643  241323 device_plugin_stub.go:227] ListAndWatch
I0714 15:00:00.957616  241323 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin564962222/device-plugin.sock.third
I0714 15:00:01.948195  241323 device_plugin_stub.go:153] GetInfo
I0714 15:00:01.952409  241323 device_plugin_stub.go:227] ListAndWatch
E0714 15:00:01.956921  241323 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Unavailable desc = transport is closing
E0714 15:00:01.957156  241323 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Unavailable desc = transport is closing
E0714 15:00:01.957727  241323 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Unavailable desc = transport is closing
I0714 15:00:01.958390  241323 fake_topology_manager.go:29] [fake topologymanager] NewFakeManager
E0714 15:00:01.972897  241323 manager.go:546] unexpected: unhealthyDevices and endpoints are out of sync
I0714 15:00:01.984593  241323 fake_topology_manager.go:29] [fake topologymanager] NewFakeManager
I0714 15:00:01.984788  241323 fake_topology_manager.go:34] [fake topologymanager] GetAffinity podUID: 9be008eb-678c-44e9-856f-8ea2ce897aac container name:  
I0714 15:00:01.984858  241323 fake_topology_manager.go:34] [fake topologymanager] GetAffinity podUID: 9be008eb-678c-44e9-856f-8ea2ce897aac container name:  
I0714 15:00:01.988316  241323 fake_topology_manager.go:34] [fake topologymanager] GetAffinity podUID: 4952d042-6045-486a-b0ee-3252964f1a5c container name:  
I0714 15:00:01.992281  241323 fake_topology_manager.go:29] [fake topologymanager] NewFakeManager
I0714 15:00:01.992474  241323 fake_topology_manager.go:34] [fake topologymanager] GetAffinity podUID: 920edb14-d436-4b99-8614-490d1f1fa932 container name:  2bd8d7a2-c25c-423f-a403-131ba13f10f7
I0714 15:00:01.995814  241323 fake_topology_manager.go:34] [fake topologymanager] GetAffinity podUID: 920edb14-d436-4b99-8614-490d1f1fa932 container name:  751f7c8c-4747-41d2-9b7b-23e3aae72269
I0714 15:00:01.999319  241323 fake_topology_manager.go:34] [fake topologymanager] GetAffinity podUID: 920edb14-d436-4b99-8614-490d1f1fa932 container name:  f4d3c1ce-e306-476d-b5b9-18c0fd615839
I0714 15:00:02.003497  241323 fake_topology_manager.go:34] [fake topologymanager] GetAffinity podUID: 920edb14-d436-4b99-8614-490d1f1fa932 container name:  ad686801-0745-4fcc-9057-56fae15b8fe8
I0714 15:00:02.010041  241323 fake_topology_manager.go:29] [fake topologymanager] NewFakeManager
I0714 15:00:02.010157  241323 fake_topology_manager.go:34] [fake topologymanager] GetAffinity podUID: ad747fc5-f962-4637-bb02-3cd79edba709 container name:  
I0714 15:00:02.017824  241323 topology_hints.go:49] [devicemanager] Resource 'testdevice' does not have a topology preference
E0714 15:00:09.815932  241323 goroutinemap.go:150] Operation for "/tmp/device_plugin983123136/device-plugin.sock" failed. No retries permitted until 2021-07-14 15:00:10.315760942 +0000 UTC m=+10.608190509 (durationBeforeRetry 500ms). Error: "RegisterPlugin error -- dial failed at socket /tmp/device_plugin983123136/device-plugin.sock, err: failed to dial socket /tmp/device_plugin983123136/device-plugin.sock, err: context deadline exceeded"
E0714 15:00:09.816266  241323 goroutinemap.go:150] Operation for "/tmp/device_plugin983123136/server.sock" failed. No retries permitted until 2021-07-14 15:00:10.316089873 +0000 UTC m=+10.608519433 (durationBeforeRetry 500ms). Error: "RegisterPlugin error -- dial failed at socket /tmp/device_plugin983123136/server.sock, err: failed to dial socket /tmp/device_plugin983123136/server.sock, err: context deadline exceeded"
Terminated
-- Test timed out at 2021-07-14 15:04:59 UTC --

				from junit_bazel.xml

Filter through log files | View test history on testgrid


Show 956 Passed Tests

Error lines from build-log.txt

+ bazel test --config=unit //... -- -//build/... -//vendor/...
$TEST_TMPDIR defined: output root default is '/bazel-scratch/.cache/bazel' and max_idle_secs default is '15'.
$TEST_TMPDIR defined: output root default is '/bazel-scratch/.cache/bazel' and max_idle_secs default is '15'.
Extracting Bazel installation...
Starting local Bazel server and connecting to it...
WARNING: Download from https://storage.googleapis.com/k8s-bazel-cache/https://github.com/bazelbuild/rules_docker/releases/download/v0.14.4/rules_docker-v0.14.4.tar.gz failed: class com.google.devtools.build.lib.bazel.repository.downloader.UnrecoverableHttpException GET returned 404 Not Found
Loading:  (1 packages loaded)
Loading: 1 packages loaded
Loading: 220 packages loaded
    currently loading: test/e2e/framework/replicaset ... (3 packages)
Loading: 1192 packages loaded
    currently loading: translations
... skipping 4 lines ...
Analyzing: 959 targets (4556 packages loaded, 29053 targets configured)
Analyzing: 959 targets (4568 packages loaded, 36329 targets configured)
Analyzing: 959 targets (4572 packages loaded, 36631 targets configured)
Analyzing: 959 targets (4572 packages loaded, 36631 targets configured)
Analyzing: 959 targets (4573 packages loaded, 36754 targets configured)
Analyzing: 959 targets (4573 packages loaded, 36754 targets configured)
DEBUG: /bazel-scratch/.cache/bazel/_bazel_root/7989b31489f31aee54f32688da2f0120/external/bazel_gazelle/internal/go_repository.bzl:189:13: org_golang_x_tools: gazelle: /bazel-scratch/.cache/bazel/_bazel_root/7989b31489f31aee54f32688da2f0120/external/org_golang_x_tools/cmd/fiximports/testdata/src/old.com/bad/bad.go: error reading go file: /bazel-scratch/.cache/bazel/_bazel_root/7989b31489f31aee54f32688da2f0120/external/org_golang_x_tools/cmd/fiximports/testdata/src/old.com/bad/bad.go:2:43: expected 'package', found 'EOF'
gazelle: found packages notinheap (notinheap.go) and issue31540 (issue31540.go) in /bazel-scratch/.cache/bazel/_bazel_root/7989b31489f31aee54f32688da2f0120/external/org_golang_x_tools/go/internal/gccgoimporter/testdata
gazelle: found packages a (a.go) and b (b.go) in /bazel-scratch/.cache/bazel/_bazel_root/7989b31489f31aee54f32688da2f0120/external/org_golang_x_tools/go/internal/gcimporter/testdata
gazelle: /bazel-scratch/.cache/bazel/_bazel_root/7989b31489f31aee54f32688da2f0120/external/org_golang_x_tools/go/loader/testdata/badpkgdecl.go: error reading go file: /bazel-scratch/.cache/bazel/_bazel_root/7989b31489f31aee54f32688da2f0120/external/org_golang_x_tools/go/loader/testdata/badpkgdecl.go:1:34: expected 'package', found 'EOF'
gazelle: /bazel-scratch/.cache/bazel/_bazel_root/7989b31489f31aee54f32688da2f0120/external/org_golang_x_tools/go/packages/packagestest/testdata/groups/two/modules/example.com/extra/geez/help.go: error reading go file: /bazel-scratch/.cache/bazel/_bazel_root/7989b31489f31aee54f32688da2f0120/external/org_golang_x_tools/go/packages/packagestest/testdata/groups/two/modules/example.com/extra/geez/help.go:1:16: expected ';', found '.'
gazelle: /bazel-scratch/.cache/bazel/_bazel_root/7989b31489f31aee54f32688da2f0120/external/org_golang_x_tools/go/packages/packagestest/testdata/groups/two/modules/example.com/extra/v2/geez/help.go: error reading go file: /bazel-scratch/.cache/bazel/_bazel_root/7989b31489f31aee54f32688da2f0120/external/org_golang_x_tools/go/packages/packagestest/testdata/groups/two/modules/example.com/extra/v2/geez/help.go:1:16: expected ';', found '.'
gazelle: /bazel-scratch/.cache/bazel/_bazel_root/7989b31489f31aee54f32688da2f0120/external/org_golang_x_tools/go/packages/packagestest/testdata/groups/two/modules/example.com/extra/v2/me.go: error reading go file: /bazel-scratch/.cache/bazel/_bazel_root/7989b31489f31aee54f32688da2f0120/external/org_golang_x_tools/go/packages/packagestest/testdata/groups/two/modules/example.com/extra/v2/me.go:1:16: expected ';', found '.'
gazelle: /bazel-scratch/.cache/bazel/_bazel_root/7989b31489f31aee54f32688da2f0120/external/org_golang_x_tools/go/packages/packagestest/testdata/groups/two/modules/example.com/extra/yo.go: error reading go file: /bazel-scratch/.cache/bazel/_bazel_root/7989b31489f31aee54f32688da2f0120/external/org_golang_x_tools/go/packages/packagestest/testdata/groups/two/modules/example.com/extra/yo.go:1:16: expected ';', found '.'
gazelle: /bazel-scratch/.cache/bazel/_bazel_root/7989b31489f31aee54f32688da2f0120/external/org_golang_x_tools/go/packages/packagestest/testdata/groups/two/modules/example.com/tempmod/main.go: error reading go file: /bazel-scratch/.cache/bazel/_bazel_root/7989b31489f31aee54f32688da2f0120/external/org_golang_x_tools/go/packages/packagestest/testdata/groups/two/modules/example.com/tempmod/main.go:1:16: expected ';', found '.'
gazelle: /bazel-scratch/.cache/bazel/_bazel_root/7989b31489f31aee54f32688da2f0120/external/org_golang_x_tools/go/packages/packagestest/testdata/groups/two/modules/example.com/what@v1.0.0/main.go: error reading go file: /bazel-scratch/.cache/bazel/_bazel_root/7989b31489f31aee54f32688da2f0120/external/org_golang_x_tools/go/packages/packagestest/testdata/groups/two/modules/example.com/what@v1.0.0/main.go:1:16: expected ';', found '.'
gazelle: /bazel-scratch/.cache/bazel/_bazel_root/7989b31489f31aee54f32688da2f0120/external/org_golang_x_tools/go/packages/packagestest/testdata/groups/two/modules/example.com/what@v1.1.0/main.go: error reading go file: /bazel-scratch/.cache/bazel/_bazel_root/7989b31489f31aee54f32688da2f0120/external/org_golang_x_tools/go/packages/packagestest/testdata/groups/two/modules/example.com/what@v1.1.0/main.go:1:16: expected ';', found '.'
gazelle: finding module path for import domain.name/importdecl: go get domain.name/importdecl: module domain.name/importdecl: reading https://proxy.golang.org/domain.name/importdecl/@v/list: 410 Gone
	server response: not found: domain.name/importdecl@latest: unrecognized import path "domain.name/importdecl": https fetch: Get "https://domain.name/importdecl?go-get=1": dial tcp: lookup domain.name on 8.8.8.8:53: no such host
gazelle: finding module path for import old.com/one: go get old.com/one: module old.com/one: reading https://proxy.golang.org/old.com/one/@v/list: 410 Gone
	server response: not found: old.com/one@latest: unrecognized import path "old.com/one": https fetch: Get "http://www.old.com/one?go-get=1": redirected from secure URL https://old.com/one?go-get=1 to insecure URL http://www.old.com/one?go-get=1
gazelle: finding module path for import titanic.biz/bar: go get titanic.biz/bar: module titanic.biz/bar: reading https://proxy.golang.org/titanic.biz/bar/@v/list: 410 Gone
	server response: not found: titanic.biz/bar@latest: unrecognized import path "titanic.biz/bar": reading https://titanic.biz/bar?go-get=1: 403 Forbidden
... skipping 77 lines ...
[12,356 / 12,373] 219 / 959 tests; GoLink cmd/genkubedocs/go_default_test_/go_default_test; 9s processwrapper-sandbox ... (8 actions, 7 running)
[13,228 / 13,244] 337 / 959 tests; Testing //pkg/master:go_default_test; 82s processwrapper-sandbox ... (8 actions, 7 running)
[13,988 / 14,001] 460 / 959 tests; Testing //cmd/kubeadm/app/phases/upgrade:go_default_test; 85s processwrapper-sandbox ... (8 actions, 7 running)
[14,847 / 14,855] 630 / 959 tests; Testing //cmd/kubeadm/app/phases/upgrade:go_default_test; 214s processwrapper-sandbox ... (8 actions running)
[16,206 / 18,964] 891 / 959 tests; Testing //pkg/kubelet/cm/devicemanager:go_default_test; 185s processwrapper-sandbox ... (8 actions, 7 running)
[18,950 / 19,020] 903 / 959 tests; Testing //pkg/kubelet/cm/devicemanager:go_default_test; 40s processwrapper-sandbox ... (8 actions, 7 running)
FAIL: //pkg/kubelet:go_default_test (see /bazel-scratch/.cache/bazel/_bazel_root/7989b31489f31aee54f32688da2f0120/execroot/io_k8s_kubernetes/bazel-out/k8-fastbuild/testlogs/pkg/kubelet/go_default_test/test_attempts/attempt_1.log)

FLAKY: //pkg/kubelet:go_default_test (Summary)
      /bazel-scratch/.cache/bazel/_bazel_root/7989b31489f31aee54f32688da2f0120/execroot/io_k8s_kubernetes/bazel-out/k8-fastbuild/testlogs/pkg/kubelet/go_default_test/test_attempts/attempt_1.log
INFO: From Testing //pkg/kubelet:go_default_test:
==================== Test output for //pkg/kubelet:go_default_test:
E0714 14:56:11.156684  241081 eviction_manager.go:260] eviction manager: failed to get summary stats: failed to get root cgroup stats: failed to get cgroup stats for "/": unexpected number of containers: 0
I0714 14:56:11.156942  241081 plugin_manager.go:114] Starting Kubelet Plugin Manager
E0714 14:56:11.178306  241081 eviction_manager.go:260] eviction manager: failed to get summary stats: failed to get root cgroup stats: failed to get cgroup stats for "/": unexpected number of containers: 0
I0714 14:56:11.178795  241081 plugin_manager.go:114] Starting Kubelet Plugin Manager
E0714 14:56:11.196910  241081 eviction_manager.go:260] eviction manager: failed to get summary stats: failed to get root cgroup stats: failed to get cgroup stats for "/": unexpected number of containers: 0
I0714 14:56:11.197190  241081 plugin_manager.go:114] Starting Kubelet Plugin Manager
E0714 14:56:12.213487  241081 kubelet_node_status.go:470] Error updating node status, will retry: error getting node "127.0.0.1": Get "http://127.0.0.1:42133/api/v1/nodes/127.0.0.1?resourceVersion=0&timeout=1s": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
E0714 14:56:13.214361  241081 kubelet_node_status.go:470] Error updating node status, will retry: error getting node "127.0.0.1": Get "http://127.0.0.1:42133/api/v1/nodes/127.0.0.1?timeout=1s": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
E0714 14:56:14.215370  241081 kubelet_node_status.go:470] Error updating node status, will retry: error getting node "127.0.0.1": Get "http://127.0.0.1:42133/api/v1/nodes/127.0.0.1?timeout=1s": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
E0714 14:56:15.216299  241081 kubelet_node_status.go:470] Error updating node status, will retry: error getting node "127.0.0.1": Get "http://127.0.0.1:42133/api/v1/nodes/127.0.0.1?timeout=1s": net/http: request canceled (Client.Timeout exceeded while awaiting headers)
E0714 14:56:16.217160  241081 kubelet_node_status.go:470] Error updating node status, will retry: error getting node "127.0.0.1": Get "http://127.0.0.1:42133/api/v1/nodes/127.0.0.1?timeout=1s": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
==================
WARNING: DATA RACE
Read at 0x00c0004956c3 by goroutine 105:
  testing.(*common).logDepth()
      GOROOT/src/testing/testing.go:744 +0xa6
  testing.(*common).log()
... skipping 40 lines ...
      vendor/github.com/cilium/ebpf/syscalls.go:188 +0x2b0
  runtime.doInit()
      GOROOT/src/runtime/proc.go:5652 +0x89
  k8s.io/kubernetes/vendor/github.com/cilium/ebpf/internal/btf.init()
      vendor/github.com/cilium/ebpf/internal/btf/btf.go:656 +0x18f
==================
E0714 14:56:16.220702  241081 eviction_manager.go:260] eviction manager: failed to get summary stats: failed to get root cgroup stats: failed to get cgroup stats for "/": unexpected number of containers: 0
I0714 14:56:16.220860  241081 plugin_manager.go:114] Starting Kubelet Plugin Manager
I0714 14:56:16.240007  241081 setters.go:555] Node became not ready: {Type:Ready Status:False LastHeartbeatTime:2021-07-14 14:55:46.239650097 +0000 UTC m=-24.485119776 LastTransitionTime:2021-07-14 14:55:46.239650097 +0000 UTC m=-24.485119776 Reason:KubeletNotReady Message:container runtime is down}
E0714 14:56:16.249411  241081 kubelet.go:2116] Container runtime sanity check failed: injected runtime status error
E0714 14:56:16.257255  241081 kubelet.go:2120] Container runtime status is nil
E0714 14:56:16.266295  241081 kubelet.go:2129] Container runtime network not ready: <nil>
E0714 14:56:16.267194  241081 kubelet.go:2140] Container runtime not ready: <nil>
E0714 14:56:16.275107  241081 kubelet.go:2140] Container runtime not ready: RuntimeReady=false reason: message:
E0714 14:56:16.289728  241081 kubelet.go:2129] Container runtime network not ready: NetworkReady=false reason: message:
I0714 14:56:16.290003  241081 setters.go:555] Node became not ready: {Type:Ready Status:False LastHeartbeatTime:2021-07-14 14:56:16.249394276 +0000 UTC m=+5.524624392 LastTransitionTime:2021-07-14 14:56:16.249394276 +0000 UTC m=+5.524624392 Reason:KubeletNotReady Message:runtime network not ready: NetworkReady=false reason: message:}
E0714 14:56:16.298448  241081 kubelet_node_status.go:470] Error updating node status, will retry: error getting node "127.0.0.1": nodes "127.0.0.1" not found
E0714 14:56:16.298530  241081 kubelet_node_status.go:470] Error updating node status, will retry: error getting node "127.0.0.1": nodes "127.0.0.1" not found
E0714 14:56:16.298590  241081 kubelet_node_status.go:470] Error updating node status, will retry: error getting node "127.0.0.1": nodes "127.0.0.1" not found
E0714 14:56:16.298636  241081 kubelet_node_status.go:470] Error updating node status, will retry: error getting node "127.0.0.1": nodes "127.0.0.1" not found
E0714 14:56:16.298681  241081 kubelet_node_status.go:470] Error updating node status, will retry: error getting node "127.0.0.1": nodes "127.0.0.1" not found
E0714 14:56:16.301101  241081 eviction_manager.go:260] eviction manager: failed to get summary stats: failed to get root cgroup stats: failed to get cgroup stats for "/": unexpected number of containers: 0
I0714 14:56:16.301382  241081 plugin_manager.go:114] Starting Kubelet Plugin Manager
I0714 14:56:16.327419  241081 kubelet_network.go:77] Setting Pod CIDR:  -> 10.0.0.0/24,2000::/10
I0714 14:56:16.447120  241081 kubelet_node_status.go:71] Attempting to register node 127.0.0.1
I0714 14:56:16.447283  241081 kubelet_node_status.go:109] Node 127.0.0.1 was previously registered
I0714 14:56:16.448584  241081 kubelet_node_status.go:74] Successfully registered node 127.0.0.1
I0714 14:56:16.453830  241081 kubelet_node_status.go:109] Node 127.0.0.1 was previously registered
I0714 14:56:16.454993  241081 kubelet_node_status.go:109] Node 127.0.0.1 was previously registered
I0714 14:56:16.455055  241081 kubelet_node_status.go:272] Controller attach-detach setting changed to false; updating existing Node
I0714 14:56:16.459426  241081 kubelet_node_status.go:109] Node 127.0.0.1 was previously registered
I0714 14:56:16.459487  241081 kubelet_node_status.go:275] Controller attach-detach setting changed to true; updating existing Node
E0714 14:56:16.463381  241081 kubelet_node_status.go:93] Unable to register node "127.0.0.1" with API server: 
E0714 14:56:16.464268  241081 kubelet_node_status.go:99] Unable to register node "127.0.0.1" with API server: error getting existing node: 
I0714 14:56:16.465149  241081 kubelet_node_status.go:109] Node 127.0.0.1 was previously registered
I0714 14:56:16.465189  241081 kubelet_node_status.go:272] Controller attach-detach setting changed to false; updating existing Node
E0714 14:56:16.465910  241081 kubelet_node_status.go:120] Unable to reconcile node "127.0.0.1" with API server: error updating node: failed to patch status "{\"metadata\":{\"annotations\":null}}" for node "127.0.0.1": 
E0714 14:56:16.470659  241081 eviction_manager.go:260] eviction manager: failed to get summary stats: failed to get root cgroup stats: failed to get cgroup stats for "/": unexpected number of containers: 0
I0714 14:56:16.470984  241081 plugin_manager.go:114] Starting Kubelet Plugin Manager
I0714 14:56:16.485074  241081 kubelet_node_status.go:168] Removing now unsupported huge page resource named: hugepages-2Mi
I0714 14:56:16.488390  241081 kubelet_node_status.go:182] Zero out resource test.com/resource1 capacity in existing node.
I0714 14:56:16.488511  241081 kubelet_node_status.go:182] Zero out resource test.com/resource2 capacity in existing node.
I0714 14:56:16.488883  241081 kubelet_node_status.go:182] Zero out resource test.com/resource1 capacity in existing node.
I0714 14:56:16.488953  241081 kubelet_node_status.go:182] Zero out resource test.com/resource2 capacity in existing node.
... skipping 14 lines ...
I0714 14:56:16.721922  241081 kubelet_resources.go:45] allocatable: map[cpu:{{6 0} {<nil>} 6 DecimalSI} memory:{{4294967296 0} {<nil>} 4Gi BinarySI}]
I0714 14:56:16.722182  241081 kubelet_resources.go:45] allocatable: map[cpu:{{6 0} {<nil>} 6 DecimalSI} memory:{{4294967296 0} {<nil>} 4Gi BinarySI}]
I0714 14:56:16.722442  241081 kubelet_resources.go:45] allocatable: map[cpu:{{6 0} {<nil>} 6 DecimalSI} memory:{{4294967296 0} {<nil>} 4Gi BinarySI}]
E0714 14:56:16.724187  241081 kubelet.go:1850] Update channel is closed. Exiting the sync loop.
I0714 14:56:16.724242  241081 kubelet.go:1770] Starting kubelet main sync loop.
E0714 14:56:16.724283  241081 kubelet.go:1850] Update channel is closed. Exiting the sync loop.
W0714 14:56:16.741591  241081 predicate.go:79] Failed to admit pod failedpod_foo(4) - Update plugin resources failed due to Allocation failed, which is unexpected.
E0714 14:56:16.745000  241081 runtime.go:207] invalid container ID: ""
E0714 14:56:16.745095  241081 runtime.go:207] invalid container ID: ""
I0714 14:56:16.750262  241081 kubelet.go:1585] Trying to delete pod foo_ns 11111111
W0714 14:56:16.750343  241081 kubelet.go:1589] Deleted mirror pod "foo_ns(11111111)" because it is outdated
W0714 14:56:16.792486  241081 kubelet_getters.go:300] Path "/tmp/kubelet_test.313009351/pods/pod1uid/volumes" does not exist
W0714 14:56:16.792580  241081 kubelet_getters.go:300] Path "/tmp/kubelet_test.313009351/pods/pod1uid/volumes" does not exist
W0714 14:56:16.792844  241081 kubelet_volumes.go:156] Cleaned up orphaned volume subpath from pod "pod1uid" at /tmp/kubelet_test.313009351/pods/pod1uid/volume-subpaths/volume/container/index
W0714 14:56:16.794691  241081 kubelet_getters.go:300] Path "/tmp/kubelet_test.680135546/pods/pod1uid/volumes" does not exist
W0714 14:56:16.794782  241081 kubelet_getters.go:300] Path "/tmp/kubelet_test.680135546/pods/pod1uid/volumes" does not exist
W0714 14:56:16.796642  241081 kubelet_getters.go:300] Path "/tmp/kubelet_test.168025745/pods/pod1uid/volumes" does not exist
W0714 14:56:16.796747  241081 kubelet_getters.go:300] Path "/tmp/kubelet_test.168025745/pods/pod1uid/volumes" does not exist
E0714 14:56:16.796980  241081 kubelet_volumes.go:179] orphaned pod "pod1uid" found, but failed to rmdir() subpath at path /tmp/kubelet_test.168025745/pods/pod1uid/volume-subpaths/volume/container/index: directory not empty : There were a total of 1 errors similar to this. Turn up verbosity to see them.
W0714 14:56:16.803422  241081 kubelet_getters.go:300] Path "/tmp/kubelet_test.121855822/pods/pod1uid/volumes" does not exist
W0714 14:56:16.803519  241081 kubelet_getters.go:300] Path "/tmp/kubelet_test.121855822/pods/pod1uid/volumes" does not exist
W0714 14:56:16.808069  241081 kubelet_volumes.go:139] Cleaned up orphaned volume from pod "pod1uid" at /tmp/kubelet_test.708482480/pods/pod1uid/volumes/plugin/name
E0714 14:56:16.810175  241081 kubelet_volumes.go:179] orphaned pod "pod1uid" found, but failed to rmdir() volume at path /tmp/kubelet_test.837628239/pods/pod1uid/volumes/plugin/name: directory not empty : There were a total of 1 errors similar to this. Turn up verbosity to see them.
W0714 14:56:16.823417  241081 kubelet_getters.go:300] Path "/tmp/kubelet_test.726215512/pods/poduid/volumes" does not exist
I0714 14:56:16.827934  241081 volume_manager.go:265] Starting Kubelet Volume Manager
I0714 14:56:16.828157  241081 desired_state_of_world_populator.go:139] Desired state populator starts to run
E0714 14:56:16.831626  241081 reflector.go:127] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.CSIDriver: unhandled watch: testing.WatchActionImpl{ActionImpl:testing.ActionImpl{Namespace:"", Verb:"watch", Resource:schema.GroupVersionResource{Group:"storage.k8s.io", Version:"v1", Resource:"csidrivers"}, Subresource:""}, WatchRestrictions:testing.WatchRestrictions{Labels:labels.internalSelector(nil), Fields:fields.andTerm{}, ResourceVersion:""}}
I0714 14:56:17.029263  241081 reconciler.go:244] operationExecutor.AttachVolume started for volume "vol1" (UniqueName: "fake/fake-device1") pod "foo" (UID: "12345678") 
I0714 14:56:17.029510  241081 operation_generator.go:361] AttachVolume.Attach succeeded for volume "vol1" (UniqueName: "fake/fake-device1") from node "127.0.0.1" 
I0714 14:56:17.029584  241081 reconciler.go:244] operationExecutor.AttachVolume started for volume "vol2" (UniqueName: "fake/fake-device2") pod "foo" (UID: "12345678") 
I0714 14:56:17.029786  241081 operation_generator.go:361] AttachVolume.Attach succeeded for volume "vol2" (UniqueName: "fake/fake-device2") from node "127.0.0.1" 
I0714 14:56:17.029858  241081 reconciler.go:157] Reconciler: start to sync state
I0714 14:56:17.130700  241081 operation_generator.go:558] MountVolume.WaitForAttach entering for volume "vol1" (UniqueName: "fake/fake-device1") pod "foo" (UID: "12345678") DevicePath "/dev/vdb-test"
... skipping 2 lines ...
I0714 14:56:17.130994  241081 operation_generator.go:567] MountVolume.WaitForAttach succeeded for volume "vol2" (UniqueName: "fake/fake-device2") pod "foo" (UID: "12345678") DevicePath "/dev/sdb"
I0714 14:56:17.131134  241081 operation_generator.go:596] MountVolume.MountDevice succeeded for volume "vol1" (UniqueName: "fake/fake-device1") pod "foo" (UID: "12345678") device mount path ""
I0714 14:56:17.131244  241081 operation_generator.go:596] MountVolume.MountDevice succeeded for volume "vol2" (UniqueName: "fake/fake-device2") pod "foo" (UID: "12345678") device mount path ""
I0714 14:56:17.428599  241081 volume_manager.go:276] Shutting down Kubelet Volume Manager
I0714 14:56:17.430516  241081 volume_manager.go:265] Starting Kubelet Volume Manager
I0714 14:56:17.430529  241081 desired_state_of_world_populator.go:139] Desired state populator starts to run
E0714 14:56:17.433380  241081 reflector.go:127] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.CSIDriver: unhandled watch: testing.WatchActionImpl{ActionImpl:testing.ActionImpl{Namespace:"", Verb:"watch", Resource:schema.GroupVersionResource{Group:"storage.k8s.io", Version:"v1", Resource:"csidrivers"}, Subresource:""}, WatchRestrictions:testing.WatchRestrictions{Labels:labels.internalSelector(nil), Fields:fields.andTerm{}, ResourceVersion:""}}
I0714 14:56:17.631561  241081 reconciler.go:244] operationExecutor.AttachVolume started for volume "vol1" (UniqueName: "fake/fake-device1") pod "pod1" (UID: "pod1uid") 
I0714 14:56:17.631795  241081 operation_generator.go:361] AttachVolume.Attach succeeded for volume "vol1" (UniqueName: "fake/fake-device1") from node "127.0.0.1" 
I0714 14:56:17.631856  241081 reconciler.go:244] operationExecutor.AttachVolume started for volume "vol2" (UniqueName: "fake/fake-device2") pod "pod2" (UID: "pod2uid") 
I0714 14:56:17.632070  241081 operation_generator.go:361] AttachVolume.Attach succeeded for volume "vol2" (UniqueName: "fake/fake-device2") from node "127.0.0.1" 
I0714 14:56:17.632361  241081 reconciler.go:244] operationExecutor.AttachVolume started for volume "vol3" (UniqueName: "fake/fake-device3") pod "pod3" (UID: "pod3uid") 
I0714 14:56:17.632415  241081 operation_generator.go:361] AttachVolume.Attach succeeded for volume "vol3" (UniqueName: "fake/fake-device3") from node "127.0.0.1" 
... skipping 7 lines ...
I0714 14:56:17.733816  241081 operation_generator.go:596] MountVolume.MountDevice succeeded for volume "vol3" (UniqueName: "fake/fake-device3") pod "pod3" (UID: "pod3uid") device mount path ""
I0714 14:56:17.733675  241081 operation_generator.go:567] MountVolume.WaitForAttach succeeded for volume "vol2" (UniqueName: "fake/fake-device2") pod "pod2" (UID: "pod2uid") DevicePath "/dev/sdb"
I0714 14:56:17.733949  241081 operation_generator.go:596] MountVolume.MountDevice succeeded for volume "vol2" (UniqueName: "fake/fake-device2") pod "pod2" (UID: "pod2uid") device mount path ""
I0714 14:56:18.031062  241081 volume_manager.go:276] Shutting down Kubelet Volume Manager
I0714 14:56:18.033379  241081 volume_manager.go:265] Starting Kubelet Volume Manager
I0714 14:56:18.033485  241081 desired_state_of_world_populator.go:139] Desired state populator starts to run
E0714 14:56:18.036131  241081 reflector.go:127] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.CSIDriver: unhandled watch: testing.WatchActionImpl{ActionImpl:testing.ActionImpl{Namespace:"", Verb:"watch", Resource:schema.GroupVersionResource{Group:"storage.k8s.io", Version:"v1", Resource:"csidrivers"}, Subresource:""}, WatchRestrictions:testing.WatchRestrictions{Labels:labels.internalSelector(nil), Fields:fields.andTerm{}, ResourceVersion:""}}
I0714 14:56:18.234284  241081 reconciler.go:244] operationExecutor.AttachVolume started for volume "vol1" (UniqueName: "fake/fake-device") pod "foo" (UID: "12345678") 
I0714 14:56:18.234396  241081 reconciler.go:157] Reconciler: start to sync state
I0714 14:56:18.234450  241081 operation_generator.go:361] AttachVolume.Attach succeeded for volume "vol1" (UniqueName: "fake/fake-device") from node "127.0.0.1" 
I0714 14:56:18.335171  241081 operation_generator.go:558] MountVolume.WaitForAttach entering for volume "vol1" (UniqueName: "fake/fake-device") pod "foo" (UID: "12345678") DevicePath "/dev/vdb-test"
I0714 14:56:18.335320  241081 operation_generator.go:567] MountVolume.WaitForAttach succeeded for volume "vol1" (UniqueName: "fake/fake-device") pod "foo" (UID: "12345678") DevicePath "/dev/sdb"
I0714 14:56:18.335383  241081 operation_generator.go:596] MountVolume.MountDevice succeeded for volume "vol1" (UniqueName: "fake/fake-device") pod "foo" (UID: "12345678") device mount path ""
I0714 14:56:18.634127  241081 volume_manager.go:276] Shutting down Kubelet Volume Manager
I0714 14:56:18.636093  241081 volume_manager.go:265] Starting Kubelet Volume Manager
I0714 14:56:18.636162  241081 desired_state_of_world_populator.go:139] Desired state populator starts to run
E0714 14:56:18.638687  241081 reflector.go:127] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.CSIDriver: unhandled watch: testing.WatchActionImpl{ActionImpl:testing.ActionImpl{Namespace:"", Verb:"watch", Resource:schema.GroupVersionResource{Group:"storage.k8s.io", Version:"v1", Resource:"csidrivers"}, Subresource:""}, WatchRestrictions:testing.WatchRestrictions{Labels:labels.internalSelector(nil), Fields:fields.andTerm{}, ResourceVersion:""}}
I0714 14:56:18.837084  241081 reconciler.go:244] operationExecutor.AttachVolume started for volume "vol1" (UniqueName: "fake/fake-device") pod "foo" (UID: "12345678") 
I0714 14:56:18.837184  241081 reconciler.go:157] Reconciler: start to sync state
I0714 14:56:18.837236  241081 operation_generator.go:361] AttachVolume.Attach succeeded for volume "vol1" (UniqueName: "fake/fake-device") from node "127.0.0.1" 
I0714 14:56:18.937862  241081 operation_generator.go:558] MountVolume.WaitForAttach entering for volume "vol1" (UniqueName: "fake/fake-device") pod "foo" (UID: "12345678") DevicePath "/dev/vdb-test"
I0714 14:56:18.937987  241081 operation_generator.go:567] MountVolume.WaitForAttach succeeded for volume "vol1" (UniqueName: "fake/fake-device") pod "foo" (UID: "12345678") DevicePath "/dev/sdb"
I0714 14:56:18.938047  241081 operation_generator.go:596] MountVolume.MountDevice succeeded for volume "vol1" (UniqueName: "fake/fake-device") pod "foo" (UID: "12345678") device mount path ""
... skipping 3 lines ...
I0714 14:56:19.339008  241081 operation_generator.go:882] UnmountDevice succeeded for volume "vol1" %!(EXTRA string=UnmountDevice succeeded for volume "vol1" (UniqueName: "fake/fake-device") on node "127.0.0.1" )
I0714 14:56:19.439407  241081 reconciler.go:333] operationExecutor.DetachVolume started for volume "vol1" (UniqueName: "fake/fake-device") on node "127.0.0.1" 
I0714 14:56:19.439580  241081 operation_generator.go:472] DetachVolume.Detach succeeded for volume "vol1" (UniqueName: "fake/fake-device") on node "127.0.0.1" 
I0714 14:56:19.487508  241081 volume_manager.go:276] Shutting down Kubelet Volume Manager
I0714 14:56:19.489925  241081 volume_manager.go:265] Starting Kubelet Volume Manager
I0714 14:56:19.490003  241081 desired_state_of_world_populator.go:139] Desired state populator starts to run
E0714 14:56:19.492213  241081 reflector.go:127] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.CSIDriver: failed to list *v1.CSIDriver: no reaction implemented for {{ list storage.k8s.io/v1, Resource=csidrivers } storage.k8s.io/v1, Kind=CSIDriver  { }}
I0714 14:56:19.690957  241081 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "vol1" (UniqueName: "fake/fake-device") pod "foo" (UID: "12345678") 
I0714 14:56:19.691057  241081 reconciler.go:157] Reconciler: start to sync state
I0714 14:56:19.691077  241081 operation_generator.go:1348] Controller attach succeeded for volume "vol1" (UniqueName: "fake/fake-device") pod "foo" (UID: "12345678") device path: "fake/path"
I0714 14:56:19.791641  241081 operation_generator.go:558] MountVolume.WaitForAttach entering for volume "vol1" (UniqueName: "fake/fake-device") pod "foo" (UID: "12345678") DevicePath "fake/path"
I0714 14:56:19.791780  241081 operation_generator.go:567] MountVolume.WaitForAttach succeeded for volume "vol1" (UniqueName: "fake/fake-device") pod "foo" (UID: "12345678") DevicePath "/dev/sdb"
I0714 14:56:19.791846  241081 operation_generator.go:596] MountVolume.MountDevice succeeded for volume "vol1" (UniqueName: "fake/fake-device") pod "foo" (UID: "12345678") device mount path ""
I0714 14:56:20.090598  241081 volume_manager.go:276] Shutting down Kubelet Volume Manager
I0714 14:56:20.092486  241081 volume_manager.go:265] Starting Kubelet Volume Manager
I0714 14:56:20.092666  241081 desired_state_of_world_populator.go:139] Desired state populator starts to run
E0714 14:56:20.094496  241081 reflector.go:127] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.CSIDriver: failed to list *v1.CSIDriver: no reaction implemented for {{ list storage.k8s.io/v1, Resource=csidrivers } storage.k8s.io/v1, Kind=CSIDriver  { }}
I0714 14:56:20.293267  241081 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "vol1" (UniqueName: "fake/fake-device") pod "foo" (UID: "12345678") 
I0714 14:56:20.293387  241081 reconciler.go:157] Reconciler: start to sync state
I0714 14:56:20.293415  241081 operation_generator.go:1348] Controller attach succeeded for volume "vol1" (UniqueName: "fake/fake-device") pod "foo" (UID: "12345678") device path: "fake/path"
I0714 14:56:20.394227  241081 operation_generator.go:558] MountVolume.WaitForAttach entering for volume "vol1" (UniqueName: "fake/fake-device") pod "foo" (UID: "12345678") DevicePath "fake/path"
I0714 14:56:20.394344  241081 operation_generator.go:567] MountVolume.WaitForAttach succeeded for volume "vol1" (UniqueName: "fake/fake-device") pod "foo" (UID: "12345678") DevicePath "/dev/sdb"
I0714 14:56:20.394412  241081 operation_generator.go:596] MountVolume.MountDevice succeeded for volume "vol1" (UniqueName: "fake/fake-device") pod "foo" (UID: "12345678") device mount path ""
I0714 14:56:20.794944  241081 reconciler.go:196] operationExecutor.UnmountVolume started for volume "vol1" (UniqueName: "fake/fake-device") pod "12345678" (UID: "12345678") 
I0714 14:56:20.795088  241081 operation_generator.go:788] UnmountVolume.TearDown succeeded for volume "fake/fake-device" (OuterVolumeSpecName: "vol1") pod "12345678" (UID: "12345678"). InnerVolumeSpecName "vol1". PluginName "fake", VolumeGidValue ""
I0714 14:56:20.895434  241081 reconciler.go:312] operationExecutor.UnmountDevice started for volume "vol1" (UniqueName: "fake/fake-device") on node "127.0.0.1" 
I0714 14:56:20.895549  241081 operation_generator.go:882] UnmountDevice succeeded for volume "vol1" %!(EXTRA string=UnmountDevice succeeded for volume "vol1" (UniqueName: "fake/fake-device") on node "127.0.0.1" )
I0714 14:56:20.995853  241081 reconciler.go:319] Volume detached for volume "vol1" (UniqueName: "fake/fake-device") on node "127.0.0.1" DevicePath "/dev/sdb"
I0714 14:56:21.094467  241081 volume_manager.go:276] Shutting down Kubelet Volume Manager
W0714 14:56:21.095780  241081 pod_container_deletor.go:79] Container "abc" not found in pod's containers
E0714 14:56:21.157397  241081 eviction_manager.go:260] eviction manager: failed to get summary stats: failed to get root cgroup stats: failed to get cgroup stats for "/": unexpected number of containers: 0
E0714 14:56:21.178995  241081 eviction_manager.go:260] eviction manager: failed to get summary stats: failed to get root cgroup stats: failed to get cgroup stats for "/": unexpected number of containers: 0
E0714 14:56:21.197546  241081 eviction_manager.go:260] eviction manager: failed to get summary stats: failed to get root cgroup stats: failed to get cgroup stats for "/": unexpected number of containers: 0
I0714 14:56:21.263271  241081 runonce.go:88] Waiting for 1 pods
I0714 14:56:21.263333  241081 runonce.go:123] pod "foo_new(12345678)" containers running
I0714 14:56:21.263463  241081 runonce.go:102] started pod "foo_new(12345678)"
I0714 14:56:21.263507  241081 runonce.go:108] 1 pods started
FAIL
================================================================================
[19,280 / 19,281] 958 / 959 tests; Testing //pkg/kubelet/cm/devicemanager:go_default_test; 236s processwrapper-sandbox
[19,280 / 19,281] 958 / 959 tests; Testing //pkg/kubelet/cm/devicemanager:go_default_test; 146s processwrapper-sandbox

TIMEOUT: //pkg/kubelet/cm/devicemanager:go_default_test (Summary)
      /bazel-scratch/.cache/bazel/_bazel_root/7989b31489f31aee54f32688da2f0120/execroot/io_k8s_kubernetes/bazel-out/k8-fastbuild/testlogs/pkg/kubelet/cm/devicemanager/go_default_test/test.log
      /bazel-scratch/.cache/bazel/_bazel_root/7989b31489f31aee54f32688da2f0120/execroot/io_k8s_kubernetes/bazel-out/k8-fastbuild/testlogs/pkg/kubelet/cm/devicemanager/go_default_test/test_attempts/attempt_1.log
      /bazel-scratch/.cache/bazel/_bazel_root/7989b31489f31aee54f32688da2f0120/execroot/io_k8s_kubernetes/bazel-out/k8-fastbuild/testlogs/pkg/kubelet/cm/devicemanager/go_default_test/test_attempts/attempt_2.log
INFO: From Testing //pkg/kubelet/cm/devicemanager:go_default_test:
==================== Test output for //pkg/kubelet/cm/devicemanager:go_default_test:
I0714 14:49:29.588002  170333 device_plugin_stub.go:131] Starting to serve on /tmp/mock.sock
I0714 14:49:29.596000  170333 device_plugin_stub.go:131] Starting to serve on /tmp/mock.sock
I0714 14:49:29.600557  170333 device_plugin_stub.go:227] ListAndWatch
E0714 14:49:29.602252  170333 endpoint.go:107] listAndWatch ended unexpectedly for device plugin mock with error rpc error: code = Canceled desc = grpc: the client connection is closing
I0714 14:49:29.604104  170333 device_plugin_stub.go:131] Starting to serve on /tmp/mock.sock
I0714 14:49:29.607436  170333 device_plugin_stub.go:227] ListAndWatch
I0714 14:49:29.608718  170333 device_plugin_stub.go:261] Allocate, &AllocateRequest{ContainerRequests:[]*ContainerAllocateRequest{&ContainerAllocateRequest{DevicesIDs:[ADeviceId],},},}
E0714 14:49:29.609805  170333 endpoint.go:107] listAndWatch ended unexpectedly for device plugin mock with error rpc error: code = Unavailable desc = transport is closing
I0714 14:49:29.611624  170333 device_plugin_stub.go:131] Starting to serve on /tmp/mock.sock
I0714 14:49:29.614089  170333 device_plugin_stub.go:227] ListAndWatch
I0714 14:49:29.615719  170333 device_plugin_stub.go:248] GetPreferredAllocation, &PreferredAllocationRequest{ContainerRequests:[]*ContainerPreferredAllocationRequest{&ContainerPreferredAllocationRequest{AvailableDeviceIDs:[],MustIncludeDeviceIDs:[],AllocationSize:-1,},},}
I0714 14:49:29.617005  170333 fake_topology_manager.go:29] [fake topologymanager] NewFakeManager
E0714 14:49:29.617119  170333 endpoint.go:107] listAndWatch ended unexpectedly for device plugin mock with error rpc error: code = Unavailable desc = transport is closing
I0714 14:49:29.630169  170333 fake_topology_manager.go:29] [fake topologymanager] NewFakeManager
W0714 14:49:29.630843  170333 manager.go:596] Failed to retrieve checkpoint for "kubelet_internal_checkpoint": checkpoint is not found
I0714 14:49:29.633685  170333 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin324036753/device-plugin.sock
I0714 14:49:29.634392  170333 fake_topology_manager.go:29] [fake topologymanager] NewFakeManager
W0714 14:49:29.634596  170333 manager.go:596] Failed to retrieve checkpoint for "kubelet_internal_checkpoint": checkpoint is not found
I0714 14:49:29.636442  170333 plugin_manager.go:114] Starting Kubelet Plugin Manager
I0714 14:49:29.638444  170333 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin376226492/device-plugin.sock
I0714 14:49:29.639689  170333 fake_topology_manager.go:29] [fake topologymanager] NewFakeManager
W0714 14:49:29.640057  170333 manager.go:596] Failed to retrieve checkpoint for "kubelet_internal_checkpoint": checkpoint is not found
I0714 14:49:29.642243  170333 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin658505963/device-plugin.sock
I0714 14:49:29.642320  170333 device_plugin_stub.go:180] Deprecation file not found. Invoke registration
I0714 14:49:29.646577  170333 manager.go:410] Got registration request from device plugin with resource name "fake-domain/resource"
I0714 14:49:29.649988  170333 device_plugin_stub.go:227] ListAndWatch
I0714 14:49:29.656306  170333 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin658505963/device-plugin.sock.new
I0714 14:49:29.656388  170333 device_plugin_stub.go:180] Deprecation file not found. Invoke registration
I0714 14:49:29.658497  170333 manager.go:410] Got registration request from device plugin with resource name "fake-domain/resource"
I0714 14:49:29.661794  170333 device_plugin_stub.go:227] ListAndWatch
I0714 14:49:29.667386  170333 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin658505963/device-plugin.sock.third
I0714 14:49:29.667473  170333 device_plugin_stub.go:180] Deprecation file not found. Invoke registration
I0714 14:49:29.672119  170333 manager.go:410] Got registration request from device plugin with resource name "fake-domain/resource"
I0714 14:49:29.675163  170333 device_plugin_stub.go:227] ListAndWatch
E0714 14:49:29.679866  170333 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Unavailable desc = transport is closing
E0714 14:49:29.680495  170333 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Unavailable desc = transport is closing
E0714 14:49:29.680984  170333 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Unavailable desc = transport is closing
I0714 14:49:29.681454  170333 fake_topology_manager.go:29] [fake topologymanager] NewFakeManager
I0714 14:49:29.684596  170333 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin658505963/device-plugin.sock
I0714 14:49:29.684674  170333 device_plugin_stub.go:180] Deprecation file not found. Invoke registration
I0714 14:49:29.686954  170333 manager.go:410] Got registration request from device plugin with resource name "fake-domain/resource"
I0714 14:49:29.691504  170333 device_plugin_stub.go:227] ListAndWatch
I0714 14:49:29.698326  170333 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin658505963/device-plugin.sock.new
I0714 14:49:29.698418  170333 device_plugin_stub.go:180] Deprecation file not found. Invoke registration
I0714 14:49:29.701380  170333 manager.go:410] Got registration request from device plugin with resource name "fake-domain/resource"
I0714 14:49:29.708076  170333 device_plugin_stub.go:227] ListAndWatch
I0714 14:49:29.715001  170333 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin658505963/device-plugin.sock.third
I0714 14:49:29.715090  170333 device_plugin_stub.go:180] Deprecation file not found. Invoke registration
I0714 14:49:29.717796  170333 manager.go:410] Got registration request from device plugin with resource name "fake-domain/resource"
I0714 14:49:29.721739  170333 device_plugin_stub.go:227] ListAndWatch
E0714 14:49:29.842367  170333 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Unavailable desc = transport is closing
E0714 14:49:29.842561  170333 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Unavailable desc = transport is closing
E0714 14:49:29.843143  170333 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Unavailable desc = transport is closing
I0714 14:49:29.843385  170333 fake_topology_manager.go:29] [fake topologymanager] NewFakeManager
I0714 14:49:29.845571  170333 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin658505963/device-plugin.sock
I0714 14:49:29.845646  170333 device_plugin_stub.go:180] Deprecation file not found. Invoke registration
I0714 14:49:29.847759  170333 manager.go:410] Got registration request from device plugin with resource name "fake-domain/resource"
I0714 14:49:29.850331  170333 device_plugin_stub.go:227] ListAndWatch
I0714 14:49:30.137987  170333 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin658505963/device-plugin.sock.new
I0714 14:49:30.138079  170333 device_plugin_stub.go:180] Deprecation file not found. Invoke registration
I0714 14:49:30.142291  170333 manager.go:410] Got registration request from device plugin with resource name "fake-domain/resource"
I0714 14:49:30.147017  170333 device_plugin_stub.go:227] ListAndWatch
I0714 14:49:30.350927  170333 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin658505963/device-plugin.sock.third
I0714 14:49:30.351005  170333 device_plugin_stub.go:180] Deprecation file not found. Invoke registration
I0714 14:49:30.353126  170333 manager.go:410] Got registration request from device plugin with resource name "fake-domain/resource"
I0714 14:49:30.356704  170333 device_plugin_stub.go:227] ListAndWatch
E0714 14:49:30.363855  170333 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Unavailable desc = transport is closing
E0714 14:49:30.364507  170333 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Unavailable desc = transport is closing
I0714 14:49:30.364661  170333 fake_topology_manager.go:29] [fake topologymanager] NewFakeManager
E0714 14:49:30.364870  170333 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Unavailable desc = transport is closing
I0714 14:49:30.366762  170333 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin658505963/device-plugin.sock
I0714 14:49:30.366842  170333 device_plugin_stub.go:180] Deprecation file not found. Invoke registration
I0714 14:49:30.368800  170333 manager.go:410] Got registration request from device plugin with resource name "fake-domain/resource"
I0714 14:49:30.374095  170333 device_plugin_stub.go:227] ListAndWatch
I0714 14:49:30.379529  170333 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin658505963/device-plugin.sock.new
I0714 14:49:30.379622  170333 device_plugin_stub.go:180] Deprecation file not found. Invoke registration
I0714 14:49:30.382216  170333 manager.go:410] Got registration request from device plugin with resource name "fake-domain/resource"
I0714 14:49:30.384734  170333 device_plugin_stub.go:227] ListAndWatch
I0714 14:49:30.390014  170333 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin658505963/device-plugin.sock.third
I0714 14:49:30.390096  170333 device_plugin_stub.go:180] Deprecation file not found. Invoke registration
I0714 14:49:30.392043  170333 manager.go:410] Got registration request from device plugin with resource name "fake-domain/resource"
I0714 14:49:30.394859  170333 device_plugin_stub.go:227] ListAndWatch
E0714 14:49:30.399291  170333 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Unavailable desc = transport is closing
E0714 14:49:30.400075  170333 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Canceled desc = grpc: the client connection is closing
E0714 14:49:30.400861  170333 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Unavailable desc = transport is closing
I0714 14:49:30.401365  170333 fake_topology_manager.go:29] [fake topologymanager] NewFakeManager
W0714 14:49:30.401835  170333 manager.go:596] Failed to retrieve checkpoint for "kubelet_internal_checkpoint": checkpoint is not found
I0714 14:49:30.403386  170333 plugin_manager.go:114] Starting Kubelet Plugin Manager
I0714 14:49:30.405757  170333 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin076308554/device-plugin.sock
I0714 14:49:30.406240  170333 device_plugin_stub.go:153] GetInfo
E0714 14:49:30.407155  170333 goroutinemap.go:150] Operation for "/tmp/device_plugin076308554/server.sock" failed. No retries permitted until 2021-07-14 14:49:30.907000807 +0000 UTC m=+1.434532795 (durationBeforeRetry 500ms). Error: "RegisterPlugin error -- failed to get plugin info using RPC GetInfo at socket /tmp/device_plugin076308554/server.sock, err: rpc error: code = Unimplemented desc = unknown service pluginregistration.Registration"
I0714 14:49:30.410320  170333 device_plugin_stub.go:227] ListAndWatch
I0714 14:49:30.417780  170333 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin076308554/device-plugin.sock.new
I0714 14:49:31.407531  170333 device_plugin_stub.go:153] GetInfo
E0714 14:49:31.408004  170333 goroutinemap.go:150] Operation for "/tmp/device_plugin076308554/server.sock" failed. No retries permitted until 2021-07-14 14:49:32.407902125 +0000 UTC m=+2.935434108 (durationBeforeRetry 1s). Error: "RegisterPlugin error -- failed to get plugin info using RPC GetInfo at socket /tmp/device_plugin076308554/server.sock, err: rpc error: code = Unimplemented desc = unknown service pluginregistration.Registration"
I0714 14:49:31.411236  170333 device_plugin_stub.go:227] ListAndWatch
I0714 14:49:31.417625  170333 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin076308554/device-plugin.sock.third
I0714 14:49:32.408029  170333 device_plugin_stub.go:153] GetInfo
I0714 14:49:32.412757  170333 device_plugin_stub.go:227] ListAndWatch
E0714 14:49:32.423447  170333 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Unavailable desc = transport is closing
E0714 14:49:32.423494  170333 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Unavailable desc = transport is closing
E0714 14:49:32.424146  170333 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Unavailable desc = transport is closing
I0714 14:49:32.424836  170333 fake_topology_manager.go:29] [fake topologymanager] NewFakeManager
E0714 14:49:32.438675  170333 manager.go:546] unexpected: unhealthyDevices and endpoints are out of sync
I0714 14:49:32.451072  170333 fake_topology_manager.go:29] [fake topologymanager] NewFakeManager
I0714 14:49:32.451232  170333 fake_topology_manager.go:34] [fake topologymanager] GetAffinity podUID: 58190b25-8d81-4705-bd59-debd0c4d2d80 container name:  
I0714 14:49:32.451299  170333 fake_topology_manager.go:34] [fake topologymanager] GetAffinity podUID: 58190b25-8d81-4705-bd59-debd0c4d2d80 container name:  
I0714 14:49:32.454703  170333 fake_topology_manager.go:34] [fake topologymanager] GetAffinity podUID: 081b1bbf-f4d3-404a-9a27-1e874ae3882e container name:  
... skipping 2 lines ...
I0714 14:49:32.461413  170333 fake_topology_manager.go:34] [fake topologymanager] GetAffinity podUID: 68fce4ae-3bd7-4d55-b875-a570471f708f container name:  306c529b-dced-474d-ab4c-5a1bc1286edd
I0714 14:49:32.464921  170333 fake_topology_manager.go:34] [fake topologymanager] GetAffinity podUID: 68fce4ae-3bd7-4d55-b875-a570471f708f container name:  f984b8d1-11ff-4d64-9c0a-3d33ab4d4ec7
I0714 14:49:32.468507  170333 fake_topology_manager.go:34] [fake topologymanager] GetAffinity podUID: 68fce4ae-3bd7-4d55-b875-a570471f708f container name:  61e68b7d-22e2-48c9-af55-6e7a027d6b94
I0714 14:49:32.473115  170333 fake_topology_manager.go:29] [fake topologymanager] NewFakeManager
I0714 14:49:32.473262  170333 fake_topology_manager.go:34] [fake topologymanager] GetAffinity podUID: 27f5ba9e-9542-4483-8f72-94e896660c5c container name:  
I0714 14:49:32.481005  170333 topology_hints.go:49] [devicemanager] Resource 'testdevice' does not have a topology preference
E0714 14:49:39.638877  170333 goroutinemap.go:150] Operation for "/tmp/device_plugin376226492/device-plugin.sock" failed. No retries permitted until 2021-07-14 14:49:40.138678811 +0000 UTC m=+10.666210809 (durationBeforeRetry 500ms). Error: "RegisterPlugin error -- dial failed at socket /tmp/device_plugin376226492/device-plugin.sock, err: failed to dial socket /tmp/device_plugin376226492/device-plugin.sock, err: context deadline exceeded"
E0714 14:49:39.639233  170333 goroutinemap.go:150] Operation for "/tmp/device_plugin376226492/server.sock" failed. No retries permitted until 2021-07-14 14:49:40.139056153 +0000 UTC m=+10.666588145 (durationBeforeRetry 500ms). Error: "RegisterPlugin error -- dial failed at socket /tmp/device_plugin376226492/server.sock, err: failed to dial socket /tmp/device_plugin376226492/server.sock, err: context deadline exceeded"
Terminated
-- Test timed out at 2021-07-14 14:54:29 UTC --
================================================================================
==================== Test output for //pkg/kubelet/cm/devicemanager:go_default_test:
I0714 14:54:44.888619  227107 device_plugin_stub.go:131] Starting to serve on /tmp/mock.sock
I0714 14:54:44.899821  227107 device_plugin_stub.go:131] Starting to serve on /tmp/mock.sock
I0714 14:54:44.904533  227107 device_plugin_stub.go:227] ListAndWatch
E0714 14:54:44.912318  227107 endpoint.go:107] listAndWatch ended unexpectedly for device plugin mock with error rpc error: code = Canceled desc = grpc: the client connection is closing
I0714 14:54:44.914046  227107 device_plugin_stub.go:131] Starting to serve on /tmp/mock.sock
I0714 14:54:44.921739  227107 device_plugin_stub.go:227] ListAndWatch
I0714 14:54:44.923000  227107 device_plugin_stub.go:261] Allocate, &AllocateRequest{ContainerRequests:[]*ContainerAllocateRequest{&ContainerAllocateRequest{DevicesIDs:[ADeviceId],},},}
E0714 14:54:44.924453  227107 endpoint.go:107] listAndWatch ended unexpectedly for device plugin mock with error rpc error: code = Unavailable desc = transport is closing
I0714 14:54:44.926842  227107 device_plugin_stub.go:131] Starting to serve on /tmp/mock.sock
I0714 14:54:44.932415  227107 device_plugin_stub.go:227] ListAndWatch
I0714 14:54:44.933645  227107 device_plugin_stub.go:248] GetPreferredAllocation, &PreferredAllocationRequest{ContainerRequests:[]*ContainerPreferredAllocationRequest{&ContainerPreferredAllocationRequest{AvailableDeviceIDs:[],MustIncludeDeviceIDs:[],AllocationSize:-1,},},}
E0714 14:54:44.935301  227107 endpoint.go:107] listAndWatch ended unexpectedly for device plugin mock with error rpc error: code = Unavailable desc = transport is closing
I0714 14:54:44.935360  227107 fake_topology_manager.go:29] [fake topologymanager] NewFakeManager
I0714 14:54:44.935943  227107 fake_topology_manager.go:29] [fake topologymanager] NewFakeManager
W0714 14:54:44.936203  227107 manager.go:596] Failed to retrieve checkpoint for "kubelet_internal_checkpoint": checkpoint is not found
I0714 14:54:44.938270  227107 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin182359387/device-plugin.sock
I0714 14:54:44.939084  227107 fake_topology_manager.go:29] [fake topologymanager] NewFakeManager
W0714 14:54:44.939362  227107 manager.go:596] Failed to retrieve checkpoint for "kubelet_internal_checkpoint": checkpoint is not found
I0714 14:54:44.944746  227107 plugin_manager.go:114] Starting Kubelet Plugin Manager
I0714 14:54:44.947564  227107 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin253872894/device-plugin.sock
I0714 14:54:44.960804  227107 fake_topology_manager.go:29] [fake topologymanager] NewFakeManager
W0714 14:54:44.961104  227107 manager.go:596] Failed to retrieve checkpoint for "kubelet_internal_checkpoint": checkpoint is not found
I0714 14:54:44.963273  227107 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin255503429/device-plugin.sock
I0714 14:54:44.963359  227107 device_plugin_stub.go:180] Deprecation file not found. Invoke registration
I0714 14:54:44.966878  227107 manager.go:410] Got registration request from device plugin with resource name "fake-domain/resource"
I0714 14:54:44.970379  227107 device_plugin_stub.go:227] ListAndWatch
I0714 14:54:44.991089  227107 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin255503429/device-plugin.sock.new
I0714 14:54:44.991483  227107 device_plugin_stub.go:180] Deprecation file not found. Invoke registration
I0714 14:54:44.994095  227107 manager.go:410] Got registration request from device plugin with resource name "fake-domain/resource"
I0714 14:54:44.998321  227107 device_plugin_stub.go:227] ListAndWatch
I0714 14:54:45.004475  227107 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin255503429/device-plugin.sock.third
I0714 14:54:45.004570  227107 device_plugin_stub.go:180] Deprecation file not found. Invoke registration
I0714 14:54:45.008714  227107 manager.go:410] Got registration request from device plugin with resource name "fake-domain/resource"
I0714 14:54:45.012382  227107 device_plugin_stub.go:227] ListAndWatch
I0714 14:54:45.019075  227107 fake_topology_manager.go:29] [fake topologymanager] NewFakeManager
E0714 14:54:45.021170  227107 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Unavailable desc = transport is closing
I0714 14:54:45.022682  227107 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin255503429/device-plugin.sock
I0714 14:54:45.022752  227107 device_plugin_stub.go:180] Deprecation file not found. Invoke registration
E0714 14:54:45.028076  227107 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Unavailable desc = transport is closing
E0714 14:54:45.028401  227107 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Unavailable desc = transport is closing
I0714 14:54:45.031468  227107 manager.go:410] Got registration request from device plugin with resource name "fake-domain/resource"
I0714 14:54:45.033903  227107 device_plugin_stub.go:227] ListAndWatch
I0714 14:54:45.046275  227107 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin255503429/device-plugin.sock.new
I0714 14:54:45.046361  227107 device_plugin_stub.go:180] Deprecation file not found. Invoke registration
I0714 14:54:45.048520  227107 manager.go:410] Got registration request from device plugin with resource name "fake-domain/resource"
I0714 14:54:45.052224  227107 device_plugin_stub.go:227] ListAndWatch
I0714 14:54:45.060613  227107 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin255503429/device-plugin.sock.third
I0714 14:54:45.060712  227107 device_plugin_stub.go:180] Deprecation file not found. Invoke registration
I0714 14:54:45.062812  227107 manager.go:410] Got registration request from device plugin with resource name "fake-domain/resource"
I0714 14:54:45.065660  227107 device_plugin_stub.go:227] ListAndWatch
E0714 14:54:45.072582  227107 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Unavailable desc = transport is closing
E0714 14:54:45.072939  227107 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Unavailable desc = transport is closing
I0714 14:54:45.073197  227107 fake_topology_manager.go:29] [fake topologymanager] NewFakeManager
E0714 14:54:45.073203  227107 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Unavailable desc = transport is closing
I0714 14:54:45.079252  227107 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin255503429/device-plugin.sock
I0714 14:54:45.079331  227107 device_plugin_stub.go:180] Deprecation file not found. Invoke registration
I0714 14:54:45.081555  227107 manager.go:410] Got registration request from device plugin with resource name "fake-domain/resource"
I0714 14:54:45.084178  227107 device_plugin_stub.go:227] ListAndWatch
I0714 14:54:45.095651  227107 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin255503429/device-plugin.sock.new
I0714 14:54:45.095981  227107 device_plugin_stub.go:180] Deprecation file not found. Invoke registration
I0714 14:54:45.098111  227107 manager.go:410] Got registration request from device plugin with resource name "fake-domain/resource"
I0714 14:54:45.101123  227107 device_plugin_stub.go:227] ListAndWatch
I0714 14:54:45.110524  227107 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin255503429/device-plugin.sock.third
I0714 14:54:45.110615  227107 device_plugin_stub.go:180] Deprecation file not found. Invoke registration
I0714 14:54:45.115971  227107 manager.go:410] Got registration request from device plugin with resource name "fake-domain/resource"
I0714 14:54:45.120079  227107 device_plugin_stub.go:227] ListAndWatch
E0714 14:54:45.129948  227107 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Unavailable desc = transport is closing
E0714 14:54:45.130960  227107 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Unavailable desc = transport is closing
I0714 14:54:45.131107  227107 fake_topology_manager.go:29] [fake topologymanager] NewFakeManager
I0714 14:54:45.134129  227107 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin255503429/device-plugin.sock
I0714 14:54:45.134198  227107 device_plugin_stub.go:180] Deprecation file not found. Invoke registration
E0714 14:54:45.134209  227107 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Unavailable desc = transport is closing
I0714 14:54:45.136393  227107 manager.go:410] Got registration request from device plugin with resource name "fake-domain/resource"
I0714 14:54:45.139275  227107 device_plugin_stub.go:227] ListAndWatch
I0714 14:54:45.148799  227107 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin255503429/device-plugin.sock.new
I0714 14:54:45.148872  227107 device_plugin_stub.go:180] Deprecation file not found. Invoke registration
I0714 14:54:45.151232  227107 manager.go:410] Got registration request from device plugin with resource name "fake-domain/resource"
I0714 14:54:45.156393  227107 device_plugin_stub.go:227] ListAndWatch
I0714 14:54:45.162941  227107 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin255503429/device-plugin.sock.third
I0714 14:54:45.163028  227107 device_plugin_stub.go:180] Deprecation file not found. Invoke registration
I0714 14:54:45.165284  227107 manager.go:410] Got registration request from device plugin with resource name "fake-domain/resource"
I0714 14:54:45.168858  227107 device_plugin_stub.go:227] ListAndWatch
E0714 14:54:45.173257  227107 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Unavailable desc = transport is closing
E0714 14:54:45.174290  227107 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Unavailable desc = transport is closing
E0714 14:54:45.174600  227107 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Unavailable desc = transport is closing
I0714 14:54:45.175523  227107 fake_topology_manager.go:29] [fake topologymanager] NewFakeManager
W0714 14:54:45.175799  227107 manager.go:596] Failed to retrieve checkpoint for "kubelet_internal_checkpoint": checkpoint is not found
I0714 14:54:45.177195  227107 plugin_manager.go:114] Starting Kubelet Plugin Manager
I0714 14:54:45.178970  227107 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin310877244/device-plugin.sock
I0714 14:54:45.180758  227107 device_plugin_stub.go:153] GetInfo
E0714 14:54:45.191534  227107 goroutinemap.go:150] Operation for "/tmp/device_plugin310877244/server.sock" failed. No retries permitted until 2021-07-14 14:54:45.691375815 +0000 UTC m=+0.932362649 (durationBeforeRetry 500ms). Error: "RegisterPlugin error -- failed to get plugin info using RPC GetInfo at socket /tmp/device_plugin310877244/server.sock, err: rpc error: code = Unimplemented desc = unknown service pluginregistration.Registration"
I0714 14:54:45.193857  227107 device_plugin_stub.go:227] ListAndWatch
I0714 14:54:45.209633  227107 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin310877244/device-plugin.sock.new
E0714 14:54:46.186367  227107 goroutinemap.go:150] Operation for "/tmp/device_plugin310877244/server.sock" failed. No retries permitted until 2021-07-14 14:54:47.186206432 +0000 UTC m=+2.427193264 (durationBeforeRetry 1s). Error: "RegisterPlugin error -- failed to get plugin info using RPC GetInfo at socket /tmp/device_plugin310877244/server.sock, err: rpc error: code = Unimplemented desc = unknown service pluginregistration.Registration"
I0714 14:54:46.188220  227107 device_plugin_stub.go:153] GetInfo
I0714 14:54:46.193653  227107 device_plugin_stub.go:227] ListAndWatch
I0714 14:54:46.207397  227107 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin310877244/device-plugin.sock.third
I0714 14:54:47.182458  227107 device_plugin_stub.go:153] GetInfo
I0714 14:54:47.188608  227107 device_plugin_stub.go:227] ListAndWatch
E0714 14:54:47.204514  227107 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Unavailable desc = transport is closing
E0714 14:54:47.205059  227107 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Unavailable desc = transport is closing
E0714 14:54:47.205184  227107 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Unavailable desc = transport is closing
I0714 14:54:47.205646  227107 fake_topology_manager.go:29] [fake topologymanager] NewFakeManager
E0714 14:54:47.221532  227107 manager.go:546] unexpected: unhealthyDevices and endpoints are out of sync
I0714 14:54:47.238633  227107 fake_topology_manager.go:29] [fake topologymanager] NewFakeManager
I0714 14:54:47.238881  227107 fake_topology_manager.go:34] [fake topologymanager] GetAffinity podUID: 82f18566-3abf-4cdd-90fa-12a578c6b27c container name:  
I0714 14:54:47.238992  227107 fake_topology_manager.go:34] [fake topologymanager] GetAffinity podUID: 82f18566-3abf-4cdd-90fa-12a578c6b27c container name:  
I0714 14:54:47.243997  227107 fake_topology_manager.go:34] [fake topologymanager] GetAffinity podUID: 25460754-20f2-408d-944f-442938de72c6 container name:  
... skipping 2 lines ...
I0714 14:54:47.252567  227107 fake_topology_manager.go:34] [fake topologymanager] GetAffinity podUID: a6a17611-5ac5-4aac-96e1-2b6b14b1fcc7 container name:  a3cd36d3-efda-4adc-9211-b3de1944a0aa
I0714 14:54:47.256245  227107 fake_topology_manager.go:34] [fake topologymanager] GetAffinity podUID: a6a17611-5ac5-4aac-96e1-2b6b14b1fcc7 container name:  de728da1-4428-488c-8773-06b2f518549c
I0714 14:54:47.260047  227107 fake_topology_manager.go:34] [fake topologymanager] GetAffinity podUID: a6a17611-5ac5-4aac-96e1-2b6b14b1fcc7 container name:  40f08755-ff3e-4e0c-a083-800958655bdf
I0714 14:54:47.271375  227107 fake_topology_manager.go:29] [fake topologymanager] NewFakeManager
I0714 14:54:47.271563  227107 fake_topology_manager.go:34] [fake topologymanager] GetAffinity podUID: 33173870-0aaa-453c-9c55-59d2d1b3fe16 container name:  
I0714 14:54:47.285394  227107 topology_hints.go:49] [devicemanager] Resource 'testdevice' does not have a topology preference
E0714 14:54:54.947350  227107 goroutinemap.go:150] Operation for "/tmp/device_plugin253872894/device-plugin.sock" failed. No retries permitted until 2021-07-14 14:54:55.447135686 +0000 UTC m=+10.688122526 (durationBeforeRetry 500ms). Error: "RegisterPlugin error -- dial failed at socket /tmp/device_plugin253872894/device-plugin.sock, err: failed to dial socket /tmp/device_plugin253872894/device-plugin.sock, err: context deadline exceeded"
E0714 14:54:54.947692  227107 goroutinemap.go:150] Operation for "/tmp/device_plugin253872894/server.sock" failed. No retries permitted until 2021-07-14 14:54:55.447549354 +0000 UTC m=+10.688536192 (durationBeforeRetry 500ms). Error: "RegisterPlugin error -- dial failed at socket /tmp/device_plugin253872894/server.sock, err: failed to dial socket /tmp/device_plugin253872894/server.sock, err: context deadline exceeded"
Terminated
-- Test timed out at 2021-07-14 14:59:44 UTC --
================================================================================
==================== Test output for //pkg/kubelet/cm/devicemanager:go_default_test:
I0714 14:59:59.787499  241323 device_plugin_stub.go:131] Starting to serve on /tmp/mock.sock
I0714 14:59:59.790840  241323 device_plugin_stub.go:131] Starting to serve on /tmp/mock.sock
I0714 14:59:59.794305  241323 device_plugin_stub.go:227] ListAndWatch
E0714 14:59:59.795569  241323 endpoint.go:107] listAndWatch ended unexpectedly for device plugin mock with error rpc error: code = Unavailable desc = transport is closing
I0714 14:59:59.797227  241323 device_plugin_stub.go:131] Starting to serve on /tmp/mock.sock
I0714 14:59:59.801028  241323 device_plugin_stub.go:227] ListAndWatch
I0714 14:59:59.802076  241323 device_plugin_stub.go:261] Allocate, &AllocateRequest{ContainerRequests:[]*ContainerAllocateRequest{&ContainerAllocateRequest{DevicesIDs:[ADeviceId],},},}
E0714 14:59:59.802903  241323 endpoint.go:107] listAndWatch ended unexpectedly for device plugin mock with error rpc error: code = Unavailable desc = transport is closing
I0714 14:59:59.804966  241323 device_plugin_stub.go:131] Starting to serve on /tmp/mock.sock
I0714 14:59:59.807112  241323 device_plugin_stub.go:227] ListAndWatch
I0714 14:59:59.808115  241323 device_plugin_stub.go:248] GetPreferredAllocation, &PreferredAllocationRequest{ContainerRequests:[]*ContainerPreferredAllocationRequest{&ContainerPreferredAllocationRequest{AvailableDeviceIDs:[],MustIncludeDeviceIDs:[],AllocationSize:-1,},},}
E0714 14:59:59.809118  241323 endpoint.go:107] listAndWatch ended unexpectedly for device plugin mock with error rpc error: code = Unavailable desc = transport is closing
I0714 14:59:59.809419  241323 fake_topology_manager.go:29] [fake topologymanager] NewFakeManager
I0714 14:59:59.809881  241323 fake_topology_manager.go:29] [fake topologymanager] NewFakeManager
W0714 14:59:59.810275  241323 manager.go:596] Failed to retrieve checkpoint for "kubelet_internal_checkpoint": checkpoint is not found
I0714 14:59:59.812126  241323 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin761044901/device-plugin.sock
I0714 14:59:59.812922  241323 fake_topology_manager.go:29] [fake topologymanager] NewFakeManager
W0714 14:59:59.813096  241323 manager.go:596] Failed to retrieve checkpoint for "kubelet_internal_checkpoint": checkpoint is not found
I0714 14:59:59.813915  241323 plugin_manager.go:114] Starting Kubelet Plugin Manager
I0714 14:59:59.816707  241323 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin983123136/device-plugin.sock
I0714 14:59:59.818982  241323 fake_topology_manager.go:29] [fake topologymanager] NewFakeManager
W0714 14:59:59.819219  241323 manager.go:596] Failed to retrieve checkpoint for "kubelet_internal_checkpoint": checkpoint is not found
I0714 14:59:59.820785  241323 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin806971679/device-plugin.sock
I0714 14:59:59.820859  241323 device_plugin_stub.go:180] Deprecation file not found. Invoke registration
I0714 14:59:59.822604  241323 manager.go:410] Got registration request from device plugin with resource name "fake-domain/resource"
I0714 14:59:59.825278  241323 device_plugin_stub.go:227] ListAndWatch
I0714 14:59:59.832540  241323 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin806971679/device-plugin.sock.new
I0714 14:59:59.832595  241323 device_plugin_stub.go:180] Deprecation file not found. Invoke registration
I0714 14:59:59.834412  241323 manager.go:410] Got registration request from device plugin with resource name "fake-domain/resource"
I0714 14:59:59.838329  241323 device_plugin_stub.go:227] ListAndWatch
I0714 14:59:59.843272  241323 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin806971679/device-plugin.sock.third
I0714 14:59:59.843333  241323 device_plugin_stub.go:180] Deprecation file not found. Invoke registration
I0714 14:59:59.845269  241323 manager.go:410] Got registration request from device plugin with resource name "fake-domain/resource"
I0714 14:59:59.847579  241323 device_plugin_stub.go:227] ListAndWatch
E0714 14:59:59.851372  241323 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Unavailable desc = transport is closing
E0714 14:59:59.851642  241323 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Unavailable desc = transport is closing
E0714 14:59:59.851964  241323 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Unavailable desc = transport is closing
I0714 14:59:59.852153  241323 fake_topology_manager.go:29] [fake topologymanager] NewFakeManager
I0714 14:59:59.854342  241323 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin806971679/device-plugin.sock
I0714 14:59:59.854446  241323 device_plugin_stub.go:180] Deprecation file not found. Invoke registration
I0714 14:59:59.856755  241323 manager.go:410] Got registration request from device plugin with resource name "fake-domain/resource"
I0714 14:59:59.860332  241323 device_plugin_stub.go:227] ListAndWatch
I0714 14:59:59.865455  241323 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin806971679/device-plugin.sock.new
I0714 14:59:59.865526  241323 device_plugin_stub.go:180] Deprecation file not found. Invoke registration
I0714 14:59:59.867068  241323 manager.go:410] Got registration request from device plugin with resource name "fake-domain/resource"
I0714 14:59:59.869569  241323 device_plugin_stub.go:227] ListAndWatch
I0714 14:59:59.874384  241323 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin806971679/device-plugin.sock.third
I0714 14:59:59.874453  241323 device_plugin_stub.go:180] Deprecation file not found. Invoke registration
I0714 14:59:59.876162  241323 manager.go:410] Got registration request from device plugin with resource name "fake-domain/resource"
I0714 14:59:59.878339  241323 device_plugin_stub.go:227] ListAndWatch
E0714 14:59:59.882387  241323 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Unavailable desc = transport is closing
E0714 14:59:59.882589  241323 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Unavailable desc = transport is closing
E0714 14:59:59.883187  241323 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Unavailable desc = transport is closing
I0714 14:59:59.883900  241323 fake_topology_manager.go:29] [fake topologymanager] NewFakeManager
I0714 14:59:59.886619  241323 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin806971679/device-plugin.sock
I0714 14:59:59.886669  241323 device_plugin_stub.go:180] Deprecation file not found. Invoke registration
I0714 14:59:59.888256  241323 manager.go:410] Got registration request from device plugin with resource name "fake-domain/resource"
I0714 14:59:59.890569  241323 device_plugin_stub.go:227] ListAndWatch
I0714 14:59:59.895278  241323 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin806971679/device-plugin.sock.new
I0714 14:59:59.895352  241323 device_plugin_stub.go:180] Deprecation file not found. Invoke registration
I0714 14:59:59.897149  241323 manager.go:410] Got registration request from device plugin with resource name "fake-domain/resource"
I0714 14:59:59.900658  241323 device_plugin_stub.go:227] ListAndWatch
I0714 14:59:59.905471  241323 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin806971679/device-plugin.sock.third
I0714 14:59:59.905538  241323 device_plugin_stub.go:180] Deprecation file not found. Invoke registration
I0714 14:59:59.906984  241323 manager.go:410] Got registration request from device plugin with resource name "fake-domain/resource"
I0714 14:59:59.909197  241323 device_plugin_stub.go:227] ListAndWatch
E0714 14:59:59.912887  241323 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Unavailable desc = transport is closing
E0714 14:59:59.913203  241323 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Unavailable desc = transport is closing
E0714 14:59:59.913628  241323 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Unavailable desc = transport is closing
I0714 14:59:59.913675  241323 fake_topology_manager.go:29] [fake topologymanager] NewFakeManager
I0714 14:59:59.915535  241323 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin806971679/device-plugin.sock
I0714 14:59:59.915609  241323 device_plugin_stub.go:180] Deprecation file not found. Invoke registration
I0714 14:59:59.917108  241323 manager.go:410] Got registration request from device plugin with resource name "fake-domain/resource"
I0714 14:59:59.919380  241323 device_plugin_stub.go:227] ListAndWatch
I0714 14:59:59.924406  241323 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin806971679/device-plugin.sock.new
I0714 14:59:59.924515  241323 device_plugin_stub.go:180] Deprecation file not found. Invoke registration
I0714 14:59:59.926831  241323 manager.go:410] Got registration request from device plugin with resource name "fake-domain/resource"
I0714 14:59:59.929021  241323 device_plugin_stub.go:227] ListAndWatch
I0714 14:59:59.933505  241323 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin806971679/device-plugin.sock.third
I0714 14:59:59.933560  241323 device_plugin_stub.go:180] Deprecation file not found. Invoke registration
I0714 14:59:59.935116  241323 manager.go:410] Got registration request from device plugin with resource name "fake-domain/resource"
I0714 14:59:59.937574  241323 device_plugin_stub.go:227] ListAndWatch
E0714 14:59:59.941112  241323 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Unavailable desc = transport is closing
E0714 14:59:59.941535  241323 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Unavailable desc = transport is closing
E0714 14:59:59.941615  241323 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Unavailable desc = transport is closing
I0714 14:59:59.942327  241323 fake_topology_manager.go:29] [fake topologymanager] NewFakeManager
W0714 14:59:59.942459  241323 manager.go:596] Failed to retrieve checkpoint for "kubelet_internal_checkpoint": checkpoint is not found
I0714 14:59:59.943228  241323 plugin_manager.go:114] Starting Kubelet Plugin Manager
I0714 14:59:59.944827  241323 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin564962222/device-plugin.sock
I0714 14:59:59.947507  241323 device_plugin_stub.go:153] GetInfo
E0714 14:59:59.948053  241323 goroutinemap.go:150] Operation for "/tmp/device_plugin564962222/server.sock" failed. No retries permitted until 2021-07-14 15:00:00.447902405 +0000 UTC m=+0.740331954 (durationBeforeRetry 500ms). Error: "RegisterPlugin error -- failed to get plugin info using RPC GetInfo at socket /tmp/device_plugin564962222/server.sock, err: rpc error: code = Unimplemented desc = unknown service pluginregistration.Registration"
I0714 14:59:59.950775  241323 device_plugin_stub.go:227] ListAndWatch
I0714 14:59:59.955483  241323 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin564962222/device-plugin.sock.new
I0714 15:00:00.947288  241323 device_plugin_stub.go:153] GetInfo
E0714 15:00:00.948187  241323 goroutinemap.go:150] Operation for "/tmp/device_plugin564962222/server.sock" failed. No retries permitted until 2021-07-14 15:00:01.948008509 +0000 UTC m=+2.240438078 (durationBeforeRetry 1s). Error: "RegisterPlugin error -- failed to get plugin info using RPC GetInfo at socket /tmp/device_plugin564962222/server.sock, err: rpc error: code = Unimplemented desc = unknown service pluginregistration.Registration"
I0714 15:00:00.952643  241323 device_plugin_stub.go:227] ListAndWatch
I0714 15:00:00.957616  241323 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin564962222/device-plugin.sock.third
I0714 15:00:01.948195  241323 device_plugin_stub.go:153] GetInfo
I0714 15:00:01.952409  241323 device_plugin_stub.go:227] ListAndWatch
E0714 15:00:01.956921  241323 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Unavailable desc = transport is closing
E0714 15:00:01.957156  241323 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Unavailable desc = transport is closing
E0714 15:00:01.957727  241323 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Unavailable desc = transport is closing
I0714 15:00:01.958390  241323 fake_topology_manager.go:29] [fake topologymanager] NewFakeManager
E0714 15:00:01.972897  241323 manager.go:546] unexpected: unhealthyDevices and endpoints are out of sync
I0714 15:00:01.984593  241323 fake_topology_manager.go:29] [fake topologymanager] NewFakeManager
I0714 15:00:01.984788  241323 fake_topology_manager.go:34] [fake topologymanager] GetAffinity podUID: 9be008eb-678c-44e9-856f-8ea2ce897aac container name:  
I0714 15:00:01.984858  241323 fake_topology_manager.go:34] [fake topologymanager] GetAffinity podUID: 9be008eb-678c-44e9-856f-8ea2ce897aac container name:  
I0714 15:00:01.988316  241323 fake_topology_manager.go:34] [fake topologymanager] GetAffinity podUID: 4952d042-6045-486a-b0ee-3252964f1a5c container name:  
... skipping 2 lines ...
I0714 15:00:01.995814  241323 fake_topology_manager.go:34] [fake topologymanager] GetAffinity podUID: 920edb14-d436-4b99-8614-490d1f1fa932 container name:  751f7c8c-4747-41d2-9b7b-23e3aae72269
I0714 15:00:01.999319  241323 fake_topology_manager.go:34] [fake topologymanager] GetAffinity podUID: 920edb14-d436-4b99-8614-490d1f1fa932 container name:  f4d3c1ce-e306-476d-b5b9-18c0fd615839
I0714 15:00:02.003497  241323 fake_topology_manager.go:34] [fake topologymanager] GetAffinity podUID: 920edb14-d436-4b99-8614-490d1f1fa932 container name:  ad686801-0745-4fcc-9057-56fae15b8fe8
I0714 15:00:02.010041  241323 fake_topology_manager.go:29] [fake topologymanager] NewFakeManager
I0714 15:00:02.010157  241323 fake_topology_manager.go:34] [fake topologymanager] GetAffinity podUID: ad747fc5-f962-4637-bb02-3cd79edba709 container name:  
I0714 15:00:02.017824  241323 topology_hints.go:49] [devicemanager] Resource 'testdevice' does not have a topology preference
E0714 15:00:09.815932  241323 goroutinemap.go:150] Operation for "/tmp/device_plugin983123136/device-plugin.sock" failed. No retries permitted until 2021-07-14 15:00:10.315760942 +0000 UTC m=+10.608190509 (durationBeforeRetry 500ms). Error: "RegisterPlugin error -- dial failed at socket /tmp/device_plugin983123136/device-plugin.sock, err: failed to dial socket /tmp/device_plugin983123136/device-plugin.sock, err: context deadline exceeded"
E0714 15:00:09.816266  241323 goroutinemap.go:150] Operation for "/tmp/device_plugin983123136/server.sock" failed. No retries permitted until 2021-07-14 15:00:10.316089873 +0000 UTC m=+10.608519433 (durationBeforeRetry 500ms). Error: "RegisterPlugin error -- dial failed at socket /tmp/device_plugin983123136/server.sock, err: failed to dial socket /tmp/device_plugin983123136/server.sock, err: context deadline exceeded"
Terminated
-- Test timed out at 2021-07-14 15:04:59 UTC --
================================================================================
INFO: Elapsed time: 1895.102s, Critical Path: 1020.77s
INFO: 17357 processes: 17357 processwrapper-sandbox.
INFO: Build completed, 1 test FAILED, 19281 total actions
//cluster:common_test                                                    PASSED in 0.1s
//cluster:kube-util_test                                                 PASSED in 0.0s
//cluster/gce/cos:go_default_test                                        PASSED in 4.5s
//cluster/gce/custom:go_default_test                                     PASSED in 6.8s
//cluster/gce/gci:go_default_test                                        PASSED in 2.9s
//cluster/gce/ubuntu:go_default_test                                     PASSED in 2.9s
... skipping 945 lines ...
//test/list:go_default_test                                              PASSED in 0.1s
//test/typecheck:go_default_test                                         PASSED in 14.6s
//test/utils/image:go_default_test                                       PASSED in 0.2s
//third_party/forked/golang/expansion:go_default_test                    PASSED in 0.2s
//third_party/forked/golang/reflect:go_default_test                      PASSED in 0.1s
//third_party/forked/gonum/graph/simple:go_default_test                  PASSED in 0.1s
//pkg/kubelet:go_default_test                                             FLAKY, failed in 1 out of 2 in 11.1s
  Stats over 2 runs: max = 11.1s, min = 10.9s, avg = 11.0s, dev = 0.1s
  /bazel-scratch/.cache/bazel/_bazel_root/7989b31489f31aee54f32688da2f0120/execroot/io_k8s_kubernetes/bazel-out/k8-fastbuild/testlogs/pkg/kubelet/go_default_test/test_attempts/attempt_1.log
//pkg/kubelet/cm/devicemanager:go_default_test                          TIMEOUT in 3 out of 3 in 315.0s
  Stats over 3 runs: max = 315.0s, min = 315.0s, avg = 315.0s, dev = 0.0s
  /bazel-scratch/.cache/bazel/_bazel_root/7989b31489f31aee54f32688da2f0120/execroot/io_k8s_kubernetes/bazel-out/k8-fastbuild/testlogs/pkg/kubelet/cm/devicemanager/go_default_test/test.log
  /bazel-scratch/.cache/bazel/_bazel_root/7989b31489f31aee54f32688da2f0120/execroot/io_k8s_kubernetes/bazel-out/k8-fastbuild/testlogs/pkg/kubelet/cm/devicemanager/go_default_test/test_attempts/attempt_1.log
  /bazel-scratch/.cache/bazel/_bazel_root/7989b31489f31aee54f32688da2f0120/execroot/io_k8s_kubernetes/bazel-out/k8-fastbuild/testlogs/pkg/kubelet/cm/devicemanager/go_default_test/test_attempts/attempt_2.log

Executed 959 out of 959 tests: 958 tests pass and 1 fails locally.
There were tests whose specified size is too big. Use the --test_verbose_timeout_warnings command line option to see which ones these are.
INFO: Build completed, 1 test FAILED, 19281 total actions
+ ../test-infra/hack/coalesce.py
+ exit 3