This job view page is being replaced by Spyglass soon. Check out the new job view.
PRdimakow: Add missing locks for device-manager in 1.19
ResultFAILURE
Tests 1 failed / 956 succeeded
Started2021-07-14 12:29
Elapsed32m17s
Revisione734cd996b5d13da90c2fb834a4085e438fe46af
Refs 103262

Test Failures


//pkg/kubelet/cm/devicemanager:go_default_test 0.00s

bazel test //pkg/kubelet/cm/devicemanager:go_default_test
exec ${PAGER:-/usr/bin/less} "$0" || exit 1
Executing tests from //pkg/kubelet/cm/devicemanager:go_default_test
-----------------------------------------------------------------------------
I0714 12:56:30.737999  241916 device_plugin_stub.go:131] Starting to serve on /tmp/mock.sock
I0714 12:56:30.742451  241916 device_plugin_stub.go:131] Starting to serve on /tmp/mock.sock
I0714 12:56:30.746296  241916 device_plugin_stub.go:227] ListAndWatch
E0714 12:56:30.747489  241916 endpoint.go:107] listAndWatch ended unexpectedly for device plugin mock with error rpc error: code = Unavailable desc = transport is closing
I0714 12:56:30.749493  241916 device_plugin_stub.go:131] Starting to serve on /tmp/mock.sock
I0714 12:56:30.752226  241916 device_plugin_stub.go:227] ListAndWatch
I0714 12:56:30.753337  241916 device_plugin_stub.go:261] Allocate, &AllocateRequest{ContainerRequests:[]*ContainerAllocateRequest{&ContainerAllocateRequest{DevicesIDs:[ADeviceId],},},}
E0714 12:56:30.754507  241916 endpoint.go:107] listAndWatch ended unexpectedly for device plugin mock with error rpc error: code = Canceled desc = grpc: the client connection is closing
I0714 12:56:30.757572  241916 device_plugin_stub.go:131] Starting to serve on /tmp/mock.sock
I0714 12:56:30.759877  241916 device_plugin_stub.go:227] ListAndWatch
I0714 12:56:30.761071  241916 device_plugin_stub.go:248] GetPreferredAllocation, &PreferredAllocationRequest{ContainerRequests:[]*ContainerPreferredAllocationRequest{&ContainerPreferredAllocationRequest{AvailableDeviceIDs:[],MustIncludeDeviceIDs:[],AllocationSize:-1,},},}
E0714 12:56:30.762326  241916 endpoint.go:107] listAndWatch ended unexpectedly for device plugin mock with error rpc error: code = Unavailable desc = transport is closing
I0714 12:56:30.762811  241916 fake_topology_manager.go:29] [fake topologymanager] NewFakeManager
I0714 12:56:30.763195  241916 fake_topology_manager.go:29] [fake topologymanager] NewFakeManager
W0714 12:56:30.763408  241916 manager.go:596] Failed to retrieve checkpoint for "kubelet_internal_checkpoint": checkpoint is not found
I0714 12:56:30.765459  241916 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin017986006/device-plugin.sock
I0714 12:56:30.766276  241916 fake_topology_manager.go:29] [fake topologymanager] NewFakeManager
W0714 12:56:30.766527  241916 manager.go:596] Failed to retrieve checkpoint for "kubelet_internal_checkpoint": checkpoint is not found
I0714 12:56:30.767723  241916 plugin_manager.go:114] Starting Kubelet Plugin Manager
I0714 12:56:30.768983  241916 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin248488253/device-plugin.sock
I0714 12:56:30.770747  241916 fake_topology_manager.go:29] [fake topologymanager] NewFakeManager
W0714 12:56:30.770963  241916 manager.go:596] Failed to retrieve checkpoint for "kubelet_internal_checkpoint": checkpoint is not found
I0714 12:56:30.773386  241916 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin191224184/device-plugin.sock
I0714 12:56:30.773464  241916 device_plugin_stub.go:180] Deprecation file not found. Invoke registration
I0714 12:56:30.776144  241916 manager.go:410] Got registration request from device plugin with resource name "fake-domain/resource"
I0714 12:56:30.779104  241916 device_plugin_stub.go:227] ListAndWatch
I0714 12:56:30.787353  241916 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin191224184/device-plugin.sock.new
I0714 12:56:30.787455  241916 device_plugin_stub.go:180] Deprecation file not found. Invoke registration
I0714 12:56:30.789406  241916 manager.go:410] Got registration request from device plugin with resource name "fake-domain/resource"
I0714 12:56:30.792199  241916 device_plugin_stub.go:227] ListAndWatch
I0714 12:56:30.800417  241916 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin191224184/device-plugin.sock.third
I0714 12:56:30.800519  241916 device_plugin_stub.go:180] Deprecation file not found. Invoke registration
I0714 12:56:30.802438  241916 manager.go:410] Got registration request from device plugin with resource name "fake-domain/resource"
I0714 12:56:30.804988  241916 device_plugin_stub.go:227] ListAndWatch
E0714 12:56:30.810115  241916 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Unavailable desc = transport is closing
E0714 12:56:30.810527  241916 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Unavailable desc = transport is closing
E0714 12:56:30.811053  241916 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Unavailable desc = transport is closing
I0714 12:56:30.811175  241916 fake_topology_manager.go:29] [fake topologymanager] NewFakeManager
I0714 12:56:30.813231  241916 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin191224184/device-plugin.sock
I0714 12:56:30.813303  241916 device_plugin_stub.go:180] Deprecation file not found. Invoke registration
I0714 12:56:30.815344  241916 manager.go:410] Got registration request from device plugin with resource name "fake-domain/resource"
I0714 12:56:30.818382  241916 device_plugin_stub.go:227] ListAndWatch
I0714 12:56:30.825057  241916 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin191224184/device-plugin.sock.new
I0714 12:56:30.825166  241916 device_plugin_stub.go:180] Deprecation file not found. Invoke registration
I0714 12:56:30.828249  241916 manager.go:410] Got registration request from device plugin with resource name "fake-domain/resource"
I0714 12:56:30.830832  241916 device_plugin_stub.go:227] ListAndWatch
I0714 12:56:30.836161  241916 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin191224184/device-plugin.sock.third
I0714 12:56:30.836250  241916 device_plugin_stub.go:180] Deprecation file not found. Invoke registration
I0714 12:56:30.838030  241916 manager.go:410] Got registration request from device plugin with resource name "fake-domain/resource"
I0714 12:56:30.840437  241916 device_plugin_stub.go:227] ListAndWatch
E0714 12:56:30.844922  241916 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Unavailable desc = transport is closing
E0714 12:56:30.845210  241916 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Unavailable desc = transport is closing
E0714 12:56:30.845556  241916 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Unavailable desc = transport is closing
I0714 12:56:30.845902  241916 fake_topology_manager.go:29] [fake topologymanager] NewFakeManager
I0714 12:56:30.847823  241916 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin191224184/device-plugin.sock
I0714 12:56:30.847919  241916 device_plugin_stub.go:180] Deprecation file not found. Invoke registration
I0714 12:56:30.850918  241916 manager.go:410] Got registration request from device plugin with resource name "fake-domain/resource"
I0714 12:56:30.853267  241916 device_plugin_stub.go:227] ListAndWatch
I0714 12:56:30.858125  241916 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin191224184/device-plugin.sock.new
I0714 12:56:30.858227  241916 device_plugin_stub.go:180] Deprecation file not found. Invoke registration
I0714 12:56:30.859860  241916 manager.go:410] Got registration request from device plugin with resource name "fake-domain/resource"
I0714 12:56:30.862166  241916 device_plugin_stub.go:227] ListAndWatch
I0714 12:56:30.867228  241916 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin191224184/device-plugin.sock.third
I0714 12:56:30.867312  241916 device_plugin_stub.go:180] Deprecation file not found. Invoke registration
I0714 12:56:30.870687  241916 manager.go:410] Got registration request from device plugin with resource name "fake-domain/resource"
I0714 12:56:30.873092  241916 device_plugin_stub.go:227] ListAndWatch
E0714 12:56:30.877991  241916 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Unavailable desc = transport is closing
E0714 12:56:30.878210  241916 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Unavailable desc = transport is closing
E0714 12:56:30.878639  241916 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Unavailable desc = transport is closing
I0714 12:56:30.878933  241916 fake_topology_manager.go:29] [fake topologymanager] NewFakeManager
I0714 12:56:30.880917  241916 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin191224184/device-plugin.sock
I0714 12:56:30.880974  241916 device_plugin_stub.go:180] Deprecation file not found. Invoke registration
I0714 12:56:30.882963  241916 manager.go:410] Got registration request from device plugin with resource name "fake-domain/resource"
I0714 12:56:30.885651  241916 device_plugin_stub.go:227] ListAndWatch
I0714 12:56:30.891161  241916 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin191224184/device-plugin.sock.new
I0714 12:56:30.891249  241916 device_plugin_stub.go:180] Deprecation file not found. Invoke registration
I0714 12:56:30.892967  241916 manager.go:410] Got registration request from device plugin with resource name "fake-domain/resource"
I0714 12:56:30.897176  241916 device_plugin_stub.go:227] ListAndWatch
I0714 12:56:30.902576  241916 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin191224184/device-plugin.sock.third
I0714 12:56:30.902671  241916 device_plugin_stub.go:180] Deprecation file not found. Invoke registration
I0714 12:56:30.904572  241916 manager.go:410] Got registration request from device plugin with resource name "fake-domain/resource"
I0714 12:56:30.907646  241916 device_plugin_stub.go:227] ListAndWatch
E0714 12:56:30.912640  241916 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Unavailable desc = transport is closing
E0714 12:56:30.913379  241916 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Unavailable desc = transport is closing
E0714 12:56:30.913375  241916 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Unavailable desc = transport is closing
I0714 12:56:30.914081  241916 fake_topology_manager.go:29] [fake topologymanager] NewFakeManager
W0714 12:56:30.914348  241916 manager.go:596] Failed to retrieve checkpoint for "kubelet_internal_checkpoint": checkpoint is not found
I0714 12:56:30.915180  241916 plugin_manager.go:114] Starting Kubelet Plugin Manager
I0714 12:56:30.916645  241916 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin142253731/device-plugin.sock
I0714 12:56:30.919844  241916 device_plugin_stub.go:153] GetInfo
E0714 12:56:30.920643  241916 goroutinemap.go:150] Operation for "/tmp/device_plugin142253731/server.sock" failed. No retries permitted until 2021-07-14 12:56:31.42050195 +0000 UTC m=+0.767437133 (durationBeforeRetry 500ms). Error: "RegisterPlugin error -- failed to get plugin info using RPC GetInfo at socket /tmp/device_plugin142253731/server.sock, err: rpc error: code = Unimplemented desc = unknown service pluginregistration.Registration"
I0714 12:56:30.923898  241916 device_plugin_stub.go:227] ListAndWatch
I0714 12:56:30.929166  241916 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin142253731/device-plugin.sock.new
I0714 12:56:31.919451  241916 device_plugin_stub.go:153] GetInfo
E0714 12:56:31.919959  241916 goroutinemap.go:150] Operation for "/tmp/device_plugin142253731/server.sock" failed. No retries permitted until 2021-07-14 12:56:32.919790318 +0000 UTC m=+2.266725508 (durationBeforeRetry 1s). Error: "RegisterPlugin error -- failed to get plugin info using RPC GetInfo at socket /tmp/device_plugin142253731/server.sock, err: rpc error: code = Unimplemented desc = unknown service pluginregistration.Registration"
I0714 12:56:31.923052  241916 device_plugin_stub.go:227] ListAndWatch
I0714 12:56:31.930311  241916 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin142253731/device-plugin.sock.third
I0714 12:56:32.920196  241916 device_plugin_stub.go:153] GetInfo
I0714 12:56:32.924817  241916 device_plugin_stub.go:227] ListAndWatch
E0714 12:56:32.929248  241916 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Unavailable desc = transport is closing
E0714 12:56:32.929460  241916 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Unavailable desc = transport is closing
E0714 12:56:32.929854  241916 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Unavailable desc = transport is closing
I0714 12:56:32.930607  241916 fake_topology_manager.go:29] [fake topologymanager] NewFakeManager
E0714 12:56:32.946224  241916 manager.go:546] unexpected: unhealthyDevices and endpoints are out of sync
I0714 12:56:32.959063  241916 fake_topology_manager.go:29] [fake topologymanager] NewFakeManager
I0714 12:56:32.959275  241916 fake_topology_manager.go:34] [fake topologymanager] GetAffinity podUID: 95713eb7-6bb7-4a34-8c80-ab2e5dfca84c container name:  
E0714 12:56:40.769495  241916 goroutinemap.go:150] Operation for "/tmp/device_plugin248488253/device-plugin.sock" failed. No retries permitted until 2021-07-14 12:56:41.269350437 +0000 UTC m=+10.616285623 (durationBeforeRetry 500ms). Error: "RegisterPlugin error -- dial failed at socket /tmp/device_plugin248488253/device-plugin.sock, err: failed to dial socket /tmp/device_plugin248488253/device-plugin.sock, err: context deadline exceeded"
E0714 12:56:40.770216  241916 goroutinemap.go:150] Operation for "/tmp/device_plugin248488253/server.sock" failed. No retries permitted until 2021-07-14 12:56:41.27004987 +0000 UTC m=+10.616985050 (durationBeforeRetry 500ms). Error: "RegisterPlugin error -- dial failed at socket /tmp/device_plugin248488253/server.sock, err: failed to dial socket /tmp/device_plugin248488253/server.sock, err: context deadline exceeded"
Terminated
-- Test timed out at 2021-07-14 13:01:30 UTC --

				from junit_bazel.xml

Filter through log files | View test history on testgrid


Show 956 Passed Tests

Error lines from build-log.txt

+ bazel test --config=unit //... -- -//build/... -//vendor/...
$TEST_TMPDIR defined: output root default is '/bazel-scratch/.cache/bazel' and max_idle_secs default is '15'.
$TEST_TMPDIR defined: output root default is '/bazel-scratch/.cache/bazel' and max_idle_secs default is '15'.
Extracting Bazel installation...
Starting local Bazel server and connecting to it...
WARNING: Download from https://storage.googleapis.com/k8s-bazel-cache/https://github.com/bazelbuild/rules_docker/releases/download/v0.14.4/rules_docker-v0.14.4.tar.gz failed: class com.google.devtools.build.lib.bazel.repository.downloader.UnrecoverableHttpException GET returned 404 Not Found
Loading:  (1 packages loaded)
Loading: 1 packages loaded
Loading: 405 packages loaded
    currently loading: vendor/github.com/aws/aws-sdk-go/aws/request
Loading: 1450 packages loaded
    currently loading: vendor/k8s.io/apiserver/pkg/admission/plugin/webhook/errors
... skipping 6 lines ...
Analyzing: 959 targets (4556 packages loaded, 29053 targets configured)
Analyzing: 959 targets (4567 packages loaded, 32306 targets configured)
Analyzing: 959 targets (4572 packages loaded, 36631 targets configured)
Analyzing: 959 targets (4572 packages loaded, 36631 targets configured)
Analyzing: 959 targets (4573 packages loaded, 36754 targets configured)
Analyzing: 959 targets (4573 packages loaded, 36754 targets configured)
DEBUG: /bazel-scratch/.cache/bazel/_bazel_root/7989b31489f31aee54f32688da2f0120/external/bazel_gazelle/internal/go_repository.bzl:189:13: org_golang_x_tools: gazelle: /bazel-scratch/.cache/bazel/_bazel_root/7989b31489f31aee54f32688da2f0120/external/org_golang_x_tools/cmd/fiximports/testdata/src/old.com/bad/bad.go: error reading go file: /bazel-scratch/.cache/bazel/_bazel_root/7989b31489f31aee54f32688da2f0120/external/org_golang_x_tools/cmd/fiximports/testdata/src/old.com/bad/bad.go:2:43: expected 'package', found 'EOF'
gazelle: found packages aliases (aliases.go) and conversions (conversions.go) in /bazel-scratch/.cache/bazel/_bazel_root/7989b31489f31aee54f32688da2f0120/external/org_golang_x_tools/go/internal/gccgoimporter/testdata
gazelle: found packages exports (exports.go) and p (issue15920.go) in /bazel-scratch/.cache/bazel/_bazel_root/7989b31489f31aee54f32688da2f0120/external/org_golang_x_tools/go/internal/gcimporter/testdata
gazelle: /bazel-scratch/.cache/bazel/_bazel_root/7989b31489f31aee54f32688da2f0120/external/org_golang_x_tools/go/loader/testdata/badpkgdecl.go: error reading go file: /bazel-scratch/.cache/bazel/_bazel_root/7989b31489f31aee54f32688da2f0120/external/org_golang_x_tools/go/loader/testdata/badpkgdecl.go:1:34: expected 'package', found 'EOF'
gazelle: /bazel-scratch/.cache/bazel/_bazel_root/7989b31489f31aee54f32688da2f0120/external/org_golang_x_tools/go/packages/packagestest/testdata/groups/two/modules/example.com/extra/geez/help.go: error reading go file: /bazel-scratch/.cache/bazel/_bazel_root/7989b31489f31aee54f32688da2f0120/external/org_golang_x_tools/go/packages/packagestest/testdata/groups/two/modules/example.com/extra/geez/help.go:1:16: expected ';', found '.'
gazelle: /bazel-scratch/.cache/bazel/_bazel_root/7989b31489f31aee54f32688da2f0120/external/org_golang_x_tools/go/packages/packagestest/testdata/groups/two/modules/example.com/extra/v2/geez/help.go: error reading go file: /bazel-scratch/.cache/bazel/_bazel_root/7989b31489f31aee54f32688da2f0120/external/org_golang_x_tools/go/packages/packagestest/testdata/groups/two/modules/example.com/extra/v2/geez/help.go:1:16: expected ';', found '.'
gazelle: /bazel-scratch/.cache/bazel/_bazel_root/7989b31489f31aee54f32688da2f0120/external/org_golang_x_tools/go/packages/packagestest/testdata/groups/two/modules/example.com/extra/v2/me.go: error reading go file: /bazel-scratch/.cache/bazel/_bazel_root/7989b31489f31aee54f32688da2f0120/external/org_golang_x_tools/go/packages/packagestest/testdata/groups/two/modules/example.com/extra/v2/me.go:1:16: expected ';', found '.'
gazelle: /bazel-scratch/.cache/bazel/_bazel_root/7989b31489f31aee54f32688da2f0120/external/org_golang_x_tools/go/packages/packagestest/testdata/groups/two/modules/example.com/extra/yo.go: error reading go file: /bazel-scratch/.cache/bazel/_bazel_root/7989b31489f31aee54f32688da2f0120/external/org_golang_x_tools/go/packages/packagestest/testdata/groups/two/modules/example.com/extra/yo.go:1:16: expected ';', found '.'
gazelle: /bazel-scratch/.cache/bazel/_bazel_root/7989b31489f31aee54f32688da2f0120/external/org_golang_x_tools/go/packages/packagestest/testdata/groups/two/modules/example.com/tempmod/main.go: error reading go file: /bazel-scratch/.cache/bazel/_bazel_root/7989b31489f31aee54f32688da2f0120/external/org_golang_x_tools/go/packages/packagestest/testdata/groups/two/modules/example.com/tempmod/main.go:1:16: expected ';', found '.'
gazelle: /bazel-scratch/.cache/bazel/_bazel_root/7989b31489f31aee54f32688da2f0120/external/org_golang_x_tools/go/packages/packagestest/testdata/groups/two/modules/example.com/what@v1.0.0/main.go: error reading go file: /bazel-scratch/.cache/bazel/_bazel_root/7989b31489f31aee54f32688da2f0120/external/org_golang_x_tools/go/packages/packagestest/testdata/groups/two/modules/example.com/what@v1.0.0/main.go:1:16: expected ';', found '.'
gazelle: /bazel-scratch/.cache/bazel/_bazel_root/7989b31489f31aee54f32688da2f0120/external/org_golang_x_tools/go/packages/packagestest/testdata/groups/two/modules/example.com/what@v1.1.0/main.go: error reading go file: /bazel-scratch/.cache/bazel/_bazel_root/7989b31489f31aee54f32688da2f0120/external/org_golang_x_tools/go/packages/packagestest/testdata/groups/two/modules/example.com/what@v1.1.0/main.go:1:16: expected ';', found '.'
gazelle: finding module path for import domain.name/importdecl: go get domain.name/importdecl: module domain.name/importdecl: reading https://proxy.golang.org/domain.name/importdecl/@v/list: 410 Gone
	server response: not found: domain.name/importdecl@latest: unrecognized import path "domain.name/importdecl": https fetch: Get "https://domain.name/importdecl?go-get=1": dial tcp: lookup domain.name on 8.8.8.8:53: no such host
gazelle: finding module path for import old.com/one: go get old.com/one: module old.com/one: reading https://proxy.golang.org/old.com/one/@v/list: 410 Gone
	server response: not found: old.com/one@latest: unrecognized import path "old.com/one": https fetch: Get "http://www.old.com/one?go-get=1": redirected from secure URL https://old.com/one?go-get=1 to insecure URL http://www.old.com/one?go-get=1
gazelle: finding module path for import titanic.biz/bar: go get titanic.biz/bar: module titanic.biz/bar: reading https://proxy.golang.org/titanic.biz/bar/@v/list: 410 Gone
	server response: not found: titanic.biz/bar@latest: unrecognized import path "titanic.biz/bar": reading https://titanic.biz/bar?go-get=1: 403 Forbidden
... skipping 89 lines ...
      /bazel-scratch/.cache/bazel/_bazel_root/7989b31489f31aee54f32688da2f0120/execroot/io_k8s_kubernetes/bazel-out/k8-fastbuild/testlogs/pkg/kubelet/cm/devicemanager/go_default_test/test_attempts/attempt_2.log
INFO: From Testing //pkg/kubelet/cm/devicemanager:go_default_test:
==================== Test output for //pkg/kubelet/cm/devicemanager:go_default_test:
I0714 12:46:00.677788  181515 device_plugin_stub.go:131] Starting to serve on /tmp/mock.sock
I0714 12:46:00.683954  181515 device_plugin_stub.go:131] Starting to serve on /tmp/mock.sock
I0714 12:46:00.689133  181515 device_plugin_stub.go:227] ListAndWatch
E0714 12:46:00.690581  181515 endpoint.go:107] listAndWatch ended unexpectedly for device plugin mock with error rpc error: code = Unavailable desc = transport is closing
I0714 12:46:00.693468  181515 device_plugin_stub.go:131] Starting to serve on /tmp/mock.sock
I0714 12:46:00.711754  181515 device_plugin_stub.go:227] ListAndWatch
I0714 12:46:00.713019  181515 device_plugin_stub.go:261] Allocate, &AllocateRequest{ContainerRequests:[]*ContainerAllocateRequest{&ContainerAllocateRequest{DevicesIDs:[ADeviceId],},},}
E0714 12:46:00.714760  181515 endpoint.go:107] listAndWatch ended unexpectedly for device plugin mock with error rpc error: code = Unavailable desc = transport is closing
I0714 12:46:00.716936  181515 device_plugin_stub.go:131] Starting to serve on /tmp/mock.sock
I0714 12:46:00.720126  181515 device_plugin_stub.go:227] ListAndWatch
I0714 12:46:00.721032  181515 device_plugin_stub.go:248] GetPreferredAllocation, &PreferredAllocationRequest{ContainerRequests:[]*ContainerPreferredAllocationRequest{&ContainerPreferredAllocationRequest{AvailableDeviceIDs:[],MustIncludeDeviceIDs:[],AllocationSize:-1,},},}
I0714 12:46:00.722380  181515 fake_topology_manager.go:29] [fake topologymanager] NewFakeManager
E0714 12:46:00.722680  181515 endpoint.go:107] listAndWatch ended unexpectedly for device plugin mock with error rpc error: code = Canceled desc = grpc: the client connection is closing
I0714 12:46:00.722961  181515 fake_topology_manager.go:29] [fake topologymanager] NewFakeManager
W0714 12:46:00.723238  181515 manager.go:596] Failed to retrieve checkpoint for "kubelet_internal_checkpoint": checkpoint is not found
I0714 12:46:00.730562  181515 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin956828074/device-plugin.sock
I0714 12:46:00.731847  181515 fake_topology_manager.go:29] [fake topologymanager] NewFakeManager
W0714 12:46:00.732355  181515 manager.go:596] Failed to retrieve checkpoint for "kubelet_internal_checkpoint": checkpoint is not found
I0714 12:46:00.734439  181515 plugin_manager.go:114] Starting Kubelet Plugin Manager
I0714 12:46:00.737000  181515 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin404643585/device-plugin.sock
I0714 12:46:00.738792  181515 fake_topology_manager.go:29] [fake topologymanager] NewFakeManager
W0714 12:46:00.739206  181515 manager.go:596] Failed to retrieve checkpoint for "kubelet_internal_checkpoint": checkpoint is not found
I0714 12:46:00.754775  181515 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin979901548/device-plugin.sock
I0714 12:46:00.754885  181515 device_plugin_stub.go:180] Deprecation file not found. Invoke registration
I0714 12:46:00.759944  181515 manager.go:410] Got registration request from device plugin with resource name "fake-domain/resource"
I0714 12:46:00.764802  181515 device_plugin_stub.go:227] ListAndWatch
I0714 12:46:00.783868  181515 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin979901548/device-plugin.sock.new
I0714 12:46:00.783970  181515 device_plugin_stub.go:180] Deprecation file not found. Invoke registration
I0714 12:46:00.787006  181515 manager.go:410] Got registration request from device plugin with resource name "fake-domain/resource"
I0714 12:46:00.791218  181515 device_plugin_stub.go:227] ListAndWatch
I0714 12:46:00.798871  181515 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin979901548/device-plugin.sock.third
I0714 12:46:00.798961  181515 device_plugin_stub.go:180] Deprecation file not found. Invoke registration
I0714 12:46:00.802202  181515 manager.go:410] Got registration request from device plugin with resource name "fake-domain/resource"
I0714 12:46:00.805606  181515 device_plugin_stub.go:227] ListAndWatch
E0714 12:46:00.810038  181515 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Unavailable desc = transport is closing
E0714 12:46:00.810589  181515 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Unavailable desc = transport is closing
E0714 12:46:00.811378  181515 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Unavailable desc = transport is closing
I0714 12:46:00.811416  181515 fake_topology_manager.go:29] [fake topologymanager] NewFakeManager
I0714 12:46:00.814022  181515 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin979901548/device-plugin.sock
I0714 12:46:00.814098  181515 device_plugin_stub.go:180] Deprecation file not found. Invoke registration
I0714 12:46:00.816382  181515 manager.go:410] Got registration request from device plugin with resource name "fake-domain/resource"
I0714 12:46:00.820342  181515 device_plugin_stub.go:227] ListAndWatch
I0714 12:46:00.826360  181515 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin979901548/device-plugin.sock.new
I0714 12:46:00.826452  181515 device_plugin_stub.go:180] Deprecation file not found. Invoke registration
I0714 12:46:00.828546  181515 manager.go:410] Got registration request from device plugin with resource name "fake-domain/resource"
I0714 12:46:00.832045  181515 device_plugin_stub.go:227] ListAndWatch
I0714 12:46:00.839969  181515 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin979901548/device-plugin.sock.third
I0714 12:46:00.840090  181515 device_plugin_stub.go:180] Deprecation file not found. Invoke registration
I0714 12:46:00.843197  181515 manager.go:410] Got registration request from device plugin with resource name "fake-domain/resource"
I0714 12:46:00.851887  181515 device_plugin_stub.go:227] ListAndWatch
E0714 12:46:00.857951  181515 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Canceled desc = grpc: the client connection is closing
I0714 12:46:00.858060  181515 fake_topology_manager.go:29] [fake topologymanager] NewFakeManager
E0714 12:46:00.862548  181515 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Unavailable desc = transport is closing
E0714 12:46:00.863708  181515 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Unavailable desc = transport is closing
I0714 12:46:00.863727  181515 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin979901548/device-plugin.sock
I0714 12:46:00.864038  181515 device_plugin_stub.go:180] Deprecation file not found. Invoke registration
I0714 12:46:00.866588  181515 manager.go:410] Got registration request from device plugin with resource name "fake-domain/resource"
I0714 12:46:00.870335  181515 device_plugin_stub.go:227] ListAndWatch
I0714 12:46:00.875800  181515 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin979901548/device-plugin.sock.new
I0714 12:46:00.875892  181515 device_plugin_stub.go:180] Deprecation file not found. Invoke registration
I0714 12:46:00.878014  181515 manager.go:410] Got registration request from device plugin with resource name "fake-domain/resource"
I0714 12:46:00.882281  181515 device_plugin_stub.go:227] ListAndWatch
I0714 12:46:00.887830  181515 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin979901548/device-plugin.sock.third
I0714 12:46:00.887917  181515 device_plugin_stub.go:180] Deprecation file not found. Invoke registration
I0714 12:46:00.890449  181515 manager.go:410] Got registration request from device plugin with resource name "fake-domain/resource"
I0714 12:46:00.895931  181515 device_plugin_stub.go:227] ListAndWatch
E0714 12:46:00.900139  181515 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Unavailable desc = transport is closing
I0714 12:46:00.900739  181515 fake_topology_manager.go:29] [fake topologymanager] NewFakeManager
E0714 12:46:00.900940  181515 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Unavailable desc = transport is closing
E0714 12:46:00.901233  181515 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Unavailable desc = transport is closing
I0714 12:46:00.903371  181515 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin979901548/device-plugin.sock
I0714 12:46:00.903459  181515 device_plugin_stub.go:180] Deprecation file not found. Invoke registration
I0714 12:46:00.905687  181515 manager.go:410] Got registration request from device plugin with resource name "fake-domain/resource"
I0714 12:46:00.908679  181515 device_plugin_stub.go:227] ListAndWatch
I0714 12:46:00.914106  181515 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin979901548/device-plugin.sock.new
I0714 12:46:00.914218  181515 device_plugin_stub.go:180] Deprecation file not found. Invoke registration
I0714 12:46:00.916994  181515 manager.go:410] Got registration request from device plugin with resource name "fake-domain/resource"
I0714 12:46:00.920499  181515 device_plugin_stub.go:227] ListAndWatch
I0714 12:46:00.925921  181515 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin979901548/device-plugin.sock.third
I0714 12:46:00.926021  181515 device_plugin_stub.go:180] Deprecation file not found. Invoke registration
I0714 12:46:00.930182  181515 manager.go:410] Got registration request from device plugin with resource name "fake-domain/resource"
I0714 12:46:00.934644  181515 device_plugin_stub.go:227] ListAndWatch
E0714 12:46:00.938920  181515 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Unavailable desc = transport is closing
E0714 12:46:00.939330  181515 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Unavailable desc = transport is closing
E0714 12:46:00.939455  181515 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Canceled desc = grpc: the client connection is closing
I0714 12:46:00.940310  181515 fake_topology_manager.go:29] [fake topologymanager] NewFakeManager
W0714 12:46:00.940601  181515 manager.go:596] Failed to retrieve checkpoint for "kubelet_internal_checkpoint": checkpoint is not found
I0714 12:46:00.941515  181515 plugin_manager.go:114] Starting Kubelet Plugin Manager
I0714 12:46:00.943953  181515 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin112359879/device-plugin.sock
I0714 12:46:00.946082  181515 device_plugin_stub.go:153] GetInfo
E0714 12:46:00.946516  181515 goroutinemap.go:150] Operation for "/tmp/device_plugin112359879/server.sock" failed. No retries permitted until 2021-07-14 12:46:01.446359989 +0000 UTC m=+0.884998783 (durationBeforeRetry 500ms). Error: "RegisterPlugin error -- failed to get plugin info using RPC GetInfo at socket /tmp/device_plugin112359879/server.sock, err: rpc error: code = Unimplemented desc = unknown service pluginregistration.Registration"
I0714 12:46:00.956549  181515 device_plugin_stub.go:227] ListAndWatch
I0714 12:46:00.963948  181515 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin112359879/device-plugin.sock.new
I0714 12:46:01.947105  181515 device_plugin_stub.go:153] GetInfo
E0714 12:46:01.947314  181515 goroutinemap.go:150] Operation for "/tmp/device_plugin112359879/server.sock" failed. No retries permitted until 2021-07-14 12:46:02.947123496 +0000 UTC m=+2.385762292 (durationBeforeRetry 1s). Error: "RegisterPlugin error -- failed to get plugin info using RPC GetInfo at socket /tmp/device_plugin112359879/server.sock, err: rpc error: code = Unimplemented desc = unknown service pluginregistration.Registration"
I0714 12:46:01.955534  181515 device_plugin_stub.go:227] ListAndWatch
I0714 12:46:01.968291  181515 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin112359879/device-plugin.sock.third
I0714 12:46:02.947308  181515 device_plugin_stub.go:153] GetInfo
I0714 12:46:02.953874  181515 device_plugin_stub.go:227] ListAndWatch
E0714 12:46:02.967897  181515 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Unavailable desc = transport is closing
E0714 12:46:02.968861  181515 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Unavailable desc = transport is closing
E0714 12:46:02.969175  181515 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Unavailable desc = transport is closing
I0714 12:46:02.970116  181515 fake_topology_manager.go:29] [fake topologymanager] NewFakeManager
E0714 12:46:02.986521  181515 manager.go:546] unexpected: unhealthyDevices and endpoints are out of sync
I0714 12:46:03.003162  181515 fake_topology_manager.go:29] [fake topologymanager] NewFakeManager
I0714 12:46:03.003388  181515 fake_topology_manager.go:34] [fake topologymanager] GetAffinity podUID: c9621f19-224f-438e-96ec-193cf3e1c08a container name:  
E0714 12:46:10.738772  181515 goroutinemap.go:150] Operation for "/tmp/device_plugin404643585/device-plugin.sock" failed. No retries permitted until 2021-07-14 12:46:11.23859764 +0000 UTC m=+10.677236499 (durationBeforeRetry 500ms). Error: "RegisterPlugin error -- dial failed at socket /tmp/device_plugin404643585/device-plugin.sock, err: failed to dial socket /tmp/device_plugin404643585/device-plugin.sock, err: context deadline exceeded"
E0714 12:46:10.749145  181515 goroutinemap.go:150] Operation for "/tmp/device_plugin404643585/server.sock" failed. No retries permitted until 2021-07-14 12:46:11.248984483 +0000 UTC m=+10.687623289 (durationBeforeRetry 500ms). Error: "RegisterPlugin error -- dial failed at socket /tmp/device_plugin404643585/server.sock, err: failed to dial socket /tmp/device_plugin404643585/server.sock, err: context deadline exceeded"
Terminated
-- Test timed out at 2021-07-14 12:51:00 UTC --
================================================================================
==================== Test output for //pkg/kubelet/cm/devicemanager:go_default_test:
I0714 12:51:15.936712  223084 device_plugin_stub.go:131] Starting to serve on /tmp/mock.sock
I0714 12:51:15.965281  223084 device_plugin_stub.go:131] Starting to serve on /tmp/mock.sock
I0714 12:51:15.974487  223084 device_plugin_stub.go:227] ListAndWatch
E0714 12:51:15.979597  223084 endpoint.go:107] listAndWatch ended unexpectedly for device plugin mock with error rpc error: code = Unavailable desc = transport is closing
I0714 12:51:15.980967  223084 device_plugin_stub.go:131] Starting to serve on /tmp/mock.sock
I0714 12:51:15.984315  223084 device_plugin_stub.go:227] ListAndWatch
I0714 12:51:15.985599  223084 device_plugin_stub.go:261] Allocate, &AllocateRequest{ContainerRequests:[]*ContainerAllocateRequest{&ContainerAllocateRequest{DevicesIDs:[ADeviceId],},},}
E0714 12:51:15.987237  223084 endpoint.go:107] listAndWatch ended unexpectedly for device plugin mock with error rpc error: code = Canceled desc = grpc: the client connection is closing
I0714 12:51:15.991471  223084 device_plugin_stub.go:131] Starting to serve on /tmp/mock.sock
I0714 12:51:15.993966  223084 device_plugin_stub.go:227] ListAndWatch
I0714 12:51:15.995189  223084 device_plugin_stub.go:248] GetPreferredAllocation, &PreferredAllocationRequest{ContainerRequests:[]*ContainerPreferredAllocationRequest{&ContainerPreferredAllocationRequest{AvailableDeviceIDs:[],MustIncludeDeviceIDs:[],AllocationSize:-1,},},}
I0714 12:51:15.996854  223084 fake_topology_manager.go:29] [fake topologymanager] NewFakeManager
I0714 12:51:15.997486  223084 fake_topology_manager.go:29] [fake topologymanager] NewFakeManager
W0714 12:51:15.997710  223084 manager.go:596] Failed to retrieve checkpoint for "kubelet_internal_checkpoint": checkpoint is not found
E0714 12:51:15.998364  223084 endpoint.go:107] listAndWatch ended unexpectedly for device plugin mock with error rpc error: code = Canceled desc = grpc: the client connection is closing
I0714 12:51:16.000604  223084 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin168260754/device-plugin.sock
I0714 12:51:16.001769  223084 fake_topology_manager.go:29] [fake topologymanager] NewFakeManager
W0714 12:51:16.002106  223084 manager.go:596] Failed to retrieve checkpoint for "kubelet_internal_checkpoint": checkpoint is not found
I0714 12:51:16.004461  223084 plugin_manager.go:114] Starting Kubelet Plugin Manager
I0714 12:51:16.038267  223084 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin636618441/device-plugin.sock
E0714 12:51:16.039689  223084 goroutinemap.go:150] Operation for "/tmp/device_plugin636618441/device-plugin.sock" failed. No retries permitted until 2021-07-14 12:51:16.539470716 +0000 UTC m=+0.759487546 (durationBeforeRetry 500ms). Error: "RegisterPlugin error -- failed to get plugin info using RPC GetInfo at socket /tmp/device_plugin636618441/device-plugin.sock, err: rpc error: code = Unavailable desc = transport is closing"
I0714 12:51:16.042210  223084 fake_topology_manager.go:29] [fake topologymanager] NewFakeManager
E0714 12:51:16.042390  223084 goroutinemap.go:150] Operation for "/tmp/device_plugin636618441/server.sock" failed. No retries permitted until 2021-07-14 12:51:16.542054749 +0000 UTC m=+0.762071582 (durationBeforeRetry 500ms). Error: "RegisterPlugin error -- failed to get plugin info using RPC GetInfo at socket /tmp/device_plugin636618441/server.sock, err: rpc error: code = Unavailable desc = transport is closing"
W0714 12:51:16.042465  223084 manager.go:596] Failed to retrieve checkpoint for "kubelet_internal_checkpoint": checkpoint is not found
I0714 12:51:16.044662  223084 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin942739348/device-plugin.sock
I0714 12:51:16.044751  223084 device_plugin_stub.go:180] Deprecation file not found. Invoke registration
I0714 12:51:16.047894  223084 manager.go:410] Got registration request from device plugin with resource name "fake-domain/resource"
I0714 12:51:16.059006  223084 device_plugin_stub.go:227] ListAndWatch
I0714 12:51:16.066701  223084 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin942739348/device-plugin.sock.new
I0714 12:51:16.066799  223084 device_plugin_stub.go:180] Deprecation file not found. Invoke registration
I0714 12:51:16.070116  223084 manager.go:410] Got registration request from device plugin with resource name "fake-domain/resource"
I0714 12:51:16.084606  223084 device_plugin_stub.go:227] ListAndWatch
I0714 12:51:16.092066  223084 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin942739348/device-plugin.sock.third
I0714 12:51:16.092162  223084 device_plugin_stub.go:180] Deprecation file not found. Invoke registration
I0714 12:51:16.094505  223084 manager.go:410] Got registration request from device plugin with resource name "fake-domain/resource"
I0714 12:51:16.099486  223084 device_plugin_stub.go:227] ListAndWatch
E0714 12:51:16.105155  223084 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Unavailable desc = transport is closing
E0714 12:51:16.105918  223084 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Unavailable desc = transport is closing
I0714 12:51:16.106405  223084 fake_topology_manager.go:29] [fake topologymanager] NewFakeManager
E0714 12:51:16.106486  223084 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Unavailable desc = transport is closing
I0714 12:51:16.108832  223084 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin942739348/device-plugin.sock
I0714 12:51:16.108918  223084 device_plugin_stub.go:180] Deprecation file not found. Invoke registration
I0714 12:51:16.111332  223084 manager.go:410] Got registration request from device plugin with resource name "fake-domain/resource"
I0714 12:51:16.119926  223084 device_plugin_stub.go:227] ListAndWatch
I0714 12:51:16.127605  223084 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin942739348/device-plugin.sock.new
I0714 12:51:16.127699  223084 device_plugin_stub.go:180] Deprecation file not found. Invoke registration
I0714 12:51:16.130442  223084 manager.go:410] Got registration request from device plugin with resource name "fake-domain/resource"
I0714 12:51:16.134306  223084 device_plugin_stub.go:227] ListAndWatch
I0714 12:51:16.142365  223084 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin942739348/device-plugin.sock.third
I0714 12:51:16.142472  223084 device_plugin_stub.go:180] Deprecation file not found. Invoke registration
I0714 12:51:16.154700  223084 manager.go:410] Got registration request from device plugin with resource name "fake-domain/resource"
I0714 12:51:16.159668  223084 device_plugin_stub.go:227] ListAndWatch
E0714 12:51:16.165573  223084 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Canceled desc = grpc: the client connection is closing
I0714 12:51:16.165936  223084 fake_topology_manager.go:29] [fake topologymanager] NewFakeManager
E0714 12:51:16.167629  223084 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Unavailable desc = transport is closing
E0714 12:51:16.167686  223084 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Unavailable desc = transport is closing
I0714 12:51:16.176915  223084 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin942739348/device-plugin.sock
I0714 12:51:16.177216  223084 device_plugin_stub.go:180] Deprecation file not found. Invoke registration
I0714 12:51:16.180404  223084 manager.go:410] Got registration request from device plugin with resource name "fake-domain/resource"
I0714 12:51:16.183957  223084 device_plugin_stub.go:227] ListAndWatch
I0714 12:51:16.195024  223084 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin942739348/device-plugin.sock.new
I0714 12:51:16.195310  223084 device_plugin_stub.go:180] Deprecation file not found. Invoke registration
I0714 12:51:16.199509  223084 manager.go:410] Got registration request from device plugin with resource name "fake-domain/resource"
I0714 12:51:16.202592  223084 device_plugin_stub.go:227] ListAndWatch
I0714 12:51:16.210223  223084 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin942739348/device-plugin.sock.third
I0714 12:51:16.210321  223084 device_plugin_stub.go:180] Deprecation file not found. Invoke registration
I0714 12:51:16.212476  223084 manager.go:410] Got registration request from device plugin with resource name "fake-domain/resource"
I0714 12:51:16.222154  223084 device_plugin_stub.go:227] ListAndWatch
I0714 12:51:16.229776  223084 fake_topology_manager.go:29] [fake topologymanager] NewFakeManager
E0714 12:51:16.229927  223084 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Unavailable desc = transport is closing
I0714 12:51:16.233535  223084 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin942739348/device-plugin.sock
I0714 12:51:16.233618  223084 device_plugin_stub.go:180] Deprecation file not found. Invoke registration
E0714 12:51:16.235733  223084 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Unavailable desc = transport is closing
E0714 12:51:16.239024  223084 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Canceled desc = grpc: the client connection is closing
I0714 12:51:16.239896  223084 manager.go:410] Got registration request from device plugin with resource name "fake-domain/resource"
I0714 12:51:16.255915  223084 device_plugin_stub.go:227] ListAndWatch
I0714 12:51:16.266481  223084 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin942739348/device-plugin.sock.new
I0714 12:51:16.266582  223084 device_plugin_stub.go:180] Deprecation file not found. Invoke registration
I0714 12:51:16.269553  223084 manager.go:410] Got registration request from device plugin with resource name "fake-domain/resource"
I0714 12:51:16.275357  223084 device_plugin_stub.go:227] ListAndWatch
I0714 12:51:16.282504  223084 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin942739348/device-plugin.sock.third
I0714 12:51:16.282599  223084 device_plugin_stub.go:180] Deprecation file not found. Invoke registration
I0714 12:51:16.288917  223084 manager.go:410] Got registration request from device plugin with resource name "fake-domain/resource"
I0714 12:51:16.292412  223084 device_plugin_stub.go:227] ListAndWatch
E0714 12:51:16.297201  223084 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Unavailable desc = transport is closing
E0714 12:51:16.298024  223084 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Canceled desc = grpc: the client connection is closing
I0714 12:51:16.298678  223084 fake_topology_manager.go:29] [fake topologymanager] NewFakeManager
E0714 12:51:16.299082  223084 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Unavailable desc = transport is closing
W0714 12:51:16.299448  223084 manager.go:596] Failed to retrieve checkpoint for "kubelet_internal_checkpoint": checkpoint is not found
I0714 12:51:16.300593  223084 plugin_manager.go:114] Starting Kubelet Plugin Manager
I0714 12:51:16.302599  223084 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin440923727/device-plugin.sock
I0714 12:51:16.306020  223084 device_plugin_stub.go:153] GetInfo
E0714 12:51:16.306036  223084 goroutinemap.go:150] Operation for "/tmp/device_plugin440923727/server.sock" failed. No retries permitted until 2021-07-14 12:51:16.805855497 +0000 UTC m=+1.025872335 (durationBeforeRetry 500ms). Error: "RegisterPlugin error -- failed to get plugin info using RPC GetInfo at socket /tmp/device_plugin440923727/server.sock, err: rpc error: code = Unimplemented desc = unknown service pluginregistration.Registration"
I0714 12:51:16.311962  223084 device_plugin_stub.go:227] ListAndWatch
I0714 12:51:16.326750  223084 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin440923727/device-plugin.sock.new
I0714 12:51:17.304890  223084 device_plugin_stub.go:153] GetInfo
E0714 12:51:17.305235  223084 goroutinemap.go:150] Operation for "/tmp/device_plugin440923727/server.sock" failed. No retries permitted until 2021-07-14 12:51:18.305061406 +0000 UTC m=+2.525078241 (durationBeforeRetry 1s). Error: "RegisterPlugin error -- failed to get plugin info using RPC GetInfo at socket /tmp/device_plugin440923727/server.sock, err: rpc error: code = Unimplemented desc = unknown service pluginregistration.Registration"
I0714 12:51:17.309693  223084 device_plugin_stub.go:227] ListAndWatch
I0714 12:51:17.323507  223084 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin440923727/device-plugin.sock.third
I0714 12:51:18.307027  223084 device_plugin_stub.go:153] GetInfo
I0714 12:51:18.313089  223084 device_plugin_stub.go:227] ListAndWatch
E0714 12:51:18.317465  223084 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Unavailable desc = transport is closing
E0714 12:51:18.318073  223084 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Unavailable desc = transport is closing
E0714 12:51:18.318501  223084 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Unavailable desc = transport is closing
I0714 12:51:18.319293  223084 fake_topology_manager.go:29] [fake topologymanager] NewFakeManager
E0714 12:51:18.336359  223084 manager.go:546] unexpected: unhealthyDevices and endpoints are out of sync
I0714 12:51:18.370494  223084 fake_topology_manager.go:29] [fake topologymanager] NewFakeManager
I0714 12:51:18.370722  223084 fake_topology_manager.go:34] [fake topologymanager] GetAffinity podUID: 2b0907ec-8388-4f95-b2ee-533b331ba180 container name:  
Terminated
-- Test timed out at 2021-07-14 12:56:15 UTC --
================================================================================
==================== Test output for //pkg/kubelet/cm/devicemanager:go_default_test:
I0714 12:56:30.737999  241916 device_plugin_stub.go:131] Starting to serve on /tmp/mock.sock
I0714 12:56:30.742451  241916 device_plugin_stub.go:131] Starting to serve on /tmp/mock.sock
I0714 12:56:30.746296  241916 device_plugin_stub.go:227] ListAndWatch
E0714 12:56:30.747489  241916 endpoint.go:107] listAndWatch ended unexpectedly for device plugin mock with error rpc error: code = Unavailable desc = transport is closing
I0714 12:56:30.749493  241916 device_plugin_stub.go:131] Starting to serve on /tmp/mock.sock
I0714 12:56:30.752226  241916 device_plugin_stub.go:227] ListAndWatch
I0714 12:56:30.753337  241916 device_plugin_stub.go:261] Allocate, &AllocateRequest{ContainerRequests:[]*ContainerAllocateRequest{&ContainerAllocateRequest{DevicesIDs:[ADeviceId],},},}
E0714 12:56:30.754507  241916 endpoint.go:107] listAndWatch ended unexpectedly for device plugin mock with error rpc error: code = Canceled desc = grpc: the client connection is closing
I0714 12:56:30.757572  241916 device_plugin_stub.go:131] Starting to serve on /tmp/mock.sock
I0714 12:56:30.759877  241916 device_plugin_stub.go:227] ListAndWatch
I0714 12:56:30.761071  241916 device_plugin_stub.go:248] GetPreferredAllocation, &PreferredAllocationRequest{ContainerRequests:[]*ContainerPreferredAllocationRequest{&ContainerPreferredAllocationRequest{AvailableDeviceIDs:[],MustIncludeDeviceIDs:[],AllocationSize:-1,},},}
E0714 12:56:30.762326  241916 endpoint.go:107] listAndWatch ended unexpectedly for device plugin mock with error rpc error: code = Unavailable desc = transport is closing
I0714 12:56:30.762811  241916 fake_topology_manager.go:29] [fake topologymanager] NewFakeManager
I0714 12:56:30.763195  241916 fake_topology_manager.go:29] [fake topologymanager] NewFakeManager
W0714 12:56:30.763408  241916 manager.go:596] Failed to retrieve checkpoint for "kubelet_internal_checkpoint": checkpoint is not found
I0714 12:56:30.765459  241916 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin017986006/device-plugin.sock
I0714 12:56:30.766276  241916 fake_topology_manager.go:29] [fake topologymanager] NewFakeManager
W0714 12:56:30.766527  241916 manager.go:596] Failed to retrieve checkpoint for "kubelet_internal_checkpoint": checkpoint is not found
I0714 12:56:30.767723  241916 plugin_manager.go:114] Starting Kubelet Plugin Manager
I0714 12:56:30.768983  241916 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin248488253/device-plugin.sock
I0714 12:56:30.770747  241916 fake_topology_manager.go:29] [fake topologymanager] NewFakeManager
W0714 12:56:30.770963  241916 manager.go:596] Failed to retrieve checkpoint for "kubelet_internal_checkpoint": checkpoint is not found
I0714 12:56:30.773386  241916 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin191224184/device-plugin.sock
I0714 12:56:30.773464  241916 device_plugin_stub.go:180] Deprecation file not found. Invoke registration
I0714 12:56:30.776144  241916 manager.go:410] Got registration request from device plugin with resource name "fake-domain/resource"
I0714 12:56:30.779104  241916 device_plugin_stub.go:227] ListAndWatch
I0714 12:56:30.787353  241916 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin191224184/device-plugin.sock.new
I0714 12:56:30.787455  241916 device_plugin_stub.go:180] Deprecation file not found. Invoke registration
I0714 12:56:30.789406  241916 manager.go:410] Got registration request from device plugin with resource name "fake-domain/resource"
I0714 12:56:30.792199  241916 device_plugin_stub.go:227] ListAndWatch
I0714 12:56:30.800417  241916 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin191224184/device-plugin.sock.third
I0714 12:56:30.800519  241916 device_plugin_stub.go:180] Deprecation file not found. Invoke registration
I0714 12:56:30.802438  241916 manager.go:410] Got registration request from device plugin with resource name "fake-domain/resource"
I0714 12:56:30.804988  241916 device_plugin_stub.go:227] ListAndWatch
E0714 12:56:30.810115  241916 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Unavailable desc = transport is closing
E0714 12:56:30.810527  241916 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Unavailable desc = transport is closing
E0714 12:56:30.811053  241916 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Unavailable desc = transport is closing
I0714 12:56:30.811175  241916 fake_topology_manager.go:29] [fake topologymanager] NewFakeManager
I0714 12:56:30.813231  241916 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin191224184/device-plugin.sock
I0714 12:56:30.813303  241916 device_plugin_stub.go:180] Deprecation file not found. Invoke registration
I0714 12:56:30.815344  241916 manager.go:410] Got registration request from device plugin with resource name "fake-domain/resource"
I0714 12:56:30.818382  241916 device_plugin_stub.go:227] ListAndWatch
I0714 12:56:30.825057  241916 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin191224184/device-plugin.sock.new
I0714 12:56:30.825166  241916 device_plugin_stub.go:180] Deprecation file not found. Invoke registration
I0714 12:56:30.828249  241916 manager.go:410] Got registration request from device plugin with resource name "fake-domain/resource"
I0714 12:56:30.830832  241916 device_plugin_stub.go:227] ListAndWatch
I0714 12:56:30.836161  241916 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin191224184/device-plugin.sock.third
I0714 12:56:30.836250  241916 device_plugin_stub.go:180] Deprecation file not found. Invoke registration
I0714 12:56:30.838030  241916 manager.go:410] Got registration request from device plugin with resource name "fake-domain/resource"
I0714 12:56:30.840437  241916 device_plugin_stub.go:227] ListAndWatch
E0714 12:56:30.844922  241916 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Unavailable desc = transport is closing
E0714 12:56:30.845210  241916 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Unavailable desc = transport is closing
E0714 12:56:30.845556  241916 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Unavailable desc = transport is closing
I0714 12:56:30.845902  241916 fake_topology_manager.go:29] [fake topologymanager] NewFakeManager
I0714 12:56:30.847823  241916 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin191224184/device-plugin.sock
I0714 12:56:30.847919  241916 device_plugin_stub.go:180] Deprecation file not found. Invoke registration
I0714 12:56:30.850918  241916 manager.go:410] Got registration request from device plugin with resource name "fake-domain/resource"
I0714 12:56:30.853267  241916 device_plugin_stub.go:227] ListAndWatch
I0714 12:56:30.858125  241916 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin191224184/device-plugin.sock.new
I0714 12:56:30.858227  241916 device_plugin_stub.go:180] Deprecation file not found. Invoke registration
I0714 12:56:30.859860  241916 manager.go:410] Got registration request from device plugin with resource name "fake-domain/resource"
I0714 12:56:30.862166  241916 device_plugin_stub.go:227] ListAndWatch
I0714 12:56:30.867228  241916 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin191224184/device-plugin.sock.third
I0714 12:56:30.867312  241916 device_plugin_stub.go:180] Deprecation file not found. Invoke registration
I0714 12:56:30.870687  241916 manager.go:410] Got registration request from device plugin with resource name "fake-domain/resource"
I0714 12:56:30.873092  241916 device_plugin_stub.go:227] ListAndWatch
E0714 12:56:30.877991  241916 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Unavailable desc = transport is closing
E0714 12:56:30.878210  241916 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Unavailable desc = transport is closing
E0714 12:56:30.878639  241916 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Unavailable desc = transport is closing
I0714 12:56:30.878933  241916 fake_topology_manager.go:29] [fake topologymanager] NewFakeManager
I0714 12:56:30.880917  241916 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin191224184/device-plugin.sock
I0714 12:56:30.880974  241916 device_plugin_stub.go:180] Deprecation file not found. Invoke registration
I0714 12:56:30.882963  241916 manager.go:410] Got registration request from device plugin with resource name "fake-domain/resource"
I0714 12:56:30.885651  241916 device_plugin_stub.go:227] ListAndWatch
I0714 12:56:30.891161  241916 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin191224184/device-plugin.sock.new
I0714 12:56:30.891249  241916 device_plugin_stub.go:180] Deprecation file not found. Invoke registration
I0714 12:56:30.892967  241916 manager.go:410] Got registration request from device plugin with resource name "fake-domain/resource"
I0714 12:56:30.897176  241916 device_plugin_stub.go:227] ListAndWatch
I0714 12:56:30.902576  241916 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin191224184/device-plugin.sock.third
I0714 12:56:30.902671  241916 device_plugin_stub.go:180] Deprecation file not found. Invoke registration
I0714 12:56:30.904572  241916 manager.go:410] Got registration request from device plugin with resource name "fake-domain/resource"
I0714 12:56:30.907646  241916 device_plugin_stub.go:227] ListAndWatch
E0714 12:56:30.912640  241916 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Unavailable desc = transport is closing
E0714 12:56:30.913379  241916 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Unavailable desc = transport is closing
E0714 12:56:30.913375  241916 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Unavailable desc = transport is closing
I0714 12:56:30.914081  241916 fake_topology_manager.go:29] [fake topologymanager] NewFakeManager
W0714 12:56:30.914348  241916 manager.go:596] Failed to retrieve checkpoint for "kubelet_internal_checkpoint": checkpoint is not found
I0714 12:56:30.915180  241916 plugin_manager.go:114] Starting Kubelet Plugin Manager
I0714 12:56:30.916645  241916 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin142253731/device-plugin.sock
I0714 12:56:30.919844  241916 device_plugin_stub.go:153] GetInfo
E0714 12:56:30.920643  241916 goroutinemap.go:150] Operation for "/tmp/device_plugin142253731/server.sock" failed. No retries permitted until 2021-07-14 12:56:31.42050195 +0000 UTC m=+0.767437133 (durationBeforeRetry 500ms). Error: "RegisterPlugin error -- failed to get plugin info using RPC GetInfo at socket /tmp/device_plugin142253731/server.sock, err: rpc error: code = Unimplemented desc = unknown service pluginregistration.Registration"
I0714 12:56:30.923898  241916 device_plugin_stub.go:227] ListAndWatch
I0714 12:56:30.929166  241916 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin142253731/device-plugin.sock.new
I0714 12:56:31.919451  241916 device_plugin_stub.go:153] GetInfo
E0714 12:56:31.919959  241916 goroutinemap.go:150] Operation for "/tmp/device_plugin142253731/server.sock" failed. No retries permitted until 2021-07-14 12:56:32.919790318 +0000 UTC m=+2.266725508 (durationBeforeRetry 1s). Error: "RegisterPlugin error -- failed to get plugin info using RPC GetInfo at socket /tmp/device_plugin142253731/server.sock, err: rpc error: code = Unimplemented desc = unknown service pluginregistration.Registration"
I0714 12:56:31.923052  241916 device_plugin_stub.go:227] ListAndWatch
I0714 12:56:31.930311  241916 device_plugin_stub.go:131] Starting to serve on /tmp/device_plugin142253731/device-plugin.sock.third
I0714 12:56:32.920196  241916 device_plugin_stub.go:153] GetInfo
I0714 12:56:32.924817  241916 device_plugin_stub.go:227] ListAndWatch
E0714 12:56:32.929248  241916 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Unavailable desc = transport is closing
E0714 12:56:32.929460  241916 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Unavailable desc = transport is closing
E0714 12:56:32.929854  241916 endpoint.go:107] listAndWatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = Unavailable desc = transport is closing
I0714 12:56:32.930607  241916 fake_topology_manager.go:29] [fake topologymanager] NewFakeManager
E0714 12:56:32.946224  241916 manager.go:546] unexpected: unhealthyDevices and endpoints are out of sync
I0714 12:56:32.959063  241916 fake_topology_manager.go:29] [fake topologymanager] NewFakeManager
I0714 12:56:32.959275  241916 fake_topology_manager.go:34] [fake topologymanager] GetAffinity podUID: 95713eb7-6bb7-4a34-8c80-ab2e5dfca84c container name:  
E0714 12:56:40.769495  241916 goroutinemap.go:150] Operation for "/tmp/device_plugin248488253/device-plugin.sock" failed. No retries permitted until 2021-07-14 12:56:41.269350437 +0000 UTC m=+10.616285623 (durationBeforeRetry 500ms). Error: "RegisterPlugin error -- dial failed at socket /tmp/device_plugin248488253/device-plugin.sock, err: failed to dial socket /tmp/device_plugin248488253/device-plugin.sock, err: context deadline exceeded"
E0714 12:56:40.770216  241916 goroutinemap.go:150] Operation for "/tmp/device_plugin248488253/server.sock" failed. No retries permitted until 2021-07-14 12:56:41.27004987 +0000 UTC m=+10.616985050 (durationBeforeRetry 500ms). Error: "RegisterPlugin error -- dial failed at socket /tmp/device_plugin248488253/server.sock, err: failed to dial socket /tmp/device_plugin248488253/server.sock, err: context deadline exceeded"
Terminated
-- Test timed out at 2021-07-14 13:01:30 UTC --
================================================================================
INFO: Elapsed time: 1906.031s, Critical Path: 1026.89s
INFO: 17356 processes: 17356 processwrapper-sandbox.
INFO: Build completed, 1 test FAILED, 19281 total actions
//cluster:common_test                                                    PASSED in 0.1s
//cluster:kube-util_test                                                 PASSED in 0.1s
//cluster/gce/cos:go_default_test                                        PASSED in 3.8s
//cluster/gce/custom:go_default_test                                     PASSED in 2.9s
//cluster/gce/gci:go_default_test                                        PASSED in 3.6s
//cluster/gce/ubuntu:go_default_test                                     PASSED in 3.8s
... skipping 954 lines ...
  /bazel-scratch/.cache/bazel/_bazel_root/7989b31489f31aee54f32688da2f0120/execroot/io_k8s_kubernetes/bazel-out/k8-fastbuild/testlogs/pkg/kubelet/cm/devicemanager/go_default_test/test.log
  /bazel-scratch/.cache/bazel/_bazel_root/7989b31489f31aee54f32688da2f0120/execroot/io_k8s_kubernetes/bazel-out/k8-fastbuild/testlogs/pkg/kubelet/cm/devicemanager/go_default_test/test_attempts/attempt_1.log
  /bazel-scratch/.cache/bazel/_bazel_root/7989b31489f31aee54f32688da2f0120/execroot/io_k8s_kubernetes/bazel-out/k8-fastbuild/testlogs/pkg/kubelet/cm/devicemanager/go_default_test/test_attempts/attempt_2.log

Executed 959 out of 959 tests: 958 tests pass and 1 fails locally.
There were tests whose specified size is too big. Use the --test_verbose_timeout_warnings command line option to see which ones these are.
INFO: Build completed, 1 test FAILED, 19281 total actions
+ ../test-infra/hack/coalesce.py
+ exit 3