This job view page is being replaced by Spyglass soon. Check out the new job view.
ResultFAILURE
Tests 6 failed / 1937 succeeded
Started2021-06-24 08:16
Elapsed30m47s
Revisionrelease-1.20

Test Failures


//pkg/controller/volume/attachdetach/go_default_test:run_1_of_2 0.00s

bazel test //pkg/controller/volume/attachdetach/go_default_test:run_1_of_2
exec ${PAGER:-/usr/bin/less} "$0" || exit 1
Executing tests from //pkg/controller/volume/attachdetach:go_default_test
-----------------------------------------------------------------------------
I0624 08:25:59.624866  116460 shared_informer.go:240] Waiting for caches to sync for attach detach
I0624 08:25:59.736817  116460 shared_informer.go:247] Caches are synced for attach detach 
I0624 08:25:59.746235  116460 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0624 08:25:59.746415  116460 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0624 08:25:59.746557  116460 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0624 08:25:59.747642  116460 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0624 08:25:59.747863  116460 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0624 08:25:59.747983  116460 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0624 08:25:59.748983  116460 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0624 08:25:59.749056  116460 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0624 08:25:59.749183  116460 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0624 08:25:59.750235  116460 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0624 08:25:59.750378  116460 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0624 08:25:59.750496  116460 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0624 08:25:59.751495  116460 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0624 08:25:59.751607  116460 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0624 08:25:59.751717  116460 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0624 08:25:59.755493  116460 reconciler.go:295] attacherDetacher.AttachVolume started for volume "volumeName" (UniqueName: "kubernetes.io/testPlugin/volumeName") from node "mynode" 
I0624 08:25:59.756018  116460 event.go:291] "Event occurred" object="mynamespace/mypod-0" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:25:59.756215  116460 event.go:291] "Event occurred" object="mynamespace/mypod-1" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:25:59.756395  116460 event.go:291] "Event occurred" object="mynamespace/mypod-2" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:25:59.756479  116460 operation_generator.go:360] AttachVolume.Attach succeeded for volume "volumeName" (UniqueName: "kubernetes.io/testPlugin/volumeName") from node "mynode" 
I0624 08:25:59.756560  116460 event.go:291] "Event occurred" object="mynamespace/mypod-3" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:25:59.756733  116460 event.go:291] "Event occurred" object="mynamespace/mypod-4" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:01.742182  116460 shared_informer.go:240] Waiting for caches to sync for attach detach
I0624 08:26:01.852064  116460 shared_informer.go:247] Caches are synced for attach detach 
I0624 08:26:01.859447  116460 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0624 08:26:01.859578  116460 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0624 08:26:01.859989  116460 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0624 08:26:01.861802  116460 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0624 08:26:01.861940  116460 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0624 08:26:01.862212  116460 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0624 08:26:01.864080  116460 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0624 08:26:01.864182  116460 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0624 08:26:01.864501  116460 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0624 08:26:01.865999  116460 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0624 08:26:01.866106  116460 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0624 08:26:01.866334  116460 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0624 08:26:01.867766  116460 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0624 08:26:01.867879  116460 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0624 08:26:01.868140  116460 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0624 08:26:01.869449  116460 operation_generator.go:360] AttachVolume.Attach succeeded for volume "volumeName" (UniqueName: "kubernetes.io/testPlugin/volumeName") from node "mynode" 
I0624 08:26:01.870511  116460 event.go:291] "Event occurred" object="mynamespace/mypod-4" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:01.870728  116460 event.go:291] "Event occurred" object="mynamespace/mypod-0" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:01.870932  116460 event.go:291] "Event occurred" object="mynamespace/mypod-1" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:01.871109  116460 event.go:291] "Event occurred" object="mynamespace/mypod-2" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:01.871280  116460 event.go:291] "Event occurred" object="mynamespace/mypod-3" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:01.871463  116460 reconciler.go:295] attacherDetacher.AttachVolume started for volume "volumeName" (UniqueName: "kubernetes.io/testPlugin/volumeName") from node "mynode" 
I0624 08:26:01.872289  116460 event.go:291] "Event occurred" object="mynamespace/newpod-1" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName2\" "
I0624 08:26:01.872547  116460 operation_generator.go:360] AttachVolume.Attach succeeded for volume "volumeName2" (UniqueName: "kubernetes.io/testPlugin/volumeName2") from node "mynode-1" 
I0624 08:26:01.872710  116460 reconciler.go:295] attacherDetacher.AttachVolume started for volume "volumeName2" (UniqueName: "kubernetes.io/testPlugin/volumeName2") from node "mynode-1" 
I0624 08:26:03.858222  116460 shared_informer.go:240] Waiting for caches to sync for attach detach
I0624 08:26:03.966824  116460 shared_informer.go:247] Caches are synced for attach detach 
I0624 08:26:03.974968  116460 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0624 08:26:03.975217  116460 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0624 08:26:03.975362  116460 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0624 08:26:03.976499  116460 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0624 08:26:03.976651  116460 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0624 08:26:03.976975  116460 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0624 08:26:03.978480  116460 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0624 08:26:03.978612  116460 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0624 08:26:03.978895  116460 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0624 08:26:03.980174  116460 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0624 08:26:03.980324  116460 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0624 08:26:03.980443  116460 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0624 08:26:03.981117  116460 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0624 08:26:03.981392  116460 reconciler.go:295] attacherDetacher.AttachVolume started for volume "volumeName" (UniqueName: "kubernetes.io/testPlugin/volumeName") from node "mynode" 
I0624 08:26:03.981605  116460 reconciler.go:295] attacherDetacher.AttachVolume started for volume "volumeName2" (UniqueName: "kubernetes.io/testPlugin/volumeName2") from node "mynode-1" 
I0624 08:26:03.981930  116460 operation_generator.go:360] AttachVolume.Attach succeeded for volume "volumeName2" (UniqueName: "kubernetes.io/testPlugin/volumeName2") from node "mynode-1" 
I0624 08:26:03.982161  116460 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0624 08:26:03.982236  116460 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0624 08:26:03.983164  116460 operation_generator.go:360] AttachVolume.Attach succeeded for volume "volumeName" (UniqueName: "kubernetes.io/testPlugin/volumeName") from node "mynode" 
I0624 08:26:03.983514  116460 event.go:291] "Event occurred" object="mynamespace/newpod-1" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName2\" "
I0624 08:26:03.983653  116460 event.go:291] "Event occurred" object="mynamespace/mypod-2" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:03.995933  116460 event.go:291] "Event occurred" object="mynamespace/mypod-3" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:03.996230  116460 event.go:291] "Event occurred" object="mynamespace/mypod-4" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:03.996413  116460 event.go:291] "Event occurred" object="mynamespace/mypod-0" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:03.996584  116460 event.go:291] "Event occurred" object="mynamespace/mypod-1" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:05.971341  116460 shared_informer.go:240] Waiting for caches to sync for attach detach
I0624 08:26:06.078693  116460 shared_informer.go:247] Caches are synced for attach detach 
I0624 08:26:06.086082  116460 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0624 08:26:06.086240  116460 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0624 08:26:06.086382  116460 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0624 08:26:06.087286  116460 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0624 08:26:06.087448  116460 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0624 08:26:06.087573  116460 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0624 08:26:06.088975  116460 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0624 08:26:06.089071  116460 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0624 08:26:06.089177  116460 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0624 08:26:06.090149  116460 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0624 08:26:06.090350  116460 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0624 08:26:06.090474  116460 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0624 08:26:06.091417  116460 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0624 08:26:06.091508  116460 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0624 08:26:06.091752  116460 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0624 08:26:06.092280  116460 operation_generator.go:360] AttachVolume.Attach succeeded for volume "volumeName2" (UniqueName: "kubernetes.io/testPlugin/volumeName2") from node "mynode-1" 
I0624 08:26:06.092710  116460 event.go:291] "Event occurred" object="mynamespace/newpod-1" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName2\" "
I0624 08:26:06.093002  116460 reconciler.go:295] attacherDetacher.AttachVolume started for volume "volumeName2" (UniqueName: "kubernetes.io/testPlugin/volumeName2") from node "mynode-1" 
I0624 08:26:06.093463  116460 operation_generator.go:360] AttachVolume.Attach succeeded for volume "volumeName3" (UniqueName: "kubernetes.io/testPlugin/volumeName3") from node "mynode-1" 
I0624 08:26:06.093735  116460 event.go:291] "Event occurred" object="mynamespace/newpod-2" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName3\" "
I0624 08:26:06.094040  116460 reconciler.go:295] attacherDetacher.AttachVolume started for volume "volumeName3" (UniqueName: "kubernetes.io/testPlugin/volumeName3") from node "mynode-1" 
I0624 08:26:06.095138  116460 operation_generator.go:360] AttachVolume.Attach succeeded for volume "volumeName" (UniqueName: "kubernetes.io/testPlugin/volumeName") from node "mynode" 
I0624 08:26:06.095580  116460 event.go:291] "Event occurred" object="mynamespace/mypod-3" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:06.096116  116460 event.go:291] "Event occurred" object="mynamespace/mypod-0" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:06.096279  116460 event.go:291] "Event occurred" object="mynamespace/mypod-1" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:06.096429  116460 event.go:291] "Event occurred" object="mynamespace/mypod-2" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:06.096598  116460 event.go:291] "Event occurred" object="mynamespace/mypod-4" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:06.096769  116460 reconciler.go:295] attacherDetacher.AttachVolume started for volume "volumeName" (UniqueName: "kubernetes.io/testPlugin/volumeName") from node "mynode" 
I0624 08:26:08.086146  116460 shared_informer.go:240] Waiting for caches to sync for attach detach
I0624 08:26:08.186739  116460 shared_informer.go:247] Caches are synced for attach detach 
I0624 08:26:08.193735  116460 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0624 08:26:08.193969  116460 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0624 08:26:08.194093  116460 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0624 08:26:08.195034  116460 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0624 08:26:08.195233  116460 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0624 08:26:08.195351  116460 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0624 08:26:08.196501  116460 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0624 08:26:08.196660  116460 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0624 08:26:08.196806  116460 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0624 08:26:08.197776  116460 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0624 08:26:08.197891  116460 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0624 08:26:08.198010  116460 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0624 08:26:08.199048  116460 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0624 08:26:08.199173  116460 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0624 08:26:08.199307  116460 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0624 08:26:08.199430  116460 reconciler.go:295] attacherDetacher.AttachVolume started for volume "volumeName" (UniqueName: "kubernetes.io/testPlugin/volumeName") from node "mynode" 
I0624 08:26:08.199702  116460 reconciler.go:295] attacherDetacher.AttachVolume started for volume "vol1" (UniqueName: "kubernetes.io/testPlugin/vol1") from node "mynode-1" 
I0624 08:26:08.200142  116460 operation_generator.go:360] AttachVolume.Attach succeeded for volume "vol1" (UniqueName: "kubernetes.io/testPlugin/vol1") from node "mynode-1" 
I0624 08:26:08.200302  116460 event.go:291] "Event occurred" object="mynamespace/mypod-4" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:08.200504  116460 event.go:291] "Event occurred" object="mynamespace/pod1" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"vol1\" "
I0624 08:26:08.200710  116460 operation_generator.go:360] AttachVolume.Attach succeeded for volume "volumeName" (UniqueName: "kubernetes.io/testPlugin/volumeName") from node "mynode" 
I0624 08:26:08.200728  116460 event.go:291] "Event occurred" object="mynamespace/mypod-0" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:08.200946  116460 event.go:291] "Event occurred" object="mynamespace/mypod-1" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:08.201120  116460 event.go:291] "Event occurred" object="mynamespace/mypod-2" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:08.201291  116460 event.go:291] "Event occurred" object="mynamespace/mypod-3" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:09.191662  116460 shared_informer.go:240] Waiting for caches to sync for attach detach
I0624 08:26:09.292201  116460 shared_informer.go:247] Caches are synced for attach detach 
I0624 08:26:09.300259  116460 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0624 08:26:09.300409  116460 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0624 08:26:09.300636  116460 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0624 08:26:09.301929  116460 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0624 08:26:09.302121  116460 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0624 08:26:09.302221  116460 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0624 08:26:09.303238  116460 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0624 08:26:09.303383  116460 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0624 08:26:09.303475  116460 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0624 08:26:09.304631  116460 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0624 08:26:09.304736  116460 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0624 08:26:09.304853  116460 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0624 08:26:09.305896  116460 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0624 08:26:09.306195  116460 reconciler.go:295] attacherDetacher.AttachVolume started for volume "volumeName" (UniqueName: "kubernetes.io/testPlugin/volumeName") from node "mynode" 
I0624 08:26:09.306425  116460 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0624 08:26:09.306550  116460 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0624 08:26:09.307351  116460 event.go:291] "Event occurred" object="mynamespace/mypod-0" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:09.307435  116460 operation_generator.go:360] AttachVolume.Attach succeeded for volume "volumeName" (UniqueName: "kubernetes.io/testPlugin/volumeName") from node "mynode" 
I0624 08:26:09.307592  116460 event.go:291] "Event occurred" object="mynamespace/mypod-1" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:09.307829  116460 event.go:291] "Event occurred" object="mynamespace/mypod-2" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:09.308024  116460 event.go:291] "Event occurred" object="mynamespace/mypod-3" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:09.308185  116460 event.go:291] "Event occurred" object="mynamespace/mypod-4" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:10.214157  116460 reconciler.go:219] attacherDetacher.DetachVolume started for volume "pv1" (UniqueName: "kubernetes.io/testPlugin/vol1") on node "mynode-1" 
I0624 08:26:10.214296  116460 operation_generator.go:1409] Verified volume is safe to detach for volume "pv1" (UniqueName: "kubernetes.io/testPlugin/vol1") on node "mynode-1" 
I0624 08:26:10.214392  116460 operation_generator.go:470] DetachVolume.Detach succeeded for volume "pv1" (UniqueName: "kubernetes.io/testPlugin/vol1") on node "mynode-1" 
I0624 08:26:10.297811  116460 shared_informer.go:240] Waiting for caches to sync for attach detach
I0624 08:26:10.398350  116460 shared_informer.go:247] Caches are synced for attach detach 
I0624 08:26:10.403482  116460 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0624 08:26:10.403709  116460 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0624 08:26:10.403855  116460 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0624 08:26:10.404620  116460 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0624 08:26:10.404882  116460 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0624 08:26:10.405051  116460 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0624 08:26:10.405743  116460 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0624 08:26:10.405903  116460 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0624 08:26:10.406034  116460 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0624 08:26:10.406714  116460 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0624 08:26:10.406850  116460 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0624 08:26:10.406938  116460 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0624 08:26:10.407922  116460 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0624 08:26:10.408012  116460 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0624 08:26:10.408110  116460 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0624 08:26:10.408240  116460 reconciler.go:295] attacherDetacher.AttachVolume started for volume "volumeName" (UniqueName: "kubernetes.io/testPlugin/volumeName") from node "mynode" 
I0624 08:26:10.408878  116460 event.go:291] "Event occurred" object="mynamespace/mypod-3" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:10.409056  116460 operation_generator.go:360] AttachVolume.Attach succeeded for volume "volumeName" (UniqueName: "kubernetes.io/testPlugin/volumeName") from node "mynode" 
I0624 08:26:10.409100  116460 event.go:291] "Event occurred" object="mynamespace/mypod-4" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:10.409288  116460 event.go:291] "Event occurred" object="mynamespace/mypod-1" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:10.409449  116460 event.go:291] "Event occurred" object="mynamespace/mypod-0" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:10.409557  116460 event.go:291] "Event occurred" object="mynamespace/mypod-2" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:11.313349  116460 reconciler.go:219] attacherDetacher.DetachVolume started for volume "pv1" (UniqueName: "kubernetes.io/testPlugin/vol1") on node "mynode-1" 
I0624 08:26:11.313470  116460 operation_generator.go:1409] Verified volume is safe to detach for volume "pv1" (UniqueName: "kubernetes.io/testPlugin/vol1") on node "mynode-1" 
I0624 08:26:11.313573  116460 operation_generator.go:470] DetachVolume.Detach succeeded for volume "pv1" (UniqueName: "kubernetes.io/testPlugin/vol1") on node "mynode-1" 
I0624 08:26:11.403936  116460 shared_informer.go:240] Waiting for caches to sync for attach detach
I0624 08:26:11.504467  116460 shared_informer.go:247] Caches are synced for attach detach 
W0624 08:26:11.506642  116460 reflector.go:436] k8s.io/client-go/informers/factory.go:134: watch of *v1.PersistentVolumeClaim ended with: very short watch: k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
W0624 08:26:11.507035  116460 reflector.go:436] k8s.io/client-go/informers/factory.go:134: watch of *v1.CSIDriver ended with: very short watch: k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
W0624 08:26:11.507281  116460 reflector.go:436] k8s.io/client-go/informers/factory.go:134: watch of *v1.Pod ended with: very short watch: k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
I0624 08:26:11.510750  116460 shared_informer.go:240] Waiting for caches to sync for attach detach
E0624 08:26:11.511949  116460 reconciler.go:228] attacherDetacher.DetachVolume failed to start for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" : DetachVolume.FindAttachablePluginByName failed for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" : no volume plugin matched name: kubernetes.io/testPlugin
E0624 08:26:11.513240  116460 reconciler.go:228] attacherDetacher.DetachVolume failed to start for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" : DetachVolume.FindAttachablePluginByName failed for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" : no volume plugin matched name: kubernetes.io/testPlugin
E0624 08:26:11.514462  116460 reconciler.go:228] attacherDetacher.DetachVolume failed to start for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" : DetachVolume.FindAttachablePluginByName failed for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" : no volume plugin matched name: kubernetes.io/testPlugin
E0624 08:26:11.515651  116460 reconciler.go:228] attacherDetacher.DetachVolume failed to start for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" : DetachVolume.FindAttachablePluginByName failed for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" : no volume plugin matched name: kubernetes.io/testPlugin
I0624 08:26:11.525766  116460 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/csi/pd.csi.storage.gke.io^projects/UNSPECIFIED/zones/UNSPECIFIED/disks/vol1") on node "mynode-1" 
I0624 08:26:11.526276  116460 reconciler.go:295] attacherDetacher.AttachVolume started for volume "pd.csi.storage.gke.io-pdName" (UniqueName: "kubernetes.io/csi/pd.csi.storage.gke.io^projects/UNSPECIFIED/zones/UNSPECIFIED/disks/pdName") from node "mynode" 
I0624 08:26:11.526387  116460 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/csi/pd.csi.storage.gke.io^projects/UNSPECIFIED/zones/UNSPECIFIED/disks/vol1") on node "mynode-1" 
==================
WARNING: DATA RACE
Write at 0x00c0002501a8 by goroutine 179:
  k8s.io/kubernetes/pkg/controller/volume/attachdetach/testing.CreateTestClient.func7()
      pkg/controller/volume/attachdetach/testing/testvolumespec.go:185 +0x207
  k8s.io/kubernetes/vendor/k8s.io/client-go/testing.(*SimpleReactor).React()
      staging/src/k8s.io/client-go/testing/fixture.go:521 +0x68
  k8s.io/kubernetes/vendor/k8s.io/client-go/testing.(*Fake).Invokes()
      staging/src/k8s.io/client-go/testing/fake.go:141 +0x327
  k8s.io/kubernetes/vendor/k8s.io/client-go/kubernetes/typed/storage/v1/fake.(*FakeVolumeAttachments).Create()
      staging/src/k8s.io/client-go/kubernetes/typed/storage/v1/fake/fake_volumeattachment.go:82 +0x246
  k8s.io/kubernetes/pkg/volume/csi.(*csiAttacher).Attach()
      pkg/volume/csi/csi_attacher.go:109 +0x5f3
  k8s.io/kubernetes/pkg/volume/util/operationexecutor.(*operationGenerator).GenerateAttachVolumeFunc.func1()
      pkg/volume/util/operationexecutor/operation_generator.go:335 +0x275
  k8s.io/kubernetes/pkg/volume/util/types.(*GeneratedOperations).Run()
      pkg/volume/util/types/types.go:54 +0x142
  k8s.io/kubernetes/pkg/volume/util/nestedpendingoperations.(*nestedPendingOperations).Run.func1()
      pkg/volume/util/nestedpendingoperations/nestedpendingoperations.go:183 +0x159

Previous read at 0x00c0002501a8 by goroutine 136:
  k8s.io/kubernetes/pkg/controller/volume/attachdetach/testing.CreateTestClient.func6()
      pkg/controller/volume/attachdetach/testing/testvolumespec.go:179 +0x104
  k8s.io/kubernetes/vendor/k8s.io/client-go/testing.(*SimpleReactor).React()
      staging/src/k8s.io/client-go/testing/fixture.go:521 +0x68
  k8s.io/kubernetes/vendor/k8s.io/client-go/testing.(*Fake).Invokes()
      staging/src/k8s.io/client-go/testing/fake.go:141 +0x327
  k8s.io/kubernetes/vendor/k8s.io/client-go/kubernetes/typed/storage/v1/fake.(*FakeVolumeAttachments).List()
      staging/src/k8s.io/client-go/kubernetes/typed/storage/v1/fake/fake_volumeattachment.go:55 +0x273
  k8s.io/kubernetes/vendor/k8s.io/client-go/informers/storage/v1.NewFilteredVolumeAttachmentInformer.func1()
      staging/src/k8s.io/client-go/informers/storage/v1/volumeattachment.go:64 +0x23c
  k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.(*ListWatch).List()
      staging/src/k8s.io/client-go/tools/cache/listwatch.go:106 +0x97
  k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch.func1.1.2()
      staging/src/k8s.io/client-go/tools/cache/reflector.go:277 +0xf4
  k8s.io/kubernetes/vendor/k8s.io/client-go/tools/pager.SimplePageFunc.func1()
      staging/src/k8s.io/client-go/tools/pager/pager.go:40 +0x97
  k8s.io/kubernetes/vendor/k8s.io/client-go/tools/pager.(*ListPager).List()
      staging/src/k8s.io/client-go/tools/pager/pager.go:91 +0x1e2
  k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch.func1.1()
      staging/src/k8s.io/client-go/tools/cache/reflector.go:302 +0x2b7

Goroutine 179 (running) created at:
  k8s.io/kubernetes/pkg/volume/util/nestedpendingoperations.(*nestedPendingOperations).Run()
      pkg/volume/util/nestedpendingoperations/nestedpendingoperations.go:178 +0x5b2
  k8s.io/kubernetes/pkg/volume/util/operationexecutor.(*operationExecutor).AttachVolume()
      pkg/volume/util/operationexecutor/operation_executor.go:675 +0x348
  k8s.io/kubernetes/pkg/controller/volume/attachdetach/reconciler.(*reconciler).attachDesiredVolumes()
      pkg/controller/volume/attachdetach/reconciler/reconciler.go:293 +0x639
  k8s.io/kubernetes/pkg/controller/volume/attachdetach/reconciler.(*reconciler).reconcile()
      pkg/controller/volume/attachdetach/reconciler/reconciler.go:233 +0x1b9
  k8s.io/kubernetes/pkg/controller/volume/attachdetach/reconciler.(*reconciler).reconciliationLoopFunc.func1()
      pkg/controller/volume/attachdetach/reconciler/reconciler.go:110 +0x5e
  k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1()
      staging/src/k8s.io/apimachinery/pkg/util/wait/wait.go:155 +0x75
  k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.BackoffUntil()
      staging/src/k8s.io/apimachinery/pkg/util/wait/wait.go:156 +0xba
  k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil()
      staging/src/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x114
  k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.Until()
      staging/src/k8s.io/apimachinery/pkg/util/wait/wait.go:90 +0x89
  k8s.io/kubernetes/pkg/controller/volume/attachdetach/reconciler.(*reconciler).Run()
      pkg/controller/volume/attachdetach/reconciler/reconciler.go:101 +0x2f

Goroutine 136 (finished) created at:
  k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch.func1()
      staging/src/k8s.io/client-go/tools/cache/reflector.go:268 +0x36f
  k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch()
      staging/src/k8s.io/client-go/tools/cache/reflector.go:360 +0x2e4
  k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.(*Reflector).Run.func1()
      staging/src/k8s.io/client-go/tools/cache/reflector.go:221 +0x53
  k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1()
      staging/src/k8s.io/apimachinery/pkg/util/wait/wait.go:155 +0x75
  k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.BackoffUntil()
      staging/src/k8s.io/apimachinery/pkg/util/wait/wait.go:156 +0xba
  k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.(*Reflector).Run()
      staging/src/k8s.io/client-go/tools/cache/reflector.go:220 +0x2c4
  k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.(*Reflector).Run-fm()
      staging/src/k8s.io/client-go/tools/cache/reflector.go:218 +0x4b
  k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).StartWithChannel.func1()
      staging/src/k8s.io/apimachinery/pkg/util/wait/wait.go:56 +0x45
  k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
      staging/src/k8s.io/apimachinery/pkg/util/wait/wait.go:73 +0x6d
==================
E0624 08:26:11.527948  116460 csi_attacher.go:455] [attachment.ID=csi-77bd5b88d32d48e5a31f0073211a7dd54ed0066f008e12364f4b6ecb61dc7370] watch channel had been closed
E0624 08:26:11.528269  116460 csi_attacher.go:455] [attachment.ID=csi-4ff6fd4b10c02f804e7435dffeb62054338b590749cdaacf18e198ee0ab6dd66] watch channel had been closed
E0624 08:26:11.544498  116460 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/csi/pd.csi.storage.gke.io^projects/UNSPECIFIED/zones/UNSPECIFIED/disks/vol1 podName: nodeName:mynode-1}" failed. No retries permitted until 2021-06-24 08:26:12.04418822 +0000 UTC m=+12.659523971 (durationBeforeRetry 500ms). Error: "DetachVolume.Detach failed for volume \"nil\" (UniqueName: \"kubernetes.io/csi/pd.csi.storage.gke.io^projects/UNSPECIFIED/zones/UNSPECIFIED/disks/vol1\") on node \"mynode-1\" : volume attachment watch channel had been closed"
E0624 08:26:11.545587  116460 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/csi/pd.csi.storage.gke.io^projects/UNSPECIFIED/zones/UNSPECIFIED/disks/pdName podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:26:12.045319608 +0000 UTC m=+12.660655353 (durationBeforeRetry 500ms). Error: "AttachVolume.Attach failed for volume \"pd.csi.storage.gke.io-pdName\" (UniqueName: \"kubernetes.io/csi/pd.csi.storage.gke.io^projects/UNSPECIFIED/zones/UNSPECIFIED/disks/pdName\") from node \"mynode\" : volume attachment watch channel had been closed"
I0624 08:26:11.546573  116460 event.go:291] "Event occurred" object="mynamespace/mypod-0" kind="Pod" apiVersion="v1" type="Warning" reason="FailedAttachVolume" message="AttachVolume.Attach failed for volume \"pd.csi.storage.gke.io-pdName\" : volume attachment watch channel had been closed"
I0624 08:26:11.547676  116460 event.go:291] "Event occurred" object="mynamespace/mypod-3" kind="Pod" apiVersion="v1" type="Warning" reason="FailedAttachVolume" message="AttachVolume.Attach failed for volume \"pd.csi.storage.gke.io-pdName\" : volume attachment watch channel had been closed"
I0624 08:26:11.547895  116460 event.go:291] "Event occurred" object="mynamespace/mypod-4" kind="Pod" apiVersion="v1" type="Warning" reason="FailedAttachVolume" message="AttachVolume.Attach failed for volume \"pd.csi.storage.gke.io-pdName\" : volume attachment watch channel had been closed"
I0624 08:26:11.548098  116460 event.go:291] "Event occurred" object="mynamespace/mypod-2" kind="Pod" apiVersion="v1" type="Warning" reason="FailedAttachVolume" message="AttachVolume.Attach failed for volume \"pd.csi.storage.gke.io-pdName\" : volume attachment watch channel had been closed"
I0624 08:26:11.548264  116460 event.go:291] "Event occurred" object="mynamespace/mypod-1" kind="Pod" apiVersion="v1" type="Warning" reason="FailedAttachVolume" message="AttachVolume.Attach failed for volume \"pd.csi.storage.gke.io-pdName\" : volume attachment watch channel had been closed"
I0624 08:26:11.611863  116460 shared_informer.go:247] Caches are synced for attach detach 
--- FAIL: Test_ADC_VolumeAttachmentRecovery (3.53s)
    --- FAIL: Test_ADC_VolumeAttachmentRecovery/Deleted_Pod_with_migrated_PV (0.11s)
        testing.go:1038: race detected during execution of test
    testing.go:1038: race detected during execution of test
W0624 08:26:11.614784  116460 reflector.go:436] k8s.io/client-go/informers/factory.go:134: watch of *v1.Pod ended with: very short watch: k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
FAIL

				from junit_bazel.xml

Filter through log files | View test history on testgrid


//pkg/controller/volume/attachdetach/go_default_test:run_1_of_2 0.00s

bazel test //pkg/controller/volume/attachdetach/go_default_test:run_1_of_2
exec ${PAGER:-/usr/bin/less} "$0" || exit 1
Executing tests from //pkg/controller/volume/attachdetach:go_default_test
-----------------------------------------------------------------------------
I0624 08:25:59.624866  116460 shared_informer.go:240] Waiting for caches to sync for attach detach
I0624 08:25:59.736817  116460 shared_informer.go:247] Caches are synced for attach detach 
I0624 08:25:59.746235  116460 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0624 08:25:59.746415  116460 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0624 08:25:59.746557  116460 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0624 08:25:59.747642  116460 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0624 08:25:59.747863  116460 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0624 08:25:59.747983  116460 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0624 08:25:59.748983  116460 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0624 08:25:59.749056  116460 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0624 08:25:59.749183  116460 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0624 08:25:59.750235  116460 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0624 08:25:59.750378  116460 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0624 08:25:59.750496  116460 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0624 08:25:59.751495  116460 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0624 08:25:59.751607  116460 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0624 08:25:59.751717  116460 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0624 08:25:59.755493  116460 reconciler.go:295] attacherDetacher.AttachVolume started for volume "volumeName" (UniqueName: "kubernetes.io/testPlugin/volumeName") from node "mynode" 
I0624 08:25:59.756018  116460 event.go:291] "Event occurred" object="mynamespace/mypod-0" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:25:59.756215  116460 event.go:291] "Event occurred" object="mynamespace/mypod-1" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:25:59.756395  116460 event.go:291] "Event occurred" object="mynamespace/mypod-2" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:25:59.756479  116460 operation_generator.go:360] AttachVolume.Attach succeeded for volume "volumeName" (UniqueName: "kubernetes.io/testPlugin/volumeName") from node "mynode" 
I0624 08:25:59.756560  116460 event.go:291] "Event occurred" object="mynamespace/mypod-3" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:25:59.756733  116460 event.go:291] "Event occurred" object="mynamespace/mypod-4" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:01.742182  116460 shared_informer.go:240] Waiting for caches to sync for attach detach
I0624 08:26:01.852064  116460 shared_informer.go:247] Caches are synced for attach detach 
I0624 08:26:01.859447  116460 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0624 08:26:01.859578  116460 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0624 08:26:01.859989  116460 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0624 08:26:01.861802  116460 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0624 08:26:01.861940  116460 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0624 08:26:01.862212  116460 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0624 08:26:01.864080  116460 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0624 08:26:01.864182  116460 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0624 08:26:01.864501  116460 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0624 08:26:01.865999  116460 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0624 08:26:01.866106  116460 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0624 08:26:01.866334  116460 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0624 08:26:01.867766  116460 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0624 08:26:01.867879  116460 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0624 08:26:01.868140  116460 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0624 08:26:01.869449  116460 operation_generator.go:360] AttachVolume.Attach succeeded for volume "volumeName" (UniqueName: "kubernetes.io/testPlugin/volumeName") from node "mynode" 
I0624 08:26:01.870511  116460 event.go:291] "Event occurred" object="mynamespace/mypod-4" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:01.870728  116460 event.go:291] "Event occurred" object="mynamespace/mypod-0" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:01.870932  116460 event.go:291] "Event occurred" object="mynamespace/mypod-1" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:01.871109  116460 event.go:291] "Event occurred" object="mynamespace/mypod-2" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:01.871280  116460 event.go:291] "Event occurred" object="mynamespace/mypod-3" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:01.871463  116460 reconciler.go:295] attacherDetacher.AttachVolume started for volume "volumeName" (UniqueName: "kubernetes.io/testPlugin/volumeName") from node "mynode" 
I0624 08:26:01.872289  116460 event.go:291] "Event occurred" object="mynamespace/newpod-1" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName2\" "
I0624 08:26:01.872547  116460 operation_generator.go:360] AttachVolume.Attach succeeded for volume "volumeName2" (UniqueName: "kubernetes.io/testPlugin/volumeName2") from node "mynode-1" 
I0624 08:26:01.872710  116460 reconciler.go:295] attacherDetacher.AttachVolume started for volume "volumeName2" (UniqueName: "kubernetes.io/testPlugin/volumeName2") from node "mynode-1" 
I0624 08:26:03.858222  116460 shared_informer.go:240] Waiting for caches to sync for attach detach
I0624 08:26:03.966824  116460 shared_informer.go:247] Caches are synced for attach detach 
I0624 08:26:03.974968  116460 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0624 08:26:03.975217  116460 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0624 08:26:03.975362  116460 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0624 08:26:03.976499  116460 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0624 08:26:03.976651  116460 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0624 08:26:03.976975  116460 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0624 08:26:03.978480  116460 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0624 08:26:03.978612  116460 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0624 08:26:03.978895  116460 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0624 08:26:03.980174  116460 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0624 08:26:03.980324  116460 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0624 08:26:03.980443  116460 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0624 08:26:03.981117  116460 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0624 08:26:03.981392  116460 reconciler.go:295] attacherDetacher.AttachVolume started for volume "volumeName" (UniqueName: "kubernetes.io/testPlugin/volumeName") from node "mynode" 
I0624 08:26:03.981605  116460 reconciler.go:295] attacherDetacher.AttachVolume started for volume "volumeName2" (UniqueName: "kubernetes.io/testPlugin/volumeName2") from node "mynode-1" 
I0624 08:26:03.981930  116460 operation_generator.go:360] AttachVolume.Attach succeeded for volume "volumeName2" (UniqueName: "kubernetes.io/testPlugin/volumeName2") from node "mynode-1" 
I0624 08:26:03.982161  116460 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0624 08:26:03.982236  116460 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0624 08:26:03.983164  116460 operation_generator.go:360] AttachVolume.Attach succeeded for volume "volumeName" (UniqueName: "kubernetes.io/testPlugin/volumeName") from node "mynode" 
I0624 08:26:03.983514  116460 event.go:291] "Event occurred" object="mynamespace/newpod-1" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName2\" "
I0624 08:26:03.983653  116460 event.go:291] "Event occurred" object="mynamespace/mypod-2" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:03.995933  116460 event.go:291] "Event occurred" object="mynamespace/mypod-3" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:03.996230  116460 event.go:291] "Event occurred" object="mynamespace/mypod-4" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:03.996413  116460 event.go:291] "Event occurred" object="mynamespace/mypod-0" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:03.996584  116460 event.go:291] "Event occurred" object="mynamespace/mypod-1" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:05.971341  116460 shared_informer.go:240] Waiting for caches to sync for attach detach
I0624 08:26:06.078693  116460 shared_informer.go:247] Caches are synced for attach detach 
I0624 08:26:06.086082  116460 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0624 08:26:06.086240  116460 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0624 08:26:06.086382  116460 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0624 08:26:06.087286  116460 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0624 08:26:06.087448  116460 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0624 08:26:06.087573  116460 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0624 08:26:06.088975  116460 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0624 08:26:06.089071  116460 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0624 08:26:06.089177  116460 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0624 08:26:06.090149  116460 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0624 08:26:06.090350  116460 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0624 08:26:06.090474  116460 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0624 08:26:06.091417  116460 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0624 08:26:06.091508  116460 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0624 08:26:06.091752  116460 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0624 08:26:06.092280  116460 operation_generator.go:360] AttachVolume.Attach succeeded for volume "volumeName2" (UniqueName: "kubernetes.io/testPlugin/volumeName2") from node "mynode-1" 
I0624 08:26:06.092710  116460 event.go:291] "Event occurred" object="mynamespace/newpod-1" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName2\" "
I0624 08:26:06.093002  116460 reconciler.go:295] attacherDetacher.AttachVolume started for volume "volumeName2" (UniqueName: "kubernetes.io/testPlugin/volumeName2") from node "mynode-1" 
I0624 08:26:06.093463  116460 operation_generator.go:360] AttachVolume.Attach succeeded for volume "volumeName3" (UniqueName: "kubernetes.io/testPlugin/volumeName3") from node "mynode-1" 
I0624 08:26:06.093735  116460 event.go:291] "Event occurred" object="mynamespace/newpod-2" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName3\" "
I0624 08:26:06.094040  116460 reconciler.go:295] attacherDetacher.AttachVolume started for volume "volumeName3" (UniqueName: "kubernetes.io/testPlugin/volumeName3") from node "mynode-1" 
I0624 08:26:06.095138  116460 operation_generator.go:360] AttachVolume.Attach succeeded for volume "volumeName" (UniqueName: "kubernetes.io/testPlugin/volumeName") from node "mynode" 
I0624 08:26:06.095580  116460 event.go:291] "Event occurred" object="mynamespace/mypod-3" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:06.096116  116460 event.go:291] "Event occurred" object="mynamespace/mypod-0" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:06.096279  116460 event.go:291] "Event occurred" object="mynamespace/mypod-1" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:06.096429  116460 event.go:291] "Event occurred" object="mynamespace/mypod-2" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:06.096598  116460 event.go:291] "Event occurred" object="mynamespace/mypod-4" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:06.096769  116460 reconciler.go:295] attacherDetacher.AttachVolume started for volume "volumeName" (UniqueName: "kubernetes.io/testPlugin/volumeName") from node "mynode" 
I0624 08:26:08.086146  116460 shared_informer.go:240] Waiting for caches to sync for attach detach
I0624 08:26:08.186739  116460 shared_informer.go:247] Caches are synced for attach detach 
I0624 08:26:08.193735  116460 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0624 08:26:08.193969  116460 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0624 08:26:08.194093  116460 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0624 08:26:08.195034  116460 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0624 08:26:08.195233  116460 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0624 08:26:08.195351  116460 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0624 08:26:08.196501  116460 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0624 08:26:08.196660  116460 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0624 08:26:08.196806  116460 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0624 08:26:08.197776  116460 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0624 08:26:08.197891  116460 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0624 08:26:08.198010  116460 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0624 08:26:08.199048  116460 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0624 08:26:08.199173  116460 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0624 08:26:08.199307  116460 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0624 08:26:08.199430  116460 reconciler.go:295] attacherDetacher.AttachVolume started for volume "volumeName" (UniqueName: "kubernetes.io/testPlugin/volumeName") from node "mynode" 
I0624 08:26:08.199702  116460 reconciler.go:295] attacherDetacher.AttachVolume started for volume "vol1" (UniqueName: "kubernetes.io/testPlugin/vol1") from node "mynode-1" 
I0624 08:26:08.200142  116460 operation_generator.go:360] AttachVolume.Attach succeeded for volume "vol1" (UniqueName: "kubernetes.io/testPlugin/vol1") from node "mynode-1" 
I0624 08:26:08.200302  116460 event.go:291] "Event occurred" object="mynamespace/mypod-4" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:08.200504  116460 event.go:291] "Event occurred" object="mynamespace/pod1" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"vol1\" "
I0624 08:26:08.200710  116460 operation_generator.go:360] AttachVolume.Attach succeeded for volume "volumeName" (UniqueName: "kubernetes.io/testPlugin/volumeName") from node "mynode" 
I0624 08:26:08.200728  116460 event.go:291] "Event occurred" object="mynamespace/mypod-0" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:08.200946  116460 event.go:291] "Event occurred" object="mynamespace/mypod-1" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:08.201120  116460 event.go:291] "Event occurred" object="mynamespace/mypod-2" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:08.201291  116460 event.go:291] "Event occurred" object="mynamespace/mypod-3" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:09.191662  116460 shared_informer.go:240] Waiting for caches to sync for attach detach
I0624 08:26:09.292201  116460 shared_informer.go:247] Caches are synced for attach detach 
I0624 08:26:09.300259  116460 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0624 08:26:09.300409  116460 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0624 08:26:09.300636  116460 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0624 08:26:09.301929  116460 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0624 08:26:09.302121  116460 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0624 08:26:09.302221  116460 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0624 08:26:09.303238  116460 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0624 08:26:09.303383  116460 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0624 08:26:09.303475  116460 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0624 08:26:09.304631  116460 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0624 08:26:09.304736  116460 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0624 08:26:09.304853  116460 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0624 08:26:09.305896  116460 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0624 08:26:09.306195  116460 reconciler.go:295] attacherDetacher.AttachVolume started for volume "volumeName" (UniqueName: "kubernetes.io/testPlugin/volumeName") from node "mynode" 
I0624 08:26:09.306425  116460 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0624 08:26:09.306550  116460 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0624 08:26:09.307351  116460 event.go:291] "Event occurred" object="mynamespace/mypod-0" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:09.307435  116460 operation_generator.go:360] AttachVolume.Attach succeeded for volume "volumeName" (UniqueName: "kubernetes.io/testPlugin/volumeName") from node "mynode" 
I0624 08:26:09.307592  116460 event.go:291] "Event occurred" object="mynamespace/mypod-1" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:09.307829  116460 event.go:291] "Event occurred" object="mynamespace/mypod-2" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:09.308024  116460 event.go:291] "Event occurred" object="mynamespace/mypod-3" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:09.308185  116460 event.go:291] "Event occurred" object="mynamespace/mypod-4" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:10.214157  116460 reconciler.go:219] attacherDetacher.DetachVolume started for volume "pv1" (UniqueName: "kubernetes.io/testPlugin/vol1") on node "mynode-1" 
I0624 08:26:10.214296  116460 operation_generator.go:1409] Verified volume is safe to detach for volume "pv1" (UniqueName: "kubernetes.io/testPlugin/vol1") on node "mynode-1" 
I0624 08:26:10.214392  116460 operation_generator.go:470] DetachVolume.Detach succeeded for volume "pv1" (UniqueName: "kubernetes.io/testPlugin/vol1") on node "mynode-1" 
I0624 08:26:10.297811  116460 shared_informer.go:240] Waiting for caches to sync for attach detach
I0624 08:26:10.398350  116460 shared_informer.go:247] Caches are synced for attach detach 
I0624 08:26:10.403482  116460 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0624 08:26:10.403709  116460 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0624 08:26:10.403855  116460 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0624 08:26:10.404620  116460 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0624 08:26:10.404882  116460 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0624 08:26:10.405051  116460 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0624 08:26:10.405743  116460 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0624 08:26:10.405903  116460 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0624 08:26:10.406034  116460 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0624 08:26:10.406714  116460 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0624 08:26:10.406850  116460 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0624 08:26:10.406938  116460 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0624 08:26:10.407922  116460 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0624 08:26:10.408012  116460 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0624 08:26:10.408110  116460 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0624 08:26:10.408240  116460 reconciler.go:295] attacherDetacher.AttachVolume started for volume "volumeName" (UniqueName: "kubernetes.io/testPlugin/volumeName") from node "mynode" 
I0624 08:26:10.408878  116460 event.go:291] "Event occurred" object="mynamespace/mypod-3" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:10.409056  116460 operation_generator.go:360] AttachVolume.Attach succeeded for volume "volumeName" (UniqueName: "kubernetes.io/testPlugin/volumeName") from node "mynode" 
I0624 08:26:10.409100  116460 event.go:291] "Event occurred" object="mynamespace/mypod-4" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:10.409288  116460 event.go:291] "Event occurred" object="mynamespace/mypod-1" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:10.409449  116460 event.go:291] "Event occurred" object="mynamespace/mypod-0" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:10.409557  116460 event.go:291] "Event occurred" object="mynamespace/mypod-2" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:11.313349  116460 reconciler.go:219] attacherDetacher.DetachVolume started for volume "pv1" (UniqueName: "kubernetes.io/testPlugin/vol1") on node "mynode-1" 
I0624 08:26:11.313470  116460 operation_generator.go:1409] Verified volume is safe to detach for volume "pv1" (UniqueName: "kubernetes.io/testPlugin/vol1") on node "mynode-1" 
I0624 08:26:11.313573  116460 operation_generator.go:470] DetachVolume.Detach succeeded for volume "pv1" (UniqueName: "kubernetes.io/testPlugin/vol1") on node "mynode-1" 
I0624 08:26:11.403936  116460 shared_informer.go:240] Waiting for caches to sync for attach detach
I0624 08:26:11.504467  116460 shared_informer.go:247] Caches are synced for attach detach 
W0624 08:26:11.506642  116460 reflector.go:436] k8s.io/client-go/informers/factory.go:134: watch of *v1.PersistentVolumeClaim ended with: very short watch: k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
W0624 08:26:11.507035  116460 reflector.go:436] k8s.io/client-go/informers/factory.go:134: watch of *v1.CSIDriver ended with: very short watch: k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
W0624 08:26:11.507281  116460 reflector.go:436] k8s.io/client-go/informers/factory.go:134: watch of *v1.Pod ended with: very short watch: k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
I0624 08:26:11.510750  116460 shared_informer.go:240] Waiting for caches to sync for attach detach
E0624 08:26:11.511949  116460 reconciler.go:228] attacherDetacher.DetachVolume failed to start for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" : DetachVolume.FindAttachablePluginByName failed for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" : no volume plugin matched name: kubernetes.io/testPlugin
E0624 08:26:11.513240  116460 reconciler.go:228] attacherDetacher.DetachVolume failed to start for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" : DetachVolume.FindAttachablePluginByName failed for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" : no volume plugin matched name: kubernetes.io/testPlugin
E0624 08:26:11.514462  116460 reconciler.go:228] attacherDetacher.DetachVolume failed to start for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" : DetachVolume.FindAttachablePluginByName failed for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" : no volume plugin matched name: kubernetes.io/testPlugin
E0624 08:26:11.515651  116460 reconciler.go:228] attacherDetacher.DetachVolume failed to start for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" : DetachVolume.FindAttachablePluginByName failed for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" : no volume plugin matched name: kubernetes.io/testPlugin
I0624 08:26:11.525766  116460 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/csi/pd.csi.storage.gke.io^projects/UNSPECIFIED/zones/UNSPECIFIED/disks/vol1") on node "mynode-1" 
I0624 08:26:11.526276  116460 reconciler.go:295] attacherDetacher.AttachVolume started for volume "pd.csi.storage.gke.io-pdName" (UniqueName: "kubernetes.io/csi/pd.csi.storage.gke.io^projects/UNSPECIFIED/zones/UNSPECIFIED/disks/pdName") from node "mynode" 
I0624 08:26:11.526387  116460 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/csi/pd.csi.storage.gke.io^projects/UNSPECIFIED/zones/UNSPECIFIED/disks/vol1") on node "mynode-1" 
==================
WARNING: DATA RACE
Write at 0x00c0002501a8 by goroutine 179:
  k8s.io/kubernetes/pkg/controller/volume/attachdetach/testing.CreateTestClient.func7()
      pkg/controller/volume/attachdetach/testing/testvolumespec.go:185 +0x207
  k8s.io/kubernetes/vendor/k8s.io/client-go/testing.(*SimpleReactor).React()
      staging/src/k8s.io/client-go/testing/fixture.go:521 +0x68
  k8s.io/kubernetes/vendor/k8s.io/client-go/testing.(*Fake).Invokes()
      staging/src/k8s.io/client-go/testing/fake.go:141 +0x327
  k8s.io/kubernetes/vendor/k8s.io/client-go/kubernetes/typed/storage/v1/fake.(*FakeVolumeAttachments).Create()
      staging/src/k8s.io/client-go/kubernetes/typed/storage/v1/fake/fake_volumeattachment.go:82 +0x246
  k8s.io/kubernetes/pkg/volume/csi.(*csiAttacher).Attach()
      pkg/volume/csi/csi_attacher.go:109 +0x5f3
  k8s.io/kubernetes/pkg/volume/util/operationexecutor.(*operationGenerator).GenerateAttachVolumeFunc.func1()
      pkg/volume/util/operationexecutor/operation_generator.go:335 +0x275
  k8s.io/kubernetes/pkg/volume/util/types.(*GeneratedOperations).Run()
      pkg/volume/util/types/types.go:54 +0x142
  k8s.io/kubernetes/pkg/volume/util/nestedpendingoperations.(*nestedPendingOperations).Run.func1()
      pkg/volume/util/nestedpendingoperations/nestedpendingoperations.go:183 +0x159

Previous read at 0x00c0002501a8 by goroutine 136:
  k8s.io/kubernetes/pkg/controller/volume/attachdetach/testing.CreateTestClient.func6()
      pkg/controller/volume/attachdetach/testing/testvolumespec.go:179 +0x104
  k8s.io/kubernetes/vendor/k8s.io/client-go/testing.(*SimpleReactor).React()
      staging/src/k8s.io/client-go/testing/fixture.go:521 +0x68
  k8s.io/kubernetes/vendor/k8s.io/client-go/testing.(*Fake).Invokes()
      staging/src/k8s.io/client-go/testing/fake.go:141 +0x327
  k8s.io/kubernetes/vendor/k8s.io/client-go/kubernetes/typed/storage/v1/fake.(*FakeVolumeAttachments).List()
      staging/src/k8s.io/client-go/kubernetes/typed/storage/v1/fake/fake_volumeattachment.go:55 +0x273
  k8s.io/kubernetes/vendor/k8s.io/client-go/informers/storage/v1.NewFilteredVolumeAttachmentInformer.func1()
      staging/src/k8s.io/client-go/informers/storage/v1/volumeattachment.go:64 +0x23c
  k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.(*ListWatch).List()
      staging/src/k8s.io/client-go/tools/cache/listwatch.go:106 +0x97
  k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch.func1.1.2()
      staging/src/k8s.io/client-go/tools/cache/reflector.go:277 +0xf4
  k8s.io/kubernetes/vendor/k8s.io/client-go/tools/pager.SimplePageFunc.func1()
      staging/src/k8s.io/client-go/tools/pager/pager.go:40 +0x97
  k8s.io/kubernetes/vendor/k8s.io/client-go/tools/pager.(*ListPager).List()
      staging/src/k8s.io/client-go/tools/pager/pager.go:91 +0x1e2
  k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch.func1.1()
      staging/src/k8s.io/client-go/tools/cache/reflector.go:302 +0x2b7

Goroutine 179 (running) created at:
  k8s.io/kubernetes/pkg/volume/util/nestedpendingoperations.(*nestedPendingOperations).Run()
      pkg/volume/util/nestedpendingoperations/nestedpendingoperations.go:178 +0x5b2
  k8s.io/kubernetes/pkg/volume/util/operationexecutor.(*operationExecutor).AttachVolume()
      pkg/volume/util/operationexecutor/operation_executor.go:675 +0x348
  k8s.io/kubernetes/pkg/controller/volume/attachdetach/reconciler.(*reconciler).attachDesiredVolumes()
      pkg/controller/volume/attachdetach/reconciler/reconciler.go:293 +0x639
  k8s.io/kubernetes/pkg/controller/volume/attachdetach/reconciler.(*reconciler).reconcile()
      pkg/controller/volume/attachdetach/reconciler/reconciler.go:233 +0x1b9
  k8s.io/kubernetes/pkg/controller/volume/attachdetach/reconciler.(*reconciler).reconciliationLoopFunc.func1()
      pkg/controller/volume/attachdetach/reconciler/reconciler.go:110 +0x5e
  k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1()
      staging/src/k8s.io/apimachinery/pkg/util/wait/wait.go:155 +0x75
  k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.BackoffUntil()
      staging/src/k8s.io/apimachinery/pkg/util/wait/wait.go:156 +0xba
  k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil()
      staging/src/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x114
  k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.Until()
      staging/src/k8s.io/apimachinery/pkg/util/wait/wait.go:90 +0x89
  k8s.io/kubernetes/pkg/controller/volume/attachdetach/reconciler.(*reconciler).Run()
      pkg/controller/volume/attachdetach/reconciler/reconciler.go:101 +0x2f

Goroutine 136 (finished) created at:
  k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch.func1()
      staging/src/k8s.io/client-go/tools/cache/reflector.go:268 +0x36f
  k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch()
      staging/src/k8s.io/client-go/tools/cache/reflector.go:360 +0x2e4
  k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.(*Reflector).Run.func1()
      staging/src/k8s.io/client-go/tools/cache/reflector.go:221 +0x53
  k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1()
      staging/src/k8s.io/apimachinery/pkg/util/wait/wait.go:155 +0x75
  k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.BackoffUntil()
      staging/src/k8s.io/apimachinery/pkg/util/wait/wait.go:156 +0xba
  k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.(*Reflector).Run()
      staging/src/k8s.io/client-go/tools/cache/reflector.go:220 +0x2c4
  k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.(*Reflector).Run-fm()
      staging/src/k8s.io/client-go/tools/cache/reflector.go:218 +0x4b
  k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).StartWithChannel.func1()
      staging/src/k8s.io/apimachinery/pkg/util/wait/wait.go:56 +0x45
  k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
      staging/src/k8s.io/apimachinery/pkg/util/wait/wait.go:73 +0x6d
==================
E0624 08:26:11.527948  116460 csi_attacher.go:455] [attachment.ID=csi-77bd5b88d32d48e5a31f0073211a7dd54ed0066f008e12364f4b6ecb61dc7370] watch channel had been closed
E0624 08:26:11.528269  116460 csi_attacher.go:455] [attachment.ID=csi-4ff6fd4b10c02f804e7435dffeb62054338b590749cdaacf18e198ee0ab6dd66] watch channel had been closed
E0624 08:26:11.544498  116460 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/csi/pd.csi.storage.gke.io^projects/UNSPECIFIED/zones/UNSPECIFIED/disks/vol1 podName: nodeName:mynode-1}" failed. No retries permitted until 2021-06-24 08:26:12.04418822 +0000 UTC m=+12.659523971 (durationBeforeRetry 500ms). Error: "DetachVolume.Detach failed for volume \"nil\" (UniqueName: \"kubernetes.io/csi/pd.csi.storage.gke.io^projects/UNSPECIFIED/zones/UNSPECIFIED/disks/vol1\") on node \"mynode-1\" : volume attachment watch channel had been closed"
E0624 08:26:11.545587  116460 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/csi/pd.csi.storage.gke.io^projects/UNSPECIFIED/zones/UNSPECIFIED/disks/pdName podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:26:12.045319608 +0000 UTC m=+12.660655353 (durationBeforeRetry 500ms). Error: "AttachVolume.Attach failed for volume \"pd.csi.storage.gke.io-pdName\" (UniqueName: \"kubernetes.io/csi/pd.csi.storage.gke.io^projects/UNSPECIFIED/zones/UNSPECIFIED/disks/pdName\") from node \"mynode\" : volume attachment watch channel had been closed"
I0624 08:26:11.546573  116460 event.go:291] "Event occurred" object="mynamespace/mypod-0" kind="Pod" apiVersion="v1" type="Warning" reason="FailedAttachVolume" message="AttachVolume.Attach failed for volume \"pd.csi.storage.gke.io-pdName\" : volume attachment watch channel had been closed"
I0624 08:26:11.547676  116460 event.go:291] "Event occurred" object="mynamespace/mypod-3" kind="Pod" apiVersion="v1" type="Warning" reason="FailedAttachVolume" message="AttachVolume.Attach failed for volume \"pd.csi.storage.gke.io-pdName\" : volume attachment watch channel had been closed"
I0624 08:26:11.547895  116460 event.go:291] "Event occurred" object="mynamespace/mypod-4" kind="Pod" apiVersion="v1" type="Warning" reason="FailedAttachVolume" message="AttachVolume.Attach failed for volume \"pd.csi.storage.gke.io-pdName\" : volume attachment watch channel had been closed"
I0624 08:26:11.548098  116460 event.go:291] "Event occurred" object="mynamespace/mypod-2" kind="Pod" apiVersion="v1" type="Warning" reason="FailedAttachVolume" message="AttachVolume.Attach failed for volume \"pd.csi.storage.gke.io-pdName\" : volume attachment watch channel had been closed"
I0624 08:26:11.548264  116460 event.go:291] "Event occurred" object="mynamespace/mypod-1" kind="Pod" apiVersion="v1" type="Warning" reason="FailedAttachVolume" message="AttachVolume.Attach failed for volume \"pd.csi.storage.gke.io-pdName\" : volume attachment watch channel had been closed"
I0624 08:26:11.611863  116460 shared_informer.go:247] Caches are synced for attach detach 
--- FAIL: Test_ADC_VolumeAttachmentRecovery (3.53s)
    --- FAIL: Test_ADC_VolumeAttachmentRecovery/Deleted_Pod_with_migrated_PV (0.11s)
        testing.go:1038: race detected during execution of test
    testing.go:1038: race detected during execution of test
W0624 08:26:11.614784  116460 reflector.go:436] k8s.io/client-go/informers/factory.go:134: watch of *v1.Pod ended with: very short watch: k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
FAIL

				from junit_bazel.xml

Filter through log files | View test history on testgrid


//pkg/controller/volume/attachdetach/go_default_test:run_2_of_2 0.00s

bazel test //pkg/controller/volume/attachdetach/go_default_test:run_2_of_2
exec ${PAGER:-/usr/bin/less} "$0" || exit 1
Executing tests from //pkg/controller/volume/attachdetach:go_default_test
-----------------------------------------------------------------------------
I0624 08:25:59.444478  116452 shared_informer.go:240] Waiting for caches to sync for attach detach
I0624 08:25:59.548002  116452 shared_informer.go:247] Caches are synced for attach detach 
I0624 08:25:59.558721  116452 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0624 08:25:59.559963  116452 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0624 08:25:59.560073  116452 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0624 08:25:59.560622  116452 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0624 08:25:59.560718  116452 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0624 08:25:59.560846  116452 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0624 08:25:59.562278  116452 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0624 08:25:59.562403  116452 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0624 08:25:59.567847  116452 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0624 08:25:59.569246  116452 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0624 08:25:59.569387  116452 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0624 08:25:59.569489  116452 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0624 08:25:59.570556  116452 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0624 08:25:59.570630  116452 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0624 08:25:59.570746  116452 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0624 08:25:59.570916  116452 reconciler.go:295] attacherDetacher.AttachVolume started for volume "volumeName" (UniqueName: "kubernetes.io/testPlugin/volumeName") from node "mynode" 
I0624 08:25:59.572186  116452 operation_generator.go:360] AttachVolume.Attach succeeded for volume "volumeName" (UniqueName: "kubernetes.io/testPlugin/volumeName") from node "mynode" 
I0624 08:25:59.572654  116452 event.go:291] "Event occurred" object="mynamespace/mypod-0" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:25:59.572845  116452 event.go:291] "Event occurred" object="mynamespace/mypod-1" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:25:59.573022  116452 event.go:291] "Event occurred" object="mynamespace/mypod-2" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:25:59.573204  116452 event.go:291] "Event occurred" object="mynamespace/mypod-3" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:25:59.573371  116452 event.go:291] "Event occurred" object="mynamespace/mypod-4" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:01.552742  116452 shared_informer.go:240] Waiting for caches to sync for attach detach
I0624 08:26:01.656153  116452 shared_informer.go:247] Caches are synced for attach detach 
I0624 08:26:01.664395  116452 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0624 08:26:01.664532  116452 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0624 08:26:01.664731  116452 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0624 08:26:01.666582  116452 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0624 08:26:01.666699  116452 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0624 08:26:01.666856  116452 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0624 08:26:01.668315  116452 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0624 08:26:01.668448  116452 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0624 08:26:01.668595  116452 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0624 08:26:01.670032  116452 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0624 08:26:01.670143  116452 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0624 08:26:01.670381  116452 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0624 08:26:01.682597  116452 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0624 08:26:01.682984  116452 reconciler.go:295] attacherDetacher.AttachVolume started for volume "volumeName2" (UniqueName: "kubernetes.io/testPlugin/volumeName2") from node "mynode-1" 
I0624 08:26:01.683280  116452 reconciler.go:295] attacherDetacher.AttachVolume started for volume "volumeName" (UniqueName: "kubernetes.io/testPlugin/volumeName") from node "mynode" 
I0624 08:26:01.683483  116452 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0624 08:26:01.683606  116452 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0624 08:26:01.684120  116452 operation_generator.go:360] AttachVolume.Attach succeeded for volume "volumeName2" (UniqueName: "kubernetes.io/testPlugin/volumeName2") from node "mynode-1" 
I0624 08:26:01.685359  116452 operation_generator.go:360] AttachVolume.Attach succeeded for volume "volumeName" (UniqueName: "kubernetes.io/testPlugin/volumeName") from node "mynode" 
I0624 08:26:01.685726  116452 event.go:291] "Event occurred" object="mynamespace/newpod-1" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName2\" "
I0624 08:26:01.686481  116452 event.go:291] "Event occurred" object="mynamespace/mypod-1" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:01.686705  116452 event.go:291] "Event occurred" object="mynamespace/mypod-2" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:01.686829  116452 event.go:291] "Event occurred" object="mynamespace/mypod-3" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:01.686960  116452 event.go:291] "Event occurred" object="mynamespace/mypod-4" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:01.687080  116452 event.go:291] "Event occurred" object="mynamespace/mypod-0" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:03.660976  116452 shared_informer.go:240] Waiting for caches to sync for attach detach
I0624 08:26:03.761576  116452 shared_informer.go:247] Caches are synced for attach detach 
I0624 08:26:03.768417  116452 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0624 08:26:03.768572  116452 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0624 08:26:03.768885  116452 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0624 08:26:03.770697  116452 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0624 08:26:03.770819  116452 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0624 08:26:03.771092  116452 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0624 08:26:03.773008  116452 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0624 08:26:03.773133  116452 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0624 08:26:03.773350  116452 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0624 08:26:03.774782  116452 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0624 08:26:03.774882  116452 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0624 08:26:03.775028  116452 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0624 08:26:03.800672  116452 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0624 08:26:03.801086  116452 reconciler.go:295] attacherDetacher.AttachVolume started for volume "volumeName" (UniqueName: "kubernetes.io/testPlugin/volumeName") from node "mynode" 
I0624 08:26:03.801328  116452 reconciler.go:295] attacherDetacher.AttachVolume started for volume "volumeName2" (UniqueName: "kubernetes.io/testPlugin/volumeName2") from node "mynode-1" 
I0624 08:26:03.801341  116452 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0624 08:26:03.801480  116452 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0624 08:26:03.801821  116452 operation_generator.go:360] AttachVolume.Attach succeeded for volume "volumeName2" (UniqueName: "kubernetes.io/testPlugin/volumeName2") from node "mynode-1" 
I0624 08:26:03.802125  116452 operation_generator.go:360] AttachVolume.Attach succeeded for volume "volumeName" (UniqueName: "kubernetes.io/testPlugin/volumeName") from node "mynode" 
I0624 08:26:03.802489  116452 event.go:291] "Event occurred" object="mynamespace/mypod-1" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:03.802739  116452 event.go:291] "Event occurred" object="mynamespace/newpod-1" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName2\" "
I0624 08:26:03.802913  116452 event.go:291] "Event occurred" object="mynamespace/mypod-2" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:03.803088  116452 event.go:291] "Event occurred" object="mynamespace/mypod-0" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:03.803251  116452 event.go:291] "Event occurred" object="mynamespace/mypod-3" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:03.803454  116452 event.go:291] "Event occurred" object="mynamespace/mypod-4" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:05.767789  116452 shared_informer.go:240] Waiting for caches to sync for attach detach
I0624 08:26:05.868251  116452 shared_informer.go:247] Caches are synced for attach detach 
I0624 08:26:05.875347  116452 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0624 08:26:05.875532  116452 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0624 08:26:05.875681  116452 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0624 08:26:05.876602  116452 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0624 08:26:05.876786  116452 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0624 08:26:05.876917  116452 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0624 08:26:05.877971  116452 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0624 08:26:05.878122  116452 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0624 08:26:05.878226  116452 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0624 08:26:05.879239  116452 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0624 08:26:05.879319  116452 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0624 08:26:05.879437  116452 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0624 08:26:05.880461  116452 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0624 08:26:05.880655  116452 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0624 08:26:05.880713  116452 reconciler.go:295] attacherDetacher.AttachVolume started for volume "volumeName" (UniqueName: "kubernetes.io/testPlugin/volumeName") from node "mynode" 
I0624 08:26:05.880764  116452 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0624 08:26:05.880945  116452 reconciler.go:295] attacherDetacher.AttachVolume started for volume "volumeName2" (UniqueName: "kubernetes.io/testPlugin/volumeName2") from node "mynode-1" 
I0624 08:26:05.881364  116452 operation_generator.go:360] AttachVolume.Attach succeeded for volume "volumeName2" (UniqueName: "kubernetes.io/testPlugin/volumeName2") from node "mynode-1" 
I0624 08:26:05.881461  116452 event.go:291] "Event occurred" object="mynamespace/mypod-2" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:05.881486  116452 operation_generator.go:360] AttachVolume.Attach succeeded for volume "volumeName" (UniqueName: "kubernetes.io/testPlugin/volumeName") from node "mynode" 
I0624 08:26:05.881829  116452 event.go:291] "Event occurred" object="mynamespace/mypod-1" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:05.882040  116452 event.go:291] "Event occurred" object="mynamespace/newpod-1" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName2\" "
I0624 08:26:05.882217  116452 event.go:291] "Event occurred" object="mynamespace/mypod-4" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:05.882385  116452 event.go:291] "Event occurred" object="mynamespace/mypod-0" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:05.882557  116452 event.go:291] "Event occurred" object="mynamespace/mypod-3" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:05.896007  116452 reconciler.go:295] attacherDetacher.AttachVolume started for volume "volumeName3" (UniqueName: "kubernetes.io/testPlugin/volumeName3") from node "mynode-1" 
I0624 08:26:05.896246  116452 operation_generator.go:360] AttachVolume.Attach succeeded for volume "volumeName3" (UniqueName: "kubernetes.io/testPlugin/volumeName3") from node "mynode-1" 
I0624 08:26:05.896655  116452 event.go:291] "Event occurred" object="mynamespace/newpod-2" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName3\" "
I0624 08:26:07.874003  116452 shared_informer.go:240] Waiting for caches to sync for attach detach
I0624 08:26:07.974629  116452 shared_informer.go:247] Caches are synced for attach detach 
I0624 08:26:07.981055  116452 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0624 08:26:07.982210  116452 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0624 08:26:07.982889  116452 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0624 08:26:07.983027  116452 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0624 08:26:07.983329  116452 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0624 08:26:07.983463  116452 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0624 08:26:07.983939  116452 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0624 08:26:07.984090  116452 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0624 08:26:07.984198  116452 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0624 08:26:07.984959  116452 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0624 08:26:07.985128  116452 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0624 08:26:07.985266  116452 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0624 08:26:07.986040  116452 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0624 08:26:07.986125  116452 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0624 08:26:07.986252  116452 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0624 08:26:07.986294  116452 reconciler.go:295] attacherDetacher.AttachVolume started for volume "volumeName" (UniqueName: "kubernetes.io/testPlugin/volumeName") from node "mynode" 
I0624 08:26:07.986520  116452 reconciler.go:295] attacherDetacher.AttachVolume started for volume "vol1" (UniqueName: "kubernetes.io/testPlugin/vol1") from node "mynode-1" 
I0624 08:26:07.986838  116452 operation_generator.go:360] AttachVolume.Attach succeeded for volume "vol1" (UniqueName: "kubernetes.io/testPlugin/vol1") from node "mynode-1" 
I0624 08:26:07.986981  116452 event.go:291] "Event occurred" object="mynamespace/mypod-0" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:07.987195  116452 event.go:291] "Event occurred" object="mynamespace/pod1" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"vol1\" "
I0624 08:26:07.987294  116452 operation_generator.go:360] AttachVolume.Attach succeeded for volume "volumeName" (UniqueName: "kubernetes.io/testPlugin/volumeName") from node "mynode" 
I0624 08:26:07.987335  116452 event.go:291] "Event occurred" object="mynamespace/mypod-1" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:07.987524  116452 event.go:291] "Event occurred" object="mynamespace/mypod-2" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:07.987633  116452 event.go:291] "Event occurred" object="mynamespace/mypod-3" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:07.987797  116452 event.go:291] "Event occurred" object="mynamespace/mypod-4" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:08.980732  116452 shared_informer.go:240] Waiting for caches to sync for attach detach
I0624 08:26:09.081316  116452 shared_informer.go:247] Caches are synced for attach detach 
I0624 08:26:09.088128  116452 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0624 08:26:09.088307  116452 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0624 08:26:09.088441  116452 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0624 08:26:09.089395  116452 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0624 08:26:09.089509  116452 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0624 08:26:09.089626  116452 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0624 08:26:09.090716  116452 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0624 08:26:09.090812  116452 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0624 08:26:09.090930  116452 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0624 08:26:09.092189  116452 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0624 08:26:09.092308  116452 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0624 08:26:09.092477  116452 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0624 08:26:09.093830  116452 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0624 08:26:09.093906  116452 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0624 08:26:09.093997  116452 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0624 08:26:09.094193  116452 reconciler.go:295] attacherDetacher.AttachVolume started for volume "volumeName" (UniqueName: "kubernetes.io/testPlugin/volumeName") from node "mynode" 
I0624 08:26:09.094858  116452 event.go:291] "Event occurred" object="mynamespace/mypod-3" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:09.094982  116452 operation_generator.go:360] AttachVolume.Attach succeeded for volume "volumeName" (UniqueName: "kubernetes.io/testPlugin/volumeName") from node "mynode" 
I0624 08:26:09.095085  116452 event.go:291] "Event occurred" object="mynamespace/mypod-4" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:09.095275  116452 event.go:291] "Event occurred" object="mynamespace/mypod-0" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:09.095465  116452 event.go:291] "Event occurred" object="mynamespace/mypod-1" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:09.095668  116452 event.go:291] "Event occurred" object="mynamespace/mypod-2" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:09.998768  116452 reconciler.go:219] attacherDetacher.DetachVolume started for volume "pv1" (UniqueName: "kubernetes.io/testPlugin/vol1") on node "mynode-1" 
I0624 08:26:09.998988  116452 operation_generator.go:1409] Verified volume is safe to detach for volume "pv1" (UniqueName: "kubernetes.io/testPlugin/vol1") on node "mynode-1" 
I0624 08:26:09.999100  116452 operation_generator.go:470] DetachVolume.Detach succeeded for volume "pv1" (UniqueName: "kubernetes.io/testPlugin/vol1") on node "mynode-1" 
I0624 08:26:10.086567  116452 shared_informer.go:240] Waiting for caches to sync for attach detach
I0624 08:26:10.189364  116452 shared_informer.go:247] Caches are synced for attach detach 
I0624 08:26:10.196341  116452 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0624 08:26:10.196468  116452 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0624 08:26:10.196658  116452 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0624 08:26:10.198016  116452 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0624 08:26:10.198153  116452 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0624 08:26:10.197824  116452 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0624 08:26:10.201154  116452 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0624 08:26:10.201259  116452 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0624 08:26:10.202683  116452 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0624 08:26:10.203932  116452 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0624 08:26:10.204055  116452 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0624 08:26:10.204173  116452 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0624 08:26:10.205179  116452 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0624 08:26:10.205385  116452 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0624 08:26:10.205485  116452 reconciler.go:295] attacherDetacher.AttachVolume started for volume "volumeName" (UniqueName: "kubernetes.io/testPlugin/volumeName") from node "mynode" 
I0624 08:26:10.205511  116452 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0624 08:26:10.206229  116452 event.go:291] "Event occurred" object="mynamespace/mypod-1" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:10.206787  116452 event.go:291] "Event occurred" object="mynamespace/mypod-2" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:10.206896  116452 operation_generator.go:360] AttachVolume.Attach succeeded for volume "volumeName" (UniqueName: "kubernetes.io/testPlugin/volumeName") from node "mynode" 
I0624 08:26:10.207025  116452 event.go:291] "Event occurred" object="mynamespace/mypod-3" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:10.207236  116452 event.go:291] "Event occurred" object="mynamespace/mypod-0" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:10.207536  116452 event.go:291] "Event occurred" object="mynamespace/mypod-4" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:11.112615  116452 reconciler.go:219] attacherDetacher.DetachVolume started for volume "pv1" (UniqueName: "kubernetes.io/testPlugin/vol1") on node "mynode-1" 
I0624 08:26:11.112752  116452 operation_generator.go:1409] Verified volume is safe to detach for volume "pv1" (UniqueName: "kubernetes.io/testPlugin/vol1") on node "mynode-1" 
I0624 08:26:11.112868  116452 operation_generator.go:470] DetachVolume.Detach succeeded for volume "pv1" (UniqueName: "kubernetes.io/testPlugin/vol1") on node "mynode-1" 
I0624 08:26:11.200897  116452 shared_informer.go:240] Waiting for caches to sync for attach detach
I0624 08:26:11.302912  116452 shared_informer.go:247] Caches are synced for attach detach 
W0624 08:26:11.304996  116452 reflector.go:436] k8s.io/client-go/informers/factory.go:134: watch of *v1.CSIDriver ended with: very short watch: k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
W0624 08:26:11.305685  116452 reflector.go:436] k8s.io/client-go/informers/factory.go:134: watch of *v1.Pod ended with: very short watch: k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
E0624 08:26:11.311777  116452 reconciler.go:228] attacherDetacher.DetachVolume failed to start for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" : DetachVolume.FindAttachablePluginByName failed for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" : no volume plugin matched name: kubernetes.io/testPlugin
I0624 08:26:11.314258  116452 shared_informer.go:240] Waiting for caches to sync for attach detach
E0624 08:26:11.314314  116452 reconciler.go:228] attacherDetacher.DetachVolume failed to start for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" : DetachVolume.FindAttachablePluginByName failed for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" : no volume plugin matched name: kubernetes.io/testPlugin
E0624 08:26:11.316281  116452 reconciler.go:228] attacherDetacher.DetachVolume failed to start for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" : DetachVolume.FindAttachablePluginByName failed for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" : no volume plugin matched name: kubernetes.io/testPlugin
E0624 08:26:11.318247  116452 reconciler.go:228] attacherDetacher.DetachVolume failed to start for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" : DetachVolume.FindAttachablePluginByName failed for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" : no volume plugin matched name: kubernetes.io/testPlugin
I0624 08:26:11.319678  116452 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/csi/pd.csi.storage.gke.io^projects/UNSPECIFIED/zones/UNSPECIFIED/disks/vol1") on node "mynode-1" 
I0624 08:26:11.319919  116452 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/csi/pd.csi.storage.gke.io^projects/UNSPECIFIED/zones/UNSPECIFIED/disks/vol1") on node "mynode-1" 
I0624 08:26:11.320255  116452 reconciler.go:295] attacherDetacher.AttachVolume started for volume "pd.csi.storage.gke.io-pdName" (UniqueName: "kubernetes.io/csi/pd.csi.storage.gke.io^projects/UNSPECIFIED/zones/UNSPECIFIED/disks/pdName") from node "mynode" 
E0624 08:26:11.320297  116452 csi_attacher.go:455] [attachment.ID=csi-77bd5b88d32d48e5a31f0073211a7dd54ed0066f008e12364f4b6ecb61dc7370] watch channel had been closed
==================
WARNING: DATA RACE
Write at 0x00c0004ec138 by goroutine 185:
  k8s.io/kubernetes/pkg/controller/volume/attachdetach/testing.CreateTestClient.func7()
      pkg/controller/volume/attachdetach/testing/testvolumespec.go:185 +0x207
  k8s.io/kubernetes/vendor/k8s.io/client-go/testing.(*SimpleReactor).React()
      staging/src/k8s.io/client-go/testing/fixture.go:521 +0x68
  k8s.io/kubernetes/vendor/k8s.io/client-go/testing.(*Fake).Invokes()
      staging/src/k8s.io/client-go/testing/fake.go:141 +0x327
  k8s.io/kubernetes/vendor/k8s.io/client-go/kubernetes/typed/storage/v1/fake.(*FakeVolumeAttachments).Create()
      staging/src/k8s.io/client-go/kubernetes/typed/storage/v1/fake/fake_volumeattachment.go:82 +0x246
  k8s.io/kubernetes/pkg/volume/csi.(*csiAttacher).Attach()
      pkg/volume/csi/csi_attacher.go:109 +0x5f3
  k8s.io/kubernetes/pkg/volume/util/operationexecutor.(*operationGenerator).GenerateAttachVolumeFunc.func1()
      pkg/volume/util/operationexecutor/operation_generator.go:335 +0x275
  k8s.io/kubernetes/pkg/volume/util/types.(*GeneratedOperations).Run()
      pkg/volume/util/types/types.go:54 +0x142
  k8s.io/kubernetes/pkg/volume/util/nestedpendingoperations.(*nestedPendingOperations).Run.func1()
      pkg/volume/util/nestedpendingoperations/nestedpendingoperations.go:183 +0x159

Previous read at 0x00c0004ec138 by goroutine 27:
  k8s.io/kubernetes/pkg/controller/volume/attachdetach/testing.CreateTestClient.func6()
      pkg/controller/volume/attachdetach/testing/testvolumespec.go:179 +0x104
  k8s.io/kubernetes/vendor/k8s.io/client-go/testing.(*SimpleReactor).React()
      staging/src/k8s.io/client-go/testing/fixture.go:521 +0x68
  k8s.io/kubernetes/vendor/k8s.io/client-go/testing.(*Fake).Invokes()
      staging/src/k8s.io/client-go/testing/fake.go:141 +0x327
  k8s.io/kubernetes/vendor/k8s.io/client-go/kubernetes/typed/storage/v1/fake.(*FakeVolumeAttachments).List()
      staging/src/k8s.io/client-go/kubernetes/typed/storage/v1/fake/fake_volumeattachment.go:55 +0x273
  k8s.io/kubernetes/vendor/k8s.io/client-go/informers/storage/v1.NewFilteredVolumeAttachmentInformer.func1()
      staging/src/k8s.io/client-go/informers/storage/v1/volumeattachment.go:64 +0x23c
  k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.(*ListWatch).List()
      staging/src/k8s.io/client-go/tools/cache/listwatch.go:106 +0x97
  k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch.func1.1.2()
      staging/src/k8s.io/client-go/tools/cache/reflector.go:277 +0xf4
  k8s.io/kubernetes/vendor/k8s.io/client-go/tools/pager.SimplePageFunc.func1()
      staging/src/k8s.io/client-go/tools/pager/pager.go:40 +0x97
  k8s.io/kubernetes/vendor/k8s.io/client-go/tools/pager.(*ListPager).List()
      staging/src/k8s.io/client-go/tools/pager/pager.go:91 +0x1e2
  k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch.func1.1()
      staging/src/k8s.io/client-go/tools/cache/reflector.go:302 +0x2b7

Goroutine 185 (running) created at:
  k8s.io/kubernetes/pkg/volume/util/nestedpendingoperations.(*nestedPendingOperations).Run()
      pkg/volume/util/nestedpendingoperations/nestedpendingoperations.go:178 +0x5b2
  k8s.io/kubernetes/pkg/volume/util/operationexecutor.(*operationExecutor).AttachVolume()
      pkg/volume/util/operationexecutor/operation_executor.go:675 +0x348
E0624 08:26:11.320776  116452 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/csi/pd.csi.storage.gke.io^projects/UNSPECIFIED/zones/UNSPECIFIED/disks/vol1 podName: nodeName:mynode-1}" failed. No retries permitted until 2021-06-24 08:26:11.820552519 +0000 UTC m=+12.559779409 (durationBeforeRetry 500ms). Error: "DetachVolume.Detach failed for volume \"nil\" (UniqueName: \"kubernetes.io/csi/pd.csi.storage.gke.io^projects/UNSPECIFIED/zones/UNSPECIFIED/disks/vol1\") on node \"mynode-1\" : volume attachment watch channel had been closed"
  k8s.io/kubernetes/pkg/controller/volume/attachdetach/reconciler.(*reconciler).attachDesiredVolumes()
      pkg/controller/volume/attachdetach/reconciler/reconciler.go:293 +0x639
  k8s.io/kubernetes/pkg/controller/volume/attachdetach/reconciler.(*reconciler).reconcile()
      pkg/controller/volume/attachdetach/reconciler/reconciler.go:233 +0x1b9
  k8s.io/kubernetes/pkg/controller/volume/attachdetach/reconciler.(*reconciler).reconciliationLoopFunc.func1()
      pkg/controller/volume/attachdetach/reconciler/reconciler.go:110 +0x5e
  k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1()
      staging/src/k8s.io/apimachinery/pkg/util/wait/wait.go:155 +0x75
  k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.BackoffUntil()
      staging/src/k8s.io/apimachinery/pkg/util/wait/wait.go:156 +0xba
  k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil()
      staging/src/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x114
  k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.Until()
      staging/src/k8s.io/apimachinery/pkg/util/wait/wait.go:90 +0x89
  k8s.io/kubernetes/pkg/controller/volume/attachdetach/reconciler.(*reconciler).Run()
      pkg/controller/volume/attachdetach/reconciler/reconciler.go:101 +0x2f

Goroutine 27 (finished) created at:
  k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch.func1()
      staging/src/k8s.io/client-go/tools/cache/reflector.go:268 +0x36f
  k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch()
      staging/src/k8s.io/client-go/tools/cache/reflector.go:360 +0x2e4
  k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.(*Reflector).Run.func1()
      staging/src/k8s.io/client-go/tools/cache/reflector.go:221 +0x53
  k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1()
      staging/src/k8s.io/apimachinery/pkg/util/wait/wait.go:155 +0x75
  k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.BackoffUntil()
      staging/src/k8s.io/apimachinery/pkg/util/wait/wait.go:156 +0xba
  k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.(*Reflector).Run()
      staging/src/k8s.io/client-go/tools/cache/reflector.go:220 +0x2c4
  k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.(*Reflector).Run-fm()
      staging/src/k8s.io/client-go/tools/cache/reflector.go:218 +0x4b
  k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).StartWithChannel.func1()
      staging/src/k8s.io/apimachinery/pkg/util/wait/wait.go:56 +0x45
  k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
      staging/src/k8s.io/apimachinery/pkg/util/wait/wait.go:73 +0x6d
==================
E0624 08:26:11.321044  116452 csi_attacher.go:455] [attachment.ID=csi-4ff6fd4b10c02f804e7435dffeb62054338b590749cdaacf18e198ee0ab6dd66] watch channel had been closed
I0624 08:26:11.321736  116452 event.go:291] "Event occurred" object="mynamespace/mypod-3" kind="Pod" apiVersion="v1" type="Warning" reason="FailedAttachVolume" message="AttachVolume.Attach failed for volume \"pd.csi.storage.gke.io-pdName\" : volume attachment watch channel had been closed"
E0624 08:26:11.321980  116452 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/csi/pd.csi.storage.gke.io^projects/UNSPECIFIED/zones/UNSPECIFIED/disks/pdName podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:26:11.82181845 +0000 UTC m=+12.561045340 (durationBeforeRetry 500ms). Error: "AttachVolume.Attach failed for volume \"pd.csi.storage.gke.io-pdName\" (UniqueName: \"kubernetes.io/csi/pd.csi.storage.gke.io^projects/UNSPECIFIED/zones/UNSPECIFIED/disks/pdName\") from node \"mynode\" : volume attachment watch channel had been closed"
I0624 08:26:11.321974  116452 event.go:291] "Event occurred" object="mynamespace/mypod-4" kind="Pod" apiVersion="v1" type="Warning" reason="FailedAttachVolume" message="AttachVolume.Attach failed for volume \"pd.csi.storage.gke.io-pdName\" : volume attachment watch channel had been closed"
I0624 08:26:11.322164  116452 event.go:291] "Event occurred" object="mynamespace/mypod-0" kind="Pod" apiVersion="v1" type="Warning" reason="FailedAttachVolume" message="AttachVolume.Attach failed for volume \"pd.csi.storage.gke.io-pdName\" : volume attachment watch channel had been closed"
I0624 08:26:11.322302  116452 event.go:291] "Event occurred" object="mynamespace/mypod-1" kind="Pod" apiVersion="v1" type="Warning" reason="FailedAttachVolume" message="AttachVolume.Attach failed for volume \"pd.csi.storage.gke.io-pdName\" : volume attachment watch channel had been closed"
I0624 08:26:11.322433  116452 event.go:291] "Event occurred" object="mynamespace/mypod-2" kind="Pod" apiVersion="v1" type="Warning" reason="FailedAttachVolume" message="AttachVolume.Attach failed for volume \"pd.csi.storage.gke.io-pdName\" : volume attachment watch channel had been closed"
I0624 08:26:11.415155  116452 shared_informer.go:247] Caches are synced for attach detach 
W0624 08:26:11.417822  116452 reflector.go:436] k8s.io/client-go/informers/factory.go:134: watch of *v1.VolumeAttachment ended with: very short watch: k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
W0624 08:26:11.418033  116452 reflector.go:436] k8s.io/client-go/informers/factory.go:134: watch of *v1.PersistentVolume ended with: very short watch: k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
W0624 08:26:11.418086  116452 reflector.go:436] k8s.io/client-go/informers/factory.go:134: watch of *v1.Node ended with: very short watch: k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
W0624 08:26:11.418551  116452 reflector.go:436] k8s.io/client-go/informers/factory.go:134: watch of *v1.Pod ended with: very short watch: k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
--- FAIL: Test_ADC_VolumeAttachmentRecovery (3.55s)
    --- FAIL: Test_ADC_VolumeAttachmentRecovery/Deleted_Pod_with_migrated_PV (0.11s)
        testing.go:1038: race detected during execution of test
    testing.go:1038: race detected during execution of test
FAIL

				from junit_bazel.xml

Filter through log files | View test history on testgrid


//pkg/controller/volume/attachdetach/go_default_test:run_2_of_2 0.00s

bazel test //pkg/controller/volume/attachdetach/go_default_test:run_2_of_2
exec ${PAGER:-/usr/bin/less} "$0" || exit 1
Executing tests from //pkg/controller/volume/attachdetach:go_default_test
-----------------------------------------------------------------------------
I0624 08:25:59.444478  116452 shared_informer.go:240] Waiting for caches to sync for attach detach
I0624 08:25:59.548002  116452 shared_informer.go:247] Caches are synced for attach detach 
I0624 08:25:59.558721  116452 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0624 08:25:59.559963  116452 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0624 08:25:59.560073  116452 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0624 08:25:59.560622  116452 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0624 08:25:59.560718  116452 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0624 08:25:59.560846  116452 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0624 08:25:59.562278  116452 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0624 08:25:59.562403  116452 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0624 08:25:59.567847  116452 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0624 08:25:59.569246  116452 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0624 08:25:59.569387  116452 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0624 08:25:59.569489  116452 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0624 08:25:59.570556  116452 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0624 08:25:59.570630  116452 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0624 08:25:59.570746  116452 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0624 08:25:59.570916  116452 reconciler.go:295] attacherDetacher.AttachVolume started for volume "volumeName" (UniqueName: "kubernetes.io/testPlugin/volumeName") from node "mynode" 
I0624 08:25:59.572186  116452 operation_generator.go:360] AttachVolume.Attach succeeded for volume "volumeName" (UniqueName: "kubernetes.io/testPlugin/volumeName") from node "mynode" 
I0624 08:25:59.572654  116452 event.go:291] "Event occurred" object="mynamespace/mypod-0" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:25:59.572845  116452 event.go:291] "Event occurred" object="mynamespace/mypod-1" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:25:59.573022  116452 event.go:291] "Event occurred" object="mynamespace/mypod-2" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:25:59.573204  116452 event.go:291] "Event occurred" object="mynamespace/mypod-3" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:25:59.573371  116452 event.go:291] "Event occurred" object="mynamespace/mypod-4" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:01.552742  116452 shared_informer.go:240] Waiting for caches to sync for attach detach
I0624 08:26:01.656153  116452 shared_informer.go:247] Caches are synced for attach detach 
I0624 08:26:01.664395  116452 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0624 08:26:01.664532  116452 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0624 08:26:01.664731  116452 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0624 08:26:01.666582  116452 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0624 08:26:01.666699  116452 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0624 08:26:01.666856  116452 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0624 08:26:01.668315  116452 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0624 08:26:01.668448  116452 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0624 08:26:01.668595  116452 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0624 08:26:01.670032  116452 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0624 08:26:01.670143  116452 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0624 08:26:01.670381  116452 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0624 08:26:01.682597  116452 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0624 08:26:01.682984  116452 reconciler.go:295] attacherDetacher.AttachVolume started for volume "volumeName2" (UniqueName: "kubernetes.io/testPlugin/volumeName2") from node "mynode-1" 
I0624 08:26:01.683280  116452 reconciler.go:295] attacherDetacher.AttachVolume started for volume "volumeName" (UniqueName: "kubernetes.io/testPlugin/volumeName") from node "mynode" 
I0624 08:26:01.683483  116452 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0624 08:26:01.683606  116452 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0624 08:26:01.684120  116452 operation_generator.go:360] AttachVolume.Attach succeeded for volume "volumeName2" (UniqueName: "kubernetes.io/testPlugin/volumeName2") from node "mynode-1" 
I0624 08:26:01.685359  116452 operation_generator.go:360] AttachVolume.Attach succeeded for volume "volumeName" (UniqueName: "kubernetes.io/testPlugin/volumeName") from node "mynode" 
I0624 08:26:01.685726  116452 event.go:291] "Event occurred" object="mynamespace/newpod-1" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName2\" "
I0624 08:26:01.686481  116452 event.go:291] "Event occurred" object="mynamespace/mypod-1" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:01.686705  116452 event.go:291] "Event occurred" object="mynamespace/mypod-2" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:01.686829  116452 event.go:291] "Event occurred" object="mynamespace/mypod-3" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:01.686960  116452 event.go:291] "Event occurred" object="mynamespace/mypod-4" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:01.687080  116452 event.go:291] "Event occurred" object="mynamespace/mypod-0" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:03.660976  116452 shared_informer.go:240] Waiting for caches to sync for attach detach
I0624 08:26:03.761576  116452 shared_informer.go:247] Caches are synced for attach detach 
I0624 08:26:03.768417  116452 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0624 08:26:03.768572  116452 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0624 08:26:03.768885  116452 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0624 08:26:03.770697  116452 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0624 08:26:03.770819  116452 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0624 08:26:03.771092  116452 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0624 08:26:03.773008  116452 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0624 08:26:03.773133  116452 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0624 08:26:03.773350  116452 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0624 08:26:03.774782  116452 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0624 08:26:03.774882  116452 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0624 08:26:03.775028  116452 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0624 08:26:03.800672  116452 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0624 08:26:03.801086  116452 reconciler.go:295] attacherDetacher.AttachVolume started for volume "volumeName" (UniqueName: "kubernetes.io/testPlugin/volumeName") from node "mynode" 
I0624 08:26:03.801328  116452 reconciler.go:295] attacherDetacher.AttachVolume started for volume "volumeName2" (UniqueName: "kubernetes.io/testPlugin/volumeName2") from node "mynode-1" 
I0624 08:26:03.801341  116452 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0624 08:26:03.801480  116452 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0624 08:26:03.801821  116452 operation_generator.go:360] AttachVolume.Attach succeeded for volume "volumeName2" (UniqueName: "kubernetes.io/testPlugin/volumeName2") from node "mynode-1" 
I0624 08:26:03.802125  116452 operation_generator.go:360] AttachVolume.Attach succeeded for volume "volumeName" (UniqueName: "kubernetes.io/testPlugin/volumeName") from node "mynode" 
I0624 08:26:03.802489  116452 event.go:291] "Event occurred" object="mynamespace/mypod-1" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:03.802739  116452 event.go:291] "Event occurred" object="mynamespace/newpod-1" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName2\" "
I0624 08:26:03.802913  116452 event.go:291] "Event occurred" object="mynamespace/mypod-2" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:03.803088  116452 event.go:291] "Event occurred" object="mynamespace/mypod-0" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:03.803251  116452 event.go:291] "Event occurred" object="mynamespace/mypod-3" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:03.803454  116452 event.go:291] "Event occurred" object="mynamespace/mypod-4" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:05.767789  116452 shared_informer.go:240] Waiting for caches to sync for attach detach
I0624 08:26:05.868251  116452 shared_informer.go:247] Caches are synced for attach detach 
I0624 08:26:05.875347  116452 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0624 08:26:05.875532  116452 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0624 08:26:05.875681  116452 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0624 08:26:05.876602  116452 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0624 08:26:05.876786  116452 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0624 08:26:05.876917  116452 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0624 08:26:05.877971  116452 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0624 08:26:05.878122  116452 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0624 08:26:05.878226  116452 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0624 08:26:05.879239  116452 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0624 08:26:05.879319  116452 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0624 08:26:05.879437  116452 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0624 08:26:05.880461  116452 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0624 08:26:05.880655  116452 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0624 08:26:05.880713  116452 reconciler.go:295] attacherDetacher.AttachVolume started for volume "volumeName" (UniqueName: "kubernetes.io/testPlugin/volumeName") from node "mynode" 
I0624 08:26:05.880764  116452 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0624 08:26:05.880945  116452 reconciler.go:295] attacherDetacher.AttachVolume started for volume "volumeName2" (UniqueName: "kubernetes.io/testPlugin/volumeName2") from node "mynode-1" 
I0624 08:26:05.881364  116452 operation_generator.go:360] AttachVolume.Attach succeeded for volume "volumeName2" (UniqueName: "kubernetes.io/testPlugin/volumeName2") from node "mynode-1" 
I0624 08:26:05.881461  116452 event.go:291] "Event occurred" object="mynamespace/mypod-2" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:05.881486  116452 operation_generator.go:360] AttachVolume.Attach succeeded for volume "volumeName" (UniqueName: "kubernetes.io/testPlugin/volumeName") from node "mynode" 
I0624 08:26:05.881829  116452 event.go:291] "Event occurred" object="mynamespace/mypod-1" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:05.882040  116452 event.go:291] "Event occurred" object="mynamespace/newpod-1" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName2\" "
I0624 08:26:05.882217  116452 event.go:291] "Event occurred" object="mynamespace/mypod-4" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:05.882385  116452 event.go:291] "Event occurred" object="mynamespace/mypod-0" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:05.882557  116452 event.go:291] "Event occurred" object="mynamespace/mypod-3" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:05.896007  116452 reconciler.go:295] attacherDetacher.AttachVolume started for volume "volumeName3" (UniqueName: "kubernetes.io/testPlugin/volumeName3") from node "mynode-1" 
I0624 08:26:05.896246  116452 operation_generator.go:360] AttachVolume.Attach succeeded for volume "volumeName3" (UniqueName: "kubernetes.io/testPlugin/volumeName3") from node "mynode-1" 
I0624 08:26:05.896655  116452 event.go:291] "Event occurred" object="mynamespace/newpod-2" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName3\" "
I0624 08:26:07.874003  116452 shared_informer.go:240] Waiting for caches to sync for attach detach
I0624 08:26:07.974629  116452 shared_informer.go:247] Caches are synced for attach detach 
I0624 08:26:07.981055  116452 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0624 08:26:07.982210  116452 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0624 08:26:07.982889  116452 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0624 08:26:07.983027  116452 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0624 08:26:07.983329  116452 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0624 08:26:07.983463  116452 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0624 08:26:07.983939  116452 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0624 08:26:07.984090  116452 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0624 08:26:07.984198  116452 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0624 08:26:07.984959  116452 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0624 08:26:07.985128  116452 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0624 08:26:07.985266  116452 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0624 08:26:07.986040  116452 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0624 08:26:07.986125  116452 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0624 08:26:07.986252  116452 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0624 08:26:07.986294  116452 reconciler.go:295] attacherDetacher.AttachVolume started for volume "volumeName" (UniqueName: "kubernetes.io/testPlugin/volumeName") from node "mynode" 
I0624 08:26:07.986520  116452 reconciler.go:295] attacherDetacher.AttachVolume started for volume "vol1" (UniqueName: "kubernetes.io/testPlugin/vol1") from node "mynode-1" 
I0624 08:26:07.986838  116452 operation_generator.go:360] AttachVolume.Attach succeeded for volume "vol1" (UniqueName: "kubernetes.io/testPlugin/vol1") from node "mynode-1" 
I0624 08:26:07.986981  116452 event.go:291] "Event occurred" object="mynamespace/mypod-0" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:07.987195  116452 event.go:291] "Event occurred" object="mynamespace/pod1" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"vol1\" "
I0624 08:26:07.987294  116452 operation_generator.go:360] AttachVolume.Attach succeeded for volume "volumeName" (UniqueName: "kubernetes.io/testPlugin/volumeName") from node "mynode" 
I0624 08:26:07.987335  116452 event.go:291] "Event occurred" object="mynamespace/mypod-1" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:07.987524  116452 event.go:291] "Event occurred" object="mynamespace/mypod-2" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:07.987633  116452 event.go:291] "Event occurred" object="mynamespace/mypod-3" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:07.987797  116452 event.go:291] "Event occurred" object="mynamespace/mypod-4" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:08.980732  116452 shared_informer.go:240] Waiting for caches to sync for attach detach
I0624 08:26:09.081316  116452 shared_informer.go:247] Caches are synced for attach detach 
I0624 08:26:09.088128  116452 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0624 08:26:09.088307  116452 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0624 08:26:09.088441  116452 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0624 08:26:09.089395  116452 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0624 08:26:09.089509  116452 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0624 08:26:09.089626  116452 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0624 08:26:09.090716  116452 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0624 08:26:09.090812  116452 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0624 08:26:09.090930  116452 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0624 08:26:09.092189  116452 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0624 08:26:09.092308  116452 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0624 08:26:09.092477  116452 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0624 08:26:09.093830  116452 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0624 08:26:09.093906  116452 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0624 08:26:09.093997  116452 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0624 08:26:09.094193  116452 reconciler.go:295] attacherDetacher.AttachVolume started for volume "volumeName" (UniqueName: "kubernetes.io/testPlugin/volumeName") from node "mynode" 
I0624 08:26:09.094858  116452 event.go:291] "Event occurred" object="mynamespace/mypod-3" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:09.094982  116452 operation_generator.go:360] AttachVolume.Attach succeeded for volume "volumeName" (UniqueName: "kubernetes.io/testPlugin/volumeName") from node "mynode" 
I0624 08:26:09.095085  116452 event.go:291] "Event occurred" object="mynamespace/mypod-4" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:09.095275  116452 event.go:291] "Event occurred" object="mynamespace/mypod-0" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:09.095465  116452 event.go:291] "Event occurred" object="mynamespace/mypod-1" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:09.095668  116452 event.go:291] "Event occurred" object="mynamespace/mypod-2" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:09.998768  116452 reconciler.go:219] attacherDetacher.DetachVolume started for volume "pv1" (UniqueName: "kubernetes.io/testPlugin/vol1") on node "mynode-1" 
I0624 08:26:09.998988  116452 operation_generator.go:1409] Verified volume is safe to detach for volume "pv1" (UniqueName: "kubernetes.io/testPlugin/vol1") on node "mynode-1" 
I0624 08:26:09.999100  116452 operation_generator.go:470] DetachVolume.Detach succeeded for volume "pv1" (UniqueName: "kubernetes.io/testPlugin/vol1") on node "mynode-1" 
I0624 08:26:10.086567  116452 shared_informer.go:240] Waiting for caches to sync for attach detach
I0624 08:26:10.189364  116452 shared_informer.go:247] Caches are synced for attach detach 
I0624 08:26:10.196341  116452 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0624 08:26:10.196468  116452 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0624 08:26:10.196658  116452 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0624 08:26:10.198016  116452 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0624 08:26:10.198153  116452 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0624 08:26:10.197824  116452 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0624 08:26:10.201154  116452 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0624 08:26:10.201259  116452 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0624 08:26:10.202683  116452 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0624 08:26:10.203932  116452 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0624 08:26:10.204055  116452 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0624 08:26:10.204173  116452 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0624 08:26:10.205179  116452 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0624 08:26:10.205385  116452 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0624 08:26:10.205485  116452 reconciler.go:295] attacherDetacher.AttachVolume started for volume "volumeName" (UniqueName: "kubernetes.io/testPlugin/volumeName") from node "mynode" 
I0624 08:26:10.205511  116452 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0624 08:26:10.206229  116452 event.go:291] "Event occurred" object="mynamespace/mypod-1" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:10.206787  116452 event.go:291] "Event occurred" object="mynamespace/mypod-2" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:10.206896  116452 operation_generator.go:360] AttachVolume.Attach succeeded for volume "volumeName" (UniqueName: "kubernetes.io/testPlugin/volumeName") from node "mynode" 
I0624 08:26:10.207025  116452 event.go:291] "Event occurred" object="mynamespace/mypod-3" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:10.207236  116452 event.go:291] "Event occurred" object="mynamespace/mypod-0" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:10.207536  116452 event.go:291] "Event occurred" object="mynamespace/mypod-4" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0624 08:26:11.112615  116452 reconciler.go:219] attacherDetacher.DetachVolume started for volume "pv1" (UniqueName: "kubernetes.io/testPlugin/vol1") on node "mynode-1" 
I0624 08:26:11.112752  116452 operation_generator.go:1409] Verified volume is safe to detach for volume "pv1" (UniqueName: "kubernetes.io/testPlugin/vol1") on node "mynode-1" 
I0624 08:26:11.112868  116452 operation_generator.go:470] DetachVolume.Detach succeeded for volume "pv1" (UniqueName: "kubernetes.io/testPlugin/vol1") on node "mynode-1" 
I0624 08:26:11.200897  116452 shared_informer.go:240] Waiting for caches to sync for attach detach
I0624 08:26:11.302912  116452 shared_informer.go:247] Caches are synced for attach detach 
W0624 08:26:11.304996  116452 reflector.go:436] k8s.io/client-go/informers/factory.go:134: watch of *v1.CSIDriver ended with: very short watch: k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
W0624 08:26:11.305685  116452 reflector.go:436] k8s.io/client-go/informers/factory.go:134: watch of *v1.Pod ended with: very short watch: k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
E0624 08:26:11.311777  116452 reconciler.go:228] attacherDetacher.DetachVolume failed to start for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" : DetachVolume.FindAttachablePluginByName failed for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" : no volume plugin matched name: kubernetes.io/testPlugin
I0624 08:26:11.314258  116452 shared_informer.go:240] Waiting for caches to sync for attach detach
E0624 08:26:11.314314  116452 reconciler.go:228] attacherDetacher.DetachVolume failed to start for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" : DetachVolume.FindAttachablePluginByName failed for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" : no volume plugin matched name: kubernetes.io/testPlugin
E0624 08:26:11.316281  116452 reconciler.go:228] attacherDetacher.DetachVolume failed to start for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" : DetachVolume.FindAttachablePluginByName failed for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" : no volume plugin matched name: kubernetes.io/testPlugin
E0624 08:26:11.318247  116452 reconciler.go:228] attacherDetacher.DetachVolume failed to start for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" : DetachVolume.FindAttachablePluginByName failed for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" : no volume plugin matched name: kubernetes.io/testPlugin
I0624 08:26:11.319678  116452 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/csi/pd.csi.storage.gke.io^projects/UNSPECIFIED/zones/UNSPECIFIED/disks/vol1") on node "mynode-1" 
I0624 08:26:11.319919  116452 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/csi/pd.csi.storage.gke.io^projects/UNSPECIFIED/zones/UNSPECIFIED/disks/vol1") on node "mynode-1" 
I0624 08:26:11.320255  116452 reconciler.go:295] attacherDetacher.AttachVolume started for volume "pd.csi.storage.gke.io-pdName" (UniqueName: "kubernetes.io/csi/pd.csi.storage.gke.io^projects/UNSPECIFIED/zones/UNSPECIFIED/disks/pdName") from node "mynode" 
E0624 08:26:11.320297  116452 csi_attacher.go:455] [attachment.ID=csi-77bd5b88d32d48e5a31f0073211a7dd54ed0066f008e12364f4b6ecb61dc7370] watch channel had been closed
==================
WARNING: DATA RACE
Write at 0x00c0004ec138 by goroutine 185:
  k8s.io/kubernetes/pkg/controller/volume/attachdetach/testing.CreateTestClient.func7()
      pkg/controller/volume/attachdetach/testing/testvolumespec.go:185 +0x207
  k8s.io/kubernetes/vendor/k8s.io/client-go/testing.(*SimpleReactor).React()
      staging/src/k8s.io/client-go/testing/fixture.go:521 +0x68
  k8s.io/kubernetes/vendor/k8s.io/client-go/testing.(*Fake).Invokes()
      staging/src/k8s.io/client-go/testing/fake.go:141 +0x327
  k8s.io/kubernetes/vendor/k8s.io/client-go/kubernetes/typed/storage/v1/fake.(*FakeVolumeAttachments).Create()
      staging/src/k8s.io/client-go/kubernetes/typed/storage/v1/fake/fake_volumeattachment.go:82 +0x246
  k8s.io/kubernetes/pkg/volume/csi.(*csiAttacher).Attach()
      pkg/volume/csi/csi_attacher.go:109 +0x5f3
  k8s.io/kubernetes/pkg/volume/util/operationexecutor.(*operationGenerator).GenerateAttachVolumeFunc.func1()
      pkg/volume/util/operationexecutor/operation_generator.go:335 +0x275
  k8s.io/kubernetes/pkg/volume/util/types.(*GeneratedOperations).Run()
      pkg/volume/util/types/types.go:54 +0x142
  k8s.io/kubernetes/pkg/volume/util/nestedpendingoperations.(*nestedPendingOperations).Run.func1()
      pkg/volume/util/nestedpendingoperations/nestedpendingoperations.go:183 +0x159

Previous read at 0x00c0004ec138 by goroutine 27:
  k8s.io/kubernetes/pkg/controller/volume/attachdetach/testing.CreateTestClient.func6()
      pkg/controller/volume/attachdetach/testing/testvolumespec.go:179 +0x104
  k8s.io/kubernetes/vendor/k8s.io/client-go/testing.(*SimpleReactor).React()
      staging/src/k8s.io/client-go/testing/fixture.go:521 +0x68
  k8s.io/kubernetes/vendor/k8s.io/client-go/testing.(*Fake).Invokes()
      staging/src/k8s.io/client-go/testing/fake.go:141 +0x327
  k8s.io/kubernetes/vendor/k8s.io/client-go/kubernetes/typed/storage/v1/fake.(*FakeVolumeAttachments).List()
      staging/src/k8s.io/client-go/kubernetes/typed/storage/v1/fake/fake_volumeattachment.go:55 +0x273
  k8s.io/kubernetes/vendor/k8s.io/client-go/informers/storage/v1.NewFilteredVolumeAttachmentInformer.func1()
      staging/src/k8s.io/client-go/informers/storage/v1/volumeattachment.go:64 +0x23c
  k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.(*ListWatch).List()
      staging/src/k8s.io/client-go/tools/cache/listwatch.go:106 +0x97
  k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch.func1.1.2()
      staging/src/k8s.io/client-go/tools/cache/reflector.go:277 +0xf4
  k8s.io/kubernetes/vendor/k8s.io/client-go/tools/pager.SimplePageFunc.func1()
      staging/src/k8s.io/client-go/tools/pager/pager.go:40 +0x97
  k8s.io/kubernetes/vendor/k8s.io/client-go/tools/pager.(*ListPager).List()
      staging/src/k8s.io/client-go/tools/pager/pager.go:91 +0x1e2
  k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch.func1.1()
      staging/src/k8s.io/client-go/tools/cache/reflector.go:302 +0x2b7

Goroutine 185 (running) created at:
  k8s.io/kubernetes/pkg/volume/util/nestedpendingoperations.(*nestedPendingOperations).Run()
      pkg/volume/util/nestedpendingoperations/nestedpendingoperations.go:178 +0x5b2
  k8s.io/kubernetes/pkg/volume/util/operationexecutor.(*operationExecutor).AttachVolume()
      pkg/volume/util/operationexecutor/operation_executor.go:675 +0x348
E0624 08:26:11.320776  116452 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/csi/pd.csi.storage.gke.io^projects/UNSPECIFIED/zones/UNSPECIFIED/disks/vol1 podName: nodeName:mynode-1}" failed. No retries permitted until 2021-06-24 08:26:11.820552519 +0000 UTC m=+12.559779409 (durationBeforeRetry 500ms). Error: "DetachVolume.Detach failed for volume \"nil\" (UniqueName: \"kubernetes.io/csi/pd.csi.storage.gke.io^projects/UNSPECIFIED/zones/UNSPECIFIED/disks/vol1\") on node \"mynode-1\" : volume attachment watch channel had been closed"
  k8s.io/kubernetes/pkg/controller/volume/attachdetach/reconciler.(*reconciler).attachDesiredVolumes()
      pkg/controller/volume/attachdetach/reconciler/reconciler.go:293 +0x639
  k8s.io/kubernetes/pkg/controller/volume/attachdetach/reconciler.(*reconciler).reconcile()
      pkg/controller/volume/attachdetach/reconciler/reconciler.go:233 +0x1b9
  k8s.io/kubernetes/pkg/controller/volume/attachdetach/reconciler.(*reconciler).reconciliationLoopFunc.func1()
      pkg/controller/volume/attachdetach/reconciler/reconciler.go:110 +0x5e
  k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1()
      staging/src/k8s.io/apimachinery/pkg/util/wait/wait.go:155 +0x75
  k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.BackoffUntil()
      staging/src/k8s.io/apimachinery/pkg/util/wait/wait.go:156 +0xba
  k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil()
      staging/src/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x114
  k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.Until()
      staging/src/k8s.io/apimachinery/pkg/util/wait/wait.go:90 +0x89
  k8s.io/kubernetes/pkg/controller/volume/attachdetach/reconciler.(*reconciler).Run()
      pkg/controller/volume/attachdetach/reconciler/reconciler.go:101 +0x2f

Goroutine 27 (finished) created at:
  k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch.func1()
      staging/src/k8s.io/client-go/tools/cache/reflector.go:268 +0x36f
  k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch()
      staging/src/k8s.io/client-go/tools/cache/reflector.go:360 +0x2e4
  k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.(*Reflector).Run.func1()
      staging/src/k8s.io/client-go/tools/cache/reflector.go:221 +0x53
  k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1()
      staging/src/k8s.io/apimachinery/pkg/util/wait/wait.go:155 +0x75
  k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.BackoffUntil()
      staging/src/k8s.io/apimachinery/pkg/util/wait/wait.go:156 +0xba
  k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.(*Reflector).Run()
      staging/src/k8s.io/client-go/tools/cache/reflector.go:220 +0x2c4
  k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.(*Reflector).Run-fm()
      staging/src/k8s.io/client-go/tools/cache/reflector.go:218 +0x4b
  k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).StartWithChannel.func1()
      staging/src/k8s.io/apimachinery/pkg/util/wait/wait.go:56 +0x45
  k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
      staging/src/k8s.io/apimachinery/pkg/util/wait/wait.go:73 +0x6d
==================
E0624 08:26:11.321044  116452 csi_attacher.go:455] [attachment.ID=csi-4ff6fd4b10c02f804e7435dffeb62054338b590749cdaacf18e198ee0ab6dd66] watch channel had been closed
I0624 08:26:11.321736  116452 event.go:291] "Event occurred" object="mynamespace/mypod-3" kind="Pod" apiVersion="v1" type="Warning" reason="FailedAttachVolume" message="AttachVolume.Attach failed for volume \"pd.csi.storage.gke.io-pdName\" : volume attachment watch channel had been closed"
E0624 08:26:11.321980  116452 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/csi/pd.csi.storage.gke.io^projects/UNSPECIFIED/zones/UNSPECIFIED/disks/pdName podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:26:11.82181845 +0000 UTC m=+12.561045340 (durationBeforeRetry 500ms). Error: "AttachVolume.Attach failed for volume \"pd.csi.storage.gke.io-pdName\" (UniqueName: \"kubernetes.io/csi/pd.csi.storage.gke.io^projects/UNSPECIFIED/zones/UNSPECIFIED/disks/pdName\") from node \"mynode\" : volume attachment watch channel had been closed"
I0624 08:26:11.321974  116452 event.go:291] "Event occurred" object="mynamespace/mypod-4" kind="Pod" apiVersion="v1" type="Warning" reason="FailedAttachVolume" message="AttachVolume.Attach failed for volume \"pd.csi.storage.gke.io-pdName\" : volume attachment watch channel had been closed"
I0624 08:26:11.322164  116452 event.go:291] "Event occurred" object="mynamespace/mypod-0" kind="Pod" apiVersion="v1" type="Warning" reason="FailedAttachVolume" message="AttachVolume.Attach failed for volume \"pd.csi.storage.gke.io-pdName\" : volume attachment watch channel had been closed"
I0624 08:26:11.322302  116452 event.go:291] "Event occurred" object="mynamespace/mypod-1" kind="Pod" apiVersion="v1" type="Warning" reason="FailedAttachVolume" message="AttachVolume.Attach failed for volume \"pd.csi.storage.gke.io-pdName\" : volume attachment watch channel had been closed"
I0624 08:26:11.322433  116452 event.go:291] "Event occurred" object="mynamespace/mypod-2" kind="Pod" apiVersion="v1" type="Warning" reason="FailedAttachVolume" message="AttachVolume.Attach failed for volume \"pd.csi.storage.gke.io-pdName\" : volume attachment watch channel had been closed"
I0624 08:26:11.415155  116452 shared_informer.go:247] Caches are synced for attach detach 
W0624 08:26:11.417822  116452 reflector.go:436] k8s.io/client-go/informers/factory.go:134: watch of *v1.VolumeAttachment ended with: very short watch: k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
W0624 08:26:11.418033  116452 reflector.go:436] k8s.io/client-go/informers/factory.go:134: watch of *v1.PersistentVolume ended with: very short watch: k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
W0624 08:26:11.418086  116452 reflector.go:436] k8s.io/client-go/informers/factory.go:134: watch of *v1.Node ended with: very short watch: k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
W0624 08:26:11.418551  116452 reflector.go:436] k8s.io/client-go/informers/factory.go:134: watch of *v1.Pod ended with: very short watch: k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
--- FAIL: Test_ADC_VolumeAttachmentRecovery (3.55s)
    --- FAIL: Test_ADC_VolumeAttachmentRecovery/Deleted_Pod_with_migrated_PV (0.11s)
        testing.go:1038: race detected during execution of test
    testing.go:1038: race detected during execution of test
FAIL

				from junit_bazel.xml

Filter through log files | View test history on testgrid


//pkg/kubelet/volumemanager/reconciler/go_default_test:run_1_of_2 0.00s

bazel test //pkg/kubelet/volumemanager/reconciler/go_default_test:run_1_of_2
exec ${PAGER:-/usr/bin/less} "$0" || exit 1
Executing tests from //pkg/kubelet/volumemanager/reconciler:go_default_test
-----------------------------------------------------------------------------
I0624 08:35:37.400941  220043 reconciler.go:157] Reconciler: start to sync state
E0624 08:35:37.401110  220043 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0624 08:35:37.402020  220043 reconciler.go:244] operationExecutor.AttachVolume started for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") 
I0624 08:35:37.402094  220043 reconciler.go:157] Reconciler: start to sync state
E0624 08:35:37.402140  220043 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0624 08:35:37.402223  220043 operation_generator.go:360] AttachVolume.Attach succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") from node "mynodename" 
I0624 08:35:37.402907  220043 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") DevicePath "/dev/vdb-test"
I0624 08:35:37.403034  220043 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:35:37.403132  220043 operation_generator.go:594] MountVolume.MountDevice succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") device mount path ""
I0624 08:35:37.502773  220043 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") 
I0624 08:35:37.502850  220043 reconciler.go:157] Reconciler: start to sync state
E0624 08:35:37.502902  220043 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0624 08:35:37.503147  220043 operation_generator.go:1357] Controller attach succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") device path: "/fake/path"
I0624 08:35:37.503701  220043 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") DevicePath "/fake/path"
I0624 08:35:37.503849  220043 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:35:37.503952  220043 operation_generator.go:594] MountVolume.MountDevice succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") device mount path ""
I0624 08:35:37.504604  220043 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:35:37.504721  220043 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:35:37.605037  220043 reconciler.go:244] operationExecutor.AttachVolume started for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") 
I0624 08:35:37.605118  220043 reconciler.go:157] Reconciler: start to sync state
E0624 08:35:37.605178  220043 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0624 08:35:37.605531  220043 operation_generator.go:360] AttachVolume.Attach succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") from node "mynodename" 
I0624 08:35:37.606168  220043 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") DevicePath "/dev/vdb-test"
I0624 08:35:37.606284  220043 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:35:37.606393  220043 operation_generator.go:594] MountVolume.MountDevice succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") device mount path ""
I0624 08:35:37.705773  220043 reconciler.go:196] operationExecutor.UnmountVolume started for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1uid" (UID: "pod1uid") 
I0624 08:35:37.705979  220043 operation_generator.go:797] UnmountVolume.TearDown succeeded for volume "fake-plugin/fake-device1" (OuterVolumeSpecName: "volume-name") pod "pod1uid" (UID: "pod1uid"). InnerVolumeSpecName "volume-name". PluginName "fake-plugin", VolumeGidValue ""
I0624 08:35:37.706521  220043 reconciler.go:312] operationExecutor.UnmountDevice started for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") on node "mynodename" 
I0624 08:35:37.706753  220043 operation_generator.go:891] UnmountDevice succeeded for volume "volume-name" %!(EXTRA string=UnmountDevice succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") on node "mynodename" )
I0624 08:35:37.707181  220043 reconciler.go:333] operationExecutor.DetachVolume started for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") on node "mynodename" 
I0624 08:35:37.707304  220043 operation_generator.go:470] DetachVolume.Detach succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") on node "mynodename" 
I0624 08:35:37.813297  220043 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") 
I0624 08:35:37.813378  220043 reconciler.go:157] Reconciler: start to sync state
E0624 08:35:37.813439  220043 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0624 08:35:37.814037  220043 operation_generator.go:1357] Controller attach succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") device path: "/fake/path"
I0624 08:35:37.814806  220043 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") DevicePath "/fake/path"
I0624 08:35:37.814974  220043 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:35:37.815199  220043 operation_generator.go:594] MountVolume.MountDevice succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") device mount path ""
I0624 08:35:37.816062  220043 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:35:37.816271  220043 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:35:37.926641  220043 reconciler.go:196] operationExecutor.UnmountVolume started for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1uid" (UID: "pod1uid") 
I0624 08:35:37.926872  220043 operation_generator.go:797] UnmountVolume.TearDown succeeded for volume "fake-plugin/fake-device1" (OuterVolumeSpecName: "volume-name") pod "pod1uid" (UID: "pod1uid"). InnerVolumeSpecName "volume-name". PluginName "fake-plugin", VolumeGidValue ""
I0624 08:35:37.927331  220043 reconciler.go:312] operationExecutor.UnmountDevice started for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") on node "mynodename" 
I0624 08:35:37.927529  220043 operation_generator.go:891] UnmountDevice succeeded for volume "volume-name" %!(EXTRA string=UnmountDevice succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") on node "mynodename" )
I0624 08:35:37.927698  220043 reconciler.go:319] Volume detached for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") on node "mynodename" DevicePath "/dev/sdb"
I0624 08:35:38.027692  220043 reconciler.go:244] operationExecutor.AttachVolume started for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") 
I0624 08:35:38.027775  220043 reconciler.go:157] Reconciler: start to sync state
E0624 08:35:38.027824  220043 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0624 08:35:38.028100  220043 operation_generator.go:360] AttachVolume.Attach succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") from node "mynodename" 
I0624 08:35:38.028652  220043 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") DevicePath "/dev/vdb-test"
I0624 08:35:38.028762  220043 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:35:38.128592  220043 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") 
I0624 08:35:38.128659  220043 reconciler.go:157] Reconciler: start to sync state
E0624 08:35:38.128710  220043 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0624 08:35:38.128979  220043 operation_generator.go:1357] Controller attach succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") device path: "/fake/path"
I0624 08:35:38.129537  220043 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") DevicePath "/fake/path"
I0624 08:35:38.129662  220043 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:35:38.229200  220043 reconciler.go:244] operationExecutor.AttachVolume started for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") 
I0624 08:35:38.229268  220043 reconciler.go:157] Reconciler: start to sync state
E0624 08:35:38.229314  220043 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0624 08:35:38.229535  220043 operation_generator.go:360] AttachVolume.Attach succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") from node "mynodename" 
I0624 08:35:38.230124  220043 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") DevicePath "/dev/vdb-test"
I0624 08:35:38.230232  220043 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:35:38.335124  220043 reconciler.go:196] operationExecutor.UnmountVolume started for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1uid" (UID: "pod1uid") 
I0624 08:35:38.335434  220043 operation_generator.go:1170] UnmapVolume succeeded for volume "fake-plugin/fake-device1" (OuterVolumeSpecName: "volume-name") pod "pod1uid" (UID: "pod1uid"). InnerVolumeSpecName "volume-name". PluginName "fake-plugin", VolumeGidValue ""
I0624 08:35:38.335884  220043 reconciler.go:312] operationExecutor.UnmountDevice started for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") on node "mynodename" 
I0624 08:35:38.336063  220043 operation_generator.go:1282] UnmapDevice succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") on node "mynodename" 
I0624 08:35:38.336416  220043 reconciler.go:333] operationExecutor.DetachVolume started for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") on node "mynodename" 
I0624 08:35:38.336601  220043 operation_generator.go:470] DetachVolume.Detach succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") on node "mynodename" 
I0624 08:35:38.436617  220043 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") 
I0624 08:35:38.436697  220043 operation_generator.go:1357] Controller attach succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") device path: "/fake/path"
I0624 08:35:38.436952  220043 reconciler.go:157] Reconciler: start to sync state
E0624 08:35:38.437048  220043 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0624 08:35:38.437717  220043 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") DevicePath "/fake/path"
I0624 08:35:38.438027  220043 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:35:38.537542  220043 reconciler.go:196] operationExecutor.UnmountVolume started for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1uid" (UID: "pod1uid") 
I0624 08:35:38.537681  220043 operation_generator.go:1170] UnmapVolume succeeded for volume "fake-plugin/fake-device1" (OuterVolumeSpecName: "volume-name") pod "pod1uid" (UID: "pod1uid"). InnerVolumeSpecName "volume-name". PluginName "fake-plugin", VolumeGidValue ""
I0624 08:35:38.538079  220043 reconciler.go:312] operationExecutor.UnmountDevice started for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") on node "mynodename" 
I0624 08:35:38.538287  220043 operation_generator.go:1282] UnmapDevice succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") on node "mynodename" 
I0624 08:35:38.538432  220043 reconciler.go:319] Volume detached for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") on node "mynodename" DevicePath "/dev/sdb"
I0624 08:35:38.645594  220043 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "pv" (UniqueName: "fake-plugin/pv") pod "pod1" (UID: "pod1uid") 
I0624 08:35:38.645691  220043 reconciler.go:157] Reconciler: start to sync state
E0624 08:35:38.645743  220043 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0624 08:35:38.646270  220043 operation_generator.go:1357] Controller attach succeeded for volume "pv" (UniqueName: "fake-plugin/pv") pod "pod1" (UID: "pod1uid") device path: "fake/path"
I0624 08:35:38.646947  220043 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "pv" (UniqueName: "fake-plugin/pv") pod "pod1" (UID: "pod1uid") DevicePath "fake/path"
I0624 08:35:38.647054  220043 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "pv" (UniqueName: "fake-plugin/pv") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:35:38.647139  220043 operation_generator.go:594] MountVolume.MountDevice succeeded for volume "pv" (UniqueName: "fake-plugin/pv") pod "pod1" (UID: "pod1uid") device mount path ""
I0624 08:35:38.753516  220043 operation_generator.go:1683] MountVolume.NodeExpandVolume succeeded for volume "pv" (UniqueName: "fake-plugin/pv") pod "pod1" (UID: "pod1uid") 
I0624 08:35:38.795231  220043 operation_generator.go:1683] MountVolume.NodeExpandVolume succeeded for volume "pv" (UniqueName: "fake-plugin/pv") pod "pod1" (UID: "pod1uid") 
I0624 08:35:38.854546  220043 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "pv" (UniqueName: "fake-plugin/pv") pod "pod1" (UID: "pod1uid") 
I0624 08:35:38.854669  220043 reconciler.go:157] Reconciler: start to sync state
I0624 08:35:38.854734  220043 operation_generator.go:1357] Controller attach succeeded for volume "pv" (UniqueName: "fake-plugin/pv") pod "pod1" (UID: "pod1uid") device path: "fake/path"
E0624 08:35:38.854735  220043 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0624 08:35:38.856768  220043 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "pv" (UniqueName: "fake-plugin/pv") pod "pod1" (UID: "pod1uid") DevicePath "fake/path"
I0624 08:35:38.856917  220043 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "pv" (UniqueName: "fake-plugin/pv") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:35:38.962891  220043 operation_generator.go:1683] MountVolume.NodeExpandVolume succeeded for volume "pv" (UniqueName: "fake-plugin/pv") pod "pod1" (UID: "pod1uid") 
I0624 08:35:39.063584  220043 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "fail-expansion-in-use" (UniqueName: "fake-plugin/fail-expansion-in-use") pod "pod1" (UID: "pod1uid") 
I0624 08:35:39.063691  220043 reconciler.go:157] Reconciler: start to sync state
I0624 08:35:39.087963  220043 operation_generator.go:1357] Controller attach succeeded for volume "fail-expansion-in-use" (UniqueName: "fake-plugin/fail-expansion-in-use") pod "pod1" (UID: "pod1uid") device path: "fake/path"
E0624 08:35:39.063764  220043 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0624 08:35:39.088925  220043 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "fail-expansion-in-use" (UniqueName: "fake-plugin/fail-expansion-in-use") pod "pod1" (UID: "pod1uid") DevicePath "fake/path"
I0624 08:35:39.089053  220043 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "fail-expansion-in-use" (UniqueName: "fake-plugin/fail-expansion-in-use") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:35:39.089157  220043 operation_generator.go:594] MountVolume.MountDevice succeeded for volume "fail-expansion-in-use" (UniqueName: "fake-plugin/fail-expansion-in-use") pod "pod1" (UID: "pod1uid") device mount path ""
E0624 08:35:39.089399  220043 operation_generator.go:1669] MountVolume.NodeExapndVolume failed with %!!(MISSING)v(MISSING) for volume "fail-expansion-in-use" (UniqueName: "fake-plugin/fail-expansion-in-use") pod "pod1" (UID: "pod1uid") : volume-in-use
I0624 08:35:39.166082  220043 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "timeout-mount-device-volume" (UniqueName: "fake-plugin/timeout-mount-device-volume") pod "pod1" (UID: "pod1uid") 
I0624 08:35:39.166157  220043 reconciler.go:157] Reconciler: start to sync state
E0624 08:35:39.166209  220043 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0624 08:35:39.166447  220043 operation_generator.go:1357] Controller attach succeeded for volume "timeout-mount-device-volume" (UniqueName: "fake-plugin/timeout-mount-device-volume") pod "pod1" (UID: "pod1uid") device path: "fake/path"
I0624 08:35:39.167080  220043 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "timeout-mount-device-volume" (UniqueName: "fake-plugin/timeout-mount-device-volume") pod "pod1" (UID: "pod1uid") DevicePath "fake/path"
I0624 08:35:39.167218  220043 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "timeout-mount-device-volume" (UniqueName: "fake-plugin/timeout-mount-device-volume") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0624 08:35:39.167632  220043 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/timeout-mount-device-volume podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:35:39.667411359 +0000 UTC m=+2.359117127 (durationBeforeRetry 500ms). Error: "MapVolume.SetUpDevice failed for volume \"timeout-mount-device-volume\" (UniqueName: \"fake-plugin/timeout-mount-device-volume\") pod \"pod1\" (UID: \"pod1uid\") : mount failed"
I0624 08:35:39.289043  220043 reconciler.go:312] operationExecutor.UnmountDevice started for volume "timeout-mount-device-volume" (UniqueName: "fake-plugin/timeout-mount-device-volume") on node "mynodename" 
I0624 08:35:39.289277  220043 operation_generator.go:1282] UnmapDevice succeeded for volume "timeout-mount-device-volume" (UniqueName: "fake-plugin/timeout-mount-device-volume") on node "mynodename" 
I0624 08:35:39.289531  220043 reconciler.go:319] Volume detached for volume "timeout-mount-device-volume" (UniqueName: "fake-plugin/timeout-mount-device-volume") on node "mynodename" DevicePath "/dev/sdb"
I0624 08:35:39.390102  220043 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "fail-mount-device-volume-name" (UniqueName: "fake-plugin/fail-mount-device-volume-name") pod "pod1" (UID: "pod1uid") 
I0624 08:35:39.390188  220043 reconciler.go:157] Reconciler: start to sync state
E0624 08:35:39.390247  220043 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0624 08:35:39.390544  220043 operation_generator.go:1357] Controller attach succeeded for volume "fail-mount-device-volume-name" (UniqueName: "fake-plugin/fail-mount-device-volume-name") pod "pod1" (UID: "pod1uid") device path: "fake/path"
I0624 08:35:39.391103  220043 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "fail-mount-device-volume-name" (UniqueName: "fake-plugin/fail-mount-device-volume-name") pod "pod1" (UID: "pod1uid") DevicePath "fake/path"
I0624 08:35:39.391244  220043 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "fail-mount-device-volume-name" (UniqueName: "fake-plugin/fail-mount-device-volume-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0624 08:35:39.391706  220043 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/fail-mount-device-volume-name podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:35:39.891506432 +0000 UTC m=+2.583212197 (durationBeforeRetry 500ms). Error: "MapVolume.SetUpDevice failed for volume \"fail-mount-device-volume-name\" (UniqueName: \"fake-plugin/fail-mount-device-volume-name\") pod \"pod1\" (UID: \"pod1uid\") : error mapping disk: fail-mount-device-volume-name"
I0624 08:35:39.505351  220043 reconciler.go:319] Volume detached for volume "fail-mount-device-volume-name" (UniqueName: "fake-plugin/fail-mount-device-volume-name") on node "mynodename" DevicePath "fake/path"
I0624 08:35:39.633406  220043 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "timeout-and-fail-mount-device-name" (UniqueName: "fake-plugin/timeout-and-fail-mount-device-name") pod "pod1" (UID: "pod1uid") 
I0624 08:35:39.633938  220043 reconciler.go:157] Reconciler: start to sync state
E0624 08:35:39.634030  220043 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0624 08:35:39.634881  220043 operation_generator.go:1357] Controller attach succeeded for volume "timeout-and-fail-mount-device-name" (UniqueName: "fake-plugin/timeout-and-fail-mount-device-name") pod "pod1" (UID: "pod1uid") device path: "fake/path"
I0624 08:35:39.635744  220043 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "timeout-and-fail-mount-device-name" (UniqueName: "fake-plugin/timeout-and-fail-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "fake/path"
I0624 08:35:39.635902  220043 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "timeout-and-fail-mount-device-name" (UniqueName: "fake-plugin/timeout-and-fail-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0624 08:35:39.636459  220043 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/timeout-and-fail-mount-device-name podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:35:40.136120019 +0000 UTC m=+2.827825798 (durationBeforeRetry 500ms). Error: "MapVolume.SetUpDevice failed for volume \"timeout-and-fail-mount-device-name\" (UniqueName: \"fake-plugin/timeout-and-fail-mount-device-name\") pod \"pod1\" (UID: \"pod1uid\") : timed out mounting error"
I0624 08:35:40.138397  220043 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "timeout-and-fail-mount-device-name" (UniqueName: "fake-plugin/timeout-and-fail-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:35:40.139144  220043 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "timeout-and-fail-mount-device-name" (UniqueName: "fake-plugin/timeout-and-fail-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0624 08:35:40.154083  220043 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/timeout-and-fail-mount-device-name podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:35:41.140360237 +0000 UTC m=+3.832066013 (durationBeforeRetry 1s). Error: "MapVolume.SetUpDevice failed for volume \"timeout-and-fail-mount-device-name\" (UniqueName: \"fake-plugin/timeout-and-fail-mount-device-name\") pod \"pod1\" (UID: \"pod1uid\") : error mapping disk: timeout-and-fail-mount-device-name"
I0624 08:35:41.143499  220043 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "timeout-and-fail-mount-device-name" (UniqueName: "fake-plugin/timeout-and-fail-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:35:41.143721  220043 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "timeout-and-fail-mount-device-name" (UniqueName: "fake-plugin/timeout-and-fail-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0624 08:35:41.144190  220043 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/timeout-and-fail-mount-device-name podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:35:43.143987511 +0000 UTC m=+5.835693286 (durationBeforeRetry 2s). Error: "MapVolume.SetUpDevice failed for volume \"timeout-and-fail-mount-device-name\" (UniqueName: \"fake-plugin/timeout-and-fail-mount-device-name\") pod \"pod1\" (UID: \"pod1uid\") : error mapping disk: timeout-and-fail-mount-device-name"
I0624 08:35:43.144436  220043 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "timeout-and-fail-mount-device-name" (UniqueName: "fake-plugin/timeout-and-fail-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:35:43.144605  220043 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "timeout-and-fail-mount-device-name" (UniqueName: "fake-plugin/timeout-and-fail-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0624 08:35:43.145004  220043 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/timeout-and-fail-mount-device-name podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:35:47.144810387 +0000 UTC m=+9.836516153 (durationBeforeRetry 4s). Error: "MapVolume.SetUpDevice failed for volume \"timeout-and-fail-mount-device-name\" (UniqueName: \"fake-plugin/timeout-and-fail-mount-device-name\") pod \"pod1\" (UID: \"pod1uid\") : error mapping disk: timeout-and-fail-mount-device-name"
I0624 08:35:47.147176  220043 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "timeout-and-fail-mount-device-name" (UniqueName: "fake-plugin/timeout-and-fail-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:35:47.148640  220043 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "timeout-and-fail-mount-device-name" (UniqueName: "fake-plugin/timeout-and-fail-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0624 08:35:47.149053  220043 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/timeout-and-fail-mount-device-name podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:35:55.148862883 +0000 UTC m=+17.840568649 (durationBeforeRetry 8s). Error: "MapVolume.SetUpDevice failed for volume \"timeout-and-fail-mount-device-name\" (UniqueName: \"fake-plugin/timeout-and-fail-mount-device-name\") pod \"pod1\" (UID: \"pod1uid\") : error mapping disk: timeout-and-fail-mount-device-name"
I0624 08:35:49.749094  220043 reconciler.go:319] Volume detached for volume "timeout-and-fail-mount-device-name" (UniqueName: "fake-plugin/timeout-and-fail-mount-device-name") on node "mynodename" DevicePath "/dev/sdb"
I0624 08:35:49.874803  220043 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "success-and-timeout-device-name" (UniqueName: "fake-plugin/success-and-timeout-device-name") pod "pod1" (UID: "pod1uid") 
I0624 08:35:49.874891  220043 reconciler.go:157] Reconciler: start to sync state
E0624 08:35:49.874943  220043 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0624 08:35:49.875262  220043 operation_generator.go:1357] Controller attach succeeded for volume "success-and-timeout-device-name" (UniqueName: "fake-plugin/success-and-timeout-device-name") pod "pod1" (UID: "pod1uid") device path: "fake/path"
I0624 08:35:49.875845  220043 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "success-and-timeout-device-name" (UniqueName: "fake-plugin/success-and-timeout-device-name") pod "pod1" (UID: "pod1uid") DevicePath "fake/path"
I0624 08:35:49.876008  220043 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "success-and-timeout-device-name" (UniqueName: "fake-plugin/success-and-timeout-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:35:49.996515  220043 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "success-and-timeout-device-name" (UniqueName: "fake-plugin/success-and-timeout-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:35:50.002856  220043 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "success-and-timeout-device-name" (UniqueName: "fake-plugin/success-and-timeout-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0624 08:35:50.003939  220043 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-timeout-device-name podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:35:50.503522187 +0000 UTC m=+13.195227996 (durationBeforeRetry 500ms). Error: "MapVolume.SetUpDevice failed for volume \"success-and-timeout-device-name\" (UniqueName: \"fake-plugin/success-and-timeout-device-name\") pod \"pod1\" (UID: \"pod1uid\") : error mounting state"
I0624 08:35:50.503904  220043 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "success-and-timeout-device-name" (UniqueName: "fake-plugin/success-and-timeout-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:35:50.504072  220043 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "success-and-timeout-device-name" (UniqueName: "fake-plugin/success-and-timeout-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0624 08:35:50.504484  220043 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-timeout-device-name podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:35:51.504261881 +0000 UTC m=+14.195967646 (durationBeforeRetry 1s). Error: "MapVolume.SetUpDevice failed for volume \"success-and-timeout-device-name\" (UniqueName: \"fake-plugin/success-and-timeout-device-name\") pod \"pod1\" (UID: \"pod1uid\") : error mounting state"
I0624 08:35:51.505518  220043 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "success-and-timeout-device-name" (UniqueName: "fake-plugin/success-and-timeout-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:35:51.505706  220043 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "success-and-timeout-device-name" (UniqueName: "fake-plugin/success-and-timeout-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0624 08:35:51.506080  220043 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-timeout-device-name podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:35:53.505894831 +0000 UTC m=+16.197600594 (durationBeforeRetry 2s). Error: "MapVolume.SetUpDevice failed for volume \"success-and-timeout-device-name\" (UniqueName: \"fake-plugin/success-and-timeout-device-name\") pod \"pod1\" (UID: \"pod1uid\") : error mounting state"
I0624 08:35:53.506282  220043 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "success-and-timeout-device-name" (UniqueName: "fake-plugin/success-and-timeout-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:35:53.506464  220043 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "success-and-timeout-device-name" (UniqueName: "fake-plugin/success-and-timeout-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0624 08:35:53.506873  220043 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-timeout-device-name podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:35:57.506665125 +0000 UTC m=+20.198370892 (durationBeforeRetry 4s). Error: "MapVolume.SetUpDevice failed for volume \"success-and-timeout-device-name\" (UniqueName: \"fake-plugin/success-and-timeout-device-name\") pod \"pod1\" (UID: \"pod1uid\") : error mounting state"
I0624 08:35:57.507231  220043 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "success-and-timeout-device-name" (UniqueName: "fake-plugin/success-and-timeout-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:35:57.507390  220043 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "success-and-timeout-device-name" (UniqueName: "fake-plugin/success-and-timeout-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0624 08:35:57.507797  220043 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-timeout-device-name podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:36:05.507606914 +0000 UTC m=+28.199312680 (durationBeforeRetry 8s). Error: "MapVolume.SetUpDevice failed for volume \"success-and-timeout-device-name\" (UniqueName: \"fake-plugin/success-and-timeout-device-name\") pod \"pod1\" (UID: \"pod1uid\") : error mounting state"
I0624 08:36:00.009646  220043 reconciler.go:196] operationExecutor.UnmountVolume started for volume "success-and-timeout-device-name" (UniqueName: "fake-plugin/success-and-timeout-device-name") pod "pod1uid" (UID: "pod1uid") 
I0624 08:36:00.009186  220043 operation_generator.go:1170] UnmapVolume succeeded for volume "fake-plugin/success-and-timeout-device-name" (OuterVolumeSpecName: "success-and-timeout-device-name") pod "pod1uid" (UID: "pod1uid"). InnerVolumeSpecName "success-and-timeout-device-name". PluginName "fake-plugin", VolumeGidValue ""
I0624 08:36:00.010880  220043 reconciler.go:312] operationExecutor.UnmountDevice started for volume "success-and-timeout-device-name" (UniqueName: "fake-plugin/success-and-timeout-device-name") on node "mynodename" 
I0624 08:36:00.010889  220043 operation_generator.go:1282] UnmapDevice succeeded for volume "success-and-timeout-device-name" (UniqueName: "fake-plugin/success-and-timeout-device-name") on node "mynodename" 
I0624 08:36:00.011222  220043 reconciler.go:319] Volume detached for volume "success-and-timeout-device-name" (UniqueName: "fake-plugin/success-and-timeout-device-name") on node "mynodename" DevicePath "/dev/sdb"
I0624 08:36:00.109048  220043 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") pod "pod1" (UID: "pod1uid") 
I0624 08:36:00.109128  220043 reconciler.go:157] Reconciler: start to sync state
E0624 08:36:00.109184  220043 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0624 08:36:00.109411  220043 operation_generator.go:1357] Controller attach succeeded for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") pod "pod1" (UID: "pod1uid") device path: "fake/path"
I0624 08:36:00.110434  220043 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "fake/path"
I0624 08:36:00.110590  220043 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:36:00.224073  220043 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:36:00.224235  220043 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0624 08:36:00.224632  220043 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-failed-mount-device-name podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:36:00.724440171 +0000 UTC m=+23.416145936 (durationBeforeRetry 500ms). Error: "MapVolume.SetUpDevice failed for volume \"success-and-failed-mount-device-name\" (UniqueName: \"fake-plugin/success-and-failed-mount-device-name\") pod \"pod1\" (UID: \"pod1uid\") : error mapping disk: success-and-failed-mount-device-name"
I0624 08:36:00.749009  220043 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:36:00.749204  220043 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0624 08:36:00.749605  220043 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-failed-mount-device-name podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:36:01.749401752 +0000 UTC m=+24.441107517 (durationBeforeRetry 1s). Error: "MapVolume.SetUpDevice failed for volume \"success-and-failed-mount-device-name\" (UniqueName: \"fake-plugin/success-and-failed-mount-device-name\") pod \"pod1\" (UID: \"pod1uid\") : error mapping disk: success-and-failed-mount-device-name"
I0624 08:36:01.785860  220043 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:36:01.786016  220043 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0624 08:36:01.786855  220043 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-failed-mount-device-name podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:36:03.786249991 +0000 UTC m=+26.477955782 (durationBeforeRetry 2s). Error: "MapVolume.SetUpDevice failed for volume \"success-and-failed-mount-device-name\" (UniqueName: \"fake-plugin/success-and-failed-mount-device-name\") pod \"pod1\" (UID: \"pod1uid\") : error mapping disk: success-and-failed-mount-device-name"
I0624 08:36:03.787354  220043 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:36:03.788251  220043 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0624 08:36:03.789596  220043 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-failed-mount-device-name podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:36:07.789183944 +0000 UTC m=+30.480889718 (durationBeforeRetry 4s). Error: "MapVolume.SetUpDevice failed for volume \"success-and-failed-mount-device-name\" (UniqueName: \"fake-plugin/success-and-failed-mount-device-name\") pod \"pod1\" (UID: \"pod1uid\") : error mapping disk: success-and-failed-mount-device-name"
I0624 08:36:07.790829  220043 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:36:07.790998  220043 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0624 08:36:07.791392  220043 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-failed-mount-device-name podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:36:15.791192475 +0000 UTC m=+38.482898241 (durationBeforeRetry 8s). Error: "MapVolume.SetUpDevice failed for volume \"success-and-failed-mount-device-name\" (UniqueName: \"fake-plugin/success-and-failed-mount-device-name\") pod \"pod1\" (UID: \"pod1uid\") : error mapping disk: success-and-failed-mount-device-name"
I0624 08:36:10.221253  220043 reconciler.go:196] operationExecutor.UnmountVolume started for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") pod "pod1uid" (UID: "pod1uid") 
I0624 08:36:10.220918  220043 operation_generator.go:1170] UnmapVolume succeeded for volume "fake-plugin/success-and-failed-mount-device-name" (OuterVolumeSpecName: "success-and-failed-mount-device-name") pod "pod1uid" (UID: "pod1uid"). InnerVolumeSpecName "success-and-failed-mount-device-name". PluginName "fake-plugin", VolumeGidValue ""
I0624 08:36:10.222886  220043 operation_generator.go:1282] UnmapDevice succeeded for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") on node "mynodename" 
I0624 08:36:10.222967  220043 reconciler.go:312] operationExecutor.UnmountDevice started for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") on node "mynodename" 
I0624 08:36:10.224204  220043 reconciler.go:319] Volume detached for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") on node "mynodename" DevicePath "/dev/sdb"
I0624 08:36:10.321190  220043 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "timeout-mount-device-volume" (UniqueName: "fake-plugin/timeout-mount-device-volume") pod "pod1" (UID: "pod1uid") 
I0624 08:36:10.321273  220043 reconciler.go:157] Reconciler: start to sync state
E0624 08:36:10.321331  220043 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0624 08:36:10.321847  220043 operation_generator.go:1357] Controller attach succeeded for volume "timeout-mount-device-volume" (UniqueName: "fake-plugin/timeout-mount-device-volume") pod "pod1" (UID: "pod1uid") device path: "fake/path"
I0624 08:36:10.322377  220043 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "timeout-mount-device-volume" (UniqueName: "fake-plugin/timeout-mount-device-volume") pod "pod1" (UID: "pod1uid") DevicePath "fake/path"
I0624 08:36:10.322497  220043 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "timeout-mount-device-volume" (UniqueName: "fake-plugin/timeout-mount-device-volume") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0624 08:36:10.322901  220043 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/timeout-mount-device-volume podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:36:10.822721484 +0000 UTC m=+33.514427259 (durationBeforeRetry 500ms). Error: "MountVolume.MountDevice failed for volume \"timeout-mount-device-volume\" (UniqueName: \"fake-plugin/timeout-mount-device-volume\") pod \"pod1\" (UID: \"pod1uid\") : mount failed"
I0624 08:36:10.440413  220043 reconciler.go:312] operationExecutor.UnmountDevice started for volume "timeout-mount-device-volume" (UniqueName: "fake-plugin/timeout-mount-device-volume") on node "mynodename" 
I0624 08:36:10.440646  220043 operation_generator.go:891] UnmountDevice succeeded for volume "timeout-mount-device-volume" %!(EXTRA string=UnmountDevice succeeded for volume "timeout-mount-device-volume" (UniqueName: "fake-plugin/timeout-mount-device-volume") on node "mynodename" )
I0624 08:36:10.440874  220043 reconciler.go:319] Volume detached for volume "timeout-mount-device-volume" (UniqueName: "fake-plugin/timeout-mount-device-volume") on node "mynodename" DevicePath "/dev/sdb"
I0624 08:36:10.546303  220043 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "fail-mount-device-volume-name" (UniqueName: "fake-plugin/fail-mount-device-volume-name") pod "pod1" (UID: "pod1uid") 
I0624 08:36:10.546375  220043 reconciler.go:157] Reconciler: start to sync state
E0624 08:36:10.546430  220043 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0624 08:36:10.546691  220043 operation_generator.go:1357] Controller attach succeeded for volume "fail-mount-device-volume-name" (UniqueName: "fake-plugin/fail-mount-device-volume-name") pod "pod1" (UID: "pod1uid") device path: "fake/path"
I0624 08:36:10.547425  220043 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "fail-mount-device-volume-name" (UniqueName: "fake-plugin/fail-mount-device-volume-name") pod "pod1" (UID: "pod1uid") DevicePath "fake/path"
I0624 08:36:10.547563  220043 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "fail-mount-device-volume-name" (UniqueName: "fake-plugin/fail-mount-device-volume-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0624 08:36:10.547987  220043 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/fail-mount-device-volume-name podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:36:11.047795139 +0000 UTC m=+33.739500912 (durationBeforeRetry 500ms). Error: "MountVolume.MountDevice failed for volume \"fail-mount-device-volume-name\" (UniqueName: \"fake-plugin/fail-mount-device-volume-name\") pod \"pod1\" (UID: \"pod1uid\") : error mounting disk: /dev/sdb"
I0624 08:36:10.686780  220043 reconciler.go:319] Volume detached for volume "fail-mount-device-volume-name" (UniqueName: "fake-plugin/fail-mount-device-volume-name") on node "mynodename" DevicePath "fake/path"
I0624 08:36:10.795267  220043 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "timeout-and-fail-mount-device-name" (UniqueName: "fake-plugin/timeout-and-fail-mount-device-name") pod "pod1" (UID: "pod1uid") 
I0624 08:36:10.795355  220043 reconciler.go:157] Reconciler: start to sync state
E0624 08:36:10.795445  220043 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0624 08:36:10.795865  220043 operation_generator.go:1357] Controller attach succeeded for volume "timeout-and-fail-mount-device-name" (UniqueName: "fake-plugin/timeout-and-fail-mount-device-name") pod "pod1" (UID: "pod1uid") device path: "fake/path"
I0624 08:36:10.796668  220043 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "timeout-and-fail-mount-device-name" (UniqueName: "fake-plugin/timeout-and-fail-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "fake/path"
I0624 08:36:10.796837  220043 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "timeout-and-fail-mount-device-name" (UniqueName: "fake-plugin/timeout-and-fail-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0624 08:36:10.797317  220043 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/timeout-and-fail-mount-device-name podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:36:11.297078386 +0000 UTC m=+33.988784151 (durationBeforeRetry 500ms). Error: "MountVolume.MountDevice failed for volume \"timeout-and-fail-mount-device-name\" (UniqueName: \"fake-plugin/timeout-and-fail-mount-device-name\") pod \"pod1\" (UID: \"pod1uid\") : timed out mounting error"
I0624 08:36:11.300145  220043 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "timeout-and-fail-mount-device-name" (UniqueName: "fake-plugin/timeout-and-fail-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:36:11.300347  220043 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "timeout-and-fail-mount-device-name" (UniqueName: "fake-plugin/timeout-and-fail-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0624 08:36:11.300863  220043 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/timeout-and-fail-mount-device-name podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:36:12.30064447 +0000 UTC m=+34.992350248 (durationBeforeRetry 1s). Error: "MountVolume.MountDevice failed for volume \"timeout-and-fail-mount-device-name\" (UniqueName: \"fake-plugin/timeout-and-fail-mount-device-name\") pod \"pod1\" (UID: \"pod1uid\") : error mounting disk: /dev/sdb"
I0624 08:36:12.406125  220043 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "timeout-and-fail-mount-device-name" (UniqueName: "fake-plugin/timeout-and-fail-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:36:12.406293  220043 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "timeout-and-fail-mount-device-name" (UniqueName: "fake-plugin/timeout-and-fail-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0624 08:36:12.406971  220043 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/timeout-and-fail-mount-device-name podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:36:14.406513167 +0000 UTC m=+37.098218932 (durationBeforeRetry 2s). Error: "MountVolume.MountDevice failed for volume \"timeout-and-fail-mount-device-name\" (UniqueName: \"fake-plugin/timeout-and-fail-mount-device-name\") pod \"pod1\" (UID: \"pod1uid\") : error mounting disk: /dev/sdb"
I0624 08:36:14.419819  220043 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "timeout-and-fail-mount-device-name" (UniqueName: "fake-plugin/timeout-and-fail-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:36:14.419994  220043 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "timeout-and-fail-mount-device-name" (UniqueName: "fake-plugin/timeout-and-fail-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0624 08:36:14.420593  220043 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/timeout-and-fail-mount-device-name podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:36:18.420215735 +0000 UTC m=+41.111921502 (durationBeforeRetry 4s). Error: "MountVolume.MountDevice failed for volume \"timeout-and-fail-mount-device-name\" (UniqueName: \"fake-plugin/timeout-and-fail-mount-device-name\") pod \"pod1\" (UID: \"pod1uid\") : error mounting disk: /dev/sdb"
I0624 08:36:18.420711  220043 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "timeout-and-fail-mount-device-name" (UniqueName: "fake-plugin/timeout-and-fail-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:36:18.420884  220043 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "timeout-and-fail-mount-device-name" (UniqueName: "fake-plugin/timeout-and-fail-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0624 08:36:18.421301  220043 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/timeout-and-fail-mount-device-name podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:36:26.421110372 +0000 UTC m=+49.112816151 (durationBeforeRetry 8s). Error: "MountVolume.MountDevice failed for volume \"timeout-and-fail-mount-device-name\" (UniqueName: \"fake-plugin/timeout-and-fail-mount-device-name\") pod \"pod1\" (UID: \"pod1uid\") : error mounting disk: /dev/sdb"
I0624 08:36:20.932039  220043 reconciler.go:319] Volume detached for volume "timeout-and-fail-mount-device-name" (UniqueName: "fake-plugin/timeout-and-fail-mount-device-name") on node "mynodename" DevicePath "/dev/sdb"
I0624 08:36:21.028959  220043 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "success-and-timeout-device-name" (UniqueName: "fake-plugin/success-and-timeout-device-name") pod "pod1" (UID: "pod1uid") 
I0624 08:36:21.029038  220043 reconciler.go:157] Reconciler: start to sync state
E0624 08:36:21.029091  220043 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0624 08:36:21.029539  220043 operation_generator.go:1357] Controller attach succeeded for volume "success-and-timeout-device-name" (UniqueName: "fake-plugin/success-and-timeout-device-name") pod "pod1" (UID: "pod1uid") device path: "fake/path"
I0624 08:36:21.030187  220043 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "success-and-timeout-device-name" (UniqueName: "fake-plugin/success-and-timeout-device-name") pod "pod1" (UID: "pod1uid") DevicePath "fake/path"
I0624 08:36:21.030317  220043 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "success-and-timeout-device-name" (UniqueName: "fake-plugin/success-and-timeout-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:36:21.030428  220043 operation_generator.go:594] MountVolume.MountDevice succeeded for volume "success-and-timeout-device-name" (UniqueName: "fake-plugin/success-and-timeout-device-name") pod "pod1" (UID: "pod1uid") device mount path ""
I0624 08:36:21.193966  220043 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "success-and-timeout-device-name" (UniqueName: "fake-plugin/success-and-timeout-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:36:21.194137  220043 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "success-and-timeout-device-name" (UniqueName: "fake-plugin/success-and-timeout-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:36:31.236349  220043 reconciler.go:196] operationExecutor.UnmountVolume started for volume "success-and-timeout-device-name" (UniqueName: "fake-plugin/success-and-timeout-device-name") pod "pod1uid" (UID: "pod1uid") 
I0624 08:36:31.236939  220043 operation_generator.go:797] UnmountVolume.TearDown succeeded for volume "fake-plugin/success-and-timeout-device-name" (OuterVolumeSpecName: "success-and-timeout-device-name") pod "pod1uid" (UID: "pod1uid"). InnerVolumeSpecName "success-and-timeout-device-name". PluginName "fake-plugin", VolumeGidValue ""
I0624 08:36:31.237452  220043 reconciler.go:312] operationExecutor.UnmountDevice started for volume "success-and-timeout-device-name" (UniqueName: "fake-plugin/success-and-timeout-device-name") on node "mynodename" 
I0624 08:36:31.237798  220043 operation_generator.go:891] UnmountDevice succeeded for volume "success-and-timeout-device-name" %!(EXTRA string=UnmountDevice succeeded for volume "success-and-timeout-device-name" (UniqueName: "fake-plugin/success-and-timeout-device-name") on node "mynodename" )
I0624 08:36:31.238033  220043 reconciler.go:319] Volume detached for volume "success-and-timeout-device-name" (UniqueName: "fake-plugin/success-and-timeout-device-name") on node "mynodename" DevicePath "/dev/sdb"
I0624 08:36:31.354058  220043 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") pod "pod1" (UID: "pod1uid") 
I0624 08:36:31.354127  220043 reconciler.go:157] Reconciler: start to sync state
E0624 08:36:31.354203  220043 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0624 08:36:31.354617  220043 operation_generator.go:1357] Controller attach succeeded for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") pod "pod1" (UID: "pod1uid") device path: "fake/path"
I0624 08:36:31.355405  220043 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "fake/path"
I0624 08:36:31.355533  220043 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:36:31.355656  220043 operation_generator.go:594] MountVolume.MountDevice succeeded for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") pod "pod1" (UID: "pod1uid") device mount path ""
I0624 08:36:31.457654  220043 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:36:31.458230  220043 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:36:31.460825  220043 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:36:31.462233  220043 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:36:41.495275  220043 reconciler.go:196] operationExecutor.UnmountVolume started for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") pod "pod1uid" (UID: "pod1uid") 
I0624 08:36:41.495509  220043 operation_generator.go:797] UnmountVolume.TearDown succeeded for volume "fake-plugin/success-and-failed-mount-device-name" (OuterVolumeSpecName: "success-and-failed-mount-device-name") pod "pod1uid" (UID: "pod1uid"). InnerVolumeSpecName "success-and-failed-mount-device-name". PluginName "fake-plugin", VolumeGidValue ""
I0624 08:36:41.496281  220043 reconciler.go:312] operationExecutor.UnmountDevice started for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") on node "mynodename" 
I0624 08:36:41.496482  220043 operation_generator.go:891] UnmountDevice succeeded for volume "success-and-failed-mount-device-name" %!(EXTRA string=UnmountDevice succeeded for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") on node "mynodename" )
I0624 08:36:41.496998  220043 reconciler.go:319] Volume detached for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") on node "mynodename" DevicePath "/dev/sdb"
I0624 08:36:41.596845  220043 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "timeout-setup-volume" (UniqueName: "fake-plugin/timeout-setup-volume") pod "pod1" (UID: "pod1uid") 
I0624 08:36:41.596934  220043 reconciler.go:157] Reconciler: start to sync state
E0624 08:36:41.596984  220043 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0624 08:36:41.597469  220043 operation_generator.go:1357] Controller attach succeeded for volume "timeout-setup-volume" (UniqueName: "fake-plugin/timeout-setup-volume") pod "pod1" (UID: "pod1uid") device path: "fake/path"
I0624 08:36:41.598358  220043 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "timeout-setup-volume" (UniqueName: "fake-plugin/timeout-setup-volume") pod "pod1" (UID: "pod1uid") DevicePath "fake/path"
I0624 08:36:41.598504  220043 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "timeout-setup-volume" (UniqueName: "fake-plugin/timeout-setup-volume") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0624 08:36:41.598878  220043 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/timeout-setup-volume podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:36:42.09868944 +0000 UTC m=+64.790395213 (durationBeforeRetry 500ms). Error: "MapVolume.MapPodDevice failed for volume \"timeout-setup-volume\" (UniqueName: \"fake-plugin/timeout-setup-volume\") pod \"pod1\" (UID: \"pod1uid\") : time out on setup"
I0624 08:36:41.704278  220043 reconciler.go:196] operationExecutor.UnmountVolume started for volume "timeout-setup-volume" (UniqueName: "fake-plugin/timeout-setup-volume") pod "pod1uid" (UID: "pod1uid") 
I0624 08:36:41.704527  220043 operation_generator.go:1170] UnmapVolume succeeded for volume "fake-plugin/timeout-setup-volume" (OuterVolumeSpecName: "timeout-setup-volume") pod "pod1uid" (UID: "pod1uid"). InnerVolumeSpecName "timeout-setup-volume". PluginName "fake-plugin", VolumeGidValue ""
I0624 08:36:41.705008  220043 reconciler.go:312] operationExecutor.UnmountDevice started for volume "timeout-setup-volume" (UniqueName: "fake-plugin/timeout-setup-volume") on node "mynodename" 
I0624 08:36:41.705165  220043 operation_generator.go:1282] UnmapDevice succeeded for volume "timeout-setup-volume" (UniqueName: "fake-plugin/timeout-setup-volume") on node "mynodename" 
I0624 08:36:41.705373  220043 reconciler.go:319] Volume detached for volume "timeout-setup-volume" (UniqueName: "fake-plugin/timeout-setup-volume") on node "mynodename" DevicePath "/dev/sdb"
I0624 08:36:41.816168  220043 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "fail-setup-volume" (UniqueName: "fake-plugin/fail-setup-volume") pod "pod1" (UID: "pod1uid") 
I0624 08:36:41.816252  220043 reconciler.go:157] Reconciler: start to sync state
E0624 08:36:41.816319  220043 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0624 08:36:41.816562  220043 operation_generator.go:1357] Controller attach succeeded for volume "fail-setup-volume" (UniqueName: "fake-plugin/fail-setup-volume") pod "pod1" (UID: "pod1uid") device path: "fake/path"
I0624 08:36:41.817368  220043 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "fail-setup-volume" (UniqueName: "fake-plugin/fail-setup-volume") pod "pod1" (UID: "pod1uid") DevicePath "fake/path"
I0624 08:36:41.817491  220043 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "fail-setup-volume" (UniqueName: "fake-plugin/fail-setup-volume") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0624 08:36:41.817832  220043 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/fail-setup-volume podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:36:42.317669998 +0000 UTC m=+65.009375775 (durationBeforeRetry 500ms). Error: "MapVolume.MapPodDevice failed for volume \"fail-setup-volume\" (UniqueName: \"fake-plugin/fail-setup-volume\") pod \"pod1\" (UID: \"pod1uid\") : mounting volume failed"
I0624 08:36:41.924426  220043 reconciler.go:312] operationExecutor.UnmountDevice started for volume "fail-setup-volume" (UniqueName: "fake-plugin/fail-setup-volume") on node "mynodename" 
I0624 08:36:41.924857  220043 operation_generator.go:1282] UnmapDevice succeeded for volume "fail-setup-volume" (UniqueName: "fake-plugin/fail-setup-volume") on node "mynodename" 
I0624 08:36:41.925100  220043 reconciler.go:319] Volume detached for volume "fail-setup-volume" (UniqueName: "fake-plugin/fail-setup-volume") on node "mynodename" DevicePath "/dev/sdb"
I0624 08:36:42.042855  220043 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") pod "pod1" (UID: "pod1uid") 
I0624 08:36:42.042950  220043 reconciler.go:157] Reconciler: start to sync state
E0624 08:36:42.043008  220043 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0624 08:36:42.045827  220043 operation_generator.go:1357] Controller attach succeeded for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") pod "pod1" (UID: "pod1uid") device path: "fake/path"
I0624 08:36:42.049964  220043 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") pod "pod1" (UID: "pod1uid") DevicePath "fake/path"
I0624 08:36:42.050107  220043 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0624 08:36:42.050535  220043 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/timeout-and-fail-setup-volume podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:36:42.550333889 +0000 UTC m=+65.242039670 (durationBeforeRetry 500ms). Error: "MapVolume.MapPodDevice failed for volume \"timeout-and-fail-setup-volume\" (UniqueName: \"fake-plugin/timeout-and-fail-setup-volume\") pod \"pod1\" (UID: \"pod1uid\") : time out on setup"
I0624 08:36:42.552750  220043 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:36:42.555216  220043 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0624 08:36:42.559699  220043 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/timeout-and-fail-setup-volume podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:36:43.556622071 +0000 UTC m=+66.248327843 (durationBeforeRetry 1s). Error: "MapVolume.MapPodDevice failed for volume \"timeout-and-fail-setup-volume\" (UniqueName: \"fake-plugin/timeout-and-fail-setup-volume\") pod \"pod1\" (UID: \"pod1uid\") : mounting volume failed"
I0624 08:36:43.571160  220043 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:36:43.584917  220043 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0624 08:36:43.585397  220043 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/timeout-and-fail-setup-volume podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:36:45.585160222 +0000 UTC m=+68.276865987 (durationBeforeRetry 2s). Error: "MapVolume.MapPodDevice failed for volume \"timeout-and-fail-setup-volume\" (UniqueName: \"fake-plugin/timeout-and-fail-setup-volume\") pod \"pod1\" (UID: \"pod1uid\") : mounting volume failed"
I0624 08:36:45.586168  220043 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:36:45.586922  220043 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0624 08:36:45.590664  220043 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/timeout-and-fail-setup-volume podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:36:49.58807603 +0000 UTC m=+72.279781813 (durationBeforeRetry 4s). Error: "MapVolume.MapPodDevice failed for volume \"timeout-and-fail-setup-volume\" (UniqueName: \"fake-plugin/timeout-and-fail-setup-volume\") pod \"pod1\" (UID: \"pod1uid\") : mounting volume failed"
I0624 08:36:49.600461  220043 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:36:49.600632  220043 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0624 08:36:49.601032  220043 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/timeout-and-fail-setup-volume podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:36:57.600854902 +0000 UTC m=+80.292560665 (durationBeforeRetry 8s). Error: "MapVolume.MapPodDevice failed for volume \"timeout-and-fail-setup-volume\" (UniqueName: \"fake-plugin/timeout-and-fail-setup-volume\") pod \"pod1\" (UID: \"pod1uid\") : mounting volume failed"
I0624 08:36:52.141921  220043 reconciler.go:312] operationExecutor.UnmountDevice started for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") on node "mynodename" 
I0624 08:36:52.142218  220043 operation_generator.go:1282] UnmapDevice succeeded for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") on node "mynodename" 
I0624 08:36:52.142438  220043 reconciler.go:319] Volume detached for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") on node "mynodename" DevicePath "/dev/sdb"
I0624 08:36:52.243442  220043 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1" (UID: "pod1uid") 
I0624 08:36:52.243519  220043 reconciler.go:157] Reconciler: start to sync state
E0624 08:36:52.243580  220043 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0624 08:36:52.264082  220043 operation_generator.go:1357] Controller attach succeeded for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1" (UID: "pod1uid") device path: "fake/path"
I0624 08:36:52.264741  220043 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1" (UID: "pod1uid") DevicePath "fake/path"
I0624 08:36:52.264891  220043 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:36:52.373253  220043 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:36:52.373467  220043 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0624 08:36:52.373952  220043 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-timeout-setup-volume-name podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:36:52.873729798 +0000 UTC m=+75.565435579 (durationBeforeRetry 500ms). Error: "MapVolume.MapPodDevice failed for volume \"success-and-timeout-setup-volume-name\" (UniqueName: \"fake-plugin/success-and-timeout-setup-volume-name\") pod \"pod1\" (UID: \"pod1uid\") : time out on setup"
I0624 08:36:52.874206  220043 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:36:52.874332  220043 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0624 08:36:52.874607  220043 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-timeout-setup-volume-name podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:36:53.874474638 +0000 UTC m=+76.566180400 (durationBeforeRetry 1s). Error: "MapVolume.MapPodDevice failed for volume \"success-and-timeout-setup-volume-name\" (UniqueName: \"fake-plugin/success-and-timeout-setup-volume-name\") pod \"pod1\" (UID: \"pod1uid\") : time out on setup"
I0624 08:36:53.913316  220043 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:36:53.915443  220043 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0624 08:36:53.932746  220043 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-timeout-setup-volume-name podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:36:55.932172519 +0000 UTC m=+78.623878298 (durationBeforeRetry 2s). Error: "MapVolume.MapPodDevice failed for volume \"success-and-timeout-setup-volume-name\" (UniqueName: \"fake-plugin/success-and-timeout-setup-volume-name\") pod \"pod1\" (UID: \"pod1uid\") : time out on setup"
I0624 08:36:55.934106  220043 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:36:55.934364  220043 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0624 08:36:55.935252  220043 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-timeout-setup-volume-name podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:36:59.934697476 +0000 UTC m=+82.626403241 (durationBeforeRetry 4s). Error: "MapVolume.MapPodDevice failed for volume \"success-and-timeout-setup-volume-name\" (UniqueName: \"fake-plugin/success-and-timeout-setup-volume-name\") pod \"pod1\" (UID: \"pod1uid\") : time out on setup"
I0624 08:36:59.956784  220043 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:36:59.956960  220043 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0624 08:36:59.957379  220043 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-timeout-setup-volume-name podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:37:07.957180372 +0000 UTC m=+90.648886139 (durationBeforeRetry 8s). Error: "MapVolume.MapPodDevice failed for volume \"success-and-timeout-setup-volume-name\" (UniqueName: \"fake-plugin/success-and-timeout-setup-volume-name\") pod \"pod1\" (UID: \"pod1uid\") : time out on setup"
I0624 08:37:02.366310  220043 reconciler.go:196] operationExecutor.UnmountVolume started for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1uid" (UID: "pod1uid") 
I0624 08:37:02.366628  220043 operation_generator.go:1170] UnmapVolume succeeded for volume "fake-plugin/success-and-timeout-setup-volume-name" (OuterVolumeSpecName: "success-and-timeout-setup-volume-name") pod "pod1uid" (UID: "pod1uid"). InnerVolumeSpecName "success-and-timeout-setup-volume-name". PluginName "fake-plugin", VolumeGidValue ""
I0624 08:37:02.367131  220043 reconciler.go:312] operationExecutor.UnmountDevice started for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") on node "mynodename" 
I0624 08:37:02.367428  220043 operation_generator.go:1282] UnmapDevice succeeded for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") on node "mynodename" 
I0624 08:37:02.367635  220043 reconciler.go:319] Volume detached for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") on node "mynodename" DevicePath "/dev/sdb"
I0624 08:37:02.470518  220043 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1" (UID: "pod1uid") 
I0624 08:37:02.470607  220043 reconciler.go:157] Reconciler: start to sync state
E0624 08:37:02.471255  220043 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0624 08:37:02.471409  220043 operation_generator.go:1357] Controller attach succeeded for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1" (UID: "pod1uid") device path: "fake/path"
I0624 08:37:02.474688  220043 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1" (UID: "pod1uid") DevicePath "fake/path"
I0624 08:37:02.474812  220043 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:37:02.585644  220043 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:37:02.585805  220043 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0624 08:37:02.592236  220043 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-failed-setup-device-name podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:37:03.092042467 +0000 UTC m=+85.783748252 (durationBeforeRetry 500ms). Error: "MapVolume.MapPodDevice failed for volume \"success-and-failed-setup-device-name\" (UniqueName: \"fake-plugin/success-and-failed-setup-device-name\") pod \"pod1\" (UID: \"pod1uid\") : mounting volume failed"
I0624 08:37:03.093639  220043 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:37:03.093803  220043 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0624 08:37:03.094204  220043 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-failed-setup-device-name podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:37:04.094013331 +0000 UTC m=+86.785719097 (durationBeforeRetry 1s). Error: "MapVolume.MapPodDevice failed for volume \"success-and-failed-setup-device-name\" (UniqueName: \"fake-plugin/success-and-failed-setup-device-name\") pod \"pod1\" (UID: \"pod1uid\") : mounting volume failed"
I0624 08:37:04.097232  220043 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:37:04.097416  220043 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0624 08:37:04.097865  220043 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-failed-setup-device-name podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:37:06.097639663 +0000 UTC m=+88.789345435 (durationBeforeRetry 2s). Error: "MapVolume.MapPodDevice failed for volume \"success-and-failed-setup-device-name\" (UniqueName: \"fake-plugin/success-and-failed-setup-device-name\") pod \"pod1\" (UID: \"pod1uid\") : mounting volume failed"
I0624 08:37:06.098453  220043 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:37:06.098632  220043 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0624 08:37:06.099226  220043 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-failed-setup-device-name podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:37:10.09892803 +0000 UTC m=+92.790633804 (durationBeforeRetry 4s). Error: "MapVolume.MapPodDevice failed for volume \"success-and-failed-setup-device-name\" (UniqueName: \"fake-plugin/success-and-failed-setup-device-name\") pod \"pod1\" (UID: \"pod1uid\") : mounting volume failed"
I0624 08:37:10.114598  220043 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:37:10.114804  220043 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0624 08:37:10.115828  220043 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-failed-setup-device-name podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:37:18.115560497 +0000 UTC m=+100.807266267 (durationBeforeRetry 8s). Error: "MapVolume.MapPodDevice failed for volume \"success-and-failed-setup-device-name\" (UniqueName: \"fake-plugin/success-and-failed-setup-device-name\") pod \"pod1\" (UID: \"pod1uid\") : mounting volume failed"
I0624 08:37:12.589507  220043 reconciler.go:196] operationExecutor.UnmountVolume started for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1uid" (UID: "pod1uid") 
I0624 08:37:12.594972  220043 operation_generator.go:1170] UnmapVolume succeeded for volume "fake-plugin/success-and-failed-setup-device-name" (OuterVolumeSpecName: "success-and-failed-setup-device-name") pod "pod1uid" (UID: "pod1uid"). InnerVolumeSpecName "success-and-failed-setup-device-name". PluginName "fake-plugin", VolumeGidValue ""
I0624 08:37:12.597013  220043 reconciler.go:196] operationExecutor.UnmountVolume started for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1uid" (UID: "pod1uid") 
I0624 08:37:12.596316  220043 operation_generator.go:1170] UnmapVolume succeeded for volume "fake-plugin/success-and-failed-setup-device-name" (OuterVolumeSpecName: "success-and-failed-setup-device-name") pod "pod1uid" (UID: "pod1uid"). InnerVolumeSpecName "success-and-failed-setup-device-name". PluginName "fake-plugin", VolumeGidValue ""
I0624 08:37:12.599804  220043 operation_generator.go:1282] UnmapDevice succeeded for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") on node "mynodename" 
I0624 08:37:12.599725  220043 reconciler.go:312] operationExecutor.UnmountDevice started for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") on node "mynodename" 
I0624 08:37:12.602014  220043 reconciler.go:319] Volume detached for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") on node "mynodename" DevicePath "/dev/sdb"
I0624 08:37:12.687137  220043 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "timeout-setup-volume" (UniqueName: "fake-plugin/timeout-setup-volume") pod "pod1" (UID: "pod1uid") 
I0624 08:37:12.687213  220043 reconciler.go:157] Reconciler: start to sync state
E0624 08:37:12.687262  220043 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0624 08:37:12.687520  220043 operation_generator.go:1357] Controller attach succeeded for volume "timeout-setup-volume" (UniqueName: "fake-plugin/timeout-setup-volume") pod "pod1" (UID: "pod1uid") device path: "fake/path"
I0624 08:37:12.688306  220043 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "timeout-setup-volume" (UniqueName: "fake-plugin/timeout-setup-volume") pod "pod1" (UID: "pod1uid") DevicePath "fake/path"
I0624 08:37:12.688453  220043 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "timeout-setup-volume" (UniqueName: "fake-plugin/timeout-setup-volume") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:37:12.688561  220043 operation_generator.go:594] MountVolume.MountDevice succeeded for volume "timeout-setup-volume" (UniqueName: "fake-plugin/timeout-setup-volume") pod "pod1" (UID: "pod1uid") device mount path ""
E0624 08:37:12.688989  220043 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/timeout-setup-volume podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:37:13.188798705 +0000 UTC m=+95.880504488 (durationBeforeRetry 500ms). Error: "MountVolume.SetUp failed for volume \"timeout-setup-volume\" (UniqueName: \"fake-plugin/timeout-setup-volume\") pod \"pod1\" (UID: \"pod1uid\") : time out on setup"
I0624 08:37:12.795120  220043 reconciler.go:196] operationExecutor.UnmountVolume started for volume "timeout-setup-volume" (UniqueName: "fake-plugin/timeout-setup-volume") pod "pod1uid" (UID: "pod1uid") 
I0624 08:37:12.795422  220043 operation_generator.go:797] UnmountVolume.TearDown succeeded for volume "fake-plugin/timeout-setup-volume" (OuterVolumeSpecName: "timeout-setup-volume") pod "pod1uid" (UID: "pod1uid"). InnerVolumeSpecName "timeout-setup-volume". PluginName "fake-plugin", VolumeGidValue ""
I0624 08:37:12.795841  220043 reconciler.go:312] operationExecutor.UnmountDevice started for volume "timeout-setup-volume" (UniqueName: "fake-plugin/timeout-setup-volume") on node "mynodename" 
I0624 08:37:12.796092  220043 operation_generator.go:891] UnmountDevice succeeded for volume "timeout-setup-volume" %!(EXTRA string=UnmountDevice succeeded for volume "timeout-setup-volume" (UniqueName: "fake-plugin/timeout-setup-volume") on node "mynodename" )
I0624 08:37:12.796304  220043 reconciler.go:319] Volume detached for volume "timeout-setup-volume" (UniqueName: "fake-plugin/timeout-setup-volume") on node "mynodename" DevicePath "/dev/sdb"
I0624 08:37:12.932408  220043 operation_generator.go:1357] Controller attach succeeded for volume "fail-setup-volume" (UniqueName: "fake-plugin/fail-setup-volume") pod "pod1" (UID: "pod1uid") device path: "fake/path"
I0624 08:37:12.932282  220043 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "fail-setup-volume" (UniqueName: "fake-plugin/fail-setup-volume") pod "pod1" (UID: "pod1uid") 
I0624 08:37:12.933313  220043 reconciler.go:157] Reconciler: start to sync state
E0624 08:37:12.933374  220043 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0624 08:37:12.934462  220043 reconciler.go:319] Volume detached for volume "fail-setup-volume" (UniqueName: "fake-plugin/fail-setup-volume") on node "mynodename" DevicePath "fake/path"
I0624 08:37:13.037477  220043 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") pod "pod1" (UID: "pod1uid") 
I0624 08:37:13.037585  220043 reconciler.go:157] Reconciler: start to sync state
E0624 08:37:13.037648  220043 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0624 08:37:13.038338  220043 operation_generator.go:1357] Controller attach succeeded for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") pod "pod1" (UID: "pod1uid") device path: "fake/path"
I0624 08:37:13.039277  220043 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") pod "pod1" (UID: "pod1uid") DevicePath "fake/path"
I0624 08:37:13.039435  220043 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:37:13.039575  220043 operation_generator.go:594] MountVolume.MountDevice succeeded for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") pod "pod1" (UID: "pod1uid") device mount path ""
E0624 08:37:13.057747  220043 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/timeout-and-fail-setup-volume podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:37:13.557376394 +0000 UTC m=+96.249082161 (durationBeforeRetry 500ms). Error: "MountVolume.SetUp failed for volume \"timeout-and-fail-setup-volume\" (UniqueName: \"fake-plugin/timeout-and-fail-setup-volume\") pod \"pod1\" (UID: \"pod1uid\") : time out on setup"
I0624 08:37:13.561927  220043 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:37:13.563060  220043 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0624 08:37:13.568771  220043 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/timeout-and-fail-setup-volume podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:37:14.565346221 +0000 UTC m=+97.257051992 (durationBeforeRetry 1s). Error: "MountVolume.SetUp failed for volume \"timeout-and-fail-setup-volume\" (UniqueName: \"fake-plugin/timeout-and-fail-setup-volume\") pod \"pod1\" (UID: \"pod1uid\") : mounting volume failed"
I0624 08:37:14.569778  220043 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:37:14.588650  220043 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0624 08:37:14.619456  220043 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/timeout-and-fail-setup-volume podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:37:16.618929695 +0000 UTC m=+99.310635456 (durationBeforeRetry 2s). Error: "MountVolume.SetUp failed for volume \"timeout-and-fail-setup-volume\" (UniqueName: \"fake-plugin/timeout-and-fail-setup-volume\") pod \"pod1\" (UID: \"pod1uid\") : mounting volume failed"
I0624 08:37:16.636786  220043 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:37:16.636951  220043 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0624 08:37:16.637370  220043 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/timeout-and-fail-setup-volume podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:37:20.637178529 +0000 UTC m=+103.328884297 (durationBeforeRetry 4s). Error: "MountVolume.SetUp failed for volume \"timeout-and-fail-setup-volume\" (UniqueName: \"fake-plugin/timeout-and-fail-setup-volume\") pod \"pod1\" (UID: \"pod1uid\") : mounting volume failed"
I0624 08:37:20.644610  220043 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:37:20.644793  220043 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0624 08:37:20.649443  220043 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/timeout-and-fail-setup-volume podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:37:28.645018393 +0000 UTC m=+111.336724164 (durationBeforeRetry 8s). Error: "MountVolume.SetUp failed for volume \"timeout-and-fail-setup-volume\" (UniqueName: \"fake-plugin/timeout-and-fail-setup-volume\") pod \"pod1\" (UID: \"pod1uid\") : mounting volume failed"
I0624 08:37:23.156424  220043 reconciler.go:312] operationExecutor.UnmountDevice started for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") on node "mynodename" 
I0624 08:37:23.156883  220043 operation_generator.go:891] UnmountDevice succeeded for volume "timeout-and-fail-setup-volume" %!(EXTRA string=UnmountDevice succeeded for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") on node "mynodename" )
I0624 08:37:23.158283  220043 reconciler.go:319] Volume detached for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") on node "mynodename" DevicePath "/dev/sdb"
I0624 08:37:23.261458  220043 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1" (UID: "pod1uid") 
I0624 08:37:23.261538  220043 reconciler.go:157] Reconciler: start to sync state
E0624 08:37:23.261612  220043 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0624 08:37:23.261860  220043 operation_generator.go:1357] Controller attach succeeded for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1" (UID: "pod1uid") device path: "fake/path"
I0624 08:37:23.262572  220043 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1" (UID: "pod1uid") DevicePath "fake/path"
I0624 08:37:23.262719  220043 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:37:23.262863  220043 operation_generator.go:594] MountVolume.MountDevice succeeded for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1" (UID: "pod1uid") device mount path ""
I0624 08:37:23.404675  220043 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:37:23.415420  220043 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0624 08:37:23.415937  220043 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-timeout-setup-volume-name podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:37:23.915708529 +0000 UTC m=+106.607414310 (durationBeforeRetry 500ms). Error: "MountVolume.SetUp failed for volume \"success-and-timeout-setup-volume-name\" (UniqueName: \"fake-plugin/success-and-timeout-setup-volume-name\") pod \"pod1\" (UID: \"pod1uid\") : time out on setup"
I0624 08:37:23.924516  220043 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:37:23.924678  220043 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0624 08:37:23.925080  220043 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-timeout-setup-volume-name podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:37:24.924894249 +0000 UTC m=+107.616600013 (durationBeforeRetry 1s). Error: "MountVolume.SetUp failed for volume \"success-and-timeout-setup-volume-name\" (UniqueName: \"fake-plugin/success-and-timeout-setup-volume-name\") pod \"pod1\" (UID: \"pod1uid\") : time out on setup"
I0624 08:37:24.960664  220043 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:37:24.960839  220043 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0624 08:37:24.961506  220043 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-timeout-setup-volume-name podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:37:26.961036165 +0000 UTC m=+109.652741929 (durationBeforeRetry 2s). Error: "MountVolume.SetUp failed for volume \"success-and-timeout-setup-volume-name\" (UniqueName: \"fake-plugin/success-and-timeout-setup-volume-name\") pod \"pod1\" (UID: \"pod1uid\") : time out on setup"
I0624 08:37:26.964802  220043 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:37:26.964971  220043 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0624 08:37:26.965375  220043 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-timeout-setup-volume-name podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:37:30.965176454 +0000 UTC m=+113.656882229 (durationBeforeRetry 4s). Error: "MountVolume.SetUp failed for volume \"success-and-timeout-setup-volume-name\" (UniqueName: \"fake-plugin/success-and-timeout-setup-volume-name\") pod \"pod1\" (UID: \"pod1uid\") : time out on setup"
I0624 08:37:30.966940  220043 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:37:30.968138  220043 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0624 08:37:30.968675  220043 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-timeout-setup-volume-name podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:37:38.968448728 +0000 UTC m=+121.660154497 (durationBeforeRetry 8s). Error: "MountVolume.SetUp failed for volume \"success-and-timeout-setup-volume-name\" (UniqueName: \"fake-plugin/success-and-timeout-setup-volume-name\") pod \"pod1\" (UID: \"pod1uid\") : time out on setup"
I0624 08:37:33.405001  220043 reconciler.go:196] operationExecutor.UnmountVolume started for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1uid" (UID: "pod1uid") 
I0624 08:37:33.405368  220043 operation_generator.go:797] UnmountVolume.TearDown succeeded for volume "fake-plugin/success-and-timeout-setup-volume-name" (OuterVolumeSpecName: "success-and-timeout-setup-volume-name") pod "pod1uid" (UID: "pod1uid"). InnerVolumeSpecName "success-and-timeout-setup-volume-name". PluginName "fake-plugin", VolumeGidValue ""
I0624 08:37:33.405870  220043 reconciler.go:312] operationExecutor.UnmountDevice started for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") on node "mynodename" 
I0624 08:37:33.406457  220043 operation_generator.go:891] UnmountDevice succeeded for volume "success-and-timeout-setup-volume-name" %!(EXTRA string=UnmountDevice succeeded for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") on node "mynodename" )
I0624 08:37:33.406753  220043 reconciler.go:319] Volume detached for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") on node "mynodename" DevicePath "/dev/sdb"
I0624 08:37:33.506377  220043 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1" (UID: "pod1uid") 
I0624 08:37:33.506464  220043 reconciler.go:157] Reconciler: start to sync state
E0624 08:37:33.506522  220043 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0624 08:37:33.507073  220043 operation_generator.go:1357] Controller attach succeeded for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1" (UID: "pod1uid") device path: "fake/path"
I0624 08:37:33.508953  220043 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1" (UID: "pod1uid") DevicePath "fake/path"
I0624 08:37:33.509498  220043 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:37:33.510010  220043 operation_generator.go:594] MountVolume.MountDevice succeeded for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1" (UID: "pod1uid") device mount path ""
I0624 08:37:33.628739  220043 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:37:33.628911  220043 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0624 08:37:33.629331  220043 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-failed-setup-device-name podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:37:34.129142287 +0000 UTC m=+116.820848035 (durationBeforeRetry 500ms). Error: "MountVolume.SetUp failed for volume \"success-and-failed-setup-device-name\" (UniqueName: \"fake-plugin/success-and-failed-setup-device-name\") pod \"pod1\" (UID: \"pod1uid\") : mounting volume failed"
I0624 08:37:34.137472  220043 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:37:34.139843  220043 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0624 08:37:34.150834  220043 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-failed-setup-device-name podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:37:35.143654145 +0000 UTC m=+117.835359919 (durationBeforeRetry 1s). Error: "MountVolume.SetUp failed for volume \"success-and-failed-setup-device-name\" (UniqueName: \"fake-plugin/success-and-failed-setup-device-name\") pod \"pod1\" (UID: \"pod1uid\") : mounting volume failed"
I0624 08:37:35.151793  220043 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:37:35.152015  220043 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0624 08:37:35.165032  220043 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-failed-setup-device-name podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:37:37.152273763 +0000 UTC m=+119.843979544 (durationBeforeRetry 2s). Error: "MountVolume.SetUp failed for volume \"success-and-failed-setup-device-name\" (UniqueName: \"fake-plugin/success-and-failed-setup-device-name\") pod \"pod1\" (UID: \"pod1uid\") : mounting volume failed"
I0624 08:37:37.153290  220043 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:37:37.153473  220043 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0624 08:37:37.154164  220043 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-failed-setup-device-name podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:37:41.153710513 +0000 UTC m=+123.845416286 (durationBeforeRetry 4s). Error: "MountVolume.SetUp failed for volume \"success-and-failed-setup-device-name\" (UniqueName: \"fake-plugin/success-and-failed-setup-device-name\") pod \"pod1\" (UID: \"pod1uid\") : mounting volume failed"
I0624 08:37:41.170520  220043 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:37:41.170729  220043 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0624 08:37:41.171292  220043 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-failed-setup-device-name podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:37:49.171007258 +0000 UTC m=+131.862713022 (durationBeforeRetry 8s). Error: "MountVolume.SetUp failed for volume \"success-and-failed-setup-device-name\" (UniqueName: \"fake-plugin/success-and-failed-setup-device-name\") pod \"pod1\" (UID: \"pod1uid\") : mounting volume failed"
I0624 08:37:43.692976  220043 reconciler.go:196] operationExecutor.UnmountVolume started for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1uid" (UID: "pod1uid") 
I0624 08:37:43.693474  220043 operation_generator.go:797] UnmountVolume.TearDown succeeded for volume "fake-plugin/success-and-failed-setup-device-name" (OuterVolumeSpecName: "success-and-failed-setup-device-name") pod "pod1uid" (UID: "pod1uid"). InnerVolumeSpecName "success-and-failed-setup-device-name". PluginName "fake-plugin", VolumeGidValue ""
I0624 08:37:43.694089  220043 reconciler.go:312] operationExecutor.UnmountDevice started for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") on node "mynodename" 
I0624 08:37:43.694416  220043 operation_generator.go:891] UnmountDevice succeeded for volume "success-and-failed-setup-device-name" %!(EXTRA string=UnmountDevice succeeded for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") on node "mynodename" )
I0624 08:37:43.694648  220043 reconciler.go:319] Volume detached for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") on node "mynodename" DevicePath "/dev/sdb"
--- FAIL: Test_UncertainVolumeMountState (62.14s)
    --- FAIL: Test_UncertainVolumeMountState/failed_operation_should_result_in_not-mounted_volume_[Filesystem] (0.11s)
        reconciler_test.go:1523: Error verifying UnMountDeviceCallCount: Expected DeviceUnmount Call 1, got 0
I0624 08:37:43.736337  220043 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") 
I0624 08:37:43.736414  220043 reconciler.go:157] Reconciler: start to sync state
E0624 08:37:43.736472  220043 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0624 08:37:43.736769  220043 operation_generator.go:1357] Controller attach succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") device path: "/fake/path"
I0624 08:37:43.737508  220043 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") DevicePath "/fake/path"
I0624 08:37:43.737620  220043 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:37:43.737722  220043 operation_generator.go:594] MountVolume.MountDevice succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") device mount path ""
I0624 08:37:43.865060  220043 reconciler.go:196] operationExecutor.UnmountVolume started for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1uid" (UID: "pod1uid") 
I0624 08:37:43.865309  220043 operation_generator.go:797] UnmountVolume.TearDown succeeded for volume "fake-plugin/fake-device1" (OuterVolumeSpecName: "volume-name") pod "pod1uid" (UID: "pod1uid"). InnerVolumeSpecName "volume-name". PluginName "fake-plugin", VolumeGidValue ""
I0624 08:37:43.865996  220043 reconciler.go:312] operationExecutor.UnmountDevice started for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") on node "mynodename" 
I0624 08:37:43.866410  220043 reconciler_test.go:1798] UnmountDevice called
I0624 08:37:43.866612  220043 operation_generator.go:891] UnmountDevice succeeded for volume "volume-name" %!(EXTRA string=UnmountDevice succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") on node "mynodename" )
I0624 08:37:43.867560  220043 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:37:43.867679  220043 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:37:43.896008  220043 operation_generator.go:594] MountVolume.MountDevice succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") device mount path ""
FAIL

				from junit_bazel.xml

Find pod1 mentions in log files | View test history on testgrid


//pkg/kubelet/volumemanager/reconciler/go_default_test:run_1_of_2 0.00s

bazel test //pkg/kubelet/volumemanager/reconciler/go_default_test:run_1_of_2
exec ${PAGER:-/usr/bin/less} "$0" || exit 1
Executing tests from //pkg/kubelet/volumemanager/reconciler:go_default_test
-----------------------------------------------------------------------------
I0624 08:35:37.400941  220043 reconciler.go:157] Reconciler: start to sync state
E0624 08:35:37.401110  220043 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0624 08:35:37.402020  220043 reconciler.go:244] operationExecutor.AttachVolume started for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") 
I0624 08:35:37.402094  220043 reconciler.go:157] Reconciler: start to sync state
E0624 08:35:37.402140  220043 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0624 08:35:37.402223  220043 operation_generator.go:360] AttachVolume.Attach succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") from node "mynodename" 
I0624 08:35:37.402907  220043 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") DevicePath "/dev/vdb-test"
I0624 08:35:37.403034  220043 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:35:37.403132  220043 operation_generator.go:594] MountVolume.MountDevice succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") device mount path ""
I0624 08:35:37.502773  220043 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") 
I0624 08:35:37.502850  220043 reconciler.go:157] Reconciler: start to sync state
E0624 08:35:37.502902  220043 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0624 08:35:37.503147  220043 operation_generator.go:1357] Controller attach succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") device path: "/fake/path"
I0624 08:35:37.503701  220043 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") DevicePath "/fake/path"
I0624 08:35:37.503849  220043 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:35:37.503952  220043 operation_generator.go:594] MountVolume.MountDevice succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") device mount path ""
I0624 08:35:37.504604  220043 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:35:37.504721  220043 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:35:37.605037  220043 reconciler.go:244] operationExecutor.AttachVolume started for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") 
I0624 08:35:37.605118  220043 reconciler.go:157] Reconciler: start to sync state
E0624 08:35:37.605178  220043 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0624 08:35:37.605531  220043 operation_generator.go:360] AttachVolume.Attach succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") from node "mynodename" 
I0624 08:35:37.606168  220043 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") DevicePath "/dev/vdb-test"
I0624 08:35:37.606284  220043 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:35:37.606393  220043 operation_generator.go:594] MountVolume.MountDevice succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") device mount path ""
I0624 08:35:37.705773  220043 reconciler.go:196] operationExecutor.UnmountVolume started for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1uid" (UID: "pod1uid") 
I0624 08:35:37.705979  220043 operation_generator.go:797] UnmountVolume.TearDown succeeded for volume "fake-plugin/fake-device1" (OuterVolumeSpecName: "volume-name") pod "pod1uid" (UID: "pod1uid"). InnerVolumeSpecName "volume-name". PluginName "fake-plugin", VolumeGidValue ""
I0624 08:35:37.706521  220043 reconciler.go:312] operationExecutor.UnmountDevice started for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") on node "mynodename" 
I0624 08:35:37.706753  220043 operation_generator.go:891] UnmountDevice succeeded for volume "volume-name" %!(EXTRA string=UnmountDevice succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") on node "mynodename" )
I0624 08:35:37.707181  220043 reconciler.go:333] operationExecutor.DetachVolume started for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") on node "mynodename" 
I0624 08:35:37.707304  220043 operation_generator.go:470] DetachVolume.Detach succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") on node "mynodename" 
I0624 08:35:37.813297  220043 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") 
I0624 08:35:37.813378  220043 reconciler.go:157] Reconciler: start to sync state
E0624 08:35:37.813439  220043 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0624 08:35:37.814037  220043 operation_generator.go:1357] Controller attach succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") device path: "/fake/path"
I0624 08:35:37.814806  220043 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") DevicePath "/fake/path"
I0624 08:35:37.814974  220043 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:35:37.815199  220043 operation_generator.go:594] MountVolume.MountDevice succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") device mount path ""
I0624 08:35:37.816062  220043 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:35:37.816271  220043 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:35:37.926641  220043 reconciler.go:196] operationExecutor.UnmountVolume started for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1uid" (UID: "pod1uid") 
I0624 08:35:37.926872  220043 operation_generator.go:797] UnmountVolume.TearDown succeeded for volume "fake-plugin/fake-device1" (OuterVolumeSpecName: "volume-name") pod "pod1uid" (UID: "pod1uid"). InnerVolumeSpecName "volume-name". PluginName "fake-plugin", VolumeGidValue ""
I0624 08:35:37.927331  220043 reconciler.go:312] operationExecutor.UnmountDevice started for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") on node "mynodename" 
I0624 08:35:37.927529  220043 operation_generator.go:891] UnmountDevice succeeded for volume "volume-name" %!(EXTRA string=UnmountDevice succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") on node "mynodename" )
I0624 08:35:37.927698  220043 reconciler.go:319] Volume detached for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") on node "mynodename" DevicePath "/dev/sdb"
I0624 08:35:38.027692  220043 reconciler.go:244] operationExecutor.AttachVolume started for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") 
I0624 08:35:38.027775  220043 reconciler.go:157] Reconciler: start to sync state
E0624 08:35:38.027824  220043 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0624 08:35:38.028100  220043 operation_generator.go:360] AttachVolume.Attach succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") from node "mynodename" 
I0624 08:35:38.028652  220043 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") DevicePath "/dev/vdb-test"
I0624 08:35:38.028762  220043 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:35:38.128592  220043 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") 
I0624 08:35:38.128659  220043 reconciler.go:157] Reconciler: start to sync state
E0624 08:35:38.128710  220043 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0624 08:35:38.128979  220043 operation_generator.go:1357] Controller attach succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") device path: "/fake/path"
I0624 08:35:38.129537  220043 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") DevicePath "/fake/path"
I0624 08:35:38.129662  220043 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:35:38.229200  220043 reconciler.go:244] operationExecutor.AttachVolume started for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") 
I0624 08:35:38.229268  220043 reconciler.go:157] Reconciler: start to sync state
E0624 08:35:38.229314  220043 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0624 08:35:38.229535  220043 operation_generator.go:360] AttachVolume.Attach succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") from node "mynodename" 
I0624 08:35:38.230124  220043 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") DevicePath "/dev/vdb-test"
I0624 08:35:38.230232  220043 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:35:38.335124  220043 reconciler.go:196] operationExecutor.UnmountVolume started for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1uid" (UID: "pod1uid") 
I0624 08:35:38.335434  220043 operation_generator.go:1170] UnmapVolume succeeded for volume "fake-plugin/fake-device1" (OuterVolumeSpecName: "volume-name") pod "pod1uid" (UID: "pod1uid"). InnerVolumeSpecName "volume-name". PluginName "fake-plugin", VolumeGidValue ""
I0624 08:35:38.335884  220043 reconciler.go:312] operationExecutor.UnmountDevice started for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") on node "mynodename" 
I0624 08:35:38.336063  220043 operation_generator.go:1282] UnmapDevice succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") on node "mynodename" 
I0624 08:35:38.336416  220043 reconciler.go:333] operationExecutor.DetachVolume started for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") on node "mynodename" 
I0624 08:35:38.336601  220043 operation_generator.go:470] DetachVolume.Detach succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") on node "mynodename" 
I0624 08:35:38.436617  220043 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") 
I0624 08:35:38.436697  220043 operation_generator.go:1357] Controller attach succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") device path: "/fake/path"
I0624 08:35:38.436952  220043 reconciler.go:157] Reconciler: start to sync state
E0624 08:35:38.437048  220043 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0624 08:35:38.437717  220043 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") DevicePath "/fake/path"
I0624 08:35:38.438027  220043 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:35:38.537542  220043 reconciler.go:196] operationExecutor.UnmountVolume started for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1uid" (UID: "pod1uid") 
I0624 08:35:38.537681  220043 operation_generator.go:1170] UnmapVolume succeeded for volume "fake-plugin/fake-device1" (OuterVolumeSpecName: "volume-name") pod "pod1uid" (UID: "pod1uid"). InnerVolumeSpecName "volume-name". PluginName "fake-plugin", VolumeGidValue ""
I0624 08:35:38.538079  220043 reconciler.go:312] operationExecutor.UnmountDevice started for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") on node "mynodename" 
I0624 08:35:38.538287  220043 operation_generator.go:1282] UnmapDevice succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") on node "mynodename" 
I0624 08:35:38.538432  220043 reconciler.go:319] Volume detached for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") on node "mynodename" DevicePath "/dev/sdb"
I0624 08:35:38.645594  220043 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "pv" (UniqueName: "fake-plugin/pv") pod "pod1" (UID: "pod1uid") 
I0624 08:35:38.645691  220043 reconciler.go:157] Reconciler: start to sync state
E0624 08:35:38.645743  220043 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0624 08:35:38.646270  220043 operation_generator.go:1357] Controller attach succeeded for volume "pv" (UniqueName: "fake-plugin/pv") pod "pod1" (UID: "pod1uid") device path: "fake/path"
I0624 08:35:38.646947  220043 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "pv" (UniqueName: "fake-plugin/pv") pod "pod1" (UID: "pod1uid") DevicePath "fake/path"
I0624 08:35:38.647054  220043 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "pv" (UniqueName: "fake-plugin/pv") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:35:38.647139  220043 operation_generator.go:594] MountVolume.MountDevice succeeded for volume "pv" (UniqueName: "fake-plugin/pv") pod "pod1" (UID: "pod1uid") device mount path ""
I0624 08:35:38.753516  220043 operation_generator.go:1683] MountVolume.NodeExpandVolume succeeded for volume "pv" (UniqueName: "fake-plugin/pv") pod "pod1" (UID: "pod1uid") 
I0624 08:35:38.795231  220043 operation_generator.go:1683] MountVolume.NodeExpandVolume succeeded for volume "pv" (UniqueName: "fake-plugin/pv") pod "pod1" (UID: "pod1uid") 
I0624 08:35:38.854546  220043 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "pv" (UniqueName: "fake-plugin/pv") pod "pod1" (UID: "pod1uid") 
I0624 08:35:38.854669  220043 reconciler.go:157] Reconciler: start to sync state
I0624 08:35:38.854734  220043 operation_generator.go:1357] Controller attach succeeded for volume "pv" (UniqueName: "fake-plugin/pv") pod "pod1" (UID: "pod1uid") device path: "fake/path"
E0624 08:35:38.854735  220043 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0624 08:35:38.856768  220043 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "pv" (UniqueName: "fake-plugin/pv") pod "pod1" (UID: "pod1uid") DevicePath "fake/path"
I0624 08:35:38.856917  220043 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "pv" (UniqueName: "fake-plugin/pv") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:35:38.962891  220043 operation_generator.go:1683] MountVolume.NodeExpandVolume succeeded for volume "pv" (UniqueName: "fake-plugin/pv") pod "pod1" (UID: "pod1uid") 
I0624 08:35:39.063584  220043 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "fail-expansion-in-use" (UniqueName: "fake-plugin/fail-expansion-in-use") pod "pod1" (UID: "pod1uid") 
I0624 08:35:39.063691  220043 reconciler.go:157] Reconciler: start to sync state
I0624 08:35:39.087963  220043 operation_generator.go:1357] Controller attach succeeded for volume "fail-expansion-in-use" (UniqueName: "fake-plugin/fail-expansion-in-use") pod "pod1" (UID: "pod1uid") device path: "fake/path"
E0624 08:35:39.063764  220043 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0624 08:35:39.088925  220043 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "fail-expansion-in-use" (UniqueName: "fake-plugin/fail-expansion-in-use") pod "pod1" (UID: "pod1uid") DevicePath "fake/path"
I0624 08:35:39.089053  220043 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "fail-expansion-in-use" (UniqueName: "fake-plugin/fail-expansion-in-use") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:35:39.089157  220043 operation_generator.go:594] MountVolume.MountDevice succeeded for volume "fail-expansion-in-use" (UniqueName: "fake-plugin/fail-expansion-in-use") pod "pod1" (UID: "pod1uid") device mount path ""
E0624 08:35:39.089399  220043 operation_generator.go:1669] MountVolume.NodeExapndVolume failed with %!!(MISSING)v(MISSING) for volume "fail-expansion-in-use" (UniqueName: "fake-plugin/fail-expansion-in-use") pod "pod1" (UID: "pod1uid") : volume-in-use
I0624 08:35:39.166082  220043 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "timeout-mount-device-volume" (UniqueName: "fake-plugin/timeout-mount-device-volume") pod "pod1" (UID: "pod1uid") 
I0624 08:35:39.166157  220043 reconciler.go:157] Reconciler: start to sync state
E0624 08:35:39.166209  220043 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0624 08:35:39.166447  220043 operation_generator.go:1357] Controller attach succeeded for volume "timeout-mount-device-volume" (UniqueName: "fake-plugin/timeout-mount-device-volume") pod "pod1" (UID: "pod1uid") device path: "fake/path"
I0624 08:35:39.167080  220043 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "timeout-mount-device-volume" (UniqueName: "fake-plugin/timeout-mount-device-volume") pod "pod1" (UID: "pod1uid") DevicePath "fake/path"
I0624 08:35:39.167218  220043 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "timeout-mount-device-volume" (UniqueName: "fake-plugin/timeout-mount-device-volume") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0624 08:35:39.167632  220043 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/timeout-mount-device-volume podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:35:39.667411359 +0000 UTC m=+2.359117127 (durationBeforeRetry 500ms). Error: "MapVolume.SetUpDevice failed for volume \"timeout-mount-device-volume\" (UniqueName: \"fake-plugin/timeout-mount-device-volume\") pod \"pod1\" (UID: \"pod1uid\") : mount failed"
I0624 08:35:39.289043  220043 reconciler.go:312] operationExecutor.UnmountDevice started for volume "timeout-mount-device-volume" (UniqueName: "fake-plugin/timeout-mount-device-volume") on node "mynodename" 
I0624 08:35:39.289277  220043 operation_generator.go:1282] UnmapDevice succeeded for volume "timeout-mount-device-volume" (UniqueName: "fake-plugin/timeout-mount-device-volume") on node "mynodename" 
I0624 08:35:39.289531  220043 reconciler.go:319] Volume detached for volume "timeout-mount-device-volume" (UniqueName: "fake-plugin/timeout-mount-device-volume") on node "mynodename" DevicePath "/dev/sdb"
I0624 08:35:39.390102  220043 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "fail-mount-device-volume-name" (UniqueName: "fake-plugin/fail-mount-device-volume-name") pod "pod1" (UID: "pod1uid") 
I0624 08:35:39.390188  220043 reconciler.go:157] Reconciler: start to sync state
E0624 08:35:39.390247  220043 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0624 08:35:39.390544  220043 operation_generator.go:1357] Controller attach succeeded for volume "fail-mount-device-volume-name" (UniqueName: "fake-plugin/fail-mount-device-volume-name") pod "pod1" (UID: "pod1uid") device path: "fake/path"
I0624 08:35:39.391103  220043 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "fail-mount-device-volume-name" (UniqueName: "fake-plugin/fail-mount-device-volume-name") pod "pod1" (UID: "pod1uid") DevicePath "fake/path"
I0624 08:35:39.391244  220043 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "fail-mount-device-volume-name" (UniqueName: "fake-plugin/fail-mount-device-volume-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0624 08:35:39.391706  220043 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/fail-mount-device-volume-name podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:35:39.891506432 +0000 UTC m=+2.583212197 (durationBeforeRetry 500ms). Error: "MapVolume.SetUpDevice failed for volume \"fail-mount-device-volume-name\" (UniqueName: \"fake-plugin/fail-mount-device-volume-name\") pod \"pod1\" (UID: \"pod1uid\") : error mapping disk: fail-mount-device-volume-name"
I0624 08:35:39.505351  220043 reconciler.go:319] Volume detached for volume "fail-mount-device-volume-name" (UniqueName: "fake-plugin/fail-mount-device-volume-name") on node "mynodename" DevicePath "fake/path"
I0624 08:35:39.633406  220043 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "timeout-and-fail-mount-device-name" (UniqueName: "fake-plugin/timeout-and-fail-mount-device-name") pod "pod1" (UID: "pod1uid") 
I0624 08:35:39.633938  220043 reconciler.go:157] Reconciler: start to sync state
E0624 08:35:39.634030  220043 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0624 08:35:39.634881  220043 operation_generator.go:1357] Controller attach succeeded for volume "timeout-and-fail-mount-device-name" (UniqueName: "fake-plugin/timeout-and-fail-mount-device-name") pod "pod1" (UID: "pod1uid") device path: "fake/path"
I0624 08:35:39.635744  220043 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "timeout-and-fail-mount-device-name" (UniqueName: "fake-plugin/timeout-and-fail-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "fake/path"
I0624 08:35:39.635902  220043 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "timeout-and-fail-mount-device-name" (UniqueName: "fake-plugin/timeout-and-fail-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0624 08:35:39.636459  220043 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/timeout-and-fail-mount-device-name podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:35:40.136120019 +0000 UTC m=+2.827825798 (durationBeforeRetry 500ms). Error: "MapVolume.SetUpDevice failed for volume \"timeout-and-fail-mount-device-name\" (UniqueName: \"fake-plugin/timeout-and-fail-mount-device-name\") pod \"pod1\" (UID: \"pod1uid\") : timed out mounting error"
I0624 08:35:40.138397  220043 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "timeout-and-fail-mount-device-name" (UniqueName: "fake-plugin/timeout-and-fail-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:35:40.139144  220043 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "timeout-and-fail-mount-device-name" (UniqueName: "fake-plugin/timeout-and-fail-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0624 08:35:40.154083  220043 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/timeout-and-fail-mount-device-name podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:35:41.140360237 +0000 UTC m=+3.832066013 (durationBeforeRetry 1s). Error: "MapVolume.SetUpDevice failed for volume \"timeout-and-fail-mount-device-name\" (UniqueName: \"fake-plugin/timeout-and-fail-mount-device-name\") pod \"pod1\" (UID: \"pod1uid\") : error mapping disk: timeout-and-fail-mount-device-name"
I0624 08:35:41.143499  220043 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "timeout-and-fail-mount-device-name" (UniqueName: "fake-plugin/timeout-and-fail-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:35:41.143721  220043 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "timeout-and-fail-mount-device-name" (UniqueName: "fake-plugin/timeout-and-fail-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0624 08:35:41.144190  220043 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/timeout-and-fail-mount-device-name podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:35:43.143987511 +0000 UTC m=+5.835693286 (durationBeforeRetry 2s). Error: "MapVolume.SetUpDevice failed for volume \"timeout-and-fail-mount-device-name\" (UniqueName: \"fake-plugin/timeout-and-fail-mount-device-name\") pod \"pod1\" (UID: \"pod1uid\") : error mapping disk: timeout-and-fail-mount-device-name"
I0624 08:35:43.144436  220043 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "timeout-and-fail-mount-device-name" (UniqueName: "fake-plugin/timeout-and-fail-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:35:43.144605  220043 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "timeout-and-fail-mount-device-name" (UniqueName: "fake-plugin/timeout-and-fail-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0624 08:35:43.145004  220043 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/timeout-and-fail-mount-device-name podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:35:47.144810387 +0000 UTC m=+9.836516153 (durationBeforeRetry 4s). Error: "MapVolume.SetUpDevice failed for volume \"timeout-and-fail-mount-device-name\" (UniqueName: \"fake-plugin/timeout-and-fail-mount-device-name\") pod \"pod1\" (UID: \"pod1uid\") : error mapping disk: timeout-and-fail-mount-device-name"
I0624 08:35:47.147176  220043 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "timeout-and-fail-mount-device-name" (UniqueName: "fake-plugin/timeout-and-fail-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:35:47.148640  220043 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "timeout-and-fail-mount-device-name" (UniqueName: "fake-plugin/timeout-and-fail-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0624 08:35:47.149053  220043 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/timeout-and-fail-mount-device-name podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:35:55.148862883 +0000 UTC m=+17.840568649 (durationBeforeRetry 8s). Error: "MapVolume.SetUpDevice failed for volume \"timeout-and-fail-mount-device-name\" (UniqueName: \"fake-plugin/timeout-and-fail-mount-device-name\") pod \"pod1\" (UID: \"pod1uid\") : error mapping disk: timeout-and-fail-mount-device-name"
I0624 08:35:49.749094  220043 reconciler.go:319] Volume detached for volume "timeout-and-fail-mount-device-name" (UniqueName: "fake-plugin/timeout-and-fail-mount-device-name") on node "mynodename" DevicePath "/dev/sdb"
I0624 08:35:49.874803  220043 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "success-and-timeout-device-name" (UniqueName: "fake-plugin/success-and-timeout-device-name") pod "pod1" (UID: "pod1uid") 
I0624 08:35:49.874891  220043 reconciler.go:157] Reconciler: start to sync state
E0624 08:35:49.874943  220043 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0624 08:35:49.875262  220043 operation_generator.go:1357] Controller attach succeeded for volume "success-and-timeout-device-name" (UniqueName: "fake-plugin/success-and-timeout-device-name") pod "pod1" (UID: "pod1uid") device path: "fake/path"
I0624 08:35:49.875845  220043 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "success-and-timeout-device-name" (UniqueName: "fake-plugin/success-and-timeout-device-name") pod "pod1" (UID: "pod1uid") DevicePath "fake/path"
I0624 08:35:49.876008  220043 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "success-and-timeout-device-name" (UniqueName: "fake-plugin/success-and-timeout-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:35:49.996515  220043 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "success-and-timeout-device-name" (UniqueName: "fake-plugin/success-and-timeout-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:35:50.002856  220043 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "success-and-timeout-device-name" (UniqueName: "fake-plugin/success-and-timeout-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0624 08:35:50.003939  220043 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-timeout-device-name podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:35:50.503522187 +0000 UTC m=+13.195227996 (durationBeforeRetry 500ms). Error: "MapVolume.SetUpDevice failed for volume \"success-and-timeout-device-name\" (UniqueName: \"fake-plugin/success-and-timeout-device-name\") pod \"pod1\" (UID: \"pod1uid\") : error mounting state"
I0624 08:35:50.503904  220043 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "success-and-timeout-device-name" (UniqueName: "fake-plugin/success-and-timeout-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:35:50.504072  220043 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "success-and-timeout-device-name" (UniqueName: "fake-plugin/success-and-timeout-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0624 08:35:50.504484  220043 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-timeout-device-name podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:35:51.504261881 +0000 UTC m=+14.195967646 (durationBeforeRetry 1s). Error: "MapVolume.SetUpDevice failed for volume \"success-and-timeout-device-name\" (UniqueName: \"fake-plugin/success-and-timeout-device-name\") pod \"pod1\" (UID: \"pod1uid\") : error mounting state"
I0624 08:35:51.505518  220043 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "success-and-timeout-device-name" (UniqueName: "fake-plugin/success-and-timeout-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:35:51.505706  220043 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "success-and-timeout-device-name" (UniqueName: "fake-plugin/success-and-timeout-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0624 08:35:51.506080  220043 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-timeout-device-name podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:35:53.505894831 +0000 UTC m=+16.197600594 (durationBeforeRetry 2s). Error: "MapVolume.SetUpDevice failed for volume \"success-and-timeout-device-name\" (UniqueName: \"fake-plugin/success-and-timeout-device-name\") pod \"pod1\" (UID: \"pod1uid\") : error mounting state"
I0624 08:35:53.506282  220043 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "success-and-timeout-device-name" (UniqueName: "fake-plugin/success-and-timeout-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:35:53.506464  220043 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "success-and-timeout-device-name" (UniqueName: "fake-plugin/success-and-timeout-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0624 08:35:53.506873  220043 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-timeout-device-name podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:35:57.506665125 +0000 UTC m=+20.198370892 (durationBeforeRetry 4s). Error: "MapVolume.SetUpDevice failed for volume \"success-and-timeout-device-name\" (UniqueName: \"fake-plugin/success-and-timeout-device-name\") pod \"pod1\" (UID: \"pod1uid\") : error mounting state"
I0624 08:35:57.507231  220043 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "success-and-timeout-device-name" (UniqueName: "fake-plugin/success-and-timeout-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:35:57.507390  220043 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "success-and-timeout-device-name" (UniqueName: "fake-plugin/success-and-timeout-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0624 08:35:57.507797  220043 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-timeout-device-name podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:36:05.507606914 +0000 UTC m=+28.199312680 (durationBeforeRetry 8s). Error: "MapVolume.SetUpDevice failed for volume \"success-and-timeout-device-name\" (UniqueName: \"fake-plugin/success-and-timeout-device-name\") pod \"pod1\" (UID: \"pod1uid\") : error mounting state"
I0624 08:36:00.009646  220043 reconciler.go:196] operationExecutor.UnmountVolume started for volume "success-and-timeout-device-name" (UniqueName: "fake-plugin/success-and-timeout-device-name") pod "pod1uid" (UID: "pod1uid") 
I0624 08:36:00.009186  220043 operation_generator.go:1170] UnmapVolume succeeded for volume "fake-plugin/success-and-timeout-device-name" (OuterVolumeSpecName: "success-and-timeout-device-name") pod "pod1uid" (UID: "pod1uid"). InnerVolumeSpecName "success-and-timeout-device-name". PluginName "fake-plugin", VolumeGidValue ""
I0624 08:36:00.010880  220043 reconciler.go:312] operationExecutor.UnmountDevice started for volume "success-and-timeout-device-name" (UniqueName: "fake-plugin/success-and-timeout-device-name") on node "mynodename" 
I0624 08:36:00.010889  220043 operation_generator.go:1282] UnmapDevice succeeded for volume "success-and-timeout-device-name" (UniqueName: "fake-plugin/success-and-timeout-device-name") on node "mynodename" 
I0624 08:36:00.011222  220043 reconciler.go:319] Volume detached for volume "success-and-timeout-device-name" (UniqueName: "fake-plugin/success-and-timeout-device-name") on node "mynodename" DevicePath "/dev/sdb"
I0624 08:36:00.109048  220043 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") pod "pod1" (UID: "pod1uid") 
I0624 08:36:00.109128  220043 reconciler.go:157] Reconciler: start to sync state
E0624 08:36:00.109184  220043 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0624 08:36:00.109411  220043 operation_generator.go:1357] Controller attach succeeded for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") pod "pod1" (UID: "pod1uid") device path: "fake/path"
I0624 08:36:00.110434  220043 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "fake/path"
I0624 08:36:00.110590  220043 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:36:00.224073  220043 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:36:00.224235  220043 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0624 08:36:00.224632  220043 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-failed-mount-device-name podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:36:00.724440171 +0000 UTC m=+23.416145936 (durationBeforeRetry 500ms). Error: "MapVolume.SetUpDevice failed for volume \"success-and-failed-mount-device-name\" (UniqueName: \"fake-plugin/success-and-failed-mount-device-name\") pod \"pod1\" (UID: \"pod1uid\") : error mapping disk: success-and-failed-mount-device-name"
I0624 08:36:00.749009  220043 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:36:00.749204  220043 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0624 08:36:00.749605  220043 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-failed-mount-device-name podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:36:01.749401752 +0000 UTC m=+24.441107517 (durationBeforeRetry 1s). Error: "MapVolume.SetUpDevice failed for volume \"success-and-failed-mount-device-name\" (UniqueName: \"fake-plugin/success-and-failed-mount-device-name\") pod \"pod1\" (UID: \"pod1uid\") : error mapping disk: success-and-failed-mount-device-name"
I0624 08:36:01.785860  220043 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:36:01.786016  220043 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0624 08:36:01.786855  220043 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-failed-mount-device-name podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:36:03.786249991 +0000 UTC m=+26.477955782 (durationBeforeRetry 2s). Error: "MapVolume.SetUpDevice failed for volume \"success-and-failed-mount-device-name\" (UniqueName: \"fake-plugin/success-and-failed-mount-device-name\") pod \"pod1\" (UID: \"pod1uid\") : error mapping disk: success-and-failed-mount-device-name"
I0624 08:36:03.787354  220043 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:36:03.788251  220043 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0624 08:36:03.789596  220043 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-failed-mount-device-name podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:36:07.789183944 +0000 UTC m=+30.480889718 (durationBeforeRetry 4s). Error: "MapVolume.SetUpDevice failed for volume \"success-and-failed-mount-device-name\" (UniqueName: \"fake-plugin/success-and-failed-mount-device-name\") pod \"pod1\" (UID: \"pod1uid\") : error mapping disk: success-and-failed-mount-device-name"
I0624 08:36:07.790829  220043 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:36:07.790998  220043 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0624 08:36:07.791392  220043 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-failed-mount-device-name podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:36:15.791192475 +0000 UTC m=+38.482898241 (durationBeforeRetry 8s). Error: "MapVolume.SetUpDevice failed for volume \"success-and-failed-mount-device-name\" (UniqueName: \"fake-plugin/success-and-failed-mount-device-name\") pod \"pod1\" (UID: \"pod1uid\") : error mapping disk: success-and-failed-mount-device-name"
I0624 08:36:10.221253  220043 reconciler.go:196] operationExecutor.UnmountVolume started for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") pod "pod1uid" (UID: "pod1uid") 
I0624 08:36:10.220918  220043 operation_generator.go:1170] UnmapVolume succeeded for volume "fake-plugin/success-and-failed-mount-device-name" (OuterVolumeSpecName: "success-and-failed-mount-device-name") pod "pod1uid" (UID: "pod1uid"). InnerVolumeSpecName "success-and-failed-mount-device-name". PluginName "fake-plugin", VolumeGidValue ""
I0624 08:36:10.222886  220043 operation_generator.go:1282] UnmapDevice succeeded for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") on node "mynodename" 
I0624 08:36:10.222967  220043 reconciler.go:312] operationExecutor.UnmountDevice started for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") on node "mynodename" 
I0624 08:36:10.224204  220043 reconciler.go:319] Volume detached for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") on node "mynodename" DevicePath "/dev/sdb"
I0624 08:36:10.321190  220043 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "timeout-mount-device-volume" (UniqueName: "fake-plugin/timeout-mount-device-volume") pod "pod1" (UID: "pod1uid") 
I0624 08:36:10.321273  220043 reconciler.go:157] Reconciler: start to sync state
E0624 08:36:10.321331  220043 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0624 08:36:10.321847  220043 operation_generator.go:1357] Controller attach succeeded for volume "timeout-mount-device-volume" (UniqueName: "fake-plugin/timeout-mount-device-volume") pod "pod1" (UID: "pod1uid") device path: "fake/path"
I0624 08:36:10.322377  220043 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "timeout-mount-device-volume" (UniqueName: "fake-plugin/timeout-mount-device-volume") pod "pod1" (UID: "pod1uid") DevicePath "fake/path"
I0624 08:36:10.322497  220043 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "timeout-mount-device-volume" (UniqueName: "fake-plugin/timeout-mount-device-volume") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0624 08:36:10.322901  220043 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/timeout-mount-device-volume podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:36:10.822721484 +0000 UTC m=+33.514427259 (durationBeforeRetry 500ms). Error: "MountVolume.MountDevice failed for volume \"timeout-mount-device-volume\" (UniqueName: \"fake-plugin/timeout-mount-device-volume\") pod \"pod1\" (UID: \"pod1uid\") : mount failed"
I0624 08:36:10.440413  220043 reconciler.go:312] operationExecutor.UnmountDevice started for volume "timeout-mount-device-volume" (UniqueName: "fake-plugin/timeout-mount-device-volume") on node "mynodename" 
I0624 08:36:10.440646  220043 operation_generator.go:891] UnmountDevice succeeded for volume "timeout-mount-device-volume" %!(EXTRA string=UnmountDevice succeeded for volume "timeout-mount-device-volume" (UniqueName: "fake-plugin/timeout-mount-device-volume") on node "mynodename" )
I0624 08:36:10.440874  220043 reconciler.go:319] Volume detached for volume "timeout-mount-device-volume" (UniqueName: "fake-plugin/timeout-mount-device-volume") on node "mynodename" DevicePath "/dev/sdb"
I0624 08:36:10.546303  220043 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "fail-mount-device-volume-name" (UniqueName: "fake-plugin/fail-mount-device-volume-name") pod "pod1" (UID: "pod1uid") 
I0624 08:36:10.546375  220043 reconciler.go:157] Reconciler: start to sync state
E0624 08:36:10.546430  220043 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0624 08:36:10.546691  220043 operation_generator.go:1357] Controller attach succeeded for volume "fail-mount-device-volume-name" (UniqueName: "fake-plugin/fail-mount-device-volume-name") pod "pod1" (UID: "pod1uid") device path: "fake/path"
I0624 08:36:10.547425  220043 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "fail-mount-device-volume-name" (UniqueName: "fake-plugin/fail-mount-device-volume-name") pod "pod1" (UID: "pod1uid") DevicePath "fake/path"
I0624 08:36:10.547563  220043 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "fail-mount-device-volume-name" (UniqueName: "fake-plugin/fail-mount-device-volume-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0624 08:36:10.547987  220043 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/fail-mount-device-volume-name podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:36:11.047795139 +0000 UTC m=+33.739500912 (durationBeforeRetry 500ms). Error: "MountVolume.MountDevice failed for volume \"fail-mount-device-volume-name\" (UniqueName: \"fake-plugin/fail-mount-device-volume-name\") pod \"pod1\" (UID: \"pod1uid\") : error mounting disk: /dev/sdb"
I0624 08:36:10.686780  220043 reconciler.go:319] Volume detached for volume "fail-mount-device-volume-name" (UniqueName: "fake-plugin/fail-mount-device-volume-name") on node "mynodename" DevicePath "fake/path"
I0624 08:36:10.795267  220043 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "timeout-and-fail-mount-device-name" (UniqueName: "fake-plugin/timeout-and-fail-mount-device-name") pod "pod1" (UID: "pod1uid") 
I0624 08:36:10.795355  220043 reconciler.go:157] Reconciler: start to sync state
E0624 08:36:10.795445  220043 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0624 08:36:10.795865  220043 operation_generator.go:1357] Controller attach succeeded for volume "timeout-and-fail-mount-device-name" (UniqueName: "fake-plugin/timeout-and-fail-mount-device-name") pod "pod1" (UID: "pod1uid") device path: "fake/path"
I0624 08:36:10.796668  220043 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "timeout-and-fail-mount-device-name" (UniqueName: "fake-plugin/timeout-and-fail-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "fake/path"
I0624 08:36:10.796837  220043 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "timeout-and-fail-mount-device-name" (UniqueName: "fake-plugin/timeout-and-fail-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0624 08:36:10.797317  220043 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/timeout-and-fail-mount-device-name podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:36:11.297078386 +0000 UTC m=+33.988784151 (durationBeforeRetry 500ms). Error: "MountVolume.MountDevice failed for volume \"timeout-and-fail-mount-device-name\" (UniqueName: \"fake-plugin/timeout-and-fail-mount-device-name\") pod \"pod1\" (UID: \"pod1uid\") : timed out mounting error"
I0624 08:36:11.300145  220043 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "timeout-and-fail-mount-device-name" (UniqueName: "fake-plugin/timeout-and-fail-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:36:11.300347  220043 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "timeout-and-fail-mount-device-name" (UniqueName: "fake-plugin/timeout-and-fail-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0624 08:36:11.300863  220043 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/timeout-and-fail-mount-device-name podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:36:12.30064447 +0000 UTC m=+34.992350248 (durationBeforeRetry 1s). Error: "MountVolume.MountDevice failed for volume \"timeout-and-fail-mount-device-name\" (UniqueName: \"fake-plugin/timeout-and-fail-mount-device-name\") pod \"pod1\" (UID: \"pod1uid\") : error mounting disk: /dev/sdb"
I0624 08:36:12.406125  220043 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "timeout-and-fail-mount-device-name" (UniqueName: "fake-plugin/timeout-and-fail-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:36:12.406293  220043 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "timeout-and-fail-mount-device-name" (UniqueName: "fake-plugin/timeout-and-fail-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0624 08:36:12.406971  220043 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/timeout-and-fail-mount-device-name podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:36:14.406513167 +0000 UTC m=+37.098218932 (durationBeforeRetry 2s). Error: "MountVolume.MountDevice failed for volume \"timeout-and-fail-mount-device-name\" (UniqueName: \"fake-plugin/timeout-and-fail-mount-device-name\") pod \"pod1\" (UID: \"pod1uid\") : error mounting disk: /dev/sdb"
I0624 08:36:14.419819  220043 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "timeout-and-fail-mount-device-name" (UniqueName: "fake-plugin/timeout-and-fail-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:36:14.419994  220043 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "timeout-and-fail-mount-device-name" (UniqueName: "fake-plugin/timeout-and-fail-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0624 08:36:14.420593  220043 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/timeout-and-fail-mount-device-name podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:36:18.420215735 +0000 UTC m=+41.111921502 (durationBeforeRetry 4s). Error: "MountVolume.MountDevice failed for volume \"timeout-and-fail-mount-device-name\" (UniqueName: \"fake-plugin/timeout-and-fail-mount-device-name\") pod \"pod1\" (UID: \"pod1uid\") : error mounting disk: /dev/sdb"
I0624 08:36:18.420711  220043 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "timeout-and-fail-mount-device-name" (UniqueName: "fake-plugin/timeout-and-fail-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:36:18.420884  220043 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "timeout-and-fail-mount-device-name" (UniqueName: "fake-plugin/timeout-and-fail-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0624 08:36:18.421301  220043 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/timeout-and-fail-mount-device-name podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:36:26.421110372 +0000 UTC m=+49.112816151 (durationBeforeRetry 8s). Error: "MountVolume.MountDevice failed for volume \"timeout-and-fail-mount-device-name\" (UniqueName: \"fake-plugin/timeout-and-fail-mount-device-name\") pod \"pod1\" (UID: \"pod1uid\") : error mounting disk: /dev/sdb"
I0624 08:36:20.932039  220043 reconciler.go:319] Volume detached for volume "timeout-and-fail-mount-device-name" (UniqueName: "fake-plugin/timeout-and-fail-mount-device-name") on node "mynodename" DevicePath "/dev/sdb"
I0624 08:36:21.028959  220043 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "success-and-timeout-device-name" (UniqueName: "fake-plugin/success-and-timeout-device-name") pod "pod1" (UID: "pod1uid") 
I0624 08:36:21.029038  220043 reconciler.go:157] Reconciler: start to sync state
E0624 08:36:21.029091  220043 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0624 08:36:21.029539  220043 operation_generator.go:1357] Controller attach succeeded for volume "success-and-timeout-device-name" (UniqueName: "fake-plugin/success-and-timeout-device-name") pod "pod1" (UID: "pod1uid") device path: "fake/path"
I0624 08:36:21.030187  220043 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "success-and-timeout-device-name" (UniqueName: "fake-plugin/success-and-timeout-device-name") pod "pod1" (UID: "pod1uid") DevicePath "fake/path"
I0624 08:36:21.030317  220043 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "success-and-timeout-device-name" (UniqueName: "fake-plugin/success-and-timeout-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:36:21.030428  220043 operation_generator.go:594] MountVolume.MountDevice succeeded for volume "success-and-timeout-device-name" (UniqueName: "fake-plugin/success-and-timeout-device-name") pod "pod1" (UID: "pod1uid") device mount path ""
I0624 08:36:21.193966  220043 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "success-and-timeout-device-name" (UniqueName: "fake-plugin/success-and-timeout-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:36:21.194137  220043 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "success-and-timeout-device-name" (UniqueName: "fake-plugin/success-and-timeout-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:36:31.236349  220043 reconciler.go:196] operationExecutor.UnmountVolume started for volume "success-and-timeout-device-name" (UniqueName: "fake-plugin/success-and-timeout-device-name") pod "pod1uid" (UID: "pod1uid") 
I0624 08:36:31.236939  220043 operation_generator.go:797] UnmountVolume.TearDown succeeded for volume "fake-plugin/success-and-timeout-device-name" (OuterVolumeSpecName: "success-and-timeout-device-name") pod "pod1uid" (UID: "pod1uid"). InnerVolumeSpecName "success-and-timeout-device-name". PluginName "fake-plugin", VolumeGidValue ""
I0624 08:36:31.237452  220043 reconciler.go:312] operationExecutor.UnmountDevice started for volume "success-and-timeout-device-name" (UniqueName: "fake-plugin/success-and-timeout-device-name") on node "mynodename" 
I0624 08:36:31.237798  220043 operation_generator.go:891] UnmountDevice succeeded for volume "success-and-timeout-device-name" %!(EXTRA string=UnmountDevice succeeded for volume "success-and-timeout-device-name" (UniqueName: "fake-plugin/success-and-timeout-device-name") on node "mynodename" )
I0624 08:36:31.238033  220043 reconciler.go:319] Volume detached for volume "success-and-timeout-device-name" (UniqueName: "fake-plugin/success-and-timeout-device-name") on node "mynodename" DevicePath "/dev/sdb"
I0624 08:36:31.354058  220043 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") pod "pod1" (UID: "pod1uid") 
I0624 08:36:31.354127  220043 reconciler.go:157] Reconciler: start to sync state
E0624 08:36:31.354203  220043 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0624 08:36:31.354617  220043 operation_generator.go:1357] Controller attach succeeded for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") pod "pod1" (UID: "pod1uid") device path: "fake/path"
I0624 08:36:31.355405  220043 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "fake/path"
I0624 08:36:31.355533  220043 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:36:31.355656  220043 operation_generator.go:594] MountVolume.MountDevice succeeded for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") pod "pod1" (UID: "pod1uid") device mount path ""
I0624 08:36:31.457654  220043 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:36:31.458230  220043 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:36:31.460825  220043 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:36:31.462233  220043 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:36:41.495275  220043 reconciler.go:196] operationExecutor.UnmountVolume started for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") pod "pod1uid" (UID: "pod1uid") 
I0624 08:36:41.495509  220043 operation_generator.go:797] UnmountVolume.TearDown succeeded for volume "fake-plugin/success-and-failed-mount-device-name" (OuterVolumeSpecName: "success-and-failed-mount-device-name") pod "pod1uid" (UID: "pod1uid"). InnerVolumeSpecName "success-and-failed-mount-device-name". PluginName "fake-plugin", VolumeGidValue ""
I0624 08:36:41.496281  220043 reconciler.go:312] operationExecutor.UnmountDevice started for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") on node "mynodename" 
I0624 08:36:41.496482  220043 operation_generator.go:891] UnmountDevice succeeded for volume "success-and-failed-mount-device-name" %!(EXTRA string=UnmountDevice succeeded for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") on node "mynodename" )
I0624 08:36:41.496998  220043 reconciler.go:319] Volume detached for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") on node "mynodename" DevicePath "/dev/sdb"
I0624 08:36:41.596845  220043 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "timeout-setup-volume" (UniqueName: "fake-plugin/timeout-setup-volume") pod "pod1" (UID: "pod1uid") 
I0624 08:36:41.596934  220043 reconciler.go:157] Reconciler: start to sync state
E0624 08:36:41.596984  220043 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0624 08:36:41.597469  220043 operation_generator.go:1357] Controller attach succeeded for volume "timeout-setup-volume" (UniqueName: "fake-plugin/timeout-setup-volume") pod "pod1" (UID: "pod1uid") device path: "fake/path"
I0624 08:36:41.598358  220043 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "timeout-setup-volume" (UniqueName: "fake-plugin/timeout-setup-volume") pod "pod1" (UID: "pod1uid") DevicePath "fake/path"
I0624 08:36:41.598504  220043 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "timeout-setup-volume" (UniqueName: "fake-plugin/timeout-setup-volume") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0624 08:36:41.598878  220043 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/timeout-setup-volume podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:36:42.09868944 +0000 UTC m=+64.790395213 (durationBeforeRetry 500ms). Error: "MapVolume.MapPodDevice failed for volume \"timeout-setup-volume\" (UniqueName: \"fake-plugin/timeout-setup-volume\") pod \"pod1\" (UID: \"pod1uid\") : time out on setup"
I0624 08:36:41.704278  220043 reconciler.go:196] operationExecutor.UnmountVolume started for volume "timeout-setup-volume" (UniqueName: "fake-plugin/timeout-setup-volume") pod "pod1uid" (UID: "pod1uid") 
I0624 08:36:41.704527  220043 operation_generator.go:1170] UnmapVolume succeeded for volume "fake-plugin/timeout-setup-volume" (OuterVolumeSpecName: "timeout-setup-volume") pod "pod1uid" (UID: "pod1uid"). InnerVolumeSpecName "timeout-setup-volume". PluginName "fake-plugin", VolumeGidValue ""
I0624 08:36:41.705008  220043 reconciler.go:312] operationExecutor.UnmountDevice started for volume "timeout-setup-volume" (UniqueName: "fake-plugin/timeout-setup-volume") on node "mynodename" 
I0624 08:36:41.705165  220043 operation_generator.go:1282] UnmapDevice succeeded for volume "timeout-setup-volume" (UniqueName: "fake-plugin/timeout-setup-volume") on node "mynodename" 
I0624 08:36:41.705373  220043 reconciler.go:319] Volume detached for volume "timeout-setup-volume" (UniqueName: "fake-plugin/timeout-setup-volume") on node "mynodename" DevicePath "/dev/sdb"
I0624 08:36:41.816168  220043 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "fail-setup-volume" (UniqueName: "fake-plugin/fail-setup-volume") pod "pod1" (UID: "pod1uid") 
I0624 08:36:41.816252  220043 reconciler.go:157] Reconciler: start to sync state
E0624 08:36:41.816319  220043 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0624 08:36:41.816562  220043 operation_generator.go:1357] Controller attach succeeded for volume "fail-setup-volume" (UniqueName: "fake-plugin/fail-setup-volume") pod "pod1" (UID: "pod1uid") device path: "fake/path"
I0624 08:36:41.817368  220043 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "fail-setup-volume" (UniqueName: "fake-plugin/fail-setup-volume") pod "pod1" (UID: "pod1uid") DevicePath "fake/path"
I0624 08:36:41.817491  220043 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "fail-setup-volume" (UniqueName: "fake-plugin/fail-setup-volume") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0624 08:36:41.817832  220043 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/fail-setup-volume podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:36:42.317669998 +0000 UTC m=+65.009375775 (durationBeforeRetry 500ms). Error: "MapVolume.MapPodDevice failed for volume \"fail-setup-volume\" (UniqueName: \"fake-plugin/fail-setup-volume\") pod \"pod1\" (UID: \"pod1uid\") : mounting volume failed"
I0624 08:36:41.924426  220043 reconciler.go:312] operationExecutor.UnmountDevice started for volume "fail-setup-volume" (UniqueName: "fake-plugin/fail-setup-volume") on node "mynodename" 
I0624 08:36:41.924857  220043 operation_generator.go:1282] UnmapDevice succeeded for volume "fail-setup-volume" (UniqueName: "fake-plugin/fail-setup-volume") on node "mynodename" 
I0624 08:36:41.925100  220043 reconciler.go:319] Volume detached for volume "fail-setup-volume" (UniqueName: "fake-plugin/fail-setup-volume") on node "mynodename" DevicePath "/dev/sdb"
I0624 08:36:42.042855  220043 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") pod "pod1" (UID: "pod1uid") 
I0624 08:36:42.042950  220043 reconciler.go:157] Reconciler: start to sync state
E0624 08:36:42.043008  220043 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0624 08:36:42.045827  220043 operation_generator.go:1357] Controller attach succeeded for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") pod "pod1" (UID: "pod1uid") device path: "fake/path"
I0624 08:36:42.049964  220043 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") pod "pod1" (UID: "pod1uid") DevicePath "fake/path"
I0624 08:36:42.050107  220043 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0624 08:36:42.050535  220043 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/timeout-and-fail-setup-volume podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:36:42.550333889 +0000 UTC m=+65.242039670 (durationBeforeRetry 500ms). Error: "MapVolume.MapPodDevice failed for volume \"timeout-and-fail-setup-volume\" (UniqueName: \"fake-plugin/timeout-and-fail-setup-volume\") pod \"pod1\" (UID: \"pod1uid\") : time out on setup"
I0624 08:36:42.552750  220043 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:36:42.555216  220043 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0624 08:36:42.559699  220043 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/timeout-and-fail-setup-volume podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:36:43.556622071 +0000 UTC m=+66.248327843 (durationBeforeRetry 1s). Error: "MapVolume.MapPodDevice failed for volume \"timeout-and-fail-setup-volume\" (UniqueName: \"fake-plugin/timeout-and-fail-setup-volume\") pod \"pod1\" (UID: \"pod1uid\") : mounting volume failed"
I0624 08:36:43.571160  220043 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:36:43.584917  220043 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0624 08:36:43.585397  220043 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/timeout-and-fail-setup-volume podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:36:45.585160222 +0000 UTC m=+68.276865987 (durationBeforeRetry 2s). Error: "MapVolume.MapPodDevice failed for volume \"timeout-and-fail-setup-volume\" (UniqueName: \"fake-plugin/timeout-and-fail-setup-volume\") pod \"pod1\" (UID: \"pod1uid\") : mounting volume failed"
I0624 08:36:45.586168  220043 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:36:45.586922  220043 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0624 08:36:45.590664  220043 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/timeout-and-fail-setup-volume podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:36:49.58807603 +0000 UTC m=+72.279781813 (durationBeforeRetry 4s). Error: "MapVolume.MapPodDevice failed for volume \"timeout-and-fail-setup-volume\" (UniqueName: \"fake-plugin/timeout-and-fail-setup-volume\") pod \"pod1\" (UID: \"pod1uid\") : mounting volume failed"
I0624 08:36:49.600461  220043 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:36:49.600632  220043 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0624 08:36:49.601032  220043 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/timeout-and-fail-setup-volume podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:36:57.600854902 +0000 UTC m=+80.292560665 (durationBeforeRetry 8s). Error: "MapVolume.MapPodDevice failed for volume \"timeout-and-fail-setup-volume\" (UniqueName: \"fake-plugin/timeout-and-fail-setup-volume\") pod \"pod1\" (UID: \"pod1uid\") : mounting volume failed"
I0624 08:36:52.141921  220043 reconciler.go:312] operationExecutor.UnmountDevice started for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") on node "mynodename" 
I0624 08:36:52.142218  220043 operation_generator.go:1282] UnmapDevice succeeded for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") on node "mynodename" 
I0624 08:36:52.142438  220043 reconciler.go:319] Volume detached for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") on node "mynodename" DevicePath "/dev/sdb"
I0624 08:36:52.243442  220043 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1" (UID: "pod1uid") 
I0624 08:36:52.243519  220043 reconciler.go:157] Reconciler: start to sync state
E0624 08:36:52.243580  220043 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0624 08:36:52.264082  220043 operation_generator.go:1357] Controller attach succeeded for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1" (UID: "pod1uid") device path: "fake/path"
I0624 08:36:52.264741  220043 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1" (UID: "pod1uid") DevicePath "fake/path"
I0624 08:36:52.264891  220043 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:36:52.373253  220043 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:36:52.373467  220043 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0624 08:36:52.373952  220043 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-timeout-setup-volume-name podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:36:52.873729798 +0000 UTC m=+75.565435579 (durationBeforeRetry 500ms). Error: "MapVolume.MapPodDevice failed for volume \"success-and-timeout-setup-volume-name\" (UniqueName: \"fake-plugin/success-and-timeout-setup-volume-name\") pod \"pod1\" (UID: \"pod1uid\") : time out on setup"
I0624 08:36:52.874206  220043 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:36:52.874332  220043 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0624 08:36:52.874607  220043 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-timeout-setup-volume-name podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:36:53.874474638 +0000 UTC m=+76.566180400 (durationBeforeRetry 1s). Error: "MapVolume.MapPodDevice failed for volume \"success-and-timeout-setup-volume-name\" (UniqueName: \"fake-plugin/success-and-timeout-setup-volume-name\") pod \"pod1\" (UID: \"pod1uid\") : time out on setup"
I0624 08:36:53.913316  220043 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:36:53.915443  220043 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0624 08:36:53.932746  220043 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-timeout-setup-volume-name podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:36:55.932172519 +0000 UTC m=+78.623878298 (durationBeforeRetry 2s). Error: "MapVolume.MapPodDevice failed for volume \"success-and-timeout-setup-volume-name\" (UniqueName: \"fake-plugin/success-and-timeout-setup-volume-name\") pod \"pod1\" (UID: \"pod1uid\") : time out on setup"
I0624 08:36:55.934106  220043 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:36:55.934364  220043 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0624 08:36:55.935252  220043 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-timeout-setup-volume-name podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:36:59.934697476 +0000 UTC m=+82.626403241 (durationBeforeRetry 4s). Error: "MapVolume.MapPodDevice failed for volume \"success-and-timeout-setup-volume-name\" (UniqueName: \"fake-plugin/success-and-timeout-setup-volume-name\") pod \"pod1\" (UID: \"pod1uid\") : time out on setup"
I0624 08:36:59.956784  220043 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:36:59.956960  220043 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0624 08:36:59.957379  220043 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-timeout-setup-volume-name podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:37:07.957180372 +0000 UTC m=+90.648886139 (durationBeforeRetry 8s). Error: "MapVolume.MapPodDevice failed for volume \"success-and-timeout-setup-volume-name\" (UniqueName: \"fake-plugin/success-and-timeout-setup-volume-name\") pod \"pod1\" (UID: \"pod1uid\") : time out on setup"
I0624 08:37:02.366310  220043 reconciler.go:196] operationExecutor.UnmountVolume started for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1uid" (UID: "pod1uid") 
I0624 08:37:02.366628  220043 operation_generator.go:1170] UnmapVolume succeeded for volume "fake-plugin/success-and-timeout-setup-volume-name" (OuterVolumeSpecName: "success-and-timeout-setup-volume-name") pod "pod1uid" (UID: "pod1uid"). InnerVolumeSpecName "success-and-timeout-setup-volume-name". PluginName "fake-plugin", VolumeGidValue ""
I0624 08:37:02.367131  220043 reconciler.go:312] operationExecutor.UnmountDevice started for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") on node "mynodename" 
I0624 08:37:02.367428  220043 operation_generator.go:1282] UnmapDevice succeeded for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") on node "mynodename" 
I0624 08:37:02.367635  220043 reconciler.go:319] Volume detached for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") on node "mynodename" DevicePath "/dev/sdb"
I0624 08:37:02.470518  220043 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1" (UID: "pod1uid") 
I0624 08:37:02.470607  220043 reconciler.go:157] Reconciler: start to sync state
E0624 08:37:02.471255  220043 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0624 08:37:02.471409  220043 operation_generator.go:1357] Controller attach succeeded for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1" (UID: "pod1uid") device path: "fake/path"
I0624 08:37:02.474688  220043 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1" (UID: "pod1uid") DevicePath "fake/path"
I0624 08:37:02.474812  220043 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:37:02.585644  220043 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:37:02.585805  220043 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0624 08:37:02.592236  220043 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-failed-setup-device-name podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:37:03.092042467 +0000 UTC m=+85.783748252 (durationBeforeRetry 500ms). Error: "MapVolume.MapPodDevice failed for volume \"success-and-failed-setup-device-name\" (UniqueName: \"fake-plugin/success-and-failed-setup-device-name\") pod \"pod1\" (UID: \"pod1uid\") : mounting volume failed"
I0624 08:37:03.093639  220043 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:37:03.093803  220043 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0624 08:37:03.094204  220043 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-failed-setup-device-name podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:37:04.094013331 +0000 UTC m=+86.785719097 (durationBeforeRetry 1s). Error: "MapVolume.MapPodDevice failed for volume \"success-and-failed-setup-device-name\" (UniqueName: \"fake-plugin/success-and-failed-setup-device-name\") pod \"pod1\" (UID: \"pod1uid\") : mounting volume failed"
I0624 08:37:04.097232  220043 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:37:04.097416  220043 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0624 08:37:04.097865  220043 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-failed-setup-device-name podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:37:06.097639663 +0000 UTC m=+88.789345435 (durationBeforeRetry 2s). Error: "MapVolume.MapPodDevice failed for volume \"success-and-failed-setup-device-name\" (UniqueName: \"fake-plugin/success-and-failed-setup-device-name\") pod \"pod1\" (UID: \"pod1uid\") : mounting volume failed"
I0624 08:37:06.098453  220043 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:37:06.098632  220043 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0624 08:37:06.099226  220043 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-failed-setup-device-name podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:37:10.09892803 +0000 UTC m=+92.790633804 (durationBeforeRetry 4s). Error: "MapVolume.MapPodDevice failed for volume \"success-and-failed-setup-device-name\" (UniqueName: \"fake-plugin/success-and-failed-setup-device-name\") pod \"pod1\" (UID: \"pod1uid\") : mounting volume failed"
I0624 08:37:10.114598  220043 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:37:10.114804  220043 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0624 08:37:10.115828  220043 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-failed-setup-device-name podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:37:18.115560497 +0000 UTC m=+100.807266267 (durationBeforeRetry 8s). Error: "MapVolume.MapPodDevice failed for volume \"success-and-failed-setup-device-name\" (UniqueName: \"fake-plugin/success-and-failed-setup-device-name\") pod \"pod1\" (UID: \"pod1uid\") : mounting volume failed"
I0624 08:37:12.589507  220043 reconciler.go:196] operationExecutor.UnmountVolume started for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1uid" (UID: "pod1uid") 
I0624 08:37:12.594972  220043 operation_generator.go:1170] UnmapVolume succeeded for volume "fake-plugin/success-and-failed-setup-device-name" (OuterVolumeSpecName: "success-and-failed-setup-device-name") pod "pod1uid" (UID: "pod1uid"). InnerVolumeSpecName "success-and-failed-setup-device-name". PluginName "fake-plugin", VolumeGidValue ""
I0624 08:37:12.597013  220043 reconciler.go:196] operationExecutor.UnmountVolume started for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1uid" (UID: "pod1uid") 
I0624 08:37:12.596316  220043 operation_generator.go:1170] UnmapVolume succeeded for volume "fake-plugin/success-and-failed-setup-device-name" (OuterVolumeSpecName: "success-and-failed-setup-device-name") pod "pod1uid" (UID: "pod1uid"). InnerVolumeSpecName "success-and-failed-setup-device-name". PluginName "fake-plugin", VolumeGidValue ""
I0624 08:37:12.599804  220043 operation_generator.go:1282] UnmapDevice succeeded for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") on node "mynodename" 
I0624 08:37:12.599725  220043 reconciler.go:312] operationExecutor.UnmountDevice started for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") on node "mynodename" 
I0624 08:37:12.602014  220043 reconciler.go:319] Volume detached for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") on node "mynodename" DevicePath "/dev/sdb"
I0624 08:37:12.687137  220043 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "timeout-setup-volume" (UniqueName: "fake-plugin/timeout-setup-volume") pod "pod1" (UID: "pod1uid") 
I0624 08:37:12.687213  220043 reconciler.go:157] Reconciler: start to sync state
E0624 08:37:12.687262  220043 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0624 08:37:12.687520  220043 operation_generator.go:1357] Controller attach succeeded for volume "timeout-setup-volume" (UniqueName: "fake-plugin/timeout-setup-volume") pod "pod1" (UID: "pod1uid") device path: "fake/path"
I0624 08:37:12.688306  220043 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "timeout-setup-volume" (UniqueName: "fake-plugin/timeout-setup-volume") pod "pod1" (UID: "pod1uid") DevicePath "fake/path"
I0624 08:37:12.688453  220043 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "timeout-setup-volume" (UniqueName: "fake-plugin/timeout-setup-volume") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:37:12.688561  220043 operation_generator.go:594] MountVolume.MountDevice succeeded for volume "timeout-setup-volume" (UniqueName: "fake-plugin/timeout-setup-volume") pod "pod1" (UID: "pod1uid") device mount path ""
E0624 08:37:12.688989  220043 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/timeout-setup-volume podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:37:13.188798705 +0000 UTC m=+95.880504488 (durationBeforeRetry 500ms). Error: "MountVolume.SetUp failed for volume \"timeout-setup-volume\" (UniqueName: \"fake-plugin/timeout-setup-volume\") pod \"pod1\" (UID: \"pod1uid\") : time out on setup"
I0624 08:37:12.795120  220043 reconciler.go:196] operationExecutor.UnmountVolume started for volume "timeout-setup-volume" (UniqueName: "fake-plugin/timeout-setup-volume") pod "pod1uid" (UID: "pod1uid") 
I0624 08:37:12.795422  220043 operation_generator.go:797] UnmountVolume.TearDown succeeded for volume "fake-plugin/timeout-setup-volume" (OuterVolumeSpecName: "timeout-setup-volume") pod "pod1uid" (UID: "pod1uid"). InnerVolumeSpecName "timeout-setup-volume". PluginName "fake-plugin", VolumeGidValue ""
I0624 08:37:12.795841  220043 reconciler.go:312] operationExecutor.UnmountDevice started for volume "timeout-setup-volume" (UniqueName: "fake-plugin/timeout-setup-volume") on node "mynodename" 
I0624 08:37:12.796092  220043 operation_generator.go:891] UnmountDevice succeeded for volume "timeout-setup-volume" %!(EXTRA string=UnmountDevice succeeded for volume "timeout-setup-volume" (UniqueName: "fake-plugin/timeout-setup-volume") on node "mynodename" )
I0624 08:37:12.796304  220043 reconciler.go:319] Volume detached for volume "timeout-setup-volume" (UniqueName: "fake-plugin/timeout-setup-volume") on node "mynodename" DevicePath "/dev/sdb"
I0624 08:37:12.932408  220043 operation_generator.go:1357] Controller attach succeeded for volume "fail-setup-volume" (UniqueName: "fake-plugin/fail-setup-volume") pod "pod1" (UID: "pod1uid") device path: "fake/path"
I0624 08:37:12.932282  220043 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "fail-setup-volume" (UniqueName: "fake-plugin/fail-setup-volume") pod "pod1" (UID: "pod1uid") 
I0624 08:37:12.933313  220043 reconciler.go:157] Reconciler: start to sync state
E0624 08:37:12.933374  220043 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0624 08:37:12.934462  220043 reconciler.go:319] Volume detached for volume "fail-setup-volume" (UniqueName: "fake-plugin/fail-setup-volume") on node "mynodename" DevicePath "fake/path"
I0624 08:37:13.037477  220043 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") pod "pod1" (UID: "pod1uid") 
I0624 08:37:13.037585  220043 reconciler.go:157] Reconciler: start to sync state
E0624 08:37:13.037648  220043 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0624 08:37:13.038338  220043 operation_generator.go:1357] Controller attach succeeded for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") pod "pod1" (UID: "pod1uid") device path: "fake/path"
I0624 08:37:13.039277  220043 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") pod "pod1" (UID: "pod1uid") DevicePath "fake/path"
I0624 08:37:13.039435  220043 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:37:13.039575  220043 operation_generator.go:594] MountVolume.MountDevice succeeded for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") pod "pod1" (UID: "pod1uid") device mount path ""
E0624 08:37:13.057747  220043 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/timeout-and-fail-setup-volume podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:37:13.557376394 +0000 UTC m=+96.249082161 (durationBeforeRetry 500ms). Error: "MountVolume.SetUp failed for volume \"timeout-and-fail-setup-volume\" (UniqueName: \"fake-plugin/timeout-and-fail-setup-volume\") pod \"pod1\" (UID: \"pod1uid\") : time out on setup"
I0624 08:37:13.561927  220043 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:37:13.563060  220043 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0624 08:37:13.568771  220043 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/timeout-and-fail-setup-volume podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:37:14.565346221 +0000 UTC m=+97.257051992 (durationBeforeRetry 1s). Error: "MountVolume.SetUp failed for volume \"timeout-and-fail-setup-volume\" (UniqueName: \"fake-plugin/timeout-and-fail-setup-volume\") pod \"pod1\" (UID: \"pod1uid\") : mounting volume failed"
I0624 08:37:14.569778  220043 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:37:14.588650  220043 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0624 08:37:14.619456  220043 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/timeout-and-fail-setup-volume podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:37:16.618929695 +0000 UTC m=+99.310635456 (durationBeforeRetry 2s). Error: "MountVolume.SetUp failed for volume \"timeout-and-fail-setup-volume\" (UniqueName: \"fake-plugin/timeout-and-fail-setup-volume\") pod \"pod1\" (UID: \"pod1uid\") : mounting volume failed"
I0624 08:37:16.636786  220043 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:37:16.636951  220043 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0624 08:37:16.637370  220043 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/timeout-and-fail-setup-volume podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:37:20.637178529 +0000 UTC m=+103.328884297 (durationBeforeRetry 4s). Error: "MountVolume.SetUp failed for volume \"timeout-and-fail-setup-volume\" (UniqueName: \"fake-plugin/timeout-and-fail-setup-volume\") pod \"pod1\" (UID: \"pod1uid\") : mounting volume failed"
I0624 08:37:20.644610  220043 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:37:20.644793  220043 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0624 08:37:20.649443  220043 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/timeout-and-fail-setup-volume podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:37:28.645018393 +0000 UTC m=+111.336724164 (durationBeforeRetry 8s). Error: "MountVolume.SetUp failed for volume \"timeout-and-fail-setup-volume\" (UniqueName: \"fake-plugin/timeout-and-fail-setup-volume\") pod \"pod1\" (UID: \"pod1uid\") : mounting volume failed"
I0624 08:37:23.156424  220043 reconciler.go:312] operationExecutor.UnmountDevice started for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") on node "mynodename" 
I0624 08:37:23.156883  220043 operation_generator.go:891] UnmountDevice succeeded for volume "timeout-and-fail-setup-volume" %!(EXTRA string=UnmountDevice succeeded for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") on node "mynodename" )
I0624 08:37:23.158283  220043 reconciler.go:319] Volume detached for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") on node "mynodename" DevicePath "/dev/sdb"
I0624 08:37:23.261458  220043 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1" (UID: "pod1uid") 
I0624 08:37:23.261538  220043 reconciler.go:157] Reconciler: start to sync state
E0624 08:37:23.261612  220043 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0624 08:37:23.261860  220043 operation_generator.go:1357] Controller attach succeeded for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1" (UID: "pod1uid") device path: "fake/path"
I0624 08:37:23.262572  220043 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1" (UID: "pod1uid") DevicePath "fake/path"
I0624 08:37:23.262719  220043 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:37:23.262863  220043 operation_generator.go:594] MountVolume.MountDevice succeeded for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1" (UID: "pod1uid") device mount path ""
I0624 08:37:23.404675  220043 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:37:23.415420  220043 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0624 08:37:23.415937  220043 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-timeout-setup-volume-name podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:37:23.915708529 +0000 UTC m=+106.607414310 (durationBeforeRetry 500ms). Error: "MountVolume.SetUp failed for volume \"success-and-timeout-setup-volume-name\" (UniqueName: \"fake-plugin/success-and-timeout-setup-volume-name\") pod \"pod1\" (UID: \"pod1uid\") : time out on setup"
I0624 08:37:23.924516  220043 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:37:23.924678  220043 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0624 08:37:23.925080  220043 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-timeout-setup-volume-name podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:37:24.924894249 +0000 UTC m=+107.616600013 (durationBeforeRetry 1s). Error: "MountVolume.SetUp failed for volume \"success-and-timeout-setup-volume-name\" (UniqueName: \"fake-plugin/success-and-timeout-setup-volume-name\") pod \"pod1\" (UID: \"pod1uid\") : time out on setup"
I0624 08:37:24.960664  220043 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:37:24.960839  220043 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0624 08:37:24.961506  220043 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-timeout-setup-volume-name podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:37:26.961036165 +0000 UTC m=+109.652741929 (durationBeforeRetry 2s). Error: "MountVolume.SetUp failed for volume \"success-and-timeout-setup-volume-name\" (UniqueName: \"fake-plugin/success-and-timeout-setup-volume-name\") pod \"pod1\" (UID: \"pod1uid\") : time out on setup"
I0624 08:37:26.964802  220043 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:37:26.964971  220043 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0624 08:37:26.965375  220043 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-timeout-setup-volume-name podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:37:30.965176454 +0000 UTC m=+113.656882229 (durationBeforeRetry 4s). Error: "MountVolume.SetUp failed for volume \"success-and-timeout-setup-volume-name\" (UniqueName: \"fake-plugin/success-and-timeout-setup-volume-name\") pod \"pod1\" (UID: \"pod1uid\") : time out on setup"
I0624 08:37:30.966940  220043 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:37:30.968138  220043 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0624 08:37:30.968675  220043 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-timeout-setup-volume-name podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:37:38.968448728 +0000 UTC m=+121.660154497 (durationBeforeRetry 8s). Error: "MountVolume.SetUp failed for volume \"success-and-timeout-setup-volume-name\" (UniqueName: \"fake-plugin/success-and-timeout-setup-volume-name\") pod \"pod1\" (UID: \"pod1uid\") : time out on setup"
I0624 08:37:33.405001  220043 reconciler.go:196] operationExecutor.UnmountVolume started for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1uid" (UID: "pod1uid") 
I0624 08:37:33.405368  220043 operation_generator.go:797] UnmountVolume.TearDown succeeded for volume "fake-plugin/success-and-timeout-setup-volume-name" (OuterVolumeSpecName: "success-and-timeout-setup-volume-name") pod "pod1uid" (UID: "pod1uid"). InnerVolumeSpecName "success-and-timeout-setup-volume-name". PluginName "fake-plugin", VolumeGidValue ""
I0624 08:37:33.405870  220043 reconciler.go:312] operationExecutor.UnmountDevice started for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") on node "mynodename" 
I0624 08:37:33.406457  220043 operation_generator.go:891] UnmountDevice succeeded for volume "success-and-timeout-setup-volume-name" %!(EXTRA string=UnmountDevice succeeded for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") on node "mynodename" )
I0624 08:37:33.406753  220043 reconciler.go:319] Volume detached for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") on node "mynodename" DevicePath "/dev/sdb"
I0624 08:37:33.506377  220043 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1" (UID: "pod1uid") 
I0624 08:37:33.506464  220043 reconciler.go:157] Reconciler: start to sync state
E0624 08:37:33.506522  220043 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0624 08:37:33.507073  220043 operation_generator.go:1357] Controller attach succeeded for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1" (UID: "pod1uid") device path: "fake/path"
I0624 08:37:33.508953  220043 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1" (UID: "pod1uid") DevicePath "fake/path"
I0624 08:37:33.509498  220043 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:37:33.510010  220043 operation_generator.go:594] MountVolume.MountDevice succeeded for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1" (UID: "pod1uid") device mount path ""
I0624 08:37:33.628739  220043 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:37:33.628911  220043 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0624 08:37:33.629331  220043 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-failed-setup-device-name podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:37:34.129142287 +0000 UTC m=+116.820848035 (durationBeforeRetry 500ms). Error: "MountVolume.SetUp failed for volume \"success-and-failed-setup-device-name\" (UniqueName: \"fake-plugin/success-and-failed-setup-device-name\") pod \"pod1\" (UID: \"pod1uid\") : mounting volume failed"
I0624 08:37:34.137472  220043 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:37:34.139843  220043 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0624 08:37:34.150834  220043 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-failed-setup-device-name podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:37:35.143654145 +0000 UTC m=+117.835359919 (durationBeforeRetry 1s). Error: "MountVolume.SetUp failed for volume \"success-and-failed-setup-device-name\" (UniqueName: \"fake-plugin/success-and-failed-setup-device-name\") pod \"pod1\" (UID: \"pod1uid\") : mounting volume failed"
I0624 08:37:35.151793  220043 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:37:35.152015  220043 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0624 08:37:35.165032  220043 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-failed-setup-device-name podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:37:37.152273763 +0000 UTC m=+119.843979544 (durationBeforeRetry 2s). Error: "MountVolume.SetUp failed for volume \"success-and-failed-setup-device-name\" (UniqueName: \"fake-plugin/success-and-failed-setup-device-name\") pod \"pod1\" (UID: \"pod1uid\") : mounting volume failed"
I0624 08:37:37.153290  220043 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:37:37.153473  220043 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0624 08:37:37.154164  220043 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-failed-setup-device-name podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:37:41.153710513 +0000 UTC m=+123.845416286 (durationBeforeRetry 4s). Error: "MountVolume.SetUp failed for volume \"success-and-failed-setup-device-name\" (UniqueName: \"fake-plugin/success-and-failed-setup-device-name\") pod \"pod1\" (UID: \"pod1uid\") : mounting volume failed"
I0624 08:37:41.170520  220043 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:37:41.170729  220043 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0624 08:37:41.171292  220043 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-failed-setup-device-name podName: nodeName:}" failed. No retries permitted until 2021-06-24 08:37:49.171007258 +0000 UTC m=+131.862713022 (durationBeforeRetry 8s). Error: "MountVolume.SetUp failed for volume \"success-and-failed-setup-device-name\" (UniqueName: \"fake-plugin/success-and-failed-setup-device-name\") pod \"pod1\" (UID: \"pod1uid\") : mounting volume failed"
I0624 08:37:43.692976  220043 reconciler.go:196] operationExecutor.UnmountVolume started for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1uid" (UID: "pod1uid") 
I0624 08:37:43.693474  220043 operation_generator.go:797] UnmountVolume.TearDown succeeded for volume "fake-plugin/success-and-failed-setup-device-name" (OuterVolumeSpecName: "success-and-failed-setup-device-name") pod "pod1uid" (UID: "pod1uid"). InnerVolumeSpecName "success-and-failed-setup-device-name". PluginName "fake-plugin", VolumeGidValue ""
I0624 08:37:43.694089  220043 reconciler.go:312] operationExecutor.UnmountDevice started for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") on node "mynodename" 
I0624 08:37:43.694416  220043 operation_generator.go:891] UnmountDevice succeeded for volume "success-and-failed-setup-device-name" %!(EXTRA string=UnmountDevice succeeded for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") on node "mynodename" )
I0624 08:37:43.694648  220043 reconciler.go:319] Volume detached for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") on node "mynodename" DevicePath "/dev/sdb"
--- FAIL: Test_UncertainVolumeMountState (62.14s)
    --- FAIL: Test_UncertainVolumeMountState/failed_operation_should_result_in_not-mounted_volume_[Filesystem] (0.11s)
        reconciler_test.go:1523: Error verifying UnMountDeviceCallCount: Expected DeviceUnmount Call 1, got 0
I0624 08:37:43.736337  220043 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") 
I0624 08:37:43.736414  220043 reconciler.go:157] Reconciler: start to sync state
E0624 08:37:43.736472  220043 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0624 08:37:43.736769  220043 operation_generator.go:1357] Controller attach succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") device path: "/fake/path"
I0624 08:37:43.737508  220043 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") DevicePath "/fake/path"
I0624 08:37:43.737620  220043 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:37:43.737722  220043 operation_generator.go:594] MountVolume.MountDevice succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") device mount path ""
I0624 08:37:43.865060  220043 reconciler.go:196] operationExecutor.UnmountVolume started for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1uid" (UID: "pod1uid") 
I0624 08:37:43.865309  220043 operation_generator.go:797] UnmountVolume.TearDown succeeded for volume "fake-plugin/fake-device1" (OuterVolumeSpecName: "volume-name") pod "pod1uid" (UID: "pod1uid"). InnerVolumeSpecName "volume-name". PluginName "fake-plugin", VolumeGidValue ""
I0624 08:37:43.865996  220043 reconciler.go:312] operationExecutor.UnmountDevice started for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") on node "mynodename" 
I0624 08:37:43.866410  220043 reconciler_test.go:1798] UnmountDevice called
I0624 08:37:43.866612  220043 operation_generator.go:891] UnmountDevice succeeded for volume "volume-name" %!(EXTRA string=UnmountDevice succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") on node "mynodename" )
I0624 08:37:43.867560  220043 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:37:43.867679  220043 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0624 08:37:43.896008  220043 operation_generator.go:594] MountVolume.MountDevice succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") device mount path ""
FAIL

				from junit_bazel.xml

Find pod1 mentions in log files | View test history on testgrid


Show 1937 Passed Tests