This job view page is being replaced by Spyglass soon. Check out the new job view.
PRnilo19: Cherry pick of #103470: fix: return empty VMAS name if using standalone VM
ResultFAILURE
Tests 4 failed / 1936 succeeded
Started2021-07-16 19:02
Elapsed30m43s
Revisiond07de023fc519cce5858ffee9e3403fa7055de9a
Refs 103600

Test Failures


//pkg/controller/volume/attachdetach/go_default_test:run_1_of_2 0.00s

bazel test //pkg/controller/volume/attachdetach/go_default_test:run_1_of_2
exec ${PAGER:-/usr/bin/less} "$0" || exit 1
Executing tests from //pkg/controller/volume/attachdetach:go_default_test
-----------------------------------------------------------------------------
I0716 19:24:27.731283  214571 shared_informer.go:240] Waiting for caches to sync for attach detach
I0716 19:24:27.832042  214571 shared_informer.go:247] Caches are synced for attach detach 
I0716 19:24:27.844369  214571 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0716 19:24:27.844500  214571 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0716 19:24:27.844863  214571 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0716 19:24:27.846748  214571 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0716 19:24:27.846860  214571 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0716 19:24:27.848001  214571 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0716 19:24:27.851015  214571 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0716 19:24:27.851129  214571 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0716 19:24:27.852246  214571 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0716 19:24:27.854648  214571 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0716 19:24:27.854776  214571 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0716 19:24:27.856257  214571 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0716 19:24:27.859028  214571 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0716 19:24:27.859141  214571 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0716 19:24:27.859544  214571 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0716 19:24:27.860708  214571 event.go:291] "Event occurred" object="mynamespace/mypod-0" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0716 19:24:27.860889  214571 operation_generator.go:360] AttachVolume.Attach succeeded for volume "volumeName" (UniqueName: "kubernetes.io/testPlugin/volumeName") from node "mynode" 
I0716 19:24:27.861212  214571 event.go:291] "Event occurred" object="mynamespace/mypod-2" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0716 19:24:27.861402  214571 event.go:291] "Event occurred" object="mynamespace/mypod-3" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0716 19:24:27.861575  214571 event.go:291] "Event occurred" object="mynamespace/mypod-1" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0716 19:24:27.861757  214571 event.go:291] "Event occurred" object="mynamespace/mypod-4" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0716 19:24:27.862529  214571 reconciler.go:295] attacherDetacher.AttachVolume started for volume "volumeName" (UniqueName: "kubernetes.io/testPlugin/volumeName") from node "mynode" 
I0716 19:24:29.837147  214571 shared_informer.go:240] Waiting for caches to sync for attach detach
I0716 19:24:29.938993  214571 shared_informer.go:247] Caches are synced for attach detach 
I0716 19:24:29.947269  214571 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0716 19:24:29.947436  214571 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0716 19:24:29.947683  214571 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0716 19:24:29.949256  214571 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0716 19:24:29.949380  214571 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0716 19:24:29.949547  214571 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0716 19:24:29.951222  214571 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0716 19:24:29.951348  214571 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0716 19:24:29.951635  214571 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0716 19:24:29.953122  214571 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0716 19:24:29.953232  214571 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0716 19:24:29.954367  214571 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0716 19:24:29.956936  214571 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0716 19:24:29.957050  214571 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0716 19:24:29.966777  214571 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0716 19:24:29.967784  214571 event.go:291] "Event occurred" object="mynamespace/mypod-0" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0716 19:24:29.968216  214571 operation_generator.go:360] AttachVolume.Attach succeeded for volume "volumeName" (UniqueName: "kubernetes.io/testPlugin/volumeName") from node "mynode" 
I0716 19:24:29.968342  214571 event.go:291] "Event occurred" object="mynamespace/mypod-1" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0716 19:24:29.968635  214571 event.go:291] "Event occurred" object="mynamespace/mypod-2" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0716 19:24:29.968820  214571 event.go:291] "Event occurred" object="mynamespace/mypod-3" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0716 19:24:29.968991  214571 event.go:291] "Event occurred" object="mynamespace/mypod-4" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0716 19:24:29.969048  214571 reconciler.go:295] attacherDetacher.AttachVolume started for volume "volumeName" (UniqueName: "kubernetes.io/testPlugin/volumeName") from node "mynode" 
I0716 19:24:29.969612  214571 operation_generator.go:360] AttachVolume.Attach succeeded for volume "volumeName2" (UniqueName: "kubernetes.io/testPlugin/volumeName2") from node "mynode-1" 
I0716 19:24:29.969951  214571 event.go:291] "Event occurred" object="mynamespace/newpod-1" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName2\" "
I0716 19:24:29.970074  214571 reconciler.go:295] attacherDetacher.AttachVolume started for volume "volumeName2" (UniqueName: "kubernetes.io/testPlugin/volumeName2") from node "mynode-1" 
I0716 19:24:31.943345  214571 shared_informer.go:240] Waiting for caches to sync for attach detach
I0716 19:24:32.043895  214571 shared_informer.go:247] Caches are synced for attach detach 
I0716 19:24:32.051687  214571 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0716 19:24:32.051842  214571 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0716 19:24:32.052141  214571 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0716 19:24:32.053539  214571 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0716 19:24:32.053658  214571 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0716 19:24:32.053924  214571 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0716 19:24:32.055398  214571 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0716 19:24:32.055517  214571 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0716 19:24:32.055677  214571 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0716 19:24:32.057060  214571 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0716 19:24:32.057172  214571 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0716 19:24:32.057447  214571 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0716 19:24:32.058958  214571 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0716 19:24:32.059070  214571 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0716 19:24:32.065482  214571 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0716 19:24:32.066731  214571 operation_generator.go:360] AttachVolume.Attach succeeded for volume "volumeName" (UniqueName: "kubernetes.io/testPlugin/volumeName") from node "mynode" 
I0716 19:24:32.067547  214571 event.go:291] "Event occurred" object="mynamespace/mypod-1" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0716 19:24:32.067758  214571 event.go:291] "Event occurred" object="mynamespace/mypod-4" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0716 19:24:32.067955  214571 event.go:291] "Event occurred" object="mynamespace/mypod-0" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0716 19:24:32.068167  214571 event.go:291] "Event occurred" object="mynamespace/mypod-2" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0716 19:24:32.068351  214571 event.go:291] "Event occurred" object="mynamespace/mypod-3" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0716 19:24:32.069110  214571 reconciler.go:295] attacherDetacher.AttachVolume started for volume "volumeName" (UniqueName: "kubernetes.io/testPlugin/volumeName") from node "mynode" 
I0716 19:24:32.069870  214571 event.go:291] "Event occurred" object="mynamespace/newpod-1" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName2\" "
I0716 19:24:32.070094  214571 operation_generator.go:360] AttachVolume.Attach succeeded for volume "volumeName2" (UniqueName: "kubernetes.io/testPlugin/volumeName2") from node "mynode-1" 
I0716 19:24:32.070510  214571 reconciler.go:295] attacherDetacher.AttachVolume started for volume "volumeName2" (UniqueName: "kubernetes.io/testPlugin/volumeName2") from node "mynode-1" 
I0716 19:24:34.048376  214571 shared_informer.go:240] Waiting for caches to sync for attach detach
I0716 19:24:34.148988  214571 shared_informer.go:247] Caches are synced for attach detach 
I0716 19:24:34.155824  214571 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0716 19:24:34.157227  214571 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0716 19:24:34.158347  214571 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0716 19:24:34.158454  214571 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0716 19:24:34.158586  214571 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0716 19:24:34.158473  214571 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0716 19:24:34.159009  214571 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0716 19:24:34.159129  214571 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0716 19:24:34.159237  214571 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0716 19:24:34.160425  214571 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0716 19:24:34.160579  214571 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0716 19:24:34.160702  214571 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0716 19:24:34.161805  214571 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0716 19:24:34.161910  214571 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0716 19:24:34.162036  214571 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0716 19:24:34.162118  214571 reconciler.go:295] attacherDetacher.AttachVolume started for volume "volumeName" (UniqueName: "kubernetes.io/testPlugin/volumeName") from node "mynode" 
I0716 19:24:34.162352  214571 reconciler.go:295] attacherDetacher.AttachVolume started for volume "volumeName2" (UniqueName: "kubernetes.io/testPlugin/volumeName2") from node "mynode-1" 
I0716 19:24:34.163143  214571 reconciler.go:295] attacherDetacher.AttachVolume started for volume "volumeName3" (UniqueName: "kubernetes.io/testPlugin/volumeName3") from node "mynode-1" 
I0716 19:24:34.163468  214571 operation_generator.go:360] AttachVolume.Attach succeeded for volume "volumeName" (UniqueName: "kubernetes.io/testPlugin/volumeName") from node "mynode" 
I0716 19:24:34.163568  214571 operation_generator.go:360] AttachVolume.Attach succeeded for volume "volumeName3" (UniqueName: "kubernetes.io/testPlugin/volumeName3") from node "mynode-1" 
I0716 19:24:34.164014  214571 operation_generator.go:360] AttachVolume.Attach succeeded for volume "volumeName2" (UniqueName: "kubernetes.io/testPlugin/volumeName2") from node "mynode-1" 
I0716 19:24:34.164386  214571 event.go:291] "Event occurred" object="mynamespace/mypod-0" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0716 19:24:34.165487  214571 event.go:291] "Event occurred" object="mynamespace/newpod-2" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName3\" "
I0716 19:24:34.165660  214571 event.go:291] "Event occurred" object="mynamespace/mypod-1" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0716 19:24:34.165818  214571 event.go:291] "Event occurred" object="mynamespace/mypod-2" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0716 19:24:34.165992  214571 event.go:291] "Event occurred" object="mynamespace/mypod-3" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0716 19:24:34.166182  214571 event.go:291] "Event occurred" object="mynamespace/mypod-4" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0716 19:24:34.166369  214571 event.go:291] "Event occurred" object="mynamespace/newpod-1" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName2\" "
I0716 19:24:36.154130  214571 shared_informer.go:240] Waiting for caches to sync for attach detach
I0716 19:24:36.166635  214571 shared_informer.go:247] Caches are synced for attach detach 
I0716 19:24:36.175046  214571 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0716 19:24:36.175184  214571 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0716 19:24:36.175381  214571 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0716 19:24:36.176913  214571 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0716 19:24:36.177042  214571 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0716 19:24:36.177305  214571 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0716 19:24:36.178885  214571 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0716 19:24:36.179012  214571 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0716 19:24:36.179249  214571 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0716 19:24:36.180830  214571 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0716 19:24:36.180934  214571 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0716 19:24:36.181094  214571 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0716 19:24:36.182583  214571 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0716 19:24:36.182692  214571 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0716 19:24:36.182915  214571 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0716 19:24:36.183358  214571 operation_generator.go:360] AttachVolume.Attach succeeded for volume "vol1" (UniqueName: "kubernetes.io/testPlugin/vol1") from node "mynode-1" 
I0716 19:24:36.183838  214571 event.go:291] "Event occurred" object="mynamespace/pod1" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"vol1\" "
I0716 19:24:36.184089  214571 reconciler.go:295] attacherDetacher.AttachVolume started for volume "vol1" (UniqueName: "kubernetes.io/testPlugin/vol1") from node "mynode-1" 
I0716 19:24:36.185239  214571 operation_generator.go:360] AttachVolume.Attach succeeded for volume "volumeName" (UniqueName: "kubernetes.io/testPlugin/volumeName") from node "mynode" 
I0716 19:24:36.185789  214571 event.go:291] "Event occurred" object="mynamespace/mypod-4" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0716 19:24:36.185989  214571 event.go:291] "Event occurred" object="mynamespace/mypod-0" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0716 19:24:36.186173  214571 event.go:291] "Event occurred" object="mynamespace/mypod-1" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0716 19:24:36.186357  214571 event.go:291] "Event occurred" object="mynamespace/mypod-2" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0716 19:24:36.186520  214571 event.go:291] "Event occurred" object="mynamespace/mypod-3" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0716 19:24:36.187101  214571 reconciler.go:295] attacherDetacher.AttachVolume started for volume "volumeName" (UniqueName: "kubernetes.io/testPlugin/volumeName") from node "mynode" 
I0716 19:24:37.171096  214571 shared_informer.go:240] Waiting for caches to sync for attach detach
I0716 19:24:37.274605  214571 shared_informer.go:247] Caches are synced for attach detach 
I0716 19:24:37.283579  214571 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0716 19:24:37.283725  214571 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0716 19:24:37.283864  214571 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0716 19:24:37.285080  214571 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0716 19:24:37.285194  214571 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0716 19:24:37.285339  214571 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0716 19:24:37.286779  214571 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0716 19:24:37.286889  214571 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0716 19:24:37.287033  214571 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0716 19:24:37.288428  214571 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0716 19:24:37.288534  214571 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0716 19:24:37.288670  214571 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0716 19:24:37.290105  214571 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0716 19:24:37.290232  214571 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0716 19:24:37.290372  214571 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0716 19:24:37.291616  214571 event.go:291] "Event occurred" object="mynamespace/mypod-0" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0716 19:24:37.291727  214571 operation_generator.go:360] AttachVolume.Attach succeeded for volume "volumeName" (UniqueName: "kubernetes.io/testPlugin/volumeName") from node "mynode" 
I0716 19:24:37.291991  214571 event.go:291] "Event occurred" object="mynamespace/mypod-3" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0716 19:24:37.292196  214571 event.go:291] "Event occurred" object="mynamespace/mypod-2" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0716 19:24:37.292293  214571 reconciler.go:295] attacherDetacher.AttachVolume started for volume "volumeName" (UniqueName: "kubernetes.io/testPlugin/volumeName") from node "mynode" 
I0716 19:24:37.292378  214571 event.go:291] "Event occurred" object="mynamespace/mypod-1" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0716 19:24:37.292551  214571 event.go:291] "Event occurred" object="mynamespace/mypod-4" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0716 19:24:38.196622  214571 reconciler.go:219] attacherDetacher.DetachVolume started for volume "pv1" (UniqueName: "kubernetes.io/testPlugin/vol1") on node "mynode-1" 
I0716 19:24:38.197139  214571 operation_generator.go:1409] Verified volume is safe to detach for volume "pv1" (UniqueName: "kubernetes.io/testPlugin/vol1") on node "mynode-1" 
I0716 19:24:38.197261  214571 operation_generator.go:470] DetachVolume.Detach succeeded for volume "pv1" (UniqueName: "kubernetes.io/testPlugin/vol1") on node "mynode-1" 
I0716 19:24:38.279615  214571 shared_informer.go:240] Waiting for caches to sync for attach detach
I0716 19:24:38.380339  214571 shared_informer.go:247] Caches are synced for attach detach 
I0716 19:24:38.389392  214571 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0716 19:24:38.389614  214571 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0716 19:24:38.390054  214571 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0716 19:24:38.391766  214571 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0716 19:24:38.391891  214571 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0716 19:24:38.392186  214571 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0716 19:24:38.393836  214571 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0716 19:24:38.393947  214571 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0716 19:24:38.394119  214571 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0716 19:24:38.396113  214571 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0716 19:24:38.396230  214571 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0716 19:24:38.396476  214571 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0716 19:24:38.398167  214571 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0716 19:24:38.398284  214571 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0716 19:24:38.398445  214571 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0716 19:24:38.399739  214571 event.go:291] "Event occurred" object="mynamespace/mypod-0" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0716 19:24:38.399930  214571 operation_generator.go:360] AttachVolume.Attach succeeded for volume "volumeName" (UniqueName: "kubernetes.io/testPlugin/volumeName") from node "mynode" 
I0716 19:24:38.400188  214571 event.go:291] "Event occurred" object="mynamespace/mypod-1" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0716 19:24:38.400382  214571 event.go:291] "Event occurred" object="mynamespace/mypod-2" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0716 19:24:38.400557  214571 event.go:291] "Event occurred" object="mynamespace/mypod-3" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0716 19:24:38.400732  214571 event.go:291] "Event occurred" object="mynamespace/mypod-4" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0716 19:24:38.400768  214571 reconciler.go:295] attacherDetacher.AttachVolume started for volume "volumeName" (UniqueName: "kubernetes.io/testPlugin/volumeName") from node "mynode" 
I0716 19:24:39.314492  214571 reconciler.go:219] attacherDetacher.DetachVolume started for volume "pv1" (UniqueName: "kubernetes.io/testPlugin/vol1") on node "mynode-1" 
I0716 19:24:39.315025  214571 operation_generator.go:1409] Verified volume is safe to detach for volume "pv1" (UniqueName: "kubernetes.io/testPlugin/vol1") on node "mynode-1" 
I0716 19:24:39.315147  214571 operation_generator.go:470] DetachVolume.Detach succeeded for volume "pv1" (UniqueName: "kubernetes.io/testPlugin/vol1") on node "mynode-1" 
I0716 19:24:39.385446  214571 shared_informer.go:240] Waiting for caches to sync for attach detach
I0716 19:24:39.485983  214571 shared_informer.go:247] Caches are synced for attach detach 
W0716 19:24:39.488619  214571 reflector.go:436] k8s.io/client-go/informers/factory.go:134: watch of *v1.PersistentVolume ended with: very short watch: k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
W0716 19:24:39.488799  214571 reflector.go:436] k8s.io/client-go/informers/factory.go:134: watch of *v1.CSINode ended with: very short watch: k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
I0716 19:24:39.492827  214571 shared_informer.go:240] Waiting for caches to sync for attach detach
E0716 19:24:39.503208  214571 reconciler.go:228] attacherDetacher.DetachVolume failed to start for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" : DetachVolume.FindAttachablePluginByName failed for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" : no volume plugin matched name: kubernetes.io/testPlugin
E0716 19:24:39.504595  214571 reconciler.go:228] attacherDetacher.DetachVolume failed to start for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" : DetachVolume.FindAttachablePluginByName failed for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" : no volume plugin matched name: kubernetes.io/testPlugin
E0716 19:24:39.505921  214571 reconciler.go:228] attacherDetacher.DetachVolume failed to start for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" : DetachVolume.FindAttachablePluginByName failed for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" : no volume plugin matched name: kubernetes.io/testPlugin
E0716 19:24:39.507265  214571 reconciler.go:228] attacherDetacher.DetachVolume failed to start for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" : DetachVolume.FindAttachablePluginByName failed for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" : no volume plugin matched name: kubernetes.io/testPlugin
I0716 19:24:39.508825  214571 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/csi/pd.csi.storage.gke.io^projects/UNSPECIFIED/zones/UNSPECIFIED/disks/vol1") on node "mynode-1" 
E0716 19:24:39.509161  214571 csi_attacher.go:455] [attachment.ID=csi-77bd5b88d32d48e5a31f0073211a7dd54ed0066f008e12364f4b6ecb61dc7370] watch channel had been closed
E0716 19:24:39.509766  214571 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/csi/pd.csi.storage.gke.io^projects/UNSPECIFIED/zones/UNSPECIFIED/disks/vol1 podName: nodeName:mynode-1}" failed. No retries permitted until 2021-07-16 19:24:40.009512842 +0000 UTC m=+12.829464580 (durationBeforeRetry 500ms). Error: "DetachVolume.Detach failed for volume \"nil\" (UniqueName: \"kubernetes.io/csi/pd.csi.storage.gke.io^projects/UNSPECIFIED/zones/UNSPECIFIED/disks/vol1\") on node \"mynode-1\" : volume attachment watch channel had been closed"
I0716 19:24:39.509884  214571 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/csi/pd.csi.storage.gke.io^projects/UNSPECIFIED/zones/UNSPECIFIED/disks/vol1") on node "mynode-1" 
==================
WARNING: DATA RACE
Write at 0x00c00041e218 by goroutine 78:
  k8s.io/kubernetes/pkg/controller/volume/attachdetach/testing.CreateTestClient.func7()
      pkg/controller/volume/attachdetach/testing/testvolumespec.go:185 +0x207
  k8s.io/kubernetes/vendor/k8s.io/client-go/testing.(*SimpleReactor).React()
      staging/src/k8s.io/client-go/testing/fixture.go:521 +0x68
  k8s.io/kubernetes/vendor/k8s.io/client-go/testing.(*Fake).Invokes()
      staging/src/k8s.io/client-go/testing/fake.go:141 +0x327
  k8s.io/kubernetes/vendor/k8s.io/client-go/kubernetes/typed/storage/v1/fake.(*FakeVolumeAttachments).Create()
      staging/src/k8s.io/client-go/kubernetes/typed/storage/v1/fake/fake_volumeattachment.go:82 +0x246
  k8s.io/kubernetes/pkg/volume/csi.(*csiAttacher).Attach()
      pkg/volume/csi/csi_attacher.go:109 +0x5f3
  k8s.io/kubernetes/pkg/volume/util/operationexecutor.(*operationGenerator).GenerateAttachVolumeFunc.func1()
      pkg/volume/util/operationexecutor/operation_generator.go:335 +0x275
  k8s.io/kubernetes/pkg/volume/util/types.(*GeneratedOperations).Run()
      pkg/volume/util/types/types.go:54 +0x142
  k8s.io/kubernetes/pkg/volume/util/nestedpendingoperations.(*nestedPendingOperations).Run.func1()
      pkg/volume/util/nestedpendingoperations/nestedpendingoperations.go:183 +0x159

Previous read at 0x00c00041e218 by goroutine 98:
  k8s.io/kubernetes/pkg/controller/volume/attachdetach/testing.CreateTestClient.func6()
      pkg/controller/volume/attachdetach/testing/testvolumespec.go:179 +0x104
  k8s.io/kubernetes/vendor/k8s.io/client-go/testing.(*SimpleReactor).React()
      staging/src/k8s.io/client-go/testing/fixture.go:521 +0x68
  k8s.io/kubernetes/vendor/k8s.io/client-go/testing.(*Fake).Invokes()
      staging/src/k8s.io/client-go/testing/fake.go:141 +0x327
  k8s.io/kubernetes/vendor/k8s.io/client-go/kubernetes/typed/storage/v1/fake.(*FakeVolumeAttachments).List()
      staging/src/k8s.io/client-go/kubernetes/typed/storage/v1/fake/fake_volumeattachment.go:55 +0x273
  k8s.io/kubernetes/vendor/k8s.io/client-go/informers/storage/v1.NewFilteredVolumeAttachmentInformer.func1()
      staging/src/k8s.io/client-go/informers/storage/v1/volumeattachment.go:64 +0x23c
  k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.(*ListWatch).List()
      staging/src/k8s.io/client-go/tools/cache/listwatch.go:106 +0x97
  k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch.func1.1.2()
      staging/src/k8s.io/client-go/tools/cache/reflector.go:277 +0xf4
  k8s.io/kubernetes/vendor/k8s.io/client-go/tools/pager.SimplePageFunc.func1()
      staging/src/k8s.io/client-go/tools/pager/pager.go:40 +0x97
  k8s.io/kubernetes/vendor/k8s.io/client-go/tools/pager.(*ListPager).List()
      staging/src/k8s.io/client-go/tools/pager/pager.go:91 +0x1e2
  k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch.func1.1()
      staging/src/k8s.io/client-go/tools/cache/reflector.go:302 +0x2b7

Goroutine 78 (running) created at:
  k8s.io/kubernetes/pkg/volume/util/nestedpendingoperations.(*nestedPendingOperations).Run()
      pkg/volume/util/nestedpendingoperations/nestedpendingoperations.go:178 +0x5b2
  k8s.io/kubernetes/pkg/volume/util/operationexecutor.(*operationExecutor).AttachVolume()
      pkg/volume/util/operationexecutor/operation_executor.go:675 +0x348
  k8s.io/kubernetes/pkg/controller/volume/attachdetach/reconciler.(*reconciler).attachDesiredVolumes()
      pkg/controller/volume/attachdetach/reconciler/reconciler.go:293 +0x639
  k8s.io/kubernetes/pkg/controller/volume/attachdetach/reconciler.(*reconciler).reconcile()
      pkg/controller/volume/attachdetach/reconciler/reconciler.go:233 +0x1b9
  k8s.io/kubernetes/pkg/controller/volume/attachdetach/reconciler.(*reconciler).reconciliationLoopFunc.func1()
      pkg/controller/volume/attachdetach/reconciler/reconciler.go:110 +0x5e
  k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1()
      staging/src/k8s.io/apimachinery/pkg/util/wait/wait.go:155 +0x75
  k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.BackoffUntil()
      staging/src/k8s.io/apimachinery/pkg/util/wait/wait.go:156 +0xba
  k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil()
      staging/src/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x114
  k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.Until()
      staging/src/k8s.io/apimachinery/pkg/util/wait/wait.go:90 +0x89
  k8s.io/kubernetes/pkg/controller/volume/attachdetach/reconciler.(*reconciler).Run()
      pkg/controller/volume/attachdetach/reconciler/reconciler.go:101 +0x2f

Goroutine 98 (finished) created at:
  k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch.func1()
      staging/src/k8s.io/client-go/tools/cache/reflector.go:268 +0x36f
  k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch()
      staging/src/k8s.io/client-go/tools/cache/reflector.go:360 +0x2e4
  k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.(*Reflector).Run.func1()
      staging/src/k8s.io/client-go/tools/cache/reflector.go:221 +0x53
  k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1()
      staging/src/k8s.io/apimachinery/pkg/util/wait/wait.go:155 +0x75
  k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.BackoffUntil()
      staging/src/k8s.io/apimachinery/pkg/util/wait/wait.go:156 +0xba
  k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.(*Reflector).Run()
      staging/src/k8s.io/client-go/tools/cache/reflector.go:220 +0x2c4
  k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.(*Reflector).Run-fm()
      staging/src/k8s.io/client-go/tools/cache/reflector.go:218 +0x4b
  k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).StartWithChannel.func1()
      staging/src/k8s.io/apimachinery/pkg/util/wait/wait.go:56 +0x45
  k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
      staging/src/k8s.io/apimachinery/pkg/util/wait/wait.go:73 +0x6d
==================
E0716 19:24:39.510930  214571 csi_attacher.go:455] [attachment.ID=csi-4ff6fd4b10c02f804e7435dffeb62054338b590749cdaacf18e198ee0ab6dd66] watch channel had been closed
I0716 19:24:39.511405  214571 reconciler.go:295] attacherDetacher.AttachVolume started for volume "pd.csi.storage.gke.io-pdName" (UniqueName: "kubernetes.io/csi/pd.csi.storage.gke.io^projects/UNSPECIFIED/zones/UNSPECIFIED/disks/pdName") from node "mynode" 
E0716 19:24:39.512468  214571 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/csi/pd.csi.storage.gke.io^projects/UNSPECIFIED/zones/UNSPECIFIED/disks/pdName podName: nodeName:}" failed. No retries permitted until 2021-07-16 19:24:40.012212995 +0000 UTC m=+12.832164728 (durationBeforeRetry 500ms). Error: "AttachVolume.Attach failed for volume \"pd.csi.storage.gke.io-pdName\" (UniqueName: \"kubernetes.io/csi/pd.csi.storage.gke.io^projects/UNSPECIFIED/zones/UNSPECIFIED/disks/pdName\") from node \"mynode\" : volume attachment watch channel had been closed"
I0716 19:24:39.512798  214571 event.go:291] "Event occurred" object="mynamespace/mypod-1" kind="Pod" apiVersion="v1" type="Warning" reason="FailedAttachVolume" message="AttachVolume.Attach failed for volume \"pd.csi.storage.gke.io-pdName\" : volume attachment watch channel had been closed"
I0716 19:24:39.513025  214571 event.go:291] "Event occurred" object="mynamespace/mypod-0" kind="Pod" apiVersion="v1" type="Warning" reason="FailedAttachVolume" message="AttachVolume.Attach failed for volume \"pd.csi.storage.gke.io-pdName\" : volume attachment watch channel had been closed"
I0716 19:24:39.513300  214571 event.go:291] "Event occurred" object="mynamespace/mypod-2" kind="Pod" apiVersion="v1" type="Warning" reason="FailedAttachVolume" message="AttachVolume.Attach failed for volume \"pd.csi.storage.gke.io-pdName\" : volume attachment watch channel had been closed"
I0716 19:24:39.513509  214571 event.go:291] "Event occurred" object="mynamespace/mypod-3" kind="Pod" apiVersion="v1" type="Warning" reason="FailedAttachVolume" message="AttachVolume.Attach failed for volume \"pd.csi.storage.gke.io-pdName\" : volume attachment watch channel had been closed"
I0716 19:24:39.513697  214571 event.go:291] "Event occurred" object="mynamespace/mypod-4" kind="Pod" apiVersion="v1" type="Warning" reason="FailedAttachVolume" message="AttachVolume.Attach failed for volume \"pd.csi.storage.gke.io-pdName\" : volume attachment watch channel had been closed"
I0716 19:24:39.603338  214571 shared_informer.go:247] Caches are synced for attach detach 
W0716 19:24:39.605887  214571 reflector.go:436] k8s.io/client-go/informers/factory.go:134: watch of *v1.CSIDriver ended with: very short watch: k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
W0716 19:24:39.606113  214571 reflector.go:436] k8s.io/client-go/informers/factory.go:134: watch of *v1.VolumeAttachment ended with: very short watch: k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
W0716 19:24:39.606517  214571 reflector.go:436] k8s.io/client-go/informers/factory.go:134: watch of *v1.PersistentVolume ended with: very short watch: k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
--- FAIL: Test_ADC_VolumeAttachmentRecovery (3.46s)
    --- FAIL: Test_ADC_VolumeAttachmentRecovery/Deleted_Pod_with_migrated_PV (0.12s)
        testing.go:1038: race detected during execution of test
    testing.go:1038: race detected during execution of test
FAIL

				from junit_bazel.xml

Filter through log files | View test history on testgrid


//pkg/controller/volume/attachdetach/go_default_test:run_1_of_2 0.00s

bazel test //pkg/controller/volume/attachdetach/go_default_test:run_1_of_2
exec ${PAGER:-/usr/bin/less} "$0" || exit 1
Executing tests from //pkg/controller/volume/attachdetach:go_default_test
-----------------------------------------------------------------------------
I0716 19:24:27.731283  214571 shared_informer.go:240] Waiting for caches to sync for attach detach
I0716 19:24:27.832042  214571 shared_informer.go:247] Caches are synced for attach detach 
I0716 19:24:27.844369  214571 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0716 19:24:27.844500  214571 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0716 19:24:27.844863  214571 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0716 19:24:27.846748  214571 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0716 19:24:27.846860  214571 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0716 19:24:27.848001  214571 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0716 19:24:27.851015  214571 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0716 19:24:27.851129  214571 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0716 19:24:27.852246  214571 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0716 19:24:27.854648  214571 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0716 19:24:27.854776  214571 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0716 19:24:27.856257  214571 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0716 19:24:27.859028  214571 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0716 19:24:27.859141  214571 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0716 19:24:27.859544  214571 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0716 19:24:27.860708  214571 event.go:291] "Event occurred" object="mynamespace/mypod-0" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0716 19:24:27.860889  214571 operation_generator.go:360] AttachVolume.Attach succeeded for volume "volumeName" (UniqueName: "kubernetes.io/testPlugin/volumeName") from node "mynode" 
I0716 19:24:27.861212  214571 event.go:291] "Event occurred" object="mynamespace/mypod-2" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0716 19:24:27.861402  214571 event.go:291] "Event occurred" object="mynamespace/mypod-3" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0716 19:24:27.861575  214571 event.go:291] "Event occurred" object="mynamespace/mypod-1" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0716 19:24:27.861757  214571 event.go:291] "Event occurred" object="mynamespace/mypod-4" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0716 19:24:27.862529  214571 reconciler.go:295] attacherDetacher.AttachVolume started for volume "volumeName" (UniqueName: "kubernetes.io/testPlugin/volumeName") from node "mynode" 
I0716 19:24:29.837147  214571 shared_informer.go:240] Waiting for caches to sync for attach detach
I0716 19:24:29.938993  214571 shared_informer.go:247] Caches are synced for attach detach 
I0716 19:24:29.947269  214571 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0716 19:24:29.947436  214571 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0716 19:24:29.947683  214571 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0716 19:24:29.949256  214571 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0716 19:24:29.949380  214571 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0716 19:24:29.949547  214571 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0716 19:24:29.951222  214571 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0716 19:24:29.951348  214571 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0716 19:24:29.951635  214571 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0716 19:24:29.953122  214571 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0716 19:24:29.953232  214571 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0716 19:24:29.954367  214571 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0716 19:24:29.956936  214571 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0716 19:24:29.957050  214571 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0716 19:24:29.966777  214571 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0716 19:24:29.967784  214571 event.go:291] "Event occurred" object="mynamespace/mypod-0" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0716 19:24:29.968216  214571 operation_generator.go:360] AttachVolume.Attach succeeded for volume "volumeName" (UniqueName: "kubernetes.io/testPlugin/volumeName") from node "mynode" 
I0716 19:24:29.968342  214571 event.go:291] "Event occurred" object="mynamespace/mypod-1" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0716 19:24:29.968635  214571 event.go:291] "Event occurred" object="mynamespace/mypod-2" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0716 19:24:29.968820  214571 event.go:291] "Event occurred" object="mynamespace/mypod-3" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0716 19:24:29.968991  214571 event.go:291] "Event occurred" object="mynamespace/mypod-4" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0716 19:24:29.969048  214571 reconciler.go:295] attacherDetacher.AttachVolume started for volume "volumeName" (UniqueName: "kubernetes.io/testPlugin/volumeName") from node "mynode" 
I0716 19:24:29.969612  214571 operation_generator.go:360] AttachVolume.Attach succeeded for volume "volumeName2" (UniqueName: "kubernetes.io/testPlugin/volumeName2") from node "mynode-1" 
I0716 19:24:29.969951  214571 event.go:291] "Event occurred" object="mynamespace/newpod-1" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName2\" "
I0716 19:24:29.970074  214571 reconciler.go:295] attacherDetacher.AttachVolume started for volume "volumeName2" (UniqueName: "kubernetes.io/testPlugin/volumeName2") from node "mynode-1" 
I0716 19:24:31.943345  214571 shared_informer.go:240] Waiting for caches to sync for attach detach
I0716 19:24:32.043895  214571 shared_informer.go:247] Caches are synced for attach detach 
I0716 19:24:32.051687  214571 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0716 19:24:32.051842  214571 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0716 19:24:32.052141  214571 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0716 19:24:32.053539  214571 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0716 19:24:32.053658  214571 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0716 19:24:32.053924  214571 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0716 19:24:32.055398  214571 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0716 19:24:32.055517  214571 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0716 19:24:32.055677  214571 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0716 19:24:32.057060  214571 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0716 19:24:32.057172  214571 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0716 19:24:32.057447  214571 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0716 19:24:32.058958  214571 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0716 19:24:32.059070  214571 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0716 19:24:32.065482  214571 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0716 19:24:32.066731  214571 operation_generator.go:360] AttachVolume.Attach succeeded for volume "volumeName" (UniqueName: "kubernetes.io/testPlugin/volumeName") from node "mynode" 
I0716 19:24:32.067547  214571 event.go:291] "Event occurred" object="mynamespace/mypod-1" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0716 19:24:32.067758  214571 event.go:291] "Event occurred" object="mynamespace/mypod-4" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0716 19:24:32.067955  214571 event.go:291] "Event occurred" object="mynamespace/mypod-0" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0716 19:24:32.068167  214571 event.go:291] "Event occurred" object="mynamespace/mypod-2" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0716 19:24:32.068351  214571 event.go:291] "Event occurred" object="mynamespace/mypod-3" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0716 19:24:32.069110  214571 reconciler.go:295] attacherDetacher.AttachVolume started for volume "volumeName" (UniqueName: "kubernetes.io/testPlugin/volumeName") from node "mynode" 
I0716 19:24:32.069870  214571 event.go:291] "Event occurred" object="mynamespace/newpod-1" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName2\" "
I0716 19:24:32.070094  214571 operation_generator.go:360] AttachVolume.Attach succeeded for volume "volumeName2" (UniqueName: "kubernetes.io/testPlugin/volumeName2") from node "mynode-1" 
I0716 19:24:32.070510  214571 reconciler.go:295] attacherDetacher.AttachVolume started for volume "volumeName2" (UniqueName: "kubernetes.io/testPlugin/volumeName2") from node "mynode-1" 
I0716 19:24:34.048376  214571 shared_informer.go:240] Waiting for caches to sync for attach detach
I0716 19:24:34.148988  214571 shared_informer.go:247] Caches are synced for attach detach 
I0716 19:24:34.155824  214571 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0716 19:24:34.157227  214571 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0716 19:24:34.158347  214571 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0716 19:24:34.158454  214571 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0716 19:24:34.158586  214571 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0716 19:24:34.158473  214571 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0716 19:24:34.159009  214571 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0716 19:24:34.159129  214571 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0716 19:24:34.159237  214571 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0716 19:24:34.160425  214571 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0716 19:24:34.160579  214571 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0716 19:24:34.160702  214571 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0716 19:24:34.161805  214571 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0716 19:24:34.161910  214571 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0716 19:24:34.162036  214571 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0716 19:24:34.162118  214571 reconciler.go:295] attacherDetacher.AttachVolume started for volume "volumeName" (UniqueName: "kubernetes.io/testPlugin/volumeName") from node "mynode" 
I0716 19:24:34.162352  214571 reconciler.go:295] attacherDetacher.AttachVolume started for volume "volumeName2" (UniqueName: "kubernetes.io/testPlugin/volumeName2") from node "mynode-1" 
I0716 19:24:34.163143  214571 reconciler.go:295] attacherDetacher.AttachVolume started for volume "volumeName3" (UniqueName: "kubernetes.io/testPlugin/volumeName3") from node "mynode-1" 
I0716 19:24:34.163468  214571 operation_generator.go:360] AttachVolume.Attach succeeded for volume "volumeName" (UniqueName: "kubernetes.io/testPlugin/volumeName") from node "mynode" 
I0716 19:24:34.163568  214571 operation_generator.go:360] AttachVolume.Attach succeeded for volume "volumeName3" (UniqueName: "kubernetes.io/testPlugin/volumeName3") from node "mynode-1" 
I0716 19:24:34.164014  214571 operation_generator.go:360] AttachVolume.Attach succeeded for volume "volumeName2" (UniqueName: "kubernetes.io/testPlugin/volumeName2") from node "mynode-1" 
I0716 19:24:34.164386  214571 event.go:291] "Event occurred" object="mynamespace/mypod-0" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0716 19:24:34.165487  214571 event.go:291] "Event occurred" object="mynamespace/newpod-2" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName3\" "
I0716 19:24:34.165660  214571 event.go:291] "Event occurred" object="mynamespace/mypod-1" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0716 19:24:34.165818  214571 event.go:291] "Event occurred" object="mynamespace/mypod-2" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0716 19:24:34.165992  214571 event.go:291] "Event occurred" object="mynamespace/mypod-3" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0716 19:24:34.166182  214571 event.go:291] "Event occurred" object="mynamespace/mypod-4" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0716 19:24:34.166369  214571 event.go:291] "Event occurred" object="mynamespace/newpod-1" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName2\" "
I0716 19:24:36.154130  214571 shared_informer.go:240] Waiting for caches to sync for attach detach
I0716 19:24:36.166635  214571 shared_informer.go:247] Caches are synced for attach detach 
I0716 19:24:36.175046  214571 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0716 19:24:36.175184  214571 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0716 19:24:36.175381  214571 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0716 19:24:36.176913  214571 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0716 19:24:36.177042  214571 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0716 19:24:36.177305  214571 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0716 19:24:36.178885  214571 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0716 19:24:36.179012  214571 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0716 19:24:36.179249  214571 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0716 19:24:36.180830  214571 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0716 19:24:36.180934  214571 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0716 19:24:36.181094  214571 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0716 19:24:36.182583  214571 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0716 19:24:36.182692  214571 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0716 19:24:36.182915  214571 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0716 19:24:36.183358  214571 operation_generator.go:360] AttachVolume.Attach succeeded for volume "vol1" (UniqueName: "kubernetes.io/testPlugin/vol1") from node "mynode-1" 
I0716 19:24:36.183838  214571 event.go:291] "Event occurred" object="mynamespace/pod1" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"vol1\" "
I0716 19:24:36.184089  214571 reconciler.go:295] attacherDetacher.AttachVolume started for volume "vol1" (UniqueName: "kubernetes.io/testPlugin/vol1") from node "mynode-1" 
I0716 19:24:36.185239  214571 operation_generator.go:360] AttachVolume.Attach succeeded for volume "volumeName" (UniqueName: "kubernetes.io/testPlugin/volumeName") from node "mynode" 
I0716 19:24:36.185789  214571 event.go:291] "Event occurred" object="mynamespace/mypod-4" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0716 19:24:36.185989  214571 event.go:291] "Event occurred" object="mynamespace/mypod-0" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0716 19:24:36.186173  214571 event.go:291] "Event occurred" object="mynamespace/mypod-1" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0716 19:24:36.186357  214571 event.go:291] "Event occurred" object="mynamespace/mypod-2" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0716 19:24:36.186520  214571 event.go:291] "Event occurred" object="mynamespace/mypod-3" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0716 19:24:36.187101  214571 reconciler.go:295] attacherDetacher.AttachVolume started for volume "volumeName" (UniqueName: "kubernetes.io/testPlugin/volumeName") from node "mynode" 
I0716 19:24:37.171096  214571 shared_informer.go:240] Waiting for caches to sync for attach detach
I0716 19:24:37.274605  214571 shared_informer.go:247] Caches are synced for attach detach 
I0716 19:24:37.283579  214571 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0716 19:24:37.283725  214571 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0716 19:24:37.283864  214571 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0716 19:24:37.285080  214571 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0716 19:24:37.285194  214571 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0716 19:24:37.285339  214571 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0716 19:24:37.286779  214571 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0716 19:24:37.286889  214571 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0716 19:24:37.287033  214571 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0716 19:24:37.288428  214571 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0716 19:24:37.288534  214571 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0716 19:24:37.288670  214571 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0716 19:24:37.290105  214571 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0716 19:24:37.290232  214571 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0716 19:24:37.290372  214571 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0716 19:24:37.291616  214571 event.go:291] "Event occurred" object="mynamespace/mypod-0" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0716 19:24:37.291727  214571 operation_generator.go:360] AttachVolume.Attach succeeded for volume "volumeName" (UniqueName: "kubernetes.io/testPlugin/volumeName") from node "mynode" 
I0716 19:24:37.291991  214571 event.go:291] "Event occurred" object="mynamespace/mypod-3" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0716 19:24:37.292196  214571 event.go:291] "Event occurred" object="mynamespace/mypod-2" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0716 19:24:37.292293  214571 reconciler.go:295] attacherDetacher.AttachVolume started for volume "volumeName" (UniqueName: "kubernetes.io/testPlugin/volumeName") from node "mynode" 
I0716 19:24:37.292378  214571 event.go:291] "Event occurred" object="mynamespace/mypod-1" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0716 19:24:37.292551  214571 event.go:291] "Event occurred" object="mynamespace/mypod-4" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0716 19:24:38.196622  214571 reconciler.go:219] attacherDetacher.DetachVolume started for volume "pv1" (UniqueName: "kubernetes.io/testPlugin/vol1") on node "mynode-1" 
I0716 19:24:38.197139  214571 operation_generator.go:1409] Verified volume is safe to detach for volume "pv1" (UniqueName: "kubernetes.io/testPlugin/vol1") on node "mynode-1" 
I0716 19:24:38.197261  214571 operation_generator.go:470] DetachVolume.Detach succeeded for volume "pv1" (UniqueName: "kubernetes.io/testPlugin/vol1") on node "mynode-1" 
I0716 19:24:38.279615  214571 shared_informer.go:240] Waiting for caches to sync for attach detach
I0716 19:24:38.380339  214571 shared_informer.go:247] Caches are synced for attach detach 
I0716 19:24:38.389392  214571 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0716 19:24:38.389614  214571 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0716 19:24:38.390054  214571 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" 
I0716 19:24:38.391766  214571 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0716 19:24:38.391891  214571 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0716 19:24:38.392186  214571 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" 
I0716 19:24:38.393836  214571 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0716 19:24:38.393947  214571 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0716 19:24:38.394119  214571 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" 
I0716 19:24:38.396113  214571 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0716 19:24:38.396230  214571 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0716 19:24:38.396476  214571 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-1" 
I0716 19:24:38.398167  214571 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0716 19:24:38.398284  214571 operation_generator.go:470] DetachVolume.Detach succeeded for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0716 19:24:38.398445  214571 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" 
I0716 19:24:38.399739  214571 event.go:291] "Event occurred" object="mynamespace/mypod-0" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0716 19:24:38.399930  214571 operation_generator.go:360] AttachVolume.Attach succeeded for volume "volumeName" (UniqueName: "kubernetes.io/testPlugin/volumeName") from node "mynode" 
I0716 19:24:38.400188  214571 event.go:291] "Event occurred" object="mynamespace/mypod-1" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0716 19:24:38.400382  214571 event.go:291] "Event occurred" object="mynamespace/mypod-2" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0716 19:24:38.400557  214571 event.go:291] "Event occurred" object="mynamespace/mypod-3" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0716 19:24:38.400732  214571 event.go:291] "Event occurred" object="mynamespace/mypod-4" kind="Pod" apiVersion="v1" type="Normal" reason="SuccessfulAttachVolume" message="AttachVolume.Attach succeeded for volume \"volumeName\" "
I0716 19:24:38.400768  214571 reconciler.go:295] attacherDetacher.AttachVolume started for volume "volumeName" (UniqueName: "kubernetes.io/testPlugin/volumeName") from node "mynode" 
I0716 19:24:39.314492  214571 reconciler.go:219] attacherDetacher.DetachVolume started for volume "pv1" (UniqueName: "kubernetes.io/testPlugin/vol1") on node "mynode-1" 
I0716 19:24:39.315025  214571 operation_generator.go:1409] Verified volume is safe to detach for volume "pv1" (UniqueName: "kubernetes.io/testPlugin/vol1") on node "mynode-1" 
I0716 19:24:39.315147  214571 operation_generator.go:470] DetachVolume.Detach succeeded for volume "pv1" (UniqueName: "kubernetes.io/testPlugin/vol1") on node "mynode-1" 
I0716 19:24:39.385446  214571 shared_informer.go:240] Waiting for caches to sync for attach detach
I0716 19:24:39.485983  214571 shared_informer.go:247] Caches are synced for attach detach 
W0716 19:24:39.488619  214571 reflector.go:436] k8s.io/client-go/informers/factory.go:134: watch of *v1.PersistentVolume ended with: very short watch: k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
W0716 19:24:39.488799  214571 reflector.go:436] k8s.io/client-go/informers/factory.go:134: watch of *v1.CSINode ended with: very short watch: k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
I0716 19:24:39.492827  214571 shared_informer.go:240] Waiting for caches to sync for attach detach
E0716 19:24:39.503208  214571 reconciler.go:228] attacherDetacher.DetachVolume failed to start for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" : DetachVolume.FindAttachablePluginByName failed for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" : no volume plugin matched name: kubernetes.io/testPlugin
E0716 19:24:39.504595  214571 reconciler.go:228] attacherDetacher.DetachVolume failed to start for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" : DetachVolume.FindAttachablePluginByName failed for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" : no volume plugin matched name: kubernetes.io/testPlugin
E0716 19:24:39.505921  214571 reconciler.go:228] attacherDetacher.DetachVolume failed to start for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" : DetachVolume.FindAttachablePluginByName failed for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" : no volume plugin matched name: kubernetes.io/testPlugin
E0716 19:24:39.507265  214571 reconciler.go:228] attacherDetacher.DetachVolume failed to start for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" : DetachVolume.FindAttachablePluginByName failed for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" : no volume plugin matched name: kubernetes.io/testPlugin
I0716 19:24:39.508825  214571 operation_generator.go:1409] Verified volume is safe to detach for volume "nil" (UniqueName: "kubernetes.io/csi/pd.csi.storage.gke.io^projects/UNSPECIFIED/zones/UNSPECIFIED/disks/vol1") on node "mynode-1" 
E0716 19:24:39.509161  214571 csi_attacher.go:455] [attachment.ID=csi-77bd5b88d32d48e5a31f0073211a7dd54ed0066f008e12364f4b6ecb61dc7370] watch channel had been closed
E0716 19:24:39.509766  214571 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/csi/pd.csi.storage.gke.io^projects/UNSPECIFIED/zones/UNSPECIFIED/disks/vol1 podName: nodeName:mynode-1}" failed. No retries permitted until 2021-07-16 19:24:40.009512842 +0000 UTC m=+12.829464580 (durationBeforeRetry 500ms). Error: "DetachVolume.Detach failed for volume \"nil\" (UniqueName: \"kubernetes.io/csi/pd.csi.storage.gke.io^projects/UNSPECIFIED/zones/UNSPECIFIED/disks/vol1\") on node \"mynode-1\" : volume attachment watch channel had been closed"
I0716 19:24:39.509884  214571 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/csi/pd.csi.storage.gke.io^projects/UNSPECIFIED/zones/UNSPECIFIED/disks/vol1") on node "mynode-1" 
==================
WARNING: DATA RACE
Write at 0x00c00041e218 by goroutine 78:
  k8s.io/kubernetes/pkg/controller/volume/attachdetach/testing.CreateTestClient.func7()
      pkg/controller/volume/attachdetach/testing/testvolumespec.go:185 +0x207
  k8s.io/kubernetes/vendor/k8s.io/client-go/testing.(*SimpleReactor).React()
      staging/src/k8s.io/client-go/testing/fixture.go:521 +0x68
  k8s.io/kubernetes/vendor/k8s.io/client-go/testing.(*Fake).Invokes()
      staging/src/k8s.io/client-go/testing/fake.go:141 +0x327
  k8s.io/kubernetes/vendor/k8s.io/client-go/kubernetes/typed/storage/v1/fake.(*FakeVolumeAttachments).Create()
      staging/src/k8s.io/client-go/kubernetes/typed/storage/v1/fake/fake_volumeattachment.go:82 +0x246
  k8s.io/kubernetes/pkg/volume/csi.(*csiAttacher).Attach()
      pkg/volume/csi/csi_attacher.go:109 +0x5f3
  k8s.io/kubernetes/pkg/volume/util/operationexecutor.(*operationGenerator).GenerateAttachVolumeFunc.func1()
      pkg/volume/util/operationexecutor/operation_generator.go:335 +0x275
  k8s.io/kubernetes/pkg/volume/util/types.(*GeneratedOperations).Run()
      pkg/volume/util/types/types.go:54 +0x142
  k8s.io/kubernetes/pkg/volume/util/nestedpendingoperations.(*nestedPendingOperations).Run.func1()
      pkg/volume/util/nestedpendingoperations/nestedpendingoperations.go:183 +0x159

Previous read at 0x00c00041e218 by goroutine 98:
  k8s.io/kubernetes/pkg/controller/volume/attachdetach/testing.CreateTestClient.func6()
      pkg/controller/volume/attachdetach/testing/testvolumespec.go:179 +0x104
  k8s.io/kubernetes/vendor/k8s.io/client-go/testing.(*SimpleReactor).React()
      staging/src/k8s.io/client-go/testing/fixture.go:521 +0x68
  k8s.io/kubernetes/vendor/k8s.io/client-go/testing.(*Fake).Invokes()
      staging/src/k8s.io/client-go/testing/fake.go:141 +0x327
  k8s.io/kubernetes/vendor/k8s.io/client-go/kubernetes/typed/storage/v1/fake.(*FakeVolumeAttachments).List()
      staging/src/k8s.io/client-go/kubernetes/typed/storage/v1/fake/fake_volumeattachment.go:55 +0x273
  k8s.io/kubernetes/vendor/k8s.io/client-go/informers/storage/v1.NewFilteredVolumeAttachmentInformer.func1()
      staging/src/k8s.io/client-go/informers/storage/v1/volumeattachment.go:64 +0x23c
  k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.(*ListWatch).List()
      staging/src/k8s.io/client-go/tools/cache/listwatch.go:106 +0x97
  k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch.func1.1.2()
      staging/src/k8s.io/client-go/tools/cache/reflector.go:277 +0xf4
  k8s.io/kubernetes/vendor/k8s.io/client-go/tools/pager.SimplePageFunc.func1()
      staging/src/k8s.io/client-go/tools/pager/pager.go:40 +0x97
  k8s.io/kubernetes/vendor/k8s.io/client-go/tools/pager.(*ListPager).List()
      staging/src/k8s.io/client-go/tools/pager/pager.go:91 +0x1e2
  k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch.func1.1()
      staging/src/k8s.io/client-go/tools/cache/reflector.go:302 +0x2b7

Goroutine 78 (running) created at:
  k8s.io/kubernetes/pkg/volume/util/nestedpendingoperations.(*nestedPendingOperations).Run()
      pkg/volume/util/nestedpendingoperations/nestedpendingoperations.go:178 +0x5b2
  k8s.io/kubernetes/pkg/volume/util/operationexecutor.(*operationExecutor).AttachVolume()
      pkg/volume/util/operationexecutor/operation_executor.go:675 +0x348
  k8s.io/kubernetes/pkg/controller/volume/attachdetach/reconciler.(*reconciler).attachDesiredVolumes()
      pkg/controller/volume/attachdetach/reconciler/reconciler.go:293 +0x639
  k8s.io/kubernetes/pkg/controller/volume/attachdetach/reconciler.(*reconciler).reconcile()
      pkg/controller/volume/attachdetach/reconciler/reconciler.go:233 +0x1b9
  k8s.io/kubernetes/pkg/controller/volume/attachdetach/reconciler.(*reconciler).reconciliationLoopFunc.func1()
      pkg/controller/volume/attachdetach/reconciler/reconciler.go:110 +0x5e
  k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1()
      staging/src/k8s.io/apimachinery/pkg/util/wait/wait.go:155 +0x75
  k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.BackoffUntil()
      staging/src/k8s.io/apimachinery/pkg/util/wait/wait.go:156 +0xba
  k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil()
      staging/src/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x114
  k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.Until()
      staging/src/k8s.io/apimachinery/pkg/util/wait/wait.go:90 +0x89
  k8s.io/kubernetes/pkg/controller/volume/attachdetach/reconciler.(*reconciler).Run()
      pkg/controller/volume/attachdetach/reconciler/reconciler.go:101 +0x2f

Goroutine 98 (finished) created at:
  k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch.func1()
      staging/src/k8s.io/client-go/tools/cache/reflector.go:268 +0x36f
  k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch()
      staging/src/k8s.io/client-go/tools/cache/reflector.go:360 +0x2e4
  k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.(*Reflector).Run.func1()
      staging/src/k8s.io/client-go/tools/cache/reflector.go:221 +0x53
  k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1()
      staging/src/k8s.io/apimachinery/pkg/util/wait/wait.go:155 +0x75
  k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.BackoffUntil()
      staging/src/k8s.io/apimachinery/pkg/util/wait/wait.go:156 +0xba
  k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.(*Reflector).Run()
      staging/src/k8s.io/client-go/tools/cache/reflector.go:220 +0x2c4
  k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.(*Reflector).Run-fm()
      staging/src/k8s.io/client-go/tools/cache/reflector.go:218 +0x4b
  k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).StartWithChannel.func1()
      staging/src/k8s.io/apimachinery/pkg/util/wait/wait.go:56 +0x45
  k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
      staging/src/k8s.io/apimachinery/pkg/util/wait/wait.go:73 +0x6d
==================
E0716 19:24:39.510930  214571 csi_attacher.go:455] [attachment.ID=csi-4ff6fd4b10c02f804e7435dffeb62054338b590749cdaacf18e198ee0ab6dd66] watch channel had been closed
I0716 19:24:39.511405  214571 reconciler.go:295] attacherDetacher.AttachVolume started for volume "pd.csi.storage.gke.io-pdName" (UniqueName: "kubernetes.io/csi/pd.csi.storage.gke.io^projects/UNSPECIFIED/zones/UNSPECIFIED/disks/pdName") from node "mynode" 
E0716 19:24:39.512468  214571 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/csi/pd.csi.storage.gke.io^projects/UNSPECIFIED/zones/UNSPECIFIED/disks/pdName podName: nodeName:}" failed. No retries permitted until 2021-07-16 19:24:40.012212995 +0000 UTC m=+12.832164728 (durationBeforeRetry 500ms). Error: "AttachVolume.Attach failed for volume \"pd.csi.storage.gke.io-pdName\" (UniqueName: \"kubernetes.io/csi/pd.csi.storage.gke.io^projects/UNSPECIFIED/zones/UNSPECIFIED/disks/pdName\") from node \"mynode\" : volume attachment watch channel had been closed"
I0716 19:24:39.512798  214571 event.go:291] "Event occurred" object="mynamespace/mypod-1" kind="Pod" apiVersion="v1" type="Warning" reason="FailedAttachVolume" message="AttachVolume.Attach failed for volume \"pd.csi.storage.gke.io-pdName\" : volume attachment watch channel had been closed"
I0716 19:24:39.513025  214571 event.go:291] "Event occurred" object="mynamespace/mypod-0" kind="Pod" apiVersion="v1" type="Warning" reason="FailedAttachVolume" message="AttachVolume.Attach failed for volume \"pd.csi.storage.gke.io-pdName\" : volume attachment watch channel had been closed"
I0716 19:24:39.513300  214571 event.go:291] "Event occurred" object="mynamespace/mypod-2" kind="Pod" apiVersion="v1" type="Warning" reason="FailedAttachVolume" message="AttachVolume.Attach failed for volume \"pd.csi.storage.gke.io-pdName\" : volume attachment watch channel had been closed"
I0716 19:24:39.513509  214571 event.go:291] "Event occurred" object="mynamespace/mypod-3" kind="Pod" apiVersion="v1" type="Warning" reason="FailedAttachVolume" message="AttachVolume.Attach failed for volume \"pd.csi.storage.gke.io-pdName\" : volume attachment watch channel had been closed"
I0716 19:24:39.513697  214571 event.go:291] "Event occurred" object="mynamespace/mypod-4" kind="Pod" apiVersion="v1" type="Warning" reason="FailedAttachVolume" message="AttachVolume.Attach failed for volume \"pd.csi.storage.gke.io-pdName\" : volume attachment watch channel had been closed"
I0716 19:24:39.603338  214571 shared_informer.go:247] Caches are synced for attach detach 
W0716 19:24:39.605887  214571 reflector.go:436] k8s.io/client-go/informers/factory.go:134: watch of *v1.CSIDriver ended with: very short watch: k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
W0716 19:24:39.606113  214571 reflector.go:436] k8s.io/client-go/informers/factory.go:134: watch of *v1.VolumeAttachment ended with: very short watch: k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
W0716 19:24:39.606517  214571 reflector.go:436] k8s.io/client-go/informers/factory.go:134: watch of *v1.PersistentVolume ended with: very short watch: k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
--- FAIL: Test_ADC_VolumeAttachmentRecovery (3.46s)
    --- FAIL: Test_ADC_VolumeAttachmentRecovery/Deleted_Pod_with_migrated_PV (0.12s)
        testing.go:1038: race detected during execution of test
    testing.go:1038: race detected during execution of test
FAIL

				from junit_bazel.xml

Filter through log files | View test history on testgrid


//pkg/kubelet/volumemanager/reconciler/go_default_test:run_2_of_2 0.00s

bazel test //pkg/kubelet/volumemanager/reconciler/go_default_test:run_2_of_2
exec ${PAGER:-/usr/bin/less} "$0" || exit 1
Executing tests from //pkg/kubelet/volumemanager/reconciler:go_default_test
-----------------------------------------------------------------------------
I0716 19:26:51.322163  231841 reconciler.go:157] Reconciler: start to sync state
E0716 19:26:51.322295  231841 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0716 19:26:51.339541  231841 reconciler.go:244] operationExecutor.AttachVolume started for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") 
I0716 19:26:51.339625  231841 reconciler.go:157] Reconciler: start to sync state
E0716 19:26:51.339678  231841 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0716 19:26:51.340052  231841 operation_generator.go:360] AttachVolume.Attach succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") from node "mynodename" 
I0716 19:26:51.340780  231841 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") DevicePath "/dev/vdb-test"
I0716 19:26:51.340891  231841 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:26:51.340987  231841 operation_generator.go:594] MountVolume.MountDevice succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") device mount path ""
I0716 19:26:51.440373  231841 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") 
I0716 19:26:51.440445  231841 reconciler.go:157] Reconciler: start to sync state
E0716 19:26:51.440496  231841 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0716 19:26:51.440917  231841 operation_generator.go:1357] Controller attach succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") device path: "/fake/path"
I0716 19:26:51.441435  231841 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") DevicePath "/fake/path"
I0716 19:26:51.441545  231841 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:26:51.441654  231841 operation_generator.go:594] MountVolume.MountDevice succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") device mount path ""
I0716 19:26:51.541177  231841 reconciler.go:244] operationExecutor.AttachVolume started for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") 
I0716 19:26:51.541284  231841 reconciler.go:157] Reconciler: start to sync state
E0716 19:26:51.541337  231841 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0716 19:26:51.541852  231841 operation_generator.go:360] AttachVolume.Attach succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") from node "mynodename" 
I0716 19:26:51.542700  231841 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") DevicePath "/dev/vdb-test"
I0716 19:26:51.542832  231841 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:26:51.542929  231841 operation_generator.go:594] MountVolume.MountDevice succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") device mount path ""
I0716 19:26:51.643677  231841 reconciler.go:196] operationExecutor.UnmountVolume started for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1uid" (UID: "pod1uid") 
I0716 19:26:51.643920  231841 operation_generator.go:797] UnmountVolume.TearDown succeeded for volume "fake-plugin/fake-device1" (OuterVolumeSpecName: "volume-name") pod "pod1uid" (UID: "pod1uid"). InnerVolumeSpecName "volume-name". PluginName "fake-plugin", VolumeGidValue ""
I0716 19:26:51.644400  231841 reconciler.go:312] operationExecutor.UnmountDevice started for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") on node "mynodename" 
I0716 19:26:51.644648  231841 operation_generator.go:891] UnmountDevice succeeded for volume "volume-name" %!(EXTRA string=UnmountDevice succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") on node "mynodename" )
I0716 19:26:51.645004  231841 reconciler.go:333] operationExecutor.DetachVolume started for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") on node "mynodename" 
I0716 19:26:51.645113  231841 operation_generator.go:470] DetachVolume.Detach succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") on node "mynodename" 
I0716 19:26:51.742320  231841 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") 
I0716 19:26:51.742400  231841 reconciler.go:157] Reconciler: start to sync state
E0716 19:26:51.742467  231841 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0716 19:26:51.742904  231841 operation_generator.go:1357] Controller attach succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") device path: "/fake/path"
I0716 19:26:51.746061  231841 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") DevicePath "/fake/path"
I0716 19:26:51.746442  231841 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:26:51.749193  231841 operation_generator.go:594] MountVolume.MountDevice succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") device mount path ""
I0716 19:26:51.848818  231841 reconciler.go:196] operationExecutor.UnmountVolume started for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1uid" (UID: "pod1uid") 
I0716 19:26:51.854538  231841 operation_generator.go:797] UnmountVolume.TearDown succeeded for volume "fake-plugin/fake-device1" (OuterVolumeSpecName: "volume-name") pod "pod1uid" (UID: "pod1uid"). InnerVolumeSpecName "volume-name". PluginName "fake-plugin", VolumeGidValue ""
I0716 19:26:51.869221  231841 operation_generator.go:797] UnmountVolume.TearDown succeeded for volume "fake-plugin/fake-device1" (OuterVolumeSpecName: "volume-name") pod "pod1uid" (UID: "pod1uid"). InnerVolumeSpecName "volume-name". PluginName "fake-plugin", VolumeGidValue ""
I0716 19:26:51.869409  231841 reconciler.go:196] operationExecutor.UnmountVolume started for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1uid" (UID: "pod1uid") 
I0716 19:26:51.870034  231841 operation_generator.go:891] UnmountDevice succeeded for volume "volume-name" %!(EXTRA string=UnmountDevice succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") on node "mynodename" )
I0716 19:26:51.870054  231841 reconciler.go:312] operationExecutor.UnmountDevice started for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") on node "mynodename" 
I0716 19:26:51.870665  231841 reconciler.go:319] Volume detached for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") on node "mynodename" DevicePath "/dev/sdb"
I0716 19:26:51.958877  231841 reconciler.go:244] operationExecutor.AttachVolume started for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") 
I0716 19:26:51.958948  231841 reconciler.go:157] Reconciler: start to sync state
E0716 19:26:51.959004  231841 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0716 19:26:51.959202  231841 operation_generator.go:360] AttachVolume.Attach succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") from node "mynodename" 
I0716 19:26:51.959854  231841 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") DevicePath "/dev/vdb-test"
I0716 19:26:51.959976  231841 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:26:52.059725  231841 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") 
I0716 19:26:52.059792  231841 reconciler.go:157] Reconciler: start to sync state
E0716 19:26:52.059840  231841 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0716 19:26:52.060275  231841 operation_generator.go:1357] Controller attach succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") device path: "/fake/path"
I0716 19:26:52.060887  231841 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") DevicePath "/fake/path"
I0716 19:26:52.061020  231841 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:26:52.065158  231841 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:26:52.065308  231841 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:26:52.163420  231841 reconciler.go:244] operationExecutor.AttachVolume started for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") 
I0716 19:26:52.163554  231841 reconciler.go:157] Reconciler: start to sync state
E0716 19:26:52.163612  231841 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0716 19:26:52.167956  231841 operation_generator.go:360] AttachVolume.Attach succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") from node "mynodename" 
I0716 19:26:52.168793  231841 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") DevicePath "/dev/vdb-test"
I0716 19:26:52.168922  231841 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:26:52.282488  231841 reconciler.go:196] operationExecutor.UnmountVolume started for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1uid" (UID: "pod1uid") 
I0716 19:26:52.311607  231841 operation_generator.go:1170] UnmapVolume succeeded for volume "fake-plugin/fake-device1" (OuterVolumeSpecName: "volume-name") pod "pod1uid" (UID: "pod1uid"). InnerVolumeSpecName "volume-name". PluginName "fake-plugin", VolumeGidValue ""
I0716 19:26:52.341852  231841 reconciler.go:312] operationExecutor.UnmountDevice started for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") on node "mynodename" 
I0716 19:26:52.350252  231841 operation_generator.go:1282] UnmapDevice succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") on node "mynodename" 
I0716 19:26:52.350807  231841 reconciler.go:333] operationExecutor.DetachVolume started for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") on node "mynodename" 
I0716 19:26:52.351124  231841 operation_generator.go:470] DetachVolume.Detach succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") on node "mynodename" 
I0716 19:26:52.368016  231841 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") 
I0716 19:26:52.368232  231841 operation_generator.go:1357] Controller attach succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") device path: "/fake/path"
I0716 19:26:52.368337  231841 reconciler.go:157] Reconciler: start to sync state
E0716 19:26:52.368401  231841 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0716 19:26:52.369435  231841 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") DevicePath "/fake/path"
I0716 19:26:52.369926  231841 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:26:52.518578  231841 reconciler.go:196] operationExecutor.UnmountVolume started for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1uid" (UID: "pod1uid") 
I0716 19:26:52.518887  231841 operation_generator.go:1170] UnmapVolume succeeded for volume "fake-plugin/fake-device1" (OuterVolumeSpecName: "volume-name") pod "pod1uid" (UID: "pod1uid"). InnerVolumeSpecName "volume-name". PluginName "fake-plugin", VolumeGidValue ""
I0716 19:26:52.519276  231841 reconciler.go:312] operationExecutor.UnmountDevice started for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") on node "mynodename" 
I0716 19:26:52.519468  231841 operation_generator.go:1282] UnmapDevice succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") on node "mynodename" 
I0716 19:26:52.519615  231841 reconciler.go:319] Volume detached for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") on node "mynodename" DevicePath "/dev/sdb"
I0716 19:26:52.634520  231841 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "pv" (UniqueName: "fake-plugin/pv") pod "pod1" (UID: "pod1uid") 
I0716 19:26:52.634692  231841 reconciler.go:157] Reconciler: start to sync state
E0716 19:26:52.634781  231841 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0716 19:26:52.634894  231841 operation_generator.go:1357] Controller attach succeeded for volume "pv" (UniqueName: "fake-plugin/pv") pod "pod1" (UID: "pod1uid") device path: "fake/path"
I0716 19:26:52.635579  231841 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "pv" (UniqueName: "fake-plugin/pv") pod "pod1" (UID: "pod1uid") DevicePath "fake/path"
I0716 19:26:52.635964  231841 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "pv" (UniqueName: "fake-plugin/pv") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:26:52.636082  231841 operation_generator.go:594] MountVolume.MountDevice succeeded for volume "pv" (UniqueName: "fake-plugin/pv") pod "pod1" (UID: "pod1uid") device mount path ""
I0716 19:26:52.741177  231841 operation_generator.go:1683] MountVolume.NodeExpandVolume succeeded for volume "pv" (UniqueName: "fake-plugin/pv") pod "pod1" (UID: "pod1uid") 
I0716 19:26:52.846913  231841 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "pv" (UniqueName: "fake-plugin/pv") pod "pod1" (UID: "pod1uid") 
I0716 19:26:52.846989  231841 reconciler.go:157] Reconciler: start to sync state
E0716 19:26:52.847042  231841 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0716 19:26:52.847418  231841 operation_generator.go:1357] Controller attach succeeded for volume "pv" (UniqueName: "fake-plugin/pv") pod "pod1" (UID: "pod1uid") device path: "fake/path"
I0716 19:26:52.847941  231841 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "pv" (UniqueName: "fake-plugin/pv") pod "pod1" (UID: "pod1uid") DevicePath "fake/path"
I0716 19:26:52.848062  231841 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "pv" (UniqueName: "fake-plugin/pv") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:26:52.981943  231841 operation_generator.go:1683] MountVolume.NodeExpandVolume succeeded for volume "pv" (UniqueName: "fake-plugin/pv") pod "pod1" (UID: "pod1uid") 
I0716 19:26:53.084228  231841 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "fail-expansion-in-use" (UniqueName: "fake-plugin/fail-expansion-in-use") pod "pod1" (UID: "pod1uid") 
I0716 19:26:53.084347  231841 reconciler.go:157] Reconciler: start to sync state
E0716 19:26:53.084394  231841 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0716 19:26:53.084938  231841 operation_generator.go:1357] Controller attach succeeded for volume "fail-expansion-in-use" (UniqueName: "fake-plugin/fail-expansion-in-use") pod "pod1" (UID: "pod1uid") device path: "fake/path"
I0716 19:26:53.085821  231841 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "fail-expansion-in-use" (UniqueName: "fake-plugin/fail-expansion-in-use") pod "pod1" (UID: "pod1uid") DevicePath "fake/path"
I0716 19:26:53.086014  231841 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "fail-expansion-in-use" (UniqueName: "fake-plugin/fail-expansion-in-use") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:26:53.086328  231841 operation_generator.go:594] MountVolume.MountDevice succeeded for volume "fail-expansion-in-use" (UniqueName: "fake-plugin/fail-expansion-in-use") pod "pod1" (UID: "pod1uid") device mount path ""
E0716 19:26:53.086787  231841 operation_generator.go:1669] MountVolume.NodeExapndVolume failed with %!!(MISSING)v(MISSING) for volume "fail-expansion-in-use" (UniqueName: "fake-plugin/fail-expansion-in-use") pod "pod1" (UID: "pod1uid") : volume-in-use
I0716 19:26:53.200864  231841 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "timeout-mount-device-volume" (UniqueName: "fake-plugin/timeout-mount-device-volume") pod "pod1" (UID: "pod1uid") 
I0716 19:26:53.200967  231841 reconciler.go:157] Reconciler: start to sync state
E0716 19:26:53.201027  231841 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0716 19:26:53.201350  231841 operation_generator.go:1357] Controller attach succeeded for volume "timeout-mount-device-volume" (UniqueName: "fake-plugin/timeout-mount-device-volume") pod "pod1" (UID: "pod1uid") device path: "fake/path"
I0716 19:26:53.202015  231841 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "timeout-mount-device-volume" (UniqueName: "fake-plugin/timeout-mount-device-volume") pod "pod1" (UID: "pod1uid") DevicePath "fake/path"
I0716 19:26:53.219518  231841 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "timeout-mount-device-volume" (UniqueName: "fake-plugin/timeout-mount-device-volume") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0716 19:26:53.220315  231841 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/timeout-mount-device-volume podName: nodeName:}" failed. No retries permitted until 2021-07-16 19:26:53.720044148 +0000 UTC m=+2.661488687 (durationBeforeRetry 500ms). Error: "MapVolume.SetUpDevice failed for volume \"timeout-mount-device-volume\" (UniqueName: \"fake-plugin/timeout-mount-device-volume\") pod \"pod1\" (UID: \"pod1uid\") : mount failed"
I0716 19:26:53.286533  231841 reconciler.go:312] operationExecutor.UnmountDevice started for volume "timeout-mount-device-volume" (UniqueName: "fake-plugin/timeout-mount-device-volume") on node "mynodename" 
I0716 19:26:53.286932  231841 operation_generator.go:1282] UnmapDevice succeeded for volume "timeout-mount-device-volume" (UniqueName: "fake-plugin/timeout-mount-device-volume") on node "mynodename" 
I0716 19:26:53.287179  231841 reconciler.go:319] Volume detached for volume "timeout-mount-device-volume" (UniqueName: "fake-plugin/timeout-mount-device-volume") on node "mynodename" DevicePath "/dev/sdb"
I0716 19:26:53.389464  231841 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "fail-mount-device-volume-name" (UniqueName: "fake-plugin/fail-mount-device-volume-name") pod "pod1" (UID: "pod1uid") 
I0716 19:26:53.389567  231841 reconciler.go:157] Reconciler: start to sync state
E0716 19:26:53.389635  231841 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0716 19:26:53.390082  231841 operation_generator.go:1357] Controller attach succeeded for volume "fail-mount-device-volume-name" (UniqueName: "fake-plugin/fail-mount-device-volume-name") pod "pod1" (UID: "pod1uid") device path: "fake/path"
I0716 19:26:53.459045  231841 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "fail-mount-device-volume-name" (UniqueName: "fake-plugin/fail-mount-device-volume-name") pod "pod1" (UID: "pod1uid") DevicePath "fake/path"
I0716 19:26:53.459201  231841 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "fail-mount-device-volume-name" (UniqueName: "fake-plugin/fail-mount-device-volume-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0716 19:26:53.459627  231841 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/fail-mount-device-volume-name podName: nodeName:}" failed. No retries permitted until 2021-07-16 19:26:53.959414933 +0000 UTC m=+2.900859468 (durationBeforeRetry 500ms). Error: "MapVolume.SetUpDevice failed for volume \"fail-mount-device-volume-name\" (UniqueName: \"fake-plugin/fail-mount-device-volume-name\") pod \"pod1\" (UID: \"pod1uid\") : error mapping disk: fail-mount-device-volume-name"
I0716 19:26:53.560999  231841 reconciler.go:319] Volume detached for volume "fail-mount-device-volume-name" (UniqueName: "fake-plugin/fail-mount-device-volume-name") on node "mynodename" DevicePath "fake/path"
I0716 19:26:53.654312  231841 operation_generator.go:1357] Controller attach succeeded for volume "timeout-and-fail-mount-device-name" (UniqueName: "fake-plugin/timeout-and-fail-mount-device-name") pod "pod1" (UID: "pod1uid") device path: "fake/path"
I0716 19:26:53.655045  231841 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "timeout-and-fail-mount-device-name" (UniqueName: "fake-plugin/timeout-and-fail-mount-device-name") pod "pod1" (UID: "pod1uid") 
I0716 19:26:53.655119  231841 reconciler.go:157] Reconciler: start to sync state
E0716 19:26:53.655177  231841 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0716 19:26:53.663007  231841 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "timeout-and-fail-mount-device-name" (UniqueName: "fake-plugin/timeout-and-fail-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "fake/path"
I0716 19:26:53.663236  231841 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "timeout-and-fail-mount-device-name" (UniqueName: "fake-plugin/timeout-and-fail-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0716 19:26:53.664519  231841 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/timeout-and-fail-mount-device-name podName: nodeName:}" failed. No retries permitted until 2021-07-16 19:26:54.164166721 +0000 UTC m=+3.105611258 (durationBeforeRetry 500ms). Error: "MapVolume.SetUpDevice failed for volume \"timeout-and-fail-mount-device-name\" (UniqueName: \"fake-plugin/timeout-and-fail-mount-device-name\") pod \"pod1\" (UID: \"pod1uid\") : timed out mounting error"
I0716 19:26:54.164605  231841 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "timeout-and-fail-mount-device-name" (UniqueName: "fake-plugin/timeout-and-fail-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:26:54.164783  231841 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "timeout-and-fail-mount-device-name" (UniqueName: "fake-plugin/timeout-and-fail-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0716 19:26:54.166952  231841 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/timeout-and-fail-mount-device-name podName: nodeName:}" failed. No retries permitted until 2021-07-16 19:26:55.166725341 +0000 UTC m=+4.108169933 (durationBeforeRetry 1s). Error: "MapVolume.SetUpDevice failed for volume \"timeout-and-fail-mount-device-name\" (UniqueName: \"fake-plugin/timeout-and-fail-mount-device-name\") pod \"pod1\" (UID: \"pod1uid\") : error mapping disk: timeout-and-fail-mount-device-name"
I0716 19:26:55.176372  231841 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "timeout-and-fail-mount-device-name" (UniqueName: "fake-plugin/timeout-and-fail-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:26:55.176565  231841 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "timeout-and-fail-mount-device-name" (UniqueName: "fake-plugin/timeout-and-fail-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0716 19:26:55.176988  231841 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/timeout-and-fail-mount-device-name podName: nodeName:}" failed. No retries permitted until 2021-07-16 19:26:57.176779712 +0000 UTC m=+6.118224248 (durationBeforeRetry 2s). Error: "MapVolume.SetUpDevice failed for volume \"timeout-and-fail-mount-device-name\" (UniqueName: \"fake-plugin/timeout-and-fail-mount-device-name\") pod \"pod1\" (UID: \"pod1uid\") : error mapping disk: timeout-and-fail-mount-device-name"
I0716 19:26:57.180235  231841 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "timeout-and-fail-mount-device-name" (UniqueName: "fake-plugin/timeout-and-fail-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:26:57.180514  231841 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "timeout-and-fail-mount-device-name" (UniqueName: "fake-plugin/timeout-and-fail-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0716 19:26:57.181382  231841 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/timeout-and-fail-mount-device-name podName: nodeName:}" failed. No retries permitted until 2021-07-16 19:27:01.180806338 +0000 UTC m=+10.122250888 (durationBeforeRetry 4s). Error: "MapVolume.SetUpDevice failed for volume \"timeout-and-fail-mount-device-name\" (UniqueName: \"fake-plugin/timeout-and-fail-mount-device-name\") pod \"pod1\" (UID: \"pod1uid\") : error mapping disk: timeout-and-fail-mount-device-name"
I0716 19:27:01.190870  231841 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "timeout-and-fail-mount-device-name" (UniqueName: "fake-plugin/timeout-and-fail-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:27:01.191049  231841 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "timeout-and-fail-mount-device-name" (UniqueName: "fake-plugin/timeout-and-fail-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0716 19:27:01.191496  231841 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/timeout-and-fail-mount-device-name podName: nodeName:}" failed. No retries permitted until 2021-07-16 19:27:09.19127461 +0000 UTC m=+18.132719170 (durationBeforeRetry 8s). Error: "MapVolume.SetUpDevice failed for volume \"timeout-and-fail-mount-device-name\" (UniqueName: \"fake-plugin/timeout-and-fail-mount-device-name\") pod \"pod1\" (UID: \"pod1uid\") : error mapping disk: timeout-and-fail-mount-device-name"
I0716 19:27:03.770421  231841 reconciler.go:319] Volume detached for volume "timeout-and-fail-mount-device-name" (UniqueName: "fake-plugin/timeout-and-fail-mount-device-name") on node "mynodename" DevicePath "/dev/sdb"
I0716 19:27:03.863409  231841 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "success-and-timeout-device-name" (UniqueName: "fake-plugin/success-and-timeout-device-name") pod "pod1" (UID: "pod1uid") 
I0716 19:27:03.863511  231841 reconciler.go:157] Reconciler: start to sync state
E0716 19:27:03.863566  231841 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0716 19:27:03.863870  231841 operation_generator.go:1357] Controller attach succeeded for volume "success-and-timeout-device-name" (UniqueName: "fake-plugin/success-and-timeout-device-name") pod "pod1" (UID: "pod1uid") device path: "fake/path"
I0716 19:27:03.864594  231841 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "success-and-timeout-device-name" (UniqueName: "fake-plugin/success-and-timeout-device-name") pod "pod1" (UID: "pod1uid") DevicePath "fake/path"
I0716 19:27:03.864738  231841 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "success-and-timeout-device-name" (UniqueName: "fake-plugin/success-and-timeout-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:27:03.984294  231841 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "success-and-timeout-device-name" (UniqueName: "fake-plugin/success-and-timeout-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:27:03.984470  231841 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "success-and-timeout-device-name" (UniqueName: "fake-plugin/success-and-timeout-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0716 19:27:03.984860  231841 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-timeout-device-name podName: nodeName:}" failed. No retries permitted until 2021-07-16 19:27:04.484666104 +0000 UTC m=+13.426110637 (durationBeforeRetry 500ms). Error: "MapVolume.SetUpDevice failed for volume \"success-and-timeout-device-name\" (UniqueName: \"fake-plugin/success-and-timeout-device-name\") pod \"pod1\" (UID: \"pod1uid\") : error mounting state"
I0716 19:27:04.485230  231841 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "success-and-timeout-device-name" (UniqueName: "fake-plugin/success-and-timeout-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:27:04.485436  231841 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "success-and-timeout-device-name" (UniqueName: "fake-plugin/success-and-timeout-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0716 19:27:04.485837  231841 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-timeout-device-name podName: nodeName:}" failed. No retries permitted until 2021-07-16 19:27:05.485651604 +0000 UTC m=+14.427096134 (durationBeforeRetry 1s). Error: "MapVolume.SetUpDevice failed for volume \"success-and-timeout-device-name\" (UniqueName: \"fake-plugin/success-and-timeout-device-name\") pod \"pod1\" (UID: \"pod1uid\") : error mounting state"
I0716 19:27:05.522537  231841 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "success-and-timeout-device-name" (UniqueName: "fake-plugin/success-and-timeout-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:27:05.522725  231841 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "success-and-timeout-device-name" (UniqueName: "fake-plugin/success-and-timeout-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0716 19:27:05.523097  231841 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-timeout-device-name podName: nodeName:}" failed. No retries permitted until 2021-07-16 19:27:07.522928129 +0000 UTC m=+16.464372664 (durationBeforeRetry 2s). Error: "MapVolume.SetUpDevice failed for volume \"success-and-timeout-device-name\" (UniqueName: \"fake-plugin/success-and-timeout-device-name\") pod \"pod1\" (UID: \"pod1uid\") : error mounting state"
I0716 19:27:07.544819  231841 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "success-and-timeout-device-name" (UniqueName: "fake-plugin/success-and-timeout-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:27:07.545001  231841 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "success-and-timeout-device-name" (UniqueName: "fake-plugin/success-and-timeout-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0716 19:27:07.545450  231841 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-timeout-device-name podName: nodeName:}" failed. No retries permitted until 2021-07-16 19:27:11.545192856 +0000 UTC m=+20.486637392 (durationBeforeRetry 4s). Error: "MapVolume.SetUpDevice failed for volume \"success-and-timeout-device-name\" (UniqueName: \"fake-plugin/success-and-timeout-device-name\") pod \"pod1\" (UID: \"pod1uid\") : error mounting state"
I0716 19:27:11.555057  231841 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "success-and-timeout-device-name" (UniqueName: "fake-plugin/success-and-timeout-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:27:11.556175  231841 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "success-and-timeout-device-name" (UniqueName: "fake-plugin/success-and-timeout-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0716 19:27:11.558334  231841 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-timeout-device-name podName: nodeName:}" failed. No retries permitted until 2021-07-16 19:27:19.557210768 +0000 UTC m=+28.498655302 (durationBeforeRetry 8s). Error: "MapVolume.SetUpDevice failed for volume \"success-and-timeout-device-name\" (UniqueName: \"fake-plugin/success-and-timeout-device-name\") pod \"pod1\" (UID: \"pod1uid\") : error mounting state"
I0716 19:27:13.996997  231841 reconciler.go:196] operationExecutor.UnmountVolume started for volume "success-and-timeout-device-name" (UniqueName: "fake-plugin/success-and-timeout-device-name") pod "pod1uid" (UID: "pod1uid") 
I0716 19:27:13.998125  231841 operation_generator.go:1170] UnmapVolume succeeded for volume "fake-plugin/success-and-timeout-device-name" (OuterVolumeSpecName: "success-and-timeout-device-name") pod "pod1uid" (UID: "pod1uid"). InnerVolumeSpecName "success-and-timeout-device-name". PluginName "fake-plugin", VolumeGidValue ""
I0716 19:27:13.998587  231841 reconciler.go:312] operationExecutor.UnmountDevice started for volume "success-and-timeout-device-name" (UniqueName: "fake-plugin/success-and-timeout-device-name") on node "mynodename" 
I0716 19:27:13.998988  231841 operation_generator.go:1282] UnmapDevice succeeded for volume "success-and-timeout-device-name" (UniqueName: "fake-plugin/success-and-timeout-device-name") on node "mynodename" 
I0716 19:27:13.999160  231841 reconciler.go:319] Volume detached for volume "success-and-timeout-device-name" (UniqueName: "fake-plugin/success-and-timeout-device-name") on node "mynodename" DevicePath "/dev/sdb"
I0716 19:27:14.114458  231841 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") pod "pod1" (UID: "pod1uid") 
I0716 19:27:14.114566  231841 reconciler.go:157] Reconciler: start to sync state
E0716 19:27:14.114628  231841 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0716 19:27:14.115173  231841 operation_generator.go:1357] Controller attach succeeded for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") pod "pod1" (UID: "pod1uid") device path: "fake/path"
I0716 19:27:14.115754  231841 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "fake/path"
I0716 19:27:14.115885  231841 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:27:14.252891  231841 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:27:14.253065  231841 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0716 19:27:14.253632  231841 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-failed-mount-device-name podName: nodeName:}" failed. No retries permitted until 2021-07-16 19:27:14.753348121 +0000 UTC m=+23.694792655 (durationBeforeRetry 500ms). Error: "MapVolume.SetUpDevice failed for volume \"success-and-failed-mount-device-name\" (UniqueName: \"fake-plugin/success-and-failed-mount-device-name\") pod \"pod1\" (UID: \"pod1uid\") : error mapping disk: success-and-failed-mount-device-name"
I0716 19:27:14.758789  231841 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:27:14.822487  231841 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0716 19:27:14.823022  231841 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-failed-mount-device-name podName: nodeName:}" failed. No retries permitted until 2021-07-16 19:27:15.822752309 +0000 UTC m=+24.764196855 (durationBeforeRetry 1s). Error: "MapVolume.SetUpDevice failed for volume \"success-and-failed-mount-device-name\" (UniqueName: \"fake-plugin/success-and-failed-mount-device-name\") pod \"pod1\" (UID: \"pod1uid\") : error mapping disk: success-and-failed-mount-device-name"
I0716 19:27:15.824769  231841 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:27:15.827004  231841 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0716 19:27:15.830968  231841 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-failed-mount-device-name podName: nodeName:}" failed. No retries permitted until 2021-07-16 19:27:17.82852714 +0000 UTC m=+26.769971674 (durationBeforeRetry 2s). Error: "MapVolume.SetUpDevice failed for volume \"success-and-failed-mount-device-name\" (UniqueName: \"fake-plugin/success-and-failed-mount-device-name\") pod \"pod1\" (UID: \"pod1uid\") : error mapping disk: success-and-failed-mount-device-name"
I0716 19:27:17.828867  231841 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:27:17.829127  231841 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0716 19:27:17.829554  231841 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-failed-mount-device-name podName: nodeName:}" failed. No retries permitted until 2021-07-16 19:27:21.829350064 +0000 UTC m=+30.770794597 (durationBeforeRetry 4s). Error: "MapVolume.SetUpDevice failed for volume \"success-and-failed-mount-device-name\" (UniqueName: \"fake-plugin/success-and-failed-mount-device-name\") pod \"pod1\" (UID: \"pod1uid\") : error mapping disk: success-and-failed-mount-device-name"
I0716 19:27:21.830124  231841 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:27:21.830315  231841 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0716 19:27:21.830748  231841 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-failed-mount-device-name podName: nodeName:}" failed. No retries permitted until 2021-07-16 19:27:29.830524526 +0000 UTC m=+38.771969073 (durationBeforeRetry 8s). Error: "MapVolume.SetUpDevice failed for volume \"success-and-failed-mount-device-name\" (UniqueName: \"fake-plugin/success-and-failed-mount-device-name\") pod \"pod1\" (UID: \"pod1uid\") : error mapping disk: success-and-failed-mount-device-name"
I0716 19:27:24.279977  231841 reconciler.go:196] operationExecutor.UnmountVolume started for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") pod "pod1uid" (UID: "pod1uid") 
I0716 19:27:24.279495  231841 operation_generator.go:1170] UnmapVolume succeeded for volume "fake-plugin/success-and-failed-mount-device-name" (OuterVolumeSpecName: "success-and-failed-mount-device-name") pod "pod1uid" (UID: "pod1uid"). InnerVolumeSpecName "success-and-failed-mount-device-name". PluginName "fake-plugin", VolumeGidValue ""
I0716 19:27:24.281395  231841 reconciler.go:312] operationExecutor.UnmountDevice started for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") on node "mynodename" 
I0716 19:27:24.281727  231841 operation_generator.go:1282] UnmapDevice succeeded for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") on node "mynodename" 
I0716 19:27:24.282920  231841 reconciler.go:319] Volume detached for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") on node "mynodename" DevicePath "/dev/sdb"
I0716 19:27:24.379451  231841 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "timeout-mount-device-volume" (UniqueName: "fake-plugin/timeout-mount-device-volume") pod "pod1" (UID: "pod1uid") 
I0716 19:27:24.379537  231841 reconciler.go:157] Reconciler: start to sync state
E0716 19:27:24.379598  231841 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0716 19:27:24.379832  231841 operation_generator.go:1357] Controller attach succeeded for volume "timeout-mount-device-volume" (UniqueName: "fake-plugin/timeout-mount-device-volume") pod "pod1" (UID: "pod1uid") device path: "fake/path"
I0716 19:27:24.380553  231841 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "timeout-mount-device-volume" (UniqueName: "fake-plugin/timeout-mount-device-volume") pod "pod1" (UID: "pod1uid") DevicePath "fake/path"
I0716 19:27:24.380680  231841 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "timeout-mount-device-volume" (UniqueName: "fake-plugin/timeout-mount-device-volume") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0716 19:27:24.381111  231841 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/timeout-mount-device-volume podName: nodeName:}" failed. No retries permitted until 2021-07-16 19:27:24.88092774 +0000 UTC m=+33.822372272 (durationBeforeRetry 500ms). Error: "MountVolume.MountDevice failed for volume \"timeout-mount-device-volume\" (UniqueName: \"fake-plugin/timeout-mount-device-volume\") pod \"pod1\" (UID: \"pod1uid\") : mount failed"
I0716 19:27:24.485465  231841 reconciler.go:312] operationExecutor.UnmountDevice started for volume "timeout-mount-device-volume" (UniqueName: "fake-plugin/timeout-mount-device-volume") on node "mynodename" 
I0716 19:27:24.485730  231841 operation_generator.go:891] UnmountDevice succeeded for volume "timeout-mount-device-volume" %!(EXTRA string=UnmountDevice succeeded for volume "timeout-mount-device-volume" (UniqueName: "fake-plugin/timeout-mount-device-volume") on node "mynodename" )
I0716 19:27:24.485969  231841 reconciler.go:319] Volume detached for volume "timeout-mount-device-volume" (UniqueName: "fake-plugin/timeout-mount-device-volume") on node "mynodename" DevicePath "/dev/sdb"
I0716 19:27:24.623374  231841 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "fail-mount-device-volume-name" (UniqueName: "fake-plugin/fail-mount-device-volume-name") pod "pod1" (UID: "pod1uid") 
I0716 19:27:24.623448  231841 reconciler.go:157] Reconciler: start to sync state
E0716 19:27:24.623507  231841 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0716 19:27:24.623875  231841 operation_generator.go:1357] Controller attach succeeded for volume "fail-mount-device-volume-name" (UniqueName: "fake-plugin/fail-mount-device-volume-name") pod "pod1" (UID: "pod1uid") device path: "fake/path"
I0716 19:27:24.624463  231841 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "fail-mount-device-volume-name" (UniqueName: "fake-plugin/fail-mount-device-volume-name") pod "pod1" (UID: "pod1uid") DevicePath "fake/path"
I0716 19:27:24.624588  231841 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "fail-mount-device-volume-name" (UniqueName: "fake-plugin/fail-mount-device-volume-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0716 19:27:24.625033  231841 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/fail-mount-device-volume-name podName: nodeName:}" failed. No retries permitted until 2021-07-16 19:27:25.124836214 +0000 UTC m=+34.066280756 (durationBeforeRetry 500ms). Error: "MountVolume.MountDevice failed for volume \"fail-mount-device-volume-name\" (UniqueName: \"fake-plugin/fail-mount-device-volume-name\") pod \"pod1\" (UID: \"pod1uid\") : error mounting disk: /dev/sdb"
I0716 19:27:24.726490  231841 reconciler.go:319] Volume detached for volume "fail-mount-device-volume-name" (UniqueName: "fake-plugin/fail-mount-device-volume-name") on node "mynodename" DevicePath "fake/path"
I0716 19:27:24.827556  231841 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "timeout-and-fail-mount-device-name" (UniqueName: "fake-plugin/timeout-and-fail-mount-device-name") pod "pod1" (UID: "pod1uid") 
I0716 19:27:24.827636  231841 reconciler.go:157] Reconciler: start to sync state
E0716 19:27:24.827694  231841 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0716 19:27:24.828504  231841 operation_generator.go:1357] Controller attach succeeded for volume "timeout-and-fail-mount-device-name" (UniqueName: "fake-plugin/timeout-and-fail-mount-device-name") pod "pod1" (UID: "pod1uid") device path: "fake/path"
I0716 19:27:24.829437  231841 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "timeout-and-fail-mount-device-name" (UniqueName: "fake-plugin/timeout-and-fail-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "fake/path"
I0716 19:27:24.829586  231841 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "timeout-and-fail-mount-device-name" (UniqueName: "fake-plugin/timeout-and-fail-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0716 19:27:24.829989  231841 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/timeout-and-fail-mount-device-name podName: nodeName:}" failed. No retries permitted until 2021-07-16 19:27:25.329791345 +0000 UTC m=+34.271235889 (durationBeforeRetry 500ms). Error: "MountVolume.MountDevice failed for volume \"timeout-and-fail-mount-device-name\" (UniqueName: \"fake-plugin/timeout-and-fail-mount-device-name\") pod \"pod1\" (UID: \"pod1uid\") : timed out mounting error"
I0716 19:27:25.330382  231841 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "timeout-and-fail-mount-device-name" (UniqueName: "fake-plugin/timeout-and-fail-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:27:25.330561  231841 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "timeout-and-fail-mount-device-name" (UniqueName: "fake-plugin/timeout-and-fail-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0716 19:27:25.330982  231841 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/timeout-and-fail-mount-device-name podName: nodeName:}" failed. No retries permitted until 2021-07-16 19:27:26.33078591 +0000 UTC m=+35.272230478 (durationBeforeRetry 1s). Error: "MountVolume.MountDevice failed for volume \"timeout-and-fail-mount-device-name\" (UniqueName: \"fake-plugin/timeout-and-fail-mount-device-name\") pod \"pod1\" (UID: \"pod1uid\") : error mounting disk: /dev/sdb"
I0716 19:27:26.337516  231841 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "timeout-and-fail-mount-device-name" (UniqueName: "fake-plugin/timeout-and-fail-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:27:26.337683  231841 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "timeout-and-fail-mount-device-name" (UniqueName: "fake-plugin/timeout-and-fail-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0716 19:27:26.338088  231841 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/timeout-and-fail-mount-device-name podName: nodeName:}" failed. No retries permitted until 2021-07-16 19:27:28.337897869 +0000 UTC m=+37.279342444 (durationBeforeRetry 2s). Error: "MountVolume.MountDevice failed for volume \"timeout-and-fail-mount-device-name\" (UniqueName: \"fake-plugin/timeout-and-fail-mount-device-name\") pod \"pod1\" (UID: \"pod1uid\") : error mounting disk: /dev/sdb"
I0716 19:27:28.387535  231841 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "timeout-and-fail-mount-device-name" (UniqueName: "fake-plugin/timeout-and-fail-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:27:28.387711  231841 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "timeout-and-fail-mount-device-name" (UniqueName: "fake-plugin/timeout-and-fail-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0716 19:27:28.388435  231841 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/timeout-and-fail-mount-device-name podName: nodeName:}" failed. No retries permitted until 2021-07-16 19:27:32.387935071 +0000 UTC m=+41.329379608 (durationBeforeRetry 4s). Error: "MountVolume.MountDevice failed for volume \"timeout-and-fail-mount-device-name\" (UniqueName: \"fake-plugin/timeout-and-fail-mount-device-name\") pod \"pod1\" (UID: \"pod1uid\") : error mounting disk: /dev/sdb"
I0716 19:27:32.391019  231841 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "timeout-and-fail-mount-device-name" (UniqueName: "fake-plugin/timeout-and-fail-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:27:32.391210  231841 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "timeout-and-fail-mount-device-name" (UniqueName: "fake-plugin/timeout-and-fail-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0716 19:27:32.391650  231841 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/timeout-and-fail-mount-device-name podName: nodeName:}" failed. No retries permitted until 2021-07-16 19:27:40.391439586 +0000 UTC m=+49.332884126 (durationBeforeRetry 8s). Error: "MountVolume.MountDevice failed for volume \"timeout-and-fail-mount-device-name\" (UniqueName: \"fake-plugin/timeout-and-fail-mount-device-name\") pod \"pod1\" (UID: \"pod1uid\") : error mounting disk: /dev/sdb"
I0716 19:27:34.935098  231841 reconciler.go:319] Volume detached for volume "timeout-and-fail-mount-device-name" (UniqueName: "fake-plugin/timeout-and-fail-mount-device-name") on node "mynodename" DevicePath "/dev/sdb"
I0716 19:27:35.079392  231841 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "success-and-timeout-device-name" (UniqueName: "fake-plugin/success-and-timeout-device-name") pod "pod1" (UID: "pod1uid") 
I0716 19:27:35.079478  231841 reconciler.go:157] Reconciler: start to sync state
E0716 19:27:35.079537  231841 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0716 19:27:35.079920  231841 operation_generator.go:1357] Controller attach succeeded for volume "success-and-timeout-device-name" (UniqueName: "fake-plugin/success-and-timeout-device-name") pod "pod1" (UID: "pod1uid") device path: "fake/path"
I0716 19:27:35.113494  231841 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "success-and-timeout-device-name" (UniqueName: "fake-plugin/success-and-timeout-device-name") pod "pod1" (UID: "pod1uid") DevicePath "fake/path"
I0716 19:27:35.113701  231841 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "success-and-timeout-device-name" (UniqueName: "fake-plugin/success-and-timeout-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:27:35.113824  231841 operation_generator.go:594] MountVolume.MountDevice succeeded for volume "success-and-timeout-device-name" (UniqueName: "fake-plugin/success-and-timeout-device-name") pod "pod1" (UID: "pod1uid") device mount path ""
I0716 19:27:35.194835  231841 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "success-and-timeout-device-name" (UniqueName: "fake-plugin/success-and-timeout-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:27:35.194991  231841 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "success-and-timeout-device-name" (UniqueName: "fake-plugin/success-and-timeout-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:27:45.179619  231841 reconciler.go:196] operationExecutor.UnmountVolume started for volume "success-and-timeout-device-name" (UniqueName: "fake-plugin/success-and-timeout-device-name") pod "pod1uid" (UID: "pod1uid") 
I0716 19:27:45.179856  231841 operation_generator.go:797] UnmountVolume.TearDown succeeded for volume "fake-plugin/success-and-timeout-device-name" (OuterVolumeSpecName: "success-and-timeout-device-name") pod "pod1uid" (UID: "pod1uid"). InnerVolumeSpecName "success-and-timeout-device-name". PluginName "fake-plugin", VolumeGidValue ""
I0716 19:27:45.180427  231841 reconciler.go:312] operationExecutor.UnmountDevice started for volume "success-and-timeout-device-name" (UniqueName: "fake-plugin/success-and-timeout-device-name") on node "mynodename" 
I0716 19:27:45.180640  231841 operation_generator.go:891] UnmountDevice succeeded for volume "success-and-timeout-device-name" %!(EXTRA string=UnmountDevice succeeded for volume "success-and-timeout-device-name" (UniqueName: "fake-plugin/success-and-timeout-device-name") on node "mynodename" )
I0716 19:27:45.180896  231841 reconciler.go:319] Volume detached for volume "success-and-timeout-device-name" (UniqueName: "fake-plugin/success-and-timeout-device-name") on node "mynodename" DevicePath "/dev/sdb"
I0716 19:27:45.290139  231841 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") pod "pod1" (UID: "pod1uid") 
I0716 19:27:45.290254  231841 reconciler.go:157] Reconciler: start to sync state
E0716 19:27:45.290310  231841 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0716 19:27:45.290573  231841 operation_generator.go:1357] Controller attach succeeded for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") pod "pod1" (UID: "pod1uid") device path: "fake/path"
I0716 19:27:45.291193  231841 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "fake/path"
I0716 19:27:45.291333  231841 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:27:45.291453  231841 operation_generator.go:594] MountVolume.MountDevice succeeded for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") pod "pod1" (UID: "pod1uid") device mount path ""
I0716 19:27:45.393874  231841 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:27:45.394033  231841 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:27:55.435050  231841 reconciler.go:196] operationExecutor.UnmountVolume started for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") pod "pod1uid" (UID: "pod1uid") 
I0716 19:27:55.466446  231841 operation_generator.go:797] UnmountVolume.TearDown succeeded for volume "fake-plugin/success-and-failed-mount-device-name" (OuterVolumeSpecName: "success-and-failed-mount-device-name") pod "pod1uid" (UID: "pod1uid"). InnerVolumeSpecName "success-and-failed-mount-device-name". PluginName "fake-plugin", VolumeGidValue ""
I0716 19:27:55.467026  231841 reconciler.go:312] operationExecutor.UnmountDevice started for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") on node "mynodename" 
I0716 19:27:55.467237  231841 operation_generator.go:891] UnmountDevice succeeded for volume "success-and-failed-mount-device-name" %!(EXTRA string=UnmountDevice succeeded for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") on node "mynodename" )
I0716 19:27:55.467682  231841 reconciler.go:319] Volume detached for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") on node "mynodename" DevicePath "/dev/sdb"
I0716 19:27:55.536289  231841 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "timeout-setup-volume" (UniqueName: "fake-plugin/timeout-setup-volume") pod "pod1" (UID: "pod1uid") 
I0716 19:27:55.536373  231841 reconciler.go:157] Reconciler: start to sync state
E0716 19:27:55.536426  231841 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0716 19:27:55.536795  231841 operation_generator.go:1357] Controller attach succeeded for volume "timeout-setup-volume" (UniqueName: "fake-plugin/timeout-setup-volume") pod "pod1" (UID: "pod1uid") device path: "fake/path"
I0716 19:27:55.537426  231841 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "timeout-setup-volume" (UniqueName: "fake-plugin/timeout-setup-volume") pod "pod1" (UID: "pod1uid") DevicePath "fake/path"
I0716 19:27:55.537554  231841 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "timeout-setup-volume" (UniqueName: "fake-plugin/timeout-setup-volume") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0716 19:27:55.537974  231841 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/timeout-setup-volume podName: nodeName:}" failed. No retries permitted until 2021-07-16 19:27:56.037764556 +0000 UTC m=+64.979209096 (durationBeforeRetry 500ms). Error: "MapVolume.MapPodDevice failed for volume \"timeout-setup-volume\" (UniqueName: \"fake-plugin/timeout-setup-volume\") pod \"pod1\" (UID: \"pod1uid\") : time out on setup"
I0716 19:27:55.656541  231841 reconciler.go:196] operationExecutor.UnmountVolume started for volume "timeout-setup-volume" (UniqueName: "fake-plugin/timeout-setup-volume") pod "pod1uid" (UID: "pod1uid") 
I0716 19:27:55.656022  231841 operation_generator.go:1170] UnmapVolume succeeded for volume "fake-plugin/timeout-setup-volume" (OuterVolumeSpecName: "timeout-setup-volume") pod "pod1uid" (UID: "pod1uid"). InnerVolumeSpecName "timeout-setup-volume". PluginName "fake-plugin", VolumeGidValue ""
I0716 19:27:55.660590  231841 reconciler.go:196] operationExecutor.UnmountVolume started for volume "timeout-setup-volume" (UniqueName: "fake-plugin/timeout-setup-volume") pod "pod1uid" (UID: "pod1uid") 
I0716 19:27:55.659299  231841 operation_generator.go:1170] UnmapVolume succeeded for volume "fake-plugin/timeout-setup-volume" (OuterVolumeSpecName: "timeout-setup-volume") pod "pod1uid" (UID: "pod1uid"). InnerVolumeSpecName "timeout-setup-volume". PluginName "fake-plugin", VolumeGidValue ""
I0716 19:27:55.673750  231841 reconciler.go:312] operationExecutor.UnmountDevice started for volume "timeout-setup-volume" (UniqueName: "fake-plugin/timeout-setup-volume") on node "mynodename" 
I0716 19:27:55.677336  231841 operation_generator.go:1282] UnmapDevice succeeded for volume "timeout-setup-volume" (UniqueName: "fake-plugin/timeout-setup-volume") on node "mynodename" 
I0716 19:27:55.677707  231841 reconciler.go:319] Volume detached for volume "timeout-setup-volume" (UniqueName: "fake-plugin/timeout-setup-volume") on node "mynodename" DevicePath "/dev/sdb"
I0716 19:27:55.747808  231841 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "fail-setup-volume" (UniqueName: "fake-plugin/fail-setup-volume") pod "pod1" (UID: "pod1uid") 
I0716 19:27:55.747893  231841 reconciler.go:157] Reconciler: start to sync state
E0716 19:27:55.747950  231841 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0716 19:27:55.748231  231841 operation_generator.go:1357] Controller attach succeeded for volume "fail-setup-volume" (UniqueName: "fake-plugin/fail-setup-volume") pod "pod1" (UID: "pod1uid") device path: "fake/path"
I0716 19:27:55.748875  231841 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "fail-setup-volume" (UniqueName: "fake-plugin/fail-setup-volume") pod "pod1" (UID: "pod1uid") DevicePath "fake/path"
I0716 19:27:55.749009  231841 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "fail-setup-volume" (UniqueName: "fake-plugin/fail-setup-volume") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0716 19:27:55.749385  231841 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/fail-setup-volume podName: nodeName:}" failed. No retries permitted until 2021-07-16 19:27:56.24919172 +0000 UTC m=+65.190636264 (durationBeforeRetry 500ms). Error: "MapVolume.MapPodDevice failed for volume \"fail-setup-volume\" (UniqueName: \"fake-plugin/fail-setup-volume\") pod \"pod1\" (UID: \"pod1uid\") : mounting volume failed"
I0716 19:27:55.851952  231841 reconciler.go:312] operationExecutor.UnmountDevice started for volume "fail-setup-volume" (UniqueName: "fake-plugin/fail-setup-volume") on node "mynodename" 
I0716 19:27:55.857628  231841 operation_generator.go:1282] UnmapDevice succeeded for volume "fail-setup-volume" (UniqueName: "fake-plugin/fail-setup-volume") on node "mynodename" 
I0716 19:27:55.857916  231841 reconciler.go:319] Volume detached for volume "fail-setup-volume" (UniqueName: "fake-plugin/fail-setup-volume") on node "mynodename" DevicePath "/dev/sdb"
I0716 19:27:55.972704  231841 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") pod "pod1" (UID: "pod1uid") 
I0716 19:27:55.972784  231841 reconciler.go:157] Reconciler: start to sync state
E0716 19:27:55.972853  231841 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0716 19:27:55.973358  231841 operation_generator.go:1357] Controller attach succeeded for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") pod "pod1" (UID: "pod1uid") device path: "fake/path"
I0716 19:27:55.974059  231841 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") pod "pod1" (UID: "pod1uid") DevicePath "fake/path"
I0716 19:27:55.974206  231841 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0716 19:27:55.974718  231841 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/timeout-and-fail-setup-volume podName: nodeName:}" failed. No retries permitted until 2021-07-16 19:27:56.474515875 +0000 UTC m=+65.415960441 (durationBeforeRetry 500ms). Error: "MapVolume.MapPodDevice failed for volume \"timeout-and-fail-setup-volume\" (UniqueName: \"fake-plugin/timeout-and-fail-setup-volume\") pod \"pod1\" (UID: \"pod1uid\") : time out on setup"
I0716 19:27:56.489983  231841 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:27:56.490185  231841 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0716 19:27:56.490607  231841 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/timeout-and-fail-setup-volume podName: nodeName:}" failed. No retries permitted until 2021-07-16 19:27:57.490411038 +0000 UTC m=+66.431855574 (durationBeforeRetry 1s). Error: "MapVolume.MapPodDevice failed for volume \"timeout-and-fail-setup-volume\" (UniqueName: \"fake-plugin/timeout-and-fail-setup-volume\") pod \"pod1\" (UID: \"pod1uid\") : mounting volume failed"
I0716 19:27:57.491075  231841 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:27:57.491250  231841 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0716 19:27:57.491629  231841 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/timeout-and-fail-setup-volume podName: nodeName:}" failed. No retries permitted until 2021-07-16 19:27:59.491450497 +0000 UTC m=+68.432895030 (durationBeforeRetry 2s). Error: "MapVolume.MapPodDevice failed for volume \"timeout-and-fail-setup-volume\" (UniqueName: \"fake-plugin/timeout-and-fail-setup-volume\") pod \"pod1\" (UID: \"pod1uid\") : mounting volume failed"
I0716 19:27:59.492975  231841 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:27:59.493147  231841 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0716 19:27:59.493560  231841 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/timeout-and-fail-setup-volume podName: nodeName:}" failed. No retries permitted until 2021-07-16 19:28:03.493366927 +0000 UTC m=+72.434811479 (durationBeforeRetry 4s). Error: "MapVolume.MapPodDevice failed for volume \"timeout-and-fail-setup-volume\" (UniqueName: \"fake-plugin/timeout-and-fail-setup-volume\") pod \"pod1\" (UID: \"pod1uid\") : mounting volume failed"
I0716 19:28:03.512459  231841 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:28:03.512652  231841 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0716 19:28:03.513053  231841 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/timeout-and-fail-setup-volume podName: nodeName:}" failed. No retries permitted until 2021-07-16 19:28:11.512867315 +0000 UTC m=+80.454311858 (durationBeforeRetry 8s). Error: "MapVolume.MapPodDevice failed for volume \"timeout-and-fail-setup-volume\" (UniqueName: \"fake-plugin/timeout-and-fail-setup-volume\") pod \"pod1\" (UID: \"pod1uid\") : mounting volume failed"
I0716 19:28:06.066671  231841 reconciler.go:312] operationExecutor.UnmountDevice started for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") on node "mynodename" 
I0716 19:28:06.066866  231841 operation_generator.go:1282] UnmapDevice succeeded for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") on node "mynodename" 
I0716 19:28:06.067090  231841 reconciler.go:319] Volume detached for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") on node "mynodename" DevicePath "/dev/sdb"
I0716 19:28:06.167599  231841 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1" (UID: "pod1uid") 
I0716 19:28:06.167682  231841 reconciler.go:157] Reconciler: start to sync state
E0716 19:28:06.167737  231841 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0716 19:28:06.168211  231841 operation_generator.go:1357] Controller attach succeeded for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1" (UID: "pod1uid") device path: "fake/path"
I0716 19:28:06.168810  231841 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1" (UID: "pod1uid") DevicePath "fake/path"
I0716 19:28:06.168960  231841 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:28:06.276884  231841 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:28:06.278393  231841 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0716 19:28:06.283517  231841 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-timeout-setup-volume-name podName: nodeName:}" failed. No retries permitted until 2021-07-16 19:28:06.78039674 +0000 UTC m=+75.721841286 (durationBeforeRetry 500ms). Error: "MapVolume.MapPodDevice failed for volume \"success-and-timeout-setup-volume-name\" (UniqueName: \"fake-plugin/success-and-timeout-setup-volume-name\") pod \"pod1\" (UID: \"pod1uid\") : time out on setup"
I0716 19:28:06.785437  231841 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:28:06.785728  231841 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0716 19:28:06.786590  231841 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-timeout-setup-volume-name podName: nodeName:}" failed. No retries permitted until 2021-07-16 19:28:07.786030119 +0000 UTC m=+76.727474655 (durationBeforeRetry 1s). Error: "MapVolume.MapPodDevice failed for volume \"success-and-timeout-setup-volume-name\" (UniqueName: \"fake-plugin/success-and-timeout-setup-volume-name\") pod \"pod1\" (UID: \"pod1uid\") : time out on setup"
I0716 19:28:07.792906  231841 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:28:07.793456  231841 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0716 19:28:07.794409  231841 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-timeout-setup-volume-name podName: nodeName:}" failed. No retries permitted until 2021-07-16 19:28:09.793885052 +0000 UTC m=+78.735329586 (durationBeforeRetry 2s). Error: "MapVolume.MapPodDevice failed for volume \"success-and-timeout-setup-volume-name\" (UniqueName: \"fake-plugin/success-and-timeout-setup-volume-name\") pod \"pod1\" (UID: \"pod1uid\") : time out on setup"
I0716 19:28:09.794793  231841 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:28:09.794959  231841 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0716 19:28:09.795373  231841 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-timeout-setup-volume-name podName: nodeName:}" failed. No retries permitted until 2021-07-16 19:28:13.795184329 +0000 UTC m=+82.736628866 (durationBeforeRetry 4s). Error: "MapVolume.MapPodDevice failed for volume \"success-and-timeout-setup-volume-name\" (UniqueName: \"fake-plugin/success-and-timeout-setup-volume-name\") pod \"pod1\" (UID: \"pod1uid\") : time out on setup"
I0716 19:28:13.796030  231841 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:28:13.796202  231841 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0716 19:28:13.796900  231841 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-timeout-setup-volume-name podName: nodeName:}" failed. No retries permitted until 2021-07-16 19:28:21.796442714 +0000 UTC m=+90.737887248 (durationBeforeRetry 8s). Error: "MapVolume.MapPodDevice failed for volume \"success-and-timeout-setup-volume-name\" (UniqueName: \"fake-plugin/success-and-timeout-setup-volume-name\") pod \"pod1\" (UID: \"pod1uid\") : time out on setup"
I0716 19:28:16.290676  231841 reconciler.go:196] operationExecutor.UnmountVolume started for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1uid" (UID: "pod1uid") 
I0716 19:28:16.291282  231841 operation_generator.go:1170] UnmapVolume succeeded for volume "fake-plugin/success-and-timeout-setup-volume-name" (OuterVolumeSpecName: "success-and-timeout-setup-volume-name") pod "pod1uid" (UID: "pod1uid"). InnerVolumeSpecName "success-and-timeout-setup-volume-name". PluginName "fake-plugin", VolumeGidValue ""
I0716 19:28:16.291773  231841 reconciler.go:312] operationExecutor.UnmountDevice started for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") on node "mynodename" 
I0716 19:28:16.292062  231841 operation_generator.go:1282] UnmapDevice succeeded for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") on node "mynodename" 
I0716 19:28:16.292277  231841 reconciler.go:319] Volume detached for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") on node "mynodename" DevicePath "/dev/sdb"
I0716 19:28:16.420754  231841 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1" (UID: "pod1uid") 
I0716 19:28:16.420858  231841 reconciler.go:157] Reconciler: start to sync state
E0716 19:28:16.420922  231841 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0716 19:28:16.421057  231841 operation_generator.go:1357] Controller attach succeeded for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1" (UID: "pod1uid") device path: "fake/path"
I0716 19:28:16.424638  231841 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1" (UID: "pod1uid") DevicePath "fake/path"
I0716 19:28:16.429574  231841 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:28:16.435187  231841 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:28:16.435371  231841 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0716 19:28:16.435781  231841 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-failed-setup-device-name podName: nodeName:}" failed. No retries permitted until 2021-07-16 19:28:16.935573339 +0000 UTC m=+85.877017872 (durationBeforeRetry 500ms). Error: "MapVolume.MapPodDevice failed for volume \"success-and-failed-setup-device-name\" (UniqueName: \"fake-plugin/success-and-failed-setup-device-name\") pod \"pod1\" (UID: \"pod1uid\") : mounting volume failed"
I0716 19:28:16.936088  231841 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:28:16.936262  231841 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0716 19:28:16.936701  231841 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-failed-setup-device-name podName: nodeName:}" failed. No retries permitted until 2021-07-16 19:28:17.936494874 +0000 UTC m=+86.877939408 (durationBeforeRetry 1s). Error: "MapVolume.MapPodDevice failed for volume \"success-and-failed-setup-device-name\" (UniqueName: \"fake-plugin/success-and-failed-setup-device-name\") pod \"pod1\" (UID: \"pod1uid\") : mounting volume failed"
I0716 19:28:17.937221  231841 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:28:17.937396  231841 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0716 19:28:17.937803  231841 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-failed-setup-device-name podName: nodeName:}" failed. No retries permitted until 2021-07-16 19:28:19.937610945 +0000 UTC m=+88.879055476 (durationBeforeRetry 2s). Error: "MapVolume.MapPodDevice failed for volume \"success-and-failed-setup-device-name\" (UniqueName: \"fake-plugin/success-and-failed-setup-device-name\") pod \"pod1\" (UID: \"pod1uid\") : mounting volume failed"
I0716 19:28:19.943120  231841 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:28:19.943329  231841 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0716 19:28:19.943737  231841 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-failed-setup-device-name podName: nodeName:}" failed. No retries permitted until 2021-07-16 19:28:23.943556594 +0000 UTC m=+92.885001148 (durationBeforeRetry 4s). Error: "MapVolume.MapPodDevice failed for volume \"success-and-failed-setup-device-name\" (UniqueName: \"fake-plugin/success-and-failed-setup-device-name\") pod \"pod1\" (UID: \"pod1uid\") : mounting volume failed"
I0716 19:28:23.944039  231841 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:28:23.944206  231841 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0716 19:28:23.944679  231841 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-failed-setup-device-name podName: nodeName:}" failed. No retries permitted until 2021-07-16 19:28:31.944443785 +0000 UTC m=+100.885888316 (durationBeforeRetry 8s). Error: "MapVolume.MapPodDevice failed for volume \"success-and-failed-setup-device-name\" (UniqueName: \"fake-plugin/success-and-failed-setup-device-name\") pod \"pod1\" (UID: \"pod1uid\") : mounting volume failed"
I0716 19:28:26.643270  231841 reconciler.go:196] operationExecutor.UnmountVolume started for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1uid" (UID: "pod1uid") 
I0716 19:28:26.643461  231841 operation_generator.go:1170] UnmapVolume succeeded for volume "fake-plugin/success-and-failed-setup-device-name" (OuterVolumeSpecName: "success-and-failed-setup-device-name") pod "pod1uid" (UID: "pod1uid"). InnerVolumeSpecName "success-and-failed-setup-device-name". PluginName "fake-plugin", VolumeGidValue ""
I0716 19:28:26.645314  231841 reconciler.go:312] operationExecutor.UnmountDevice started for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") on node "mynodename" 
I0716 19:28:26.646179  231841 operation_generator.go:1282] UnmapDevice succeeded for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") on node "mynodename" 
I0716 19:28:26.646508  231841 reconciler.go:319] Volume detached for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") on node "mynodename" DevicePath "/dev/sdb"
I0716 19:28:26.651487  231841 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "timeout-setup-volume" (UniqueName: "fake-plugin/timeout-setup-volume") pod "pod1" (UID: "pod1uid") 
I0716 19:28:26.651568  231841 reconciler.go:157] Reconciler: start to sync state
E0716 19:28:26.651625  231841 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0716 19:28:26.651891  231841 operation_generator.go:1357] Controller attach succeeded for volume "timeout-setup-volume" (UniqueName: "fake-plugin/timeout-setup-volume") pod "pod1" (UID: "pod1uid") device path: "fake/path"
I0716 19:28:26.652666  231841 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "timeout-setup-volume" (UniqueName: "fake-plugin/timeout-setup-volume") pod "pod1" (UID: "pod1uid") DevicePath "fake/path"
I0716 19:28:26.652789  231841 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "timeout-setup-volume" (UniqueName: "fake-plugin/timeout-setup-volume") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:28:26.652888  231841 operation_generator.go:594] MountVolume.MountDevice succeeded for volume "timeout-setup-volume" (UniqueName: "fake-plugin/timeout-setup-volume") pod "pod1" (UID: "pod1uid") device mount path ""
E0716 19:28:26.653319  231841 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/timeout-setup-volume podName: nodeName:}" failed. No retries permitted until 2021-07-16 19:28:27.153148247 +0000 UTC m=+96.094592781 (durationBeforeRetry 500ms). Error: "MountVolume.SetUp failed for volume \"timeout-setup-volume\" (UniqueName: \"fake-plugin/timeout-setup-volume\") pod \"pod1\" (UID: \"pod1uid\") : time out on setup"
I0716 19:28:26.812097  231841 reconciler.go:196] operationExecutor.UnmountVolume started for volume "timeout-setup-volume" (UniqueName: "fake-plugin/timeout-setup-volume") pod "pod1uid" (UID: "pod1uid") 
I0716 19:28:26.812273  231841 operation_generator.go:797] UnmountVolume.TearDown succeeded for volume "fake-plugin/timeout-setup-volume" (OuterVolumeSpecName: "timeout-setup-volume") pod "pod1uid" (UID: "pod1uid"). InnerVolumeSpecName "timeout-setup-volume". PluginName "fake-plugin", VolumeGidValue ""
I0716 19:28:26.812766  231841 reconciler.go:312] operationExecutor.UnmountDevice started for volume "timeout-setup-volume" (UniqueName: "fake-plugin/timeout-setup-volume") on node "mynodename" 
I0716 19:28:26.812933  231841 operation_generator.go:891] UnmountDevice succeeded for volume "timeout-setup-volume" %!(EXTRA string=UnmountDevice succeeded for volume "timeout-setup-volume" (UniqueName: "fake-plugin/timeout-setup-volume") on node "mynodename" )
I0716 19:28:26.813171  231841 reconciler.go:319] Volume detached for volume "timeout-setup-volume" (UniqueName: "fake-plugin/timeout-setup-volume") on node "mynodename" DevicePath "/dev/sdb"
I0716 19:28:26.887855  231841 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "fail-setup-volume" (UniqueName: "fake-plugin/fail-setup-volume") pod "pod1" (UID: "pod1uid") 
I0716 19:28:26.888617  231841 operation_generator.go:1357] Controller attach succeeded for volume "fail-setup-volume" (UniqueName: "fake-plugin/fail-setup-volume") pod "pod1" (UID: "pod1uid") device path: "fake/path"
I0716 19:28:26.888752  231841 reconciler.go:157] Reconciler: start to sync state
E0716 19:28:26.889514  231841 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0716 19:28:26.891369  231841 reconciler.go:319] Volume detached for volume "fail-setup-volume" (UniqueName: "fake-plugin/fail-setup-volume") on node "mynodename" DevicePath "fake/path"
I0716 19:28:27.012983  231841 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") pod "pod1" (UID: "pod1uid") 
I0716 19:28:27.013086  231841 reconciler.go:157] Reconciler: start to sync state
E0716 19:28:27.013146  231841 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0716 19:28:27.013651  231841 operation_generator.go:1357] Controller attach succeeded for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") pod "pod1" (UID: "pod1uid") device path: "fake/path"
I0716 19:28:27.014369  231841 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") pod "pod1" (UID: "pod1uid") DevicePath "fake/path"
I0716 19:28:27.014504  231841 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:28:27.014636  231841 operation_generator.go:594] MountVolume.MountDevice succeeded for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") pod "pod1" (UID: "pod1uid") device mount path ""
E0716 19:28:27.015063  231841 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/timeout-and-fail-setup-volume podName: nodeName:}" failed. No retries permitted until 2021-07-16 19:28:27.514881654 +0000 UTC m=+96.456326185 (durationBeforeRetry 500ms). Error: "MountVolume.SetUp failed for volume \"timeout-and-fail-setup-volume\" (UniqueName: \"fake-plugin/timeout-and-fail-setup-volume\") pod \"pod1\" (UID: \"pod1uid\") : time out on setup"
I0716 19:28:27.520459  231841 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:28:27.520618  231841 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0716 19:28:27.521018  231841 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/timeout-and-fail-setup-volume podName: nodeName:}" failed. No retries permitted until 2021-07-16 19:28:28.520823009 +0000 UTC m=+97.462267550 (durationBeforeRetry 1s). Error: "MountVolume.SetUp failed for volume \"timeout-and-fail-setup-volume\" (UniqueName: \"fake-plugin/timeout-and-fail-setup-volume\") pod \"pod1\" (UID: \"pod1uid\") : mounting volume failed"
I0716 19:28:28.527490  231841 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:28:28.530119  231841 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0716 19:28:28.559156  231841 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/timeout-and-fail-setup-volume podName: nodeName:}" failed. No retries permitted until 2021-07-16 19:28:30.558886364 +0000 UTC m=+99.500330921 (durationBeforeRetry 2s). Error: "MountVolume.SetUp failed for volume \"timeout-and-fail-setup-volume\" (UniqueName: \"fake-plugin/timeout-and-fail-setup-volume\") pod \"pod1\" (UID: \"pod1uid\") : mounting volume failed"
I0716 19:28:30.559303  231841 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:28:30.559466  231841 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0716 19:28:30.559881  231841 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/timeout-and-fail-setup-volume podName: nodeName:}" failed. No retries permitted until 2021-07-16 19:28:34.55968325 +0000 UTC m=+103.501127782 (durationBeforeRetry 4s). Error: "MountVolume.SetUp failed for volume \"timeout-and-fail-setup-volume\" (UniqueName: \"fake-plugin/timeout-and-fail-setup-volume\") pod \"pod1\" (UID: \"pod1uid\") : mounting volume failed"
I0716 19:28:34.575064  231841 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:28:34.575225  231841 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0716 19:28:34.575634  231841 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/timeout-and-fail-setup-volume podName: nodeName:}" failed. No retries permitted until 2021-07-16 19:28:42.575450454 +0000 UTC m=+111.516894985 (durationBeforeRetry 8s). Error: "MountVolume.SetUp failed for volume \"timeout-and-fail-setup-volume\" (UniqueName: \"fake-plugin/timeout-and-fail-setup-volume\") pod \"pod1\" (UID: \"pod1uid\") : mounting volume failed"
I0716 19:28:37.142435  231841 reconciler.go:312] operationExecutor.UnmountDevice started for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") on node "mynodename" 
I0716 19:28:37.142807  231841 operation_generator.go:891] UnmountDevice succeeded for volume "timeout-and-fail-setup-volume" %!(EXTRA string=UnmountDevice succeeded for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") on node "mynodename" )
I0716 19:28:37.143062  231841 reconciler.go:319] Volume detached for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") on node "mynodename" DevicePath "/dev/sdb"
I0716 19:28:37.274037  231841 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1" (UID: "pod1uid") 
I0716 19:28:37.274131  231841 reconciler.go:157] Reconciler: start to sync state
E0716 19:28:37.274224  231841 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0716 19:28:37.274988  231841 operation_generator.go:1357] Controller attach succeeded for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1" (UID: "pod1uid") device path: "fake/path"
I0716 19:28:37.275640  231841 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1" (UID: "pod1uid") DevicePath "fake/path"
I0716 19:28:37.275778  231841 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:28:37.275909  231841 operation_generator.go:594] MountVolume.MountDevice succeeded for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1" (UID: "pod1uid") device mount path ""
I0716 19:28:37.382903  231841 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:28:37.383083  231841 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0716 19:28:37.383720  231841 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-timeout-setup-volume-name podName: nodeName:}" failed. No retries permitted until 2021-07-16 19:28:37.883466512 +0000 UTC m=+106.824911056 (durationBeforeRetry 500ms). Error: "MountVolume.SetUp failed for volume \"success-and-timeout-setup-volume-name\" (UniqueName: \"fake-plugin/success-and-timeout-setup-volume-name\") pod \"pod1\" (UID: \"pod1uid\") : time out on setup"
I0716 19:28:37.883931  231841 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:28:37.884120  231841 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0716 19:28:37.884610  231841 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-timeout-setup-volume-name podName: nodeName:}" failed. No retries permitted until 2021-07-16 19:28:38.88434348 +0000 UTC m=+107.825788022 (durationBeforeRetry 1s). Error: "MountVolume.SetUp failed for volume \"success-and-timeout-setup-volume-name\" (UniqueName: \"fake-plugin/success-and-timeout-setup-volume-name\") pod \"pod1\" (UID: \"pod1uid\") : time out on setup"
I0716 19:28:38.887070  231841 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:28:38.887238  231841 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0716 19:28:38.887653  231841 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-timeout-setup-volume-name podName: nodeName:}" failed. No retries permitted until 2021-07-16 19:28:40.887446208 +0000 UTC m=+109.828890739 (durationBeforeRetry 2s). Error: "MountVolume.SetUp failed for volume \"success-and-timeout-setup-volume-name\" (UniqueName: \"fake-plugin/success-and-timeout-setup-volume-name\") pod \"pod1\" (UID: \"pod1uid\") : time out on setup"
I0716 19:28:40.887980  231841 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:28:40.888163  231841 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0716 19:28:40.888587  231841 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-timeout-setup-volume-name podName: nodeName:}" failed. No retries permitted until 2021-07-16 19:28:44.888389097 +0000 UTC m=+113.829833631 (durationBeforeRetry 4s). Error: "MountVolume.SetUp failed for volume \"success-and-timeout-setup-volume-name\" (UniqueName: \"fake-plugin/success-and-timeout-setup-volume-name\") pod \"pod1\" (UID: \"pod1uid\") : time out on setup"
I0716 19:28:44.891250  231841 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:28:44.896011  231841 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0716 19:28:44.915996  231841 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-timeout-setup-volume-name podName: nodeName:}" failed. No retries permitted until 2021-07-16 19:28:52.915768713 +0000 UTC m=+121.857213258 (durationBeforeRetry 8s). Error: "MountVolume.SetUp failed for volume \"success-and-timeout-setup-volume-name\" (UniqueName: \"fake-plugin/success-and-timeout-setup-volume-name\") pod \"pod1\" (UID: \"pod1uid\") : time out on setup"
I0716 19:28:47.423302  231841 reconciler.go:196] operationExecutor.UnmountVolume started for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1uid" (UID: "pod1uid") 
I0716 19:28:47.423557  231841 operation_generator.go:797] UnmountVolume.TearDown succeeded for volume "fake-plugin/success-and-timeout-setup-volume-name" (OuterVolumeSpecName: "success-and-timeout-setup-volume-name") pod "pod1uid" (UID: "pod1uid"). InnerVolumeSpecName "success-and-timeout-setup-volume-name". PluginName "fake-plugin", VolumeGidValue ""
I0716 19:28:47.424077  231841 reconciler.go:312] operationExecutor.UnmountDevice started for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") on node "mynodename" 
I0716 19:28:47.424447  231841 operation_generator.go:891] UnmountDevice succeeded for volume "success-and-timeout-setup-volume-name" %!(EXTRA string=UnmountDevice succeeded for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") on node "mynodename" )
I0716 19:28:47.424670  231841 reconciler.go:319] Volume detached for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") on node "mynodename" DevicePath "/dev/sdb"
I0716 19:28:47.528917  231841 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1" (UID: "pod1uid") 
I0716 19:28:47.528996  231841 reconciler.go:157] Reconciler: start to sync state
E0716 19:28:47.529056  231841 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0716 19:28:47.529290  231841 operation_generator.go:1357] Controller attach succeeded for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1" (UID: "pod1uid") device path: "fake/path"
I0716 19:28:47.530060  231841 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1" (UID: "pod1uid") DevicePath "fake/path"
I0716 19:28:47.530229  231841 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:28:47.530366  231841 operation_generator.go:594] MountVolume.MountDevice succeeded for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1" (UID: "pod1uid") device mount path ""
I0716 19:28:47.631641  231841 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:28:47.631819  231841 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0716 19:28:47.632509  231841 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-failed-setup-device-name podName: nodeName:}" failed. No retries permitted until 2021-07-16 19:28:48.132047864 +0000 UTC m=+117.073492395 (durationBeforeRetry 500ms). Error: "MountVolume.SetUp failed for volume \"success-and-failed-setup-device-name\" (UniqueName: \"fake-plugin/success-and-failed-setup-device-name\") pod \"pod1\" (UID: \"pod1uid\") : mounting volume failed"
I0716 19:28:48.133059  231841 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:28:48.133281  231841 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0716 19:28:48.133833  231841 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-failed-setup-device-name podName: nodeName:}" failed. No retries permitted until 2021-07-16 19:28:49.133573398 +0000 UTC m=+118.075017937 (durationBeforeRetry 1s). Error: "MountVolume.SetUp failed for volume \"success-and-failed-setup-device-name\" (UniqueName: \"fake-plugin/success-and-failed-setup-device-name\") pod \"pod1\" (UID: \"pod1uid\") : mounting volume failed"
I0716 19:28:49.135607  231841 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:28:49.140086  231841 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0716 19:28:49.144440  231841 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-failed-setup-device-name podName: nodeName:}" failed. No retries permitted until 2021-07-16 19:28:51.141629313 +0000 UTC m=+120.083073847 (durationBeforeRetry 2s). Error: "MountVolume.SetUp failed for volume \"success-and-failed-setup-device-name\" (UniqueName: \"fake-plugin/success-and-failed-setup-device-name\") pod \"pod1\" (UID: \"pod1uid\") : mounting volume failed"
I0716 19:28:51.145026  231841 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:28:51.145236  231841 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0716 19:28:51.148000  231841 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-failed-setup-device-name podName: nodeName:}" failed. No retries permitted until 2021-07-16 19:28:55.146293524 +0000 UTC m=+124.087738063 (durationBeforeRetry 4s). Error: "MountVolume.SetUp failed for volume \"success-and-failed-setup-device-name\" (UniqueName: \"fake-plugin/success-and-failed-setup-device-name\") pod \"pod1\" (UID: \"pod1uid\") : mounting volume failed"
I0716 19:28:55.159332  231841 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:28:55.159553  231841 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0716 19:28:55.160065  231841 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-failed-setup-device-name podName: nodeName:}" failed. No retries permitted until 2021-07-16 19:29:03.159803288 +0000 UTC m=+132.101247834 (durationBeforeRetry 8s). Error: "MountVolume.SetUp failed for volume \"success-and-failed-setup-device-name\" (UniqueName: \"fake-plugin/success-and-failed-setup-device-name\") pod \"pod1\" (UID: \"pod1uid\") : mounting volume failed"
I0716 19:28:57.641643  231841 reconciler.go:196] operationExecutor.UnmountVolume started for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1uid" (UID: "pod1uid") 
I0716 19:28:57.642789  231841 operation_generator.go:797] UnmountVolume.TearDown succeeded for volume "fake-plugin/success-and-failed-setup-device-name" (OuterVolumeSpecName: "success-and-failed-setup-device-name") pod "pod1uid" (UID: "pod1uid"). InnerVolumeSpecName "success-and-failed-setup-device-name". PluginName "fake-plugin", VolumeGidValue ""
I0716 19:28:57.662555  231841 reconciler.go:312] operationExecutor.UnmountDevice started for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") on node "mynodename" 
I0716 19:28:57.662877  231841 operation_generator.go:891] UnmountDevice succeeded for volume "success-and-failed-setup-device-name" %!(EXTRA string=UnmountDevice succeeded for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") on node "mynodename" )
I0716 19:28:57.663192  231841 reconciler.go:319] Volume detached for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") on node "mynodename" DevicePath "/dev/sdb"
--- FAIL: Test_UncertainVolumeMountState (62.21s)
    --- FAIL: Test_UncertainVolumeMountState/failed_operation_should_result_in_not-mounted_volume_[Filesystem] (0.13s)
        reconciler_test.go:1523: Error verifying UnMountDeviceCallCount: Expected DeviceUnmount Call 1, got 0
I0716 19:28:57.742282  231841 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") 
I0716 19:28:57.742358  231841 reconciler.go:157] Reconciler: start to sync state
E0716 19:28:57.742415  231841 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0716 19:28:57.742823  231841 operation_generator.go:1357] Controller attach succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") device path: "/fake/path"
I0716 19:28:57.743473  231841 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") DevicePath "/fake/path"
I0716 19:28:57.743594  231841 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:28:57.743695  231841 operation_generator.go:594] MountVolume.MountDevice succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") device mount path ""
I0716 19:28:57.847859  231841 reconciler.go:196] operationExecutor.UnmountVolume started for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1uid" (UID: "pod1uid") 
I0716 19:28:57.848291  231841 operation_generator.go:797] UnmountVolume.TearDown succeeded for volume "fake-plugin/fake-device1" (OuterVolumeSpecName: "volume-name") pod "pod1uid" (UID: "pod1uid"). InnerVolumeSpecName "volume-name". PluginName "fake-plugin", VolumeGidValue ""
I0716 19:28:57.848954  231841 reconciler.go:312] operationExecutor.UnmountDevice started for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") on node "mynodename" 
I0716 19:28:57.849136  231841 reconciler_test.go:1798] UnmountDevice called
I0716 19:28:57.849327  231841 operation_generator.go:891] UnmountDevice succeeded for volume "volume-name" %!(EXTRA string=UnmountDevice succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") on node "mynodename" )
I0716 19:28:57.850064  231841 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:28:57.850217  231841 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:28:57.850316  231841 operation_generator.go:594] MountVolume.MountDevice succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") device mount path ""
FAIL

				from junit_bazel.xml

Find pod1 mentions in log files | View test history on testgrid


//pkg/kubelet/volumemanager/reconciler/go_default_test:run_2_of_2 0.00s

bazel test //pkg/kubelet/volumemanager/reconciler/go_default_test:run_2_of_2
exec ${PAGER:-/usr/bin/less} "$0" || exit 1
Executing tests from //pkg/kubelet/volumemanager/reconciler:go_default_test
-----------------------------------------------------------------------------
I0716 19:26:51.322163  231841 reconciler.go:157] Reconciler: start to sync state
E0716 19:26:51.322295  231841 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0716 19:26:51.339541  231841 reconciler.go:244] operationExecutor.AttachVolume started for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") 
I0716 19:26:51.339625  231841 reconciler.go:157] Reconciler: start to sync state
E0716 19:26:51.339678  231841 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0716 19:26:51.340052  231841 operation_generator.go:360] AttachVolume.Attach succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") from node "mynodename" 
I0716 19:26:51.340780  231841 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") DevicePath "/dev/vdb-test"
I0716 19:26:51.340891  231841 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:26:51.340987  231841 operation_generator.go:594] MountVolume.MountDevice succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") device mount path ""
I0716 19:26:51.440373  231841 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") 
I0716 19:26:51.440445  231841 reconciler.go:157] Reconciler: start to sync state
E0716 19:26:51.440496  231841 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0716 19:26:51.440917  231841 operation_generator.go:1357] Controller attach succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") device path: "/fake/path"
I0716 19:26:51.441435  231841 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") DevicePath "/fake/path"
I0716 19:26:51.441545  231841 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:26:51.441654  231841 operation_generator.go:594] MountVolume.MountDevice succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") device mount path ""
I0716 19:26:51.541177  231841 reconciler.go:244] operationExecutor.AttachVolume started for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") 
I0716 19:26:51.541284  231841 reconciler.go:157] Reconciler: start to sync state
E0716 19:26:51.541337  231841 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0716 19:26:51.541852  231841 operation_generator.go:360] AttachVolume.Attach succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") from node "mynodename" 
I0716 19:26:51.542700  231841 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") DevicePath "/dev/vdb-test"
I0716 19:26:51.542832  231841 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:26:51.542929  231841 operation_generator.go:594] MountVolume.MountDevice succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") device mount path ""
I0716 19:26:51.643677  231841 reconciler.go:196] operationExecutor.UnmountVolume started for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1uid" (UID: "pod1uid") 
I0716 19:26:51.643920  231841 operation_generator.go:797] UnmountVolume.TearDown succeeded for volume "fake-plugin/fake-device1" (OuterVolumeSpecName: "volume-name") pod "pod1uid" (UID: "pod1uid"). InnerVolumeSpecName "volume-name". PluginName "fake-plugin", VolumeGidValue ""
I0716 19:26:51.644400  231841 reconciler.go:312] operationExecutor.UnmountDevice started for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") on node "mynodename" 
I0716 19:26:51.644648  231841 operation_generator.go:891] UnmountDevice succeeded for volume "volume-name" %!(EXTRA string=UnmountDevice succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") on node "mynodename" )
I0716 19:26:51.645004  231841 reconciler.go:333] operationExecutor.DetachVolume started for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") on node "mynodename" 
I0716 19:26:51.645113  231841 operation_generator.go:470] DetachVolume.Detach succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") on node "mynodename" 
I0716 19:26:51.742320  231841 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") 
I0716 19:26:51.742400  231841 reconciler.go:157] Reconciler: start to sync state
E0716 19:26:51.742467  231841 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0716 19:26:51.742904  231841 operation_generator.go:1357] Controller attach succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") device path: "/fake/path"
I0716 19:26:51.746061  231841 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") DevicePath "/fake/path"
I0716 19:26:51.746442  231841 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:26:51.749193  231841 operation_generator.go:594] MountVolume.MountDevice succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") device mount path ""
I0716 19:26:51.848818  231841 reconciler.go:196] operationExecutor.UnmountVolume started for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1uid" (UID: "pod1uid") 
I0716 19:26:51.854538  231841 operation_generator.go:797] UnmountVolume.TearDown succeeded for volume "fake-plugin/fake-device1" (OuterVolumeSpecName: "volume-name") pod "pod1uid" (UID: "pod1uid"). InnerVolumeSpecName "volume-name". PluginName "fake-plugin", VolumeGidValue ""
I0716 19:26:51.869221  231841 operation_generator.go:797] UnmountVolume.TearDown succeeded for volume "fake-plugin/fake-device1" (OuterVolumeSpecName: "volume-name") pod "pod1uid" (UID: "pod1uid"). InnerVolumeSpecName "volume-name". PluginName "fake-plugin", VolumeGidValue ""
I0716 19:26:51.869409  231841 reconciler.go:196] operationExecutor.UnmountVolume started for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1uid" (UID: "pod1uid") 
I0716 19:26:51.870034  231841 operation_generator.go:891] UnmountDevice succeeded for volume "volume-name" %!(EXTRA string=UnmountDevice succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") on node "mynodename" )
I0716 19:26:51.870054  231841 reconciler.go:312] operationExecutor.UnmountDevice started for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") on node "mynodename" 
I0716 19:26:51.870665  231841 reconciler.go:319] Volume detached for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") on node "mynodename" DevicePath "/dev/sdb"
I0716 19:26:51.958877  231841 reconciler.go:244] operationExecutor.AttachVolume started for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") 
I0716 19:26:51.958948  231841 reconciler.go:157] Reconciler: start to sync state
E0716 19:26:51.959004  231841 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0716 19:26:51.959202  231841 operation_generator.go:360] AttachVolume.Attach succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") from node "mynodename" 
I0716 19:26:51.959854  231841 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") DevicePath "/dev/vdb-test"
I0716 19:26:51.959976  231841 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:26:52.059725  231841 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") 
I0716 19:26:52.059792  231841 reconciler.go:157] Reconciler: start to sync state
E0716 19:26:52.059840  231841 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0716 19:26:52.060275  231841 operation_generator.go:1357] Controller attach succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") device path: "/fake/path"
I0716 19:26:52.060887  231841 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") DevicePath "/fake/path"
I0716 19:26:52.061020  231841 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:26:52.065158  231841 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:26:52.065308  231841 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:26:52.163420  231841 reconciler.go:244] operationExecutor.AttachVolume started for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") 
I0716 19:26:52.163554  231841 reconciler.go:157] Reconciler: start to sync state
E0716 19:26:52.163612  231841 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0716 19:26:52.167956  231841 operation_generator.go:360] AttachVolume.Attach succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") from node "mynodename" 
I0716 19:26:52.168793  231841 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") DevicePath "/dev/vdb-test"
I0716 19:26:52.168922  231841 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:26:52.282488  231841 reconciler.go:196] operationExecutor.UnmountVolume started for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1uid" (UID: "pod1uid") 
I0716 19:26:52.311607  231841 operation_generator.go:1170] UnmapVolume succeeded for volume "fake-plugin/fake-device1" (OuterVolumeSpecName: "volume-name") pod "pod1uid" (UID: "pod1uid"). InnerVolumeSpecName "volume-name". PluginName "fake-plugin", VolumeGidValue ""
I0716 19:26:52.341852  231841 reconciler.go:312] operationExecutor.UnmountDevice started for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") on node "mynodename" 
I0716 19:26:52.350252  231841 operation_generator.go:1282] UnmapDevice succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") on node "mynodename" 
I0716 19:26:52.350807  231841 reconciler.go:333] operationExecutor.DetachVolume started for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") on node "mynodename" 
I0716 19:26:52.351124  231841 operation_generator.go:470] DetachVolume.Detach succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") on node "mynodename" 
I0716 19:26:52.368016  231841 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") 
I0716 19:26:52.368232  231841 operation_generator.go:1357] Controller attach succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") device path: "/fake/path"
I0716 19:26:52.368337  231841 reconciler.go:157] Reconciler: start to sync state
E0716 19:26:52.368401  231841 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0716 19:26:52.369435  231841 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") DevicePath "/fake/path"
I0716 19:26:52.369926  231841 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:26:52.518578  231841 reconciler.go:196] operationExecutor.UnmountVolume started for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1uid" (UID: "pod1uid") 
I0716 19:26:52.518887  231841 operation_generator.go:1170] UnmapVolume succeeded for volume "fake-plugin/fake-device1" (OuterVolumeSpecName: "volume-name") pod "pod1uid" (UID: "pod1uid"). InnerVolumeSpecName "volume-name". PluginName "fake-plugin", VolumeGidValue ""
I0716 19:26:52.519276  231841 reconciler.go:312] operationExecutor.UnmountDevice started for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") on node "mynodename" 
I0716 19:26:52.519468  231841 operation_generator.go:1282] UnmapDevice succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") on node "mynodename" 
I0716 19:26:52.519615  231841 reconciler.go:319] Volume detached for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") on node "mynodename" DevicePath "/dev/sdb"
I0716 19:26:52.634520  231841 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "pv" (UniqueName: "fake-plugin/pv") pod "pod1" (UID: "pod1uid") 
I0716 19:26:52.634692  231841 reconciler.go:157] Reconciler: start to sync state
E0716 19:26:52.634781  231841 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0716 19:26:52.634894  231841 operation_generator.go:1357] Controller attach succeeded for volume "pv" (UniqueName: "fake-plugin/pv") pod "pod1" (UID: "pod1uid") device path: "fake/path"
I0716 19:26:52.635579  231841 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "pv" (UniqueName: "fake-plugin/pv") pod "pod1" (UID: "pod1uid") DevicePath "fake/path"
I0716 19:26:52.635964  231841 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "pv" (UniqueName: "fake-plugin/pv") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:26:52.636082  231841 operation_generator.go:594] MountVolume.MountDevice succeeded for volume "pv" (UniqueName: "fake-plugin/pv") pod "pod1" (UID: "pod1uid") device mount path ""
I0716 19:26:52.741177  231841 operation_generator.go:1683] MountVolume.NodeExpandVolume succeeded for volume "pv" (UniqueName: "fake-plugin/pv") pod "pod1" (UID: "pod1uid") 
I0716 19:26:52.846913  231841 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "pv" (UniqueName: "fake-plugin/pv") pod "pod1" (UID: "pod1uid") 
I0716 19:26:52.846989  231841 reconciler.go:157] Reconciler: start to sync state
E0716 19:26:52.847042  231841 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0716 19:26:52.847418  231841 operation_generator.go:1357] Controller attach succeeded for volume "pv" (UniqueName: "fake-plugin/pv") pod "pod1" (UID: "pod1uid") device path: "fake/path"
I0716 19:26:52.847941  231841 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "pv" (UniqueName: "fake-plugin/pv") pod "pod1" (UID: "pod1uid") DevicePath "fake/path"
I0716 19:26:52.848062  231841 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "pv" (UniqueName: "fake-plugin/pv") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:26:52.981943  231841 operation_generator.go:1683] MountVolume.NodeExpandVolume succeeded for volume "pv" (UniqueName: "fake-plugin/pv") pod "pod1" (UID: "pod1uid") 
I0716 19:26:53.084228  231841 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "fail-expansion-in-use" (UniqueName: "fake-plugin/fail-expansion-in-use") pod "pod1" (UID: "pod1uid") 
I0716 19:26:53.084347  231841 reconciler.go:157] Reconciler: start to sync state
E0716 19:26:53.084394  231841 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0716 19:26:53.084938  231841 operation_generator.go:1357] Controller attach succeeded for volume "fail-expansion-in-use" (UniqueName: "fake-plugin/fail-expansion-in-use") pod "pod1" (UID: "pod1uid") device path: "fake/path"
I0716 19:26:53.085821  231841 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "fail-expansion-in-use" (UniqueName: "fake-plugin/fail-expansion-in-use") pod "pod1" (UID: "pod1uid") DevicePath "fake/path"
I0716 19:26:53.086014  231841 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "fail-expansion-in-use" (UniqueName: "fake-plugin/fail-expansion-in-use") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:26:53.086328  231841 operation_generator.go:594] MountVolume.MountDevice succeeded for volume "fail-expansion-in-use" (UniqueName: "fake-plugin/fail-expansion-in-use") pod "pod1" (UID: "pod1uid") device mount path ""
E0716 19:26:53.086787  231841 operation_generator.go:1669] MountVolume.NodeExapndVolume failed with %!!(MISSING)v(MISSING) for volume "fail-expansion-in-use" (UniqueName: "fake-plugin/fail-expansion-in-use") pod "pod1" (UID: "pod1uid") : volume-in-use
I0716 19:26:53.200864  231841 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "timeout-mount-device-volume" (UniqueName: "fake-plugin/timeout-mount-device-volume") pod "pod1" (UID: "pod1uid") 
I0716 19:26:53.200967  231841 reconciler.go:157] Reconciler: start to sync state
E0716 19:26:53.201027  231841 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0716 19:26:53.201350  231841 operation_generator.go:1357] Controller attach succeeded for volume "timeout-mount-device-volume" (UniqueName: "fake-plugin/timeout-mount-device-volume") pod "pod1" (UID: "pod1uid") device path: "fake/path"
I0716 19:26:53.202015  231841 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "timeout-mount-device-volume" (UniqueName: "fake-plugin/timeout-mount-device-volume") pod "pod1" (UID: "pod1uid") DevicePath "fake/path"
I0716 19:26:53.219518  231841 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "timeout-mount-device-volume" (UniqueName: "fake-plugin/timeout-mount-device-volume") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0716 19:26:53.220315  231841 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/timeout-mount-device-volume podName: nodeName:}" failed. No retries permitted until 2021-07-16 19:26:53.720044148 +0000 UTC m=+2.661488687 (durationBeforeRetry 500ms). Error: "MapVolume.SetUpDevice failed for volume \"timeout-mount-device-volume\" (UniqueName: \"fake-plugin/timeout-mount-device-volume\") pod \"pod1\" (UID: \"pod1uid\") : mount failed"
I0716 19:26:53.286533  231841 reconciler.go:312] operationExecutor.UnmountDevice started for volume "timeout-mount-device-volume" (UniqueName: "fake-plugin/timeout-mount-device-volume") on node "mynodename" 
I0716 19:26:53.286932  231841 operation_generator.go:1282] UnmapDevice succeeded for volume "timeout-mount-device-volume" (UniqueName: "fake-plugin/timeout-mount-device-volume") on node "mynodename" 
I0716 19:26:53.287179  231841 reconciler.go:319] Volume detached for volume "timeout-mount-device-volume" (UniqueName: "fake-plugin/timeout-mount-device-volume") on node "mynodename" DevicePath "/dev/sdb"
I0716 19:26:53.389464  231841 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "fail-mount-device-volume-name" (UniqueName: "fake-plugin/fail-mount-device-volume-name") pod "pod1" (UID: "pod1uid") 
I0716 19:26:53.389567  231841 reconciler.go:157] Reconciler: start to sync state
E0716 19:26:53.389635  231841 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0716 19:26:53.390082  231841 operation_generator.go:1357] Controller attach succeeded for volume "fail-mount-device-volume-name" (UniqueName: "fake-plugin/fail-mount-device-volume-name") pod "pod1" (UID: "pod1uid") device path: "fake/path"
I0716 19:26:53.459045  231841 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "fail-mount-device-volume-name" (UniqueName: "fake-plugin/fail-mount-device-volume-name") pod "pod1" (UID: "pod1uid") DevicePath "fake/path"
I0716 19:26:53.459201  231841 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "fail-mount-device-volume-name" (UniqueName: "fake-plugin/fail-mount-device-volume-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0716 19:26:53.459627  231841 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/fail-mount-device-volume-name podName: nodeName:}" failed. No retries permitted until 2021-07-16 19:26:53.959414933 +0000 UTC m=+2.900859468 (durationBeforeRetry 500ms). Error: "MapVolume.SetUpDevice failed for volume \"fail-mount-device-volume-name\" (UniqueName: \"fake-plugin/fail-mount-device-volume-name\") pod \"pod1\" (UID: \"pod1uid\") : error mapping disk: fail-mount-device-volume-name"
I0716 19:26:53.560999  231841 reconciler.go:319] Volume detached for volume "fail-mount-device-volume-name" (UniqueName: "fake-plugin/fail-mount-device-volume-name") on node "mynodename" DevicePath "fake/path"
I0716 19:26:53.654312  231841 operation_generator.go:1357] Controller attach succeeded for volume "timeout-and-fail-mount-device-name" (UniqueName: "fake-plugin/timeout-and-fail-mount-device-name") pod "pod1" (UID: "pod1uid") device path: "fake/path"
I0716 19:26:53.655045  231841 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "timeout-and-fail-mount-device-name" (UniqueName: "fake-plugin/timeout-and-fail-mount-device-name") pod "pod1" (UID: "pod1uid") 
I0716 19:26:53.655119  231841 reconciler.go:157] Reconciler: start to sync state
E0716 19:26:53.655177  231841 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0716 19:26:53.663007  231841 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "timeout-and-fail-mount-device-name" (UniqueName: "fake-plugin/timeout-and-fail-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "fake/path"
I0716 19:26:53.663236  231841 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "timeout-and-fail-mount-device-name" (UniqueName: "fake-plugin/timeout-and-fail-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0716 19:26:53.664519  231841 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/timeout-and-fail-mount-device-name podName: nodeName:}" failed. No retries permitted until 2021-07-16 19:26:54.164166721 +0000 UTC m=+3.105611258 (durationBeforeRetry 500ms). Error: "MapVolume.SetUpDevice failed for volume \"timeout-and-fail-mount-device-name\" (UniqueName: \"fake-plugin/timeout-and-fail-mount-device-name\") pod \"pod1\" (UID: \"pod1uid\") : timed out mounting error"
I0716 19:26:54.164605  231841 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "timeout-and-fail-mount-device-name" (UniqueName: "fake-plugin/timeout-and-fail-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:26:54.164783  231841 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "timeout-and-fail-mount-device-name" (UniqueName: "fake-plugin/timeout-and-fail-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0716 19:26:54.166952  231841 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/timeout-and-fail-mount-device-name podName: nodeName:}" failed. No retries permitted until 2021-07-16 19:26:55.166725341 +0000 UTC m=+4.108169933 (durationBeforeRetry 1s). Error: "MapVolume.SetUpDevice failed for volume \"timeout-and-fail-mount-device-name\" (UniqueName: \"fake-plugin/timeout-and-fail-mount-device-name\") pod \"pod1\" (UID: \"pod1uid\") : error mapping disk: timeout-and-fail-mount-device-name"
I0716 19:26:55.176372  231841 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "timeout-and-fail-mount-device-name" (UniqueName: "fake-plugin/timeout-and-fail-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:26:55.176565  231841 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "timeout-and-fail-mount-device-name" (UniqueName: "fake-plugin/timeout-and-fail-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0716 19:26:55.176988  231841 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/timeout-and-fail-mount-device-name podName: nodeName:}" failed. No retries permitted until 2021-07-16 19:26:57.176779712 +0000 UTC m=+6.118224248 (durationBeforeRetry 2s). Error: "MapVolume.SetUpDevice failed for volume \"timeout-and-fail-mount-device-name\" (UniqueName: \"fake-plugin/timeout-and-fail-mount-device-name\") pod \"pod1\" (UID: \"pod1uid\") : error mapping disk: timeout-and-fail-mount-device-name"
I0716 19:26:57.180235  231841 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "timeout-and-fail-mount-device-name" (UniqueName: "fake-plugin/timeout-and-fail-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:26:57.180514  231841 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "timeout-and-fail-mount-device-name" (UniqueName: "fake-plugin/timeout-and-fail-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0716 19:26:57.181382  231841 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/timeout-and-fail-mount-device-name podName: nodeName:}" failed. No retries permitted until 2021-07-16 19:27:01.180806338 +0000 UTC m=+10.122250888 (durationBeforeRetry 4s). Error: "MapVolume.SetUpDevice failed for volume \"timeout-and-fail-mount-device-name\" (UniqueName: \"fake-plugin/timeout-and-fail-mount-device-name\") pod \"pod1\" (UID: \"pod1uid\") : error mapping disk: timeout-and-fail-mount-device-name"
I0716 19:27:01.190870  231841 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "timeout-and-fail-mount-device-name" (UniqueName: "fake-plugin/timeout-and-fail-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:27:01.191049  231841 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "timeout-and-fail-mount-device-name" (UniqueName: "fake-plugin/timeout-and-fail-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0716 19:27:01.191496  231841 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/timeout-and-fail-mount-device-name podName: nodeName:}" failed. No retries permitted until 2021-07-16 19:27:09.19127461 +0000 UTC m=+18.132719170 (durationBeforeRetry 8s). Error: "MapVolume.SetUpDevice failed for volume \"timeout-and-fail-mount-device-name\" (UniqueName: \"fake-plugin/timeout-and-fail-mount-device-name\") pod \"pod1\" (UID: \"pod1uid\") : error mapping disk: timeout-and-fail-mount-device-name"
I0716 19:27:03.770421  231841 reconciler.go:319] Volume detached for volume "timeout-and-fail-mount-device-name" (UniqueName: "fake-plugin/timeout-and-fail-mount-device-name") on node "mynodename" DevicePath "/dev/sdb"
I0716 19:27:03.863409  231841 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "success-and-timeout-device-name" (UniqueName: "fake-plugin/success-and-timeout-device-name") pod "pod1" (UID: "pod1uid") 
I0716 19:27:03.863511  231841 reconciler.go:157] Reconciler: start to sync state
E0716 19:27:03.863566  231841 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0716 19:27:03.863870  231841 operation_generator.go:1357] Controller attach succeeded for volume "success-and-timeout-device-name" (UniqueName: "fake-plugin/success-and-timeout-device-name") pod "pod1" (UID: "pod1uid") device path: "fake/path"
I0716 19:27:03.864594  231841 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "success-and-timeout-device-name" (UniqueName: "fake-plugin/success-and-timeout-device-name") pod "pod1" (UID: "pod1uid") DevicePath "fake/path"
I0716 19:27:03.864738  231841 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "success-and-timeout-device-name" (UniqueName: "fake-plugin/success-and-timeout-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:27:03.984294  231841 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "success-and-timeout-device-name" (UniqueName: "fake-plugin/success-and-timeout-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:27:03.984470  231841 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "success-and-timeout-device-name" (UniqueName: "fake-plugin/success-and-timeout-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0716 19:27:03.984860  231841 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-timeout-device-name podName: nodeName:}" failed. No retries permitted until 2021-07-16 19:27:04.484666104 +0000 UTC m=+13.426110637 (durationBeforeRetry 500ms). Error: "MapVolume.SetUpDevice failed for volume \"success-and-timeout-device-name\" (UniqueName: \"fake-plugin/success-and-timeout-device-name\") pod \"pod1\" (UID: \"pod1uid\") : error mounting state"
I0716 19:27:04.485230  231841 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "success-and-timeout-device-name" (UniqueName: "fake-plugin/success-and-timeout-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:27:04.485436  231841 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "success-and-timeout-device-name" (UniqueName: "fake-plugin/success-and-timeout-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0716 19:27:04.485837  231841 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-timeout-device-name podName: nodeName:}" failed. No retries permitted until 2021-07-16 19:27:05.485651604 +0000 UTC m=+14.427096134 (durationBeforeRetry 1s). Error: "MapVolume.SetUpDevice failed for volume \"success-and-timeout-device-name\" (UniqueName: \"fake-plugin/success-and-timeout-device-name\") pod \"pod1\" (UID: \"pod1uid\") : error mounting state"
I0716 19:27:05.522537  231841 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "success-and-timeout-device-name" (UniqueName: "fake-plugin/success-and-timeout-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:27:05.522725  231841 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "success-and-timeout-device-name" (UniqueName: "fake-plugin/success-and-timeout-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0716 19:27:05.523097  231841 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-timeout-device-name podName: nodeName:}" failed. No retries permitted until 2021-07-16 19:27:07.522928129 +0000 UTC m=+16.464372664 (durationBeforeRetry 2s). Error: "MapVolume.SetUpDevice failed for volume \"success-and-timeout-device-name\" (UniqueName: \"fake-plugin/success-and-timeout-device-name\") pod \"pod1\" (UID: \"pod1uid\") : error mounting state"
I0716 19:27:07.544819  231841 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "success-and-timeout-device-name" (UniqueName: "fake-plugin/success-and-timeout-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:27:07.545001  231841 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "success-and-timeout-device-name" (UniqueName: "fake-plugin/success-and-timeout-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0716 19:27:07.545450  231841 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-timeout-device-name podName: nodeName:}" failed. No retries permitted until 2021-07-16 19:27:11.545192856 +0000 UTC m=+20.486637392 (durationBeforeRetry 4s). Error: "MapVolume.SetUpDevice failed for volume \"success-and-timeout-device-name\" (UniqueName: \"fake-plugin/success-and-timeout-device-name\") pod \"pod1\" (UID: \"pod1uid\") : error mounting state"
I0716 19:27:11.555057  231841 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "success-and-timeout-device-name" (UniqueName: "fake-plugin/success-and-timeout-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:27:11.556175  231841 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "success-and-timeout-device-name" (UniqueName: "fake-plugin/success-and-timeout-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0716 19:27:11.558334  231841 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-timeout-device-name podName: nodeName:}" failed. No retries permitted until 2021-07-16 19:27:19.557210768 +0000 UTC m=+28.498655302 (durationBeforeRetry 8s). Error: "MapVolume.SetUpDevice failed for volume \"success-and-timeout-device-name\" (UniqueName: \"fake-plugin/success-and-timeout-device-name\") pod \"pod1\" (UID: \"pod1uid\") : error mounting state"
I0716 19:27:13.996997  231841 reconciler.go:196] operationExecutor.UnmountVolume started for volume "success-and-timeout-device-name" (UniqueName: "fake-plugin/success-and-timeout-device-name") pod "pod1uid" (UID: "pod1uid") 
I0716 19:27:13.998125  231841 operation_generator.go:1170] UnmapVolume succeeded for volume "fake-plugin/success-and-timeout-device-name" (OuterVolumeSpecName: "success-and-timeout-device-name") pod "pod1uid" (UID: "pod1uid"). InnerVolumeSpecName "success-and-timeout-device-name". PluginName "fake-plugin", VolumeGidValue ""
I0716 19:27:13.998587  231841 reconciler.go:312] operationExecutor.UnmountDevice started for volume "success-and-timeout-device-name" (UniqueName: "fake-plugin/success-and-timeout-device-name") on node "mynodename" 
I0716 19:27:13.998988  231841 operation_generator.go:1282] UnmapDevice succeeded for volume "success-and-timeout-device-name" (UniqueName: "fake-plugin/success-and-timeout-device-name") on node "mynodename" 
I0716 19:27:13.999160  231841 reconciler.go:319] Volume detached for volume "success-and-timeout-device-name" (UniqueName: "fake-plugin/success-and-timeout-device-name") on node "mynodename" DevicePath "/dev/sdb"
I0716 19:27:14.114458  231841 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") pod "pod1" (UID: "pod1uid") 
I0716 19:27:14.114566  231841 reconciler.go:157] Reconciler: start to sync state
E0716 19:27:14.114628  231841 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0716 19:27:14.115173  231841 operation_generator.go:1357] Controller attach succeeded for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") pod "pod1" (UID: "pod1uid") device path: "fake/path"
I0716 19:27:14.115754  231841 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "fake/path"
I0716 19:27:14.115885  231841 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:27:14.252891  231841 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:27:14.253065  231841 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0716 19:27:14.253632  231841 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-failed-mount-device-name podName: nodeName:}" failed. No retries permitted until 2021-07-16 19:27:14.753348121 +0000 UTC m=+23.694792655 (durationBeforeRetry 500ms). Error: "MapVolume.SetUpDevice failed for volume \"success-and-failed-mount-device-name\" (UniqueName: \"fake-plugin/success-and-failed-mount-device-name\") pod \"pod1\" (UID: \"pod1uid\") : error mapping disk: success-and-failed-mount-device-name"
I0716 19:27:14.758789  231841 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:27:14.822487  231841 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0716 19:27:14.823022  231841 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-failed-mount-device-name podName: nodeName:}" failed. No retries permitted until 2021-07-16 19:27:15.822752309 +0000 UTC m=+24.764196855 (durationBeforeRetry 1s). Error: "MapVolume.SetUpDevice failed for volume \"success-and-failed-mount-device-name\" (UniqueName: \"fake-plugin/success-and-failed-mount-device-name\") pod \"pod1\" (UID: \"pod1uid\") : error mapping disk: success-and-failed-mount-device-name"
I0716 19:27:15.824769  231841 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:27:15.827004  231841 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0716 19:27:15.830968  231841 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-failed-mount-device-name podName: nodeName:}" failed. No retries permitted until 2021-07-16 19:27:17.82852714 +0000 UTC m=+26.769971674 (durationBeforeRetry 2s). Error: "MapVolume.SetUpDevice failed for volume \"success-and-failed-mount-device-name\" (UniqueName: \"fake-plugin/success-and-failed-mount-device-name\") pod \"pod1\" (UID: \"pod1uid\") : error mapping disk: success-and-failed-mount-device-name"
I0716 19:27:17.828867  231841 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:27:17.829127  231841 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0716 19:27:17.829554  231841 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-failed-mount-device-name podName: nodeName:}" failed. No retries permitted until 2021-07-16 19:27:21.829350064 +0000 UTC m=+30.770794597 (durationBeforeRetry 4s). Error: "MapVolume.SetUpDevice failed for volume \"success-and-failed-mount-device-name\" (UniqueName: \"fake-plugin/success-and-failed-mount-device-name\") pod \"pod1\" (UID: \"pod1uid\") : error mapping disk: success-and-failed-mount-device-name"
I0716 19:27:21.830124  231841 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:27:21.830315  231841 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0716 19:27:21.830748  231841 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-failed-mount-device-name podName: nodeName:}" failed. No retries permitted until 2021-07-16 19:27:29.830524526 +0000 UTC m=+38.771969073 (durationBeforeRetry 8s). Error: "MapVolume.SetUpDevice failed for volume \"success-and-failed-mount-device-name\" (UniqueName: \"fake-plugin/success-and-failed-mount-device-name\") pod \"pod1\" (UID: \"pod1uid\") : error mapping disk: success-and-failed-mount-device-name"
I0716 19:27:24.279977  231841 reconciler.go:196] operationExecutor.UnmountVolume started for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") pod "pod1uid" (UID: "pod1uid") 
I0716 19:27:24.279495  231841 operation_generator.go:1170] UnmapVolume succeeded for volume "fake-plugin/success-and-failed-mount-device-name" (OuterVolumeSpecName: "success-and-failed-mount-device-name") pod "pod1uid" (UID: "pod1uid"). InnerVolumeSpecName "success-and-failed-mount-device-name". PluginName "fake-plugin", VolumeGidValue ""
I0716 19:27:24.281395  231841 reconciler.go:312] operationExecutor.UnmountDevice started for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") on node "mynodename" 
I0716 19:27:24.281727  231841 operation_generator.go:1282] UnmapDevice succeeded for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") on node "mynodename" 
I0716 19:27:24.282920  231841 reconciler.go:319] Volume detached for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") on node "mynodename" DevicePath "/dev/sdb"
I0716 19:27:24.379451  231841 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "timeout-mount-device-volume" (UniqueName: "fake-plugin/timeout-mount-device-volume") pod "pod1" (UID: "pod1uid") 
I0716 19:27:24.379537  231841 reconciler.go:157] Reconciler: start to sync state
E0716 19:27:24.379598  231841 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0716 19:27:24.379832  231841 operation_generator.go:1357] Controller attach succeeded for volume "timeout-mount-device-volume" (UniqueName: "fake-plugin/timeout-mount-device-volume") pod "pod1" (UID: "pod1uid") device path: "fake/path"
I0716 19:27:24.380553  231841 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "timeout-mount-device-volume" (UniqueName: "fake-plugin/timeout-mount-device-volume") pod "pod1" (UID: "pod1uid") DevicePath "fake/path"
I0716 19:27:24.380680  231841 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "timeout-mount-device-volume" (UniqueName: "fake-plugin/timeout-mount-device-volume") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0716 19:27:24.381111  231841 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/timeout-mount-device-volume podName: nodeName:}" failed. No retries permitted until 2021-07-16 19:27:24.88092774 +0000 UTC m=+33.822372272 (durationBeforeRetry 500ms). Error: "MountVolume.MountDevice failed for volume \"timeout-mount-device-volume\" (UniqueName: \"fake-plugin/timeout-mount-device-volume\") pod \"pod1\" (UID: \"pod1uid\") : mount failed"
I0716 19:27:24.485465  231841 reconciler.go:312] operationExecutor.UnmountDevice started for volume "timeout-mount-device-volume" (UniqueName: "fake-plugin/timeout-mount-device-volume") on node "mynodename" 
I0716 19:27:24.485730  231841 operation_generator.go:891] UnmountDevice succeeded for volume "timeout-mount-device-volume" %!(EXTRA string=UnmountDevice succeeded for volume "timeout-mount-device-volume" (UniqueName: "fake-plugin/timeout-mount-device-volume") on node "mynodename" )
I0716 19:27:24.485969  231841 reconciler.go:319] Volume detached for volume "timeout-mount-device-volume" (UniqueName: "fake-plugin/timeout-mount-device-volume") on node "mynodename" DevicePath "/dev/sdb"
I0716 19:27:24.623374  231841 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "fail-mount-device-volume-name" (UniqueName: "fake-plugin/fail-mount-device-volume-name") pod "pod1" (UID: "pod1uid") 
I0716 19:27:24.623448  231841 reconciler.go:157] Reconciler: start to sync state
E0716 19:27:24.623507  231841 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0716 19:27:24.623875  231841 operation_generator.go:1357] Controller attach succeeded for volume "fail-mount-device-volume-name" (UniqueName: "fake-plugin/fail-mount-device-volume-name") pod "pod1" (UID: "pod1uid") device path: "fake/path"
I0716 19:27:24.624463  231841 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "fail-mount-device-volume-name" (UniqueName: "fake-plugin/fail-mount-device-volume-name") pod "pod1" (UID: "pod1uid") DevicePath "fake/path"
I0716 19:27:24.624588  231841 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "fail-mount-device-volume-name" (UniqueName: "fake-plugin/fail-mount-device-volume-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0716 19:27:24.625033  231841 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/fail-mount-device-volume-name podName: nodeName:}" failed. No retries permitted until 2021-07-16 19:27:25.124836214 +0000 UTC m=+34.066280756 (durationBeforeRetry 500ms). Error: "MountVolume.MountDevice failed for volume \"fail-mount-device-volume-name\" (UniqueName: \"fake-plugin/fail-mount-device-volume-name\") pod \"pod1\" (UID: \"pod1uid\") : error mounting disk: /dev/sdb"
I0716 19:27:24.726490  231841 reconciler.go:319] Volume detached for volume "fail-mount-device-volume-name" (UniqueName: "fake-plugin/fail-mount-device-volume-name") on node "mynodename" DevicePath "fake/path"
I0716 19:27:24.827556  231841 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "timeout-and-fail-mount-device-name" (UniqueName: "fake-plugin/timeout-and-fail-mount-device-name") pod "pod1" (UID: "pod1uid") 
I0716 19:27:24.827636  231841 reconciler.go:157] Reconciler: start to sync state
E0716 19:27:24.827694  231841 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0716 19:27:24.828504  231841 operation_generator.go:1357] Controller attach succeeded for volume "timeout-and-fail-mount-device-name" (UniqueName: "fake-plugin/timeout-and-fail-mount-device-name") pod "pod1" (UID: "pod1uid") device path: "fake/path"
I0716 19:27:24.829437  231841 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "timeout-and-fail-mount-device-name" (UniqueName: "fake-plugin/timeout-and-fail-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "fake/path"
I0716 19:27:24.829586  231841 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "timeout-and-fail-mount-device-name" (UniqueName: "fake-plugin/timeout-and-fail-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0716 19:27:24.829989  231841 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/timeout-and-fail-mount-device-name podName: nodeName:}" failed. No retries permitted until 2021-07-16 19:27:25.329791345 +0000 UTC m=+34.271235889 (durationBeforeRetry 500ms). Error: "MountVolume.MountDevice failed for volume \"timeout-and-fail-mount-device-name\" (UniqueName: \"fake-plugin/timeout-and-fail-mount-device-name\") pod \"pod1\" (UID: \"pod1uid\") : timed out mounting error"
I0716 19:27:25.330382  231841 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "timeout-and-fail-mount-device-name" (UniqueName: "fake-plugin/timeout-and-fail-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:27:25.330561  231841 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "timeout-and-fail-mount-device-name" (UniqueName: "fake-plugin/timeout-and-fail-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0716 19:27:25.330982  231841 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/timeout-and-fail-mount-device-name podName: nodeName:}" failed. No retries permitted until 2021-07-16 19:27:26.33078591 +0000 UTC m=+35.272230478 (durationBeforeRetry 1s). Error: "MountVolume.MountDevice failed for volume \"timeout-and-fail-mount-device-name\" (UniqueName: \"fake-plugin/timeout-and-fail-mount-device-name\") pod \"pod1\" (UID: \"pod1uid\") : error mounting disk: /dev/sdb"
I0716 19:27:26.337516  231841 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "timeout-and-fail-mount-device-name" (UniqueName: "fake-plugin/timeout-and-fail-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:27:26.337683  231841 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "timeout-and-fail-mount-device-name" (UniqueName: "fake-plugin/timeout-and-fail-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0716 19:27:26.338088  231841 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/timeout-and-fail-mount-device-name podName: nodeName:}" failed. No retries permitted until 2021-07-16 19:27:28.337897869 +0000 UTC m=+37.279342444 (durationBeforeRetry 2s). Error: "MountVolume.MountDevice failed for volume \"timeout-and-fail-mount-device-name\" (UniqueName: \"fake-plugin/timeout-and-fail-mount-device-name\") pod \"pod1\" (UID: \"pod1uid\") : error mounting disk: /dev/sdb"
I0716 19:27:28.387535  231841 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "timeout-and-fail-mount-device-name" (UniqueName: "fake-plugin/timeout-and-fail-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:27:28.387711  231841 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "timeout-and-fail-mount-device-name" (UniqueName: "fake-plugin/timeout-and-fail-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0716 19:27:28.388435  231841 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/timeout-and-fail-mount-device-name podName: nodeName:}" failed. No retries permitted until 2021-07-16 19:27:32.387935071 +0000 UTC m=+41.329379608 (durationBeforeRetry 4s). Error: "MountVolume.MountDevice failed for volume \"timeout-and-fail-mount-device-name\" (UniqueName: \"fake-plugin/timeout-and-fail-mount-device-name\") pod \"pod1\" (UID: \"pod1uid\") : error mounting disk: /dev/sdb"
I0716 19:27:32.391019  231841 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "timeout-and-fail-mount-device-name" (UniqueName: "fake-plugin/timeout-and-fail-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:27:32.391210  231841 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "timeout-and-fail-mount-device-name" (UniqueName: "fake-plugin/timeout-and-fail-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0716 19:27:32.391650  231841 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/timeout-and-fail-mount-device-name podName: nodeName:}" failed. No retries permitted until 2021-07-16 19:27:40.391439586 +0000 UTC m=+49.332884126 (durationBeforeRetry 8s). Error: "MountVolume.MountDevice failed for volume \"timeout-and-fail-mount-device-name\" (UniqueName: \"fake-plugin/timeout-and-fail-mount-device-name\") pod \"pod1\" (UID: \"pod1uid\") : error mounting disk: /dev/sdb"
I0716 19:27:34.935098  231841 reconciler.go:319] Volume detached for volume "timeout-and-fail-mount-device-name" (UniqueName: "fake-plugin/timeout-and-fail-mount-device-name") on node "mynodename" DevicePath "/dev/sdb"
I0716 19:27:35.079392  231841 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "success-and-timeout-device-name" (UniqueName: "fake-plugin/success-and-timeout-device-name") pod "pod1" (UID: "pod1uid") 
I0716 19:27:35.079478  231841 reconciler.go:157] Reconciler: start to sync state
E0716 19:27:35.079537  231841 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0716 19:27:35.079920  231841 operation_generator.go:1357] Controller attach succeeded for volume "success-and-timeout-device-name" (UniqueName: "fake-plugin/success-and-timeout-device-name") pod "pod1" (UID: "pod1uid") device path: "fake/path"
I0716 19:27:35.113494  231841 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "success-and-timeout-device-name" (UniqueName: "fake-plugin/success-and-timeout-device-name") pod "pod1" (UID: "pod1uid") DevicePath "fake/path"
I0716 19:27:35.113701  231841 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "success-and-timeout-device-name" (UniqueName: "fake-plugin/success-and-timeout-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:27:35.113824  231841 operation_generator.go:594] MountVolume.MountDevice succeeded for volume "success-and-timeout-device-name" (UniqueName: "fake-plugin/success-and-timeout-device-name") pod "pod1" (UID: "pod1uid") device mount path ""
I0716 19:27:35.194835  231841 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "success-and-timeout-device-name" (UniqueName: "fake-plugin/success-and-timeout-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:27:35.194991  231841 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "success-and-timeout-device-name" (UniqueName: "fake-plugin/success-and-timeout-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:27:45.179619  231841 reconciler.go:196] operationExecutor.UnmountVolume started for volume "success-and-timeout-device-name" (UniqueName: "fake-plugin/success-and-timeout-device-name") pod "pod1uid" (UID: "pod1uid") 
I0716 19:27:45.179856  231841 operation_generator.go:797] UnmountVolume.TearDown succeeded for volume "fake-plugin/success-and-timeout-device-name" (OuterVolumeSpecName: "success-and-timeout-device-name") pod "pod1uid" (UID: "pod1uid"). InnerVolumeSpecName "success-and-timeout-device-name". PluginName "fake-plugin", VolumeGidValue ""
I0716 19:27:45.180427  231841 reconciler.go:312] operationExecutor.UnmountDevice started for volume "success-and-timeout-device-name" (UniqueName: "fake-plugin/success-and-timeout-device-name") on node "mynodename" 
I0716 19:27:45.180640  231841 operation_generator.go:891] UnmountDevice succeeded for volume "success-and-timeout-device-name" %!(EXTRA string=UnmountDevice succeeded for volume "success-and-timeout-device-name" (UniqueName: "fake-plugin/success-and-timeout-device-name") on node "mynodename" )
I0716 19:27:45.180896  231841 reconciler.go:319] Volume detached for volume "success-and-timeout-device-name" (UniqueName: "fake-plugin/success-and-timeout-device-name") on node "mynodename" DevicePath "/dev/sdb"
I0716 19:27:45.290139  231841 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") pod "pod1" (UID: "pod1uid") 
I0716 19:27:45.290254  231841 reconciler.go:157] Reconciler: start to sync state
E0716 19:27:45.290310  231841 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0716 19:27:45.290573  231841 operation_generator.go:1357] Controller attach succeeded for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") pod "pod1" (UID: "pod1uid") device path: "fake/path"
I0716 19:27:45.291193  231841 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "fake/path"
I0716 19:27:45.291333  231841 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:27:45.291453  231841 operation_generator.go:594] MountVolume.MountDevice succeeded for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") pod "pod1" (UID: "pod1uid") device mount path ""
I0716 19:27:45.393874  231841 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:27:45.394033  231841 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:27:55.435050  231841 reconciler.go:196] operationExecutor.UnmountVolume started for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") pod "pod1uid" (UID: "pod1uid") 
I0716 19:27:55.466446  231841 operation_generator.go:797] UnmountVolume.TearDown succeeded for volume "fake-plugin/success-and-failed-mount-device-name" (OuterVolumeSpecName: "success-and-failed-mount-device-name") pod "pod1uid" (UID: "pod1uid"). InnerVolumeSpecName "success-and-failed-mount-device-name". PluginName "fake-plugin", VolumeGidValue ""
I0716 19:27:55.467026  231841 reconciler.go:312] operationExecutor.UnmountDevice started for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") on node "mynodename" 
I0716 19:27:55.467237  231841 operation_generator.go:891] UnmountDevice succeeded for volume "success-and-failed-mount-device-name" %!(EXTRA string=UnmountDevice succeeded for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") on node "mynodename" )
I0716 19:27:55.467682  231841 reconciler.go:319] Volume detached for volume "success-and-failed-mount-device-name" (UniqueName: "fake-plugin/success-and-failed-mount-device-name") on node "mynodename" DevicePath "/dev/sdb"
I0716 19:27:55.536289  231841 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "timeout-setup-volume" (UniqueName: "fake-plugin/timeout-setup-volume") pod "pod1" (UID: "pod1uid") 
I0716 19:27:55.536373  231841 reconciler.go:157] Reconciler: start to sync state
E0716 19:27:55.536426  231841 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0716 19:27:55.536795  231841 operation_generator.go:1357] Controller attach succeeded for volume "timeout-setup-volume" (UniqueName: "fake-plugin/timeout-setup-volume") pod "pod1" (UID: "pod1uid") device path: "fake/path"
I0716 19:27:55.537426  231841 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "timeout-setup-volume" (UniqueName: "fake-plugin/timeout-setup-volume") pod "pod1" (UID: "pod1uid") DevicePath "fake/path"
I0716 19:27:55.537554  231841 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "timeout-setup-volume" (UniqueName: "fake-plugin/timeout-setup-volume") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0716 19:27:55.537974  231841 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/timeout-setup-volume podName: nodeName:}" failed. No retries permitted until 2021-07-16 19:27:56.037764556 +0000 UTC m=+64.979209096 (durationBeforeRetry 500ms). Error: "MapVolume.MapPodDevice failed for volume \"timeout-setup-volume\" (UniqueName: \"fake-plugin/timeout-setup-volume\") pod \"pod1\" (UID: \"pod1uid\") : time out on setup"
I0716 19:27:55.656541  231841 reconciler.go:196] operationExecutor.UnmountVolume started for volume "timeout-setup-volume" (UniqueName: "fake-plugin/timeout-setup-volume") pod "pod1uid" (UID: "pod1uid") 
I0716 19:27:55.656022  231841 operation_generator.go:1170] UnmapVolume succeeded for volume "fake-plugin/timeout-setup-volume" (OuterVolumeSpecName: "timeout-setup-volume") pod "pod1uid" (UID: "pod1uid"). InnerVolumeSpecName "timeout-setup-volume". PluginName "fake-plugin", VolumeGidValue ""
I0716 19:27:55.660590  231841 reconciler.go:196] operationExecutor.UnmountVolume started for volume "timeout-setup-volume" (UniqueName: "fake-plugin/timeout-setup-volume") pod "pod1uid" (UID: "pod1uid") 
I0716 19:27:55.659299  231841 operation_generator.go:1170] UnmapVolume succeeded for volume "fake-plugin/timeout-setup-volume" (OuterVolumeSpecName: "timeout-setup-volume") pod "pod1uid" (UID: "pod1uid"). InnerVolumeSpecName "timeout-setup-volume". PluginName "fake-plugin", VolumeGidValue ""
I0716 19:27:55.673750  231841 reconciler.go:312] operationExecutor.UnmountDevice started for volume "timeout-setup-volume" (UniqueName: "fake-plugin/timeout-setup-volume") on node "mynodename" 
I0716 19:27:55.677336  231841 operation_generator.go:1282] UnmapDevice succeeded for volume "timeout-setup-volume" (UniqueName: "fake-plugin/timeout-setup-volume") on node "mynodename" 
I0716 19:27:55.677707  231841 reconciler.go:319] Volume detached for volume "timeout-setup-volume" (UniqueName: "fake-plugin/timeout-setup-volume") on node "mynodename" DevicePath "/dev/sdb"
I0716 19:27:55.747808  231841 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "fail-setup-volume" (UniqueName: "fake-plugin/fail-setup-volume") pod "pod1" (UID: "pod1uid") 
I0716 19:27:55.747893  231841 reconciler.go:157] Reconciler: start to sync state
E0716 19:27:55.747950  231841 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0716 19:27:55.748231  231841 operation_generator.go:1357] Controller attach succeeded for volume "fail-setup-volume" (UniqueName: "fake-plugin/fail-setup-volume") pod "pod1" (UID: "pod1uid") device path: "fake/path"
I0716 19:27:55.748875  231841 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "fail-setup-volume" (UniqueName: "fake-plugin/fail-setup-volume") pod "pod1" (UID: "pod1uid") DevicePath "fake/path"
I0716 19:27:55.749009  231841 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "fail-setup-volume" (UniqueName: "fake-plugin/fail-setup-volume") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0716 19:27:55.749385  231841 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/fail-setup-volume podName: nodeName:}" failed. No retries permitted until 2021-07-16 19:27:56.24919172 +0000 UTC m=+65.190636264 (durationBeforeRetry 500ms). Error: "MapVolume.MapPodDevice failed for volume \"fail-setup-volume\" (UniqueName: \"fake-plugin/fail-setup-volume\") pod \"pod1\" (UID: \"pod1uid\") : mounting volume failed"
I0716 19:27:55.851952  231841 reconciler.go:312] operationExecutor.UnmountDevice started for volume "fail-setup-volume" (UniqueName: "fake-plugin/fail-setup-volume") on node "mynodename" 
I0716 19:27:55.857628  231841 operation_generator.go:1282] UnmapDevice succeeded for volume "fail-setup-volume" (UniqueName: "fake-plugin/fail-setup-volume") on node "mynodename" 
I0716 19:27:55.857916  231841 reconciler.go:319] Volume detached for volume "fail-setup-volume" (UniqueName: "fake-plugin/fail-setup-volume") on node "mynodename" DevicePath "/dev/sdb"
I0716 19:27:55.972704  231841 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") pod "pod1" (UID: "pod1uid") 
I0716 19:27:55.972784  231841 reconciler.go:157] Reconciler: start to sync state
E0716 19:27:55.972853  231841 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0716 19:27:55.973358  231841 operation_generator.go:1357] Controller attach succeeded for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") pod "pod1" (UID: "pod1uid") device path: "fake/path"
I0716 19:27:55.974059  231841 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") pod "pod1" (UID: "pod1uid") DevicePath "fake/path"
I0716 19:27:55.974206  231841 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0716 19:27:55.974718  231841 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/timeout-and-fail-setup-volume podName: nodeName:}" failed. No retries permitted until 2021-07-16 19:27:56.474515875 +0000 UTC m=+65.415960441 (durationBeforeRetry 500ms). Error: "MapVolume.MapPodDevice failed for volume \"timeout-and-fail-setup-volume\" (UniqueName: \"fake-plugin/timeout-and-fail-setup-volume\") pod \"pod1\" (UID: \"pod1uid\") : time out on setup"
I0716 19:27:56.489983  231841 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:27:56.490185  231841 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0716 19:27:56.490607  231841 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/timeout-and-fail-setup-volume podName: nodeName:}" failed. No retries permitted until 2021-07-16 19:27:57.490411038 +0000 UTC m=+66.431855574 (durationBeforeRetry 1s). Error: "MapVolume.MapPodDevice failed for volume \"timeout-and-fail-setup-volume\" (UniqueName: \"fake-plugin/timeout-and-fail-setup-volume\") pod \"pod1\" (UID: \"pod1uid\") : mounting volume failed"
I0716 19:27:57.491075  231841 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:27:57.491250  231841 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0716 19:27:57.491629  231841 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/timeout-and-fail-setup-volume podName: nodeName:}" failed. No retries permitted until 2021-07-16 19:27:59.491450497 +0000 UTC m=+68.432895030 (durationBeforeRetry 2s). Error: "MapVolume.MapPodDevice failed for volume \"timeout-and-fail-setup-volume\" (UniqueName: \"fake-plugin/timeout-and-fail-setup-volume\") pod \"pod1\" (UID: \"pod1uid\") : mounting volume failed"
I0716 19:27:59.492975  231841 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:27:59.493147  231841 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0716 19:27:59.493560  231841 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/timeout-and-fail-setup-volume podName: nodeName:}" failed. No retries permitted until 2021-07-16 19:28:03.493366927 +0000 UTC m=+72.434811479 (durationBeforeRetry 4s). Error: "MapVolume.MapPodDevice failed for volume \"timeout-and-fail-setup-volume\" (UniqueName: \"fake-plugin/timeout-and-fail-setup-volume\") pod \"pod1\" (UID: \"pod1uid\") : mounting volume failed"
I0716 19:28:03.512459  231841 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:28:03.512652  231841 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0716 19:28:03.513053  231841 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/timeout-and-fail-setup-volume podName: nodeName:}" failed. No retries permitted until 2021-07-16 19:28:11.512867315 +0000 UTC m=+80.454311858 (durationBeforeRetry 8s). Error: "MapVolume.MapPodDevice failed for volume \"timeout-and-fail-setup-volume\" (UniqueName: \"fake-plugin/timeout-and-fail-setup-volume\") pod \"pod1\" (UID: \"pod1uid\") : mounting volume failed"
I0716 19:28:06.066671  231841 reconciler.go:312] operationExecutor.UnmountDevice started for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") on node "mynodename" 
I0716 19:28:06.066866  231841 operation_generator.go:1282] UnmapDevice succeeded for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") on node "mynodename" 
I0716 19:28:06.067090  231841 reconciler.go:319] Volume detached for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") on node "mynodename" DevicePath "/dev/sdb"
I0716 19:28:06.167599  231841 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1" (UID: "pod1uid") 
I0716 19:28:06.167682  231841 reconciler.go:157] Reconciler: start to sync state
E0716 19:28:06.167737  231841 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0716 19:28:06.168211  231841 operation_generator.go:1357] Controller attach succeeded for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1" (UID: "pod1uid") device path: "fake/path"
I0716 19:28:06.168810  231841 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1" (UID: "pod1uid") DevicePath "fake/path"
I0716 19:28:06.168960  231841 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:28:06.276884  231841 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:28:06.278393  231841 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0716 19:28:06.283517  231841 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-timeout-setup-volume-name podName: nodeName:}" failed. No retries permitted until 2021-07-16 19:28:06.78039674 +0000 UTC m=+75.721841286 (durationBeforeRetry 500ms). Error: "MapVolume.MapPodDevice failed for volume \"success-and-timeout-setup-volume-name\" (UniqueName: \"fake-plugin/success-and-timeout-setup-volume-name\") pod \"pod1\" (UID: \"pod1uid\") : time out on setup"
I0716 19:28:06.785437  231841 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:28:06.785728  231841 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0716 19:28:06.786590  231841 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-timeout-setup-volume-name podName: nodeName:}" failed. No retries permitted until 2021-07-16 19:28:07.786030119 +0000 UTC m=+76.727474655 (durationBeforeRetry 1s). Error: "MapVolume.MapPodDevice failed for volume \"success-and-timeout-setup-volume-name\" (UniqueName: \"fake-plugin/success-and-timeout-setup-volume-name\") pod \"pod1\" (UID: \"pod1uid\") : time out on setup"
I0716 19:28:07.792906  231841 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:28:07.793456  231841 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0716 19:28:07.794409  231841 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-timeout-setup-volume-name podName: nodeName:}" failed. No retries permitted until 2021-07-16 19:28:09.793885052 +0000 UTC m=+78.735329586 (durationBeforeRetry 2s). Error: "MapVolume.MapPodDevice failed for volume \"success-and-timeout-setup-volume-name\" (UniqueName: \"fake-plugin/success-and-timeout-setup-volume-name\") pod \"pod1\" (UID: \"pod1uid\") : time out on setup"
I0716 19:28:09.794793  231841 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:28:09.794959  231841 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0716 19:28:09.795373  231841 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-timeout-setup-volume-name podName: nodeName:}" failed. No retries permitted until 2021-07-16 19:28:13.795184329 +0000 UTC m=+82.736628866 (durationBeforeRetry 4s). Error: "MapVolume.MapPodDevice failed for volume \"success-and-timeout-setup-volume-name\" (UniqueName: \"fake-plugin/success-and-timeout-setup-volume-name\") pod \"pod1\" (UID: \"pod1uid\") : time out on setup"
I0716 19:28:13.796030  231841 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:28:13.796202  231841 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0716 19:28:13.796900  231841 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-timeout-setup-volume-name podName: nodeName:}" failed. No retries permitted until 2021-07-16 19:28:21.796442714 +0000 UTC m=+90.737887248 (durationBeforeRetry 8s). Error: "MapVolume.MapPodDevice failed for volume \"success-and-timeout-setup-volume-name\" (UniqueName: \"fake-plugin/success-and-timeout-setup-volume-name\") pod \"pod1\" (UID: \"pod1uid\") : time out on setup"
I0716 19:28:16.290676  231841 reconciler.go:196] operationExecutor.UnmountVolume started for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1uid" (UID: "pod1uid") 
I0716 19:28:16.291282  231841 operation_generator.go:1170] UnmapVolume succeeded for volume "fake-plugin/success-and-timeout-setup-volume-name" (OuterVolumeSpecName: "success-and-timeout-setup-volume-name") pod "pod1uid" (UID: "pod1uid"). InnerVolumeSpecName "success-and-timeout-setup-volume-name". PluginName "fake-plugin", VolumeGidValue ""
I0716 19:28:16.291773  231841 reconciler.go:312] operationExecutor.UnmountDevice started for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") on node "mynodename" 
I0716 19:28:16.292062  231841 operation_generator.go:1282] UnmapDevice succeeded for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") on node "mynodename" 
I0716 19:28:16.292277  231841 reconciler.go:319] Volume detached for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") on node "mynodename" DevicePath "/dev/sdb"
I0716 19:28:16.420754  231841 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1" (UID: "pod1uid") 
I0716 19:28:16.420858  231841 reconciler.go:157] Reconciler: start to sync state
E0716 19:28:16.420922  231841 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0716 19:28:16.421057  231841 operation_generator.go:1357] Controller attach succeeded for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1" (UID: "pod1uid") device path: "fake/path"
I0716 19:28:16.424638  231841 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1" (UID: "pod1uid") DevicePath "fake/path"
I0716 19:28:16.429574  231841 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:28:16.435187  231841 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:28:16.435371  231841 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0716 19:28:16.435781  231841 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-failed-setup-device-name podName: nodeName:}" failed. No retries permitted until 2021-07-16 19:28:16.935573339 +0000 UTC m=+85.877017872 (durationBeforeRetry 500ms). Error: "MapVolume.MapPodDevice failed for volume \"success-and-failed-setup-device-name\" (UniqueName: \"fake-plugin/success-and-failed-setup-device-name\") pod \"pod1\" (UID: \"pod1uid\") : mounting volume failed"
I0716 19:28:16.936088  231841 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:28:16.936262  231841 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0716 19:28:16.936701  231841 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-failed-setup-device-name podName: nodeName:}" failed. No retries permitted until 2021-07-16 19:28:17.936494874 +0000 UTC m=+86.877939408 (durationBeforeRetry 1s). Error: "MapVolume.MapPodDevice failed for volume \"success-and-failed-setup-device-name\" (UniqueName: \"fake-plugin/success-and-failed-setup-device-name\") pod \"pod1\" (UID: \"pod1uid\") : mounting volume failed"
I0716 19:28:17.937221  231841 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:28:17.937396  231841 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0716 19:28:17.937803  231841 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-failed-setup-device-name podName: nodeName:}" failed. No retries permitted until 2021-07-16 19:28:19.937610945 +0000 UTC m=+88.879055476 (durationBeforeRetry 2s). Error: "MapVolume.MapPodDevice failed for volume \"success-and-failed-setup-device-name\" (UniqueName: \"fake-plugin/success-and-failed-setup-device-name\") pod \"pod1\" (UID: \"pod1uid\") : mounting volume failed"
I0716 19:28:19.943120  231841 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:28:19.943329  231841 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0716 19:28:19.943737  231841 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-failed-setup-device-name podName: nodeName:}" failed. No retries permitted until 2021-07-16 19:28:23.943556594 +0000 UTC m=+92.885001148 (durationBeforeRetry 4s). Error: "MapVolume.MapPodDevice failed for volume \"success-and-failed-setup-device-name\" (UniqueName: \"fake-plugin/success-and-failed-setup-device-name\") pod \"pod1\" (UID: \"pod1uid\") : mounting volume failed"
I0716 19:28:23.944039  231841 operation_generator.go:973] MapVolume.WaitForAttach entering for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:28:23.944206  231841 operation_generator.go:982] MapVolume.WaitForAttach succeeded for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0716 19:28:23.944679  231841 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-failed-setup-device-name podName: nodeName:}" failed. No retries permitted until 2021-07-16 19:28:31.944443785 +0000 UTC m=+100.885888316 (durationBeforeRetry 8s). Error: "MapVolume.MapPodDevice failed for volume \"success-and-failed-setup-device-name\" (UniqueName: \"fake-plugin/success-and-failed-setup-device-name\") pod \"pod1\" (UID: \"pod1uid\") : mounting volume failed"
I0716 19:28:26.643270  231841 reconciler.go:196] operationExecutor.UnmountVolume started for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1uid" (UID: "pod1uid") 
I0716 19:28:26.643461  231841 operation_generator.go:1170] UnmapVolume succeeded for volume "fake-plugin/success-and-failed-setup-device-name" (OuterVolumeSpecName: "success-and-failed-setup-device-name") pod "pod1uid" (UID: "pod1uid"). InnerVolumeSpecName "success-and-failed-setup-device-name". PluginName "fake-plugin", VolumeGidValue ""
I0716 19:28:26.645314  231841 reconciler.go:312] operationExecutor.UnmountDevice started for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") on node "mynodename" 
I0716 19:28:26.646179  231841 operation_generator.go:1282] UnmapDevice succeeded for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") on node "mynodename" 
I0716 19:28:26.646508  231841 reconciler.go:319] Volume detached for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") on node "mynodename" DevicePath "/dev/sdb"
I0716 19:28:26.651487  231841 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "timeout-setup-volume" (UniqueName: "fake-plugin/timeout-setup-volume") pod "pod1" (UID: "pod1uid") 
I0716 19:28:26.651568  231841 reconciler.go:157] Reconciler: start to sync state
E0716 19:28:26.651625  231841 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0716 19:28:26.651891  231841 operation_generator.go:1357] Controller attach succeeded for volume "timeout-setup-volume" (UniqueName: "fake-plugin/timeout-setup-volume") pod "pod1" (UID: "pod1uid") device path: "fake/path"
I0716 19:28:26.652666  231841 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "timeout-setup-volume" (UniqueName: "fake-plugin/timeout-setup-volume") pod "pod1" (UID: "pod1uid") DevicePath "fake/path"
I0716 19:28:26.652789  231841 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "timeout-setup-volume" (UniqueName: "fake-plugin/timeout-setup-volume") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:28:26.652888  231841 operation_generator.go:594] MountVolume.MountDevice succeeded for volume "timeout-setup-volume" (UniqueName: "fake-plugin/timeout-setup-volume") pod "pod1" (UID: "pod1uid") device mount path ""
E0716 19:28:26.653319  231841 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/timeout-setup-volume podName: nodeName:}" failed. No retries permitted until 2021-07-16 19:28:27.153148247 +0000 UTC m=+96.094592781 (durationBeforeRetry 500ms). Error: "MountVolume.SetUp failed for volume \"timeout-setup-volume\" (UniqueName: \"fake-plugin/timeout-setup-volume\") pod \"pod1\" (UID: \"pod1uid\") : time out on setup"
I0716 19:28:26.812097  231841 reconciler.go:196] operationExecutor.UnmountVolume started for volume "timeout-setup-volume" (UniqueName: "fake-plugin/timeout-setup-volume") pod "pod1uid" (UID: "pod1uid") 
I0716 19:28:26.812273  231841 operation_generator.go:797] UnmountVolume.TearDown succeeded for volume "fake-plugin/timeout-setup-volume" (OuterVolumeSpecName: "timeout-setup-volume") pod "pod1uid" (UID: "pod1uid"). InnerVolumeSpecName "timeout-setup-volume". PluginName "fake-plugin", VolumeGidValue ""
I0716 19:28:26.812766  231841 reconciler.go:312] operationExecutor.UnmountDevice started for volume "timeout-setup-volume" (UniqueName: "fake-plugin/timeout-setup-volume") on node "mynodename" 
I0716 19:28:26.812933  231841 operation_generator.go:891] UnmountDevice succeeded for volume "timeout-setup-volume" %!(EXTRA string=UnmountDevice succeeded for volume "timeout-setup-volume" (UniqueName: "fake-plugin/timeout-setup-volume") on node "mynodename" )
I0716 19:28:26.813171  231841 reconciler.go:319] Volume detached for volume "timeout-setup-volume" (UniqueName: "fake-plugin/timeout-setup-volume") on node "mynodename" DevicePath "/dev/sdb"
I0716 19:28:26.887855  231841 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "fail-setup-volume" (UniqueName: "fake-plugin/fail-setup-volume") pod "pod1" (UID: "pod1uid") 
I0716 19:28:26.888617  231841 operation_generator.go:1357] Controller attach succeeded for volume "fail-setup-volume" (UniqueName: "fake-plugin/fail-setup-volume") pod "pod1" (UID: "pod1uid") device path: "fake/path"
I0716 19:28:26.888752  231841 reconciler.go:157] Reconciler: start to sync state
E0716 19:28:26.889514  231841 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0716 19:28:26.891369  231841 reconciler.go:319] Volume detached for volume "fail-setup-volume" (UniqueName: "fake-plugin/fail-setup-volume") on node "mynodename" DevicePath "fake/path"
I0716 19:28:27.012983  231841 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") pod "pod1" (UID: "pod1uid") 
I0716 19:28:27.013086  231841 reconciler.go:157] Reconciler: start to sync state
E0716 19:28:27.013146  231841 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0716 19:28:27.013651  231841 operation_generator.go:1357] Controller attach succeeded for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") pod "pod1" (UID: "pod1uid") device path: "fake/path"
I0716 19:28:27.014369  231841 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") pod "pod1" (UID: "pod1uid") DevicePath "fake/path"
I0716 19:28:27.014504  231841 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:28:27.014636  231841 operation_generator.go:594] MountVolume.MountDevice succeeded for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") pod "pod1" (UID: "pod1uid") device mount path ""
E0716 19:28:27.015063  231841 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/timeout-and-fail-setup-volume podName: nodeName:}" failed. No retries permitted until 2021-07-16 19:28:27.514881654 +0000 UTC m=+96.456326185 (durationBeforeRetry 500ms). Error: "MountVolume.SetUp failed for volume \"timeout-and-fail-setup-volume\" (UniqueName: \"fake-plugin/timeout-and-fail-setup-volume\") pod \"pod1\" (UID: \"pod1uid\") : time out on setup"
I0716 19:28:27.520459  231841 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:28:27.520618  231841 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0716 19:28:27.521018  231841 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/timeout-and-fail-setup-volume podName: nodeName:}" failed. No retries permitted until 2021-07-16 19:28:28.520823009 +0000 UTC m=+97.462267550 (durationBeforeRetry 1s). Error: "MountVolume.SetUp failed for volume \"timeout-and-fail-setup-volume\" (UniqueName: \"fake-plugin/timeout-and-fail-setup-volume\") pod \"pod1\" (UID: \"pod1uid\") : mounting volume failed"
I0716 19:28:28.527490  231841 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:28:28.530119  231841 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0716 19:28:28.559156  231841 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/timeout-and-fail-setup-volume podName: nodeName:}" failed. No retries permitted until 2021-07-16 19:28:30.558886364 +0000 UTC m=+99.500330921 (durationBeforeRetry 2s). Error: "MountVolume.SetUp failed for volume \"timeout-and-fail-setup-volume\" (UniqueName: \"fake-plugin/timeout-and-fail-setup-volume\") pod \"pod1\" (UID: \"pod1uid\") : mounting volume failed"
I0716 19:28:30.559303  231841 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:28:30.559466  231841 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0716 19:28:30.559881  231841 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/timeout-and-fail-setup-volume podName: nodeName:}" failed. No retries permitted until 2021-07-16 19:28:34.55968325 +0000 UTC m=+103.501127782 (durationBeforeRetry 4s). Error: "MountVolume.SetUp failed for volume \"timeout-and-fail-setup-volume\" (UniqueName: \"fake-plugin/timeout-and-fail-setup-volume\") pod \"pod1\" (UID: \"pod1uid\") : mounting volume failed"
I0716 19:28:34.575064  231841 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:28:34.575225  231841 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0716 19:28:34.575634  231841 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/timeout-and-fail-setup-volume podName: nodeName:}" failed. No retries permitted until 2021-07-16 19:28:42.575450454 +0000 UTC m=+111.516894985 (durationBeforeRetry 8s). Error: "MountVolume.SetUp failed for volume \"timeout-and-fail-setup-volume\" (UniqueName: \"fake-plugin/timeout-and-fail-setup-volume\") pod \"pod1\" (UID: \"pod1uid\") : mounting volume failed"
I0716 19:28:37.142435  231841 reconciler.go:312] operationExecutor.UnmountDevice started for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") on node "mynodename" 
I0716 19:28:37.142807  231841 operation_generator.go:891] UnmountDevice succeeded for volume "timeout-and-fail-setup-volume" %!(EXTRA string=UnmountDevice succeeded for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") on node "mynodename" )
I0716 19:28:37.143062  231841 reconciler.go:319] Volume detached for volume "timeout-and-fail-setup-volume" (UniqueName: "fake-plugin/timeout-and-fail-setup-volume") on node "mynodename" DevicePath "/dev/sdb"
I0716 19:28:37.274037  231841 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1" (UID: "pod1uid") 
I0716 19:28:37.274131  231841 reconciler.go:157] Reconciler: start to sync state
E0716 19:28:37.274224  231841 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0716 19:28:37.274988  231841 operation_generator.go:1357] Controller attach succeeded for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1" (UID: "pod1uid") device path: "fake/path"
I0716 19:28:37.275640  231841 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1" (UID: "pod1uid") DevicePath "fake/path"
I0716 19:28:37.275778  231841 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:28:37.275909  231841 operation_generator.go:594] MountVolume.MountDevice succeeded for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1" (UID: "pod1uid") device mount path ""
I0716 19:28:37.382903  231841 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:28:37.383083  231841 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0716 19:28:37.383720  231841 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-timeout-setup-volume-name podName: nodeName:}" failed. No retries permitted until 2021-07-16 19:28:37.883466512 +0000 UTC m=+106.824911056 (durationBeforeRetry 500ms). Error: "MountVolume.SetUp failed for volume \"success-and-timeout-setup-volume-name\" (UniqueName: \"fake-plugin/success-and-timeout-setup-volume-name\") pod \"pod1\" (UID: \"pod1uid\") : time out on setup"
I0716 19:28:37.883931  231841 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:28:37.884120  231841 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0716 19:28:37.884610  231841 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-timeout-setup-volume-name podName: nodeName:}" failed. No retries permitted until 2021-07-16 19:28:38.88434348 +0000 UTC m=+107.825788022 (durationBeforeRetry 1s). Error: "MountVolume.SetUp failed for volume \"success-and-timeout-setup-volume-name\" (UniqueName: \"fake-plugin/success-and-timeout-setup-volume-name\") pod \"pod1\" (UID: \"pod1uid\") : time out on setup"
I0716 19:28:38.887070  231841 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:28:38.887238  231841 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0716 19:28:38.887653  231841 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-timeout-setup-volume-name podName: nodeName:}" failed. No retries permitted until 2021-07-16 19:28:40.887446208 +0000 UTC m=+109.828890739 (durationBeforeRetry 2s). Error: "MountVolume.SetUp failed for volume \"success-and-timeout-setup-volume-name\" (UniqueName: \"fake-plugin/success-and-timeout-setup-volume-name\") pod \"pod1\" (UID: \"pod1uid\") : time out on setup"
I0716 19:28:40.887980  231841 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:28:40.888163  231841 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0716 19:28:40.888587  231841 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-timeout-setup-volume-name podName: nodeName:}" failed. No retries permitted until 2021-07-16 19:28:44.888389097 +0000 UTC m=+113.829833631 (durationBeforeRetry 4s). Error: "MountVolume.SetUp failed for volume \"success-and-timeout-setup-volume-name\" (UniqueName: \"fake-plugin/success-and-timeout-setup-volume-name\") pod \"pod1\" (UID: \"pod1uid\") : time out on setup"
I0716 19:28:44.891250  231841 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:28:44.896011  231841 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0716 19:28:44.915996  231841 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-timeout-setup-volume-name podName: nodeName:}" failed. No retries permitted until 2021-07-16 19:28:52.915768713 +0000 UTC m=+121.857213258 (durationBeforeRetry 8s). Error: "MountVolume.SetUp failed for volume \"success-and-timeout-setup-volume-name\" (UniqueName: \"fake-plugin/success-and-timeout-setup-volume-name\") pod \"pod1\" (UID: \"pod1uid\") : time out on setup"
I0716 19:28:47.423302  231841 reconciler.go:196] operationExecutor.UnmountVolume started for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") pod "pod1uid" (UID: "pod1uid") 
I0716 19:28:47.423557  231841 operation_generator.go:797] UnmountVolume.TearDown succeeded for volume "fake-plugin/success-and-timeout-setup-volume-name" (OuterVolumeSpecName: "success-and-timeout-setup-volume-name") pod "pod1uid" (UID: "pod1uid"). InnerVolumeSpecName "success-and-timeout-setup-volume-name". PluginName "fake-plugin", VolumeGidValue ""
I0716 19:28:47.424077  231841 reconciler.go:312] operationExecutor.UnmountDevice started for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") on node "mynodename" 
I0716 19:28:47.424447  231841 operation_generator.go:891] UnmountDevice succeeded for volume "success-and-timeout-setup-volume-name" %!(EXTRA string=UnmountDevice succeeded for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") on node "mynodename" )
I0716 19:28:47.424670  231841 reconciler.go:319] Volume detached for volume "success-and-timeout-setup-volume-name" (UniqueName: "fake-plugin/success-and-timeout-setup-volume-name") on node "mynodename" DevicePath "/dev/sdb"
I0716 19:28:47.528917  231841 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1" (UID: "pod1uid") 
I0716 19:28:47.528996  231841 reconciler.go:157] Reconciler: start to sync state
E0716 19:28:47.529056  231841 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0716 19:28:47.529290  231841 operation_generator.go:1357] Controller attach succeeded for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1" (UID: "pod1uid") device path: "fake/path"
I0716 19:28:47.530060  231841 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1" (UID: "pod1uid") DevicePath "fake/path"
I0716 19:28:47.530229  231841 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:28:47.530366  231841 operation_generator.go:594] MountVolume.MountDevice succeeded for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1" (UID: "pod1uid") device mount path ""
I0716 19:28:47.631641  231841 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:28:47.631819  231841 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0716 19:28:47.632509  231841 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-failed-setup-device-name podName: nodeName:}" failed. No retries permitted until 2021-07-16 19:28:48.132047864 +0000 UTC m=+117.073492395 (durationBeforeRetry 500ms). Error: "MountVolume.SetUp failed for volume \"success-and-failed-setup-device-name\" (UniqueName: \"fake-plugin/success-and-failed-setup-device-name\") pod \"pod1\" (UID: \"pod1uid\") : mounting volume failed"
I0716 19:28:48.133059  231841 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:28:48.133281  231841 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0716 19:28:48.133833  231841 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-failed-setup-device-name podName: nodeName:}" failed. No retries permitted until 2021-07-16 19:28:49.133573398 +0000 UTC m=+118.075017937 (durationBeforeRetry 1s). Error: "MountVolume.SetUp failed for volume \"success-and-failed-setup-device-name\" (UniqueName: \"fake-plugin/success-and-failed-setup-device-name\") pod \"pod1\" (UID: \"pod1uid\") : mounting volume failed"
I0716 19:28:49.135607  231841 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:28:49.140086  231841 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0716 19:28:49.144440  231841 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-failed-setup-device-name podName: nodeName:}" failed. No retries permitted until 2021-07-16 19:28:51.141629313 +0000 UTC m=+120.083073847 (durationBeforeRetry 2s). Error: "MountVolume.SetUp failed for volume \"success-and-failed-setup-device-name\" (UniqueName: \"fake-plugin/success-and-failed-setup-device-name\") pod \"pod1\" (UID: \"pod1uid\") : mounting volume failed"
I0716 19:28:51.145026  231841 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:28:51.145236  231841 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0716 19:28:51.148000  231841 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-failed-setup-device-name podName: nodeName:}" failed. No retries permitted until 2021-07-16 19:28:55.146293524 +0000 UTC m=+124.087738063 (durationBeforeRetry 4s). Error: "MountVolume.SetUp failed for volume \"success-and-failed-setup-device-name\" (UniqueName: \"fake-plugin/success-and-failed-setup-device-name\") pod \"pod1\" (UID: \"pod1uid\") : mounting volume failed"
I0716 19:28:55.159332  231841 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:28:55.159553  231841 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
E0716 19:28:55.160065  231841 nestedpendingoperations.go:301] Operation for "{volumeName:fake-plugin/success-and-failed-setup-device-name podName: nodeName:}" failed. No retries permitted until 2021-07-16 19:29:03.159803288 +0000 UTC m=+132.101247834 (durationBeforeRetry 8s). Error: "MountVolume.SetUp failed for volume \"success-and-failed-setup-device-name\" (UniqueName: \"fake-plugin/success-and-failed-setup-device-name\") pod \"pod1\" (UID: \"pod1uid\") : mounting volume failed"
I0716 19:28:57.641643  231841 reconciler.go:196] operationExecutor.UnmountVolume started for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") pod "pod1uid" (UID: "pod1uid") 
I0716 19:28:57.642789  231841 operation_generator.go:797] UnmountVolume.TearDown succeeded for volume "fake-plugin/success-and-failed-setup-device-name" (OuterVolumeSpecName: "success-and-failed-setup-device-name") pod "pod1uid" (UID: "pod1uid"). InnerVolumeSpecName "success-and-failed-setup-device-name". PluginName "fake-plugin", VolumeGidValue ""
I0716 19:28:57.662555  231841 reconciler.go:312] operationExecutor.UnmountDevice started for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") on node "mynodename" 
I0716 19:28:57.662877  231841 operation_generator.go:891] UnmountDevice succeeded for volume "success-and-failed-setup-device-name" %!(EXTRA string=UnmountDevice succeeded for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") on node "mynodename" )
I0716 19:28:57.663192  231841 reconciler.go:319] Volume detached for volume "success-and-failed-setup-device-name" (UniqueName: "fake-plugin/success-and-failed-setup-device-name") on node "mynodename" DevicePath "/dev/sdb"
--- FAIL: Test_UncertainVolumeMountState (62.21s)
    --- FAIL: Test_UncertainVolumeMountState/failed_operation_should_result_in_not-mounted_volume_[Filesystem] (0.13s)
        reconciler_test.go:1523: Error verifying UnMountDeviceCallCount: Expected DeviceUnmount Call 1, got 0
I0716 19:28:57.742282  231841 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") 
I0716 19:28:57.742358  231841 reconciler.go:157] Reconciler: start to sync state
E0716 19:28:57.742415  231841 reconciler.go:389] Cannot get volumes from disk open fake-dir: no such file or directory
I0716 19:28:57.742823  231841 operation_generator.go:1357] Controller attach succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") device path: "/fake/path"
I0716 19:28:57.743473  231841 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") DevicePath "/fake/path"
I0716 19:28:57.743594  231841 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:28:57.743695  231841 operation_generator.go:594] MountVolume.MountDevice succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") device mount path ""
I0716 19:28:57.847859  231841 reconciler.go:196] operationExecutor.UnmountVolume started for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1uid" (UID: "pod1uid") 
I0716 19:28:57.848291  231841 operation_generator.go:797] UnmountVolume.TearDown succeeded for volume "fake-plugin/fake-device1" (OuterVolumeSpecName: "volume-name") pod "pod1uid" (UID: "pod1uid"). InnerVolumeSpecName "volume-name". PluginName "fake-plugin", VolumeGidValue ""
I0716 19:28:57.848954  231841 reconciler.go:312] operationExecutor.UnmountDevice started for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") on node "mynodename" 
I0716 19:28:57.849136  231841 reconciler_test.go:1798] UnmountDevice called
I0716 19:28:57.849327  231841 operation_generator.go:891] UnmountDevice succeeded for volume "volume-name" %!(EXTRA string=UnmountDevice succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") on node "mynodename" )
I0716 19:28:57.850064  231841 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:28:57.850217  231841 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") DevicePath "/dev/sdb"
I0716 19:28:57.850316  231841 operation_generator.go:594] MountVolume.MountDevice succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") pod "pod1" (UID: "pod1uid") device mount path ""
FAIL

				from junit_bazel.xml

Find pod1 mentions in log files | View test history on testgrid


Show 1936 Passed Tests