kubernetes-sigs/azuredisk-csi-driver PR #1691 pull-azuredisk-csi-driver-external-e2e-single-az-mainv2 #1615887395163475968
Recent runs || View in Spyglass
Recent runs || View in Spyglass
| PR | hccheng72: [V2] fix: only count replica attachments w/t deletion timestamp from cache client |
| Result | FAILURE |
| Tests | 1 failed / 13 succeeded |
| Started | |
| Elapsed | 1h23m |
| Revision | fc1d4af86f4d998851e82aba94b558ed70947b9b |
| Refs |
1691 |
| job-version | v1.27.0-alpha.0.1082+8d4a91198f3cf5 |
| kubetest-version | v20230117-50d6df3625 |
| revision | v1.27.0-alpha.0.1082+8d4a91198f3cf5 |
error during make e2e-test: exit status 2
from junit_runner.xml
Filter through log files | View test history on testgrid
kubetest Check APIReachability
kubetest Deferred TearDown
kubetest DumpClusterLogs
kubetest GetDeployer
kubetest IsUp
kubetest Prepare
kubetest TearDown
kubetest TearDown Previous
kubetest Timeout
kubetest Up
kubetest kubectl version
kubetest list nodes
kubetest test setup
... skipping 14 lines ... 2023/01/19 01:48:07 aksengine.go:783: CWD /home/prow/go/src/k8s.io/kubernetes 2023/01/19 01:48:07 aksengine.go:801: Attempting Docker login with azure cred. 2023/01/19 01:48:08 aksengine.go:810: Docker login success. 2023/01/19 01:48:08 aksengine.go:1141: Deleting resource group: kubetest-iei3lcrr. 2023/01/19 01:48:08 aksengine_helpers.go:425: downloading /root/tmp1374176058/kubernetes.json from https://raw.githubusercontent.com/kubernetes-sigs/azuredisk-csi-driver/main_v2/test/external-e2e/manifest/single-az.json 2023/01/19 01:48:08 util.go:70: curl https://raw.githubusercontent.com/kubernetes-sigs/azuredisk-csi-driver/main_v2/test/external-e2e/manifest/single-az.json 2023/01/19 01:48:15 aksengine_helpers.go:433: url=https://raw.githubusercontent.com/kubernetes-sigs/azuredisk-csi-driver/main_v2/test/external-e2e/manifest/single-az.json failed get /root/tmp1374176058/kubernetes.json: %!w(<nil>) 2023/01/19 01:48:15 aksengine_helpers.go:425: downloading /root/tmp1374176058/kubernetes.json from https://raw.githubusercontent.com/kubernetes-sigs/azuredisk-csi-driver/main_v2/test/external-e2e/manifest/single-az.json 2023/01/19 01:48:15 util.go:70: curl https://raw.githubusercontent.com/kubernetes-sigs/azuredisk-csi-driver/main_v2/test/external-e2e/manifest/single-az.json 2023/01/19 01:48:22 aksengine.go:645: downloading /root/aks-engine.tar.gz from https://github.com/Azure/aks-engine/releases/download/nightly/aks-engine-nightly-linux-amd64.tar.gz. 2023/01/19 01:48:22 util.go:70: curl https://github.com/Azure/aks-engine/releases/download/nightly/aks-engine-nightly-linux-amd64.tar.gz 2023/01/19 01:48:22 aksengine.go:672: Extracting tar file /root/aks-engine.tar.gz into directory /tmp . 2023/01/19 01:48:22 process.go:153: Running: tar -xzf /root/aks-engine.tar.gz --strip 1 -C /tmp ... skipping 220 lines ... 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 11345 100 11345 0 0 120k 0 --:--:-- --:--:-- --:--:-- 119k Downloading https://get.helm.sh/helm-v3.11.0-linux-amd64.tar.gz Verifying checksum... Done. Preparing to install helm into /usr/local/bin helm installed into /usr/local/bin/helm docker pull k8sprow.azurecr.io/azuredisk-csi:latest-v2-ec131e57ca07be2e17dba65bd8a030b3e7a9034d || make container-all push-manifest Error response from daemon: manifest for k8sprow.azurecr.io/azuredisk-csi:latest-v2-ec131e57ca07be2e17dba65bd8a030b3e7a9034d not found: manifest unknown: manifest tagged by "latest-v2-ec131e57ca07be2e17dba65bd8a030b3e7a9034d" is not found make[2]: Entering directory '/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver' CGO_ENABLED=0 GOOS=windows go build -a -ldflags "-X sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.driverVersion=latest-v2-ec131e57ca07be2e17dba65bd8a030b3e7a9034d -X sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.gitCommit=ec131e57ca07be2e17dba65bd8a030b3e7a9034d -X sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.buildDate=2023-01-19T01:53:27Z -extldflags "-static"" -tags azurediskv2 -mod vendor -o _output/amd64/azurediskpluginv2.exe ./pkg/azurediskplugin docker buildx rm container-builder || true ERROR: no builder "container-builder" found docker buildx create --use --name=container-builder container-builder # enable qemu for arm64 build # https://github.com/docker/buildx/issues/464#issuecomment-741507760 docker run --privileged --rm tonistiigi/binfmt --uninstall qemu-aarch64 Unable to find image 'tonistiigi/binfmt:latest' locally ... skipping 1492 lines ... type: string type: object oneOf: - required: ["persistentVolumeClaimName"] - required: ["volumeSnapshotContentName"] volumeSnapshotClassName: description: 'VolumeSnapshotClassName is the name of the VolumeSnapshotClass requested by the VolumeSnapshot. VolumeSnapshotClassName may be left nil to indicate that the default SnapshotClass should be used. A given cluster may have multiple default Volume SnapshotClasses: one default per CSI Driver. If a VolumeSnapshot does not specify a SnapshotClass, VolumeSnapshotSource will be checked to figure out what the associated CSI Driver is, and the default VolumeSnapshotClass associated with that CSI Driver will be used. If more than one VolumeSnapshotClass exist for a given CSI Driver and more than one have been marked as default, CreateSnapshot will fail and generate an event. Empty string is not allowed for this field.' type: string required: - source type: object status: description: status represents the current information of a snapshot. Consumers must verify binding between VolumeSnapshot and VolumeSnapshotContent objects is successful (by validating that both VolumeSnapshot and VolumeSnapshotContent point at each other) before using this object. ... skipping 2 lines ... description: 'boundVolumeSnapshotContentName is the name of the VolumeSnapshotContent object to which this VolumeSnapshot object intends to bind to. If not specified, it indicates that the VolumeSnapshot object has not been successfully bound to a VolumeSnapshotContent object yet. NOTE: To avoid possible security issues, consumers must verify binding between VolumeSnapshot and VolumeSnapshotContent objects is successful (by validating that both VolumeSnapshot and VolumeSnapshotContent point at each other) before using this object.' type: string creationTime: description: creationTime is the timestamp when the point-in-time snapshot is taken by the underlying storage system. In dynamic snapshot creation case, this field will be filled in by the snapshot controller with the "creation_time" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "creation_time" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. If not specified, it may indicate that the creation time of the snapshot is unknown. format: date-time type: string error: description: error is the last observed error during snapshot creation, if any. This field could be helpful to upper level controllers(i.e., application controller) to decide whether they should continue on waiting for the snapshot to be created based on the type of error reported. The snapshot controller will keep retrying when an error occurs during the snapshot creation. Upon success, this error field will be cleared. properties: message: description: 'message is a string detailing the encountered error during snapshot creation if specified. NOTE: message may be logged, and it should not contain sensitive information.' type: string time: description: time is the timestamp when the error was encountered. format: date-time type: string type: object readyToUse: description: readyToUse indicates if the snapshot is ready to be used to restore a volume. In dynamic snapshot creation case, this field will be filled in by the snapshot controller with the "ready_to_use" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "ready_to_use" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it, otherwise, this field will be set to "True". If not specified, it means the readiness of a snapshot is unknown. type: boolean restoreSize: type: string description: restoreSize represents the minimum size of volume required to create a volume from this snapshot. In dynamic snapshot creation case, this field will be filled in by the snapshot controller with the "size_bytes" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "size_bytes" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. When restoring a volume from this snapshot, the size of the volume MUST NOT be smaller than the restoreSize if it is specified, otherwise the restoration will fail. If not specified, it indicates that the size is unknown. pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ x-kubernetes-int-or-string: true type: object required: - spec type: object ... skipping 60 lines ... type: string volumeSnapshotContentName: description: volumeSnapshotContentName specifies the name of a pre-existing VolumeSnapshotContent object representing an existing volume snapshot. This field should be set if the snapshot already exists and only needs a representation in Kubernetes. This field is immutable. type: string type: object volumeSnapshotClassName: description: 'VolumeSnapshotClassName is the name of the VolumeSnapshotClass requested by the VolumeSnapshot. VolumeSnapshotClassName may be left nil to indicate that the default SnapshotClass should be used. A given cluster may have multiple default Volume SnapshotClasses: one default per CSI Driver. If a VolumeSnapshot does not specify a SnapshotClass, VolumeSnapshotSource will be checked to figure out what the associated CSI Driver is, and the default VolumeSnapshotClass associated with that CSI Driver will be used. If more than one VolumeSnapshotClass exist for a given CSI Driver and more than one have been marked as default, CreateSnapshot will fail and generate an event. Empty string is not allowed for this field.' type: string required: - source type: object status: description: status represents the current information of a snapshot. Consumers must verify binding between VolumeSnapshot and VolumeSnapshotContent objects is successful (by validating that both VolumeSnapshot and VolumeSnapshotContent point at each other) before using this object. ... skipping 2 lines ... description: 'boundVolumeSnapshotContentName is the name of the VolumeSnapshotContent object to which this VolumeSnapshot object intends to bind to. If not specified, it indicates that the VolumeSnapshot object has not been successfully bound to a VolumeSnapshotContent object yet. NOTE: To avoid possible security issues, consumers must verify binding between VolumeSnapshot and VolumeSnapshotContent objects is successful (by validating that both VolumeSnapshot and VolumeSnapshotContent point at each other) before using this object.' type: string creationTime: description: creationTime is the timestamp when the point-in-time snapshot is taken by the underlying storage system. In dynamic snapshot creation case, this field will be filled in by the snapshot controller with the "creation_time" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "creation_time" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. If not specified, it may indicate that the creation time of the snapshot is unknown. format: date-time type: string error: description: error is the last observed error during snapshot creation, if any. This field could be helpful to upper level controllers(i.e., application controller) to decide whether they should continue on waiting for the snapshot to be created based on the type of error reported. The snapshot controller will keep retrying when an error occurs during the snapshot creation. Upon success, this error field will be cleared. properties: message: description: 'message is a string detailing the encountered error during snapshot creation if specified. NOTE: message may be logged, and it should not contain sensitive information.' type: string time: description: time is the timestamp when the error was encountered. format: date-time type: string type: object readyToUse: description: readyToUse indicates if the snapshot is ready to be used to restore a volume. In dynamic snapshot creation case, this field will be filled in by the snapshot controller with the "ready_to_use" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "ready_to_use" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it, otherwise, this field will be set to "True". If not specified, it means the readiness of a snapshot is unknown. type: boolean restoreSize: type: string description: restoreSize represents the minimum size of volume required to create a volume from this snapshot. In dynamic snapshot creation case, this field will be filled in by the snapshot controller with the "size_bytes" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "size_bytes" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. When restoring a volume from this snapshot, the size of the volume MUST NOT be smaller than the restoreSize if it is specified, otherwise the restoration will fail. If not specified, it indicates that the size is unknown. pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ x-kubernetes-int-or-string: true type: object required: - spec type: object ... skipping 254 lines ... description: status represents the current information of a snapshot. properties: creationTime: description: creationTime is the timestamp when the point-in-time snapshot is taken by the underlying storage system. In dynamic snapshot creation case, this field will be filled in by the CSI snapshotter sidecar with the "creation_time" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "creation_time" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. If not specified, it indicates the creation time is unknown. The format of this field is a Unix nanoseconds time encoded as an int64. On Unix, the command `date +%s%N` returns the current time in nanoseconds since 1970-01-01 00:00:00 UTC. format: int64 type: integer error: description: error is the last observed error during snapshot creation, if any. Upon success after retry, this error field will be cleared. properties: message: description: 'message is a string detailing the encountered error during snapshot creation if specified. NOTE: message may be logged, and it should not contain sensitive information.' type: string time: description: time is the timestamp when the error was encountered. format: date-time type: string type: object readyToUse: description: readyToUse indicates if a snapshot is ready to be used to restore a volume. In dynamic snapshot creation case, this field will be filled in by the CSI snapshotter sidecar with the "ready_to_use" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "ready_to_use" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it, otherwise, this field will be set to "True". If not specified, it means the readiness of a snapshot is unknown. type: boolean restoreSize: description: restoreSize represents the complete size of the snapshot in bytes. In dynamic snapshot creation case, this field will be filled in by the CSI snapshotter sidecar with the "size_bytes" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "size_bytes" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. When restoring a volume from this snapshot, the size of the volume MUST NOT be smaller than the restoreSize if it is specified, otherwise the restoration will fail. If not specified, it indicates that the size is unknown. format: int64 minimum: 0 type: integer snapshotHandle: description: snapshotHandle is the CSI "snapshot_id" of a snapshot on the underlying storage system. If not specified, it indicates that dynamic snapshot creation has either failed or it is still in progress. type: string type: object required: - spec type: object served: true ... skipping 108 lines ... description: status represents the current information of a snapshot. properties: creationTime: description: creationTime is the timestamp when the point-in-time snapshot is taken by the underlying storage system. In dynamic snapshot creation case, this field will be filled in by the CSI snapshotter sidecar with the "creation_time" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "creation_time" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. If not specified, it indicates the creation time is unknown. The format of this field is a Unix nanoseconds time encoded as an int64. On Unix, the command `date +%s%N` returns the current time in nanoseconds since 1970-01-01 00:00:00 UTC. format: int64 type: integer error: description: error is the last observed error during snapshot creation, if any. Upon success after retry, this error field will be cleared. properties: message: description: 'message is a string detailing the encountered error during snapshot creation if specified. NOTE: message may be logged, and it should not contain sensitive information.' type: string time: description: time is the timestamp when the error was encountered. format: date-time type: string type: object readyToUse: description: readyToUse indicates if a snapshot is ready to be used to restore a volume. In dynamic snapshot creation case, this field will be filled in by the CSI snapshotter sidecar with the "ready_to_use" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "ready_to_use" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it, otherwise, this field will be set to "True". If not specified, it means the readiness of a snapshot is unknown. type: boolean restoreSize: description: restoreSize represents the complete size of the snapshot in bytes. In dynamic snapshot creation case, this field will be filled in by the CSI snapshotter sidecar with the "size_bytes" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "size_bytes" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. When restoring a volume from this snapshot, the size of the volume MUST NOT be smaller than the restoreSize if it is specified, otherwise the restoration will fail. If not specified, it indicates that the size is unknown. format: int64 minimum: 0 type: integer snapshotHandle: description: snapshotHandle is the CSI "snapshot_id" of a snapshot on the underlying storage system. If not specified, it indicates that dynamic snapshot creation has either failed or it is still in progress. type: string type: object required: - spec type: object served: true ... skipping 359 lines ... - volumeName - volume_context - volume_id type: object status: description: status represents the current state of AzVolumeAttachment. includes error, state, and attachment status Required properties: detail: description: Status summarizes the current attachment state of the volume attachment Nil Status indicates that the volume has not yet been attached to the node properties: ... skipping 7 lines ... role: description: The current attachment role. type: string required: - role type: object error: description: Error occurred during attach/detach of volume properties: code: type: string message: type: string parameters: ... skipping 90 lines ... - volumeName - volume_context - volume_id type: object status: description: status represents the current state of AzVolumeAttachment. includes error, state, and attachment status properties: annotation: additionalProperties: type: string description: Annotations contains additional resource information to guide driver actions ... skipping 13 lines ... role: description: The current attachment role. type: string required: - role type: object error: description: Error occurred during attach/detach of volume properties: code: type: string message: type: string parameters: ... skipping 169 lines ... - maxMountReplicaCount - volumeCapability - volumeName type: object status: description: status represents the current state of AzVolume. includes error, state, and volume status properties: detail: description: Current status detail of the AzVolume Nil detail indicates that the volume has not been created properties: accessible_topology: ... skipping 28 lines ... type: string required: - capacity_bytes - node_expansion_required - volume_id type: object error: description: Error occurred during creation/deletion of volume properties: code: type: string message: type: string parameters: ... skipping 154 lines ... - maxMountReplicaCount - volumeCapability - volumeName type: object status: description: status represents the current state of AzVolume. includes error, state, and volume status properties: annotation: additionalProperties: type: string description: Annotations contains additional resource information to guide driver actions ... skipping 34 lines ... type: string required: - capacity_bytes - node_expansion_required - volume_id type: object error: description: Error occurred during creation/deletion of volume properties: code: type: string message: type: string parameters: ... skipping 1069 lines ... image: "mcr.microsoft.com/oss/kubernetes-csi/csi-resizer:v1.5.0" args: - "-csi-address=$(ADDRESS)" - "-v=2" - "-leader-election" - "--leader-election-namespace=kube-system" - '-handle-volume-inuse-error=false' - '-feature-gates=RecoverVolumeExpansionFailure=true' - "-timeout=240s" env: - name: ADDRESS value: /csi/csi.sock volumeMounts: ... skipping 453 lines ... test/e2e/framework/framework.go:188 Jan 19 02:05:50.447: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready [1mSTEP[0m: Destroying namespace "volumelimits-1131" for this suite. [32m•[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (filesystem volmode)] volumeLimits should verify that all csinodes have volume limits","total":37,"completed":1,"skipped":90,"failed":0} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [BeforeEach] [Testpattern: Inline-volume (default fs)] subPath test/e2e/storage/framework/testsuite.go:51 Jan 19 02:05:50.699: INFO: Driver "test.csi.azure.com" does not support volume type "InlineVolume" - skipping ... skipping 62 lines ... Jan 19 02:05:49.663: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comccfn5] to have phase Bound Jan 19 02:05:49.775: INFO: PersistentVolumeClaim test.csi.azure.comccfn5 found but phase is Pending instead of Bound. Jan 19 02:05:51.886: INFO: PersistentVolumeClaim test.csi.azure.comccfn5 found but phase is Pending instead of Bound. Jan 19 02:05:53.995: INFO: PersistentVolumeClaim test.csi.azure.comccfn5 found and phase=Bound (4.331953263s) [1mSTEP[0m: Creating pod pod-subpath-test-dynamicpv-tv66 [1mSTEP[0m: Creating a pod to test subpath Jan 19 02:05:54.323: INFO: Waiting up to 5m0s for pod "pod-subpath-test-dynamicpv-tv66" in namespace "provisioning-7848" to be "Succeeded or Failed" Jan 19 02:05:54.431: INFO: Pod "pod-subpath-test-dynamicpv-tv66": Phase="Pending", Reason="", readiness=false. Elapsed: 108.129055ms Jan 19 02:05:56.540: INFO: Pod "pod-subpath-test-dynamicpv-tv66": Phase="Pending", Reason="", readiness=false. Elapsed: 2.2173201s Jan 19 02:05:58.650: INFO: Pod "pod-subpath-test-dynamicpv-tv66": Phase="Pending", Reason="", readiness=false. Elapsed: 4.326853594s Jan 19 02:06:00.758: INFO: Pod "pod-subpath-test-dynamicpv-tv66": Phase="Pending", Reason="", readiness=false. Elapsed: 6.4352322s Jan 19 02:06:02.867: INFO: Pod "pod-subpath-test-dynamicpv-tv66": Phase="Pending", Reason="", readiness=false. Elapsed: 8.543786102s Jan 19 02:06:04.976: INFO: Pod "pod-subpath-test-dynamicpv-tv66": Phase="Pending", Reason="", readiness=false. Elapsed: 10.653746096s Jan 19 02:06:07.087: INFO: Pod "pod-subpath-test-dynamicpv-tv66": Phase="Pending", Reason="", readiness=false. Elapsed: 12.764541392s Jan 19 02:06:09.196: INFO: Pod "pod-subpath-test-dynamicpv-tv66": Phase="Pending", Reason="", readiness=false. Elapsed: 14.873418702s Jan 19 02:06:11.307: INFO: Pod "pod-subpath-test-dynamicpv-tv66": Phase="Pending", Reason="", readiness=false. Elapsed: 16.984240719s Jan 19 02:06:13.417: INFO: Pod "pod-subpath-test-dynamicpv-tv66": Phase="Pending", Reason="", readiness=false. Elapsed: 19.094215804s Jan 19 02:06:15.526: INFO: Pod "pod-subpath-test-dynamicpv-tv66": Phase="Pending", Reason="", readiness=false. Elapsed: 21.203466129s Jan 19 02:06:17.638: INFO: Pod "pod-subpath-test-dynamicpv-tv66": Phase="Succeeded", Reason="", readiness=false. Elapsed: 23.314985134s [1mSTEP[0m: Saw pod success Jan 19 02:06:17.638: INFO: Pod "pod-subpath-test-dynamicpv-tv66" satisfied condition "Succeeded or Failed" Jan 19 02:06:17.746: INFO: Trying to get logs from node k8s-agentpool1-29517166-vmss000000 pod pod-subpath-test-dynamicpv-tv66 container test-container-subpath-dynamicpv-tv66: <nil> [1mSTEP[0m: delete the pod Jan 19 02:06:17.979: INFO: Waiting for pod pod-subpath-test-dynamicpv-tv66 to disappear Jan 19 02:06:18.087: INFO: Pod pod-subpath-test-dynamicpv-tv66 no longer exists [1mSTEP[0m: Deleting pod pod-subpath-test-dynamicpv-tv66 Jan 19 02:06:18.087: INFO: Deleting pod "pod-subpath-test-dynamicpv-tv66" in namespace "provisioning-7848" ... skipping 23 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (default fs)] subPath [90mtest/e2e/storage/framework/testsuite.go:50[0m should support existing single file [LinuxOnly] [90mtest/e2e/storage/testsuites/subpath.go:221[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should support existing single file [LinuxOnly]","total":32,"completed":1,"skipped":0,"failed":0} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Dynamic PV (default fs)] subPath[0m [1mshould fail if subpath with backstepping is outside the volume [Slow][LinuxOnly][0m [37mtest/e2e/storage/testsuites/subpath.go:280[0m [BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath test/e2e/storage/framework/testsuite.go:51 [BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath test/e2e/framework/framework.go:187 ... skipping 2 lines ... [1mSTEP[0m: Building a namespace api object, basename provisioning W0119 02:05:49.258141 26102 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ Jan 19 02:05:49.258: INFO: Found PodSecurityPolicies; testing pod creation to see if PodSecurityPolicy is enabled Jan 19 02:05:49.366: INFO: No PSP annotation exists on dry run pod; assuming PodSecurityPolicy is disabled [1mSTEP[0m: Waiting for a default service account to be provisioned in namespace [1mSTEP[0m: Waiting for kube-root-ca.crt to be provisioned in namespace [It] should fail if subpath with backstepping is outside the volume [Slow][LinuxOnly] test/e2e/storage/testsuites/subpath.go:280 Jan 19 02:05:49.792: INFO: Creating resource for dynamic PV Jan 19 02:05:49.792: INFO: Using claimSize:5Gi, test suite supported size:{ 1Mi}, driver(test.csi.azure.com) supported size:{ 1Mi} [1mSTEP[0m: creating a StorageClass provisioning-9892-e2e-sc8kknf [1mSTEP[0m: creating a claim Jan 19 02:05:49.901: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil Jan 19 02:05:50.011: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comvqbxt] to have phase Bound Jan 19 02:05:50.119: INFO: PersistentVolumeClaim test.csi.azure.comvqbxt found but phase is Pending instead of Bound. Jan 19 02:05:52.227: INFO: PersistentVolumeClaim test.csi.azure.comvqbxt found but phase is Pending instead of Bound. Jan 19 02:05:54.338: INFO: PersistentVolumeClaim test.csi.azure.comvqbxt found and phase=Bound (4.327095336s) [1mSTEP[0m: Creating pod pod-subpath-test-dynamicpv-9499 [1mSTEP[0m: Checking for subpath error in container status Jan 19 02:06:16.907: INFO: Deleting pod "pod-subpath-test-dynamicpv-9499" in namespace "provisioning-9892" Jan 19 02:06:17.016: INFO: Wait up to 5m0s for pod "pod-subpath-test-dynamicpv-9499" to be fully deleted [1mSTEP[0m: Deleting pod Jan 19 02:06:19.236: INFO: Deleting pod "pod-subpath-test-dynamicpv-9499" in namespace "provisioning-9892" [1mSTEP[0m: Deleting pvc Jan 19 02:06:19.344: INFO: Deleting PersistentVolumeClaim "test.csi.azure.comvqbxt" ... skipping 16 lines ... [32m• [SLOW TEST:72.150 seconds][0m External Storage [Driver: test.csi.azure.com] [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (default fs)] subPath [90mtest/e2e/storage/framework/testsuite.go:50[0m should fail if subpath with backstepping is outside the volume [Slow][LinuxOnly] [90mtest/e2e/storage/testsuites/subpath.go:280[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should fail if subpath with backstepping is outside the volume [Slow][LinuxOnly]","total":35,"completed":1,"skipped":13,"failed":0} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [BeforeEach] [Testpattern: Inline-volume (ext4)] volumes test/e2e/storage/framework/testsuite.go:51 Jan 19 02:07:00.996: INFO: Driver "test.csi.azure.com" does not support volume type "InlineVolume" - skipping ... skipping 137 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy [90mtest/e2e/storage/framework/testsuite.go:50[0m (OnRootMismatch)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed via chgrp in first pod, new pod with same fsgroup skips ownership changes to the volume contents [90mtest/e2e/storage/testsuites/fsgroupchangepolicy.go:216[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy (OnRootMismatch)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed via chgrp in first pod, new pod with same fsgroup skips ownership changes to the volume contents","total":33,"completed":1,"skipped":0,"failed":0} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral[0m [1mshould create read/write inline ephemeral volume[0m [37mtest/e2e/storage/testsuites/ephemeral.go:196[0m ... skipping 55 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral [90mtest/e2e/storage/framework/testsuite.go:50[0m should create read/write inline ephemeral volume [90mtest/e2e/storage/testsuites/ephemeral.go:196[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral should create read/write inline ephemeral volume","total":32,"completed":1,"skipped":64,"failed":0} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [BeforeEach] [Testpattern: Pre-provisioned PV (default fs)] subPath test/e2e/storage/framework/testsuite.go:51 Jan 19 02:07:58.875: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping ... skipping 3 lines ... [36m[1mS [SKIPPING] in Spec Setup (BeforeEach) [0.001 seconds][0m External Storage [Driver: test.csi.azure.com] [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Pre-provisioned PV (default fs)] subPath [90mtest/e2e/storage/framework/testsuite.go:50[0m [36m[1mshould fail if non-existent subpath is outside the volume [Slow][LinuxOnly] [BeforeEach][0m [90mtest/e2e/storage/testsuites/subpath.go:269[0m [36mDriver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping[0m test/e2e/storage/external/external.go:262 [90m------------------------------[0m ... skipping 105 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy [90mtest/e2e/storage/framework/testsuite.go:50[0m (OnRootMismatch)[LinuxOnly], pod created with an initial fsgroup, new pod fsgroup applied to volume contents [90mtest/e2e/storage/testsuites/fsgroupchangepolicy.go:216[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy (OnRootMismatch)[LinuxOnly], pod created with an initial fsgroup, new pod fsgroup applied to volume contents","total":30,"completed":1,"skipped":29,"failed":0} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Generic Ephemeral-volume (default fs) (immediate-binding)] ephemeral[0m [1mshould create read-only inline ephemeral volume[0m [37mtest/e2e/storage/testsuites/ephemeral.go:175[0m ... skipping 67 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Generic Ephemeral-volume (default fs) (immediate-binding)] ephemeral [90mtest/e2e/storage/framework/testsuite.go:50[0m should create read-only inline ephemeral volume [90mtest/e2e/storage/testsuites/ephemeral.go:175[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Generic Ephemeral-volume (default fs) (immediate-binding)] ephemeral should create read-only inline ephemeral volume","total":31,"completed":1,"skipped":6,"failed":0} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Dynamic PV (default fs)] subPath[0m [1mshould fail if subpath directory is outside the volume [Slow][LinuxOnly][0m [37mtest/e2e/storage/testsuites/subpath.go:242[0m [BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath test/e2e/storage/framework/testsuite.go:51 [BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath test/e2e/framework/framework.go:187 [1mSTEP[0m: Creating a kubernetes client Jan 19 02:07:01.071: INFO: >>> kubeConfig: /root/tmp1374176058/kubeconfig/kubeconfig.westeurope.json [1mSTEP[0m: Building a namespace api object, basename provisioning [1mSTEP[0m: Waiting for a default service account to be provisioned in namespace [1mSTEP[0m: Waiting for kube-root-ca.crt to be provisioned in namespace [It] should fail if subpath directory is outside the volume [Slow][LinuxOnly] test/e2e/storage/testsuites/subpath.go:242 Jan 19 02:07:01.821: INFO: Creating resource for dynamic PV Jan 19 02:07:01.821: INFO: Using claimSize:5Gi, test suite supported size:{ 1Mi}, driver(test.csi.azure.com) supported size:{ 1Mi} [1mSTEP[0m: creating a StorageClass provisioning-8282-e2e-sc8s9ld [1mSTEP[0m: creating a claim Jan 19 02:07:01.929: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil Jan 19 02:07:02.040: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.combdpkj] to have phase Bound Jan 19 02:07:02.147: INFO: PersistentVolumeClaim test.csi.azure.combdpkj found but phase is Pending instead of Bound. Jan 19 02:07:04.255: INFO: PersistentVolumeClaim test.csi.azure.combdpkj found but phase is Pending instead of Bound. Jan 19 02:07:06.365: INFO: PersistentVolumeClaim test.csi.azure.combdpkj found and phase=Bound (4.325534731s) [1mSTEP[0m: Creating pod pod-subpath-test-dynamicpv-tbn7 [1mSTEP[0m: Checking for subpath error in container status Jan 19 02:08:16.909: INFO: Deleting pod "pod-subpath-test-dynamicpv-tbn7" in namespace "provisioning-8282" Jan 19 02:08:17.020: INFO: Wait up to 5m0s for pod "pod-subpath-test-dynamicpv-tbn7" to be fully deleted [1mSTEP[0m: Deleting pod Jan 19 02:08:19.237: INFO: Deleting pod "pod-subpath-test-dynamicpv-tbn7" in namespace "provisioning-8282" [1mSTEP[0m: Deleting pvc Jan 19 02:08:19.344: INFO: Deleting PersistentVolumeClaim "test.csi.azure.combdpkj" ... skipping 22 lines ... [32m• [SLOW TEST:150.577 seconds][0m External Storage [Driver: test.csi.azure.com] [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (default fs)] subPath [90mtest/e2e/storage/framework/testsuite.go:50[0m should fail if subpath directory is outside the volume [Slow][LinuxOnly] [90mtest/e2e/storage/testsuites/subpath.go:242[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should fail if subpath directory is outside the volume [Slow][LinuxOnly]","total":35,"completed":2,"skipped":99,"failed":0} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy[0m [1m(OnRootMismatch)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed via chgrp in first pod, new pod with different fsgroup applied to the volume contents[0m [37mtest/e2e/storage/testsuites/fsgroupchangepolicy.go:216[0m ... skipping 119 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy [90mtest/e2e/storage/framework/testsuite.go:50[0m (OnRootMismatch)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed via chgrp in first pod, new pod with different fsgroup applied to the volume contents [90mtest/e2e/storage/testsuites/fsgroupchangepolicy.go:216[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy (OnRootMismatch)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed via chgrp in first pod, new pod with different fsgroup applied to the volume contents","total":37,"completed":2,"skipped":144,"failed":0} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [BeforeEach] [Testpattern: Inline-volume (default fs)] subPath test/e2e/storage/framework/testsuite.go:51 Jan 19 02:09:34.649: INFO: Driver "test.csi.azure.com" does not support volume type "InlineVolume" - skipping ... skipping 59 lines ... Jan 19 02:09:32.649: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.coms7ltw] to have phase Bound Jan 19 02:09:32.757: INFO: PersistentVolumeClaim test.csi.azure.coms7ltw found but phase is Pending instead of Bound. Jan 19 02:09:34.867: INFO: PersistentVolumeClaim test.csi.azure.coms7ltw found but phase is Pending instead of Bound. Jan 19 02:09:36.976: INFO: PersistentVolumeClaim test.csi.azure.coms7ltw found and phase=Bound (4.32647605s) [1mSTEP[0m: Creating pod pod-subpath-test-dynamicpv-bfr5 [1mSTEP[0m: Creating a pod to test multi_subpath Jan 19 02:09:37.302: INFO: Waiting up to 5m0s for pod "pod-subpath-test-dynamicpv-bfr5" in namespace "provisioning-4919" to be "Succeeded or Failed" Jan 19 02:09:37.410: INFO: Pod "pod-subpath-test-dynamicpv-bfr5": Phase="Pending", Reason="", readiness=false. Elapsed: 107.988578ms Jan 19 02:09:39.519: INFO: Pod "pod-subpath-test-dynamicpv-bfr5": Phase="Pending", Reason="", readiness=false. Elapsed: 2.216496335s Jan 19 02:09:41.627: INFO: Pod "pod-subpath-test-dynamicpv-bfr5": Phase="Pending", Reason="", readiness=false. Elapsed: 4.32528633s Jan 19 02:09:43.737: INFO: Pod "pod-subpath-test-dynamicpv-bfr5": Phase="Pending", Reason="", readiness=false. Elapsed: 6.434712708s Jan 19 02:09:45.847: INFO: Pod "pod-subpath-test-dynamicpv-bfr5": Phase="Pending", Reason="", readiness=false. Elapsed: 8.5450986s Jan 19 02:09:47.958: INFO: Pod "pod-subpath-test-dynamicpv-bfr5": Phase="Pending", Reason="", readiness=false. Elapsed: 10.655915971s ... skipping 2 lines ... Jan 19 02:09:54.289: INFO: Pod "pod-subpath-test-dynamicpv-bfr5": Phase="Pending", Reason="", readiness=false. Elapsed: 16.986722197s Jan 19 02:09:56.398: INFO: Pod "pod-subpath-test-dynamicpv-bfr5": Phase="Pending", Reason="", readiness=false. Elapsed: 19.096275199s Jan 19 02:09:58.507: INFO: Pod "pod-subpath-test-dynamicpv-bfr5": Phase="Pending", Reason="", readiness=false. Elapsed: 21.205066278s Jan 19 02:10:00.617: INFO: Pod "pod-subpath-test-dynamicpv-bfr5": Phase="Pending", Reason="", readiness=false. Elapsed: 23.314799559s Jan 19 02:10:02.726: INFO: Pod "pod-subpath-test-dynamicpv-bfr5": Phase="Succeeded", Reason="", readiness=false. Elapsed: 25.424249011s [1mSTEP[0m: Saw pod success Jan 19 02:10:02.727: INFO: Pod "pod-subpath-test-dynamicpv-bfr5" satisfied condition "Succeeded or Failed" Jan 19 02:10:02.835: INFO: Trying to get logs from node k8s-agentpool1-29517166-vmss000002 pod pod-subpath-test-dynamicpv-bfr5 container test-container-subpath-dynamicpv-bfr5: <nil> [1mSTEP[0m: delete the pod Jan 19 02:10:03.081: INFO: Waiting for pod pod-subpath-test-dynamicpv-bfr5 to disappear Jan 19 02:10:03.189: INFO: Pod pod-subpath-test-dynamicpv-bfr5 no longer exists [1mSTEP[0m: Deleting pod Jan 19 02:10:03.189: INFO: Deleting pod "pod-subpath-test-dynamicpv-bfr5" in namespace "provisioning-4919" ... skipping 27 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (default fs)] subPath [90mtest/e2e/storage/framework/testsuite.go:50[0m should support creating multiple subpath from same volumes [Slow] [90mtest/e2e/storage/testsuites/subpath.go:296[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should support creating multiple subpath from same volumes [Slow]","total":35,"completed":3,"skipped":117,"failed":0} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [BeforeEach] [Testpattern: Pre-provisioned PV (default fs)] subPath test/e2e/storage/framework/testsuite.go:51 Jan 19 02:11:15.686: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping ... skipping 3 lines ... [36m[1mS [SKIPPING] in Spec Setup (BeforeEach) [0.001 seconds][0m External Storage [Driver: test.csi.azure.com] [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Pre-provisioned PV (default fs)] subPath [90mtest/e2e/storage/framework/testsuite.go:50[0m [36m[1mshould fail if subpath file is outside the volume [Slow][LinuxOnly] [BeforeEach][0m [90mtest/e2e/storage/testsuites/subpath.go:258[0m [36mDriver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping[0m test/e2e/storage/external/external.go:262 [90m------------------------------[0m ... skipping 199 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow] [90mtest/e2e/storage/framework/testsuite.go:50[0m should access to two volumes with the same volume mode and retain data across pod recreation on different node [90mtest/e2e/storage/testsuites/multivolume.go:168[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow] should access to two volumes with the same volume mode and retain data across pod recreation on different node","total":32,"completed":2,"skipped":164,"failed":0} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [BeforeEach] [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow] test/e2e/storage/framework/testsuite.go:51 [BeforeEach] [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow] ... skipping 35 lines ... [36m[1mS [SKIPPING] in Spec Setup (BeforeEach) [0.002 seconds][0m External Storage [Driver: test.csi.azure.com] [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Inline-volume (default fs)] subPath [90mtest/e2e/storage/framework/testsuite.go:50[0m [36m[1mshould fail if subpath file is outside the volume [Slow][LinuxOnly] [BeforeEach][0m [90mtest/e2e/storage/testsuites/subpath.go:258[0m [36mDriver "test.csi.azure.com" does not support volume type "InlineVolume" - skipping[0m test/e2e/storage/external/external.go:262 [90m------------------------------[0m ... skipping 117 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] [90mtest/e2e/storage/framework/testsuite.go:50[0m should concurrently access the volume and its clone from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS] [90mtest/e2e/storage/testsuites/multivolume.go:378[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] should concurrently access the volume and its clone from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS]","total":33,"completed":2,"skipped":92,"failed":0} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow][0m [1mshould access to two volumes with different volume mode and retain data across pod recreation on the same node[0m [37mtest/e2e/storage/testsuites/multivolume.go:209[0m ... skipping 195 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow] [90mtest/e2e/storage/framework/testsuite.go:50[0m should access to two volumes with different volume mode and retain data across pod recreation on the same node [90mtest/e2e/storage/testsuites/multivolume.go:209[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow] should access to two volumes with different volume mode and retain data across pod recreation on the same node","total":31,"completed":2,"skipped":116,"failed":0} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [BeforeEach] [Testpattern: Dynamic PV (delayed binding)] topology test/e2e/storage/framework/testsuite.go:51 [BeforeEach] [Testpattern: Dynamic PV (delayed binding)] topology test/e2e/framework/framework.go:187 [1mSTEP[0m: Creating a kubernetes client Jan 19 02:11:57.662: INFO: >>> kubeConfig: /root/tmp1374176058/kubeconfig/kubeconfig.westeurope.json [1mSTEP[0m: Building a namespace api object, basename topology [1mSTEP[0m: Waiting for a default service account to be provisioned in namespace [1mSTEP[0m: Waiting for kube-root-ca.crt to be provisioned in namespace [It] should fail to schedule a pod which has topologies that conflict with AllowedTopologies test/e2e/storage/testsuites/topology.go:194 Jan 19 02:11:58.422: INFO: Driver didn't provide topology keys -- skipping [AfterEach] [Testpattern: Dynamic PV (delayed binding)] topology test/e2e/framework/framework.go:188 Jan 19 02:11:58.423: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready [1mSTEP[0m: Destroying namespace "topology-5766" for this suite. [36m[1mS [SKIPPING] [0.981 seconds][0m External Storage [Driver: test.csi.azure.com] [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (delayed binding)] topology [90mtest/e2e/storage/framework/testsuite.go:50[0m [36m[1mshould fail to schedule a pod which has topologies that conflict with AllowedTopologies [Measurement][0m [90mtest/e2e/storage/testsuites/topology.go:194[0m [36mDriver didn't provide topology keys -- skipping[0m test/e2e/storage/testsuites/topology.go:126 [90m------------------------------[0m ... skipping 21 lines ... Jan 19 02:09:35.625: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil Jan 19 02:09:35.734: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comb7stb] to have phase Bound Jan 19 02:09:35.842: INFO: PersistentVolumeClaim test.csi.azure.comb7stb found but phase is Pending instead of Bound. Jan 19 02:09:37.950: INFO: PersistentVolumeClaim test.csi.azure.comb7stb found but phase is Pending instead of Bound. Jan 19 02:09:40.060: INFO: PersistentVolumeClaim test.csi.azure.comb7stb found and phase=Bound (4.325151262s) [1mSTEP[0m: Creating pod to format volume volume-prep-provisioning-7403 Jan 19 02:09:40.387: INFO: Waiting up to 5m0s for pod "volume-prep-provisioning-7403" in namespace "provisioning-7403" to be "Succeeded or Failed" Jan 19 02:09:40.495: INFO: Pod "volume-prep-provisioning-7403": Phase="Pending", Reason="", readiness=false. Elapsed: 107.719028ms Jan 19 02:09:42.604: INFO: Pod "volume-prep-provisioning-7403": Phase="Pending", Reason="", readiness=false. Elapsed: 2.21694278s Jan 19 02:09:44.714: INFO: Pod "volume-prep-provisioning-7403": Phase="Pending", Reason="", readiness=false. Elapsed: 4.327405429s Jan 19 02:09:46.824: INFO: Pod "volume-prep-provisioning-7403": Phase="Pending", Reason="", readiness=false. Elapsed: 6.437401955s Jan 19 02:09:48.933: INFO: Pod "volume-prep-provisioning-7403": Phase="Pending", Reason="", readiness=false. Elapsed: 8.546207891s Jan 19 02:09:51.043: INFO: Pod "volume-prep-provisioning-7403": Phase="Pending", Reason="", readiness=false. Elapsed: 10.655535488s ... skipping 4 lines ... Jan 19 02:10:01.588: INFO: Pod "volume-prep-provisioning-7403": Phase="Pending", Reason="", readiness=false. Elapsed: 21.201074164s Jan 19 02:10:03.700: INFO: Pod "volume-prep-provisioning-7403": Phase="Pending", Reason="", readiness=false. Elapsed: 23.312555224s Jan 19 02:10:05.818: INFO: Pod "volume-prep-provisioning-7403": Phase="Pending", Reason="", readiness=false. Elapsed: 25.430478173s Jan 19 02:10:07.928: INFO: Pod "volume-prep-provisioning-7403": Phase="Pending", Reason="", readiness=false. Elapsed: 27.540733396s Jan 19 02:10:10.037: INFO: Pod "volume-prep-provisioning-7403": Phase="Succeeded", Reason="", readiness=false. Elapsed: 29.649457809s [1mSTEP[0m: Saw pod success Jan 19 02:10:10.037: INFO: Pod "volume-prep-provisioning-7403" satisfied condition "Succeeded or Failed" Jan 19 02:10:10.037: INFO: Deleting pod "volume-prep-provisioning-7403" in namespace "provisioning-7403" Jan 19 02:10:10.150: INFO: Wait up to 5m0s for pod "volume-prep-provisioning-7403" to be fully deleted [1mSTEP[0m: Creating pod pod-subpath-test-dynamicpv-hnhc [1mSTEP[0m: Checking for subpath error in container status Jan 19 02:11:32.587: INFO: Deleting pod "pod-subpath-test-dynamicpv-hnhc" in namespace "provisioning-7403" Jan 19 02:11:32.696: INFO: Wait up to 5m0s for pod "pod-subpath-test-dynamicpv-hnhc" to be fully deleted [1mSTEP[0m: Deleting pod Jan 19 02:11:34.913: INFO: Deleting pod "pod-subpath-test-dynamicpv-hnhc" in namespace "provisioning-7403" [1mSTEP[0m: Deleting pvc Jan 19 02:11:35.022: INFO: Deleting PersistentVolumeClaim "test.csi.azure.comb7stb" ... skipping 18 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (default fs)] subPath [90mtest/e2e/storage/framework/testsuite.go:50[0m should verify container cannot write to subpath readonly volumes [Slow] [90mtest/e2e/storage/testsuites/subpath.go:425[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should verify container cannot write to subpath readonly volumes [Slow]","total":37,"completed":3,"skipped":263,"failed":0} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Dynamic PV (block volmode)] volumes[0m [1mshould store data[0m [37mtest/e2e/storage/testsuites/volumes.go:161[0m ... skipping 113 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (block volmode)] volumes [90mtest/e2e/storage/framework/testsuite.go:50[0m should store data [90mtest/e2e/storage/testsuites/volumes.go:161[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] volumes should store data","total":30,"completed":2,"skipped":61,"failed":0} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [BeforeEach] [Testpattern: Pre-provisioned PV (default fs)] subPath test/e2e/storage/framework/testsuite.go:51 Jan 19 02:12:22.809: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping ... skipping 260 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow] [90mtest/e2e/storage/framework/testsuite.go:50[0m should concurrently access the volume and restored snapshot from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS] [90mtest/e2e/storage/testsuites/multivolume.go:323[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow] should concurrently access the volume and restored snapshot from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS]","total":32,"completed":2,"skipped":35,"failed":0} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [BeforeEach] [Testpattern: Inline-volume (ntfs)][Feature:Windows] volumes test/e2e/storage/framework/testsuite.go:51 Jan 19 02:12:24.071: INFO: Driver "test.csi.azure.com" does not support volume type "InlineVolume" - skipping ... skipping 38 lines ... Jan 19 02:11:16.853: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comw9p8p] to have phase Bound Jan 19 02:11:16.962: INFO: PersistentVolumeClaim test.csi.azure.comw9p8p found but phase is Pending instead of Bound. Jan 19 02:11:19.071: INFO: PersistentVolumeClaim test.csi.azure.comw9p8p found but phase is Pending instead of Bound. Jan 19 02:11:21.180: INFO: PersistentVolumeClaim test.csi.azure.comw9p8p found and phase=Bound (4.326117534s) [1mSTEP[0m: Creating pod pod-subpath-test-dynamicpv-smnh [1mSTEP[0m: Creating a pod to test subpath Jan 19 02:11:21.511: INFO: Waiting up to 5m0s for pod "pod-subpath-test-dynamicpv-smnh" in namespace "provisioning-170" to be "Succeeded or Failed" Jan 19 02:11:21.621: INFO: Pod "pod-subpath-test-dynamicpv-smnh": Phase="Pending", Reason="", readiness=false. Elapsed: 110.681865ms Jan 19 02:11:23.731: INFO: Pod "pod-subpath-test-dynamicpv-smnh": Phase="Pending", Reason="", readiness=false. Elapsed: 2.220035737s Jan 19 02:11:25.840: INFO: Pod "pod-subpath-test-dynamicpv-smnh": Phase="Pending", Reason="", readiness=false. Elapsed: 4.329792973s Jan 19 02:11:27.950: INFO: Pod "pod-subpath-test-dynamicpv-smnh": Phase="Pending", Reason="", readiness=false. Elapsed: 6.439504636s Jan 19 02:11:30.060: INFO: Pod "pod-subpath-test-dynamicpv-smnh": Phase="Pending", Reason="", readiness=false. Elapsed: 8.549360818s Jan 19 02:11:32.172: INFO: Pod "pod-subpath-test-dynamicpv-smnh": Phase="Pending", Reason="", readiness=false. Elapsed: 10.661036012s Jan 19 02:11:34.281: INFO: Pod "pod-subpath-test-dynamicpv-smnh": Phase="Pending", Reason="", readiness=false. Elapsed: 12.770558548s Jan 19 02:11:36.399: INFO: Pod "pod-subpath-test-dynamicpv-smnh": Phase="Pending", Reason="", readiness=false. Elapsed: 14.888140006s Jan 19 02:11:38.509: INFO: Pod "pod-subpath-test-dynamicpv-smnh": Phase="Pending", Reason="", readiness=false. Elapsed: 16.997980883s Jan 19 02:11:40.618: INFO: Pod "pod-subpath-test-dynamicpv-smnh": Phase="Pending", Reason="", readiness=false. Elapsed: 19.107831131s Jan 19 02:11:42.731: INFO: Pod "pod-subpath-test-dynamicpv-smnh": Phase="Pending", Reason="", readiness=false. Elapsed: 21.220027866s Jan 19 02:11:44.844: INFO: Pod "pod-subpath-test-dynamicpv-smnh": Phase="Succeeded", Reason="", readiness=false. Elapsed: 23.333058457s [1mSTEP[0m: Saw pod success Jan 19 02:11:44.844: INFO: Pod "pod-subpath-test-dynamicpv-smnh" satisfied condition "Succeeded or Failed" Jan 19 02:11:44.952: INFO: Trying to get logs from node k8s-agentpool1-29517166-vmss000000 pod pod-subpath-test-dynamicpv-smnh container test-container-volume-dynamicpv-smnh: <nil> [1mSTEP[0m: delete the pod Jan 19 02:11:45.212: INFO: Waiting for pod pod-subpath-test-dynamicpv-smnh to disappear Jan 19 02:11:45.320: INFO: Pod pod-subpath-test-dynamicpv-smnh no longer exists [1mSTEP[0m: Deleting pod pod-subpath-test-dynamicpv-smnh Jan 19 02:11:45.320: INFO: Deleting pod "pod-subpath-test-dynamicpv-smnh" in namespace "provisioning-170" ... skipping 29 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (default fs)] subPath [90mtest/e2e/storage/framework/testsuite.go:50[0m should support non-existent path [90mtest/e2e/storage/testsuites/subpath.go:196[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should support non-existent path","total":35,"completed":4,"skipped":251,"failed":0} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [BeforeEach] [Testpattern: Dynamic PV (ntfs)][Feature:Windows] subPath test/e2e/storage/framework/testsuite.go:51 Jan 19 02:12:57.847: INFO: Distro debian doesn't support ntfs -- skipping ... skipping 218 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy [90mtest/e2e/storage/framework/testsuite.go:50[0m (Always)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed via chgrp in first pod, new pod with different fsgroup applied to the volume contents [90mtest/e2e/storage/testsuites/fsgroupchangepolicy.go:216[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy (Always)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed via chgrp in first pod, new pod with different fsgroup applied to the volume contents","total":32,"completed":3,"skipped":227,"failed":0} [36mS[0m[36mS[0m [90m------------------------------[0m [BeforeEach] [Testpattern: Dynamic Snapshot (delete policy)] snapshottable[Feature:VolumeSnapshotDataSource] test/e2e/storage/framework/testsuite.go:51 [BeforeEach] [Testpattern: Dynamic Snapshot (delete policy)] snapshottable[Feature:VolumeSnapshotDataSource] ... skipping 259 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (ext4)] multiVolume [Slow] [90mtest/e2e/storage/framework/testsuite.go:50[0m should access to two volumes with different volume mode and retain data across pod recreation on the same node [90mtest/e2e/storage/testsuites/multivolume.go:209[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext4)] multiVolume [Slow] should access to two volumes with different volume mode and retain data across pod recreation on the same node","total":31,"completed":3,"skipped":191,"failed":0} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [BeforeEach] [Testpattern: Dynamic PV (ntfs)][Feature:Windows] subPath test/e2e/storage/framework/testsuite.go:51 Jan 19 02:14:26.479: INFO: Distro debian doesn't support ntfs -- skipping ... skipping 3 lines ... [36m[1mS [SKIPPING] in Spec Setup (BeforeEach) [0.001 seconds][0m External Storage [Driver: test.csi.azure.com] [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (ntfs)][Feature:Windows] subPath [90mtest/e2e/storage/framework/testsuite.go:50[0m [36m[1mshould fail if non-existent subpath is outside the volume [Slow][LinuxOnly] [BeforeEach][0m [90mtest/e2e/storage/testsuites/subpath.go:269[0m [36mDistro debian doesn't support ntfs -- skipping[0m test/e2e/storage/framework/testsuite.go:127 [90m------------------------------[0m ... skipping 148 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (default fs)] provisioning [90mtest/e2e/storage/framework/testsuite.go:50[0m should provision storage with pvc data source [90mtest/e2e/storage/testsuites/provisioning.go:421[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] provisioning should provision storage with pvc data source","total":37,"completed":4,"skipped":276,"failed":0} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [BeforeEach] [Testpattern: Pre-provisioned PV (block volmode)] volumeMode test/e2e/storage/framework/testsuite.go:51 Jan 19 02:14:41.342: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping ... skipping 77 lines ... [36m[1mS [SKIPPING] in Spec Setup (BeforeEach) [0.001 seconds][0m External Storage [Driver: test.csi.azure.com] [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Pre-provisioned PV (default fs)] subPath [90mtest/e2e/storage/framework/testsuite.go:50[0m [36m[1mshould fail if subpath with backstepping is outside the volume [Slow][LinuxOnly] [BeforeEach][0m [90mtest/e2e/storage/testsuites/subpath.go:280[0m [36mDriver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping[0m test/e2e/storage/external/external.go:262 [90m------------------------------[0m ... skipping 70 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (default fs)] subPath [90mtest/e2e/storage/framework/testsuite.go:50[0m should support restarting containers using file as subpath [Slow][LinuxOnly] [90mtest/e2e/storage/testsuites/subpath.go:333[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should support restarting containers using file as subpath [Slow][LinuxOnly]","total":33,"completed":3,"skipped":118,"failed":0} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [BeforeEach] [Testpattern: Inline-volume (xfs)][Slow] volumes test/e2e/storage/framework/testsuite.go:51 Jan 19 02:15:03.848: INFO: Driver "test.csi.azure.com" does not support volume type "InlineVolume" - skipping ... skipping 337 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow] [90mtest/e2e/storage/framework/testsuite.go:50[0m should access to two volumes with the same volume mode and retain data across pod recreation on different node [90mtest/e2e/storage/testsuites/multivolume.go:168[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow] should access to two volumes with the same volume mode and retain data across pod recreation on different node","total":32,"completed":3,"skipped":125,"failed":0} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [BeforeEach] [Testpattern: Inline-volume (default fs)] subPath test/e2e/storage/framework/testsuite.go:51 Jan 19 02:15:54.247: INFO: Driver "test.csi.azure.com" does not support volume type "InlineVolume" - skipping ... skipping 217 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (ext4)] multiVolume [Slow] [90mtest/e2e/storage/framework/testsuite.go:50[0m should access to two volumes with the same volume mode and retain data across pod recreation on different node [90mtest/e2e/storage/testsuites/multivolume.go:168[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext4)] multiVolume [Slow] should access to two volumes with the same volume mode and retain data across pod recreation on different node","total":30,"completed":3,"skipped":300,"failed":0} [36mS[0m[36mS[0m [90m------------------------------[0m [0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Dynamic PV (block volmode)] provisioning[0m [1mshould provision storage with pvc data source[0m [37mtest/e2e/storage/testsuites/provisioning.go:421[0m ... skipping 103 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (block volmode)] provisioning [90mtest/e2e/storage/framework/testsuite.go:50[0m should provision storage with pvc data source [90mtest/e2e/storage/testsuites/provisioning.go:421[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] provisioning should provision storage with pvc data source","total":32,"completed":4,"skipped":351,"failed":0} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [BeforeEach] [Testpattern: Dynamic PV (block volmode)] provisioning test/e2e/storage/framework/testsuite.go:51 [BeforeEach] [Testpattern: Dynamic PV (block volmode)] provisioning ... skipping 227 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow] [90mtest/e2e/storage/framework/testsuite.go:50[0m should concurrently access the single volume from pods on the same node [90mtest/e2e/storage/testsuites/multivolume.go:298[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow] should concurrently access the single volume from pods on the same node","total":31,"completed":4,"skipped":401,"failed":0} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [BeforeEach] [Testpattern: CSI Ephemeral-volume (default fs)] ephemeral test/e2e/storage/framework/testsuite.go:51 Jan 19 02:16:41.696: INFO: Driver "test.csi.azure.com" does not support volume type "CSIInlineVolume" - skipping ... skipping 118 lines ... Jan 19 02:14:43.379: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.combb82b] to have phase Bound Jan 19 02:14:43.486: INFO: PersistentVolumeClaim test.csi.azure.combb82b found but phase is Pending instead of Bound. Jan 19 02:14:45.594: INFO: PersistentVolumeClaim test.csi.azure.combb82b found but phase is Pending instead of Bound. Jan 19 02:14:47.703: INFO: PersistentVolumeClaim test.csi.azure.combb82b found and phase=Bound (4.324418431s) [1mSTEP[0m: Creating pod pod-subpath-test-dynamicpv-j68g [1mSTEP[0m: Creating a pod to test subpath Jan 19 02:14:48.027: INFO: Waiting up to 5m0s for pod "pod-subpath-test-dynamicpv-j68g" in namespace "provisioning-235" to be "Succeeded or Failed" Jan 19 02:14:48.134: INFO: Pod "pod-subpath-test-dynamicpv-j68g": Phase="Pending", Reason="", readiness=false. Elapsed: 107.198417ms Jan 19 02:14:50.242: INFO: Pod "pod-subpath-test-dynamicpv-j68g": Phase="Pending", Reason="", readiness=false. Elapsed: 2.21566081s Jan 19 02:14:52.350: INFO: Pod "pod-subpath-test-dynamicpv-j68g": Phase="Pending", Reason="", readiness=false. Elapsed: 4.322906246s Jan 19 02:14:54.457: INFO: Pod "pod-subpath-test-dynamicpv-j68g": Phase="Pending", Reason="", readiness=false. Elapsed: 6.430697177s Jan 19 02:14:56.567: INFO: Pod "pod-subpath-test-dynamicpv-j68g": Phase="Pending", Reason="", readiness=false. Elapsed: 8.539939665s Jan 19 02:14:58.675: INFO: Pod "pod-subpath-test-dynamicpv-j68g": Phase="Pending", Reason="", readiness=false. Elapsed: 10.647864448s ... skipping 27 lines ... Jan 19 02:15:57.748: INFO: Pod "pod-subpath-test-dynamicpv-j68g": Phase="Pending", Reason="", readiness=false. Elapsed: 1m9.72166927s Jan 19 02:15:59.858: INFO: Pod "pod-subpath-test-dynamicpv-j68g": Phase="Pending", Reason="", readiness=false. Elapsed: 1m11.831322414s Jan 19 02:16:01.968: INFO: Pod "pod-subpath-test-dynamicpv-j68g": Phase="Pending", Reason="", readiness=false. Elapsed: 1m13.941768263s Jan 19 02:16:04.078: INFO: Pod "pod-subpath-test-dynamicpv-j68g": Phase="Pending", Reason="", readiness=false. Elapsed: 1m16.051560127s Jan 19 02:16:06.189: INFO: Pod "pod-subpath-test-dynamicpv-j68g": Phase="Succeeded", Reason="", readiness=false. Elapsed: 1m18.16271459s [1mSTEP[0m: Saw pod success Jan 19 02:16:06.189: INFO: Pod "pod-subpath-test-dynamicpv-j68g" satisfied condition "Succeeded or Failed" Jan 19 02:16:06.302: INFO: Trying to get logs from node k8s-agentpool1-29517166-vmss000000 pod pod-subpath-test-dynamicpv-j68g container test-container-volume-dynamicpv-j68g: <nil> [1mSTEP[0m: delete the pod Jan 19 02:16:06.547: INFO: Waiting for pod pod-subpath-test-dynamicpv-j68g to disappear Jan 19 02:16:06.654: INFO: Pod pod-subpath-test-dynamicpv-j68g no longer exists [1mSTEP[0m: Deleting pod pod-subpath-test-dynamicpv-j68g Jan 19 02:16:06.654: INFO: Deleting pod "pod-subpath-test-dynamicpv-j68g" in namespace "provisioning-235" ... skipping 29 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (default fs)] subPath [90mtest/e2e/storage/framework/testsuite.go:50[0m should support existing directory [90mtest/e2e/storage/testsuites/subpath.go:207[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should support existing directory","total":37,"completed":5,"skipped":364,"failed":0} [36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Generic Ephemeral-volume (default fs) (immediate-binding)] ephemeral[0m [1mshould support two pods which have the same volume definition[0m [37mtest/e2e/storage/testsuites/ephemeral.go:216[0m ... skipping 70 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Generic Ephemeral-volume (default fs) (immediate-binding)] ephemeral [90mtest/e2e/storage/framework/testsuite.go:50[0m should support two pods which have the same volume definition [90mtest/e2e/storage/testsuites/ephemeral.go:216[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Generic Ephemeral-volume (default fs) (immediate-binding)] ephemeral should support two pods which have the same volume definition","total":35,"completed":5,"skipped":476,"failed":0} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [BeforeEach] [Testpattern: Dynamic PV (ntfs)][Feature:Windows] subPath test/e2e/storage/framework/testsuite.go:51 Jan 19 02:17:21.670: INFO: Distro debian doesn't support ntfs -- skipping ... skipping 166 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (ext4)] multiVolume [Slow] [90mtest/e2e/storage/framework/testsuite.go:50[0m should concurrently access the single read-only volume from pods on the same node [90mtest/e2e/storage/testsuites/multivolume.go:423[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext4)] multiVolume [Slow] should concurrently access the single read-only volume from pods on the same node","total":32,"completed":5,"skipped":568,"failed":0} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Dynamic PV (ext3)] volumes[0m [1mshould allow exec of files on the volume[0m [37mtest/e2e/storage/testsuites/volumes.go:198[0m ... skipping 17 lines ... Jan 19 02:15:55.257: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.combcptj] to have phase Bound Jan 19 02:15:55.364: INFO: PersistentVolumeClaim test.csi.azure.combcptj found but phase is Pending instead of Bound. Jan 19 02:15:57.471: INFO: PersistentVolumeClaim test.csi.azure.combcptj found but phase is Pending instead of Bound. Jan 19 02:15:59.579: INFO: PersistentVolumeClaim test.csi.azure.combcptj found and phase=Bound (4.32224409s) [1mSTEP[0m: Creating pod exec-volume-test-dynamicpv-7rhg [1mSTEP[0m: Creating a pod to test exec-volume-test Jan 19 02:15:59.902: INFO: Waiting up to 5m0s for pod "exec-volume-test-dynamicpv-7rhg" in namespace "volume-2250" to be "Succeeded or Failed" Jan 19 02:16:00.009: INFO: Pod "exec-volume-test-dynamicpv-7rhg": Phase="Pending", Reason="", readiness=false. Elapsed: 107.146925ms Jan 19 02:16:02.118: INFO: Pod "exec-volume-test-dynamicpv-7rhg": Phase="Pending", Reason="", readiness=false. Elapsed: 2.215330534s Jan 19 02:16:04.225: INFO: Pod "exec-volume-test-dynamicpv-7rhg": Phase="Pending", Reason="", readiness=false. Elapsed: 4.322417122s Jan 19 02:16:06.332: INFO: Pod "exec-volume-test-dynamicpv-7rhg": Phase="Pending", Reason="", readiness=false. Elapsed: 6.429635235s Jan 19 02:16:08.440: INFO: Pod "exec-volume-test-dynamicpv-7rhg": Phase="Pending", Reason="", readiness=false. Elapsed: 8.538074181s Jan 19 02:16:10.549: INFO: Pod "exec-volume-test-dynamicpv-7rhg": Phase="Pending", Reason="", readiness=false. Elapsed: 10.646560514s ... skipping 14 lines ... Jan 19 02:16:42.185: INFO: Pod "exec-volume-test-dynamicpv-7rhg": Phase="Pending", Reason="", readiness=false. Elapsed: 42.283043976s Jan 19 02:16:44.294: INFO: Pod "exec-volume-test-dynamicpv-7rhg": Phase="Pending", Reason="", readiness=false. Elapsed: 44.391339329s Jan 19 02:16:46.402: INFO: Pod "exec-volume-test-dynamicpv-7rhg": Phase="Pending", Reason="", readiness=false. Elapsed: 46.50001593s Jan 19 02:16:48.511: INFO: Pod "exec-volume-test-dynamicpv-7rhg": Phase="Pending", Reason="", readiness=false. Elapsed: 48.608536976s Jan 19 02:16:50.620: INFO: Pod "exec-volume-test-dynamicpv-7rhg": Phase="Succeeded", Reason="", readiness=false. Elapsed: 50.717814043s [1mSTEP[0m: Saw pod success Jan 19 02:16:50.620: INFO: Pod "exec-volume-test-dynamicpv-7rhg" satisfied condition "Succeeded or Failed" Jan 19 02:16:50.728: INFO: Trying to get logs from node k8s-agentpool1-29517166-vmss000000 pod exec-volume-test-dynamicpv-7rhg container exec-container-dynamicpv-7rhg: <nil> [1mSTEP[0m: delete the pod Jan 19 02:16:50.960: INFO: Waiting for pod exec-volume-test-dynamicpv-7rhg to disappear Jan 19 02:16:51.067: INFO: Pod exec-volume-test-dynamicpv-7rhg no longer exists [1mSTEP[0m: Deleting pod exec-volume-test-dynamicpv-7rhg Jan 19 02:16:51.068: INFO: Deleting pod "exec-volume-test-dynamicpv-7rhg" in namespace "volume-2250" ... skipping 27 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (ext3)] volumes [90mtest/e2e/storage/framework/testsuite.go:50[0m should allow exec of files on the volume [90mtest/e2e/storage/testsuites/volumes.go:198[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext3)] volumes should allow exec of files on the volume","total":32,"completed":4,"skipped":184,"failed":0} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [BeforeEach] [Testpattern: Ephemeral Snapshot (delete policy)] snapshottable[Feature:VolumeSnapshotDataSource] test/e2e/storage/framework/testsuite.go:51 [BeforeEach] [Testpattern: Ephemeral Snapshot (delete policy)] snapshottable[Feature:VolumeSnapshotDataSource] ... skipping 103 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (default fs)] subPath [90mtest/e2e/storage/framework/testsuite.go:50[0m should support restarting containers using directory as subpath [Slow] [90mtest/e2e/storage/testsuites/subpath.go:322[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should support restarting containers using directory as subpath [Slow]","total":33,"completed":4,"skipped":436,"failed":0} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [BeforeEach] [Testpattern: Pre-provisioned PV (filesystem volmode)] multiVolume [Slow] test/e2e/storage/framework/testsuite.go:51 Jan 19 02:18:19.320: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping ... skipping 209 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] [90mtest/e2e/storage/framework/testsuite.go:50[0m should access to two volumes with the same volume mode and retain data across pod recreation on the same node [90mtest/e2e/storage/testsuites/multivolume.go:138[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] should access to two volumes with the same volume mode and retain data across pod recreation on the same node","total":30,"completed":4,"skipped":302,"failed":0} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Dynamic PV (default fs)] volumeIO[0m [1mshould write files of various sizes, verify size, validate content [Slow][0m [37mtest/e2e/storage/testsuites/volume_io.go:149[0m ... skipping 58 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (default fs)] volumeIO [90mtest/e2e/storage/framework/testsuite.go:50[0m should write files of various sizes, verify size, validate content [Slow] [90mtest/e2e/storage/testsuites/volume_io.go:149[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] volumeIO should write files of various sizes, verify size, validate content [Slow]","total":35,"completed":6,"skipped":562,"failed":0} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [BeforeEach] [Testpattern: Inline-volume (default fs)] volumeIO test/e2e/storage/framework/testsuite.go:51 Jan 19 02:19:55.011: INFO: Driver "test.csi.azure.com" does not support volume type "InlineVolume" - skipping ... skipping 24 lines ... [36m[1mS [SKIPPING] in Spec Setup (BeforeEach) [0.001 seconds][0m External Storage [Driver: test.csi.azure.com] [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (ntfs)][Feature:Windows] subPath [90mtest/e2e/storage/framework/testsuite.go:50[0m [36m[1mshould fail if subpath directory is outside the volume [Slow][LinuxOnly] [BeforeEach][0m [90mtest/e2e/storage/testsuites/subpath.go:242[0m [36mDistro debian doesn't support ntfs -- skipping[0m test/e2e/storage/framework/testsuite.go:127 [90m------------------------------[0m ... skipping 121 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] [90mtest/e2e/storage/framework/testsuite.go:50[0m should concurrently access the volume and restored snapshot from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS] [90mtest/e2e/storage/testsuites/multivolume.go:323[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] should concurrently access the volume and restored snapshot from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS]","total":31,"completed":5,"skipped":498,"failed":0} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [BeforeEach] [Testpattern: Pre-provisioned PV (block volmode)] multiVolume [Slow] test/e2e/storage/framework/testsuite.go:51 Jan 19 02:20:47.463: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping ... skipping 221 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow] [90mtest/e2e/storage/framework/testsuite.go:50[0m should access to two volumes with the same volume mode and retain data across pod recreation on the same node [90mtest/e2e/storage/testsuites/multivolume.go:138[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow] should access to two volumes with the same volume mode and retain data across pod recreation on the same node","total":32,"completed":5,"skipped":205,"failed":0} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Dynamic PV (xfs)][Slow] volumes[0m [1mshould allow exec of files on the volume[0m [37mtest/e2e/storage/testsuites/volumes.go:198[0m ... skipping 17 lines ... Jan 19 02:18:20.325: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.coms6qrc] to have phase Bound Jan 19 02:18:20.432: INFO: PersistentVolumeClaim test.csi.azure.coms6qrc found but phase is Pending instead of Bound. Jan 19 02:18:22.545: INFO: PersistentVolumeClaim test.csi.azure.coms6qrc found but phase is Pending instead of Bound. Jan 19 02:18:24.652: INFO: PersistentVolumeClaim test.csi.azure.coms6qrc found and phase=Bound (4.326935374s) [1mSTEP[0m: Creating pod exec-volume-test-dynamicpv-tx7d [1mSTEP[0m: Creating a pod to test exec-volume-test Jan 19 02:18:24.974: INFO: Waiting up to 5m0s for pod "exec-volume-test-dynamicpv-tx7d" in namespace "volume-3445" to be "Succeeded or Failed" Jan 19 02:18:25.083: INFO: Pod "exec-volume-test-dynamicpv-tx7d": Phase="Pending", Reason="", readiness=false. Elapsed: 108.694205ms Jan 19 02:18:27.194: INFO: Pod "exec-volume-test-dynamicpv-tx7d": Phase="Pending", Reason="", readiness=false. Elapsed: 2.21923192s Jan 19 02:18:29.303: INFO: Pod "exec-volume-test-dynamicpv-tx7d": Phase="Pending", Reason="", readiness=false. Elapsed: 4.32814575s Jan 19 02:18:31.411: INFO: Pod "exec-volume-test-dynamicpv-tx7d": Phase="Pending", Reason="", readiness=false. Elapsed: 6.436215038s Jan 19 02:18:33.519: INFO: Pod "exec-volume-test-dynamicpv-tx7d": Phase="Pending", Reason="", readiness=false. Elapsed: 8.544067052s Jan 19 02:18:35.627: INFO: Pod "exec-volume-test-dynamicpv-tx7d": Phase="Pending", Reason="", readiness=false. Elapsed: 10.652183025s ... skipping 15 lines ... Jan 19 02:19:09.378: INFO: Pod "exec-volume-test-dynamicpv-tx7d": Phase="Pending", Reason="", readiness=false. Elapsed: 44.403634062s Jan 19 02:19:11.487: INFO: Pod "exec-volume-test-dynamicpv-tx7d": Phase="Pending", Reason="", readiness=false. Elapsed: 46.512486243s Jan 19 02:19:13.594: INFO: Pod "exec-volume-test-dynamicpv-tx7d": Phase="Pending", Reason="", readiness=false. Elapsed: 48.619861601s Jan 19 02:19:15.704: INFO: Pod "exec-volume-test-dynamicpv-tx7d": Phase="Pending", Reason="", readiness=false. Elapsed: 50.728973066s Jan 19 02:19:17.815: INFO: Pod "exec-volume-test-dynamicpv-tx7d": Phase="Succeeded", Reason="", readiness=false. Elapsed: 52.840814286s [1mSTEP[0m: Saw pod success Jan 19 02:19:17.815: INFO: Pod "exec-volume-test-dynamicpv-tx7d" satisfied condition "Succeeded or Failed" Jan 19 02:19:17.923: INFO: Trying to get logs from node k8s-agentpool1-29517166-vmss000002 pod exec-volume-test-dynamicpv-tx7d container exec-container-dynamicpv-tx7d: <nil> [1mSTEP[0m: delete the pod Jan 19 02:19:18.168: INFO: Waiting for pod exec-volume-test-dynamicpv-tx7d to disappear Jan 19 02:19:18.275: INFO: Pod exec-volume-test-dynamicpv-tx7d no longer exists [1mSTEP[0m: Deleting pod exec-volume-test-dynamicpv-tx7d Jan 19 02:19:18.275: INFO: Deleting pod "exec-volume-test-dynamicpv-tx7d" in namespace "volume-3445" ... skipping 39 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (xfs)][Slow] volumes [90mtest/e2e/storage/framework/testsuite.go:50[0m should allow exec of files on the volume [90mtest/e2e/storage/testsuites/volumes.go:198[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (xfs)][Slow] volumes should allow exec of files on the volume","total":33,"completed":5,"skipped":580,"failed":0} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Dynamic PV (block volmode)] multiVolume [Slow][0m [1mshould access to two volumes with different volume mode and retain data across pod recreation on the same node[0m [37mtest/e2e/storage/testsuites/multivolume.go:209[0m ... skipping 194 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] [90mtest/e2e/storage/framework/testsuite.go:50[0m should access to two volumes with different volume mode and retain data across pod recreation on the same node [90mtest/e2e/storage/testsuites/multivolume.go:209[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] should access to two volumes with different volume mode and retain data across pod recreation on the same node","total":35,"completed":7,"skipped":600,"failed":0} [36mS[0m[36mS[0m [90m------------------------------[0m [BeforeEach] [Testpattern: Inline-volume (default fs)] subPath test/e2e/storage/framework/testsuite.go:51 Jan 19 02:23:55.654: INFO: Driver "test.csi.azure.com" does not support volume type "InlineVolume" - skipping ... skipping 387 lines ... Jan 19 02:20:48.453: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.com7wzmq] to have phase Bound Jan 19 02:20:48.560: INFO: PersistentVolumeClaim test.csi.azure.com7wzmq found but phase is Pending instead of Bound. Jan 19 02:20:50.667: INFO: PersistentVolumeClaim test.csi.azure.com7wzmq found but phase is Pending instead of Bound. Jan 19 02:20:52.774: INFO: PersistentVolumeClaim test.csi.azure.com7wzmq found and phase=Bound (4.321266653s) [1mSTEP[0m: Creating pod pod-subpath-test-dynamicpv-9skw [1mSTEP[0m: Creating a pod to test subpath Jan 19 02:20:53.105: INFO: Waiting up to 5m0s for pod "pod-subpath-test-dynamicpv-9skw" in namespace "provisioning-7729" to be "Succeeded or Failed" Jan 19 02:20:53.212: INFO: Pod "pod-subpath-test-dynamicpv-9skw": Phase="Pending", Reason="", readiness=false. Elapsed: 107.527432ms Jan 19 02:20:55.320: INFO: Pod "pod-subpath-test-dynamicpv-9skw": Phase="Pending", Reason="", readiness=false. Elapsed: 2.215314605s Jan 19 02:20:57.427: INFO: Pod "pod-subpath-test-dynamicpv-9skw": Phase="Pending", Reason="", readiness=false. Elapsed: 4.322844717s Jan 19 02:20:59.537: INFO: Pod "pod-subpath-test-dynamicpv-9skw": Phase="Pending", Reason="", readiness=false. Elapsed: 6.432552531s Jan 19 02:21:01.646: INFO: Pod "pod-subpath-test-dynamicpv-9skw": Phase="Pending", Reason="", readiness=false. Elapsed: 8.541085609s Jan 19 02:21:03.753: INFO: Pod "pod-subpath-test-dynamicpv-9skw": Phase="Pending", Reason="", readiness=false. Elapsed: 10.648746808s ... skipping 21 lines ... Jan 19 02:21:50.155: INFO: Pod "pod-subpath-test-dynamicpv-9skw": Phase="Pending", Reason="", readiness=false. Elapsed: 57.050094883s Jan 19 02:21:52.265: INFO: Pod "pod-subpath-test-dynamicpv-9skw": Phase="Pending", Reason="", readiness=false. Elapsed: 59.160844985s Jan 19 02:21:54.375: INFO: Pod "pod-subpath-test-dynamicpv-9skw": Phase="Pending", Reason="", readiness=false. Elapsed: 1m1.270375259s Jan 19 02:21:56.491: INFO: Pod "pod-subpath-test-dynamicpv-9skw": Phase="Pending", Reason="", readiness=false. Elapsed: 1m3.386758469s Jan 19 02:21:58.600: INFO: Pod "pod-subpath-test-dynamicpv-9skw": Phase="Succeeded", Reason="", readiness=false. Elapsed: 1m5.495619061s [1mSTEP[0m: Saw pod success Jan 19 02:21:58.600: INFO: Pod "pod-subpath-test-dynamicpv-9skw" satisfied condition "Succeeded or Failed" Jan 19 02:21:58.708: INFO: Trying to get logs from node k8s-agentpool1-29517166-vmss000002 pod pod-subpath-test-dynamicpv-9skw container test-container-subpath-dynamicpv-9skw: <nil> [1mSTEP[0m: delete the pod Jan 19 02:21:58.953: INFO: Waiting for pod pod-subpath-test-dynamicpv-9skw to disappear Jan 19 02:21:59.059: INFO: Pod pod-subpath-test-dynamicpv-9skw no longer exists [1mSTEP[0m: Deleting pod pod-subpath-test-dynamicpv-9skw Jan 19 02:21:59.059: INFO: Deleting pod "pod-subpath-test-dynamicpv-9skw" in namespace "provisioning-7729" [1mSTEP[0m: Creating pod pod-subpath-test-dynamicpv-9skw [1mSTEP[0m: Creating a pod to test subpath Jan 19 02:21:59.279: INFO: Waiting up to 5m0s for pod "pod-subpath-test-dynamicpv-9skw" in namespace "provisioning-7729" to be "Succeeded or Failed" Jan 19 02:21:59.386: INFO: Pod "pod-subpath-test-dynamicpv-9skw": Phase="Pending", Reason="", readiness=false. Elapsed: 107.01361ms Jan 19 02:22:01.493: INFO: Pod "pod-subpath-test-dynamicpv-9skw": Phase="Pending", Reason="", readiness=false. Elapsed: 2.214402494s Jan 19 02:22:03.602: INFO: Pod "pod-subpath-test-dynamicpv-9skw": Phase="Pending", Reason="", readiness=false. Elapsed: 4.323292733s Jan 19 02:22:05.712: INFO: Pod "pod-subpath-test-dynamicpv-9skw": Phase="Pending", Reason="", readiness=false. Elapsed: 6.43316639s Jan 19 02:22:07.819: INFO: Pod "pod-subpath-test-dynamicpv-9skw": Phase="Pending", Reason="", readiness=false. Elapsed: 8.540412714s Jan 19 02:22:09.927: INFO: Pod "pod-subpath-test-dynamicpv-9skw": Phase="Pending", Reason="", readiness=false. Elapsed: 10.64870975s ... skipping 59 lines ... Jan 19 02:24:16.437: INFO: Pod "pod-subpath-test-dynamicpv-9skw": Phase="Pending", Reason="", readiness=false. Elapsed: 2m17.157858786s Jan 19 02:24:18.545: INFO: Pod "pod-subpath-test-dynamicpv-9skw": Phase="Pending", Reason="", readiness=false. Elapsed: 2m19.266506841s Jan 19 02:24:20.653: INFO: Pod "pod-subpath-test-dynamicpv-9skw": Phase="Pending", Reason="", readiness=false. Elapsed: 2m21.374027893s Jan 19 02:24:22.760: INFO: Pod "pod-subpath-test-dynamicpv-9skw": Phase="Pending", Reason="", readiness=false. Elapsed: 2m23.481200852s Jan 19 02:24:24.868: INFO: Pod "pod-subpath-test-dynamicpv-9skw": Phase="Succeeded", Reason="", readiness=false. Elapsed: 2m25.589726714s [1mSTEP[0m: Saw pod success Jan 19 02:24:24.869: INFO: Pod "pod-subpath-test-dynamicpv-9skw" satisfied condition "Succeeded or Failed" Jan 19 02:24:24.976: INFO: Trying to get logs from node k8s-agentpool1-29517166-vmss000000 pod pod-subpath-test-dynamicpv-9skw container test-container-subpath-dynamicpv-9skw: <nil> [1mSTEP[0m: delete the pod Jan 19 02:24:25.229: INFO: Waiting for pod pod-subpath-test-dynamicpv-9skw to disappear Jan 19 02:24:25.336: INFO: Pod pod-subpath-test-dynamicpv-9skw no longer exists [1mSTEP[0m: Deleting pod pod-subpath-test-dynamicpv-9skw Jan 19 02:24:25.336: INFO: Deleting pod "pod-subpath-test-dynamicpv-9skw" in namespace "provisioning-7729" ... skipping 23 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (default fs)] subPath [90mtest/e2e/storage/framework/testsuite.go:50[0m should support existing directories when readOnly specified in the volumeSource [90mtest/e2e/storage/testsuites/subpath.go:397[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should support existing directories when readOnly specified in the volumeSource","total":31,"completed":6,"skipped":535,"failed":0} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [BeforeEach] [Testpattern: Dynamic PV (ntfs)][Feature:Windows] provisioning test/e2e/storage/framework/testsuite.go:51 Jan 19 02:25:07.195: INFO: Distro debian doesn't support ntfs -- skipping ... skipping 101 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow] [90mtest/e2e/storage/framework/testsuite.go:50[0m should concurrently access the single read-only volume from pods on the same node [90mtest/e2e/storage/testsuites/multivolume.go:423[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow] should concurrently access the single read-only volume from pods on the same node","total":33,"completed":6,"skipped":586,"failed":0} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Dynamic PV (default fs)] subPath[0m [1mshould support readOnly file specified in the volumeMount [LinuxOnly][0m [37mtest/e2e/storage/testsuites/subpath.go:382[0m ... skipping 17 lines ... Jan 19 02:18:29.940: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.com28gl4] to have phase Bound Jan 19 02:18:30.047: INFO: PersistentVolumeClaim test.csi.azure.com28gl4 found but phase is Pending instead of Bound. Jan 19 02:18:32.155: INFO: PersistentVolumeClaim test.csi.azure.com28gl4 found but phase is Pending instead of Bound. Jan 19 02:18:34.263: INFO: PersistentVolumeClaim test.csi.azure.com28gl4 found and phase=Bound (4.323189424s) [1mSTEP[0m: Creating pod pod-subpath-test-dynamicpv-mfv5 [1mSTEP[0m: Creating a pod to test subpath Jan 19 02:18:34.588: INFO: Waiting up to 5m0s for pod "pod-subpath-test-dynamicpv-mfv5" in namespace "provisioning-1042" to be "Succeeded or Failed" Jan 19 02:18:34.695: INFO: Pod "pod-subpath-test-dynamicpv-mfv5": Phase="Pending", Reason="", readiness=false. Elapsed: 107.088073ms Jan 19 02:18:36.807: INFO: Pod "pod-subpath-test-dynamicpv-mfv5": Phase="Pending", Reason="", readiness=false. Elapsed: 2.218884971s Jan 19 02:18:38.916: INFO: Pod "pod-subpath-test-dynamicpv-mfv5": Phase="Pending", Reason="", readiness=false. Elapsed: 4.328070272s Jan 19 02:18:41.024: INFO: Pod "pod-subpath-test-dynamicpv-mfv5": Phase="Pending", Reason="", readiness=false. Elapsed: 6.436234633s Jan 19 02:18:43.132: INFO: Pod "pod-subpath-test-dynamicpv-mfv5": Phase="Pending", Reason="", readiness=false. Elapsed: 8.54451935s Jan 19 02:18:45.240: INFO: Pod "pod-subpath-test-dynamicpv-mfv5": Phase="Pending", Reason="", readiness=false. Elapsed: 10.652288853s ... skipping 59 lines ... Jan 19 02:20:51.781: INFO: Pod "pod-subpath-test-dynamicpv-mfv5": Phase="Pending", Reason="", readiness=false. Elapsed: 2m17.192935831s Jan 19 02:20:53.890: INFO: Pod "pod-subpath-test-dynamicpv-mfv5": Phase="Pending", Reason="", readiness=false. Elapsed: 2m19.302198047s Jan 19 02:20:55.998: INFO: Pod "pod-subpath-test-dynamicpv-mfv5": Phase="Pending", Reason="", readiness=false. Elapsed: 2m21.410584034s Jan 19 02:20:58.107: INFO: Pod "pod-subpath-test-dynamicpv-mfv5": Phase="Pending", Reason="", readiness=false. Elapsed: 2m23.519667825s Jan 19 02:21:00.215: INFO: Pod "pod-subpath-test-dynamicpv-mfv5": Phase="Succeeded", Reason="", readiness=false. Elapsed: 2m25.627296127s [1mSTEP[0m: Saw pod success Jan 19 02:21:00.215: INFO: Pod "pod-subpath-test-dynamicpv-mfv5" satisfied condition "Succeeded or Failed" Jan 19 02:21:00.323: INFO: Trying to get logs from node k8s-agentpool1-29517166-vmss000000 pod pod-subpath-test-dynamicpv-mfv5 container test-container-subpath-dynamicpv-mfv5: <nil> [1mSTEP[0m: delete the pod Jan 19 02:21:00.569: INFO: Waiting for pod pod-subpath-test-dynamicpv-mfv5 to disappear Jan 19 02:21:00.676: INFO: Pod pod-subpath-test-dynamicpv-mfv5 no longer exists [1mSTEP[0m: Deleting pod pod-subpath-test-dynamicpv-mfv5 Jan 19 02:21:00.676: INFO: Deleting pod "pod-subpath-test-dynamicpv-mfv5" in namespace "provisioning-1042" ... skipping 66 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (default fs)] subPath [90mtest/e2e/storage/framework/testsuite.go:50[0m should support readOnly file specified in the volumeMount [LinuxOnly] [90mtest/e2e/storage/testsuites/subpath.go:382[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should support readOnly file specified in the volumeMount [LinuxOnly]","total":30,"completed":5,"skipped":338,"failed":0} [BeforeEach] [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow] test/e2e/storage/framework/testsuite.go:51 [BeforeEach] [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow] test/e2e/framework/framework.go:187 [1mSTEP[0m: Creating a kubernetes client Jan 19 02:25:22.216: INFO: >>> kubeConfig: /root/tmp1374176058/kubeconfig/kubeconfig.westeurope.json ... skipping 63 lines ... test/e2e/storage/external/external.go:262 [90m------------------------------[0m [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Dynamic PV (block volmode)] volumeMode[0m [1mshould fail to use a volume in a pod with mismatched mode [Slow][0m [37mtest/e2e/storage/testsuites/volumemode.go:299[0m [BeforeEach] [Testpattern: Dynamic PV (block volmode)] volumeMode test/e2e/storage/framework/testsuite.go:51 [BeforeEach] [Testpattern: Dynamic PV (block volmode)] volumeMode test/e2e/framework/framework.go:187 [1mSTEP[0m: Creating a kubernetes client Jan 19 02:20:57.636: INFO: >>> kubeConfig: /root/tmp1374176058/kubeconfig/kubeconfig.westeurope.json [1mSTEP[0m: Building a namespace api object, basename volumemode [1mSTEP[0m: Waiting for a default service account to be provisioned in namespace [1mSTEP[0m: Waiting for kube-root-ca.crt to be provisioned in namespace [It] should fail to use a volume in a pod with mismatched mode [Slow] test/e2e/storage/testsuites/volumemode.go:299 Jan 19 02:20:58.392: INFO: Creating resource for dynamic PV Jan 19 02:20:58.392: INFO: Using claimSize:5Gi, test suite supported size:{ 1Mi}, driver(test.csi.azure.com) supported size:{ 1Mi} [1mSTEP[0m: creating a StorageClass volumemode-198-e2e-sc6fcvn [1mSTEP[0m: creating a claim Jan 19 02:20:58.611: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comhvj87] to have phase Bound Jan 19 02:20:58.720: INFO: PersistentVolumeClaim test.csi.azure.comhvj87 found but phase is Pending instead of Bound. Jan 19 02:21:00.828: INFO: PersistentVolumeClaim test.csi.azure.comhvj87 found but phase is Pending instead of Bound. Jan 19 02:21:02.936: INFO: PersistentVolumeClaim test.csi.azure.comhvj87 found and phase=Bound (4.324992522s) [1mSTEP[0m: Creating pod [1mSTEP[0m: Waiting for the pod to fail Jan 19 02:21:05.588: INFO: Deleting pod "pod-677b154a-05eb-4f20-bbf6-0db694d379a9" in namespace "volumemode-198" Jan 19 02:21:05.702: INFO: Wait up to 5m0s for pod "pod-677b154a-05eb-4f20-bbf6-0db694d379a9" to be fully deleted [1mSTEP[0m: Deleting pvc Jan 19 02:21:07.920: INFO: Deleting PersistentVolumeClaim "test.csi.azure.comhvj87" Jan 19 02:21:08.033: INFO: Waiting up to 5m0s for PersistentVolume pvc-f3805438-21b2-490c-aafd-4ced2e14c1d0 to get deleted Jan 19 02:21:08.141: INFO: PersistentVolume pvc-f3805438-21b2-490c-aafd-4ced2e14c1d0 found and phase=Released (108.006219ms) ... skipping 57 lines ... [32m• [SLOW TEST:271.662 seconds][0m External Storage [Driver: test.csi.azure.com] [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (block volmode)] volumeMode [90mtest/e2e/storage/framework/testsuite.go:50[0m should fail to use a volume in a pod with mismatched mode [Slow] [90mtest/e2e/storage/testsuites/volumemode.go:299[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] volumeMode should fail to use a volume in a pod with mismatched mode [Slow]","total":32,"completed":6,"skipped":242,"failed":0} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral[0m [1mshould support two pods which have the same volume definition[0m [37mtest/e2e/storage/testsuites/ephemeral.go:216[0m ... skipping 114 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral [90mtest/e2e/storage/framework/testsuite.go:50[0m should support two pods which have the same volume definition [90mtest/e2e/storage/testsuites/ephemeral.go:216[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral should support two pods which have the same volume definition","total":37,"completed":6,"skipped":368,"failed":0} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [BeforeEach] [Testpattern: Pre-provisioned PV (filesystem volmode)] volumeMode test/e2e/storage/framework/testsuite.go:51 Jan 19 02:25:58.009: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping ... skipping 3 lines ... [36m[1mS [SKIPPING] in Spec Setup (BeforeEach) [0.001 seconds][0m External Storage [Driver: test.csi.azure.com] [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Pre-provisioned PV (filesystem volmode)] volumeMode [90mtest/e2e/storage/framework/testsuite.go:50[0m [36m[1mshould fail to use a volume in a pod with mismatched mode [Slow] [BeforeEach][0m [90mtest/e2e/storage/testsuites/volumemode.go:299[0m [36mDriver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping[0m test/e2e/storage/external/external.go:262 [90m------------------------------[0m ... skipping 26 lines ... test/e2e/framework/framework.go:187 [1mSTEP[0m: Creating a kubernetes client Jan 19 02:25:58.068: INFO: >>> kubeConfig: /root/tmp1374176058/kubeconfig/kubeconfig.westeurope.json [1mSTEP[0m: Building a namespace api object, basename topology [1mSTEP[0m: Waiting for a default service account to be provisioned in namespace [1mSTEP[0m: Waiting for kube-root-ca.crt to be provisioned in namespace [It] should fail to schedule a pod which has topologies that conflict with AllowedTopologies test/e2e/storage/testsuites/topology.go:194 Jan 19 02:25:58.833: INFO: Driver didn't provide topology keys -- skipping [AfterEach] [Testpattern: Dynamic PV (immediate binding)] topology test/e2e/framework/framework.go:188 Jan 19 02:25:58.834: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready [1mSTEP[0m: Destroying namespace "topology-5403" for this suite. [36m[1mS [SKIPPING] [0.983 seconds][0m External Storage [Driver: test.csi.azure.com] [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (immediate binding)] topology [90mtest/e2e/storage/framework/testsuite.go:50[0m [36m[1mshould fail to schedule a pod which has topologies that conflict with AllowedTopologies [Measurement][0m [90mtest/e2e/storage/testsuites/topology.go:194[0m [36mDriver didn't provide topology keys -- skipping[0m test/e2e/storage/testsuites/topology.go:126 [90m------------------------------[0m ... skipping 174 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow] [90mtest/e2e/storage/framework/testsuite.go:50[0m should concurrently access the single volume from pods on the same node [90mtest/e2e/storage/testsuites/multivolume.go:298[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow] should concurrently access the single volume from pods on the same node","total":33,"completed":7,"skipped":595,"failed":0} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [BeforeEach] [Testpattern: Dynamic PV (ntfs)][Feature:Windows] multiVolume [Slow] test/e2e/storage/framework/testsuite.go:51 Jan 19 02:26:50.743: INFO: Distro debian doesn't support ntfs -- skipping ... skipping 101 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Generic Ephemeral-volume (block volmode) (late-binding)] ephemeral [90mtest/e2e/storage/framework/testsuite.go:50[0m should support two pods which have the same volume definition [90mtest/e2e/storage/testsuites/ephemeral.go:216[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Generic Ephemeral-volume (block volmode) (late-binding)] ephemeral should support two pods which have the same volume definition","total":35,"completed":8,"skipped":691,"failed":0} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [BeforeEach] [Testpattern: Pre-provisioned PV (block volmode)] multiVolume [Slow] test/e2e/storage/framework/testsuite.go:51 Jan 19 02:29:38.843: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping ... skipping 138 lines ... Jan 19 02:34:24.041: INFO: PersistentVolume pvc-f7f1e685-5e57-4dc7-9d6e-c0440a3db932 found and phase=Released (4m35.975675997s) Jan 19 02:34:29.148: INFO: PersistentVolume pvc-f7f1e685-5e57-4dc7-9d6e-c0440a3db932 found and phase=Released (4m41.083653058s) Jan 19 02:34:34.257: INFO: PersistentVolume pvc-f7f1e685-5e57-4dc7-9d6e-c0440a3db932 found and phase=Released (4m46.191810583s) Jan 19 02:34:39.364: INFO: PersistentVolume pvc-f7f1e685-5e57-4dc7-9d6e-c0440a3db932 found and phase=Released (4m51.299614011s) Jan 19 02:34:44.474: INFO: PersistentVolume pvc-f7f1e685-5e57-4dc7-9d6e-c0440a3db932 found and phase=Released (4m56.408720384s) [1mSTEP[0m: Deleting sc Jan 19 02:34:49.584: FAIL: while cleanup resource Unexpected error: <errors.aggregate | len:1, cap:1>: [ [ { msg: "persistent Volume pvc-f7f1e685-5e57-4dc7-9d6e-c0440a3db932 not deleted by dynamic provisioner: PersistentVolume pvc-f7f1e685-5e57-4dc7-9d6e-c0440a3db932 still exists within 5m0s", err: { s: "PersistentVolume pvc-f7f1e685-5e57-4dc7-9d6e-c0440a3db932 still exists within 5m0s", ... skipping 28 lines ... Jan 19 02:34:49.799: INFO: At 2023-01-19 02:25:29 +0000 UTC - event for pod-27fef38f-eb79-47dd-91c3-d85d18e7db4c: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-f7f1e685-5e57-4dc7-9d6e-c0440a3db932" Jan 19 02:34:49.799: INFO: At 2023-01-19 02:25:43 +0000 UTC - event for pod-27fef38f-eb79-47dd-91c3-d85d18e7db4c: {kubelet k8s-agentpool1-29517166-vmss000002} Started: Started container write-pod Jan 19 02:34:49.799: INFO: At 2023-01-19 02:25:43 +0000 UTC - event for pod-27fef38f-eb79-47dd-91c3-d85d18e7db4c: {kubelet k8s-agentpool1-29517166-vmss000002} Pulled: Successfully pulled image "k8s.gcr.io/e2e-test-images/busybox:1.29-2" in 329.024865ms Jan 19 02:34:49.799: INFO: At 2023-01-19 02:25:43 +0000 UTC - event for pod-27fef38f-eb79-47dd-91c3-d85d18e7db4c: {kubelet k8s-agentpool1-29517166-vmss000002} Pulling: Pulling image "k8s.gcr.io/e2e-test-images/busybox:1.29-2" Jan 19 02:34:49.799: INFO: At 2023-01-19 02:25:43 +0000 UTC - event for pod-27fef38f-eb79-47dd-91c3-d85d18e7db4c: {kubelet k8s-agentpool1-29517166-vmss000002} Created: Created container write-pod Jan 19 02:34:49.799: INFO: At 2023-01-19 02:25:45 +0000 UTC - event for pod-27fef38f-eb79-47dd-91c3-d85d18e7db4c: {kubelet k8s-agentpool1-29517166-vmss000002} Killing: Stopping container write-pod Jan 19 02:34:49.799: INFO: At 2023-01-19 02:25:48 +0000 UTC - event for pod-27fef38f-eb79-47dd-91c3-d85d18e7db4c: {kubelet k8s-agentpool1-29517166-vmss000002} FailedKillPod: error killing pod: failed to "KillContainer" for "write-pod" with KillContainerError: "rpc error: code = Unknown desc = Error response from daemon: No such container: e0194cd37a0a2ccb2180fb1b17dcdab0686e9cb1e3577ff6ddd4ad9dd11ae3bb" Jan 19 02:34:49.799: INFO: At 2023-01-19 02:25:49 +0000 UTC - event for pod-973c92ca-b1bd-4988-8695-e1fc85294761: {attachdetach-controller } FailedAttachVolume: Multi-Attach error for volume "pvc-f7f1e685-5e57-4dc7-9d6e-c0440a3db932" Volume is already exclusively attached to one node and can't be attached to another Jan 19 02:34:49.799: INFO: At 2023-01-19 02:25:49 +0000 UTC - event for pod-973c92ca-b1bd-4988-8695-e1fc85294761: {default-scheduler } Scheduled: Successfully assigned multivolume-9364/pod-973c92ca-b1bd-4988-8695-e1fc85294761 to k8s-agentpool1-29517166-vmss000000 Jan 19 02:34:49.799: INFO: At 2023-01-19 02:27:52 +0000 UTC - event for pod-973c92ca-b1bd-4988-8695-e1fc85294761: {kubelet k8s-agentpool1-29517166-vmss000000} FailedMount: Unable to attach or mount volumes: unmounted volumes=[volume1], unattached volumes=[volume1 kube-api-access-vdd7w]: timed out waiting for the condition Jan 19 02:34:49.799: INFO: At 2023-01-19 02:28:26 +0000 UTC - event for pod-973c92ca-b1bd-4988-8695-e1fc85294761: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-f7f1e685-5e57-4dc7-9d6e-c0440a3db932" Jan 19 02:34:49.799: INFO: At 2023-01-19 02:29:27 +0000 UTC - event for pod-973c92ca-b1bd-4988-8695-e1fc85294761: {kubelet k8s-agentpool1-29517166-vmss000000} FileSystemResizeFailed: MountVolume.NodeExpandVolume failed for volume "pvc-f7f1e685-5e57-4dc7-9d6e-c0440a3db932" requested read-only file system Jan 19 02:34:49.799: INFO: At 2023-01-19 02:29:27 +0000 UTC - event for test.csi.azure.comtjwrz: {kubelet k8s-agentpool1-29517166-vmss000000} FileSystemResizeFailed: MountVolume.NodeExpandVolume failed for volume "pvc-f7f1e685-5e57-4dc7-9d6e-c0440a3db932" requested read-only file system Jan 19 02:34:49.799: INFO: At 2023-01-19 02:29:28 +0000 UTC - event for pod-973c92ca-b1bd-4988-8695-e1fc85294761: {kubelet k8s-agentpool1-29517166-vmss000000} Pulling: Pulling image "k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.5" Jan 19 02:34:49.799: INFO: At 2023-01-19 02:29:29 +0000 UTC - event for pod-973c92ca-b1bd-4988-8695-e1fc85294761: {kubelet k8s-agentpool1-29517166-vmss000000} Created: Created container write-pod Jan 19 02:34:49.799: INFO: At 2023-01-19 02:29:29 +0000 UTC - event for pod-973c92ca-b1bd-4988-8695-e1fc85294761: {kubelet k8s-agentpool1-29517166-vmss000000} Started: Started container write-pod Jan 19 02:34:49.799: INFO: At 2023-01-19 02:29:29 +0000 UTC - event for pod-973c92ca-b1bd-4988-8695-e1fc85294761: {kubelet k8s-agentpool1-29517166-vmss000000} Pulled: Successfully pulled image "k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.5" in 1.262995112s Jan 19 02:34:49.799: INFO: At 2023-01-19 02:29:32 +0000 UTC - event for pod-8ca30c7d-6f91-4c72-ba32-c3f80f75396f: {default-scheduler } Scheduled: Successfully assigned multivolume-9364/pod-8ca30c7d-6f91-4c72-ba32-c3f80f75396f to k8s-agentpool1-29517166-vmss000000 Jan 19 02:34:49.799: INFO: At 2023-01-19 02:29:35 +0000 UTC - event for pod-8ca30c7d-6f91-4c72-ba32-c3f80f75396f: {kubelet k8s-agentpool1-29517166-vmss000000} Pulled: Successfully pulled image "k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.5" in 361.23494ms Jan 19 02:34:49.799: INFO: At 2023-01-19 02:29:35 +0000 UTC - event for pod-8ca30c7d-6f91-4c72-ba32-c3f80f75396f: {kubelet k8s-agentpool1-29517166-vmss000000} Pulling: Pulling image "k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.5" Jan 19 02:34:49.799: INFO: At 2023-01-19 02:29:35 +0000 UTC - event for pod-8ca30c7d-6f91-4c72-ba32-c3f80f75396f: {kubelet k8s-agentpool1-29517166-vmss000000} Created: Created container write-pod Jan 19 02:34:49.799: INFO: At 2023-01-19 02:29:35 +0000 UTC - event for pod-8ca30c7d-6f91-4c72-ba32-c3f80f75396f: {kubelet k8s-agentpool1-29517166-vmss000000} FileSystemResizeFailed: MountVolume.NodeExpandVolume failed for volume "pvc-f7f1e685-5e57-4dc7-9d6e-c0440a3db932" requested read-only file system Jan 19 02:34:49.799: INFO: At 2023-01-19 02:29:36 +0000 UTC - event for pod-8ca30c7d-6f91-4c72-ba32-c3f80f75396f: {kubelet k8s-agentpool1-29517166-vmss000000} Started: Started container write-pod Jan 19 02:34:49.799: INFO: At 2023-01-19 02:29:41 +0000 UTC - event for pod-8ca30c7d-6f91-4c72-ba32-c3f80f75396f: {kubelet k8s-agentpool1-29517166-vmss000000} Killing: Stopping container write-pod Jan 19 02:34:49.799: INFO: At 2023-01-19 02:29:43 +0000 UTC - event for pod-8ca30c7d-6f91-4c72-ba32-c3f80f75396f: {kubelet k8s-agentpool1-29517166-vmss000000} FailedKillPod: error killing pod: failed to "KillContainer" for "write-pod" with KillContainerError: "rpc error: code = Unknown desc = Error response from daemon: No such container: 1ea8bcc9934fb5c172a5080e5bfd66922b943983c62876ccdfe93d4e9d566937" Jan 19 02:34:49.799: INFO: At 2023-01-19 02:29:45 +0000 UTC - event for pod-973c92ca-b1bd-4988-8695-e1fc85294761: {kubelet k8s-agentpool1-29517166-vmss000000} Killing: Stopping container write-pod Jan 19 02:34:49.799: INFO: At 2023-01-19 02:29:47 +0000 UTC - event for pod-973c92ca-b1bd-4988-8695-e1fc85294761: {kubelet k8s-agentpool1-29517166-vmss000000} FailedKillPod: error killing pod: failed to "KillContainer" for "write-pod" with KillContainerError: "rpc error: code = Unknown desc = Error response from daemon: No such container: 71050a2aa6f6731559a6c6d434fd71b78d77a299118d46941ae092179c904204" Jan 19 02:34:49.906: INFO: POD NODE PHASE GRACE CONDITIONS Jan 19 02:34:49.906: INFO: Jan 19 02:34:50.120: INFO: Logging node info for node k8s-agentpool1-29517166-vmss000000 Jan 19 02:34:50.228: INFO: Node Info: &Node{ObjectMeta:{k8s-agentpool1-29517166-vmss000000 010e38ff-80ae-4d5c-a0fc-6820676d4797 17329 0 2023-01-19 01:51:01 +0000 UTC <nil> <nil> map[agentpool:agentpool1 beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_D8s_v3 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westeurope failure-domain.beta.kubernetes.io/zone:0 kubernetes.azure.com/cluster:kubetest-iei3lcrr kubernetes.azure.com/role:agent kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-agentpool1-29517166-vmss000000 kubernetes.io/os:linux kubernetes.io/role:agent node-role.kubernetes.io/agent: node.kubernetes.io/instance-type:Standard_D8s_v3 storageprofile:managed storagetier:Premium_LRS topology.kubernetes.io/region:westeurope topology.kubernetes.io/zone:0 topology.test.csi.azure.com/zone:] map[csi.volume.kubernetes.io/nodeid:{"test.csi.azure.com":"k8s-agentpool1-29517166-vmss000000"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-01-19 01:51:00 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:agentpool":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.azure.com/cluster":{},"f:kubernetes.azure.com/role":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:storageprofile":{},"f:storagetier":{}}}} } {kubectl-label Update v1 2023-01-19 01:51:09 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/agent":{}}}} } {kube-controller-manager Update v1 2023-01-19 01:51:18 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"10.244.3.0/24\"":{}}}} } {cloud-controller-manager Update v1 2023-01-19 01:52:09 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {cloud-node-manager Update v1 2023-01-19 01:52:43 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kube-controller-manager Update v1 2023-01-19 02:17:27 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-01-19 02:17:33 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.test.csi.azure.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:10.244.3.0/24,DoNotUseExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-iei3lcrr/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool1-29517166-vmss/virtualMachines/0,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[10.244.3.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{8 0} {<nil>} 8 DecimalSI},ephemeral-storage: {{31025332224 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{33672708096 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{8 0} {<nil>} 8 DecimalSI},ephemeral-storage: {{27922798956 0} {<nil>} 27922798956 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{32886276096 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-01-19 01:52:09 +0000 UTC,LastTransitionTime:2023-01-19 01:52:09 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-19 02:29:48 +0000 UTC,LastTransitionTime:2023-01-19 01:50:56 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-19 02:29:48 +0000 UTC,LastTransitionTime:2023-01-19 01:50:56 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-19 02:29:48 +0000 UTC,LastTransitionTime:2023-01-19 01:50:56 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-19 02:29:48 +0000 UTC,LastTransitionTime:2023-01-19 01:51:18 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.240.0.4,},NodeAddress{Type:Hostname,Address:k8s-agentpool1-29517166-vmss000000,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:850534128dbd4d1889d8150ebc315078,SystemUUID:762e608f-d263-4940-a3bd-27947897842e,BootID:8c627fcc-5295-44a8-8883-11801edd48e5,KernelVersion:5.4.0-1100-azure,OSImage:Ubuntu 18.04.6 LTS,ContainerRuntimeVersion:docker://20.10.14+azure-1,KubeletVersion:v1.23.15,KubeProxyVersion:v1.23.15,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:11e6a66017ba4e4b938c1612b7a54a3befcefd354796c04e1dba76873a13518e k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.5],SizeBytes:253346057,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/azuredisk-csi@sha256:423eb6cf602c064c8b2deefead5ceadd6324ed41b3d995dab5d0f6f0f4d4710f mcr.microsoft.com/oss/kubernetes-csi/azuredisk-csi:v1.10.0],SizeBytes:245959792,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/azurefile-csi@sha256:9e2ecabcf9dd9943e6600eb9fb460f45b4dc61af7cabe95d115082a029db2aaf mcr.microsoft.com/oss/kubernetes-csi/azurefile-csi:v1.9.0],SizeBytes:230470852,},ContainerImage{Names:[mcr.microsoft.com/containernetworking/azure-npm@sha256:106f669f48e5e80c4ec0afb49858ead72cf4b901cd8664e7bf81f8d789e56e12 mcr.microsoft.com/containernetworking/azure-npm:v1.2.2_hotfix],SizeBytes:175230380,},ContainerImage{Names:[k8sprow.azurecr.io/azuredisk-csi@sha256:407401f03a843a20acbdf97cb221e57990fdccdfc86b5468b73befb74e7c6526 k8sprow.azurecr.io/azuredisk-csi:latest-v2-ec131e57ca07be2e17dba65bd8a030b3e7a9034d],SizeBytes:146883742,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-apiserver@sha256:7fd087358ea7b38817f1a2d79868bda3f6fbc8811a060309a9ed3fc3b2d697a4 mcr.microsoft.com/oss/kubernetes/kube-apiserver:v1.23.15],SizeBytes:135200982,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:f5241226198f5a54d22540acf2b3933ea0f49458f90c51fc75833d0c428687b8 k8s.gcr.io/e2e-test-images/agnhost:2.36],SizeBytes:125426639,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-controller-manager@sha256:afa8e9af98f63778def44a8e731472702e8a309d6a37d6f18cef93250700d42f mcr.microsoft.com/oss/kubernetes/kube-controller-manager:v1.23.15],SizeBytes:125006676,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/secrets-store/driver@sha256:c0d040a1c4fbfceb65663e31c09ea40f4f78e356437610cbc3fbb4bb409bd6f1 mcr.microsoft.com/oss/kubernetes-csi/secrets-store/driver:v0.0.19],SizeBytes:123229697,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-proxy@sha256:08c2bde6c8c0f9eda629bee3f5cd2e9c0cd991b377cdf3f94576dc1115ef80bd mcr.microsoft.com/oss/kubernetes/kube-proxy:v1.23.15],SizeBytes:112320997,},ContainerImage{Names:[mcr.microsoft.com/oss/azure/secrets-store/provider-azure@sha256:6f67f3d0c7cdde5702f8ce7f101b6519daa0237f0c34fecb7c058b6af8c22ad1 mcr.microsoft.com/oss/azure/secrets-store/provider-azure:0.0.12],SizeBytes:101061355,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/autoscaler/cluster-autoscaler@sha256:6f0c680d375c62e74351f8ff3ed6ddb9b72ca759e0645c329b95f64264654a6d mcr.microsoft.com/oss/kubernetes/autoscaler/cluster-autoscaler:v1.22.1],SizeBytes:99962810,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-addon-manager@sha256:32e2836018c96e73533bd4642fe438e465b81dcbfa8b7b61935a6f4d0246c7ae mcr.microsoft.com/oss/kubernetes/kube-addon-manager:v9.1.3],SizeBytes:86832059,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-addon-manager@sha256:92c2c5aad9012ee32d2a43a74966cc0adc6ccb1705ad15abb10485ecf406d88b mcr.microsoft.com/oss/kubernetes/kube-addon-manager:v9.1.5],SizeBytes:84094027,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/metrics-server@sha256:1ef9d57ce41ffcc328b92494c3bfafe401e0b9a1694a295301a1385337d52815 mcr.microsoft.com/oss/kubernetes/metrics-server:v0.5.2],SizeBytes:64327621,},ContainerImage{Names:[mcr.microsoft.com/oss/nvidia/k8s-device-plugin@sha256:0f5b52bf28239234e831697d96db63ac03cde70fe68058f964504ab7564ee810 mcr.microsoft.com/oss/nvidia/k8s-device-plugin:1.0.0-beta6],SizeBytes:64160241,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-provisioner@sha256:2f9a214fe823b4b5099e7cff4f4d16e7e4298cd086478880845de8858d49a3b3 mcr.microsoft.com/oss/kubernetes-csi/csi-provisioner:v3.2.0],SizeBytes:59035449,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-provisioner@sha256:e9ddadc44ba87a4a27f67e54760a14f9986885b534b3dff170a14eae1e35d213 mcr.microsoft.com/oss/kubernetes-csi/csi-provisioner:v3.0.0],SizeBytes:56881280,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-resizer@sha256:544e74bd67c649fd49500e195ff4a4ee675cfd26768574262dc6fa0250373d59 mcr.microsoft.com/oss/kubernetes-csi/csi-resizer:v1.5.0],SizeBytes:56173099,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-attacher@sha256:7e5af2ed16e053822e58f6576423c0bb77e59050c3698986f319d257b4551023 mcr.microsoft.com/oss/kubernetes-csi/csi-attacher:v3.5.0],SizeBytes:55590278,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-snapshotter@sha256:a889e925e15f9423f7842f1b769f64cbcf6a20b6956122836fc835cf22d9073f mcr.microsoft.com/oss/kubernetes-csi/csi-snapshotter:v5.0.1],SizeBytes:55314459,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-resizer@sha256:c5bb71ceaac60b1a4b58739fa07b709f6248c452ff6272a384d2f7648895a750 mcr.microsoft.com/oss/kubernetes-csi/csi-resizer:v1.3.0],SizeBytes:54313772,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-snapshotter@sha256:61849a026511cf332c87d73d0a7aed803b510c3ede197ec755389686d490de72 mcr.microsoft.com/oss/kubernetes-csi/csi-snapshotter:v4.2.1],SizeBytes:54210936,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-attacher@sha256:6b41e7153ebdfdc1501aa65184624bc15fd33a52d93f88ec3a758d0f8c9b8c10 mcr.microsoft.com/oss/kubernetes-csi/csi-attacher:v3.3.0],SizeBytes:53842561,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-scheduler@sha256:6dc978765fd2478cd9d7ba5ee02d7e4860f37a47d5067d5b6dc777a1bfd4e76c mcr.microsoft.com/oss/kubernetes/kube-scheduler:v1.23.15],SizeBytes:53494612,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/snapshot-controller@sha256:8c3fc3c2667004ad6bbdf723bb64c5da66a5cb8b11d4ee59b67179b686223b13 mcr.microsoft.com/oss/kubernetes-csi/snapshot-controller:v5.0.1],SizeBytes:52732401,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/snapshot-controller@sha256:be5a8dc1f990828f653c77bd0a0f1bbd13197c3019f6e1d99d590389bac36705 mcr.microsoft.com/oss/kubernetes-csi/snapshot-controller:v4.2.1],SizeBytes:51575245,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager@sha256:949db4eea46ea31f9afa0b49ad0b55f7a31f91aa03b408e0ac9b7720c8ef7d29 mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager:v1.23.11],SizeBytes:51376235,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager@sha256:72fbc9d17947d1a54c09a16671e6810fd7c0d8f09c0502fdbdd212c2192451a7 mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager:v1.1.14],SizeBytes:51044459,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager@sha256:075ea1f8270312350f1396ab6677251e803e61a523822d5abfa5e6acd180cfab mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager:v1.23.11],SizeBytes:50806891,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager@sha256:eb7d329559d954093f7a234bc0ff322764de48a0b2a1007229032fb53e7bb9eb mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager:v1.24.0],SizeBytes:49070187,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager@sha256:2dfff4dc10f6d25b0a34c01bf4d166540cbf1ff2bccd52635350039138eb23c4 mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager:v1.0.18],SizeBytes:48775275,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager@sha256:3e23b9ab6bfee42a1751019c21a60aa474964bd4cba00cbed5055430424242cc mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager:v1.24.0],SizeBytes:48500843,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager@sha256:acec9ee6626c2cdea89107ba191ae36e6f60339c9b3b3215b5a59bf0e2e730ba mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager:v1.1.14],SizeBytes:47853675,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/coredns@sha256:f873bf7f0928461efe10697fa76cf0ad7a1ae3041c5b57b50dd3d0b72d273f8c mcr.microsoft.com/oss/kubernetes/coredns:1.8.6],SizeBytes:46804601,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager@sha256:759ac7020e982884b99121583bd577658b5663b1c1ed69746356aa6ce9879da4 mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager:v1.0.18],SizeBytes:45850731,},ContainerImage{Names:[mcr.microsoft.com/oss/etcd-io/etcd@sha256:cf587862e3f1b6fa4d9a2565520a34f164bdf72c50f37af8c3c668160593246e mcr.microsoft.com/oss/etcd-io/etcd:v3.3.25],SizeBytes:41832119,},ContainerImage{Names:[mcr.microsoft.com/k8s/aad-pod-identity/mic@sha256:bd9465be94966b9a917e1e3904fa5e63dd91772ccadf304e18ffd8e4ad8ccedd mcr.microsoft.com/k8s/aad-pod-identity/mic:1.6.1],SizeBytes:41374894,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/autoscaler/cluster-proportional-autoscaler@sha256:c849d75d61943ce7f51b4c049f1a79d19a08253966c8f49c4cfb6414cc33db8b mcr.microsoft.com/oss/kubernetes/autoscaler/cluster-proportional-autoscaler:1.8.5],SizeBytes:40661903,},ContainerImage{Names:[mcr.microsoft.com/k8s/aad-pod-identity/nmi@sha256:02128fefcdb7593ac53fc342e2c53a0fc6fabd813036bf60457bf43cc2940116 mcr.microsoft.com/k8s/aad-pod-identity/nmi:1.6.1],SizeBytes:38007982,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/ip-masq-agent@sha256:c05561241a4a87f5526ab093746558f43ee7754a4a5911a6264819e252515ea0 mcr.microsoft.com/oss/kubernetes/ip-masq-agent:v2.8.0],SizeBytes:34483129,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:921f301c44dda06a325164accf22e78ecc570b5c7d9d6ee4c66bd8cbb2b60b9a mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.16],SizeBytes:26970670,},ContainerImage{Names:[mcr.microsoft.com/k8s/kms/keyvault@sha256:1a27e175f8c125209e32d2957b5509fe20757bd8cb309ff9da598799b56326fb mcr.microsoft.com/k8s/kms/keyvault:v0.0.10],SizeBytes:23077387,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-node-driver-registrar@sha256:2fbd1e1a0538a06f2061afd45975df70c942654aa7f86e920720169ee439c2d6 mcr.microsoft.com/oss/kubernetes-csi/csi-node-driver-registrar:v2.5.1],SizeBytes:19743433,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-node-driver-registrar@sha256:dbec3a8166686b09b242176ab5b99e993da4126438bbce68147c3fd654f35662 mcr.microsoft.com/oss/kubernetes-csi/csi-node-driver-registrar:v2.4.0],SizeBytes:19547289,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/livenessprobe@sha256:31547791294872570393470991481c2477a311031d3a03e0ae54eb164347dc34 mcr.microsoft.com/oss/kubernetes-csi/livenessprobe:v2.7.0],SizeBytes:17857485,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/livenessprobe@sha256:c96a6255c42766f6b8bb1a7cda02b0060ab1b20b2e2dafcc64ec09e7646745a6 mcr.microsoft.com/oss/kubernetes-csi/livenessprobe:v2.5.0],SizeBytes:17573341,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:16028126,},ContainerImage{Names:[mcr.microsoft.com/oss/busybox/busybox@sha256:582a641242b49809af3a1a522f9aae8c3f047d1c6ca1dd9d8cdabd349e45b1a9 mcr.microsoft.com/oss/busybox/busybox:1.33.1],SizeBytes:1235829,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8],SizeBytes:1159025,},},VolumesInUse:[kubernetes.io/csi/test.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-iei3lcrr/providers/Microsoft.Compute/disks/pvc-d50823a8-8f4a-4a25-bf58-4b7def955165],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/test.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-iei3lcrr/providers/Microsoft.Compute/disks/pvc-d50823a8-8f4a-4a25-bf58-4b7def955165,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/test.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-iei3lcrr/providers/Microsoft.Compute/disks/pvc-ead47477-2e82-42be-b42b-1d3a21778bbe,DevicePath:,},},Config:nil,},} Jan 19 02:34:50.228: INFO: ... skipping 115 lines ... [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow] [90mtest/e2e/storage/framework/testsuite.go:50[0m [91m[1mshould concurrently access the single read-only volume from pods on the same node [Measurement][0m [90mtest/e2e/storage/testsuites/multivolume.go:423[0m [91mJan 19 02:34:49.584: while cleanup resource Unexpected error: <errors.aggregate | len:1, cap:1>: [ [ { msg: "persistent Volume pvc-f7f1e685-5e57-4dc7-9d6e-c0440a3db932 not deleted by dynamic provisioner: PersistentVolume pvc-f7f1e685-5e57-4dc7-9d6e-c0440a3db932 still exists within 5m0s", err: { s: "PersistentVolume pvc-f7f1e685-5e57-4dc7-9d6e-c0440a3db932 still exists within 5m0s", ... skipping 3 lines ... ] persistent Volume pvc-f7f1e685-5e57-4dc7-9d6e-c0440a3db932 not deleted by dynamic provisioner: PersistentVolume pvc-f7f1e685-5e57-4dc7-9d6e-c0440a3db932 still exists within 5m0s occurred[0m test/e2e/storage/testsuites/multivolume.go:129 [90m------------------------------[0m {"msg":"FAILED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow] should concurrently access the single read-only volume from pods on the same node","total":30,"completed":5,"skipped":385,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow] should concurrently access the single read-only volume from pods on the same node"]} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Dynamic PV (ext4)] multiVolume [Slow][0m [1mshould access to two volumes with the same volume mode and retain data across pod recreation on the same node[0m [37mtest/e2e/storage/testsuites/multivolume.go:138[0m ... skipping 226 lines ... Jan 19 02:35:09.896: INFO: Deleting PersistentVolumeClaim "test.csi.azure.com665v5" Jan 19 02:35:10.005: INFO: Waiting up to 5m0s for PersistentVolume pvc-22a63d2a-fae2-4d68-9a13-2485b715c31e to get deleted Jan 19 02:35:10.112: INFO: PersistentVolume pvc-22a63d2a-fae2-4d68-9a13-2485b715c31e found and phase=Released (107.211811ms) Jan 19 02:35:15.220: INFO: PersistentVolume pvc-22a63d2a-fae2-4d68-9a13-2485b715c31e found and phase=Released (5.215403479s) Jan 19 02:35:20.328: INFO: PersistentVolume pvc-22a63d2a-fae2-4d68-9a13-2485b715c31e was removed [1mSTEP[0m: Deleting sc Jan 19 02:35:20.436: FAIL: while cleanup resource Unexpected error: <errors.aggregate | len:1, cap:1>: [ [ { msg: "persistent Volume pvc-02d2e695-f4ec-4283-b74e-8b328ec3a2c9 not deleted by dynamic provisioner: PersistentVolume pvc-02d2e695-f4ec-4283-b74e-8b328ec3a2c9 still exists within 5m0s", err: { s: "PersistentVolume pvc-02d2e695-f4ec-4283-b74e-8b328ec3a2c9 still exists within 5m0s", ... skipping 33 lines ... Jan 19 02:35:20.650: INFO: At 2023-01-19 02:29:04 +0000 UTC - event for pod-789ad6fd-afcb-4eee-89fd-3667052742ab: {kubelet k8s-agentpool1-29517166-vmss000002} FailedMount: Unable to attach or mount volumes: unmounted volumes=[volume1 volume2], unattached volumes=[volume1 volume2 kube-api-access-47szw]: timed out waiting for the condition Jan 19 02:35:20.650: INFO: At 2023-01-19 02:29:28 +0000 UTC - event for pod-789ad6fd-afcb-4eee-89fd-3667052742ab: {kubelet k8s-agentpool1-29517166-vmss000002} Started: Started container write-pod Jan 19 02:35:20.650: INFO: At 2023-01-19 02:29:28 +0000 UTC - event for pod-789ad6fd-afcb-4eee-89fd-3667052742ab: {kubelet k8s-agentpool1-29517166-vmss000002} Pulling: Pulling image "k8s.gcr.io/e2e-test-images/busybox:1.29-2" Jan 19 02:35:20.650: INFO: At 2023-01-19 02:29:28 +0000 UTC - event for pod-789ad6fd-afcb-4eee-89fd-3667052742ab: {kubelet k8s-agentpool1-29517166-vmss000002} Created: Created container write-pod Jan 19 02:35:20.650: INFO: At 2023-01-19 02:29:28 +0000 UTC - event for pod-789ad6fd-afcb-4eee-89fd-3667052742ab: {kubelet k8s-agentpool1-29517166-vmss000002} Pulled: Successfully pulled image "k8s.gcr.io/e2e-test-images/busybox:1.29-2" in 335.769592ms Jan 19 02:35:20.650: INFO: At 2023-01-19 02:29:40 +0000 UTC - event for pod-789ad6fd-afcb-4eee-89fd-3667052742ab: {kubelet k8s-agentpool1-29517166-vmss000002} Killing: Stopping container write-pod Jan 19 02:35:20.650: INFO: At 2023-01-19 02:29:42 +0000 UTC - event for pod-789ad6fd-afcb-4eee-89fd-3667052742ab: {kubelet k8s-agentpool1-29517166-vmss000002} FailedKillPod: error killing pod: failed to "KillContainer" for "write-pod" with KillContainerError: "rpc error: code = Unknown desc = Error response from daemon: No such container: 2084698d1edf56e2c4d3e07327be6de3acfbf2f3638fda1fa19288dae05f3a07" Jan 19 02:35:20.650: INFO: At 2023-01-19 02:29:43 +0000 UTC - event for pod-53120dd7-bfd4-44e2-8730-c6190cafd916: {default-scheduler } Scheduled: Successfully assigned multivolume-257/pod-53120dd7-bfd4-44e2-8730-c6190cafd916 to k8s-agentpool1-29517166-vmss000002 Jan 19 02:35:20.650: INFO: At 2023-01-19 02:29:49 +0000 UTC - event for pod-53120dd7-bfd4-44e2-8730-c6190cafd916: {kubelet k8s-agentpool1-29517166-vmss000002} Started: Started container write-pod Jan 19 02:35:20.650: INFO: At 2023-01-19 02:29:49 +0000 UTC - event for pod-53120dd7-bfd4-44e2-8730-c6190cafd916: {kubelet k8s-agentpool1-29517166-vmss000002} Created: Created container write-pod Jan 19 02:35:20.650: INFO: At 2023-01-19 02:29:49 +0000 UTC - event for pod-53120dd7-bfd4-44e2-8730-c6190cafd916: {kubelet k8s-agentpool1-29517166-vmss000002} Pulled: Successfully pulled image "k8s.gcr.io/e2e-test-images/busybox:1.29-2" in 319.589611ms Jan 19 02:35:20.650: INFO: At 2023-01-19 02:29:49 +0000 UTC - event for pod-53120dd7-bfd4-44e2-8730-c6190cafd916: {kubelet k8s-agentpool1-29517166-vmss000002} Pulling: Pulling image "k8s.gcr.io/e2e-test-images/busybox:1.29-2" Jan 19 02:35:20.650: INFO: At 2023-01-19 02:30:03 +0000 UTC - event for pod-53120dd7-bfd4-44e2-8730-c6190cafd916: {kubelet k8s-agentpool1-29517166-vmss000002} Killing: Stopping container write-pod ... skipping 119 lines ... [Testpattern: Dynamic PV (ext4)] multiVolume [Slow] [90mtest/e2e/storage/framework/testsuite.go:50[0m [91m[1mshould access to two volumes with the same volume mode and retain data across pod recreation on the same node [Measurement][0m [90mtest/e2e/storage/testsuites/multivolume.go:138[0m [91mJan 19 02:35:20.436: while cleanup resource Unexpected error: <errors.aggregate | len:1, cap:1>: [ [ { msg: "persistent Volume pvc-02d2e695-f4ec-4283-b74e-8b328ec3a2c9 not deleted by dynamic provisioner: PersistentVolume pvc-02d2e695-f4ec-4283-b74e-8b328ec3a2c9 still exists within 5m0s", err: { s: "PersistentVolume pvc-02d2e695-f4ec-4283-b74e-8b328ec3a2c9 still exists within 5m0s", ... skipping 3 lines ... ] persistent Volume pvc-02d2e695-f4ec-4283-b74e-8b328ec3a2c9 not deleted by dynamic provisioner: PersistentVolume pvc-02d2e695-f4ec-4283-b74e-8b328ec3a2c9 still exists within 5m0s occurred[0m test/e2e/storage/testsuites/multivolume.go:129 [90m------------------------------[0m {"msg":"FAILED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext4)] multiVolume [Slow] should access to two volumes with the same volume mode and retain data across pod recreation on the same node","total":33,"completed":7,"skipped":609,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext4)] multiVolume [Slow] should access to two volumes with the same volume mode and retain data across pod recreation on the same node"]} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [BeforeEach] [Testpattern: CSI Ephemeral-volume (default fs)] ephemeral test/e2e/storage/framework/testsuite.go:51 Jan 19 02:35:24.058: INFO: Driver "test.csi.azure.com" does not support volume type "CSIInlineVolume" - skipping ... skipping 38 lines ... Jan 19 02:25:08.167: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.com2xh6h] to have phase Bound Jan 19 02:25:08.274: INFO: PersistentVolumeClaim test.csi.azure.com2xh6h found but phase is Pending instead of Bound. Jan 19 02:25:10.383: INFO: PersistentVolumeClaim test.csi.azure.com2xh6h found but phase is Pending instead of Bound. Jan 19 02:25:12.491: INFO: PersistentVolumeClaim test.csi.azure.com2xh6h found and phase=Bound (4.323440936s) [1mSTEP[0m: [init] starting a pod to use the claim [1mSTEP[0m: [init] check pod success Jan 19 02:25:12.919: INFO: Waiting up to 15m0s for pod "pvc-snapshottable-tester-rs6pj" in namespace "snapshotting-4953" to be "Succeeded or Failed" Jan 19 02:25:13.025: INFO: Pod "pvc-snapshottable-tester-rs6pj": Phase="Pending", Reason="", readiness=false. Elapsed: 106.773913ms Jan 19 02:25:15.133: INFO: Pod "pvc-snapshottable-tester-rs6pj": Phase="Pending", Reason="", readiness=false. Elapsed: 2.213913608s Jan 19 02:25:17.241: INFO: Pod "pvc-snapshottable-tester-rs6pj": Phase="Pending", Reason="", readiness=false. Elapsed: 4.321848529s Jan 19 02:25:19.348: INFO: Pod "pvc-snapshottable-tester-rs6pj": Phase="Pending", Reason="", readiness=false. Elapsed: 6.429584814s Jan 19 02:25:21.468: INFO: Pod "pvc-snapshottable-tester-rs6pj": Phase="Pending", Reason="", readiness=false. Elapsed: 8.549553979s Jan 19 02:25:23.577: INFO: Pod "pvc-snapshottable-tester-rs6pj": Phase="Pending", Reason="", readiness=false. Elapsed: 10.658117845s ... skipping 5 lines ... Jan 19 02:25:36.227: INFO: Pod "pvc-snapshottable-tester-rs6pj": Phase="Pending", Reason="", readiness=false. Elapsed: 23.307897019s Jan 19 02:25:38.334: INFO: Pod "pvc-snapshottable-tester-rs6pj": Phase="Pending", Reason="", readiness=false. Elapsed: 25.415225026s Jan 19 02:25:40.442: INFO: Pod "pvc-snapshottable-tester-rs6pj": Phase="Pending", Reason="", readiness=false. Elapsed: 27.523803268s Jan 19 02:25:42.551: INFO: Pod "pvc-snapshottable-tester-rs6pj": Phase="Pending", Reason="", readiness=false. Elapsed: 29.632617298s Jan 19 02:25:44.659: INFO: Pod "pvc-snapshottable-tester-rs6pj": Phase="Succeeded", Reason="", readiness=false. Elapsed: 31.740090012s [1mSTEP[0m: Saw pod success Jan 19 02:25:44.659: INFO: Pod "pvc-snapshottable-tester-rs6pj" satisfied condition "Succeeded or Failed" [1mSTEP[0m: [init] checking the claim Jan 19 02:25:44.766: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.com2xh6h] to have phase Bound Jan 19 02:25:44.872: INFO: PersistentVolumeClaim test.csi.azure.com2xh6h found and phase=Bound (105.992116ms) [1mSTEP[0m: [init] checking the PV [1mSTEP[0m: [init] deleting the pod Jan 19 02:25:45.195: INFO: Pod pvc-snapshottable-tester-rs6pj has the following logs: ... skipping 32 lines ... Jan 19 02:25:51.361: INFO: WaitUntil finished successfully after 109.938097ms [1mSTEP[0m: getting the snapshot and snapshot content [1mSTEP[0m: checking the snapshot [1mSTEP[0m: checking the SnapshotContent [1mSTEP[0m: Modifying source data test [1mSTEP[0m: modifying the data in the source PVC Jan 19 02:25:51.900: INFO: Waiting up to 15m0s for pod "pvc-snapshottable-data-tester-fb7bt" in namespace "snapshotting-4953" to be "Succeeded or Failed" Jan 19 02:25:52.007: INFO: Pod "pvc-snapshottable-data-tester-fb7bt": Phase="Pending", Reason="", readiness=false. Elapsed: 106.833047ms Jan 19 02:25:54.116: INFO: Pod "pvc-snapshottable-data-tester-fb7bt": Phase="Pending", Reason="", readiness=false. Elapsed: 2.216049172s Jan 19 02:25:56.223: INFO: Pod "pvc-snapshottable-data-tester-fb7bt": Phase="Pending", Reason="", readiness=false. Elapsed: 4.323260885s Jan 19 02:25:58.331: INFO: Pod "pvc-snapshottable-data-tester-fb7bt": Phase="Pending", Reason="", readiness=false. Elapsed: 6.431112794s Jan 19 02:26:00.440: INFO: Pod "pvc-snapshottable-data-tester-fb7bt": Phase="Pending", Reason="", readiness=false. Elapsed: 8.539983084s Jan 19 02:26:02.548: INFO: Pod "pvc-snapshottable-data-tester-fb7bt": Phase="Pending", Reason="", readiness=false. Elapsed: 10.648452134s ... skipping 66 lines ... Jan 19 02:28:23.836: INFO: Pod "pvc-snapshottable-data-tester-fb7bt": Phase="Pending", Reason="", readiness=false. Elapsed: 2m31.935625928s Jan 19 02:28:25.944: INFO: Pod "pvc-snapshottable-data-tester-fb7bt": Phase="Pending", Reason="", readiness=false. Elapsed: 2m34.044343744s Jan 19 02:28:28.052: INFO: Pod "pvc-snapshottable-data-tester-fb7bt": Phase="Pending", Reason="", readiness=false. Elapsed: 2m36.152286371s Jan 19 02:28:30.160: INFO: Pod "pvc-snapshottable-data-tester-fb7bt": Phase="Pending", Reason="", readiness=false. Elapsed: 2m38.260160728s Jan 19 02:28:32.269: INFO: Pod "pvc-snapshottable-data-tester-fb7bt": Phase="Succeeded", Reason="", readiness=false. Elapsed: 2m40.368647388s [1mSTEP[0m: Saw pod success Jan 19 02:28:32.269: INFO: Pod "pvc-snapshottable-data-tester-fb7bt" satisfied condition "Succeeded or Failed" Jan 19 02:28:32.509: INFO: Pod pvc-snapshottable-data-tester-fb7bt has the following logs: Jan 19 02:28:32.509: INFO: Deleting pod "pvc-snapshottable-data-tester-fb7bt" in namespace "snapshotting-4953" Jan 19 02:28:32.619: INFO: Wait up to 5m0s for pod "pvc-snapshottable-data-tester-fb7bt" to be fully deleted [1mSTEP[0m: creating a pvc from the snapshot [1mSTEP[0m: starting a pod to use the snapshot Jan 19 02:29:53.158: INFO: Running '/usr/local/bin/kubectl --server=https://kubetest-iei3lcrr.westeurope.cloudapp.azure.com --kubeconfig=/root/tmp1374176058/kubeconfig/kubeconfig.westeurope.json --namespace=snapshotting-4953 exec restored-pvc-tester-7dfbr --namespace=snapshotting-4953 -- cat /mnt/test/data' ... skipping 85 lines ... Jan 19 02:35:02.399: INFO: PersistentVolume pvc-78b85abd-0251-4a7c-a888-927ee9fe65c4 found and phase=Released (4m36.012875196s) Jan 19 02:35:07.507: INFO: PersistentVolume pvc-78b85abd-0251-4a7c-a888-927ee9fe65c4 found and phase=Released (4m41.120751501s) Jan 19 02:35:12.615: INFO: PersistentVolume pvc-78b85abd-0251-4a7c-a888-927ee9fe65c4 found and phase=Released (4m46.22875049s) Jan 19 02:35:17.722: INFO: PersistentVolume pvc-78b85abd-0251-4a7c-a888-927ee9fe65c4 found and phase=Released (4m51.336494379s) Jan 19 02:35:22.830: INFO: PersistentVolume pvc-78b85abd-0251-4a7c-a888-927ee9fe65c4 found and phase=Released (4m56.44446545s) [1mSTEP[0m: Deleting sc Jan 19 02:35:27.960: FAIL: Unexpected error: <errors.aggregate | len:1, cap:1>: [ { msg: "persistent Volume pvc-78b85abd-0251-4a7c-a888-927ee9fe65c4 not deleted by dynamic provisioner: PersistentVolume pvc-78b85abd-0251-4a7c-a888-927ee9fe65c4 still exists within 5m0s", err: { s: "PersistentVolume pvc-78b85abd-0251-4a7c-a888-927ee9fe65c4 still exists within 5m0s", }, ... skipping 180 lines ... [90mtest/e2e/storage/testsuites/snapshottable.go:113[0m [91m[1m [It][0m [90mtest/e2e/storage/testsuites/snapshottable.go:176[0m should check snapshot fields, check restore correctly works after modifying source data, check deletion (persistent) [90mtest/e2e/storage/testsuites/snapshottable.go:278[0m [91mJan 19 02:35:27.960: Unexpected error: <errors.aggregate | len:1, cap:1>: [ { msg: "persistent Volume pvc-78b85abd-0251-4a7c-a888-927ee9fe65c4 not deleted by dynamic provisioner: PersistentVolume pvc-78b85abd-0251-4a7c-a888-927ee9fe65c4 still exists within 5m0s", err: { s: "PersistentVolume pvc-78b85abd-0251-4a7c-a888-927ee9fe65c4 still exists within 5m0s", }, }, ] persistent Volume pvc-78b85abd-0251-4a7c-a888-927ee9fe65c4 not deleted by dynamic provisioner: PersistentVolume pvc-78b85abd-0251-4a7c-a888-927ee9fe65c4 still exists within 5m0s occurred[0m test/e2e/storage/testsuites/snapshottable.go:142 [90m------------------------------[0m {"msg":"FAILED External Storage [Driver: test.csi.azure.com] [Testpattern: Pre-provisioned Snapshot (delete policy)] snapshottable[Feature:VolumeSnapshotDataSource] volume snapshot controller should check snapshot fields, check restore correctly works after modifying source data, check deletion (persistent)","total":31,"completed":6,"skipped":545,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Pre-provisioned Snapshot (delete policy)] snapshottable[Feature:VolumeSnapshotDataSource] volume snapshot controller should check snapshot fields, check restore correctly works after modifying source data, check deletion (persistent)"]} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [BeforeEach] [Testpattern: Pre-provisioned PV (block volmode)] multiVolume [Slow] test/e2e/storage/framework/testsuite.go:51 Jan 19 02:35:31.599: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping ... skipping 90 lines ... [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Dynamic PV (default fs)] provisioning[0m [1mshould provision storage with pvc data source in parallel [Slow][0m [37mtest/e2e/storage/testsuites/provisioning.go:459[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext4)] multiVolume [Slow] should access to two volumes with different volume mode and retain data across pod recreation on different node","total":32,"completed":6,"skipped":623,"failed":0} [BeforeEach] [Testpattern: Dynamic PV (default fs)] provisioning test/e2e/storage/framework/testsuite.go:51 [BeforeEach] [Testpattern: Dynamic PV (default fs)] provisioning test/e2e/framework/framework.go:187 [1mSTEP[0m: Creating a kubernetes client Jan 19 02:24:35.738: INFO: >>> kubeConfig: /root/tmp1374176058/kubeconfig/kubeconfig.westeurope.json ... skipping 508 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (default fs)] provisioning [90mtest/e2e/storage/framework/testsuite.go:50[0m should provision storage with pvc data source in parallel [Slow] [90mtest/e2e/storage/testsuites/provisioning.go:459[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] provisioning should provision storage with pvc data source in parallel [Slow]","total":32,"completed":7,"skipped":623,"failed":0} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Dynamic PV (block volmode)] multiVolume [Slow][0m [1mshould concurrently access the single volume from pods on the same node[0m [37mtest/e2e/storage/testsuites/multivolume.go:298[0m ... skipping 15 lines ... [1mSTEP[0m: creating a claim Jan 19 02:29:39.870: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comt45nd] to have phase Bound Jan 19 02:29:39.977: INFO: PersistentVolumeClaim test.csi.azure.comt45nd found but phase is Pending instead of Bound. Jan 19 02:29:42.084: INFO: PersistentVolumeClaim test.csi.azure.comt45nd found but phase is Pending instead of Bound. Jan 19 02:29:44.191: INFO: PersistentVolumeClaim test.csi.azure.comt45nd found and phase=Bound (4.320744356s) [1mSTEP[0m: Creating pod1 with a volume on {Name: Selector:map[] Affinity:nil} Jan 19 02:34:44.842: FAIL: Unexpected error: <*errors.errorString | 0xc0026ea780>: { s: "pod \"pod-86cc9afc-bf62-45c5-a4f5-c7dd3768e326\" is not Running: timed out waiting for the condition", } pod "pod-86cc9afc-bf62-45c5-a4f5-c7dd3768e326" is not Running: timed out waiting for the condition occurred ... skipping 34 lines ... Jan 19 02:35:44.687: INFO: At 2023-01-19 02:29:39 +0000 UTC - event for test.csi.azure.comt45nd: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "test.csi.azure.com" or manually created by system administrator Jan 19 02:35:44.687: INFO: At 2023-01-19 02:29:39 +0000 UTC - event for test.csi.azure.comt45nd: {test.csi.azure.com_k8s-agentpool1-29517166-vmss000000_53759c0c-6de5-4277-9d52-f675b7576acb } Provisioning: External provisioner is provisioning volume for claim "multivolume-4004/test.csi.azure.comt45nd" Jan 19 02:35:44.687: INFO: At 2023-01-19 02:29:42 +0000 UTC - event for test.csi.azure.comt45nd: {test.csi.azure.com_k8s-agentpool1-29517166-vmss000000_53759c0c-6de5-4277-9d52-f675b7576acb } ProvisioningSucceeded: Successfully provisioned volume pvc-d50823a8-8f4a-4a25-bf58-4b7def955165 Jan 19 02:35:44.687: INFO: At 2023-01-19 02:29:44 +0000 UTC - event for pod-86cc9afc-bf62-45c5-a4f5-c7dd3768e326: {default-scheduler } Scheduled: Successfully assigned multivolume-4004/pod-86cc9afc-bf62-45c5-a4f5-c7dd3768e326 to k8s-agentpool1-29517166-vmss000000 Jan 19 02:35:44.687: INFO: At 2023-01-19 02:29:45 +0000 UTC - event for pod-86cc9afc-bf62-45c5-a4f5-c7dd3768e326: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-d50823a8-8f4a-4a25-bf58-4b7def955165" Jan 19 02:35:44.687: INFO: At 2023-01-19 02:31:47 +0000 UTC - event for pod-86cc9afc-bf62-45c5-a4f5-c7dd3768e326: {kubelet k8s-agentpool1-29517166-vmss000000} FailedMount: Unable to attach or mount volumes: unmounted volumes=[volume1], unattached volumes=[kube-api-access-gz6lw volume1]: timed out waiting for the condition Jan 19 02:35:44.687: INFO: At 2023-01-19 02:31:48 +0000 UTC - event for pod-86cc9afc-bf62-45c5-a4f5-c7dd3768e326: {kubelet k8s-agentpool1-29517166-vmss000000} FailedMapVolume: MapVolume.SetUpDevice failed for volume "pvc-d50823a8-8f4a-4a25-bf58-4b7def955165" : rpc error: code = Internal desc = failed to wait for volume (/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-iei3lcrr/providers/Microsoft.Compute/disks/pvc-d50823a8-8f4a-4a25-bf58-4b7def955165) to be attached to node (k8s-agentpool1-29517166-vmss000000): context deadline exceeded Jan 19 02:35:44.687: INFO: At 2023-01-19 02:33:49 +0000 UTC - event for pod-86cc9afc-bf62-45c5-a4f5-c7dd3768e326: {kubelet k8s-agentpool1-29517166-vmss000000} FailedMapVolume: MapVolume.SetUpDevice failed for volume "pvc-d50823a8-8f4a-4a25-bf58-4b7def955165" : rpc error: code = DeadlineExceeded desc = context deadline exceeded Jan 19 02:35:44.687: INFO: At 2023-01-19 02:34:05 +0000 UTC - event for pod-86cc9afc-bf62-45c5-a4f5-c7dd3768e326: {kubelet k8s-agentpool1-29517166-vmss000000} FailedMount: Unable to attach or mount volumes: unmounted volumes=[volume1], unattached volumes=[volume1 kube-api-access-gz6lw]: timed out waiting for the condition Jan 19 02:35:44.687: INFO: At 2023-01-19 02:34:44 +0000 UTC - event for pod-86cc9afc-bf62-45c5-a4f5-c7dd3768e326: {kubelet k8s-agentpool1-29517166-vmss000000} FailedMapVolume: MapVolume.SetUpDevice failed for volume "pvc-d50823a8-8f4a-4a25-bf58-4b7def955165" : rpc error: code = Internal desc = failed to wait for volume (/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-iei3lcrr/providers/Microsoft.Compute/disks/pvc-d50823a8-8f4a-4a25-bf58-4b7def955165) to be attached to node (k8s-agentpool1-29517166-vmss000000): rpc error: code = Unknown desc = context deadline exceeded Jan 19 02:35:44.687: INFO: At 2023-01-19 02:34:48 +0000 UTC - event for pod-86cc9afc-bf62-45c5-a4f5-c7dd3768e326: {kubelet k8s-agentpool1-29517166-vmss000000} SuccessfulMountVolume: MapVolume.MapPodDevice succeeded for volume "pvc-d50823a8-8f4a-4a25-bf58-4b7def955165" globalMapPath "/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/pvc-d50823a8-8f4a-4a25-bf58-4b7def955165/dev" Jan 19 02:35:44.687: INFO: At 2023-01-19 02:34:48 +0000 UTC - event for pod-86cc9afc-bf62-45c5-a4f5-c7dd3768e326: {kubelet k8s-agentpool1-29517166-vmss000000} SuccessfulMountVolume: MapVolume.MapPodDevice succeeded for volume "pvc-d50823a8-8f4a-4a25-bf58-4b7def955165" volumeMapPath "/var/lib/kubelet/pods/06a3e360-61ed-409a-8fa4-523a314e5349/volumeDevices/kubernetes.io~csi" Jan 19 02:35:44.687: INFO: At 2023-01-19 02:34:49 +0000 UTC - event for pod-86cc9afc-bf62-45c5-a4f5-c7dd3768e326: {kubelet k8s-agentpool1-29517166-vmss000000} Started: Started container write-pod Jan 19 02:35:44.687: INFO: At 2023-01-19 02:34:49 +0000 UTC - event for pod-86cc9afc-bf62-45c5-a4f5-c7dd3768e326: {kubelet k8s-agentpool1-29517166-vmss000000} Created: Created container write-pod Jan 19 02:35:44.687: INFO: At 2023-01-19 02:34:49 +0000 UTC - event for pod-86cc9afc-bf62-45c5-a4f5-c7dd3768e326: {kubelet k8s-agentpool1-29517166-vmss000000} Pulling: Pulling image "k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.5" Jan 19 02:35:44.687: INFO: At 2023-01-19 02:34:49 +0000 UTC - event for pod-86cc9afc-bf62-45c5-a4f5-c7dd3768e326: {kubelet k8s-agentpool1-29517166-vmss000000} Pulled: Successfully pulled image "k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.5" in 297.271729ms ... skipping 123 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] [90mtest/e2e/storage/framework/testsuite.go:50[0m [91m[1mshould concurrently access the single volume from pods on the same node [Measurement][0m [90mtest/e2e/storage/testsuites/multivolume.go:298[0m [91mJan 19 02:34:44.842: Unexpected error: <*errors.errorString | 0xc0026ea780>: { s: "pod \"pod-86cc9afc-bf62-45c5-a4f5-c7dd3768e326\" is not Running: timed out waiting for the condition", } pod "pod-86cc9afc-bf62-45c5-a4f5-c7dd3768e326" is not Running: timed out waiting for the condition occurred[0m test/e2e/storage/testsuites/multivolume.go:577 [90m------------------------------[0m {"msg":"FAILED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] should concurrently access the single volume from pods on the same node","total":35,"completed":8,"skipped":846,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] should concurrently access the single volume from pods on the same node"]} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Dynamic PV (default fs)] volumes[0m [1mshould store data[0m [37mtest/e2e/storage/testsuites/volumes.go:161[0m ... skipping 121 lines ... Jan 19 02:34:54.210: INFO: Waiting for pod external-client to disappear Jan 19 02:34:54.318: INFO: Pod external-client still exists Jan 19 02:34:56.209: INFO: Waiting for pod external-client to disappear Jan 19 02:34:56.317: INFO: Pod external-client still exists Jan 19 02:34:58.209: INFO: Waiting for pod external-client to disappear Jan 19 02:34:58.316: INFO: Pod external-client no longer exists Jan 19 02:34:58.316: FAIL: Failed to create client pod: timed out waiting for the condition Full Stack Trace k8s.io/kubernetes/test/e2e/framework/volume.TestVolumeClient(...) test/e2e/framework/volume/fixtures.go:580 k8s.io/kubernetes/test/e2e/storage/testsuites.(*volumesTestSuite).DefineTests.func3() test/e2e/storage/testsuites/volumes.go:190 +0x525 ... skipping 27 lines ... Jan 19 02:35:44.833: INFO: At 2023-01-19 02:25:30 +0000 UTC - event for test.csi.azure.comwmhgd: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "test.csi.azure.com" or manually created by system administrator Jan 19 02:35:44.833: INFO: At 2023-01-19 02:25:30 +0000 UTC - event for test.csi.azure.comwmhgd: {test.csi.azure.com_k8s-agentpool1-29517166-vmss000000_53759c0c-6de5-4277-9d52-f675b7576acb } Provisioning: External provisioner is provisioning volume for claim "volume-1076/test.csi.azure.comwmhgd" Jan 19 02:35:44.833: INFO: At 2023-01-19 02:25:32 +0000 UTC - event for test.csi.azure.comwmhgd: {test.csi.azure.com_k8s-agentpool1-29517166-vmss000000_53759c0c-6de5-4277-9d52-f675b7576acb } ProvisioningSucceeded: Successfully provisioned volume pvc-b60a43d5-32ea-4dc5-94e9-b57828b77226 Jan 19 02:35:44.833: INFO: At 2023-01-19 02:25:34 +0000 UTC - event for external-injector: {default-scheduler } Scheduled: Successfully assigned volume-1076/external-injector to k8s-agentpool1-29517166-vmss000000 Jan 19 02:35:44.833: INFO: At 2023-01-19 02:25:39 +0000 UTC - event for external-injector: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-b60a43d5-32ea-4dc5-94e9-b57828b77226" Jan 19 02:35:44.833: INFO: At 2023-01-19 02:27:37 +0000 UTC - event for external-injector: {kubelet k8s-agentpool1-29517166-vmss000000} FailedMount: Unable to attach or mount volumes: unmounted volumes=[external-volume-0], unattached volumes=[kube-api-access-55jl4 external-volume-0]: timed out waiting for the condition Jan 19 02:35:44.833: INFO: At 2023-01-19 02:27:44 +0000 UTC - event for external-injector: {kubelet k8s-agentpool1-29517166-vmss000000} FailedMount: MountVolume.MountDevice failed for volume "pvc-b60a43d5-32ea-4dc5-94e9-b57828b77226" : rpc error: code = DeadlineExceeded desc = context deadline exceeded Jan 19 02:35:44.833: INFO: At 2023-01-19 02:28:28 +0000 UTC - event for external-injector: {kubelet k8s-agentpool1-29517166-vmss000000} Pulling: Pulling image "k8s.gcr.io/e2e-test-images/busybox:1.29-2" Jan 19 02:35:44.833: INFO: At 2023-01-19 02:28:29 +0000 UTC - event for external-injector: {kubelet k8s-agentpool1-29517166-vmss000000} Started: Started container external-injector Jan 19 02:35:44.833: INFO: At 2023-01-19 02:28:29 +0000 UTC - event for external-injector: {kubelet k8s-agentpool1-29517166-vmss000000} Created: Created container external-injector Jan 19 02:35:44.833: INFO: At 2023-01-19 02:28:29 +0000 UTC - event for external-injector: {kubelet k8s-agentpool1-29517166-vmss000000} Pulled: Successfully pulled image "k8s.gcr.io/e2e-test-images/busybox:1.29-2" in 322.382706ms Jan 19 02:35:44.833: INFO: At 2023-01-19 02:28:35 +0000 UTC - event for external-injector: {kubelet k8s-agentpool1-29517166-vmss000000} Killing: Stopping container external-injector Jan 19 02:35:44.833: INFO: At 2023-01-19 02:28:39 +0000 UTC - event for external-client: {default-scheduler } Scheduled: Successfully assigned volume-1076/external-client to k8s-agentpool1-29517166-vmss000001 Jan 19 02:35:44.833: INFO: At 2023-01-19 02:29:50 +0000 UTC - event for external-client: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-b60a43d5-32ea-4dc5-94e9-b57828b77226" Jan 19 02:35:44.833: INFO: At 2023-01-19 02:30:42 +0000 UTC - event for external-client: {kubelet k8s-agentpool1-29517166-vmss000001} FailedMount: Unable to attach or mount volumes: unmounted volumes=[external-volume-0], unattached volumes=[external-volume-0 kube-api-access-dx5n8]: timed out waiting for the condition Jan 19 02:35:44.833: INFO: At 2023-01-19 02:31:50 +0000 UTC - event for external-client: {kubelet k8s-agentpool1-29517166-vmss000001} FailedMount: MountVolume.MountDevice failed for volume "pvc-b60a43d5-32ea-4dc5-94e9-b57828b77226" : rpc error: code = DeadlineExceeded desc = context deadline exceeded Jan 19 02:35:44.833: INFO: At 2023-01-19 02:34:48 +0000 UTC - event for external-client: {kubelet k8s-agentpool1-29517166-vmss000001} FailedMount: MountVolume.MountDevice failed for volume "pvc-b60a43d5-32ea-4dc5-94e9-b57828b77226" : rpc error: code = Internal desc = failed to wait for volume (/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-iei3lcrr/providers/Microsoft.Compute/disks/pvc-b60a43d5-32ea-4dc5-94e9-b57828b77226) to be attached to node (k8s-agentpool1-29517166-vmss000001): rpc error: code = Unknown desc = context deadline exceeded Jan 19 02:35:44.833: INFO: At 2023-01-19 02:34:53 +0000 UTC - event for external-client: {kubelet k8s-agentpool1-29517166-vmss000001} Pulling: Pulling image "k8s.gcr.io/e2e-test-images/busybox:1.29-2" Jan 19 02:35:44.833: INFO: At 2023-01-19 02:34:53 +0000 UTC - event for external-client: {kubelet k8s-agentpool1-29517166-vmss000001} Started: Started container external-client Jan 19 02:35:44.833: INFO: At 2023-01-19 02:34:53 +0000 UTC - event for external-client: {kubelet k8s-agentpool1-29517166-vmss000001} Created: Created container external-client Jan 19 02:35:44.833: INFO: At 2023-01-19 02:34:53 +0000 UTC - event for external-client: {kubelet k8s-agentpool1-29517166-vmss000001} Pulled: Successfully pulled image "k8s.gcr.io/e2e-test-images/busybox:1.29-2" in 342.462928ms Jan 19 02:35:44.833: INFO: At 2023-01-19 02:34:54 +0000 UTC - event for external-client: {kubelet k8s-agentpool1-29517166-vmss000001} Killing: Stopping container external-client Jan 19 02:35:44.940: INFO: POD NODE PHASE GRACE CONDITIONS ... skipping 121 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (default fs)] volumes [90mtest/e2e/storage/framework/testsuite.go:50[0m [91m[1mshould store data [Measurement][0m [90mtest/e2e/storage/testsuites/volumes.go:161[0m [91mJan 19 02:34:58.316: Failed to create client pod: timed out waiting for the condition[0m test/e2e/framework/volume/fixtures.go:580 [90m------------------------------[0m {"msg":"FAILED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] volumes should store data","total":32,"completed":6,"skipped":251,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] volumes should store data"]} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [BeforeEach] [Testpattern: Pre-provisioned PV (default fs)] subPath test/e2e/storage/framework/testsuite.go:51 Jan 19 02:35:48.194: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping ... skipping 225 lines ... Jan 19 02:36:17.065: INFO: Waiting for pod external-client to disappear Jan 19 02:36:17.173: INFO: Pod external-client still exists Jan 19 02:36:19.064: INFO: Waiting for pod external-client to disappear Jan 19 02:36:19.172: INFO: Pod external-client still exists Jan 19 02:36:21.065: INFO: Waiting for pod external-client to disappear Jan 19 02:36:21.173: INFO: Pod external-client no longer exists Jan 19 02:36:21.173: FAIL: Failed to create client pod: timed out waiting for the condition Full Stack Trace k8s.io/kubernetes/test/e2e/framework/volume.TestVolumeClient(...) test/e2e/framework/volume/fixtures.go:580 k8s.io/kubernetes/test/e2e/storage/testsuites.(*volumesTestSuite).DefineTests.func3() test/e2e/storage/testsuites/volumes.go:190 +0x525 ... skipping 18 lines ... [1mSTEP[0m: Collecting events from namespace "volume-5841". [1mSTEP[0m: Found 16 events. Jan 19 02:36:31.829: INFO: At 2023-01-19 02:26:00 +0000 UTC - event for test.csi.azure.com2qn2j: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "test.csi.azure.com" or manually created by system administrator Jan 19 02:36:31.829: INFO: At 2023-01-19 02:26:00 +0000 UTC - event for test.csi.azure.com2qn2j: {test.csi.azure.com_k8s-agentpool1-29517166-vmss000000_53759c0c-6de5-4277-9d52-f675b7576acb } Provisioning: External provisioner is provisioning volume for claim "volume-5841/test.csi.azure.com2qn2j" Jan 19 02:36:31.829: INFO: At 2023-01-19 02:26:02 +0000 UTC - event for test.csi.azure.com2qn2j: {test.csi.azure.com_k8s-agentpool1-29517166-vmss000000_53759c0c-6de5-4277-9d52-f675b7576acb } ProvisioningSucceeded: Successfully provisioned volume pvc-ead47477-2e82-42be-b42b-1d3a21778bbe Jan 19 02:36:31.829: INFO: At 2023-01-19 02:26:04 +0000 UTC - event for external-injector: {default-scheduler } Scheduled: Successfully assigned volume-5841/external-injector to k8s-agentpool1-29517166-vmss000000 Jan 19 02:36:31.829: INFO: At 2023-01-19 02:28:04 +0000 UTC - event for external-injector: {attachdetach-controller } FailedAttachVolume: AttachVolume.Attach failed for volume "pvc-ead47477-2e82-42be-b42b-1d3a21778bbe" : Attach timeout for volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-iei3lcrr/providers/Microsoft.Compute/disks/pvc-ead47477-2e82-42be-b42b-1d3a21778bbe Jan 19 02:36:31.829: INFO: At 2023-01-19 02:28:07 +0000 UTC - event for external-injector: {kubelet k8s-agentpool1-29517166-vmss000000} FailedMount: Unable to attach or mount volumes: unmounted volumes=[external-volume-0], unattached volumes=[external-volume-0 kube-api-access-wqml7]: timed out waiting for the condition Jan 19 02:36:31.829: INFO: At 2023-01-19 02:28:27 +0000 UTC - event for external-injector: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-ead47477-2e82-42be-b42b-1d3a21778bbe" Jan 19 02:36:31.829: INFO: At 2023-01-19 02:29:28 +0000 UTC - event for external-injector: {kubelet k8s-agentpool1-29517166-vmss000000} Pulling: Pulling image "k8s.gcr.io/e2e-test-images/busybox:1.29-2" Jan 19 02:36:31.829: INFO: At 2023-01-19 02:29:30 +0000 UTC - event for external-injector: {kubelet k8s-agentpool1-29517166-vmss000000} Pulled: Successfully pulled image "k8s.gcr.io/e2e-test-images/busybox:1.29-2" in 1.467706231s Jan 19 02:36:31.829: INFO: At 2023-01-19 02:29:30 +0000 UTC - event for external-injector: {kubelet k8s-agentpool1-29517166-vmss000000} Created: Created container external-injector Jan 19 02:36:31.829: INFO: At 2023-01-19 02:29:30 +0000 UTC - event for external-injector: {kubelet k8s-agentpool1-29517166-vmss000000} Started: Started container external-injector Jan 19 02:36:31.829: INFO: At 2023-01-19 02:29:38 +0000 UTC - event for external-injector: {kubelet k8s-agentpool1-29517166-vmss000000} Killing: Stopping container external-injector Jan 19 02:36:31.829: INFO: At 2023-01-19 02:29:40 +0000 UTC - event for external-client: {attachdetach-controller } FailedAttachVolume: Multi-Attach error for volume "pvc-ead47477-2e82-42be-b42b-1d3a21778bbe" Volume is already exclusively attached to one node and can't be attached to another Jan 19 02:36:31.829: INFO: At 2023-01-19 02:29:40 +0000 UTC - event for external-client: {default-scheduler } Scheduled: Successfully assigned volume-5841/external-client to k8s-agentpool1-29517166-vmss000001 Jan 19 02:36:31.829: INFO: At 2023-01-19 02:31:43 +0000 UTC - event for external-client: {kubelet k8s-agentpool1-29517166-vmss000001} FailedMount: Unable to attach or mount volumes: unmounted volumes=[external-volume-0], unattached volumes=[external-volume-0 kube-api-access-9jmp9]: timed out waiting for the condition Jan 19 02:36:31.829: INFO: At 2023-01-19 02:36:17 +0000 UTC - event for external-client: {kubelet k8s-agentpool1-29517166-vmss000001} FailedMount: Unable to attach or mount volumes: unmounted volumes=[external-volume-0], unattached volumes=[kube-api-access-9jmp9 external-volume-0]: timed out waiting for the condition Jan 19 02:36:31.936: INFO: POD NODE PHASE GRACE CONDITIONS Jan 19 02:36:31.936: INFO: Jan 19 02:36:32.151: INFO: ... skipping 123 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (ext4)] volumes [90mtest/e2e/storage/framework/testsuite.go:50[0m [91m[1mshould store data [Measurement][0m [90mtest/e2e/storage/testsuites/volumes.go:161[0m [91mJan 19 02:36:21.173: Failed to create client pod: timed out waiting for the condition[0m test/e2e/framework/volume/fixtures.go:580 [90m------------------------------[0m {"msg":"FAILED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext4)] volumes should store data","total":37,"completed":6,"skipped":492,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext4)] volumes should store data"]} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Dynamic PV (xfs)][Slow] volumes[0m [1mshould store data[0m [37mtest/e2e/storage/testsuites/volumes.go:161[0m ... skipping 114 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (xfs)][Slow] volumes [90mtest/e2e/storage/framework/testsuite.go:50[0m should store data [90mtest/e2e/storage/testsuites/volumes.go:161[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (xfs)][Slow] volumes should store data","total":30,"completed":6,"skipped":399,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow] should concurrently access the single read-only volume from pods on the same node"]} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [BeforeEach] [Testpattern: CSI Ephemeral-volume (default fs)] ephemeral test/e2e/storage/framework/testsuite.go:51 Jan 19 02:38:32.251: INFO: Driver "test.csi.azure.com" does not support volume type "CSIInlineVolume" - skipping ... skipping 69 lines ... test/e2e/framework/framework.go:188 Jan 19 02:38:34.518: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready [1mSTEP[0m: Destroying namespace "volumelimits-3391" for this suite. [32m•[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Generic Ephemeral-volume (default fs)] volumeLimits should verify that all csinodes have volume limits","total":30,"completed":7,"skipped":546,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow] should concurrently access the single read-only volume from pods on the same node"]} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Dynamic PV (block volmode)] provisioning[0m [1mshould provision storage with snapshot data source [Feature:VolumeSnapshotDataSource][0m [37mtest/e2e/storage/testsuites/provisioning.go:208[0m ... skipping 128 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (block volmode)] provisioning [90mtest/e2e/storage/framework/testsuite.go:50[0m should provision storage with snapshot data source [Feature:VolumeSnapshotDataSource] [90mtest/e2e/storage/testsuites/provisioning.go:208[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] provisioning should provision storage with snapshot data source [Feature:VolumeSnapshotDataSource]","total":31,"completed":7,"skipped":698,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Pre-provisioned Snapshot (delete policy)] snapshottable[Feature:VolumeSnapshotDataSource] volume snapshot controller should check snapshot fields, check restore correctly works after modifying source data, check deletion (persistent)"]} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [BeforeEach] [Testpattern: Dynamic PV (delayed binding)] topology test/e2e/storage/framework/testsuite.go:51 [BeforeEach] [Testpattern: Dynamic PV (delayed binding)] topology ... skipping 95 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral [90mtest/e2e/storage/framework/testsuite.go:50[0m should support multiple inline ephemeral volumes [90mtest/e2e/storage/testsuites/ephemeral.go:254[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral should support multiple inline ephemeral volumes","total":35,"completed":9,"skipped":851,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] should concurrently access the single volume from pods on the same node"]} [36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [BeforeEach] [Testpattern: Pre-provisioned PV (default fs)] volumes test/e2e/storage/framework/testsuite.go:51 Jan 19 02:38:58.466: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping ... skipping 252 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow] [90mtest/e2e/storage/framework/testsuite.go:50[0m should access to two volumes with different volume mode and retain data across pod recreation on different node [90mtest/e2e/storage/testsuites/multivolume.go:248[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow] should access to two volumes with different volume mode and retain data across pod recreation on different node","total":33,"completed":8,"skipped":659,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext4)] multiVolume [Slow] should access to two volumes with the same volume mode and retain data across pod recreation on the same node"]} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [BeforeEach] [Testpattern: Inline-volume (default fs)] subPath test/e2e/storage/framework/testsuite.go:51 Jan 19 02:39:04.112: INFO: Driver "test.csi.azure.com" does not support volume type "InlineVolume" - skipping ... skipping 202 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy [90mtest/e2e/storage/framework/testsuite.go:50[0m (Always)[LinuxOnly], pod created with an initial fsgroup, new pod fsgroup applied to volume contents [90mtest/e2e/storage/testsuites/fsgroupchangepolicy.go:216[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy (Always)[LinuxOnly], pod created with an initial fsgroup, new pod fsgroup applied to volume contents","total":32,"completed":8,"skipped":663,"failed":0} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [BeforeEach] [Testpattern: Pre-provisioned PV (default fs)] subPath test/e2e/storage/framework/testsuite.go:51 Jan 19 02:39:51.922: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping ... skipping 3 lines ... [36m[1mS [SKIPPING] in Spec Setup (BeforeEach) [0.001 seconds][0m External Storage [Driver: test.csi.azure.com] [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Pre-provisioned PV (default fs)] subPath [90mtest/e2e/storage/framework/testsuite.go:50[0m [36m[1mshould fail if subpath directory is outside the volume [Slow][LinuxOnly] [BeforeEach][0m [90mtest/e2e/storage/testsuites/subpath.go:242[0m [36mDriver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping[0m test/e2e/storage/external/external.go:262 [90m------------------------------[0m ... skipping 91 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow] [90mtest/e2e/storage/framework/testsuite.go:50[0m should concurrently access the volume and its clone from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS] [90mtest/e2e/storage/testsuites/multivolume.go:378[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow] should concurrently access the volume and its clone from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS]","total":32,"completed":7,"skipped":292,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] volumes should store data"]} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral[0m [1mshould create read-only inline ephemeral volume[0m [37mtest/e2e/storage/testsuites/ephemeral.go:175[0m ... skipping 52 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral [90mtest/e2e/storage/framework/testsuite.go:50[0m should create read-only inline ephemeral volume [90mtest/e2e/storage/testsuites/ephemeral.go:175[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral should create read-only inline ephemeral volume","total":35,"completed":10,"skipped":965,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] should concurrently access the single volume from pods on the same node"]} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [BeforeEach] [Testpattern: Dynamic PV (ntfs)][Feature:Windows] subPath test/e2e/storage/framework/testsuite.go:51 Jan 19 02:41:42.284: INFO: Distro debian doesn't support ntfs -- skipping ... skipping 41 lines ... [It] should check snapshot fields, check restore correctly works, check deletion (ephemeral) test/e2e/storage/testsuites/snapshottable.go:177 Jan 19 02:38:35.537: INFO: Creating resource for dynamic PV Jan 19 02:38:35.537: INFO: Using claimSize:5Gi, test suite supported size:{ 1Mi}, driver(test.csi.azure.com) supported size:{ 1Mi} [1mSTEP[0m: creating a StorageClass snapshotting-6651-e2e-sclv2rl [1mSTEP[0m: [init] starting a pod to use the claim Jan 19 02:38:35.756: INFO: Waiting up to 15m0s for pod "pvc-snapshottable-tester-wq4pg" in namespace "snapshotting-6651" to be "Succeeded or Failed" Jan 19 02:38:35.864: INFO: Pod "pvc-snapshottable-tester-wq4pg": Phase="Pending", Reason="", readiness=false. Elapsed: 108.13952ms Jan 19 02:38:37.972: INFO: Pod "pvc-snapshottable-tester-wq4pg": Phase="Pending", Reason="", readiness=false. Elapsed: 2.216041768s Jan 19 02:38:40.080: INFO: Pod "pvc-snapshottable-tester-wq4pg": Phase="Pending", Reason="", readiness=false. Elapsed: 4.324160244s Jan 19 02:38:42.188: INFO: Pod "pvc-snapshottable-tester-wq4pg": Phase="Pending", Reason="", readiness=false. Elapsed: 6.431880576s Jan 19 02:38:44.297: INFO: Pod "pvc-snapshottable-tester-wq4pg": Phase="Pending", Reason="", readiness=false. Elapsed: 8.541290072s Jan 19 02:38:46.406: INFO: Pod "pvc-snapshottable-tester-wq4pg": Phase="Pending", Reason="", readiness=false. Elapsed: 10.650382146s ... skipping 12 lines ... Jan 19 02:39:13.828: INFO: Pod "pvc-snapshottable-tester-wq4pg": Phase="Pending", Reason="", readiness=false. Elapsed: 38.072104032s Jan 19 02:39:15.937: INFO: Pod "pvc-snapshottable-tester-wq4pg": Phase="Pending", Reason="", readiness=false. Elapsed: 40.181488576s Jan 19 02:39:18.046: INFO: Pod "pvc-snapshottable-tester-wq4pg": Phase="Pending", Reason="", readiness=false. Elapsed: 42.289863673s Jan 19 02:39:20.155: INFO: Pod "pvc-snapshottable-tester-wq4pg": Phase="Pending", Reason="", readiness=false. Elapsed: 44.398773567s Jan 19 02:39:22.263: INFO: Pod "pvc-snapshottable-tester-wq4pg": Phase="Succeeded", Reason="", readiness=false. Elapsed: 46.507289435s [1mSTEP[0m: Saw pod success Jan 19 02:39:22.263: INFO: Pod "pvc-snapshottable-tester-wq4pg" satisfied condition "Succeeded or Failed" [1mSTEP[0m: [init] checking the claim [1mSTEP[0m: creating a SnapshotClass [1mSTEP[0m: creating a dynamic VolumeSnapshot Jan 19 02:39:22.698: INFO: Waiting up to 5m0s for VolumeSnapshot snapshot-pdz2h to become ready Jan 19 02:39:22.805: INFO: VolumeSnapshot snapshot-pdz2h found but is not ready. Jan 19 02:39:24.913: INFO: VolumeSnapshot snapshot-pdz2h found but is not ready. ... skipping 49 lines ... [90mtest/e2e/storage/testsuites/snapshottable.go:113[0m [90mtest/e2e/storage/testsuites/snapshottable.go:176[0m should check snapshot fields, check restore correctly works, check deletion (ephemeral) [90mtest/e2e/storage/testsuites/snapshottable.go:177[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Ephemeral Snapshot (delete policy)] snapshottable[Feature:VolumeSnapshotDataSource] volume snapshot controller should check snapshot fields, check restore correctly works, check deletion (ephemeral)","total":30,"completed":8,"skipped":595,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow] should concurrently access the single read-only volume from pods on the same node"]} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Dynamic PV (filesystem volmode)] volumeMode[0m [1mshould fail to use a volume in a pod with mismatched mode [Slow][0m [37mtest/e2e/storage/testsuites/volumemode.go:299[0m [BeforeEach] [Testpattern: Dynamic PV (filesystem volmode)] volumeMode test/e2e/storage/framework/testsuite.go:51 [BeforeEach] [Testpattern: Dynamic PV (filesystem volmode)] volumeMode test/e2e/framework/framework.go:187 [1mSTEP[0m: Creating a kubernetes client Jan 19 02:40:37.190: INFO: >>> kubeConfig: /root/tmp1374176058/kubeconfig/kubeconfig.westeurope.json [1mSTEP[0m: Building a namespace api object, basename volumemode [1mSTEP[0m: Waiting for a default service account to be provisioned in namespace [1mSTEP[0m: Waiting for kube-root-ca.crt to be provisioned in namespace [It] should fail to use a volume in a pod with mismatched mode [Slow] test/e2e/storage/testsuites/volumemode.go:299 Jan 19 02:40:37.943: INFO: Creating resource for dynamic PV Jan 19 02:40:37.943: INFO: Using claimSize:5Gi, test suite supported size:{ 1Mi}, driver(test.csi.azure.com) supported size:{ 1Mi} [1mSTEP[0m: creating a StorageClass volumemode-5157-e2e-scbpbpb [1mSTEP[0m: creating a claim Jan 19 02:40:38.163: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.com45ttb] to have phase Bound Jan 19 02:40:38.271: INFO: PersistentVolumeClaim test.csi.azure.com45ttb found but phase is Pending instead of Bound. Jan 19 02:40:40.380: INFO: PersistentVolumeClaim test.csi.azure.com45ttb found but phase is Pending instead of Bound. Jan 19 02:40:42.491: INFO: PersistentVolumeClaim test.csi.azure.com45ttb found and phase=Bound (4.32793186s) [1mSTEP[0m: Creating pod [1mSTEP[0m: Waiting for the pod to fail Jan 19 02:40:45.139: INFO: Deleting pod "pod-21e17638-e00c-4583-a96a-76e89967933b" in namespace "volumemode-5157" Jan 19 02:40:45.249: INFO: Wait up to 5m0s for pod "pod-21e17638-e00c-4583-a96a-76e89967933b" to be fully deleted [1mSTEP[0m: Deleting pvc Jan 19 02:40:47.466: INFO: Deleting PersistentVolumeClaim "test.csi.azure.com45ttb" Jan 19 02:40:47.574: INFO: Waiting up to 5m0s for PersistentVolume pvc-b5af44a4-cbfc-46ea-9fd4-a09f8b914201 to get deleted Jan 19 02:40:47.682: INFO: PersistentVolume pvc-b5af44a4-cbfc-46ea-9fd4-a09f8b914201 found and phase=Released (108.225413ms) ... skipping 32 lines ... [32m• [SLOW TEST:143.858 seconds][0m External Storage [Driver: test.csi.azure.com] [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (filesystem volmode)] volumeMode [90mtest/e2e/storage/framework/testsuite.go:50[0m should fail to use a volume in a pod with mismatched mode [Slow] [90mtest/e2e/storage/testsuites/volumemode.go:299[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (filesystem volmode)] volumeMode should fail to use a volume in a pod with mismatched mode [Slow]","total":32,"completed":8,"skipped":313,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] volumes should store data"]} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [BeforeEach] [Testpattern: Dynamic PV (ntfs)][Feature:Windows] subPath test/e2e/storage/framework/testsuite.go:51 Jan 19 02:43:01.074: INFO: Distro debian doesn't support ntfs -- skipping ... skipping 3 lines ... [36m[1mS [SKIPPING] in Spec Setup (BeforeEach) [0.001 seconds][0m External Storage [Driver: test.csi.azure.com] [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (ntfs)][Feature:Windows] subPath [90mtest/e2e/storage/framework/testsuite.go:50[0m [36m[1mshould fail if subpath with backstepping is outside the volume [Slow][LinuxOnly] [BeforeEach][0m [90mtest/e2e/storage/testsuites/subpath.go:280[0m [36mDistro debian doesn't support ntfs -- skipping[0m test/e2e/storage/framework/testsuite.go:127 [90m------------------------------[0m ... skipping 105 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] [90mtest/e2e/storage/framework/testsuite.go:50[0m should concurrently access the single read-only volume from pods on the same node [90mtest/e2e/storage/testsuites/multivolume.go:423[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] should concurrently access the single read-only volume from pods on the same node","total":31,"completed":8,"skipped":748,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Pre-provisioned Snapshot (delete policy)] snapshottable[Feature:VolumeSnapshotDataSource] volume snapshot controller should check snapshot fields, check restore correctly works after modifying source data, check deletion (persistent)"]} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Generic Ephemeral-volume (block volmode) (late-binding)] ephemeral[0m [1mshould support multiple inline ephemeral volumes[0m [37mtest/e2e/storage/testsuites/ephemeral.go:254[0m ... skipping 63 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Generic Ephemeral-volume (block volmode) (late-binding)] ephemeral [90mtest/e2e/storage/framework/testsuite.go:50[0m should support multiple inline ephemeral volumes [90mtest/e2e/storage/testsuites/ephemeral.go:254[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Generic Ephemeral-volume (block volmode) (late-binding)] ephemeral should support multiple inline ephemeral volumes","total":33,"completed":9,"skipped":825,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext4)] multiVolume [Slow] should access to two volumes with the same volume mode and retain data across pod recreation on the same node"]} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Dynamic PV (block volmode)] multiVolume [Slow][0m [1mshould access to two volumes with the same volume mode and retain data across pod recreation on different node[0m [37mtest/e2e/storage/testsuites/multivolume.go:168[0m ... skipping 206 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] [90mtest/e2e/storage/framework/testsuite.go:50[0m should access to two volumes with the same volume mode and retain data across pod recreation on different node [90mtest/e2e/storage/testsuites/multivolume.go:168[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] should access to two volumes with the same volume mode and retain data across pod recreation on different node","total":37,"completed":7,"skipped":500,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext4)] volumes should store data"]} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [BeforeEach] [Testpattern: Inline-volume (ext3)] volumes test/e2e/storage/framework/testsuite.go:51 Jan 19 02:43:21.441: INFO: Driver "test.csi.azure.com" does not support volume type "InlineVolume" - skipping ... skipping 127 lines ... [36m[1mS [SKIPPING] in Spec Setup (BeforeEach) [0.001 seconds][0m External Storage [Driver: test.csi.azure.com] [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Inline-volume (default fs)] subPath [90mtest/e2e/storage/framework/testsuite.go:50[0m [36m[1mshould fail if subpath with backstepping is outside the volume [Slow][LinuxOnly] [BeforeEach][0m [90mtest/e2e/storage/testsuites/subpath.go:280[0m [36mDriver "test.csi.azure.com" does not support volume type "InlineVolume" - skipping[0m test/e2e/storage/external/external.go:262 [90m------------------------------[0m ... skipping 192 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (ext4)] multiVolume [Slow] [90mtest/e2e/storage/framework/testsuite.go:50[0m should concurrently access the single volume from pods on the same node [90mtest/e2e/storage/testsuites/multivolume.go:298[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext4)] multiVolume [Slow] should concurrently access the single volume from pods on the same node","total":32,"completed":9,"skipped":691,"failed":0} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [BeforeEach] [Testpattern: Dynamic PV (default fs)] capacity test/e2e/storage/framework/testsuite.go:51 Jan 19 02:43:56.149: INFO: Driver test.csi.azure.com doesn't publish storage capacity -- skipping ... skipping 24 lines ... [36m[1mS [SKIPPING] in Spec Setup (BeforeEach) [0.001 seconds][0m External Storage [Driver: test.csi.azure.com] [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Inline-volume (default fs)] subPath [90mtest/e2e/storage/framework/testsuite.go:50[0m [36m[1mshould fail if non-existent subpath is outside the volume [Slow][LinuxOnly] [BeforeEach][0m [90mtest/e2e/storage/testsuites/subpath.go:269[0m [36mDriver "test.csi.azure.com" does not support volume type "InlineVolume" - skipping[0m test/e2e/storage/external/external.go:262 [90m------------------------------[0m ... skipping 68 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (block volmode)] volumeMode [90mtest/e2e/storage/framework/testsuite.go:50[0m should not mount / map unused volumes in a pod [LinuxOnly] [90mtest/e2e/storage/testsuites/volumemode.go:354[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] volumeMode should not mount / map unused volumes in a pod [LinuxOnly]","total":33,"completed":10,"skipped":838,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext4)] multiVolume [Slow] should access to two volumes with the same volume mode and retain data across pod recreation on the same node"]} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [BeforeEach] [Testpattern: Inline-volume (default fs)] subPath test/e2e/storage/framework/testsuite.go:51 Jan 19 02:44:13.561: INFO: Driver "test.csi.azure.com" does not support volume type "InlineVolume" - skipping ... skipping 3 lines ... [36m[1mS [SKIPPING] in Spec Setup (BeforeEach) [0.001 seconds][0m External Storage [Driver: test.csi.azure.com] [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Inline-volume (default fs)] subPath [90mtest/e2e/storage/framework/testsuite.go:50[0m [36m[1mshould fail if subpath directory is outside the volume [Slow][LinuxOnly] [BeforeEach][0m [90mtest/e2e/storage/testsuites/subpath.go:242[0m [36mDriver "test.csi.azure.com" does not support volume type "InlineVolume" - skipping[0m test/e2e/storage/external/external.go:262 [90m------------------------------[0m ... skipping 57 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Generic Ephemeral-volume (block volmode) (late-binding)] ephemeral [90mtest/e2e/storage/framework/testsuite.go:50[0m should create read/write inline ephemeral volume [90mtest/e2e/storage/testsuites/ephemeral.go:196[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Generic Ephemeral-volume (block volmode) (late-binding)] ephemeral should create read/write inline ephemeral volume","total":30,"completed":9,"skipped":630,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow] should concurrently access the single read-only volume from pods on the same node"]} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Dynamic PV (ext3)] volumes[0m [1mshould store data[0m [37mtest/e2e/storage/testsuites/volumes.go:161[0m ... skipping 108 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (ext3)] volumes [90mtest/e2e/storage/framework/testsuite.go:50[0m should store data [90mtest/e2e/storage/testsuites/volumes.go:161[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext3)] volumes should store data","total":32,"completed":9,"skipped":352,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] volumes should store data"]} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [BeforeEach] [Testpattern: Dynamic PV (ntfs)(allowExpansion)][Feature:Windows] volume-expand test/e2e/storage/framework/testsuite.go:51 Jan 19 02:44:37.272: INFO: Distro debian doesn't support ntfs -- skipping ... skipping 88 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Generic Ephemeral-volume (default fs) (immediate-binding)] ephemeral [90mtest/e2e/storage/framework/testsuite.go:50[0m should create read/write inline ephemeral volume [90mtest/e2e/storage/testsuites/ephemeral.go:196[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Generic Ephemeral-volume (default fs) (immediate-binding)] ephemeral should create read/write inline ephemeral volume","total":37,"completed":8,"skipped":763,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext4)] volumes should store data"]} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [BeforeEach] [Testpattern: Pre-provisioned PV (default fs)] volumes test/e2e/storage/framework/testsuite.go:51 Jan 19 02:45:06.684: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping ... skipping 125 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow] [90mtest/e2e/storage/framework/testsuite.go:50[0m should concurrently access the volume and its clone from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS] [90mtest/e2e/storage/testsuites/multivolume.go:378[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow] should concurrently access the volume and its clone from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS]","total":31,"completed":9,"skipped":763,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Pre-provisioned Snapshot (delete policy)] snapshottable[Feature:VolumeSnapshotDataSource] volume snapshot controller should check snapshot fields, check restore correctly works after modifying source data, check deletion (persistent)"]} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [BeforeEach] [Testpattern: Dynamic PV (block volmode)] volume-expand test/e2e/storage/framework/testsuite.go:51 Jan 19 02:45:07.952: INFO: Driver "test.csi.azure.com" does not support volume expansion - skipping ... skipping 38 lines ... Jan 19 02:43:57.229: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comj86bt] to have phase Bound Jan 19 02:43:57.337: INFO: PersistentVolumeClaim test.csi.azure.comj86bt found but phase is Pending instead of Bound. Jan 19 02:43:59.446: INFO: PersistentVolumeClaim test.csi.azure.comj86bt found but phase is Pending instead of Bound. Jan 19 02:44:01.557: INFO: PersistentVolumeClaim test.csi.azure.comj86bt found and phase=Bound (4.32781776s) [1mSTEP[0m: Creating pod exec-volume-test-dynamicpv-dptx [1mSTEP[0m: Creating a pod to test exec-volume-test Jan 19 02:44:01.884: INFO: Waiting up to 5m0s for pod "exec-volume-test-dynamicpv-dptx" in namespace "volume-6424" to be "Succeeded or Failed" Jan 19 02:44:01.992: INFO: Pod "exec-volume-test-dynamicpv-dptx": Phase="Pending", Reason="", readiness=false. Elapsed: 108.053618ms Jan 19 02:44:04.102: INFO: Pod "exec-volume-test-dynamicpv-dptx": Phase="Pending", Reason="", readiness=false. Elapsed: 2.218742504s Jan 19 02:44:06.212: INFO: Pod "exec-volume-test-dynamicpv-dptx": Phase="Pending", Reason="", readiness=false. Elapsed: 4.328566028s Jan 19 02:44:08.326: INFO: Pod "exec-volume-test-dynamicpv-dptx": Phase="Pending", Reason="", readiness=false. Elapsed: 6.442124362s Jan 19 02:44:10.436: INFO: Pod "exec-volume-test-dynamicpv-dptx": Phase="Pending", Reason="", readiness=false. Elapsed: 8.552451249s Jan 19 02:44:12.546: INFO: Pod "exec-volume-test-dynamicpv-dptx": Phase="Pending", Reason="", readiness=false. Elapsed: 10.661932621s ... skipping 4 lines ... Jan 19 02:44:23.090: INFO: Pod "exec-volume-test-dynamicpv-dptx": Phase="Pending", Reason="", readiness=false. Elapsed: 21.206780932s Jan 19 02:44:25.200: INFO: Pod "exec-volume-test-dynamicpv-dptx": Phase="Pending", Reason="", readiness=false. Elapsed: 23.316658714s Jan 19 02:44:27.310: INFO: Pod "exec-volume-test-dynamicpv-dptx": Phase="Pending", Reason="", readiness=false. Elapsed: 25.426512254s Jan 19 02:44:29.420: INFO: Pod "exec-volume-test-dynamicpv-dptx": Phase="Pending", Reason="", readiness=false. Elapsed: 27.536745891s Jan 19 02:44:31.530: INFO: Pod "exec-volume-test-dynamicpv-dptx": Phase="Succeeded", Reason="", readiness=false. Elapsed: 29.646764938s [1mSTEP[0m: Saw pod success Jan 19 02:44:31.530: INFO: Pod "exec-volume-test-dynamicpv-dptx" satisfied condition "Succeeded or Failed" Jan 19 02:44:31.639: INFO: Trying to get logs from node k8s-agentpool1-29517166-vmss000000 pod exec-volume-test-dynamicpv-dptx container exec-container-dynamicpv-dptx: <nil> [1mSTEP[0m: delete the pod Jan 19 02:44:31.867: INFO: Waiting for pod exec-volume-test-dynamicpv-dptx to disappear Jan 19 02:44:31.988: INFO: Pod exec-volume-test-dynamicpv-dptx no longer exists [1mSTEP[0m: Deleting pod exec-volume-test-dynamicpv-dptx Jan 19 02:44:31.988: INFO: Deleting pod "exec-volume-test-dynamicpv-dptx" in namespace "volume-6424" ... skipping 21 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (default fs)] volumes [90mtest/e2e/storage/framework/testsuite.go:50[0m should allow exec of files on the volume [90mtest/e2e/storage/testsuites/volumes.go:198[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] volumes should allow exec of files on the volume","total":32,"completed":10,"skipped":814,"failed":0} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [BeforeEach] [Testpattern: Pre-provisioned PV (block volmode)] multiVolume [Slow] test/e2e/storage/framework/testsuite.go:51 Jan 19 02:45:13.768: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping ... skipping 68 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (default fs)] subPath [90mtest/e2e/storage/framework/testsuite.go:50[0m should be able to unmount after the subpath directory is deleted [LinuxOnly] [90mtest/e2e/storage/testsuites/subpath.go:447[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should be able to unmount after the subpath directory is deleted [LinuxOnly]","total":32,"completed":11,"skipped":837,"failed":0} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [BeforeEach] [Testpattern: Dynamic PV (default fs)] provisioning test/e2e/storage/framework/testsuite.go:51 [BeforeEach] [Testpattern: Dynamic PV (default fs)] provisioning ... skipping 116 lines ... Jan 19 02:46:08.020: INFO: Running Cleanup Action: k8s.io/kubernetes/test/e2e/storage/vsphere.glob..func9.2 Jan 19 02:46:08.020: INFO: Running Cleanup Action: k8s.io/kubernetes/test/e2e/storage/vsphere.glob..func4.2 Jan 19 02:46:08.020: INFO: Running Cleanup Action: k8s.io/kubernetes/test/e2e/storage/vsphere.glob..func1.3 [0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Dynamic PV (default fs)] subPath[0m [1mshould fail if non-existent subpath is outside the volume [Slow][LinuxOnly][0m [37mtest/e2e/storage/testsuites/subpath.go:269[0m [BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath test/e2e/storage/framework/testsuite.go:51 [BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath test/e2e/framework/framework.go:187 [1mSTEP[0m: Creating a kubernetes client Jan 19 02:44:32.167: INFO: >>> kubeConfig: /root/tmp1374176058/kubeconfig/kubeconfig.westeurope.json [1mSTEP[0m: Building a namespace api object, basename provisioning [1mSTEP[0m: Waiting for a default service account to be provisioned in namespace [1mSTEP[0m: Waiting for kube-root-ca.crt to be provisioned in namespace [It] should fail if non-existent subpath is outside the volume [Slow][LinuxOnly] test/e2e/storage/testsuites/subpath.go:269 Jan 19 02:44:32.929: INFO: Creating resource for dynamic PV Jan 19 02:44:32.929: INFO: Using claimSize:5Gi, test suite supported size:{ 1Mi}, driver(test.csi.azure.com) supported size:{ 1Mi} [1mSTEP[0m: creating a StorageClass provisioning-6524-e2e-sc7bl9b [1mSTEP[0m: creating a claim Jan 19 02:44:33.038: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil Jan 19 02:44:33.149: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.com9jfr8] to have phase Bound Jan 19 02:44:33.259: INFO: PersistentVolumeClaim test.csi.azure.com9jfr8 found but phase is Pending instead of Bound. Jan 19 02:44:35.368: INFO: PersistentVolumeClaim test.csi.azure.com9jfr8 found but phase is Pending instead of Bound. Jan 19 02:44:37.477: INFO: PersistentVolumeClaim test.csi.azure.com9jfr8 found and phase=Bound (4.328186288s) [1mSTEP[0m: Creating pod pod-subpath-test-dynamicpv-dfl2 [1mSTEP[0m: Checking for subpath error in container status Jan 19 02:45:06.022: INFO: Deleting pod "pod-subpath-test-dynamicpv-dfl2" in namespace "provisioning-6524" Jan 19 02:45:06.132: INFO: Wait up to 5m0s for pod "pod-subpath-test-dynamicpv-dfl2" to be fully deleted [1mSTEP[0m: Deleting pod Jan 19 02:45:08.363: INFO: Deleting pod "pod-subpath-test-dynamicpv-dfl2" in namespace "provisioning-6524" [1mSTEP[0m: Deleting pvc Jan 19 02:45:08.472: INFO: Deleting PersistentVolumeClaim "test.csi.azure.com9jfr8" ... skipping 22 lines ... [32m• [SLOW TEST:108.604 seconds][0m External Storage [Driver: test.csi.azure.com] [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (default fs)] subPath [90mtest/e2e/storage/framework/testsuite.go:50[0m should fail if non-existent subpath is outside the volume [Slow][LinuxOnly] [90mtest/e2e/storage/testsuites/subpath.go:269[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should fail if non-existent subpath is outside the volume [Slow][LinuxOnly]","total":30,"completed":10,"skipped":679,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow] should concurrently access the single read-only volume from pods on the same node"]} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Ephemeral Snapshot (retain policy)] snapshottable[Feature:VolumeSnapshotDataSource][0m [0mvolume snapshot controller[0m [90m[0m [1mshould check snapshot fields, check restore correctly works, check deletion (ephemeral)[0m [37mtest/e2e/storage/testsuites/snapshottable.go:177[0m ... skipping 10 lines ... [It] should check snapshot fields, check restore correctly works, check deletion (ephemeral) test/e2e/storage/testsuites/snapshottable.go:177 Jan 19 02:44:38.173: INFO: Creating resource for dynamic PV Jan 19 02:44:38.173: INFO: Using claimSize:5Gi, test suite supported size:{ 1Mi}, driver(test.csi.azure.com) supported size:{ 1Mi} [1mSTEP[0m: creating a StorageClass snapshotting-1841-e2e-scfjrpn [1mSTEP[0m: [init] starting a pod to use the claim Jan 19 02:44:38.394: INFO: Waiting up to 15m0s for pod "pvc-snapshottable-tester-f76t7" in namespace "snapshotting-1841" to be "Succeeded or Failed" Jan 19 02:44:38.502: INFO: Pod "pvc-snapshottable-tester-f76t7": Phase="Pending", Reason="", readiness=false. Elapsed: 108.21049ms Jan 19 02:44:40.612: INFO: Pod "pvc-snapshottable-tester-f76t7": Phase="Pending", Reason="", readiness=false. Elapsed: 2.217659372s Jan 19 02:44:42.721: INFO: Pod "pvc-snapshottable-tester-f76t7": Phase="Pending", Reason="", readiness=false. Elapsed: 4.326506738s Jan 19 02:44:44.832: INFO: Pod "pvc-snapshottable-tester-f76t7": Phase="Pending", Reason="", readiness=false. Elapsed: 6.437839639s Jan 19 02:44:46.941: INFO: Pod "pvc-snapshottable-tester-f76t7": Phase="Pending", Reason="", readiness=false. Elapsed: 8.546549812s Jan 19 02:44:49.050: INFO: Pod "pvc-snapshottable-tester-f76t7": Phase="Pending", Reason="", readiness=false. Elapsed: 10.65589945s ... skipping 8 lines ... Jan 19 02:45:08.037: INFO: Pod "pvc-snapshottable-tester-f76t7": Phase="Pending", Reason="", readiness=false. Elapsed: 29.642965546s Jan 19 02:45:10.148: INFO: Pod "pvc-snapshottable-tester-f76t7": Phase="Pending", Reason="", readiness=false. Elapsed: 31.754082149s Jan 19 02:45:12.258: INFO: Pod "pvc-snapshottable-tester-f76t7": Phase="Pending", Reason="", readiness=false. Elapsed: 33.86406757s Jan 19 02:45:14.368: INFO: Pod "pvc-snapshottable-tester-f76t7": Phase="Pending", Reason="", readiness=false. Elapsed: 35.97380193s Jan 19 02:45:16.477: INFO: Pod "pvc-snapshottable-tester-f76t7": Phase="Succeeded", Reason="", readiness=false. Elapsed: 38.082928452s [1mSTEP[0m: Saw pod success Jan 19 02:45:16.477: INFO: Pod "pvc-snapshottable-tester-f76t7" satisfied condition "Succeeded or Failed" [1mSTEP[0m: [init] checking the claim [1mSTEP[0m: creating a SnapshotClass [1mSTEP[0m: creating a dynamic VolumeSnapshot Jan 19 02:45:16.915: INFO: Waiting up to 5m0s for VolumeSnapshot snapshot-8vd48 to become ready Jan 19 02:45:17.024: INFO: VolumeSnapshot snapshot-8vd48 found but is not ready. Jan 19 02:45:19.135: INFO: VolumeSnapshot snapshot-8vd48 found but is not ready. ... skipping 40 lines ... Jan 19 02:46:07.086: INFO: volumesnapshotcontents snapcontent-2f38a956-472d-4f3b-a60a-4996451a60e2 has been found and is not deleted Jan 19 02:46:08.196: INFO: volumesnapshotcontents snapcontent-2f38a956-472d-4f3b-a60a-4996451a60e2 has been found and is not deleted Jan 19 02:46:09.306: INFO: volumesnapshotcontents snapcontent-2f38a956-472d-4f3b-a60a-4996451a60e2 has been found and is not deleted Jan 19 02:46:10.414: INFO: volumesnapshotcontents snapcontent-2f38a956-472d-4f3b-a60a-4996451a60e2 has been found and is not deleted Jan 19 02:46:11.525: INFO: volumesnapshotcontents snapcontent-2f38a956-472d-4f3b-a60a-4996451a60e2 has been found and is not deleted Jan 19 02:46:12.634: INFO: volumesnapshotcontents snapcontent-2f38a956-472d-4f3b-a60a-4996451a60e2 has been found and is not deleted Jan 19 02:46:13.634: INFO: WaitUntil failed after reaching the timeout 30s [AfterEach] volume snapshot controller test/e2e/storage/testsuites/snapshottable.go:172 Jan 19 02:46:13.747: INFO: Pod restored-pvc-tester-nsfkr has the following logs: Jan 19 02:46:13.747: INFO: Deleting pod "restored-pvc-tester-nsfkr" in namespace "snapshotting-1841" Jan 19 02:46:13.858: INFO: Wait up to 5m0s for pod "restored-pvc-tester-nsfkr" to be fully deleted Jan 19 02:46:46.075: INFO: deleting snapshot "snapshotting-1841"/"snapshot-8vd48" ... skipping 26 lines ... [90mtest/e2e/storage/testsuites/snapshottable.go:113[0m [90mtest/e2e/storage/testsuites/snapshottable.go:176[0m should check snapshot fields, check restore correctly works, check deletion (ephemeral) [90mtest/e2e/storage/testsuites/snapshottable.go:177[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Ephemeral Snapshot (retain policy)] snapshottable[Feature:VolumeSnapshotDataSource] volume snapshot controller should check snapshot fields, check restore correctly works, check deletion (ephemeral)","total":32,"completed":10,"skipped":601,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] volumes should store data"]} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [BeforeEach] [Testpattern: Pre-provisioned PV (ntfs)][Feature:Windows] volumes test/e2e/storage/framework/testsuite.go:51 Jan 19 02:46:53.972: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping ... skipping 38 lines ... Jan 19 02:44:14.539: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comzd48c] to have phase Bound Jan 19 02:44:14.646: INFO: PersistentVolumeClaim test.csi.azure.comzd48c found but phase is Pending instead of Bound. Jan 19 02:44:16.755: INFO: PersistentVolumeClaim test.csi.azure.comzd48c found but phase is Pending instead of Bound. Jan 19 02:44:18.863: INFO: PersistentVolumeClaim test.csi.azure.comzd48c found and phase=Bound (4.323563826s) [1mSTEP[0m: [init] starting a pod to use the claim [1mSTEP[0m: [init] check pod success Jan 19 02:44:19.294: INFO: Waiting up to 15m0s for pod "pvc-snapshottable-tester-7dlbh" in namespace "snapshotting-9690" to be "Succeeded or Failed" Jan 19 02:44:19.405: INFO: Pod "pvc-snapshottable-tester-7dlbh": Phase="Pending", Reason="", readiness=false. Elapsed: 111.101133ms Jan 19 02:44:21.512: INFO: Pod "pvc-snapshottable-tester-7dlbh": Phase="Pending", Reason="", readiness=false. Elapsed: 2.218708934s Jan 19 02:44:23.620: INFO: Pod "pvc-snapshottable-tester-7dlbh": Phase="Pending", Reason="", readiness=false. Elapsed: 4.326710693s Jan 19 02:44:25.728: INFO: Pod "pvc-snapshottable-tester-7dlbh": Phase="Pending", Reason="", readiness=false. Elapsed: 6.434842336s Jan 19 02:44:27.837: INFO: Pod "pvc-snapshottable-tester-7dlbh": Phase="Pending", Reason="", readiness=false. Elapsed: 8.543515333s Jan 19 02:44:29.946: INFO: Pod "pvc-snapshottable-tester-7dlbh": Phase="Pending", Reason="", readiness=false. Elapsed: 10.652464685s Jan 19 02:44:32.053: INFO: Pod "pvc-snapshottable-tester-7dlbh": Phase="Pending", Reason="", readiness=false. Elapsed: 12.759890568s Jan 19 02:44:34.161: INFO: Pod "pvc-snapshottable-tester-7dlbh": Phase="Pending", Reason="", readiness=false. Elapsed: 14.867743086s Jan 19 02:44:36.269: INFO: Pod "pvc-snapshottable-tester-7dlbh": Phase="Pending", Reason="", readiness=false. Elapsed: 16.975465703s Jan 19 02:44:38.377: INFO: Pod "pvc-snapshottable-tester-7dlbh": Phase="Pending", Reason="", readiness=false. Elapsed: 19.083234657s Jan 19 02:44:40.485: INFO: Pod "pvc-snapshottable-tester-7dlbh": Phase="Succeeded", Reason="", readiness=false. Elapsed: 21.191576373s [1mSTEP[0m: Saw pod success Jan 19 02:44:40.485: INFO: Pod "pvc-snapshottable-tester-7dlbh" satisfied condition "Succeeded or Failed" [1mSTEP[0m: [init] checking the claim Jan 19 02:44:40.593: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comzd48c] to have phase Bound Jan 19 02:44:40.700: INFO: PersistentVolumeClaim test.csi.azure.comzd48c found and phase=Bound (106.935259ms) [1mSTEP[0m: [init] checking the PV [1mSTEP[0m: [init] deleting the pod Jan 19 02:44:41.027: INFO: Pod pvc-snapshottable-tester-7dlbh has the following logs: ... skipping 12 lines ... Jan 19 02:44:46.221: INFO: received snapshotStatus map[boundVolumeSnapshotContentName:snapcontent-6b26c96f-92f4-46b6-a12e-c6d9693b6c66 creationTime:2023-01-19T02:44:41Z readyToUse:true restoreSize:5Gi] Jan 19 02:44:46.221: INFO: snapshotContentName snapcontent-6b26c96f-92f4-46b6-a12e-c6d9693b6c66 [1mSTEP[0m: checking the snapshot [1mSTEP[0m: checking the SnapshotContent [1mSTEP[0m: Modifying source data test [1mSTEP[0m: modifying the data in the source PVC Jan 19 02:44:46.655: INFO: Waiting up to 15m0s for pod "pvc-snapshottable-data-tester-p6k4l" in namespace "snapshotting-9690" to be "Succeeded or Failed" Jan 19 02:44:46.762: INFO: Pod "pvc-snapshottable-data-tester-p6k4l": Phase="Pending", Reason="", readiness=false. Elapsed: 106.730034ms Jan 19 02:44:48.870: INFO: Pod "pvc-snapshottable-data-tester-p6k4l": Phase="Pending", Reason="", readiness=false. Elapsed: 2.214747701s Jan 19 02:44:50.978: INFO: Pod "pvc-snapshottable-data-tester-p6k4l": Phase="Pending", Reason="", readiness=false. Elapsed: 4.322708562s Jan 19 02:44:53.086: INFO: Pod "pvc-snapshottable-data-tester-p6k4l": Phase="Pending", Reason="", readiness=false. Elapsed: 6.430785034s Jan 19 02:44:55.194: INFO: Pod "pvc-snapshottable-data-tester-p6k4l": Phase="Pending", Reason="", readiness=false. Elapsed: 8.538496062s Jan 19 02:44:57.302: INFO: Pod "pvc-snapshottable-data-tester-p6k4l": Phase="Pending", Reason="", readiness=false. Elapsed: 10.64698682s ... skipping 23 lines ... Jan 19 02:45:47.912: INFO: Pod "pvc-snapshottable-data-tester-p6k4l": Phase="Pending", Reason="", readiness=false. Elapsed: 1m1.256586429s Jan 19 02:45:50.021: INFO: Pod "pvc-snapshottable-data-tester-p6k4l": Phase="Pending", Reason="", readiness=false. Elapsed: 1m3.365730905s Jan 19 02:45:52.129: INFO: Pod "pvc-snapshottable-data-tester-p6k4l": Phase="Pending", Reason="", readiness=false. Elapsed: 1m5.473697603s Jan 19 02:45:54.237: INFO: Pod "pvc-snapshottable-data-tester-p6k4l": Phase="Pending", Reason="", readiness=false. Elapsed: 1m7.581604519s Jan 19 02:45:56.347: INFO: Pod "pvc-snapshottable-data-tester-p6k4l": Phase="Succeeded", Reason="", readiness=false. Elapsed: 1m9.691068516s [1mSTEP[0m: Saw pod success Jan 19 02:45:56.347: INFO: Pod "pvc-snapshottable-data-tester-p6k4l" satisfied condition "Succeeded or Failed" Jan 19 02:45:56.589: INFO: Pod pvc-snapshottable-data-tester-p6k4l has the following logs: Jan 19 02:45:56.589: INFO: Deleting pod "pvc-snapshottable-data-tester-p6k4l" in namespace "snapshotting-9690" Jan 19 02:45:56.703: INFO: Wait up to 5m0s for pod "pvc-snapshottable-data-tester-p6k4l" to be fully deleted [1mSTEP[0m: creating a pvc from the snapshot [1mSTEP[0m: starting a pod to use the snapshot Jan 19 02:46:27.244: INFO: Running '/usr/local/bin/kubectl --server=https://kubetest-iei3lcrr.westeurope.cloudapp.azure.com --kubeconfig=/root/tmp1374176058/kubeconfig/kubeconfig.westeurope.json --namespace=snapshotting-9690 exec restored-pvc-tester-gjsnq --namespace=snapshotting-9690 -- cat /mnt/test/data' ... skipping 47 lines ... [90mtest/e2e/storage/testsuites/snapshottable.go:113[0m [90mtest/e2e/storage/testsuites/snapshottable.go:176[0m should check snapshot fields, check restore correctly works after modifying source data, check deletion (persistent) [90mtest/e2e/storage/testsuites/snapshottable.go:278[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic Snapshot (delete policy)] snapshottable[Feature:VolumeSnapshotDataSource] volume snapshot controller should check snapshot fields, check restore correctly works after modifying source data, check deletion (persistent)","total":33,"completed":11,"skipped":873,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext4)] multiVolume [Slow] should access to two volumes with the same volume mode and retain data across pod recreation on the same node"]} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [BeforeEach] [Testpattern: Dynamic PV (ntfs)][Feature:Windows] subPath test/e2e/storage/framework/testsuite.go:51 Jan 19 02:47:11.363: INFO: Distro debian doesn't support ntfs -- skipping ... skipping 138 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy [90mtest/e2e/storage/framework/testsuite.go:50[0m (Always)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed via chgrp in first pod, new pod with same fsgroup applied to the volume contents [90mtest/e2e/storage/testsuites/fsgroupchangepolicy.go:216[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy (Always)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed via chgrp in first pod, new pod with same fsgroup applied to the volume contents","total":31,"completed":10,"skipped":929,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Pre-provisioned Snapshot (delete policy)] snapshottable[Feature:VolumeSnapshotDataSource] volume snapshot controller should check snapshot fields, check restore correctly works after modifying source data, check deletion (persistent)"]} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow][0m [1mshould access to two volumes with the same volume mode and retain data across pod recreation on the same node[0m [37mtest/e2e/storage/testsuites/multivolume.go:138[0m ... skipping 194 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow] [90mtest/e2e/storage/framework/testsuite.go:50[0m should access to two volumes with the same volume mode and retain data across pod recreation on the same node [90mtest/e2e/storage/testsuites/multivolume.go:138[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow] should access to two volumes with the same volume mode and retain data across pod recreation on the same node","total":37,"completed":9,"skipped":870,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext4)] volumes should store data"]} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Dynamic PV (default fs)] subPath[0m [1mshould support file as subpath [LinuxOnly][0m [37mtest/e2e/storage/testsuites/subpath.go:232[0m ... skipping 17 lines ... Jan 19 02:46:54.957: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.com4z9qw] to have phase Bound Jan 19 02:46:55.075: INFO: PersistentVolumeClaim test.csi.azure.com4z9qw found but phase is Pending instead of Bound. Jan 19 02:46:57.185: INFO: PersistentVolumeClaim test.csi.azure.com4z9qw found but phase is Pending instead of Bound. Jan 19 02:46:59.294: INFO: PersistentVolumeClaim test.csi.azure.com4z9qw found and phase=Bound (4.336865012s) [1mSTEP[0m: Creating pod pod-subpath-test-dynamicpv-pvvq [1mSTEP[0m: Creating a pod to test atomic-volume-subpath Jan 19 02:46:59.618: INFO: Waiting up to 5m0s for pod "pod-subpath-test-dynamicpv-pvvq" in namespace "provisioning-7220" to be "Succeeded or Failed" Jan 19 02:46:59.729: INFO: Pod "pod-subpath-test-dynamicpv-pvvq": Phase="Pending", Reason="", readiness=false. Elapsed: 110.574525ms Jan 19 02:47:01.839: INFO: Pod "pod-subpath-test-dynamicpv-pvvq": Phase="Pending", Reason="", readiness=false. Elapsed: 2.220569189s Jan 19 02:47:03.949: INFO: Pod "pod-subpath-test-dynamicpv-pvvq": Phase="Pending", Reason="", readiness=false. Elapsed: 4.330486146s Jan 19 02:47:06.061: INFO: Pod "pod-subpath-test-dynamicpv-pvvq": Phase="Pending", Reason="", readiness=false. Elapsed: 6.442514512s Jan 19 02:47:08.170: INFO: Pod "pod-subpath-test-dynamicpv-pvvq": Phase="Pending", Reason="", readiness=false. Elapsed: 8.551445084s Jan 19 02:47:10.280: INFO: Pod "pod-subpath-test-dynamicpv-pvvq": Phase="Pending", Reason="", readiness=false. Elapsed: 10.661455972s ... skipping 24 lines ... Jan 19 02:48:03.020: INFO: Pod "pod-subpath-test-dynamicpv-pvvq": Phase="Running", Reason="", readiness=true. Elapsed: 1m3.401847124s Jan 19 02:48:05.129: INFO: Pod "pod-subpath-test-dynamicpv-pvvq": Phase="Running", Reason="", readiness=true. Elapsed: 1m5.510740722s Jan 19 02:48:07.238: INFO: Pod "pod-subpath-test-dynamicpv-pvvq": Phase="Running", Reason="", readiness=true. Elapsed: 1m7.619718404s Jan 19 02:48:09.355: INFO: Pod "pod-subpath-test-dynamicpv-pvvq": Phase="Running", Reason="", readiness=false. Elapsed: 1m9.736946865s Jan 19 02:48:11.464: INFO: Pod "pod-subpath-test-dynamicpv-pvvq": Phase="Succeeded", Reason="", readiness=false. Elapsed: 1m11.84585701s [1mSTEP[0m: Saw pod success Jan 19 02:48:11.464: INFO: Pod "pod-subpath-test-dynamicpv-pvvq" satisfied condition "Succeeded or Failed" Jan 19 02:48:11.572: INFO: Trying to get logs from node k8s-agentpool1-29517166-vmss000000 pod pod-subpath-test-dynamicpv-pvvq container test-container-subpath-dynamicpv-pvvq: <nil> [1mSTEP[0m: delete the pod Jan 19 02:48:11.834: INFO: Waiting for pod pod-subpath-test-dynamicpv-pvvq to disappear Jan 19 02:48:11.941: INFO: Pod pod-subpath-test-dynamicpv-pvvq no longer exists [1mSTEP[0m: Deleting pod pod-subpath-test-dynamicpv-pvvq Jan 19 02:48:11.941: INFO: Deleting pod "pod-subpath-test-dynamicpv-pvvq" in namespace "provisioning-7220" ... skipping 23 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (default fs)] subPath [90mtest/e2e/storage/framework/testsuite.go:50[0m should support file as subpath [LinuxOnly] [90mtest/e2e/storage/testsuites/subpath.go:232[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should support file as subpath [LinuxOnly]","total":32,"completed":11,"skipped":639,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] volumes should store data"]} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [BeforeEach] [Testpattern: Pre-provisioned PV (xfs)][Slow] volumes test/e2e/storage/framework/testsuite.go:51 Jan 19 02:48:53.800: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping ... skipping 38 lines ... Jan 19 02:47:12.330: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.com6q65q] to have phase Bound Jan 19 02:47:12.437: INFO: PersistentVolumeClaim test.csi.azure.com6q65q found but phase is Pending instead of Bound. Jan 19 02:47:14.548: INFO: PersistentVolumeClaim test.csi.azure.com6q65q found but phase is Pending instead of Bound. Jan 19 02:47:16.655: INFO: PersistentVolumeClaim test.csi.azure.com6q65q found and phase=Bound (4.325604873s) [1mSTEP[0m: Creating pod pod-subpath-test-dynamicpv-hw8z [1mSTEP[0m: Creating a pod to test subpath Jan 19 02:47:16.979: INFO: Waiting up to 5m0s for pod "pod-subpath-test-dynamicpv-hw8z" in namespace "provisioning-8432" to be "Succeeded or Failed" Jan 19 02:47:17.088: INFO: Pod "pod-subpath-test-dynamicpv-hw8z": Phase="Pending", Reason="", readiness=false. Elapsed: 108.764091ms Jan 19 02:47:19.196: INFO: Pod "pod-subpath-test-dynamicpv-hw8z": Phase="Pending", Reason="", readiness=false. Elapsed: 2.217277306s Jan 19 02:47:21.305: INFO: Pod "pod-subpath-test-dynamicpv-hw8z": Phase="Pending", Reason="", readiness=false. Elapsed: 4.326125667s Jan 19 02:47:23.414: INFO: Pod "pod-subpath-test-dynamicpv-hw8z": Phase="Pending", Reason="", readiness=false. Elapsed: 6.434439144s Jan 19 02:47:25.522: INFO: Pod "pod-subpath-test-dynamicpv-hw8z": Phase="Pending", Reason="", readiness=false. Elapsed: 8.543094385s Jan 19 02:47:27.631: INFO: Pod "pod-subpath-test-dynamicpv-hw8z": Phase="Pending", Reason="", readiness=false. Elapsed: 10.651365011s ... skipping 21 lines ... Jan 19 02:48:14.041: INFO: Pod "pod-subpath-test-dynamicpv-hw8z": Phase="Pending", Reason="", readiness=false. Elapsed: 57.061430493s Jan 19 02:48:16.150: INFO: Pod "pod-subpath-test-dynamicpv-hw8z": Phase="Pending", Reason="", readiness=false. Elapsed: 59.170337906s Jan 19 02:48:18.258: INFO: Pod "pod-subpath-test-dynamicpv-hw8z": Phase="Pending", Reason="", readiness=false. Elapsed: 1m1.278669957s Jan 19 02:48:20.366: INFO: Pod "pod-subpath-test-dynamicpv-hw8z": Phase="Pending", Reason="", readiness=false. Elapsed: 1m3.386888968s Jan 19 02:48:22.475: INFO: Pod "pod-subpath-test-dynamicpv-hw8z": Phase="Succeeded", Reason="", readiness=false. Elapsed: 1m5.495399019s [1mSTEP[0m: Saw pod success Jan 19 02:48:22.475: INFO: Pod "pod-subpath-test-dynamicpv-hw8z" satisfied condition "Succeeded or Failed" Jan 19 02:48:22.582: INFO: Trying to get logs from node k8s-agentpool1-29517166-vmss000000 pod pod-subpath-test-dynamicpv-hw8z container test-container-subpath-dynamicpv-hw8z: <nil> [1mSTEP[0m: delete the pod Jan 19 02:48:22.808: INFO: Waiting for pod pod-subpath-test-dynamicpv-hw8z to disappear Jan 19 02:48:22.915: INFO: Pod pod-subpath-test-dynamicpv-hw8z no longer exists [1mSTEP[0m: Deleting pod pod-subpath-test-dynamicpv-hw8z Jan 19 02:48:22.915: INFO: Deleting pod "pod-subpath-test-dynamicpv-hw8z" in namespace "provisioning-8432" ... skipping 29 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (default fs)] subPath [90mtest/e2e/storage/framework/testsuite.go:50[0m should support readOnly directory specified in the volumeMount [90mtest/e2e/storage/testsuites/subpath.go:367[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should support readOnly directory specified in the volumeMount","total":33,"completed":12,"skipped":911,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext4)] multiVolume [Slow] should access to two volumes with the same volume mode and retain data across pod recreation on the same node"]} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Dynamic PV (filesystem volmode)] volumeMode[0m [1mshould not mount / map unused volumes in a pod [LinuxOnly][0m [37mtest/e2e/storage/testsuites/volumemode.go:354[0m ... skipping 69 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (filesystem volmode)] volumeMode [90mtest/e2e/storage/framework/testsuite.go:50[0m should not mount / map unused volumes in a pod [LinuxOnly] [90mtest/e2e/storage/testsuites/volumemode.go:354[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (filesystem volmode)] volumeMode should not mount / map unused volumes in a pod [LinuxOnly]","total":37,"completed":10,"skipped":936,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext4)] volumes should store data"]} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [BeforeEach] [Testpattern: Dynamic PV (ntfs)][Feature:Windows] volumeIO test/e2e/storage/framework/testsuite.go:51 Jan 19 02:49:51.071: INFO: Distro debian doesn't support ntfs -- skipping ... skipping 225 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] [90mtest/e2e/storage/framework/testsuite.go:50[0m should access to two volumes with different volume mode and retain data across pod recreation on different node [90mtest/e2e/storage/testsuites/multivolume.go:248[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] should access to two volumes with different volume mode and retain data across pod recreation on different node","total":30,"completed":11,"skipped":876,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow] should concurrently access the single read-only volume from pods on the same node"]} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [BeforeEach] [Testpattern: Pre-provisioned PV (default fs)] subPath test/e2e/storage/framework/testsuite.go:51 Jan 19 02:50:23.839: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping ... skipping 118 lines ... Jan 19 02:50:25.896: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comtlgc4] to have phase Bound Jan 19 02:50:26.003: INFO: PersistentVolumeClaim test.csi.azure.comtlgc4 found but phase is Pending instead of Bound. Jan 19 02:50:28.111: INFO: PersistentVolumeClaim test.csi.azure.comtlgc4 found but phase is Pending instead of Bound. Jan 19 02:50:30.219: INFO: PersistentVolumeClaim test.csi.azure.comtlgc4 found and phase=Bound (4.323352024s) [1mSTEP[0m: Creating pod exec-volume-test-dynamicpv-rlv9 [1mSTEP[0m: Creating a pod to test exec-volume-test Jan 19 02:50:30.544: INFO: Waiting up to 5m0s for pod "exec-volume-test-dynamicpv-rlv9" in namespace "volume-4020" to be "Succeeded or Failed" Jan 19 02:50:30.651: INFO: Pod "exec-volume-test-dynamicpv-rlv9": Phase="Pending", Reason="", readiness=false. Elapsed: 106.913019ms Jan 19 02:50:32.760: INFO: Pod "exec-volume-test-dynamicpv-rlv9": Phase="Pending", Reason="", readiness=false. Elapsed: 2.215456794s Jan 19 02:50:34.869: INFO: Pod "exec-volume-test-dynamicpv-rlv9": Phase="Pending", Reason="", readiness=false. Elapsed: 4.324512507s Jan 19 02:50:36.977: INFO: Pod "exec-volume-test-dynamicpv-rlv9": Phase="Pending", Reason="", readiness=false. Elapsed: 6.433116899s Jan 19 02:50:39.087: INFO: Pod "exec-volume-test-dynamicpv-rlv9": Phase="Pending", Reason="", readiness=false. Elapsed: 8.542720446s Jan 19 02:50:41.195: INFO: Pod "exec-volume-test-dynamicpv-rlv9": Phase="Pending", Reason="", readiness=false. Elapsed: 10.650531629s ... skipping 9 lines ... Jan 19 02:51:02.283: INFO: Pod "exec-volume-test-dynamicpv-rlv9": Phase="Pending", Reason="", readiness=false. Elapsed: 31.738366516s Jan 19 02:51:04.394: INFO: Pod "exec-volume-test-dynamicpv-rlv9": Phase="Pending", Reason="", readiness=false. Elapsed: 33.84995237s Jan 19 02:51:06.509: INFO: Pod "exec-volume-test-dynamicpv-rlv9": Phase="Pending", Reason="", readiness=false. Elapsed: 35.964538948s Jan 19 02:51:08.616: INFO: Pod "exec-volume-test-dynamicpv-rlv9": Phase="Pending", Reason="", readiness=false. Elapsed: 38.072124753s Jan 19 02:51:10.724: INFO: Pod "exec-volume-test-dynamicpv-rlv9": Phase="Succeeded", Reason="", readiness=false. Elapsed: 40.179922001s [1mSTEP[0m: Saw pod success Jan 19 02:51:10.724: INFO: Pod "exec-volume-test-dynamicpv-rlv9" satisfied condition "Succeeded or Failed" Jan 19 02:51:10.832: INFO: Trying to get logs from node k8s-agentpool1-29517166-vmss000001 pod exec-volume-test-dynamicpv-rlv9 container exec-container-dynamicpv-rlv9: <nil> [1mSTEP[0m: delete the pod Jan 19 02:51:11.079: INFO: Waiting for pod exec-volume-test-dynamicpv-rlv9 to disappear Jan 19 02:51:11.187: INFO: Pod exec-volume-test-dynamicpv-rlv9 no longer exists [1mSTEP[0m: Deleting pod exec-volume-test-dynamicpv-rlv9 Jan 19 02:51:11.187: INFO: Deleting pod "exec-volume-test-dynamicpv-rlv9" in namespace "volume-4020" ... skipping 21 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (ext4)] volumes [90mtest/e2e/storage/framework/testsuite.go:50[0m should allow exec of files on the volume [90mtest/e2e/storage/testsuites/volumes.go:198[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext4)] volumes should allow exec of files on the volume","total":30,"completed":12,"skipped":972,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow] should concurrently access the single read-only volume from pods on the same node"]} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [BeforeEach] [Testpattern: Pre-provisioned PV (xfs)][Slow] volumes test/e2e/storage/framework/testsuite.go:51 Jan 19 02:51:52.927: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping ... skipping 48 lines ... Jan 19 02:47:43.118: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comljbg5] to have phase Bound Jan 19 02:47:43.226: INFO: PersistentVolumeClaim test.csi.azure.comljbg5 found but phase is Pending instead of Bound. Jan 19 02:47:45.335: INFO: PersistentVolumeClaim test.csi.azure.comljbg5 found but phase is Pending instead of Bound. Jan 19 02:47:47.444: INFO: PersistentVolumeClaim test.csi.azure.comljbg5 found and phase=Bound (4.325907913s) [1mSTEP[0m: [init] starting a pod to use the claim [1mSTEP[0m: [init] check pod success Jan 19 02:47:47.880: INFO: Waiting up to 15m0s for pod "pvc-snapshottable-tester-llxw7" in namespace "snapshotting-6034" to be "Succeeded or Failed" Jan 19 02:47:47.990: INFO: Pod "pvc-snapshottable-tester-llxw7": Phase="Pending", Reason="", readiness=false. Elapsed: 109.093259ms Jan 19 02:47:50.101: INFO: Pod "pvc-snapshottable-tester-llxw7": Phase="Pending", Reason="", readiness=false. Elapsed: 2.220670262s Jan 19 02:47:52.211: INFO: Pod "pvc-snapshottable-tester-llxw7": Phase="Pending", Reason="", readiness=false. Elapsed: 4.330420791s Jan 19 02:47:54.320: INFO: Pod "pvc-snapshottable-tester-llxw7": Phase="Pending", Reason="", readiness=false. Elapsed: 6.439934149s Jan 19 02:47:56.430: INFO: Pod "pvc-snapshottable-tester-llxw7": Phase="Pending", Reason="", readiness=false. Elapsed: 8.54922699s Jan 19 02:47:58.539: INFO: Pod "pvc-snapshottable-tester-llxw7": Phase="Pending", Reason="", readiness=false. Elapsed: 10.658299432s Jan 19 02:48:00.649: INFO: Pod "pvc-snapshottable-tester-llxw7": Phase="Pending", Reason="", readiness=false. Elapsed: 12.768091827s Jan 19 02:48:02.758: INFO: Pod "pvc-snapshottable-tester-llxw7": Phase="Pending", Reason="", readiness=false. Elapsed: 14.877178981s Jan 19 02:48:04.867: INFO: Pod "pvc-snapshottable-tester-llxw7": Phase="Pending", Reason="", readiness=false. Elapsed: 16.986139399s Jan 19 02:48:06.976: INFO: Pod "pvc-snapshottable-tester-llxw7": Phase="Succeeded", Reason="", readiness=false. Elapsed: 19.095857799s [1mSTEP[0m: Saw pod success Jan 19 02:48:06.976: INFO: Pod "pvc-snapshottable-tester-llxw7" satisfied condition "Succeeded or Failed" [1mSTEP[0m: [init] checking the claim Jan 19 02:48:07.085: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comljbg5] to have phase Bound Jan 19 02:48:07.193: INFO: PersistentVolumeClaim test.csi.azure.comljbg5 found and phase=Bound (108.004466ms) [1mSTEP[0m: [init] checking the PV [1mSTEP[0m: [init] deleting the pod Jan 19 02:48:07.542: INFO: Pod pvc-snapshottable-tester-llxw7 has the following logs: ... skipping 32 lines ... Jan 19 02:48:13.758: INFO: WaitUntil finished successfully after 109.288484ms [1mSTEP[0m: getting the snapshot and snapshot content [1mSTEP[0m: checking the snapshot [1mSTEP[0m: checking the SnapshotContent [1mSTEP[0m: Modifying source data test [1mSTEP[0m: modifying the data in the source PVC Jan 19 02:48:14.317: INFO: Waiting up to 15m0s for pod "pvc-snapshottable-data-tester-vmpql" in namespace "snapshotting-6034" to be "Succeeded or Failed" Jan 19 02:48:14.425: INFO: Pod "pvc-snapshottable-data-tester-vmpql": Phase="Pending", Reason="", readiness=false. Elapsed: 108.387778ms Jan 19 02:48:16.534: INFO: Pod "pvc-snapshottable-data-tester-vmpql": Phase="Pending", Reason="", readiness=false. Elapsed: 2.217397748s Jan 19 02:48:18.644: INFO: Pod "pvc-snapshottable-data-tester-vmpql": Phase="Pending", Reason="", readiness=false. Elapsed: 4.327484234s Jan 19 02:48:20.754: INFO: Pod "pvc-snapshottable-data-tester-vmpql": Phase="Pending", Reason="", readiness=false. Elapsed: 6.437490524s Jan 19 02:48:22.863: INFO: Pod "pvc-snapshottable-data-tester-vmpql": Phase="Pending", Reason="", readiness=false. Elapsed: 8.54642075s Jan 19 02:48:24.973: INFO: Pod "pvc-snapshottable-data-tester-vmpql": Phase="Pending", Reason="", readiness=false. Elapsed: 10.656455276s ... skipping 34 lines ... Jan 19 02:49:38.827: INFO: Pod "pvc-snapshottable-data-tester-vmpql": Phase="Pending", Reason="", readiness=false. Elapsed: 1m24.510396857s Jan 19 02:49:40.937: INFO: Pod "pvc-snapshottable-data-tester-vmpql": Phase="Pending", Reason="", readiness=false. Elapsed: 1m26.620368366s Jan 19 02:49:43.047: INFO: Pod "pvc-snapshottable-data-tester-vmpql": Phase="Pending", Reason="", readiness=false. Elapsed: 1m28.730389816s Jan 19 02:49:45.160: INFO: Pod "pvc-snapshottable-data-tester-vmpql": Phase="Pending", Reason="", readiness=false. Elapsed: 1m30.843315112s Jan 19 02:49:47.269: INFO: Pod "pvc-snapshottable-data-tester-vmpql": Phase="Succeeded", Reason="", readiness=false. Elapsed: 1m32.952289748s [1mSTEP[0m: Saw pod success Jan 19 02:49:47.269: INFO: Pod "pvc-snapshottable-data-tester-vmpql" satisfied condition "Succeeded or Failed" Jan 19 02:49:47.490: INFO: Pod pvc-snapshottable-data-tester-vmpql has the following logs: Jan 19 02:49:47.490: INFO: Deleting pod "pvc-snapshottable-data-tester-vmpql" in namespace "snapshotting-6034" Jan 19 02:49:47.603: INFO: Wait up to 5m0s for pod "pvc-snapshottable-data-tester-vmpql" to be fully deleted [1mSTEP[0m: creating a pvc from the snapshot [1mSTEP[0m: starting a pod to use the snapshot Jan 19 02:51:06.156: INFO: Running '/usr/local/bin/kubectl --server=https://kubetest-iei3lcrr.westeurope.cloudapp.azure.com --kubeconfig=/root/tmp1374176058/kubeconfig/kubeconfig.westeurope.json --namespace=snapshotting-6034 exec restored-pvc-tester-d7r6l --namespace=snapshotting-6034 -- cat /mnt/test/data' ... skipping 33 lines ... Jan 19 02:51:32.431: INFO: volumesnapshotcontents pre-provisioned-snapcontent-2da0ad61-9d1d-4abe-a324-f2f00fb3e8ec has been found and is not deleted Jan 19 02:51:33.539: INFO: volumesnapshotcontents pre-provisioned-snapcontent-2da0ad61-9d1d-4abe-a324-f2f00fb3e8ec has been found and is not deleted Jan 19 02:51:34.649: INFO: volumesnapshotcontents pre-provisioned-snapcontent-2da0ad61-9d1d-4abe-a324-f2f00fb3e8ec has been found and is not deleted Jan 19 02:51:35.758: INFO: volumesnapshotcontents pre-provisioned-snapcontent-2da0ad61-9d1d-4abe-a324-f2f00fb3e8ec has been found and is not deleted Jan 19 02:51:36.872: INFO: volumesnapshotcontents pre-provisioned-snapcontent-2da0ad61-9d1d-4abe-a324-f2f00fb3e8ec has been found and is not deleted Jan 19 02:51:37.984: INFO: volumesnapshotcontents pre-provisioned-snapcontent-2da0ad61-9d1d-4abe-a324-f2f00fb3e8ec has been found and is not deleted Jan 19 02:51:38.984: INFO: WaitUntil failed after reaching the timeout 30s [AfterEach] volume snapshot controller test/e2e/storage/testsuites/snapshottable.go:172 Jan 19 02:51:39.124: INFO: Pod restored-pvc-tester-d7r6l has the following logs: Jan 19 02:51:39.124: INFO: Deleting pod "restored-pvc-tester-d7r6l" in namespace "snapshotting-6034" Jan 19 02:51:39.232: INFO: Wait up to 5m0s for pod "restored-pvc-tester-d7r6l" to be fully deleted Jan 19 02:51:41.449: INFO: deleting claim "snapshotting-6034"/"pvc-q9kwj" ... skipping 31 lines ... [90mtest/e2e/storage/testsuites/snapshottable.go:113[0m [90mtest/e2e/storage/testsuites/snapshottable.go:176[0m should check snapshot fields, check restore correctly works after modifying source data, check deletion (persistent) [90mtest/e2e/storage/testsuites/snapshottable.go:278[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Pre-provisioned Snapshot (retain policy)] snapshottable[Feature:VolumeSnapshotDataSource] volume snapshot controller should check snapshot fields, check restore correctly works after modifying source data, check deletion (persistent)","total":31,"completed":11,"skipped":964,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Pre-provisioned Snapshot (delete policy)] snapshottable[Feature:VolumeSnapshotDataSource] volume snapshot controller should check snapshot fields, check restore correctly works after modifying source data, check deletion (persistent)"]} [36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Dynamic PV (block volmode)] provisioning[0m [1mshould provision storage with pvc data source in parallel [Slow][0m [37mtest/e2e/storage/testsuites/provisioning.go:459[0m ... skipping 318 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (block volmode)] provisioning [90mtest/e2e/storage/framework/testsuite.go:50[0m should provision storage with pvc data source in parallel [Slow] [90mtest/e2e/storage/testsuites/provisioning.go:459[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] provisioning should provision storage with pvc data source in parallel [Slow]","total":33,"completed":13,"skipped":938,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext4)] multiVolume [Slow] should access to two volumes with the same volume mode and retain data across pod recreation on the same node"]} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [BeforeEach] [Testpattern: Pre-provisioned PV (block volmode)] volumeMode test/e2e/storage/framework/testsuite.go:51 Jan 19 02:53:01.932: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping ... skipping 3 lines ... [36m[1mS [SKIPPING] in Spec Setup (BeforeEach) [0.001 seconds][0m External Storage [Driver: test.csi.azure.com] [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Pre-provisioned PV (block volmode)] volumeMode [90mtest/e2e/storage/framework/testsuite.go:50[0m [36m[1mshould fail to use a volume in a pod with mismatched mode [Slow] [BeforeEach][0m [90mtest/e2e/storage/testsuites/volumemode.go:299[0m [36mDriver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping[0m test/e2e/storage/external/external.go:262 [90m------------------------------[0m ... skipping 113 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (ext4)] multiVolume [Slow] [90mtest/e2e/storage/framework/testsuite.go:50[0m should concurrently access the volume and restored snapshot from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS] [90mtest/e2e/storage/testsuites/multivolume.go:323[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext4)] multiVolume [Slow] should concurrently access the volume and restored snapshot from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS]","total":32,"completed":12,"skipped":669,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] volumes should store data"]} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Dynamic PV (default fs)] subPath[0m [1mshould fail if subpath file is outside the volume [Slow][LinuxOnly][0m [37mtest/e2e/storage/testsuites/subpath.go:258[0m [BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath test/e2e/storage/framework/testsuite.go:51 [BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath test/e2e/framework/framework.go:187 [1mSTEP[0m: Creating a kubernetes client Jan 19 02:53:01.944: INFO: >>> kubeConfig: /root/tmp1374176058/kubeconfig/kubeconfig.westeurope.json [1mSTEP[0m: Building a namespace api object, basename provisioning [1mSTEP[0m: Waiting for a default service account to be provisioned in namespace [1mSTEP[0m: Waiting for kube-root-ca.crt to be provisioned in namespace [It] should fail if subpath file is outside the volume [Slow][LinuxOnly] test/e2e/storage/testsuites/subpath.go:258 Jan 19 02:53:02.692: INFO: Creating resource for dynamic PV Jan 19 02:53:02.692: INFO: Using claimSize:5Gi, test suite supported size:{ 1Mi}, driver(test.csi.azure.com) supported size:{ 1Mi} [1mSTEP[0m: creating a StorageClass provisioning-1615-e2e-scb58bn [1mSTEP[0m: creating a claim Jan 19 02:53:02.801: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil Jan 19 02:53:02.911: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.combxrc6] to have phase Bound Jan 19 02:53:03.017: INFO: PersistentVolumeClaim test.csi.azure.combxrc6 found but phase is Pending instead of Bound. Jan 19 02:53:05.125: INFO: PersistentVolumeClaim test.csi.azure.combxrc6 found but phase is Pending instead of Bound. Jan 19 02:53:07.238: INFO: PersistentVolumeClaim test.csi.azure.combxrc6 found and phase=Bound (4.327222456s) [1mSTEP[0m: Creating pod pod-subpath-test-dynamicpv-sb99 [1mSTEP[0m: Checking for subpath error in container status Jan 19 02:53:39.790: INFO: Deleting pod "pod-subpath-test-dynamicpv-sb99" in namespace "provisioning-1615" Jan 19 02:53:39.901: INFO: Wait up to 5m0s for pod "pod-subpath-test-dynamicpv-sb99" to be fully deleted [1mSTEP[0m: Deleting pod Jan 19 02:53:42.124: INFO: Deleting pod "pod-subpath-test-dynamicpv-sb99" in namespace "provisioning-1615" [1mSTEP[0m: Deleting pvc Jan 19 02:53:42.231: INFO: Deleting PersistentVolumeClaim "test.csi.azure.combxrc6" ... skipping 16 lines ... [32m• [SLOW TEST:81.921 seconds][0m External Storage [Driver: test.csi.azure.com] [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (default fs)] subPath [90mtest/e2e/storage/framework/testsuite.go:50[0m should fail if subpath file is outside the volume [Slow][LinuxOnly] [90mtest/e2e/storage/testsuites/subpath.go:258[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should fail if subpath file is outside the volume [Slow][LinuxOnly]","total":33,"completed":14,"skipped":991,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext4)] multiVolume [Slow] should access to two volumes with the same volume mode and retain data across pod recreation on the same node"]} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m Jan 19 02:54:23.895: INFO: Running AfterSuite actions on all nodes Jan 19 02:54:23.896: INFO: Running Cleanup Action: k8s.io/kubernetes/test/e2e/storage.glob..func19.2 Jan 19 02:54:23.896: INFO: Running Cleanup Action: k8s.io/kubernetes/test/e2e/storage.glob..func9.2 ... skipping 27 lines ... Jan 19 02:52:00.640: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comcxzf5] to have phase Bound Jan 19 02:52:00.748: INFO: PersistentVolumeClaim test.csi.azure.comcxzf5 found but phase is Pending instead of Bound. Jan 19 02:52:02.857: INFO: PersistentVolumeClaim test.csi.azure.comcxzf5 found but phase is Pending instead of Bound. Jan 19 02:52:04.965: INFO: PersistentVolumeClaim test.csi.azure.comcxzf5 found and phase=Bound (4.325722346s) [1mSTEP[0m: [init] starting a pod to use the claim [1mSTEP[0m: [init] check pod success Jan 19 02:52:05.400: INFO: Waiting up to 15m0s for pod "pvc-snapshottable-tester-vscwk" in namespace "snapshotting-8130" to be "Succeeded or Failed" Jan 19 02:52:05.509: INFO: Pod "pvc-snapshottable-tester-vscwk": Phase="Pending", Reason="", readiness=false. Elapsed: 108.845359ms Jan 19 02:52:07.619: INFO: Pod "pvc-snapshottable-tester-vscwk": Phase="Pending", Reason="", readiness=false. Elapsed: 2.218245695s Jan 19 02:52:09.729: INFO: Pod "pvc-snapshottable-tester-vscwk": Phase="Pending", Reason="", readiness=false. Elapsed: 4.32839504s Jan 19 02:52:11.839: INFO: Pod "pvc-snapshottable-tester-vscwk": Phase="Pending", Reason="", readiness=false. Elapsed: 6.438372462s Jan 19 02:52:13.948: INFO: Pod "pvc-snapshottable-tester-vscwk": Phase="Pending", Reason="", readiness=false. Elapsed: 8.547573915s Jan 19 02:52:16.058: INFO: Pod "pvc-snapshottable-tester-vscwk": Phase="Pending", Reason="", readiness=false. Elapsed: 10.657157034s Jan 19 02:52:18.168: INFO: Pod "pvc-snapshottable-tester-vscwk": Phase="Pending", Reason="", readiness=false. Elapsed: 12.767499588s Jan 19 02:52:20.276: INFO: Pod "pvc-snapshottable-tester-vscwk": Phase="Pending", Reason="", readiness=false. Elapsed: 14.875903784s Jan 19 02:52:22.386: INFO: Pod "pvc-snapshottable-tester-vscwk": Phase="Pending", Reason="", readiness=false. Elapsed: 16.985138392s Jan 19 02:52:24.507: INFO: Pod "pvc-snapshottable-tester-vscwk": Phase="Pending", Reason="", readiness=false. Elapsed: 19.106108464s Jan 19 02:52:26.617: INFO: Pod "pvc-snapshottable-tester-vscwk": Phase="Pending", Reason="", readiness=false. Elapsed: 21.216597734s Jan 19 02:52:28.726: INFO: Pod "pvc-snapshottable-tester-vscwk": Phase="Succeeded", Reason="", readiness=false. Elapsed: 23.325436922s [1mSTEP[0m: Saw pod success Jan 19 02:52:28.726: INFO: Pod "pvc-snapshottable-tester-vscwk" satisfied condition "Succeeded or Failed" [1mSTEP[0m: [init] checking the claim Jan 19 02:52:28.835: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comcxzf5] to have phase Bound Jan 19 02:52:28.943: INFO: PersistentVolumeClaim test.csi.azure.comcxzf5 found and phase=Bound (108.362701ms) [1mSTEP[0m: [init] checking the PV [1mSTEP[0m: [init] deleting the pod Jan 19 02:52:29.273: INFO: Pod pvc-snapshottable-tester-vscwk has the following logs: ... skipping 13 lines ... Jan 19 02:52:36.700: INFO: received snapshotStatus map[boundVolumeSnapshotContentName:snapcontent-938afd56-0702-411f-b730-07a3f4c97396 creationTime:2023-01-19T02:52:32Z readyToUse:true restoreSize:5Gi] Jan 19 02:52:36.700: INFO: snapshotContentName snapcontent-938afd56-0702-411f-b730-07a3f4c97396 [1mSTEP[0m: checking the snapshot [1mSTEP[0m: checking the SnapshotContent [1mSTEP[0m: Modifying source data test [1mSTEP[0m: modifying the data in the source PVC Jan 19 02:52:37.137: INFO: Waiting up to 15m0s for pod "pvc-snapshottable-data-tester-rk79l" in namespace "snapshotting-8130" to be "Succeeded or Failed" Jan 19 02:52:37.246: INFO: Pod "pvc-snapshottable-data-tester-rk79l": Phase="Pending", Reason="", readiness=false. Elapsed: 109.314166ms Jan 19 02:52:39.356: INFO: Pod "pvc-snapshottable-data-tester-rk79l": Phase="Pending", Reason="", readiness=false. Elapsed: 2.218810623s Jan 19 02:52:41.465: INFO: Pod "pvc-snapshottable-data-tester-rk79l": Phase="Pending", Reason="", readiness=false. Elapsed: 4.328370814s Jan 19 02:52:43.575: INFO: Pod "pvc-snapshottable-data-tester-rk79l": Phase="Pending", Reason="", readiness=false. Elapsed: 6.437687635s Jan 19 02:52:45.685: INFO: Pod "pvc-snapshottable-data-tester-rk79l": Phase="Pending", Reason="", readiness=false. Elapsed: 8.547456057s Jan 19 02:52:47.793: INFO: Pod "pvc-snapshottable-data-tester-rk79l": Phase="Pending", Reason="", readiness=false. Elapsed: 10.656123538s ... skipping 4 lines ... Jan 19 02:52:58.341: INFO: Pod "pvc-snapshottable-data-tester-rk79l": Phase="Pending", Reason="", readiness=false. Elapsed: 21.203622617s Jan 19 02:53:00.450: INFO: Pod "pvc-snapshottable-data-tester-rk79l": Phase="Pending", Reason="", readiness=false. Elapsed: 23.312996224s Jan 19 02:53:02.560: INFO: Pod "pvc-snapshottable-data-tester-rk79l": Phase="Pending", Reason="", readiness=false. Elapsed: 25.422720509s Jan 19 02:53:04.669: INFO: Pod "pvc-snapshottable-data-tester-rk79l": Phase="Pending", Reason="", readiness=false. Elapsed: 27.531788626s Jan 19 02:53:06.779: INFO: Pod "pvc-snapshottable-data-tester-rk79l": Phase="Succeeded", Reason="", readiness=false. Elapsed: 29.642248339s [1mSTEP[0m: Saw pod success Jan 19 02:53:06.779: INFO: Pod "pvc-snapshottable-data-tester-rk79l" satisfied condition "Succeeded or Failed" Jan 19 02:53:07.003: INFO: Pod pvc-snapshottable-data-tester-rk79l has the following logs: Jan 19 02:53:07.003: INFO: Deleting pod "pvc-snapshottable-data-tester-rk79l" in namespace "snapshotting-8130" Jan 19 02:53:07.128: INFO: Wait up to 5m0s for pod "pvc-snapshottable-data-tester-rk79l" to be fully deleted [1mSTEP[0m: creating a pvc from the snapshot [1mSTEP[0m: starting a pod to use the snapshot Jan 19 02:53:53.717: INFO: Running '/usr/local/bin/kubectl --server=https://kubetest-iei3lcrr.westeurope.cloudapp.azure.com --kubeconfig=/root/tmp1374176058/kubeconfig/kubeconfig.westeurope.json --namespace=snapshotting-8130 exec restored-pvc-tester-2x49w --namespace=snapshotting-8130 -- cat /mnt/test/data' ... skipping 33 lines ... Jan 19 02:54:19.995: INFO: volumesnapshotcontents snapcontent-938afd56-0702-411f-b730-07a3f4c97396 has been found and is not deleted Jan 19 02:54:21.104: INFO: volumesnapshotcontents snapcontent-938afd56-0702-411f-b730-07a3f4c97396 has been found and is not deleted Jan 19 02:54:22.212: INFO: volumesnapshotcontents snapcontent-938afd56-0702-411f-b730-07a3f4c97396 has been found and is not deleted Jan 19 02:54:23.321: INFO: volumesnapshotcontents snapcontent-938afd56-0702-411f-b730-07a3f4c97396 has been found and is not deleted Jan 19 02:54:24.429: INFO: volumesnapshotcontents snapcontent-938afd56-0702-411f-b730-07a3f4c97396 has been found and is not deleted Jan 19 02:54:25.537: INFO: volumesnapshotcontents snapcontent-938afd56-0702-411f-b730-07a3f4c97396 has been found and is not deleted Jan 19 02:54:26.538: INFO: WaitUntil failed after reaching the timeout 30s [AfterEach] volume snapshot controller test/e2e/storage/testsuites/snapshottable.go:172 Jan 19 02:54:26.645: INFO: Error getting logs for pod restored-pvc-tester-2x49w: the server could not find the requested resource (get pods restored-pvc-tester-2x49w) Jan 19 02:54:26.645: INFO: Deleting pod "restored-pvc-tester-2x49w" in namespace "snapshotting-8130" Jan 19 02:54:26.752: INFO: deleting claim "snapshotting-8130"/"pvc-zdrgk" Jan 19 02:54:26.859: INFO: deleting snapshot "snapshotting-8130"/"snapshot-s98k2" Jan 19 02:54:26.968: INFO: deleting snapshot content "snapcontent-938afd56-0702-411f-b730-07a3f4c97396" Jan 19 02:54:27.298: INFO: Waiting up to 5m0s for volumesnapshotcontents snapcontent-938afd56-0702-411f-b730-07a3f4c97396 to be deleted Jan 19 02:54:27.406: INFO: volumesnapshotcontents snapcontent-938afd56-0702-411f-b730-07a3f4c97396 has been found and is not deleted ... skipping 27 lines ... [90mtest/e2e/storage/testsuites/snapshottable.go:113[0m [90mtest/e2e/storage/testsuites/snapshottable.go:176[0m should check snapshot fields, check restore correctly works after modifying source data, check deletion (persistent) [90mtest/e2e/storage/testsuites/snapshottable.go:278[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic Snapshot (retain policy)] snapshottable[Feature:VolumeSnapshotDataSource] volume snapshot controller should check snapshot fields, check restore correctly works after modifying source data, check deletion (persistent)","total":31,"completed":12,"skipped":967,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Pre-provisioned Snapshot (delete policy)] snapshottable[Feature:VolumeSnapshotDataSource] volume snapshot controller should check snapshot fields, check restore correctly works after modifying source data, check deletion (persistent)"]} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [BeforeEach] [Testpattern: Inline-volume (default fs)] subPath test/e2e/storage/framework/testsuite.go:51 Jan 19 02:54:44.970: INFO: Driver "test.csi.azure.com" does not support volume type "InlineVolume" - skipping ... skipping 174 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (default fs)] provisioning [90mtest/e2e/storage/framework/testsuite.go:50[0m should provision storage with snapshot data source [Feature:VolumeSnapshotDataSource] [90mtest/e2e/storage/testsuites/provisioning.go:208[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] provisioning should provision storage with snapshot data source [Feature:VolumeSnapshotDataSource]","total":32,"completed":13,"skipped":784,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] volumes should store data"]} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [BeforeEach] [Testpattern: Dynamic PV (ntfs)][Feature:Windows] provisioning test/e2e/storage/framework/testsuite.go:51 Jan 19 02:57:27.761: INFO: Distro debian doesn't support ntfs -- skipping ... skipping 45 lines ... [36m[1mS [SKIPPING] in Spec Setup (BeforeEach) [0.001 seconds][0m External Storage [Driver: test.csi.azure.com] [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (ntfs)][Feature:Windows] subPath [90mtest/e2e/storage/framework/testsuite.go:50[0m [36m[1mshould fail if subpath file is outside the volume [Slow][LinuxOnly] [BeforeEach][0m [90mtest/e2e/storage/testsuites/subpath.go:258[0m [36mDistro debian doesn't support ntfs -- skipping[0m test/e2e/storage/framework/testsuite.go:127 [90m------------------------------[0m ... skipping 128 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (ext4)] multiVolume [Slow] [90mtest/e2e/storage/framework/testsuite.go:50[0m should concurrently access the volume and its clone from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS] [90mtest/e2e/storage/testsuites/multivolume.go:378[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext4)] multiVolume [Slow] should concurrently access the volume and its clone from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS]","total":32,"completed":14,"skipped":934,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] volumes should store data"]} [36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [BeforeEach] [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow] test/e2e/storage/framework/testsuite.go:51 [BeforeEach] [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow] ... skipping 79 lines ... Jan 19 02:59:07.315: INFO: Running AfterSuite actions on node 1 [91m[1mSummarizing 6 Failures:[0m [91m[1m[Fail] [0m[90mExternal Storage [Driver: test.csi.azure.com] [0m[0m[Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow] [0m[91m[1m[Measurement] should concurrently access the single read-only volume from pods on the same node [0m [37mtest/e2e/storage/testsuites/multivolume.go:129[0m [91m[1m[Fail] [0m[90mExternal Storage [Driver: test.csi.azure.com] [0m[0m[Testpattern: Dynamic PV (ext4)] multiVolume [Slow] [0m[91m[1m[Measurement] should access to two volumes with the same volume mode and retain data across pod recreation on the same node [0m [37mtest/e2e/storage/testsuites/multivolume.go:129[0m [91m[1m[Fail] [0m[90mExternal Storage [Driver: test.csi.azure.com] [0m[0m[Testpattern: Pre-provisioned Snapshot (delete policy)] snapshottable[Feature:VolumeSnapshotDataSource] [0m[90mvolume snapshot controller [0m[91m[1m[It] [0m[90mshould check snapshot fields, check restore correctly works after modifying source data, check deletion (persistent) [0m [37mtest/e2e/storage/testsuites/snapshottable.go:142[0m [91m[1m[Fail] [0m[90mExternal Storage [Driver: test.csi.azure.com] [0m[0m[Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] [0m[91m[1m[Measurement] should concurrently access the single volume from pods on the same node [0m [37mtest/e2e/storage/testsuites/multivolume.go:577[0m [91m[1m[Fail] [0m[90mExternal Storage [Driver: test.csi.azure.com] [0m[0m[Testpattern: Dynamic PV (default fs)] volumes [0m[91m[1m[Measurement] should store data [0m [37mtest/e2e/framework/volume/fixtures.go:580[0m [91m[1m[Fail] [0m[90mExternal Storage [Driver: test.csi.azure.com] [0m[0m[Testpattern: Dynamic PV (ext4)] volumes [0m[91m[1m[Measurement] should store data [0m [37mtest/e2e/framework/volume/fixtures.go:580[0m [1m[91mRan 89 of 7227 Specs in 3200.392 seconds[0m [1m[91mFAIL![0m -- [32m[1m83 Passed[0m | [91m[1m6 Failed[0m | [33m[1m0 Pending[0m | [36m[1m7138 Skipped[0m Ginkgo ran 1 suite in 53m25.229186365s Test Suite Failed + print_logs + sed -i s/disk.csi.azure.com/test.csi.azure.com/g deploy/example/storageclass-azuredisk-csi.yaml + bash ./hack/verify-examples.sh linux azurepubliccloud ephemeral test begin to create deployment examples ... storageclass.storage.k8s.io/managed-csi created Applying config "deploy/example/deployment.yaml" ... skipping 101 lines ... I0119 02:05:37.447948 1 reflector.go:257] Listing and watching *v1beta2.AzVolumeAttachment from sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132 I0119 02:05:37.447959 1 reflector.go:221] Starting reflector *v1beta2.AzVolume (30s) from sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132 I0119 02:05:37.447968 1 reflector.go:257] Listing and watching *v1beta2.AzVolume from sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132 I0119 02:05:37.548072 1 shared_informer.go:303] caches populated I0119 02:05:37.548113 1 azuredisk_v2.go:225] driver userAgent: test.csi.azure.com/latest-v2-ec131e57ca07be2e17dba65bd8a030b3e7a9034d e2e-test I0119 02:05:37.548125 1 azure_disk_utils.go:564] reading cloud config from secret kube-system/azure-cloud-provider I0119 02:05:37.551076 1 azure_disk_utils.go:571] InitializeCloudFromSecret: failed to get cloud config from secret kube-system/azure-cloud-provider: failed to get secret kube-system/azure-cloud-provider: secrets "azure-cloud-provider" not found I0119 02:05:37.551095 1 azure_disk_utils.go:576] could not read cloud config from secret kube-system/azure-cloud-provider I0119 02:05:37.551101 1 azure_disk_utils.go:586] use default AZURE_CREDENTIAL_FILE env var: /etc/kubernetes/azure.json I0119 02:05:37.551135 1 azure_disk_utils.go:594] read cloud config from file: /etc/kubernetes/azure.json successfully I0119 02:05:37.551819 1 azure_auth.go:253] Using AzurePublicCloud environment I0119 02:05:37.551837 1 azure_auth.go:104] azure: using managed identity extension to retrieve access token I0119 02:05:37.551842 1 azure_auth.go:110] azure: using User Assigned MSI ID to retrieve access token ... skipping 29 lines ... I0119 02:05:37.552224 1 azure_vmasclient.go:73] Azure AvailabilitySetsClient (write ops) using rate limit config: QPS=100, bucket=1000 I0119 02:05:37.552269 1 azure.go:1024] attach/detach disk operation rate limiter configuration - Enabled: true QPS: 1.333000, Bucket: 240 I0119 02:05:37.552328 1 azuredisk_v2.go:248] disable UseInstanceMetadata for controller I0119 02:05:37.552342 1 azuredisk_v2.go:258] DisableAvailabilitySetNodes for controller since current VMType is vmss I0119 02:05:37.552351 1 azuredisk_v2.go:264] cloud: AzurePublicCloud, location: westeurope, rg: kubetest-iei3lcrr, VMType: vmss, PrimaryScaleSetName: k8s-agentpool1-29517166-vmss, PrimaryAvailabilitySetName: , DisableAvailabilitySetNodes: true I0119 02:05:37.552360 1 skus.go:123] NewNodeInfo: Starting to populate node and disk sku information. E0119 02:05:37.552378 1 azuredisk_v2.go:272] Failed to get node info. Error: NewNodeInfo: Failed to get instance type from Azure cloud provider, nodeName: , error: not a vmss instance I0119 02:05:37.552474 1 mount_linux.go:208] Detected OS without systemd I0119 02:05:37.552490 1 driver.go:81] Enabling controller service capability: CREATE_DELETE_VOLUME I0119 02:05:37.552498 1 driver.go:81] Enabling controller service capability: PUBLISH_UNPUBLISH_VOLUME I0119 02:05:37.552503 1 driver.go:81] Enabling controller service capability: CREATE_DELETE_SNAPSHOT I0119 02:05:37.552509 1 driver.go:81] Enabling controller service capability: CLONE_VOLUME I0119 02:05:37.552515 1 driver.go:81] Enabling controller service capability: EXPAND_VOLUME ... skipping 32995 lines ... 51822 I0119 02:09:54.483849 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:09:54.483900 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="56488684-979e-11ed-af75-002248822ac5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-c5c646a5-a7e9-44a3-a60c-830d891ca7a1" "latency"=11826424515 I0119 02:09:54.483921 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="56488684-979e-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=11826514117 I0119 02:09:54.637396 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdd under /dev/disk/azure/scsi1/ I0119 02:09:54.637456 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType I0119 02:09:54.637474 1 utils.go:85] GRPC response: {} I0119 02:09:54.641274 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities ... skipping 259 lines ... I0119 02:11:18.144078 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities I0119 02:11:18.144094 1 utils.go:79] GRPC request: {} I0119 02:11:18.144131 1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]} I0119 02:11:18.144939 1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume I0119 02:11:18.144959 1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-3953852e-5d09-4b69-aeac-18f10ea101ae/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-3953852e-5d09-4b69-aeac-18f10ea101ae","csi.storage.k8s.io/pvc/name":"test.csi.azure.comb7stb","csi.storage.k8s.io/pvc/namespace":"provisioning-7403","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1674093937787-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-iei3lcrr/providers/Microsoft.Compute/disks/pvc-3953852e-5d09-4b69-aeac-18f10ea101ae"} I0119 02:11:18.145165 1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-3953852e-5d09-4b69-aeac-18f10ea101ae-k8s-agentpool1-29517166-vmss000000-attachment) I0119 02:11:27.662530 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:11:27.662600 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="8f32cc5a-979e-11ed-af75-002248822ac5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-3953852e-5d09-4b69-aeac-18f10ea101ae" "latency"=9517359618 I0119 02:11:27.662634 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="8f32cc5a-979e-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9517480119 I0119 02:11:28.304828 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities I0119 02:11:28.304851 1 utils.go:79] GRPC request: {} I0119 02:11:28.304893 1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]} I0119 02:11:28.307882 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities ... skipping 55 lines ... W0119 02:11:33.041849 1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-3953852e-5d09-4b69-aeac-18f10ea101ae/globalmount" is not a mountpoint, deleting I0119 02:11:33.041911 1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-3953852e-5d09-4b69-aeac-18f10ea101ae/globalmount successfully I0119 02:11:33.041925 1 utils.go:85] GRPC response: {} I0119 02:11:34.220786 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:11:34.220821 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:11:34.220860 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:11:34.220887 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:11:34.220891 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:11:34.279178 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:11:37.999689 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:11:37.999798 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="9541c535-979e-11ed-af75-002248822ac5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-128fcf9c-cde5-4a63-8682-a2d815b75b81" "latency"=9690056181 I0119 02:11:37.999833 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="9541c535-979e-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9690228683 I0119 02:11:38.565254 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdd under /dev/disk/azure/scsi1/ I0119 02:11:38.565299 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType I0119 02:11:38.565324 1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun1 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-128fcf9c-cde5-4a63-8682-a2d815b75b81/globalmount with mount options([]) I0119 02:11:38.565332 1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun1" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun1]) ... skipping 61 lines ... I0119 02:11:56.598473 1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]} I0119 02:11:56.599109 1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume I0119 02:11:56.599125 1 utils.go:79] GRPC request: {"publish_context":{"LUN":"2"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-4a1f8cf6-9776-4f0f-9fa9-1ed08ab81381/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-4a1f8cf6-9776-4f0f-9fa9-1ed08ab81381","csi.storage.k8s.io/pvc/name":"test.csi.azure.comzwm6b","csi.storage.k8s.io/pvc/namespace":"provisioning-8888","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1674093937787-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-iei3lcrr/providers/Microsoft.Compute/disks/pvc-4a1f8cf6-9776-4f0f-9fa9-1ed08ab81381"} I0119 02:11:56.599298 1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-4a1f8cf6-9776-4f0f-9fa9-1ed08ab81381-k8s-agentpool1-29517166-vmss000000-attachment) I0119 02:12:04.221188 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:12:04.221218 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:12:04.221313 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:12:04.221352 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:12:04.221356 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:12:04.279719 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:12:14.779949 1 azuredisk_v2.go:578] "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-19T02:12:14Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="b0f49376-979e-11ed-af75-002248822ac5" I0119 02:12:14.793783 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-29517166-vmss000000/status 200 OK in 13 milliseconds I0119 02:12:14.794046 1 azuredisk_v2.go:565] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="b0f49376-979e-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=14133390 ... skipping 3 lines ... I0119 02:12:29.209334 1 reflector.go:559] k8s.io/client-go/informers/factory.go:150: Watch close - *v1.Node total 113 items received I0119 02:12:29.210799 1 round_trippers.go:553] GET https://10.0.0.1:443/api/v1/nodes?allowWatchBookmarks=true&resourceVersion=6672&timeout=9m39s&timeoutSeconds=579&watch=true 200 OK in 1 milliseconds I0119 02:12:34.222132 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:12:34.222166 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:12:34.222201 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:12:34.222243 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:12:34.222292 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:12:34.280590 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:12:44.050904 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:12:44.050968 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="a61e6f05-979e-11ed-af75-002248822ac5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-4a1f8cf6-9776-4f0f-9fa9-1ed08ab81381" "latency"=47451598923 I0119 02:12:44.051003 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="a61e6f05-979e-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=47451710625 I0119 02:12:44.778610 1 azuredisk_v2.go:578] "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-19T02:12:44Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="c2d60234-979e-11ed-af75-002248822ac5" I0119 02:12:44.787693 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-29517166-vmss000000/status 200 OK in 8 milliseconds I0119 02:12:44.787922 1 azuredisk_v2.go:565] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="c2d60234-979e-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=9347270 I0119 02:12:45.041990 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities ... skipping 56 lines ... I0119 02:12:46.416453 1 nodeserver_v2.go:353] NodePublishVolume: mount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-4a1f8cf6-9776-4f0f-9fa9-1ed08ab81381/globalmount at /var/lib/kubelet/pods/b8c05870-b8de-4891-a9eb-28cfeed699ed/volumes/kubernetes.io~csi/pvc-4a1f8cf6-9776-4f0f-9fa9-1ed08ab81381/mount successfully I0119 02:12:46.416467 1 utils.go:85] GRPC response: {} I0119 02:13:04.223053 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:13:04.223142 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:13:04.223168 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:13:04.223186 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:13:04.223267 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:13:04.223294 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:13:04.281442 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:13:05.072758 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:13:05.072822 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="c2ff0452-979e-11ed-af75-002248822ac5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-caf4d040-a20b-42e3-bd47-23095f505e7b" "latency"=20025429455 I0119 02:13:05.072842 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="c2ff0452-979e-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=20025527555 I0119 02:13:05.078356 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:13:05.078413 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="c2fefee1-979e-11ed-af75-002248822ac5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-6338796d-185e-4c9a-a6c3-e3d4496b6d84" "latency"=20031161920 I0119 02:13:05.078450 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="c2fefee1-979e-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=20031269620 I0119 02:13:05.650397 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sde under /dev/disk/azure/scsi1/ I0119 02:13:05.650408 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdd under /dev/disk/azure/scsi1/ I0119 02:13:05.650447 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType I0119 02:13:05.650459 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType ... skipping 173 lines ... I0119 02:14:21.561586 1 nodeserver_v2.go:257] NodeUnstageVolume: unmounting /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-4a1f8cf6-9776-4f0f-9fa9-1ed08ab81381/globalmount I0119 02:14:21.561615 1 mount_helper_common.go:99] "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-4a1f8cf6-9776-4f0f-9fa9-1ed08ab81381/globalmount" is a mountpoint, unmounting I0119 02:14:21.561632 1 mount_linux.go:294] Unmounting /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-4a1f8cf6-9776-4f0f-9fa9-1ed08ab81381/globalmount W0119 02:14:21.567814 1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-4a1f8cf6-9776-4f0f-9fa9-1ed08ab81381/globalmount" is not a mountpoint, deleting I0119 02:14:21.567888 1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-4a1f8cf6-9776-4f0f-9fa9-1ed08ab81381/globalmount successfully I0119 02:14:21.567903 1 utils.go:85] GRPC response: {} I0119 02:14:25.176219 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:14:25.176341 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="f2b05c3c-979e-11ed-af75-002248822ac5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-da1b00f3-5874-46e2-90ff-f463da550578" "latency"=20113725771 I0119 02:14:25.176385 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="f2b05c3c-979e-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=20113899072 I0119 02:14:25.185265 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:14:25.185322 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="f2cceee9-979e-11ed-af75-002248822ac5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-da7ad20b-e1f4-4037-bd0f-67e616879e07" "latency"=19935526123 I0119 02:14:25.185351 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="f2cceee9-979e-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=19935621224 I0119 02:14:25.776159 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdd under /dev/disk/azure/scsi1/ I0119 02:14:25.776207 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType I0119 02:14:25.776251 1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun1 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-da1b00f3-5874-46e2-90ff-f463da550578/globalmount with mount options([]) I0119 02:14:25.776269 1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun1" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun1]) ... skipping 115 lines ... I0119 02:14:44.778308 1 azuredisk_v2.go:578] "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-19T02:14:44Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="0a5c80b0-979f-11ed-af75-002248822ac5" I0119 02:14:44.786756 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-29517166-vmss000000/status 200 OK in 8 milliseconds I0119 02:14:44.787136 1 azuredisk_v2.go:565] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="0a5c80b0-979f-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=8907695 I0119 02:14:45.393390 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I0119 02:14:45.393420 1 utils.go:79] GRPC request: {} I0119 02:14:45.393468 1 utils.go:85] GRPC response: {"ready":{"value":true}} I0119 02:15:00.873678 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:15:00.873740 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="07183975-979f-11ed-af75-002248822ac5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-bb2f8285-70ed-4775-8117-a6722bcbb2dc" "latency"=21576074999 I0119 02:15:00.873773 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="07183975-979f-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=21576183800 I0119 02:15:01.442423 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun3 by sdc under /dev/disk/azure/scsi1/ I0119 02:15:01.442469 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun3. perfProfile none accountType I0119 02:15:01.442494 1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun3 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-bb2f8285-70ed-4775-8117-a6722bcbb2dc/globalmount with mount options([]) I0119 02:15:01.442503 1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun3" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun3]) ... skipping 37 lines ... I0119 02:15:01.955833 1 mount_linux.go:183] Mounting cmd (mount) with arguments ( -o bind,remount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-bb2f8285-70ed-4775-8117-a6722bcbb2dc/globalmount /var/lib/kubelet/pods/36c09b9e-74dd-4838-bce4-8045efd3f4db/volumes/kubernetes.io~csi/pvc-bb2f8285-70ed-4775-8117-a6722bcbb2dc/mount) I0119 02:15:01.956568 1 nodeserver_v2.go:353] NodePublishVolume: mount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-bb2f8285-70ed-4775-8117-a6722bcbb2dc/globalmount at /var/lib/kubelet/pods/36c09b9e-74dd-4838-bce4-8045efd3f4db/volumes/kubernetes.io~csi/pvc-bb2f8285-70ed-4775-8117-a6722bcbb2dc/mount successfully I0119 02:15:01.956580 1 utils.go:85] GRPC response: {} I0119 02:15:04.224340 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:15:04.224357 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:15:04.224433 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:15:04.224451 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:15:04.224530 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:15:04.283836 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:15:04.965345 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities I0119 02:15:04.965365 1 utils.go:79] GRPC request: {} I0119 02:15:04.965407 1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]} I0119 02:15:04.968178 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities ... skipping 41 lines ... I0119 02:15:27.822623 1 mount_linux.go:294] Unmounting /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-bb2f8285-70ed-4775-8117-a6722bcbb2dc/globalmount W0119 02:15:27.832592 1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-bb2f8285-70ed-4775-8117-a6722bcbb2dc/globalmount" is not a mountpoint, deleting I0119 02:15:27.832651 1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-bb2f8285-70ed-4775-8117-a6722bcbb2dc/globalmount successfully I0119 02:15:27.832665 1 utils.go:85] GRPC response: {} I0119 02:15:34.224774 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:15:34.224866 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:15:34.224873 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:15:34.224907 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:15:34.224934 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:15:34.284334 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:15:44.779331 1 azuredisk_v2.go:578] "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-19T02:15:44Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="2e1feff3-979f-11ed-af75-002248822ac5" I0119 02:15:44.788899 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-29517166-vmss000000/status 200 OK in 9 milliseconds I0119 02:15:44.789194 1 azuredisk_v2.go:565] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="2e1feff3-979f-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=9898017 I0119 02:15:45.393309 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I0119 02:15:45.393336 1 utils.go:79] GRPC request: {} I0119 02:15:45.393386 1 utils.go:85] GRPC response: {"ready":{"value":true}} I0119 02:15:56.635222 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:15:56.635279 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="1479dec2-979f-11ed-af75-002248822ac5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-0d305e33-54b1-4895-b2e7-75aa9a6978e5" "latency"=54887306743 I0119 02:15:56.635309 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="1479dec2-979f-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=54887406345 I0119 02:15:57.217011 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun2 by sdd under /dev/disk/azure/scsi1/ I0119 02:15:57.217228 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun2. perfProfile none accountType I0119 02:15:57.217268 1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun2 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-0d305e33-54b1-4895-b2e7-75aa9a6978e5/globalmount with mount options([]) I0119 02:15:57.217282 1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun2" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun2]) ... skipping 56 lines ... W0119 02:16:03.790533 1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-0d305e33-54b1-4895-b2e7-75aa9a6978e5/globalmount" is not a mountpoint, deleting I0119 02:16:03.790600 1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-0d305e33-54b1-4895-b2e7-75aa9a6978e5/globalmount successfully I0119 02:16:03.790610 1 utils.go:85] GRPC response: {} I0119 02:16:04.225081 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:16:04.225128 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:16:04.225090 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:16:04.225211 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:16:04.225122 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:16:04.284838 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:16:14.780075 1 azuredisk_v2.go:578] "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-19T02:16:14Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="4001afe3-979f-11ed-af75-002248822ac5" I0119 02:16:14.789456 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-29517166-vmss000000/status 200 OK in 9 milliseconds I0119 02:16:14.791807 1 azuredisk_v2.go:565] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="4001afe3-979f-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=11779340 I0119 02:16:15.394153 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I0119 02:16:15.394174 1 utils.go:79] GRPC request: {} I0119 02:16:15.394222 1 utils.go:85] GRPC response: {"ready":{"value":true}} I0119 02:16:34.226185 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:16:34.226279 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:16:34.226349 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:16:34.226397 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:16:34.226399 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:16:34.285792 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:16:42.267829 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:16:42.267897 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="38317d52-979f-11ed-af75-002248822ac5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-be390e40-c0ed-4193-8a95-3f3f361de228" "latency"=40596292682 I0119 02:16:42.267924 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="38317d52-979f-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=40596400883 I0119 02:16:42.861801 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/ I0119 02:16:42.861849 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType I0119 02:16:42.861877 1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-be390e40-c0ed-4193-8a95-3f3f361de228/globalmount with mount options([]) I0119 02:16:42.861886 1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0]) ... skipping 59 lines ... I0119 02:16:48.121915 1 nodeserver_v2.go:257] NodeUnstageVolume: unmounting /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-be390e40-c0ed-4193-8a95-3f3f361de228/globalmount I0119 02:16:48.121954 1 mount_helper_common.go:99] "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-be390e40-c0ed-4193-8a95-3f3f361de228/globalmount" is a mountpoint, unmounting I0119 02:16:48.121970 1 mount_linux.go:294] Unmounting /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-be390e40-c0ed-4193-8a95-3f3f361de228/globalmount W0119 02:16:48.164361 1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-be390e40-c0ed-4193-8a95-3f3f361de228/globalmount" is not a mountpoint, deleting I0119 02:16:48.164454 1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-be390e40-c0ed-4193-8a95-3f3f361de228/globalmount successfully I0119 02:16:48.164476 1 utils.go:85] GRPC response: {} I0119 02:16:52.528515 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:16:52.528587 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="510103bd-979f-11ed-af75-002248822ac5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-72809247-2735-442f-a6a1-48c354c09409" "latency"=9231618482 I0119 02:16:52.528623 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="510103bd-979f-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9231742084 I0119 02:16:53.119625 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sde under /dev/disk/azure/scsi1/ I0119 02:16:53.119667 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType I0119 02:16:53.119702 1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun1 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-72809247-2735-442f-a6a1-48c354c09409/globalmount with mount options([]) I0119 02:16:53.119713 1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun1" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun1]) ... skipping 150 lines ... I0119 02:17:34.134274 1 utils.go:79] GRPC request: {} I0119 02:17:34.134301 1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]} I0119 02:17:34.135002 1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume I0119 02:17:34.135018 1 utils.go:79] GRPC request: {"publish_context":{"LUN":"2"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-3bb49a06-23fd-429a-bcd8-32bc8e9ba8c2/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-3bb49a06-23fd-429a-bcd8-32bc8e9ba8c2","csi.storage.k8s.io/pvc/name":"inline-volume-tester-57jdb-my-volume-0","csi.storage.k8s.io/pvc/namespace":"ephemeral-4718","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1674093937787-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-iei3lcrr/providers/Microsoft.Compute/disks/pvc-3bb49a06-23fd-429a-bcd8-32bc8e9ba8c2"} I0119 02:17:34.135182 1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-3bb49a06-23fd-429a-bcd8-32bc8e9ba8c2-k8s-agentpool1-29517166-vmss000000-attachment) I0119 02:17:34.227234 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:17:34.227351 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:17:34.227360 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:17:34.227379 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:17:34.227424 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:17:34.286659 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:17:44.778550 1 azuredisk_v2.go:578] "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-19T02:17:44Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="75a65d90-979f-11ed-af75-002248822ac5" I0119 02:17:44.786775 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-29517166-vmss000000/status 200 OK in 8 milliseconds I0119 02:17:44.789138 1 azuredisk_v2.go:565] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="75a65d90-979f-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=10630918 I0119 02:17:45.393723 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I0119 02:17:45.393745 1 utils.go:79] GRPC request: {} I0119 02:17:45.393792 1 utils.go:85] GRPC response: {"ready":{"value":true}} I0119 02:17:54.208715 1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: Watch close - *v1beta2.AzDriverNode total 63 items received I0119 02:17:54.212233 1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azdrivernodes?allowWatchBookmarks=true&resourceVersion=10096&timeout=5m21s&timeoutSeconds=321&watch=true 200 OK in 3 milliseconds I0119 02:18:03.711228 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:18:03.711296 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="6f4e5273-979f-11ed-af75-002248822ac5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-3bb49a06-23fd-429a-bcd8-32bc8e9ba8c2" "latency"=29576048007 I0119 02:18:03.711331 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="6f4e5273-979f-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=29576157907 I0119 02:18:04.227627 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:18:04.227709 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:18:04.227753 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:18:04.227802 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync ... skipping 50 lines ... I0119 02:18:06.107722 1 mount_linux.go:183] Mounting cmd (mount) with arguments ( -o bind /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-3bb49a06-23fd-429a-bcd8-32bc8e9ba8c2/globalmount /var/lib/kubelet/pods/9170ec5c-d4bf-4d9f-a71c-d1a9f8f8572c/volumes/kubernetes.io~csi/pvc-3bb49a06-23fd-429a-bcd8-32bc8e9ba8c2/mount) I0119 02:18:06.108545 1 mount_linux.go:183] Mounting cmd (mount) with arguments ( -o bind,remount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-3bb49a06-23fd-429a-bcd8-32bc8e9ba8c2/globalmount /var/lib/kubelet/pods/9170ec5c-d4bf-4d9f-a71c-d1a9f8f8572c/volumes/kubernetes.io~csi/pvc-3bb49a06-23fd-429a-bcd8-32bc8e9ba8c2/mount) I0119 02:18:06.109371 1 nodeserver_v2.go:353] NodePublishVolume: mount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-3bb49a06-23fd-429a-bcd8-32bc8e9ba8c2/globalmount at /var/lib/kubelet/pods/9170ec5c-d4bf-4d9f-a71c-d1a9f8f8572c/volumes/kubernetes.io~csi/pvc-3bb49a06-23fd-429a-bcd8-32bc8e9ba8c2/mount successfully I0119 02:18:06.109389 1 utils.go:85] GRPC response: {} I0119 02:18:11.209758 1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: Watch close - *v1beta2.AzVolumeAttachment total 280 items received I0119 02:18:11.213835 1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumeattachments?allowWatchBookmarks=true&resourceVersion=10250&timeout=7m55s&timeoutSeconds=475&watch=true 200 OK in 3 milliseconds I0119 02:18:13.998839 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:18:13.998907 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="818fa45e-979f-11ed-af75-002248822ac5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-706b4078-c1dc-4e1b-8a0c-a29165252163" "latency"=9236573355 I0119 02:18:13.998942 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="818fa45e-979f-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9236699957 I0119 02:18:14.002437 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:18:14.002504 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="81ad9f92-979f-11ed-af75-002248822ac5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-d2f050dd-5ba5-4c0d-ba73-6b0e46e54b8a" "latency"=9043711239 I0119 02:18:14.002543 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="81ad9f92-979f-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9043815741 I0119 02:18:14.571387 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sde under /dev/disk/azure/scsi1/ I0119 02:18:14.571440 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType I0119 02:18:14.571454 1 utils.go:85] GRPC response: {} I0119 02:18:14.571497 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdd under /dev/disk/azure/scsi1/ ... skipping 74 lines ... I0119 02:18:25.217908 1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]} I0119 02:18:25.218692 1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume I0119 02:18:25.218709 1 utils.go:79] GRPC request: {"publish_context":{"LUN":"3"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-ec6c6785-6757-4a2d-8984-2445795fa681","volume_capability":{"AccessType":{"Block":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-ec6c6785-6757-4a2d-8984-2445795fa681","csi.storage.k8s.io/pvc/name":"test.csi.azure.comg7gfw-restored","csi.storage.k8s.io/pvc/namespace":"multivolume-5379","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1674093937787-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-iei3lcrr/providers/Microsoft.Compute/disks/pvc-ec6c6785-6757-4a2d-8984-2445795fa681"} I0119 02:18:25.218937 1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-ec6c6785-6757-4a2d-8984-2445795fa681-k8s-agentpool1-29517166-vmss000000-attachment) I0119 02:18:34.229145 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:18:34.229226 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:18:34.229256 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:18:34.229283 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:18:34.229319 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:18:34.287728 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:18:39.959417 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:18:39.959495 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="8dc11441-979f-11ed-af75-002248822ac5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-ec6c6785-6757-4a2d-8984-2445795fa681" "latency"=14740476030 I0119 02:18:39.959526 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="8dc11441-979f-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=14740634231 I0119 02:18:40.551213 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun3 by sdf under /dev/disk/azure/scsi1/ I0119 02:18:40.551259 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun3. perfProfile none accountType I0119 02:18:40.551270 1 utils.go:85] GRPC response: {} I0119 02:18:40.554486 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities ... skipping 63 lines ... I0119 02:18:51.420366 1 utils.go:85] GRPC response: {} I0119 02:18:53.274641 1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: Watch close - *v1beta2.AzDriverNode total 18 items received I0119 02:18:53.275913 1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dk8s-agentpool1-29517166-vmss000000&resourceVersion=10808&timeout=9m38s&timeoutSeconds=578&watch=true 200 OK in 1 milliseconds I0119 02:19:04.231890 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:19:04.231957 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:19:04.231933 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:19:04.232027 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:19:04.232057 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:19:04.288396 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:19:14.778903 1 azuredisk_v2.go:578] "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-19T02:19:14Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="ab4b545e-979f-11ed-af75-002248822ac5" I0119 02:19:14.793448 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-29517166-vmss000000/status 200 OK in 14 milliseconds I0119 02:19:14.793606 1 azuredisk_v2.go:565] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="ab4b545e-979f-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=14736662 I0119 02:19:15.393404 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe ... skipping 6 lines ... I0119 02:19:15.428505 1 utils.go:79] GRPC request: {"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-iei3lcrr/providers/Microsoft.Compute/disks/pvc-3bb49a06-23fd-429a-bcd8-32bc8e9ba8c2","volume_path":"/var/lib/kubelet/pods/9170ec5c-d4bf-4d9f-a71c-d1a9f8f8572c/volumes/kubernetes.io~csi/pvc-3bb49a06-23fd-429a-bcd8-32bc8e9ba8c2/mount"} I0119 02:19:15.428568 1 utils.go:85] GRPC response: {"usage":[{"available":5179580416,"total":5196382208,"unit":1,"used":24576},{"available":327669,"total":327680,"unit":2,"used":11}]} I0119 02:19:34.235199 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:19:34.235254 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:19:34.235280 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:19:34.235320 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:19:34.235415 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:19:34.288534 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:19:39.211438 1 reflector.go:559] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: Watch close - *v1.CustomResourceDefinition total 0 items received I0119 02:19:39.216602 1 round_trippers.go:553] GET https://10.0.0.1:443/apis/apiextensions.k8s.io/v1/customresourcedefinitions?allowWatchBookmarks=true&resourceVersion=2543&timeout=8m11s&timeoutSeconds=491&watch=true 200 OK in 4 milliseconds I0119 02:19:44.781195 1 azuredisk_v2.go:578] "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-19T02:19:44Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="bd2d50bd-979f-11ed-af75-002248822ac5" I0119 02:19:44.788242 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-29517166-vmss000000/status 200 OK in 6 milliseconds I0119 02:19:44.788889 1 azuredisk_v2.go:565] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="bd2d50bd-979f-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=7741085 I0119 02:19:45.393304 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I0119 02:19:45.393330 1 utils.go:79] GRPC request: {} I0119 02:19:45.393377 1 utils.go:85] GRPC response: {"ready":{"value":true}} I0119 02:20:04.235509 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:20:04.235593 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:20:04.235631 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:20:04.235670 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:20:04.235726 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:20:04.288982 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:20:14.779949 1 azuredisk_v2.go:578] "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-19T02:20:14Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="cf0ec34c-979f-11ed-af75-002248822ac5" I0119 02:20:14.789504 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-29517166-vmss000000/status 200 OK in 9 milliseconds I0119 02:20:14.789691 1 azuredisk_v2.go:565] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="cf0ec34c-979f-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=9772507 I0119 02:20:15.394379 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I0119 02:20:15.394406 1 utils.go:79] GRPC request: {} I0119 02:20:15.394451 1 utils.go:85] GRPC response: {"ready":{"value":true}} I0119 02:20:34.236206 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:20:34.236250 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:20:34.236315 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:20:34.236368 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:20:34.236480 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:20:34.289624 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync E0119 02:20:40.651235 1 conditionwaiter.go:50] "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [context canceled]" "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="96f484ff-979f-11ed-af75-002248822ac5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-0b3ffbf0-e6e8-4166-ace3-1410c7124696" "latency"=119995535991 E0119 02:20:40.651303 1 crdprovisioner.go:743] "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [context canceled]" "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="96f484ff-979f-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=119995785694 E0119 02:20:40.651324 1 utils.go:83] GRPC error: rpc error: code = Internal desc = failed to wait for volume (/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-iei3lcrr/providers/Microsoft.Compute/disks/pvc-0b3ffbf0-e6e8-4166-ace3-1410c7124696) to be attached to node (k8s-agentpool1-29517166-vmss000000): context canceled I0119 02:20:41.232207 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities I0119 02:20:41.232229 1 utils.go:79] GRPC request: {} I0119 02:20:41.232270 1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]} I0119 02:20:41.235167 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities I0119 02:20:41.235188 1 utils.go:79] GRPC request: {} I0119 02:20:41.235223 1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]} ... skipping 6 lines ... I0119 02:20:44.778294 1 azuredisk_v2.go:578] "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-19T02:20:44Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="e0f02546-979f-11ed-af75-002248822ac5" I0119 02:20:44.794523 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-29517166-vmss000000/status 200 OK in 16 milliseconds I0119 02:20:44.795169 1 azuredisk_v2.go:565] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="e0f02546-979f-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=16927185 I0119 02:20:45.393199 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I0119 02:20:45.393220 1 utils.go:79] GRPC request: {} I0119 02:20:45.393270 1 utils.go:85] GRPC response: {"ready":{"value":true}} I0119 02:20:51.522508 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:20:51.522574 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="ded3c4b9-979f-11ed-af75-002248822ac5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-0b3ffbf0-e6e8-4166-ace3-1410c7124696" "latency"=10285682486 I0119 02:20:51.522607 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="ded3c4b9-979f-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=10285785688 I0119 02:20:52.131850 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun4 by sdd under /dev/disk/azure/scsi1/ I0119 02:20:52.131905 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun4. perfProfile none accountType I0119 02:20:52.131942 1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun4 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-0b3ffbf0-e6e8-4166-ace3-1410c7124696/globalmount with mount options([]) I0119 02:20:52.131958 1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun4" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun4]) ... skipping 74 lines ... I0119 02:21:03.012139 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetVolumeStats I0119 02:21:03.012154 1 utils.go:79] GRPC request: {"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-iei3lcrr/providers/Microsoft.Compute/disks/pvc-3bb49a06-23fd-429a-bcd8-32bc8e9ba8c2","volume_path":"/var/lib/kubelet/pods/9170ec5c-d4bf-4d9f-a71c-d1a9f8f8572c/volumes/kubernetes.io~csi/pvc-3bb49a06-23fd-429a-bcd8-32bc8e9ba8c2/mount"} I0119 02:21:03.012253 1 utils.go:85] GRPC response: {"usage":[{"available":5179580416,"total":5196382208,"unit":1,"used":24576},{"available":327668,"total":327680,"unit":2,"used":12}]} I0119 02:21:04.236413 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:21:04.236474 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:21:04.236533 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:21:04.236592 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:21:04.236538 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:21:04.236619 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:21:04.289796 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:21:14.780099 1 azuredisk_v2.go:578] "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-19T02:21:14Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="f2d20f43-979f-11ed-af75-002248822ac5" I0119 02:21:14.793482 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-29517166-vmss000000/status 200 OK in 12 milliseconds I0119 02:21:14.793653 1 azuredisk_v2.go:565] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="f2d20f43-979f-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=13591949 I0119 02:21:15.393577 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I0119 02:21:15.393600 1 utils.go:79] GRPC request: {} I0119 02:21:15.393649 1 utils.go:85] GRPC response: {"ready":{"value":true}} I0119 02:21:34.237539 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:21:34.237636 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:21:34.237686 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:21:34.237712 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:21:34.237721 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:21:34.237754 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:21:34.290127 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:21:35.005241 1 utils.go:78] GRPC call: /csi.v1.Node/NodeUnpublishVolume I0119 02:21:35.005261 1 utils.go:79] GRPC request: {"target_path":"/var/lib/kubelet/pods/9170ec5c-d4bf-4d9f-a71c-d1a9f8f8572c/volumes/kubernetes.io~csi/pvc-3bb49a06-23fd-429a-bcd8-32bc8e9ba8c2/mount","volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-iei3lcrr/providers/Microsoft.Compute/disks/pvc-3bb49a06-23fd-429a-bcd8-32bc8e9ba8c2"} I0119 02:21:35.005300 1 nodeserver_v2.go:369] NodeUnpublishVolume: unmounting volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-iei3lcrr/providers/Microsoft.Compute/disks/pvc-3bb49a06-23fd-429a-bcd8-32bc8e9ba8c2 on /var/lib/kubelet/pods/9170ec5c-d4bf-4d9f-a71c-d1a9f8f8572c/volumes/kubernetes.io~csi/pvc-3bb49a06-23fd-429a-bcd8-32bc8e9ba8c2/mount I0119 02:21:35.005333 1 mount_helper_common.go:99] "/var/lib/kubelet/pods/9170ec5c-d4bf-4d9f-a71c-d1a9f8f8572c/volumes/kubernetes.io~csi/pvc-3bb49a06-23fd-429a-bcd8-32bc8e9ba8c2/mount" is a mountpoint, unmounting I0119 02:21:35.005351 1 mount_linux.go:294] Unmounting /var/lib/kubelet/pods/9170ec5c-d4bf-4d9f-a71c-d1a9f8f8572c/volumes/kubernetes.io~csi/pvc-3bb49a06-23fd-429a-bcd8-32bc8e9ba8c2/mount ... skipping 14 lines ... I0119 02:21:44.781243 1 azuredisk_v2.go:578] "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-19T02:21:44Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="04b3dead-97a0-11ed-af75-002248822ac5" I0119 02:21:44.788437 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-29517166-vmss000000/status 200 OK in 6 milliseconds I0119 02:21:44.788614 1 azuredisk_v2.go:565] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="04b3dead-97a0-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=7415182 I0119 02:21:45.393670 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I0119 02:21:45.393694 1 utils.go:79] GRPC request: {} I0119 02:21:45.393742 1 utils.go:85] GRPC response: {"ready":{"value":true}} I0119 02:21:52.120256 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:21:52.120326 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="e5e9667c-979f-11ed-af75-002248822ac5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-4d3e1a2c-d8ba-4c91-8f8e-7cbe8951e2dd" "latency"=58997602102 I0119 02:21:52.120354 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="e5e9667c-979f-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=58997716904 I0119 02:21:52.120504 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:21:52.120555 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="e55fea6e-979f-11ed-af75-002248822ac5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-7d8237a4-f85f-40c3-a328-ae27db8d8607" "latency"=59898849494 I0119 02:21:52.120583 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="e55fea6e-979f-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=59898965895 I0119 02:21:52.533283 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities I0119 02:21:52.533309 1 utils.go:79] GRPC request: {} I0119 02:21:52.533354 1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]} I0119 02:21:52.536508 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities ... skipping 68 lines ... I0119 02:21:52.789714 1 nodeserver_v2.go:353] NodePublishVolume: mount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-4d3e1a2c-d8ba-4c91-8f8e-7cbe8951e2dd/globalmount at /var/lib/kubelet/pods/35f350a4-5677-411a-bf88-a1df1b284293/volumes/kubernetes.io~csi/pvc-4d3e1a2c-d8ba-4c91-8f8e-7cbe8951e2dd/mount successfully I0119 02:21:52.789731 1 utils.go:85] GRPC response: {} I0119 02:22:04.241519 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:22:04.241571 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:22:04.241553 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:22:04.241652 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:22:04.241672 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:22:04.291029 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:22:08.208694 1 reflector.go:559] k8s.io/client-go/informers/factory.go:150: Watch close - *v1.Node total 144 items received I0119 02:22:08.211376 1 round_trippers.go:553] GET https://10.0.0.1:443/api/v1/nodes?allowWatchBookmarks=true&resourceVersion=12253&timeout=8m0s&timeoutSeconds=480&watch=true 200 OK in 2 milliseconds I0119 02:22:10.248926 1 utils.go:78] GRPC call: /csi.v1.Node/NodeUnpublishVolume I0119 02:22:10.248965 1 utils.go:79] GRPC request: {"target_path":"/var/lib/kubelet/pods/35f350a4-5677-411a-bf88-a1df1b284293/volumes/kubernetes.io~csi/pvc-4d3e1a2c-d8ba-4c91-8f8e-7cbe8951e2dd/mount","volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-iei3lcrr/providers/Microsoft.Compute/disks/pvc-4d3e1a2c-d8ba-4c91-8f8e-7cbe8951e2dd"} I0119 02:22:10.248931 1 utils.go:78] GRPC call: /csi.v1.Node/NodeUnpublishVolume ... skipping 39 lines ... I0119 02:22:15.393457 1 utils.go:79] GRPC request: {} I0119 02:22:15.393506 1 utils.go:85] GRPC response: {"ready":{"value":true}} I0119 02:22:34.242202 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:22:34.242229 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:22:34.242257 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:22:34.242210 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:22:34.242345 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:22:34.291564 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:22:44.782338 1 azuredisk_v2.go:578] "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-19T02:22:44Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="28775008-97a0-11ed-af75-002248822ac5" I0119 02:22:44.794115 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-29517166-vmss000000/status 200 OK in 11 milliseconds I0119 02:22:44.794641 1 azuredisk_v2.go:565] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="28775008-97a0-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=12330325 I0119 02:22:45.393745 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I0119 02:22:45.393767 1 utils.go:79] GRPC request: {} I0119 02:22:45.393814 1 utils.go:85] GRPC response: {"ready":{"value":true}} I0119 02:23:04.243185 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:23:04.243233 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:23:04.243275 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:23:04.243325 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:23:04.243381 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:23:04.292675 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:23:14.778296 1 azuredisk_v2.go:578] "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-19T02:23:14Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="3a5854e2-97a0-11ed-af75-002248822ac5" I0119 02:23:14.786927 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-29517166-vmss000000/status 200 OK in 8 milliseconds I0119 02:23:14.788067 1 azuredisk_v2.go:565] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="3a5854e2-97a0-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=9804524 I0119 02:23:15.212902 1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: Watch close - *v1beta2.AzDriverNode total 51 items received I0119 02:23:15.221192 1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azdrivernodes?allowWatchBookmarks=true&resourceVersion=12635&timeout=7m53s&timeoutSeconds=473&watch=true 200 OK in 8 milliseconds I0119 02:23:15.392920 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I0119 02:23:15.392946 1 utils.go:79] GRPC request: {} I0119 02:23:15.392990 1 utils.go:85] GRPC response: {"ready":{"value":true}} I0119 02:23:34.243861 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:23:34.243897 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:23:34.243875 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:23:34.243884 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:23:34.244054 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:23:34.293156 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:23:44.778344 1 azuredisk_v2.go:578] "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-19T02:23:44Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="4c39fa1c-97a0-11ed-af75-002248822ac5" I0119 02:23:44.794309 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-29517166-vmss000000/status 200 OK in 15 milliseconds I0119 02:23:44.794894 1 azuredisk_v2.go:565] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="4c39fa1c-97a0-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=16576774 I0119 02:23:45.392860 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I0119 02:23:45.392881 1 utils.go:79] GRPC request: {} I0119 02:23:45.392923 1 utils.go:85] GRPC response: {"ready":{"value":true}} E0119 02:23:52.533453 1 conditionwaiter.go:50] "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [context canceled]" "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="095381e8-97a0-11ed-af75-002248822ac5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-ce37279a-4421-4908-8238-a57c45cd9cc2" "latency"=119995049754 E0119 02:23:52.533503 1 crdprovisioner.go:743] "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [context canceled]" "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="095381e8-97a0-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=119995222757 E0119 02:23:52.533525 1 utils.go:83] GRPC error: rpc error: code = Internal desc = failed to wait for volume (/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-iei3lcrr/providers/Microsoft.Compute/disks/pvc-ce37279a-4421-4908-8238-a57c45cd9cc2) to be attached to node (k8s-agentpool1-29517166-vmss000000): context canceled I0119 02:23:53.106248 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities I0119 02:23:53.106272 1 utils.go:79] GRPC request: {} I0119 02:23:53.106307 1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]} I0119 02:23:53.109919 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities I0119 02:23:53.109934 1 utils.go:79] GRPC request: {} I0119 02:23:53.109964 1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]} ... skipping 2 lines ... I0119 02:23:53.110685 1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]} I0119 02:23:53.111434 1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume I0119 02:23:53.111451 1 utils.go:79] GRPC request: {"publish_context":{"LUN":"5"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-ce37279a-4421-4908-8238-a57c45cd9cc2/globalmount","volume_capability":{"AccessType":{"Mount":{"fs_type":"xfs"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-ce37279a-4421-4908-8238-a57c45cd9cc2","csi.storage.k8s.io/pvc/name":"test.csi.azure.com2kr8t","csi.storage.k8s.io/pvc/namespace":"multivolume-8573","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1674093937787-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-iei3lcrr/providers/Microsoft.Compute/disks/pvc-ce37279a-4421-4908-8238-a57c45cd9cc2"} I0119 02:23:53.111606 1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-ce37279a-4421-4908-8238-a57c45cd9cc2-k8s-agentpool1-29517166-vmss000000-attachment) I0119 02:23:54.211313 1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: Watch close - *v1beta2.AzVolume total 199 items received I0119 02:23:54.213633 1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumes?allowWatchBookmarks=true&resourceVersion=12819&timeout=9m59s&timeoutSeconds=599&watch=true 200 OK in 2 milliseconds I0119 02:24:03.635674 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:24:03.635741 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="5131890e-97a0-11ed-af75-002248822ac5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-ce37279a-4421-4908-8238-a57c45cd9cc2" "latency"=10524078238 I0119 02:24:03.635762 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="5131890e-97a0-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=10524173240 I0119 02:24:04.212139 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun5 by sdd under /dev/disk/azure/scsi1/ I0119 02:24:04.212194 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun5. perfProfile none accountType I0119 02:24:04.212224 1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun5 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-ce37279a-4421-4908-8238-a57c45cd9cc2/globalmount with mount options([nouuid]) I0119 02:24:04.212235 1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun5" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun5]) ... skipping 122 lines ... I0119 02:24:16.979142 1 utils.go:79] GRPC request: {"publish_context":{"LUN":"5"},"readonly":true,"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-ce37279a-4421-4908-8238-a57c45cd9cc2/globalmount","target_path":"/var/lib/kubelet/pods/8cd426cb-7307-4116-9724-d2a9f9e30a82/volumes/kubernetes.io~csi/pvc-ce37279a-4421-4908-8238-a57c45cd9cc2/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"xfs"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-ce37279a-4421-4908-8238-a57c45cd9cc2","csi.storage.k8s.io/pvc/name":"test.csi.azure.com2kr8t","csi.storage.k8s.io/pvc/namespace":"multivolume-8573","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1674093937787-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-iei3lcrr/providers/Microsoft.Compute/disks/pvc-ce37279a-4421-4908-8238-a57c45cd9cc2"} I0119 02:24:16.979314 1 nodeserver_v2.go:348] NodePublishVolume: mounting /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-ce37279a-4421-4908-8238-a57c45cd9cc2/globalmount at /var/lib/kubelet/pods/8cd426cb-7307-4116-9724-d2a9f9e30a82/volumes/kubernetes.io~csi/pvc-ce37279a-4421-4908-8238-a57c45cd9cc2/mount I0119 02:24:16.979336 1 mount_linux.go:183] Mounting cmd (mount) with arguments ( -o bind /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-ce37279a-4421-4908-8238-a57c45cd9cc2/globalmount /var/lib/kubelet/pods/8cd426cb-7307-4116-9724-d2a9f9e30a82/volumes/kubernetes.io~csi/pvc-ce37279a-4421-4908-8238-a57c45cd9cc2/mount) I0119 02:24:16.980262 1 mount_linux.go:183] Mounting cmd (mount) with arguments ( -o bind,remount,ro /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-ce37279a-4421-4908-8238-a57c45cd9cc2/globalmount /var/lib/kubelet/pods/8cd426cb-7307-4116-9724-d2a9f9e30a82/volumes/kubernetes.io~csi/pvc-ce37279a-4421-4908-8238-a57c45cd9cc2/mount) I0119 02:24:16.981165 1 nodeserver_v2.go:353] NodePublishVolume: mount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-ce37279a-4421-4908-8238-a57c45cd9cc2/globalmount at /var/lib/kubelet/pods/8cd426cb-7307-4116-9724-d2a9f9e30a82/volumes/kubernetes.io~csi/pvc-ce37279a-4421-4908-8238-a57c45cd9cc2/mount successfully I0119 02:24:16.981183 1 utils.go:85] GRPC response: {} I0119 02:24:18.874761 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:24:18.874822 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="58c22fd9-97a0-11ed-af75-002248822ac5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-f30c1e94-f9c0-4e74-b991-a4a797f63e4d" "latency"=13071110497 I0119 02:24:18.874849 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="58c22fd9-97a0-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=13071216400 I0119 02:24:19.446558 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sde under /dev/disk/azure/scsi1/ I0119 02:24:19.446622 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType I0119 02:24:19.446650 1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-f30c1e94-f9c0-4e74-b991-a4a797f63e4d/globalmount with mount options([]) I0119 02:24:19.446660 1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0]) ... skipping 80 lines ... I0119 02:24:33.089668 1 utils.go:79] GRPC request: {"publish_context":{"LUN":"1"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-76be1297-f626-49b5-a3c8-64b0365cb2d7","volume_capability":{"AccessType":{"Block":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-76be1297-f626-49b5-a3c8-64b0365cb2d7","csi.storage.k8s.io/pvc/name":"inline-volume-tester2-vlz7b-my-volume-0","csi.storage.k8s.io/pvc/namespace":"ephemeral-3530","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1674093937787-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-iei3lcrr/providers/Microsoft.Compute/disks/pvc-76be1297-f626-49b5-a3c8-64b0365cb2d7"} I0119 02:24:33.089865 1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-76be1297-f626-49b5-a3c8-64b0365cb2d7-k8s-agentpool1-29517166-vmss000000-attachment) I0119 02:24:34.244994 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:24:34.245092 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:24:34.245099 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:24:34.245123 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:24:34.245188 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:24:34.294421 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:24:44.779577 1 azuredisk_v2.go:578] "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-19T02:24:44Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="6ffd6fbf-97a0-11ed-af75-002248822ac5" I0119 02:24:44.790203 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-29517166-vmss000000/status 200 OK in 10 milliseconds I0119 02:24:44.791026 1 azuredisk_v2.go:565] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="6ffd6fbf-97a0-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=11491840 I0119 02:24:45.393715 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I0119 02:24:45.393742 1 utils.go:79] GRPC request: {} I0119 02:24:45.393787 1 utils.go:85] GRPC response: {"ready":{"value":true}} I0119 02:25:00.350992 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:25:00.351045 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="6905bbaf-97a0-11ed-af75-002248822ac5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-76be1297-f626-49b5-a3c8-64b0365cb2d7" "latency"=27261126770 I0119 02:25:00.351077 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="6905bbaf-97a0-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=27261223672 I0119 02:25:00.627250 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities I0119 02:25:00.627292 1 utils.go:79] GRPC request: {} I0119 02:25:00.627554 1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]} I0119 02:25:00.633473 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities ... skipping 22 lines ... I0119 02:25:04.066984 1 nodeserver_v2.go:353] NodePublishVolume: mount /dev/disk/azure/scsi1/lun1 at /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-76be1297-f626-49b5-a3c8-64b0365cb2d7/5c3a78a7-b00a-4425-a8d8-b4e2e5ead23d successfully I0119 02:25:04.067002 1 utils.go:85] GRPC response: {} I0119 02:25:04.245795 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:25:04.245824 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:25:04.245854 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:25:04.245807 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:25:04.245983 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:25:04.295069 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:25:10.719820 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:25:10.719868 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="7970cbf3-97a0-11ed-af75-002248822ac5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-6e99af13-5dce-459f-b34d-d74d4e41ed2f" "latency"=10084750382 I0119 02:25:10.719888 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="7970cbf3-97a0-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=10084845784 I0119 02:25:11.303898 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdd under /dev/disk/azure/scsi1/ I0119 02:25:11.303947 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType I0119 02:25:11.303975 1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6e99af13-5dce-459f-b34d-d74d4e41ed2f/globalmount with mount options([]) I0119 02:25:11.303984 1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0]) ... skipping 61 lines ... I0119 02:25:21.067525 1 mount_linux.go:294] Unmounting /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6e99af13-5dce-459f-b34d-d74d4e41ed2f/globalmount W0119 02:25:21.077830 1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6e99af13-5dce-459f-b34d-d74d4e41ed2f/globalmount" is not a mountpoint, deleting I0119 02:25:21.077913 1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6e99af13-5dce-459f-b34d-d74d4e41ed2f/globalmount successfully I0119 02:25:21.077928 1 utils.go:85] GRPC response: {} I0119 02:25:34.246983 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:25:34.247079 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:25:34.247124 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:25:34.247143 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:25:34.247165 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:25:34.295568 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:25:38.320148 1 utils.go:78] GRPC call: /csi.v1.Node/NodeUnpublishVolume I0119 02:25:38.320170 1 utils.go:79] GRPC request: {"target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-76be1297-f626-49b5-a3c8-64b0365cb2d7/5c3a78a7-b00a-4425-a8d8-b4e2e5ead23d","volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-iei3lcrr/providers/Microsoft.Compute/disks/pvc-76be1297-f626-49b5-a3c8-64b0365cb2d7"} I0119 02:25:38.320234 1 nodeserver_v2.go:369] NodeUnpublishVolume: unmounting volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-iei3lcrr/providers/Microsoft.Compute/disks/pvc-76be1297-f626-49b5-a3c8-64b0365cb2d7 on /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-76be1297-f626-49b5-a3c8-64b0365cb2d7/5c3a78a7-b00a-4425-a8d8-b4e2e5ead23d ... skipping 8 lines ... I0119 02:25:38.393728 1 utils.go:78] GRPC call: /csi.v1.Node/NodeUnstageVolume I0119 02:25:38.393756 1 utils.go:79] GRPC request: {"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-76be1297-f626-49b5-a3c8-64b0365cb2d7","volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-iei3lcrr/providers/Microsoft.Compute/disks/pvc-76be1297-f626-49b5-a3c8-64b0365cb2d7"} I0119 02:25:38.393805 1 nodeserver_v2.go:257] NodeUnstageVolume: unmounting /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-76be1297-f626-49b5-a3c8-64b0365cb2d7 W0119 02:25:38.394711 1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-76be1297-f626-49b5-a3c8-64b0365cb2d7" is not a mountpoint, deleting I0119 02:25:38.394782 1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-76be1297-f626-49b5-a3c8-64b0365cb2d7 successfully I0119 02:25:38.394800 1 utils.go:85] GRPC response: {} I0119 02:25:39.277576 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:25:39.277644 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="817b3b43-97a0-11ed-af75-002248822ac5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-78b85abd-0251-4a7c-a888-927ee9fe65c4" "latency"=25152360070 I0119 02:25:39.277673 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="817b3b43-97a0-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=25152466472 I0119 02:25:39.866121 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun2 by sde under /dev/disk/azure/scsi1/ I0119 02:25:39.866172 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun2. perfProfile none accountType I0119 02:25:39.866210 1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun2 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-78b85abd-0251-4a7c-a888-927ee9fe65c4/globalmount with mount options([]) I0119 02:25:39.866230 1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun2" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun2]) ... skipping 62 lines ... I0119 02:25:45.392940 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I0119 02:25:45.392963 1 utils.go:79] GRPC request: {} I0119 02:25:45.393009 1 utils.go:85] GRPC response: {"ready":{"value":true}} I0119 02:26:04.248050 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:26:04.248100 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:26:04.248155 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:26:04.248213 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:26:04.248163 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:26:04.296592 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:26:06.223213 1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: Watch close - *v1beta2.AzVolumeAttachment total 187 items received I0119 02:26:06.226046 1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumeattachments?allowWatchBookmarks=true&resourceVersion=14205&timeout=5m56s&timeoutSeconds=356&watch=true 200 OK in 2 milliseconds I0119 02:26:14.778298 1 azuredisk_v2.go:578] "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-19T02:26:14Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="a5a2274f-97a0-11ed-af75-002248822ac5" I0119 02:26:14.798495 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-29517166-vmss000000/status 200 OK in 20 milliseconds ... skipping 2 lines ... I0119 02:26:15.393723 1 utils.go:79] GRPC request: {} I0119 02:26:15.393766 1 utils.go:85] GRPC response: {"ready":{"value":true}} I0119 02:26:34.249302 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:26:34.249371 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:26:34.249391 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:26:34.249448 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:26:34.249490 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:26:34.296707 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:26:44.778891 1 azuredisk_v2.go:578] "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-19T02:26:44Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="b783e02f-97a0-11ed-af75-002248822ac5" I0119 02:26:44.788833 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-29517166-vmss000000/status 200 OK in 9 milliseconds I0119 02:26:44.789038 1 azuredisk_v2.go:565] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="b783e02f-97a0-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=10206370 I0119 02:26:45.393165 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I0119 02:26:45.393187 1 utils.go:79] GRPC request: {} I0119 02:26:45.393234 1 utils.go:85] GRPC response: {"ready":{"value":true}} I0119 02:27:04.250187 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:27:04.250224 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:27:04.250286 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:27:04.250315 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:27:04.250411 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:27:04.297746 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:27:14.778552 1 azuredisk_v2.go:578] "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-19T02:27:14Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="c96576fa-97a0-11ed-af75-002248822ac5" I0119 02:27:14.788756 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-29517166-vmss000000/status 200 OK in 10 milliseconds I0119 02:27:14.790035 1 azuredisk_v2.go:565] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="c96576fa-97a0-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=11528786 I0119 02:27:15.393241 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I0119 02:27:15.393270 1 utils.go:79] GRPC request: {} I0119 02:27:15.393316 1 utils.go:85] GRPC response: {"ready":{"value":true}} I0119 02:27:34.253263 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:27:34.253347 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:27:34.253373 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:27:34.253422 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:27:34.253461 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:27:34.298810 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync E0119 02:27:44.353253 1 conditionwaiter.go:50] "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [context deadline exceeded]" "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="93837727-97a0-11ed-af75-002248822ac5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-b60a43d5-32ea-4dc5-94e9-b57828b77226" "latency"=119974940525 E0119 02:27:44.353311 1 crdprovisioner.go:743] "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [context deadline exceeded]" "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="93837727-97a0-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=119975143828 E0119 02:27:44.353326 1 utils.go:83] GRPC error: rpc error: code = Internal desc = failed to wait for volume (/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-iei3lcrr/providers/Microsoft.Compute/disks/pvc-b60a43d5-32ea-4dc5-94e9-b57828b77226) to be attached to node (k8s-agentpool1-29517166-vmss000000): context deadline exceeded I0119 02:27:44.779007 1 azuredisk_v2.go:578] "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-19T02:27:44Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="db472b8e-97a0-11ed-af75-002248822ac5" I0119 02:27:44.791638 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-29517166-vmss000000/status 200 OK in 12 milliseconds I0119 02:27:44.791790 1 azuredisk_v2.go:565] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="db472b8e-97a0-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=12826061 I0119 02:27:44.908278 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities I0119 02:27:44.908302 1 utils.go:79] GRPC request: {} I0119 02:27:44.908341 1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]} ... skipping 12 lines ... I0119 02:27:50.219960 1 reflector.go:559] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: Watch close - *v1.CustomResourceDefinition total 8 items received I0119 02:27:50.223065 1 round_trippers.go:553] GET https://10.0.0.1:443/apis/apiextensions.k8s.io/v1/customresourcedefinitions?allowWatchBookmarks=true&resourceVersion=11762&timeout=8m4s&timeoutSeconds=484&watch=true 200 OK in 2 milliseconds I0119 02:28:04.255576 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:28:04.255625 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:28:04.255626 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:28:04.255650 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:28:04.255781 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:28:04.298910 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:28:14.783248 1 azuredisk_v2.go:578] "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-19T02:28:14Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="ed2972c0-97a0-11ed-af75-002248822ac5" I0119 02:28:14.788427 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-29517166-vmss000000/status 200 OK in 4 milliseconds I0119 02:28:14.789093 1 azuredisk_v2.go:565] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="ed2972c0-97a0-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=5898902 I0119 02:28:15.392958 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I0119 02:28:15.392982 1 utils.go:79] GRPC request: {} I0119 02:28:15.393021 1 utils.go:85] GRPC response: {"ready":{"value":true}} I0119 02:28:25.988911 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:28:25.988979 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="db5ba545-97a0-11ed-af75-002248822ac5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-b60a43d5-32ea-4dc5-94e9-b57828b77226" "latency"=41075757064 I0119 02:28:25.989015 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="db5ba545-97a0-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=41075864465 I0119 02:28:26.818714 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities I0119 02:28:26.818739 1 utils.go:79] GRPC request: {} I0119 02:28:26.818786 1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]} I0119 02:28:26.821914 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities ... skipping 81 lines ... I0119 02:28:30.723217 1 utils.go:85] GRPC response: {"usage":[{"available":5179580416,"total":5196382208,"unit":1,"used":24576},{"available":327669,"total":327680,"unit":2,"used":11}]} I0119 02:28:31.279293 1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: Watch close - *v1beta2.AzDriverNode total 30 items received I0119 02:28:31.282318 1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dk8s-agentpool1-29517166-vmss000000&resourceVersion=15175&timeout=7m51s&timeoutSeconds=471&watch=true 200 OK in 2 milliseconds I0119 02:28:34.255907 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:28:34.255965 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:28:34.255998 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:28:34.256008 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:28:34.256039 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:28:34.256051 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:28:34.256063 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:28:34.256083 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:28:34.299304 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:28:36.786823 1 utils.go:78] GRPC call: /csi.v1.Node/NodeUnpublishVolume I0119 02:28:36.786852 1 utils.go:79] GRPC request: {"target_path":"/var/lib/kubelet/pods/0df8e692-072b-4d8b-ae67-6d79a331f6ac/volumes/kubernetes.io~csi/pvc-b60a43d5-32ea-4dc5-94e9-b57828b77226/mount","volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-iei3lcrr/providers/Microsoft.Compute/disks/pvc-b60a43d5-32ea-4dc5-94e9-b57828b77226"} I0119 02:28:36.786896 1 nodeserver_v2.go:369] NodeUnpublishVolume: unmounting volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-iei3lcrr/providers/Microsoft.Compute/disks/pvc-b60a43d5-32ea-4dc5-94e9-b57828b77226 on /var/lib/kubelet/pods/0df8e692-072b-4d8b-ae67-6d79a331f6ac/volumes/kubernetes.io~csi/pvc-b60a43d5-32ea-4dc5-94e9-b57828b77226/mount I0119 02:28:36.786925 1 mount_helper_common.go:99] "/var/lib/kubelet/pods/0df8e692-072b-4d8b-ae67-6d79a331f6ac/volumes/kubernetes.io~csi/pvc-b60a43d5-32ea-4dc5-94e9-b57828b77226/mount" is a mountpoint, unmounting ... skipping 19 lines ... I0119 02:28:45.393381 1 utils.go:79] GRPC request: {} I0119 02:28:45.393431 1 utils.go:85] GRPC response: {"ready":{"value":true}} I0119 02:29:04.256809 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:29:04.256853 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:29:04.256900 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:29:04.256953 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:29:04.257025 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:29:04.257078 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:29:04.257111 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:29:04.257136 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:29:04.300312 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:29:14.778887 1 azuredisk_v2.go:578] "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-19T02:29:14Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="10ec100a-97a1-11ed-af75-002248822ac5" I0119 02:29:14.784826 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-29517166-vmss000000/status 200 OK in 5 milliseconds I0119 02:29:14.785113 1 azuredisk_v2.go:565] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="10ec100a-97a1-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=6215585 I0119 02:29:15.393533 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I0119 02:29:15.393557 1 utils.go:79] GRPC request: {} I0119 02:29:15.393606 1 utils.go:85] GRPC response: {"ready":{"value":true}} I0119 02:29:26.704122 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:29:26.704171 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="f456ab4f-97a0-11ed-af75-002248822ac5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-f7f1e685-5e57-4dc7-9d6e-c0440a3db932" "latency"=59880515688 I0119 02:29:26.704190 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="f456ab4f-97a0-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=59880620090 I0119 02:29:26.708474 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:29:26.708531 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="f494349e-97a0-11ed-af75-002248822ac5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-ead47477-2e82-42be-b42b-1d3a21778bbe" "latency"=59481612615 I0119 02:29:26.708550 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="f494349e-97a0-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=59481696716 I0119 02:29:26.711933 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:29:26.711994 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="f4a383e6-97a0-11ed-af75-002248822ac5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-d6d3cfdf-7507-4255-a78f-9a42be313618" "latency"=59384729662 I0119 02:29:26.712032 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="f4a383e6-97a0-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=59384837863 I0119 02:29:26.713149 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:29:26.713207 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="f4ffe7e0-97a0-11ed-af75-002248822ac5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-eba1dce7-2545-4988-810a-1b3ba0f29f90" "latency"=58780450617 I0119 02:29:26.713237 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="f4ffe7e0-97a0-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=58780552718 I0119 02:29:27.628764 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun2 by sdg under /dev/disk/azure/scsi1/ I0119 02:29:27.628829 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun2. perfProfile none accountType I0119 02:29:27.628834 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdf under /dev/disk/azure/scsi1/ I0119 02:29:27.628855 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun4 by sde under /dev/disk/azure/scsi1/ ... skipping 238 lines ... I0119 02:29:48.935124 1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume I0119 02:29:48.935192 1 utils.go:79] GRPC request: {"publish_context":{"LUN":"5"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-d50823a8-8f4a-4a25-bf58-4b7def955165","volume_capability":{"AccessType":{"Block":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-d50823a8-8f4a-4a25-bf58-4b7def955165","csi.storage.k8s.io/pvc/name":"test.csi.azure.comt45nd","csi.storage.k8s.io/pvc/namespace":"multivolume-4004","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1674093937787-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-iei3lcrr/providers/Microsoft.Compute/disks/pvc-d50823a8-8f4a-4a25-bf58-4b7def955165"} I0119 02:29:48.935475 1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-d50823a8-8f4a-4a25-bf58-4b7def955165-k8s-agentpool1-29517166-vmss000000-attachment) I0119 02:30:04.262311 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:30:04.262389 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:30:04.262454 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:30:04.262459 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:30:04.262497 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:30:04.301775 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:30:08.216419 1 reflector.go:559] k8s.io/client-go/informers/factory.go:150: Watch close - *v1.Node total 85 items received I0119 02:30:08.222314 1 round_trippers.go:553] GET https://10.0.0.1:443/api/v1/nodes?allowWatchBookmarks=true&resourceVersion=15918&timeout=9m27s&timeoutSeconds=567&watch=true 200 OK in 5 milliseconds I0119 02:30:14.778347 1 azuredisk_v2.go:578] "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-19T02:30:14Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="34af4107-97a1-11ed-af75-002248822ac5" I0119 02:30:14.790257 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-29517166-vmss000000/status 200 OK in 11 milliseconds I0119 02:30:14.791521 1 azuredisk_v2.go:565] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="34af4107-97a1-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=13205373 I0119 02:30:15.393528 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I0119 02:30:15.393557 1 utils.go:79] GRPC request: {} I0119 02:30:15.393607 1 utils.go:85] GRPC response: {"ready":{"value":true}} I0119 02:30:34.262819 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:30:34.262893 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:30:34.262907 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:30:34.262995 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:30:34.263002 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:30:34.302236 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:30:44.784353 1 azuredisk_v2.go:578] "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-19T02:30:44Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="4691cec6-97a1-11ed-af75-002248822ac5" I0119 02:30:44.797411 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-29517166-vmss000000/status 200 OK in 12 milliseconds I0119 02:30:44.798559 1 azuredisk_v2.go:565] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="4691cec6-97a1-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=14226597 I0119 02:30:45.394079 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I0119 02:30:45.394103 1 utils.go:79] GRPC request: {} I0119 02:30:45.394143 1 utils.go:85] GRPC response: {"ready":{"value":true}} I0119 02:31:04.269312 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:31:04.269344 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:31:04.269321 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:31:04.269336 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:31:04.269436 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:31:04.302684 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:31:08.223245 1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: Watch close - *v1beta2.AzDriverNode total 72 items received I0119 02:31:08.226847 1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azdrivernodes?allowWatchBookmarks=true&resourceVersion=16312&timeout=7m55s&timeoutSeconds=475&watch=true 200 OK in 3 milliseconds I0119 02:31:14.783520 1 azuredisk_v2.go:578] "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-19T02:31:14Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="58735119-97a1-11ed-af75-002248822ac5" I0119 02:31:14.798519 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-29517166-vmss000000/status 200 OK in 14 milliseconds I0119 02:31:14.798769 1 azuredisk_v2.go:565] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="58735119-97a1-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=15281192 I0119 02:31:15.392962 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I0119 02:31:15.392985 1 utils.go:79] GRPC request: {} I0119 02:31:15.393031 1 utils.go:85] GRPC response: {"ready":{"value":true}} I0119 02:31:34.271143 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:31:34.271205 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:31:34.271292 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:31:34.271306 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:31:34.271308 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:31:34.303529 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:31:44.784292 1 azuredisk_v2.go:578] "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-19T02:31:44Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="6a551256-97a1-11ed-af75-002248822ac5" I0119 02:31:44.797220 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-29517166-vmss000000/status 200 OK in 12 milliseconds I0119 02:31:44.797829 1 azuredisk_v2.go:565] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="6a551256-97a1-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=13557967 I0119 02:31:45.393615 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I0119 02:31:45.393636 1 utils.go:79] GRPC request: {} I0119 02:31:45.393679 1 utils.go:85] GRPC response: {"ready":{"value":true}} E0119 02:31:48.933255 1 conditionwaiter.go:50] "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [context deadline exceeded]" "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="2547f1ab-97a1-11ed-af75-002248822ac5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-d50823a8-8f4a-4a25-bf58-4b7def955165" "latency"=119997668016 E0119 02:31:48.933322 1 crdprovisioner.go:743] "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [context deadline exceeded]" "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="2547f1ab-97a1-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=119997874718 E0119 02:31:48.933341 1 utils.go:83] GRPC error: rpc error: code = Internal desc = failed to wait for volume (/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-iei3lcrr/providers/Microsoft.Compute/disks/pvc-d50823a8-8f4a-4a25-bf58-4b7def955165) to be attached to node (k8s-agentpool1-29517166-vmss000000): context deadline exceeded I0119 02:31:49.538810 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities I0119 02:31:49.538835 1 utils.go:79] GRPC request: {} I0119 02:31:49.538878 1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]} I0119 02:31:49.539699 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities I0119 02:31:49.539716 1 utils.go:79] GRPC request: {} I0119 02:31:49.539743 1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]} ... skipping 3 lines ... I0119 02:32:02.225976 1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: Watch close - *v1beta2.AzVolumeAttachment total 104 items received I0119 02:32:02.229258 1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumeattachments?allowWatchBookmarks=true&resourceVersion=16159&timeout=7m21s&timeoutSeconds=441&watch=true 200 OK in 3 milliseconds I0119 02:32:04.272386 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:32:04.272442 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:32:04.272504 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:32:04.272507 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:32:04.272603 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:32:04.303750 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:32:14.783351 1 azuredisk_v2.go:578] "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-19T02:32:14Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="7c369089-97a1-11ed-af75-002248822ac5" I0119 02:32:14.802112 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-29517166-vmss000000/status 200 OK in 18 milliseconds I0119 02:32:14.802515 1 azuredisk_v2.go:565] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="7c369089-97a1-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=19195041 I0119 02:32:15.393163 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I0119 02:32:15.393185 1 utils.go:79] GRPC request: {} I0119 02:32:15.393231 1 utils.go:85] GRPC response: {"ready":{"value":true}} I0119 02:32:34.272499 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:32:34.272602 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:32:34.272637 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:32:34.272638 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:32:34.272792 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:32:34.303934 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:32:44.781184 1 azuredisk_v2.go:578] "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-19T02:32:44Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="8e17deb0-97a1-11ed-af75-002248822ac5" I0119 02:32:44.799129 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-29517166-vmss000000/status 200 OK in 17 milliseconds I0119 02:32:44.799275 1 azuredisk_v2.go:565] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="8e17deb0-97a1-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=18119008 I0119 02:32:45.393626 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I0119 02:32:45.393653 1 utils.go:79] GRPC request: {} I0119 02:32:45.393695 1 utils.go:85] GRPC response: {"ready":{"value":true}} I0119 02:33:04.278321 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:33:04.278360 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:33:04.278456 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:33:04.278466 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:33:04.278475 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:33:04.304894 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:33:14.784816 1 azuredisk_v2.go:578] "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-19T02:33:14Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="9ffa0fd6-97a1-11ed-af75-002248822ac5" I0119 02:33:14.791338 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-29517166-vmss000000/status 200 OK in 6 milliseconds I0119 02:33:14.792037 1 azuredisk_v2.go:565] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="9ffa0fd6-97a1-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=7249783 I0119 02:33:15.393628 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I0119 02:33:15.393649 1 utils.go:79] GRPC request: {} I0119 02:33:15.393697 1 utils.go:85] GRPC response: {"ready":{"value":true}} I0119 02:33:34.280901 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:33:34.280996 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:33:34.281016 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:33:34.281046 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:33:34.281158 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:33:34.305365 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:33:44.778486 1 azuredisk_v2.go:578] "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-19T02:33:44Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="b1dabbec-97a1-11ed-af75-002248822ac5" I0119 02:33:44.782952 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-29517166-vmss000000/status 200 OK in 4 milliseconds I0119 02:33:44.783578 1 azuredisk_v2.go:565] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="b1dabbec-97a1-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=5110159 I0119 02:33:45.393166 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I0119 02:33:45.393193 1 utils.go:79] GRPC request: {} I0119 02:33:45.393241 1 utils.go:85] GRPC response: {"ready":{"value":true}} E0119 02:33:49.538453 1 conditionwaiter.go:50] "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [context deadline exceeded]" "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="6d2ad331-97a1-11ed-af75-002248822ac5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-d50823a8-8f4a-4a25-bf58-4b7def955165" "latency"=119997788424 E0119 02:33:49.538507 1 crdprovisioner.go:743] "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [context deadline exceeded]" "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="6d2ad331-97a1-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=119997953526 E0119 02:33:49.538529 1 utils.go:83] GRPC error: rpc error: code = Internal desc = failed to wait for volume (/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-iei3lcrr/providers/Microsoft.Compute/disks/pvc-d50823a8-8f4a-4a25-bf58-4b7def955165) to be attached to node (k8s-agentpool1-29517166-vmss000000): context deadline exceeded I0119 02:33:50.648506 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities I0119 02:33:50.648531 1 utils.go:79] GRPC request: {} I0119 02:33:50.648569 1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]} I0119 02:33:50.649329 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities I0119 02:33:50.649346 1 utils.go:79] GRPC request: {} I0119 02:33:50.649376 1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]} ... skipping 3 lines ... I0119 02:33:53.214684 1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: Watch close - *v1beta2.AzVolume total 114 items received I0119 02:33:53.221005 1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumes?allowWatchBookmarks=true&resourceVersion=16421&timeout=7m42s&timeoutSeconds=462&watch=true 200 OK in 6 milliseconds I0119 02:34:04.281889 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:34:04.282030 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:34:04.282035 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:34:04.282133 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:34:04.282194 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:34:04.306339 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:34:14.782266 1 azuredisk_v2.go:578] "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-19T02:34:14Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="c3bcf226-97a1-11ed-af75-002248822ac5" I0119 02:34:14.801917 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-29517166-vmss000000/status 200 OK in 19 milliseconds I0119 02:34:14.802891 1 azuredisk_v2.go:565] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="c3bcf226-97a1-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=20644748 I0119 02:34:15.393614 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I0119 02:34:15.393637 1 utils.go:79] GRPC request: {} I0119 02:34:15.393683 1 utils.go:85] GRPC response: {"ready":{"value":true}} I0119 02:34:34.283038 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:34:34.283121 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:34:34.283157 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:34:34.283187 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:34:34.283163 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:34:34.306430 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:34:44.552004 1 conditionwatcher.go:173] condition result: succeeded: false, error: rpc error: code = Unknown desc = context deadline exceeded E0119 02:34:44.552127 1 conditionwaiter.go:50] "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [rpc error: code = Unknown desc = context deadline exceeded]" "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="b55ab46e-97a1-11ed-af75-002248822ac5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-d50823a8-8f4a-4a25-bf58-4b7def955165" "latency"=53901725828 E0119 02:34:44.552174 1 crdprovisioner.go:743] "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [rpc error: code = Unknown desc = context deadline exceeded]" "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="b55ab46e-97a1-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=53901874630 E0119 02:34:44.552192 1 utils.go:83] GRPC error: rpc error: code = Internal desc = failed to wait for volume (/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-iei3lcrr/providers/Microsoft.Compute/disks/pvc-d50823a8-8f4a-4a25-bf58-4b7def955165) to be attached to node (k8s-agentpool1-29517166-vmss000000): rpc error: code = Unknown desc = context deadline exceeded I0119 02:34:44.779115 1 azuredisk_v2.go:578] "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-19T02:34:44Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="d59e1a24-97a1-11ed-af75-002248822ac5" I0119 02:34:44.786616 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-29517166-vmss000000/status 200 OK in 7 milliseconds I0119 02:34:44.787234 1 azuredisk_v2.go:565] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="d59e1a24-97a1-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=8147594 I0119 02:34:45.393337 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I0119 02:34:45.393362 1 utils.go:79] GRPC request: {} I0119 02:34:45.393413 1 utils.go:85] GRPC response: {"ready":{"value":true}} ... skipping 5 lines ... I0119 02:34:46.603575 1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]} I0119 02:34:46.604330 1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume I0119 02:34:46.604346 1 utils.go:79] GRPC request: {"publish_context":{"LUN":"5"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-d50823a8-8f4a-4a25-bf58-4b7def955165","volume_capability":{"AccessType":{"Block":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-d50823a8-8f4a-4a25-bf58-4b7def955165","csi.storage.k8s.io/pvc/name":"test.csi.azure.comt45nd","csi.storage.k8s.io/pvc/namespace":"multivolume-4004","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1674093937787-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-iei3lcrr/providers/Microsoft.Compute/disks/pvc-d50823a8-8f4a-4a25-bf58-4b7def955165"} I0119 02:34:46.613042 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-d50823a8-8f4a-4a25-bf58-4b7def955165-k8s-agentpool1-29517166-vmss000000-attachment/status 200 OK in 8 milliseconds I0119 02:34:46.613274 1 azure_disk_utils.go:1103] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/request-id"="d6b4a2b1-97a1-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "latency"=8747401 "number_of_net_retry"=0 "number_of_retry"=0 I0119 02:34:46.613290 1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-d50823a8-8f4a-4a25-bf58-4b7def955165-k8s-agentpool1-29517166-vmss000000-attachment) I0119 02:34:46.614058 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:34:46.619132 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:34:46.712523 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:34:47.725902 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:34:47.725983 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="d6b4a2b1-97a1-11ed-af75-002248822ac5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-d50823a8-8f4a-4a25-bf58-4b7def955165" "latency"=1112631237 I0119 02:34:47.726012 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="d6b4a2b1-97a1-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=1121535240 I0119 02:34:48.311340 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun5 by sdc under /dev/disk/azure/scsi1/ I0119 02:34:48.311387 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun5. perfProfile none accountType I0119 02:34:48.311398 1 utils.go:85] GRPC response: {} I0119 02:34:48.319368 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities ... skipping 49 lines ... I0119 02:35:15.393211 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I0119 02:35:15.393234 1 utils.go:79] GRPC request: {} I0119 02:35:15.393272 1 utils.go:85] GRPC response: {"ready":{"value":true}} I0119 02:35:34.284489 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:35:34.284550 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:35:34.284569 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:35:34.284614 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:35:34.284640 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:35:34.306938 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:35:44.779207 1 azuredisk_v2.go:578] "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-19T02:35:44Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="f961622d-97a1-11ed-af75-002248822ac5" I0119 02:35:44.792597 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-29517166-vmss000000/status 200 OK in 13 milliseconds I0119 02:35:44.793046 1 azuredisk_v2.go:565] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="f961622d-97a1-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=13894158 I0119 02:35:45.393084 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I0119 02:35:45.393109 1 utils.go:79] GRPC request: {} I0119 02:35:45.393150 1 utils.go:85] GRPC response: {"ready":{"value":true}} I0119 02:35:47.171438 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:35:47.171496 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="e2d78275-97a1-11ed-af75-002248822ac5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-8512c96c-52c9-47a5-b805-c1a2164b9779" "latency"=40205734595 I0119 02:35:47.171517 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="e2d78275-97a1-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=40205837197 I0119 02:35:48.249429 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities I0119 02:35:48.249452 1 utils.go:79] GRPC request: {} I0119 02:35:48.249495 1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]} I0119 02:35:48.252864 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities ... skipping 46 lines ... I0119 02:35:49.302485 1 utils.go:79] GRPC request: {"publish_context":{"LUN":"3"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8512c96c-52c9-47a5-b805-c1a2164b9779/globalmount","target_path":"/var/lib/kubelet/pods/22ef29b8-a0b0-4e77-9565-ee68effc658a/volumes/kubernetes.io~csi/pvc-8512c96c-52c9-47a5-b805-c1a2164b9779/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"xfs"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-8512c96c-52c9-47a5-b805-c1a2164b9779","csi.storage.k8s.io/pvc/name":"test.csi.azure.comzxdj8","csi.storage.k8s.io/pvc/namespace":"volume-9217","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1674093937787-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-iei3lcrr/providers/Microsoft.Compute/disks/pvc-8512c96c-52c9-47a5-b805-c1a2164b9779"} I0119 02:35:49.302654 1 nodeserver_v2.go:348] NodePublishVolume: mounting /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8512c96c-52c9-47a5-b805-c1a2164b9779/globalmount at /var/lib/kubelet/pods/22ef29b8-a0b0-4e77-9565-ee68effc658a/volumes/kubernetes.io~csi/pvc-8512c96c-52c9-47a5-b805-c1a2164b9779/mount I0119 02:35:49.302672 1 mount_linux.go:183] Mounting cmd (mount) with arguments ( -o bind /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8512c96c-52c9-47a5-b805-c1a2164b9779/globalmount /var/lib/kubelet/pods/22ef29b8-a0b0-4e77-9565-ee68effc658a/volumes/kubernetes.io~csi/pvc-8512c96c-52c9-47a5-b805-c1a2164b9779/mount) I0119 02:35:49.303494 1 mount_linux.go:183] Mounting cmd (mount) with arguments ( -o bind,remount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8512c96c-52c9-47a5-b805-c1a2164b9779/globalmount /var/lib/kubelet/pods/22ef29b8-a0b0-4e77-9565-ee68effc658a/volumes/kubernetes.io~csi/pvc-8512c96c-52c9-47a5-b805-c1a2164b9779/mount) I0119 02:35:49.304240 1 nodeserver_v2.go:353] NodePublishVolume: mount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8512c96c-52c9-47a5-b805-c1a2164b9779/globalmount at /var/lib/kubelet/pods/22ef29b8-a0b0-4e77-9565-ee68effc658a/volumes/kubernetes.io~csi/pvc-8512c96c-52c9-47a5-b805-c1a2164b9779/mount successfully I0119 02:35:49.304262 1 utils.go:85] GRPC response: {} I0119 02:35:53.703910 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:35:53.703989 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="fb73ad2c-97a1-11ed-af75-002248822ac5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-8485d7eb-1250-4db3-9511-4ab959eecf1e" "latency"=5449448712 I0119 02:35:53.704018 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="fb73ad2c-97a1-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=5449559013 I0119 02:35:53.707491 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:35:53.707556 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="fb92550d-97a1-11ed-af75-002248822ac5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-71cbc491-43d0-433c-97cd-d94e4663b3b2" "latency"=5252112480 I0119 02:35:53.707595 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="fb92550d-97a1-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=5252228882 I0119 02:35:54.244371 1 reflector.go:559] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: Watch close - *v1.CustomResourceDefinition total 6 items received I0119 02:35:54.269138 1 round_trippers.go:553] GET https://10.0.0.1:443/apis/apiextensions.k8s.io/v1/customresourcedefinitions?allowWatchBookmarks=true&resourceVersion=16255&timeout=6m51s&timeoutSeconds=411&watch=true 200 OK in 24 milliseconds I0119 02:35:54.271421 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdd under /dev/disk/azure/scsi1/ I0119 02:35:54.271460 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType ... skipping 127 lines ... I0119 02:36:01.464389 1 utils.go:79] GRPC request: {"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-iei3lcrr/providers/Microsoft.Compute/disks/pvc-8485d7eb-1250-4db3-9511-4ab959eecf1e","volume_path":"/var/lib/kubelet/pods/37bf676a-2b25-42d9-8d7e-602d9383f86b/volumes/kubernetes.io~csi/pvc-8485d7eb-1250-4db3-9511-4ab959eecf1e/mount"} I0119 02:36:01.464472 1 utils.go:85] GRPC response: {"usage":[{"available":5286961152,"total":5358223360,"unit":1,"used":71262208},{"available":2621437,"total":2621440,"unit":2,"used":3}]} I0119 02:36:04.286138 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:36:04.286154 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:36:04.286172 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:36:04.286209 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:36:04.286248 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:36:04.286282 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:36:04.286314 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:36:04.307534 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:36:05.985661 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:36:05.985730 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="0234e53b-97a2-11ed-af75-002248822ac5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-c1059538-b4f0-4bd1-8857-5cc7423f2b65" "latency"=6398584897 I0119 02:36:05.985753 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="0234e53b-97a2-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=6398690498 I0119 02:36:05.995002 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:36:05.995061 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="0215c6b2-97a2-11ed-af75-002248822ac5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-c89145e2-7612-4672-9c28-3737baab61d1" "latency"=6611857043 I0119 02:36:05.995079 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:36:05.995093 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="0215c6b2-97a2-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=6611968545 I0119 02:36:05.995127 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="0252bf23-97a2-11ed-af75-002248822ac5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-6a72dd82-95cc-4837-8325-6d46cac2f32a" "latency"=6212353261 I0119 02:36:05.995155 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="0252bf23-97a2-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=6212458662 I0119 02:36:06.919049 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun4 by sdh under /dev/disk/azure/scsi1/ I0119 02:36:06.919064 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun2 by sdg under /dev/disk/azure/scsi1/ I0119 02:36:06.919094 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun2. perfProfile none accountType ... skipping 164 lines ... I0119 02:36:19.186360 1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-99e5899c-d022-49ca-992b-f919f6daf035-k8s-agentpool1-29517166-vmss000000-attachment) I0119 02:36:22.280646 1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: Watch close - *v1beta2.AzDriverNode total 25 items received I0119 02:36:22.284186 1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dk8s-agentpool1-29517166-vmss000000&resourceVersion=18559&timeout=8m48s&timeoutSeconds=528&watch=true 200 OK in 3 milliseconds I0119 02:36:34.290577 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:36:34.290615 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:36:34.290654 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:36:34.290670 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:36:34.290684 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:36:34.308063 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:36:42.269502 1 utils.go:78] GRPC call: /csi.v1.Node/NodeUnpublishVolume I0119 02:36:42.269574 1 utils.go:78] GRPC call: /csi.v1.Node/NodeUnpublishVolume I0119 02:36:42.269530 1 utils.go:79] GRPC request: {"target_path":"/var/lib/kubelet/pods/f0ec29e7-d202-446f-848d-72982c8556df/volumes/kubernetes.io~csi/pvc-c89145e2-7612-4672-9c28-3737baab61d1/mount","volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-iei3lcrr/providers/Microsoft.Compute/disks/pvc-c89145e2-7612-4672-9c28-3737baab61d1"} I0119 02:36:42.269604 1 nodeserver_v2.go:369] NodeUnpublishVolume: unmounting volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-iei3lcrr/providers/Microsoft.Compute/disks/pvc-c89145e2-7612-4672-9c28-3737baab61d1 on /var/lib/kubelet/pods/f0ec29e7-d202-446f-848d-72982c8556df/volumes/kubernetes.io~csi/pvc-c89145e2-7612-4672-9c28-3737baab61d1/mount ... skipping 38 lines ... I0119 02:36:45.393202 1 utils.go:79] GRPC request: {} I0119 02:36:45.393240 1 utils.go:85] GRPC response: {"ready":{"value":true}} I0119 02:37:04.291057 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:37:04.291120 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:37:04.291155 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:37:04.291201 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:37:04.291281 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:37:04.308515 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:37:05.478399 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:37:05.478459 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="0de381d4-97a2-11ed-af75-002248822ac5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-99e5899c-d022-49ca-992b-f919f6daf035" "latency"=46292027017 I0119 02:37:05.478487 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="0de381d4-97a2-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=46292146018 I0119 02:37:05.784675 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities I0119 02:37:05.784696 1 utils.go:79] GRPC request: {} I0119 02:37:05.784730 1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]} I0119 02:37:05.788019 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities ... skipping 63 lines ... I0119 02:37:19.625531 1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-99e5899c-d022-49ca-992b-f919f6daf035 successfully I0119 02:37:19.625539 1 utils.go:85] GRPC response: {} I0119 02:37:34.299891 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:37:34.299925 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:37:34.299965 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:37:34.300025 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:37:34.300032 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:37:34.300052 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:37:34.300063 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:37:34.309179 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:37:44.784319 1 azuredisk_v2.go:578] "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-19T02:37:44Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="40e8b723-97a2-11ed-af75-002248822ac5" I0119 02:37:44.792482 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-29517166-vmss000000/status 200 OK in 7 milliseconds I0119 02:37:44.792644 1 azuredisk_v2.go:565] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="40e8b723-97a2-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=8364293 I0119 02:37:45.393362 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I0119 02:37:45.393386 1 utils.go:79] GRPC request: {} I0119 02:37:45.393433 1 utils.go:85] GRPC response: {"ready":{"value":true}} I0119 02:38:04.300629 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:38:04.300675 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:38:04.300713 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:38:04.300803 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:38:04.300818 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:38:04.300849 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:38:04.300864 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:38:04.310158 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:38:14.778325 1 azuredisk_v2.go:578] "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-19T02:38:14Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="52c97051-97a2-11ed-af75-002248822ac5" I0119 02:38:14.784864 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-29517166-vmss000000/status 200 OK in 6 milliseconds I0119 02:38:14.785505 1 azuredisk_v2.go:565] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="52c97051-97a2-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=7219381 I0119 02:38:15.393600 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I0119 02:38:15.393629 1 utils.go:79] GRPC request: {} I0119 02:38:15.393679 1 utils.go:85] GRPC response: {"ready":{"value":true}} I0119 02:38:21.381832 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:38:21.381897 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="29e84977-97a2-11ed-af75-002248822ac5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-be1e4442-0d18-404d-b36c-8089b08cbe32" "latency"=75187933862 I0119 02:38:21.381922 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="29e84977-97a2-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=75188054563 I0119 02:38:21.389148 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:38:21.389216 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="29e84a29-97a2-11ed-af75-002248822ac5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-9a960e5e-a3d4-416b-a791-4a8e35348ee6" "latency"=75195224943 I0119 02:38:21.389250 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="29e84a29-97a2-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=75195357844 I0119 02:38:21.389411 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:38:21.389465 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="29aaa3b1-97a2-11ed-af75-002248822ac5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-542309de-2e70-4a0d-b31a-1e9aef6d43c1" "latency"=75599523138 I0119 02:38:21.389502 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="29aaa3b1-97a2-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=75599643640 I0119 02:38:22.287463 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun8 by sdc under /dev/disk/azure/scsi1/ I0119 02:38:22.287688 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun8. perfProfile none accountType I0119 02:38:22.287709 1 utils.go:85] GRPC response: {} I0119 02:38:22.287736 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun3 by sde under /dev/disk/azure/scsi1/ ... skipping 134 lines ... I0119 02:39:03.226494 1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: Watch close - *v1beta2.AzDriverNode total 71 items received I0119 02:39:03.229479 1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azdrivernodes?allowWatchBookmarks=true&resourceVersion=19880&timeout=6m16s&timeoutSeconds=376&watch=true 200 OK in 2 milliseconds I0119 02:39:04.302118 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:39:04.302151 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:39:04.302206 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:39:04.302207 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:39:04.302278 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:39:04.311523 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:39:13.583060 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:39:13.583116 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="630f8d25-97a2-11ed-af75-002248822ac5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-cfc1118f-500a-4ca4-8e5f-2d991d0895cf" "latency"=31501700613 I0119 02:39:13.583141 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="630f8d25-97a2-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=31501819216 I0119 02:39:14.189325 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdf under /dev/disk/azure/scsi1/ I0119 02:39:14.189370 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType I0119 02:39:14.189409 1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-cfc1118f-500a-4ca4-8e5f-2d991d0895cf/globalmount with mount options([]) I0119 02:39:14.189419 1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0]) ... skipping 80 lines ... I0119 02:39:23.228541 1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: Watch close - *v1beta2.AzVolumeAttachment total 197 items received I0119 02:39:23.230171 1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumeattachments?allowWatchBookmarks=true&resourceVersion=20220&timeout=9m5s&timeoutSeconds=545&watch=true 200 OK in 1 milliseconds I0119 02:39:34.302627 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:39:34.302703 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:39:34.302795 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:39:34.302798 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:39:34.302915 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:39:34.312140 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:39:35.221135 1 reflector.go:559] k8s.io/client-go/informers/factory.go:150: Watch close - *v1.Node total 152 items received I0119 02:39:35.224002 1 round_trippers.go:553] GET https://10.0.0.1:443/api/v1/nodes?allowWatchBookmarks=true&resourceVersion=20317&timeout=9m32s&timeoutSeconds=572&watch=true 200 OK in 2 milliseconds I0119 02:39:44.779785 1 azuredisk_v2.go:578] "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-19T02:39:44Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="886e921d-97a2-11ed-af75-002248822ac5" I0119 02:39:44.794648 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-29517166-vmss000000/status 200 OK in 14 milliseconds I0119 02:39:44.796543 1 azuredisk_v2.go:565] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="886e921d-97a2-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=16805697 I0119 02:39:45.393636 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I0119 02:39:45.393663 1 utils.go:79] GRPC request: {} I0119 02:39:45.393714 1 utils.go:85] GRPC response: {"ready":{"value":true}} I0119 02:40:04.303181 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:40:04.303237 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:40:04.303285 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:40:04.303296 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:40:04.303471 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:40:04.312783 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:40:14.784217 1 azuredisk_v2.go:578] "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-19T02:40:14Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="9a50e23f-97a2-11ed-af75-002248822ac5" I0119 02:40:14.795641 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-29517166-vmss000000/status 200 OK in 11 milliseconds I0119 02:40:14.795807 1 azuredisk_v2.go:565] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="9a50e23f-97a2-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=11627077 I0119 02:40:15.393046 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I0119 02:40:15.393093 1 utils.go:79] GRPC request: {} I0119 02:40:15.393146 1 utils.go:85] GRPC response: {"ready":{"value":true}} I0119 02:40:34.304195 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:40:34.304298 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:40:34.304325 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:40:34.304347 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:40:34.304354 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:40:34.313614 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:40:36.594820 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:40:36.594893 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="7636e3fe-97a2-11ed-af75-002248822ac5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-103d1c80-cf60-400b-8bfa-e87969753277" "latency"=82378943187 I0119 02:40:36.594915 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="7636e3fe-97a2-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=82379071891 I0119 02:40:37.447280 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities I0119 02:40:37.447304 1 utils.go:79] GRPC request: {} I0119 02:40:37.447339 1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]} I0119 02:40:37.450782 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities ... skipping 78 lines ... I0119 02:40:56.787765 1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-103d1c80-cf60-400b-8bfa-e87969753277 successfully I0119 02:40:56.787782 1 utils.go:85] GRPC response: {} I0119 02:41:04.305749 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:41:04.305791 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:41:04.305826 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:41:04.305889 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:41:04.305934 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:41:04.306017 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:41:04.314189 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:41:11.935514 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:41:11.935569 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="a7d3c6f6-97a2-11ed-af75-002248822ac5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-d6053011-dd2c-4a8d-a51d-da31163b4334" "latency"=34483109724 I0119 02:41:11.935595 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="a7d3c6f6-97a2-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=34483216826 I0119 02:41:11.940589 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:41:11.940658 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="a88bc057-97a2-11ed-af75-002248822ac5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-7c681bc6-a9ef-4175-a6c5-bc7a91546cc3" "latency"=33282511322 I0119 02:41:11.940692 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="a88bc057-97a2-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=33282620525 I0119 02:41:12.542325 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun2 by sdd under /dev/disk/azure/scsi1/ I0119 02:41:12.542377 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun2. perfProfile none accountType I0119 02:41:12.542410 1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun2 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-d6053011-dd2c-4a8d-a51d-da31163b4334/globalmount with mount options([]) I0119 02:41:12.542419 1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun2" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun2]) ... skipping 158 lines ... I0119 02:42:15.392752 1 utils.go:79] GRPC request: {} I0119 02:42:15.392801 1 utils.go:85] GRPC response: {"ready":{"value":true}} I0119 02:42:34.306794 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:42:34.306819 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:42:34.306795 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:42:34.306888 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:42:34.306898 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:42:34.316138 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:42:42.648701 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:42:42.648776 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="dcd58798-97a2-11ed-af75-002248822ac5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-7d5f55b2-cacb-4082-83b6-f50e28b2871b" "latency"=36265562416 I0119 02:42:42.648813 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="dcd58798-97a2-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=36265695017 I0119 02:42:43.257849 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun3 by sdc under /dev/disk/azure/scsi1/ I0119 02:42:43.257900 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun3. perfProfile none accountType I0119 02:42:43.257912 1 utils.go:85] GRPC response: {} I0119 02:42:43.263532 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities ... skipping 43 lines ... I0119 02:43:14.779242 1 azuredisk_v2.go:578] "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-19T02:43:14Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="0599f17e-97a3-11ed-af75-002248822ac5" I0119 02:43:14.786378 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-29517166-vmss000000/status 200 OK in 6 milliseconds I0119 02:43:14.786570 1 azuredisk_v2.go:565] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="0599f17e-97a3-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=7367319 I0119 02:43:15.392811 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I0119 02:43:15.392833 1 utils.go:79] GRPC request: {} I0119 02:43:15.392877 1 utils.go:85] GRPC response: {"ready":{"value":true}} I0119 02:43:18.099335 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:43:18.099396 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="01f28a15-97a3-11ed-af75-002248822ac5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-5655f9b5-8fae-4d01-bc80-66da9f67984a" "latency"=9450370361 I0119 02:43:18.099423 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="01f28a15-97a3-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9450497864 I0119 02:43:18.135882 1 utils.go:78] GRPC call: /csi.v1.Node/NodeUnpublishVolume I0119 02:43:18.135900 1 utils.go:79] GRPC request: {"target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-7d5f55b2-cacb-4082-83b6-f50e28b2871b/5cbf6f4c-0acc-4301-b3ef-1855af7f2009","volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-iei3lcrr/providers/Microsoft.Compute/disks/pvc-7d5f55b2-cacb-4082-83b6-f50e28b2871b"} I0119 02:43:18.135960 1 nodeserver_v2.go:369] NodeUnpublishVolume: unmounting volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-iei3lcrr/providers/Microsoft.Compute/disks/pvc-7d5f55b2-cacb-4082-83b6-f50e28b2871b on /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-7d5f55b2-cacb-4082-83b6-f50e28b2871b/5cbf6f4c-0acc-4301-b3ef-1855af7f2009 I0119 02:43:18.135988 1 mount_helper_common.go:99] "/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-7d5f55b2-cacb-4082-83b6-f50e28b2871b/5cbf6f4c-0acc-4301-b3ef-1855af7f2009" is a mountpoint, unmounting ... skipping 73 lines ... W0119 02:43:29.797591 1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-5655f9b5-8fae-4d01-bc80-66da9f67984a/globalmount" is not a mountpoint, deleting I0119 02:43:29.797659 1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-5655f9b5-8fae-4d01-bc80-66da9f67984a/globalmount successfully I0119 02:43:29.797678 1 utils.go:85] GRPC response: {} I0119 02:43:34.310437 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:43:34.310489 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:43:34.310506 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:43:34.310536 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:43:34.310558 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:43:34.317737 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:43:38.249606 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities I0119 02:43:38.249629 1 utils.go:79] GRPC request: {} I0119 02:43:38.249667 1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]} I0119 02:43:38.253279 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities ... skipping 38 lines ... I0119 02:43:44.780060 1 azuredisk_v2.go:578] "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-19T02:43:44Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="177bb4b5-97a3-11ed-af75-002248822ac5" I0119 02:43:44.793957 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-29517166-vmss000000/status 200 OK in 13 milliseconds I0119 02:43:44.794327 1 azuredisk_v2.go:565] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="177bb4b5-97a3-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=14303832 I0119 02:43:45.393228 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I0119 02:43:45.393262 1 utils.go:79] GRPC request: {} I0119 02:43:45.393313 1 utils.go:85] GRPC response: {"ready":{"value":true}} I0119 02:43:47.881832 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:43:47.881906 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="0d977062-97a3-11ed-af75-002248822ac5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-0b99137b-d6a4-4f2f-95e1-2392f67376ec" "latency"=19697278144 I0119 02:43:47.881943 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="0d977062-97a3-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=19697388646 I0119 02:43:48.469040 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sde under /dev/disk/azure/scsi1/ I0119 02:43:48.469119 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType I0119 02:43:48.469155 1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun1 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-0b99137b-d6a4-4f2f-95e1-2392f67376ec/globalmount with mount options([]) I0119 02:43:48.469173 1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun1" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun1]) ... skipping 89 lines ... I0119 02:44:23.764917 1 nodeserver_v2.go:257] NodeUnstageVolume: unmounting /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-0b99137b-d6a4-4f2f-95e1-2392f67376ec/globalmount I0119 02:44:23.764943 1 mount_helper_common.go:99] "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-0b99137b-d6a4-4f2f-95e1-2392f67376ec/globalmount" is a mountpoint, unmounting I0119 02:44:23.764955 1 mount_linux.go:294] Unmounting /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-0b99137b-d6a4-4f2f-95e1-2392f67376ec/globalmount W0119 02:44:23.774336 1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-0b99137b-d6a4-4f2f-95e1-2392f67376ec/globalmount" is not a mountpoint, deleting I0119 02:44:23.774399 1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-0b99137b-d6a4-4f2f-95e1-2392f67376ec/globalmount successfully I0119 02:44:23.774413 1 utils.go:85] GRPC response: {} I0119 02:44:24.414857 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:44:24.414939 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="2616b67e-97a3-11ed-af75-002248822ac5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-7ba391a5-305a-45fd-85c5-071d88d1e462" "latency"=15130863909 I0119 02:44:24.414974 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="2616b67e-97a3-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=15131000511 I0119 02:44:25.011304 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun2 by sdc under /dev/disk/azure/scsi1/ I0119 02:44:25.011360 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun2. perfProfile none accountType I0119 02:44:25.011405 1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun2 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-7ba391a5-305a-45fd-85c5-071d88d1e462/globalmount with mount options([]) I0119 02:44:25.011418 1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun2" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun2]) ... skipping 57 lines ... I0119 02:44:29.077653 1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-7ba391a5-305a-45fd-85c5-071d88d1e462/globalmount successfully I0119 02:44:29.077663 1 utils.go:85] GRPC response: {} I0119 02:44:34.312276 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:44:34.312389 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:44:34.312397 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:44:34.312433 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:44:34.312446 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:44:34.319682 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:44:34.802513 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:44:34.802594 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="2fa9ec3e-97a3-11ed-af75-002248822ac5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-62398535-c59c-4669-aa53-4a0739a3ad50" "latency"=9454291383 I0119 02:44:34.802629 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="2fa9ec3e-97a3-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9454404684 I0119 02:44:35.394839 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdd under /dev/disk/azure/scsi1/ I0119 02:44:35.394889 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType I0119 02:44:35.394924 1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-62398535-c59c-4669-aa53-4a0739a3ad50/globalmount with mount options([]) I0119 02:44:35.394935 1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0]) ... skipping 59 lines ... I0119 02:44:44.780730 1 azuredisk_v2.go:578] "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-19T02:44:44Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="3b3f142d-97a3-11ed-af75-002248822ac5" I0119 02:44:44.793379 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-29517166-vmss000000/status 200 OK in 12 milliseconds I0119 02:44:44.793631 1 azuredisk_v2.go:565] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="3b3f142d-97a3-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=12956882 I0119 02:44:45.393601 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I0119 02:44:45.393629 1 utils.go:79] GRPC request: {} I0119 02:44:45.393679 1 utils.go:85] GRPC response: {"ready":{"value":true}} I0119 02:45:00.902917 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:45:00.902984 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="385d0de3-97a3-11ed-af75-002248822ac5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-f74a9f16-617c-4ca3-9ff9-5960de0a082c" "latency"=20958962041 I0119 02:45:00.903012 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="385d0de3-97a3-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=20959064443 I0119 02:45:01.500872 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun3 by sdc under /dev/disk/azure/scsi1/ I0119 02:45:01.500932 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun3. perfProfile none accountType I0119 02:45:01.500964 1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun3 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-f74a9f16-617c-4ca3-9ff9-5960de0a082c/globalmount with mount options([]) I0119 02:45:01.500974 1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun3" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun3]) ... skipping 38 lines ... I0119 02:45:01.995152 1 nodeserver_v2.go:353] NodePublishVolume: mount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-f74a9f16-617c-4ca3-9ff9-5960de0a082c/globalmount at /var/lib/kubelet/pods/2d033158-cc90-4319-818f-88807514df5c/volumes/kubernetes.io~csi/pvc-f74a9f16-617c-4ca3-9ff9-5960de0a082c/mount successfully I0119 02:45:01.995176 1 utils.go:85] GRPC response: {} I0119 02:45:04.312757 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:45:04.312841 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:45:04.312864 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:45:04.312898 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:45:04.312926 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:45:04.320158 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:45:06.527828 1 utils.go:78] GRPC call: /csi.v1.Node/NodeUnpublishVolume I0119 02:45:06.527852 1 utils.go:79] GRPC request: {"target_path":"/var/lib/kubelet/pods/2d033158-cc90-4319-818f-88807514df5c/volumes/kubernetes.io~csi/pvc-f74a9f16-617c-4ca3-9ff9-5960de0a082c/mount","volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-iei3lcrr/providers/Microsoft.Compute/disks/pvc-f74a9f16-617c-4ca3-9ff9-5960de0a082c"} I0119 02:45:06.527893 1 nodeserver_v2.go:369] NodeUnpublishVolume: unmounting volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-iei3lcrr/providers/Microsoft.Compute/disks/pvc-f74a9f16-617c-4ca3-9ff9-5960de0a082c on /var/lib/kubelet/pods/2d033158-cc90-4319-818f-88807514df5c/volumes/kubernetes.io~csi/pvc-f74a9f16-617c-4ca3-9ff9-5960de0a082c/mount I0119 02:45:06.527923 1 mount_helper_common.go:99] "/var/lib/kubelet/pods/2d033158-cc90-4319-818f-88807514df5c/volumes/kubernetes.io~csi/pvc-f74a9f16-617c-4ca3-9ff9-5960de0a082c/mount" is a mountpoint, unmounting I0119 02:45:06.527938 1 mount_linux.go:294] Unmounting /var/lib/kubelet/pods/2d033158-cc90-4319-818f-88807514df5c/volumes/kubernetes.io~csi/pvc-f74a9f16-617c-4ca3-9ff9-5960de0a082c/mount ... skipping 10 lines ... I0119 02:45:06.630097 1 mount_linux.go:294] Unmounting /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-f74a9f16-617c-4ca3-9ff9-5960de0a082c/globalmount W0119 02:45:06.642394 1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-f74a9f16-617c-4ca3-9ff9-5960de0a082c/globalmount" is not a mountpoint, deleting I0119 02:45:06.642458 1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-f74a9f16-617c-4ca3-9ff9-5960de0a082c/globalmount successfully I0119 02:45:06.642468 1 utils.go:85] GRPC response: {} I0119 02:45:10.286818 1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: Watch close - *v1beta2.AzDriverNode total 27 items received I0119 02:45:10.289128 1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dk8s-agentpool1-29517166-vmss000000&resourceVersion=23254&timeout=9m33s&timeoutSeconds=573&watch=true 200 OK in 2 milliseconds I0119 02:45:11.399091 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:45:11.399152 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="45649a10-97a3-11ed-af75-002248822ac5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-712cce25-c4af-4f0e-8ed0-2529ecd68b96" "latency"=9595278125 I0119 02:45:11.399180 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="45649a10-97a3-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9595385727 I0119 02:45:11.976413 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sde under /dev/disk/azure/scsi1/ I0119 02:45:11.976462 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType I0119 02:45:11.976490 1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun1 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-712cce25-c4af-4f0e-8ed0-2529ecd68b96/globalmount with mount options([]) I0119 02:45:11.976499 1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun1" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun1]) ... skipping 65 lines ... I0119 02:45:20.950200 1 utils.go:79] GRPC request: {"publish_context":{"LUN":"2"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-9c2075a6-57ea-402e-9991-285259525d8d/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-9c2075a6-57ea-402e-9991-285259525d8d","csi.storage.k8s.io/pvc/name":"test.csi.azure.com25r7t","csi.storage.k8s.io/pvc/namespace":"fsgroupchangepolicy-7808","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1674093937787-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-iei3lcrr/providers/Microsoft.Compute/disks/pvc-9c2075a6-57ea-402e-9991-285259525d8d"} I0119 02:45:20.950369 1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-9c2075a6-57ea-402e-9991-285259525d8d-k8s-agentpool1-29517166-vmss000000-attachment) I0119 02:45:34.313887 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:45:34.313978 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:45:34.314001 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:45:34.314033 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:45:34.314131 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:45:34.321200 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:45:44.782095 1 azuredisk_v2.go:578] "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-19T02:45:44Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="5f02907d-97a3-11ed-af75-002248822ac5" I0119 02:45:44.791714 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-29517166-vmss000000/status 200 OK in 9 milliseconds I0119 02:45:44.791903 1 azuredisk_v2.go:565] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="5f02907d-97a3-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=9841610 I0119 02:45:45.393071 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I0119 02:45:45.393100 1 utils.go:79] GRPC request: {} I0119 02:45:45.393144 1 utils.go:85] GRPC response: {"ready":{"value":true}} I0119 02:46:02.360682 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:46:02.360744 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="50ce2299-97a3-11ed-af75-002248822ac5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-9c2075a6-57ea-402e-9991-285259525d8d" "latency"=41410314898 I0119 02:46:02.360778 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="50ce2299-97a3-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=41410419000 I0119 02:46:04.160011 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun2 by sdc under /dev/disk/azure/scsi1/ I0119 02:46:04.160067 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun2. perfProfile none accountType I0119 02:46:04.160098 1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun2 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-9c2075a6-57ea-402e-9991-285259525d8d/globalmount with mount options([]) I0119 02:46:04.160110 1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun2" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun2]) ... skipping 97 lines ... I0119 02:46:22.695317 1 mount_linux.go:490] Output: "DEVNAME=/dev/disk/azure/scsi1/lun2\nTYPE=ext4\n" I0119 02:46:22.695335 1 mount_linux.go:376] Checking for issues with fsck on disk: /dev/disk/azure/scsi1/lun2 I0119 02:46:22.708883 1 mount_linux.go:477] Attempting to mount disk /dev/disk/azure/scsi1/lun2 in ext4 format at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-9c2075a6-57ea-402e-9991-285259525d8d/globalmount I0119 02:46:22.708911 1 mount_linux.go:183] Mounting cmd (mount) with arguments (-t ext4 -o defaults /dev/disk/azure/scsi1/lun2 /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-9c2075a6-57ea-402e-9991-285259525d8d/globalmount) I0119 02:46:22.712385 1 nodeserver_v2.go:215] NodeStageVolume: format /dev/disk/azure/scsi1/lun2 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-9c2075a6-57ea-402e-9991-285259525d8d/globalmount successfully. I0119 02:46:22.712912 1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun2" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun2]) I0119 02:46:22.714008 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:46:22.714073 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="6afd6978-97a3-11ed-af75-002248822ac5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-63e861e1-db89-497c-b596-3521418a7be9" "latency"=17833007004 I0119 02:46:22.714111 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="6afd6978-97a3-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=17833160906 I0119 02:46:22.717319 1 mount_linux.go:490] Output: "DEVNAME=/dev/disk/azure/scsi1/lun2\nTYPE=ext4\n" I0119 02:46:22.717338 1 resizefs_linux.go:124] ResizeFs.needResize - checking mounted volume /dev/disk/azure/scsi1/lun2 I0119 02:46:22.718382 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:46:22.718439 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="6afd765a-97a3-11ed-af75-002248822ac5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-6abccbed-2a99-4a8c-9d46-a550098c6125" "latency"=17837075669 I0119 02:46:22.718480 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="6afd765a-97a3-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=17837195971 I0119 02:46:22.718658 1 resizefs_linux.go:128] Ext size: filesystem size=5368709120, block size=4096 I0119 02:46:22.718679 1 resizefs_linux.go:143] Volume /dev/disk/azure/scsi1/lun2: device size=5368709120, filesystem size=5368709120, block size=4096 I0119 02:46:22.718693 1 utils.go:85] GRPC response: {} I0119 02:46:22.722886 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities ... skipping 117 lines ... I0119 02:46:33.389664 1 utils.go:79] GRPC request: {"publish_context":{"LUN":"3"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-ff0c3c20-3611-4d61-bd57-17ac092a3f1d/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-ff0c3c20-3611-4d61-bd57-17ac092a3f1d","csi.storage.k8s.io/pvc/name":"test.csi.azure.comj9cpn","csi.storage.k8s.io/pvc/namespace":"multivolume-3499","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1674093937787-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-iei3lcrr/providers/Microsoft.Compute/disks/pvc-ff0c3c20-3611-4d61-bd57-17ac092a3f1d"} I0119 02:46:33.389801 1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-ff0c3c20-3611-4d61-bd57-17ac092a3f1d-k8s-agentpool1-29517166-vmss000000-attachment) I0119 02:46:34.315662 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:46:34.315764 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:46:34.315769 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:46:34.315794 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:46:34.315866 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:46:34.315909 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:46:34.322225 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:46:38.520055 1 utils.go:78] GRPC call: /csi.v1.Node/NodeUnpublishVolume I0119 02:46:38.520074 1 utils.go:79] GRPC request: {"target_path":"/var/lib/kubelet/pods/717455d3-c0fd-44aa-b92f-f1e9fd034766/volumes/kubernetes.io~csi/pvc-6abccbed-2a99-4a8c-9d46-a550098c6125/mount","volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-iei3lcrr/providers/Microsoft.Compute/disks/pvc-6abccbed-2a99-4a8c-9d46-a550098c6125"} I0119 02:46:38.520134 1 nodeserver_v2.go:369] NodeUnpublishVolume: unmounting volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-iei3lcrr/providers/Microsoft.Compute/disks/pvc-6abccbed-2a99-4a8c-9d46-a550098c6125 on /var/lib/kubelet/pods/717455d3-c0fd-44aa-b92f-f1e9fd034766/volumes/kubernetes.io~csi/pvc-6abccbed-2a99-4a8c-9d46-a550098c6125/mount I0119 02:46:38.520142 1 utils.go:78] GRPC call: /csi.v1.Node/NodeUnpublishVolume I0119 02:46:38.520165 1 mount_helper_common.go:99] "/var/lib/kubelet/pods/717455d3-c0fd-44aa-b92f-f1e9fd034766/volumes/kubernetes.io~csi/pvc-6abccbed-2a99-4a8c-9d46-a550098c6125/mount" is a mountpoint, unmounting ... skipping 111 lines ... I0119 02:46:42.996408 1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-63e861e1-db89-497c-b596-3521418a7be9/globalmount","target_path":"/var/lib/kubelet/pods/6b3a1176-1e0d-4dc8-bf17-a10125007be0/volumes/kubernetes.io~csi/pvc-63e861e1-db89-497c-b596-3521418a7be9/mount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-63e861e1-db89-497c-b596-3521418a7be9","csi.storage.k8s.io/pvc/name":"test.csi.azure.comw7fzk","csi.storage.k8s.io/pvc/namespace":"multivolume-7541","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1674093937787-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-iei3lcrr/providers/Microsoft.Compute/disks/pvc-63e861e1-db89-497c-b596-3521418a7be9"} I0119 02:46:42.996557 1 nodeserver_v2.go:348] NodePublishVolume: mounting /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-63e861e1-db89-497c-b596-3521418a7be9/globalmount at /var/lib/kubelet/pods/6b3a1176-1e0d-4dc8-bf17-a10125007be0/volumes/kubernetes.io~csi/pvc-63e861e1-db89-497c-b596-3521418a7be9/mount I0119 02:46:42.996569 1 mount_linux.go:183] Mounting cmd (mount) with arguments ( -o bind /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-63e861e1-db89-497c-b596-3521418a7be9/globalmount /var/lib/kubelet/pods/6b3a1176-1e0d-4dc8-bf17-a10125007be0/volumes/kubernetes.io~csi/pvc-63e861e1-db89-497c-b596-3521418a7be9/mount) I0119 02:46:42.997513 1 mount_linux.go:183] Mounting cmd (mount) with arguments ( -o bind,remount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-63e861e1-db89-497c-b596-3521418a7be9/globalmount /var/lib/kubelet/pods/6b3a1176-1e0d-4dc8-bf17-a10125007be0/volumes/kubernetes.io~csi/pvc-63e861e1-db89-497c-b596-3521418a7be9/mount) I0119 02:46:42.998358 1 nodeserver_v2.go:353] NodePublishVolume: mount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-63e861e1-db89-497c-b596-3521418a7be9/globalmount at /var/lib/kubelet/pods/6b3a1176-1e0d-4dc8-bf17-a10125007be0/volumes/kubernetes.io~csi/pvc-63e861e1-db89-497c-b596-3521418a7be9/mount successfully I0119 02:46:42.998380 1 utils.go:85] GRPC response: {} I0119 02:46:42.999093 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:46:42.999338 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="7bfaaf60-97a3-11ed-af75-002248822ac5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-66f5f690-224e-4d3c-b4f1-57a8145c8f7d" "latency"=9614735876 I0119 02:46:42.999580 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="7bfaaf60-97a3-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9615034879 I0119 02:46:43.003222 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:46:43.003269 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="7bfb8429-97a3-11ed-af75-002248822ac5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-ff0c3c20-3611-4d61-bd57-17ac092a3f1d" "latency"=9613425549 I0119 02:46:43.003296 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="7bfb8429-97a3-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9613503451 I0119 02:46:43.598819 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun3 by sdg under /dev/disk/azure/scsi1/ I0119 02:46:43.598872 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun3. perfProfile none accountType I0119 02:46:43.598913 1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun3 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-ff0c3c20-3611-4d61-bd57-17ac092a3f1d/globalmount with mount options([]) I0119 02:46:43.598934 1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun3" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun3]) ... skipping 134 lines ... I0119 02:47:02.530014 1 utils.go:79] GRPC request: {"publish_context":{"LUN":"5"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-e7e20f6b-f562-461b-9ad6-4b6e33e7931d/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-e7e20f6b-f562-461b-9ad6-4b6e33e7931d","csi.storage.k8s.io/pvc/name":"test.csi.azure.com4z9qw","csi.storage.k8s.io/pvc/namespace":"provisioning-7220","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1674093937787-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-iei3lcrr/providers/Microsoft.Compute/disks/pvc-e7e20f6b-f562-461b-9ad6-4b6e33e7931d"} I0119 02:47:02.530161 1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-e7e20f6b-f562-461b-9ad6-4b6e33e7931d-k8s-agentpool1-29517166-vmss000000-attachment) I0119 02:47:04.317038 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:47:04.317083 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:47:04.317118 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:47:04.317073 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:47:04.317210 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:47:04.322294 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:47:14.780177 1 azuredisk_v2.go:578] "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-19T02:47:14Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="94a72e07-97a3-11ed-af75-002248822ac5" I0119 02:47:14.812332 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-29517166-vmss000000/status 200 OK in 32 milliseconds I0119 02:47:14.812626 1 azuredisk_v2.go:565] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="94a72e07-97a3-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=32492225 I0119 02:47:15.393279 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I0119 02:47:15.393299 1 utils.go:79] GRPC request: {} I0119 02:47:15.393338 1 utils.go:85] GRPC response: {"ready":{"value":true}} I0119 02:47:34.318598 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:47:34.318633 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:47:34.318763 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:47:34.318804 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:47:34.318905 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:47:34.323153 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:47:43.678744 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:47:43.678812 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="8d59fb79-97a3-11ed-af75-002248822ac5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-e7e20f6b-f562-461b-9ad6-4b6e33e7931d" "latency"=41148570839 I0119 02:47:43.678842 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="8d59fb79-97a3-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=41148687541 I0119 02:47:44.263870 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun5 by sdc under /dev/disk/azure/scsi1/ I0119 02:47:44.263923 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun5. perfProfile none accountType I0119 02:47:44.263961 1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun5 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-e7e20f6b-f562-461b-9ad6-4b6e33e7931d/globalmount with mount options([]) I0119 02:47:44.263977 1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun5" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun5]) ... skipping 46 lines ... I0119 02:47:53.252126 1 reflector.go:559] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: Watch close - *v1.CustomResourceDefinition total 0 items received I0119 02:47:53.256486 1 round_trippers.go:553] GET https://10.0.0.1:443/apis/apiextensions.k8s.io/v1/customresourcedefinitions?allowWatchBookmarks=true&resourceVersion=20948&timeout=6m17s&timeoutSeconds=377&watch=true 200 OK in 4 milliseconds I0119 02:48:04.319748 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:48:04.319797 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:48:04.319821 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:48:04.319853 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:48:04.319884 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:48:04.324035 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:48:09.083753 1 utils.go:78] GRPC call: /csi.v1.Node/NodeUnpublishVolume I0119 02:48:09.083774 1 utils.go:79] GRPC request: {"target_path":"/var/lib/kubelet/pods/5ae3f871-3312-49b8-a3cf-e5e4ce4be001/volumes/kubernetes.io~csi/pvc-e7e20f6b-f562-461b-9ad6-4b6e33e7931d/mount","volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-iei3lcrr/providers/Microsoft.Compute/disks/pvc-e7e20f6b-f562-461b-9ad6-4b6e33e7931d"} I0119 02:48:09.083837 1 nodeserver_v2.go:369] NodeUnpublishVolume: unmounting volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-iei3lcrr/providers/Microsoft.Compute/disks/pvc-e7e20f6b-f562-461b-9ad6-4b6e33e7931d on /var/lib/kubelet/pods/5ae3f871-3312-49b8-a3cf-e5e4ce4be001/volumes/kubernetes.io~csi/pvc-e7e20f6b-f562-461b-9ad6-4b6e33e7931d/mount I0119 02:48:09.083874 1 mount_helper_common.go:99] "/var/lib/kubelet/pods/5ae3f871-3312-49b8-a3cf-e5e4ce4be001/volumes/kubernetes.io~csi/pvc-e7e20f6b-f562-461b-9ad6-4b6e33e7931d/mount" is a mountpoint, unmounting I0119 02:48:09.083905 1 mount_linux.go:294] Unmounting /var/lib/kubelet/pods/5ae3f871-3312-49b8-a3cf-e5e4ce4be001/volumes/kubernetes.io~csi/pvc-e7e20f6b-f562-461b-9ad6-4b6e33e7931d/mount ... skipping 8 lines ... I0119 02:48:09.182492 1 nodeserver_v2.go:257] NodeUnstageVolume: unmounting /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-e7e20f6b-f562-461b-9ad6-4b6e33e7931d/globalmount I0119 02:48:09.182533 1 mount_helper_common.go:99] "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-e7e20f6b-f562-461b-9ad6-4b6e33e7931d/globalmount" is a mountpoint, unmounting I0119 02:48:09.182544 1 mount_linux.go:294] Unmounting /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-e7e20f6b-f562-461b-9ad6-4b6e33e7931d/globalmount W0119 02:48:09.192740 1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-e7e20f6b-f562-461b-9ad6-4b6e33e7931d/globalmount" is not a mountpoint, deleting I0119 02:48:09.192816 1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-e7e20f6b-f562-461b-9ad6-4b6e33e7931d/globalmount successfully I0119 02:48:09.192833 1 utils.go:85] GRPC response: {} I0119 02:48:14.256979 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:48:14.257086 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="a7185530-97a3-11ed-af75-002248822ac5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-a03ba1ea-dfc1-47f1-85c3-942f8ebaa8af" "latency"=28536294725 I0119 02:48:14.257120 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="a7185530-97a3-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=28536443727 I0119 02:48:14.778403 1 azuredisk_v2.go:578] "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-19T02:48:14Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="b86a2eb3-97a3-11ed-af75-002248822ac5" I0119 02:48:14.788286 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-29517166-vmss000000/status 200 OK in 9 milliseconds I0119 02:48:14.789005 1 azuredisk_v2.go:565] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="b86a2eb3-97a3-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=10640237 I0119 02:48:14.835956 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun2 by sdd under /dev/disk/azure/scsi1/ ... skipping 75 lines ... I0119 02:48:45.393287 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I0119 02:48:45.393310 1 utils.go:79] GRPC request: {} I0119 02:48:45.393356 1 utils.go:85] GRPC response: {"ready":{"value":true}} I0119 02:49:04.324074 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:49:04.324109 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:49:04.324164 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:49:04.324168 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:49:04.324195 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:49:04.325288 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:49:07.224747 1 reflector.go:559] k8s.io/client-go/informers/factory.go:150: Watch close - *v1.Node total 115 items received I0119 02:49:07.228269 1 round_trippers.go:553] GET https://10.0.0.1:443/api/v1/nodes?allowWatchBookmarks=true&resourceVersion=25469&timeout=5m1s&timeoutSeconds=301&watch=true 200 OK in 3 milliseconds I0119 02:49:11.227914 1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: Watch close - *v1beta2.AzVolume total 183 items received I0119 02:49:11.229044 1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumes?allowWatchBookmarks=true&resourceVersion=25391&timeout=8m43s&timeoutSeconds=523&watch=true 200 OK in 1 milliseconds ... skipping 5 lines ... I0119 02:49:15.393195 1 utils.go:85] GRPC response: {"ready":{"value":true}} I0119 02:49:34.326667 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:49:34.326707 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:49:34.326726 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:49:34.326733 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:49:34.326772 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:49:34.326778 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:49:41.508843 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:49:41.508912 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="c85c33cc-97a3-11ed-af75-002248822ac5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-d2cc607e-eeb0-4910-953e-da5c936183e6" "latency"=59978534963 I0119 02:49:41.508937 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="c85c33cc-97a3-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=59978663264 I0119 02:49:42.171671 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities I0119 02:49:42.171693 1 utils.go:79] GRPC request: {} I0119 02:49:42.171736 1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]} I0119 02:49:42.174564 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities ... skipping 62 lines ... I0119 02:49:46.503013 1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-d2cc607e-eeb0-4910-953e-da5c936183e6/globalmount successfully I0119 02:49:46.503029 1 utils.go:85] GRPC response: {} I0119 02:50:04.327715 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:50:04.327773 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:50:04.327785 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:50:04.327828 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:50:04.327854 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:50:04.327861 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:50:14.778973 1 azuredisk_v2.go:578] "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-19T02:50:14Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="fff0d039-97a3-11ed-af75-002248822ac5" I0119 02:50:14.787314 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-29517166-vmss000000/status 200 OK in 8 milliseconds I0119 02:50:14.787506 1 azuredisk_v2.go:565] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="fff0d039-97a3-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=8596601 I0119 02:50:15.394377 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I0119 02:50:15.394405 1 utils.go:79] GRPC request: {} I0119 02:50:15.394454 1 utils.go:85] GRPC response: {"ready":{"value":true}} I0119 02:50:16.887375 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:50:16.887429 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="ec8209cd-97a3-11ed-af75-002248822ac5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-916281c9-25e6-44cf-bddf-fa1807ff6080" "latency"=34711118848 I0119 02:50:16.887450 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="ec8209cd-97a3-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=34711239450 I0119 02:50:17.483792 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdd under /dev/disk/azure/scsi1/ I0119 02:50:17.483837 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType I0119 02:50:17.484007 1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-916281c9-25e6-44cf-bddf-fa1807ff6080/globalmount with mount options([]) I0119 02:50:17.484024 1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0]) ... skipping 55 lines ... I0119 02:50:25.482061 1 mount_linux.go:294] Unmounting /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-916281c9-25e6-44cf-bddf-fa1807ff6080/globalmount W0119 02:50:25.491992 1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-916281c9-25e6-44cf-bddf-fa1807ff6080/globalmount" is not a mountpoint, deleting I0119 02:50:25.492072 1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-916281c9-25e6-44cf-bddf-fa1807ff6080/globalmount successfully I0119 02:50:25.492084 1 utils.go:85] GRPC response: {} I0119 02:50:34.328069 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:50:34.328103 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:50:34.328182 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:50:34.328202 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:50:34.328208 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:50:34.328214 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:50:44.783129 1 azuredisk_v2.go:578] "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-19T02:50:44Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="11d31679-97a4-11ed-af75-002248822ac5" I0119 02:50:44.797975 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-29517166-vmss000000/status 200 OK in 14 milliseconds I0119 02:50:44.798374 1 azuredisk_v2.go:565] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="11d31679-97a4-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=15290780 I0119 02:50:45.392952 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I0119 02:50:45.392982 1 utils.go:79] GRPC request: {} I0119 02:50:45.393029 1 utils.go:85] GRPC response: {"ready":{"value":true}} I0119 02:51:02.877164 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:51:02.877243 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="01d2696b-97a4-11ed-af75-002248822ac5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-73ced8dc-841d-4be8-8a2d-f784ee14408e" "latency"=44942060379 I0119 02:51:02.877285 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="01d2696b-97a4-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=44942175180 I0119 02:51:04.275668 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities I0119 02:51:04.275690 1 utils.go:79] GRPC request: {} I0119 02:51:04.275734 1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]} I0119 02:51:04.328357 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync ... skipping 44 lines ... I0119 02:51:14.781198 1 azuredisk_v2.go:578] "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-19T02:51:14Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="23b46d46-97a4-11ed-af75-002248822ac5" I0119 02:51:14.788658 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-29517166-vmss000000/status 200 OK in 7 milliseconds I0119 02:51:14.788867 1 azuredisk_v2.go:565] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="23b46d46-97a4-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=7728988 I0119 02:51:15.393713 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I0119 02:51:15.393741 1 utils.go:79] GRPC request: {} I0119 02:51:15.393789 1 utils.go:85] GRPC response: {"ready":{"value":true}} I0119 02:51:18.258769 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:51:18.258834 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="1d8ce2c3-97a4-11ed-af75-002248822ac5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-916281c9-25e6-44cf-bddf-fa1807ff6080" "latency"=13803088100 I0119 02:51:18.258862 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="1d8ce2c3-97a4-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=13803199302 I0119 02:51:18.580269 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities I0119 02:51:18.580293 1 utils.go:79] GRPC request: {} I0119 02:51:18.580333 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities I0119 02:51:18.580346 1 utils.go:79] GRPC request: {} ... skipping 45 lines ... I0119 02:51:31.364321 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities I0119 02:51:31.364384 1 utils.go:79] GRPC request: {} I0119 02:51:31.364437 1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]} I0119 02:51:31.365226 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetVolumeStats I0119 02:51:31.365246 1 utils.go:79] GRPC request: {"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-iei3lcrr/providers/Microsoft.Compute/disks/pvc-73ced8dc-841d-4be8-8a2d-f784ee14408e","volume_path":"/var/lib/kubelet/pods/8ff7f947-5645-42c8-891e-174202aaa81c/volumes/kubernetes.io~csi/pvc-73ced8dc-841d-4be8-8a2d-f784ee14408e/mount"} I0119 02:51:31.365323 1 utils.go:85] GRPC response: {"usage":[{"available":5179576320,"total":5196382208,"unit":1,"used":28672},{"available":327668,"total":327680,"unit":2,"used":12}]} I0119 02:51:33.781798 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:51:33.781886 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="25f87e30-97a4-11ed-af75-002248822ac5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-e67af1c2-8b30-493a-863c-bcc303585ba8" "latency"=15199126284 I0119 02:51:33.781929 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="25f87e30-97a4-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=15199274686 I0119 02:51:33.786479 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:51:33.786554 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="25f8a3c3-97a4-11ed-af75-002248822ac5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-83b1af5f-e964-40a6-a19b-ac3a169bec6a" "latency"=15202860426 I0119 02:51:33.786593 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="25f8a3c3-97a4-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=15202976328 I0119 02:51:34.328858 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:51:34.328887 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:51:34.328893 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:51:34.328858 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync ... skipping 98 lines ... I0119 02:51:48.452803 1 utils.go:78] GRPC call: /csi.v1.Node/NodeUnstageVolume I0119 02:51:48.452822 1 utils.go:79] GRPC request: {"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-83b1af5f-e964-40a6-a19b-ac3a169bec6a","volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-iei3lcrr/providers/Microsoft.Compute/disks/pvc-83b1af5f-e964-40a6-a19b-ac3a169bec6a"} I0119 02:51:48.452883 1 nodeserver_v2.go:257] NodeUnstageVolume: unmounting /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-83b1af5f-e964-40a6-a19b-ac3a169bec6a W0119 02:51:48.453271 1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-83b1af5f-e964-40a6-a19b-ac3a169bec6a" is not a mountpoint, deleting I0119 02:51:48.453333 1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-83b1af5f-e964-40a6-a19b-ac3a169bec6a successfully I0119 02:51:48.453347 1 utils.go:85] GRPC response: {} I0119 02:51:59.113527 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:51:59.113601 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="2f6c1ae6-97a4-11ed-af75-002248822ac5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-be832c74-9614-43bd-8a0f-c853362ced80" "latency"=24673709067 I0119 02:51:59.113634 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="2f6c1ae6-97a4-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=24673808969 I0119 02:51:59.689847 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun4 by sdg under /dev/disk/azure/scsi1/ I0119 02:51:59.689905 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun4. perfProfile none accountType I0119 02:51:59.689944 1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun4 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-be832c74-9614-43bd-8a0f-c853362ced80/globalmount with mount options([]) I0119 02:51:59.689962 1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun4" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun4]) ... skipping 110 lines ... I0119 02:53:14.780482 1 azuredisk_v2.go:578] "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-19T02:53:14Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="6b3addac-97a4-11ed-af75-002248822ac5" I0119 02:53:14.796294 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-29517166-vmss000000/status 200 OK in 15 milliseconds I0119 02:53:14.796470 1 azuredisk_v2.go:565] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="6b3addac-97a4-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=16033874 I0119 02:53:15.393209 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I0119 02:53:15.393257 1 utils.go:79] GRPC request: {} I0119 02:53:15.393302 1 utils.go:85] GRPC response: {"ready":{"value":true}} I0119 02:53:34.103481 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:53:34.103556 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="68a1c2b4-97a4-11ed-af75-002248822ac5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-46b1b7a6-86b9-48bd-b392-a9294cdb3fab" "latency"=23681839323 I0119 02:53:34.103590 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="68a1c2b4-97a4-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=23681997825 I0119 02:53:34.332965 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:53:34.333081 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:53:34.333104 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:53:34.333106 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync ... skipping 64 lines ... I0119 02:53:44.781954 1 azuredisk_v2.go:578] "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-19T02:53:44Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="7d1cbb0e-97a4-11ed-af75-002248822ac5" I0119 02:53:44.789858 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-29517166-vmss000000/status 200 OK in 7 milliseconds I0119 02:53:44.790324 1 azuredisk_v2.go:565] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="7d1cbb0e-97a4-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=8401694 I0119 02:53:45.393748 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I0119 02:53:45.393775 1 utils.go:79] GRPC request: {} I0119 02:53:45.393822 1 utils.go:85] GRPC response: {"ready":{"value":true}} I0119 02:53:49.471664 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:53:49.471720 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="77e49c6a-97a4-11ed-af75-002248822ac5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-e1414edf-005c-4f28-a4bd-3e9d6b53dd93" "latency"=13446121212 I0119 02:53:49.471743 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="77e49c6a-97a4-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=13446223914 I0119 02:53:50.045635 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdd under /dev/disk/azure/scsi1/ I0119 02:53:50.045684 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType I0119 02:53:50.045711 1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun1 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-e1414edf-005c-4f28-a4bd-3e9d6b53dd93/globalmount with mount options([]) I0119 02:53:50.045721 1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun1" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun1]) ... skipping 202 lines ... I0119 02:59:44.778176 1 azuredisk_v2.go:578] "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-19T02:59:44Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="53afca9a-97a5-11ed-af75-002248822ac5" I0119 02:59:44.792213 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-29517166-vmss000000/status 200 OK in 13 milliseconds I0119 02:59:44.792815 1 azuredisk_v2.go:565] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="53afca9a-97a5-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=14686071 I0119 02:59:45.393097 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I0119 02:59:45.393124 1 utils.go:79] GRPC request: {} I0119 02:59:45.393175 1 utils.go:85] GRPC response: {"ready":{"value":true}} I0119 02:59:50.386772 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:59:50.386825 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="50f6f94d-97a5-11ed-af75-002248822ac5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-6be1ce99-4200-49de-b3ac-2c2007ba0cd1" "latency"=10175270498 I0119 02:59:50.386853 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="50f6f94d-97a5-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=10175399501 I0119 02:59:52.030160 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/ I0119 02:59:52.030213 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType StandardSSD_LRS I0119 02:59:52.030241 1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6be1ce99-4200-49de-b3ac-2c2007ba0cd1/globalmount with mount options([]) I0119 02:59:52.030251 1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0]) ... skipping 58 lines ... I0119 03:00:31.070968 1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-17464c85-4188-4cae-9c70-489c21ec4bae-k8s-agentpool1-29517166-vmss000000-attachment) I0119 03:00:34.381283 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 03:00:34.381315 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 03:00:34.381320 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 03:00:34.381354 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 03:00:34.381296 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 03:00:34.381399 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 03:00:44.781265 1 azuredisk_v2.go:578] "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-19T03:00:44Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="777389e6-97a5-11ed-af75-002248822ac5" I0119 03:00:44.800592 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-29517166-vmss000000/status 200 OK in 19 milliseconds I0119 03:00:44.816409 1 azuredisk_v2.go:565] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="777389e6-97a5-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=35171867 I0119 03:00:45.392917 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I0119 03:00:45.392940 1 utils.go:79] GRPC request: {} I0119 03:00:45.392980 1 utils.go:85] GRPC response: {"ready":{"value":true}} I0119 03:01:02.528801 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 03:01:02.528871 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="6f4784a0-97a5-11ed-af75-002248822ac5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-17464c85-4188-4cae-9c70-489c21ec4bae" "latency"=31457841795 I0119 03:01:02.528894 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="6f4784a0-97a5-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=31457942096 I0119 03:01:03.106271 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdd under /dev/disk/azure/scsi1/ I0119 03:01:03.106320 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType StandardSSD_LRS I0119 03:01:03.106350 1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun1 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-17464c85-4188-4cae-9c70-489c21ec4bae/globalmount with mount options([]) I0119 03:01:03.106362 1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun1" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun1]) ... skipping 45 lines ... I0119 03:01:14.779004 1 azuredisk_v2.go:578] "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-19T03:01:14Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="8954d36d-97a5-11ed-af75-002248822ac5" I0119 03:01:14.788257 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-29517166-vmss000000/status 200 OK in 9 milliseconds I0119 03:01:14.788467 1 azuredisk_v2.go:565] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="8954d36d-97a5-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=9516691 I0119 03:01:15.393077 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I0119 03:01:15.393102 1 utils.go:79] GRPC request: {} I0119 03:01:15.393151 1 utils.go:85] GRPC response: {"ready":{"value":true}} I0119 03:01:23.147752 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 03:01:23.147813 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000000" "disk.csi.azure.com/request-id"="885f60d9-97a5-11ed-af75-002248822ac5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-b2b7a71c-c960-4149-be30-a1f31f981d42" "latency"=9977347809 I0119 03:01:23.147838 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="885f60d9-97a5-11ed-af75-002248822ac5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9977473012 I0119 03:01:23.722293 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun2 by sde under /dev/disk/azure/scsi1/ I0119 03:01:23.722344 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun2. perfProfile none accountType StandardSSD_LRS I0119 03:01:23.722391 1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun2 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-b2b7a71c-c960-4149-be30-a1f31f981d42/globalmount with mount options([]) I0119 03:01:23.722401 1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun2" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun2]) ... skipping 152 lines ... I0119 02:05:40.818301 1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azdrivernodes?allowWatchBookmarks=true&resourceVersion=2808&timeout=6m4s&timeoutSeconds=364&watch=true 200 OK in 1 milliseconds I0119 02:05:40.821334 1 round_trippers.go:553] GET https://10.0.0.1:443/apis/apiextensions.k8s.io/v1/customresourcedefinitions?allowWatchBookmarks=true&resourceVersion=2543&timeout=6m12s&timeoutSeconds=372&watch=true 200 OK in 2 milliseconds I0119 02:05:40.845674 1 shared_informer.go:303] caches populated I0119 02:05:40.845729 1 azuredisk_v2.go:225] driver userAgent: test.csi.azure.com/latest-v2-ec131e57ca07be2e17dba65bd8a030b3e7a9034d e2e-test I0119 02:05:40.845749 1 azure_disk_utils.go:564] reading cloud config from secret kube-system/azure-cloud-provider I0119 02:05:40.852293 1 round_trippers.go:553] GET https://10.0.0.1:443/api/v1/namespaces/kube-system/secrets/azure-cloud-provider 404 Not Found in 6 milliseconds I0119 02:05:40.852503 1 azure_disk_utils.go:571] InitializeCloudFromSecret: failed to get cloud config from secret kube-system/azure-cloud-provider: failed to get secret kube-system/azure-cloud-provider: secrets "azure-cloud-provider" not found I0119 02:05:40.852519 1 azure_disk_utils.go:576] could not read cloud config from secret kube-system/azure-cloud-provider I0119 02:05:40.852524 1 azure_disk_utils.go:586] use default AZURE_CREDENTIAL_FILE env var: /etc/kubernetes/azure.json I0119 02:05:40.852544 1 azure_disk_utils.go:594] read cloud config from file: /etc/kubernetes/azure.json successfully I0119 02:05:40.854068 1 azure_auth.go:253] Using AzurePublicCloud environment I0119 02:05:40.854087 1 azure_auth.go:104] azure: using managed identity extension to retrieve access token I0119 02:05:40.854092 1 azure_auth.go:110] azure: using User Assigned MSI ID to retrieve access token ... skipping 91 lines ... I0119 02:06:00.807065 1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume I0119 02:06:00.807083 1 utils.go:79] GRPC request: {"publish_context":{"LUN":"1"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-2c47869e-7d50-4b13-8d9a-9ad81e217b5a/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-2c47869e-7d50-4b13-8d9a-9ad81e217b5a","csi.storage.k8s.io/pvc/name":"inline-volume-tester-66x8t-my-volume-0","csi.storage.k8s.io/pvc/namespace":"ephemeral-1350","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1674093937787-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-iei3lcrr/providers/Microsoft.Compute/disks/pvc-2c47869e-7d50-4b13-8d9a-9ad81e217b5a"} I0119 02:06:00.807436 1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-2c47869e-7d50-4b13-8d9a-9ad81e217b5a-k8s-agentpool1-29517166-vmss000001-attachment) I0119 02:06:00.807503 1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume I0119 02:06:00.807516 1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-3e807688-567d-47e2-97cf-186882661fdf/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-3e807688-567d-47e2-97cf-186882661fdf","csi.storage.k8s.io/pvc/name":"test.csi.azure.comjw9ks","csi.storage.k8s.io/pvc/namespace":"fsgroupchangepolicy-6594","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1674093937787-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-iei3lcrr/providers/Microsoft.Compute/disks/pvc-3e807688-567d-47e2-97cf-186882661fdf"} I0119 02:06:00.807688 1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-3e807688-567d-47e2-97cf-186882661fdf-k8s-agentpool1-29517166-vmss000001-attachment) I0119 02:06:08.373189 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:06:08.373287 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="d20ce6b0-979d-11ed-be91-002248822c79" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-2c47869e-7d50-4b13-8d9a-9ad81e217b5a" "latency"=7565754985 I0119 02:06:08.373334 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="d20ce6b0-979d-11ed-be91-002248822c79" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=7565923387 I0119 02:06:08.376734 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:06:08.376786 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="d20cf12b-979d-11ed-be91-002248822c79" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-3e807688-567d-47e2-97cf-186882661fdf" "latency"=7569047831 I0119 02:06:08.376824 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="d20cf12b-979d-11ed-be91-002248822c79" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=7569138233 I0119 02:06:08.860825 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities I0119 02:06:08.860847 1 utils.go:79] GRPC request: {} I0119 02:06:08.860884 1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]} I0119 02:06:08.865616 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities ... skipping 67 lines ... I0119 02:06:09.452911 1 mount_linux.go:183] Mounting cmd (mount) with arguments ( -o bind,remount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-3e807688-567d-47e2-97cf-186882661fdf/globalmount /var/lib/kubelet/pods/5f7e3619-b2a1-40e5-ad67-e62b6b9e434c/volumes/kubernetes.io~csi/pvc-3e807688-567d-47e2-97cf-186882661fdf/mount) I0119 02:06:09.453310 1 nodeserver_v2.go:353] NodePublishVolume: mount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-2c47869e-7d50-4b13-8d9a-9ad81e217b5a/globalmount at /var/lib/kubelet/pods/040f393a-e866-4877-8e68-35d16c486485/volumes/kubernetes.io~csi/pvc-2c47869e-7d50-4b13-8d9a-9ad81e217b5a/mount successfully I0119 02:06:09.453327 1 utils.go:85] GRPC response: {} I0119 02:06:09.453718 1 nodeserver_v2.go:353] NodePublishVolume: mount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-3e807688-567d-47e2-97cf-186882661fdf/globalmount at /var/lib/kubelet/pods/5f7e3619-b2a1-40e5-ad67-e62b6b9e434c/volumes/kubernetes.io~csi/pvc-3e807688-567d-47e2-97cf-186882661fdf/mount successfully I0119 02:06:09.453736 1 utils.go:85] GRPC response: {} I0119 02:06:10.808639 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:06:10.808728 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:06:10.816066 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:06:10.817211 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:06:10.819373 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:06:10.881764 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:06:15.185808 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I0119 02:06:15.185832 1 utils.go:79] GRPC request: {} ... skipping 14 lines ... I0119 02:06:18.123575 1 nodeserver_v2.go:257] NodeUnstageVolume: unmounting /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-3e807688-567d-47e2-97cf-186882661fdf/globalmount I0119 02:06:18.123603 1 mount_helper_common.go:99] "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-3e807688-567d-47e2-97cf-186882661fdf/globalmount" is a mountpoint, unmounting I0119 02:06:18.123616 1 mount_linux.go:294] Unmounting /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-3e807688-567d-47e2-97cf-186882661fdf/globalmount W0119 02:06:18.137715 1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-3e807688-567d-47e2-97cf-186882661fdf/globalmount" is not a mountpoint, deleting I0119 02:06:18.137775 1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-3e807688-567d-47e2-97cf-186882661fdf/globalmount successfully I0119 02:06:18.137785 1 utils.go:85] GRPC response: {} I0119 02:06:18.770807 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:06:18.770870 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="d6dab999-979d-11ed-be91-002248822c79" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-b7a7b01e-0c32-4fcf-b260-77fda1a7ba63" "latency"=9903620888 I0119 02:06:18.770893 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="d6dab999-979d-11ed-be91-002248822c79" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9903715788 I0119 02:06:19.370081 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun2 by sde under /dev/disk/azure/scsi1/ I0119 02:06:19.370136 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun2. perfProfile none accountType I0119 02:06:19.370171 1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun2 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-b7a7b01e-0c32-4fcf-b260-77fda1a7ba63/globalmount with mount options([]) I0119 02:06:19.370192 1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun2" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun2]) ... skipping 87 lines ... I0119 02:07:05.374082 1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-b7a7b01e-0c32-4fcf-b260-77fda1a7ba63/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-b7a7b01e-0c32-4fcf-b260-77fda1a7ba63","csi.storage.k8s.io/pvc/name":"test.csi.azure.com5ttw5","csi.storage.k8s.io/pvc/namespace":"fsgroupchangepolicy-3640","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1674093937787-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-iei3lcrr/providers/Microsoft.Compute/disks/pvc-b7a7b01e-0c32-4fcf-b260-77fda1a7ba63"} I0119 02:07:05.374245 1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-b7a7b01e-0c32-4fcf-b260-77fda1a7ba63-k8s-agentpool1-29517166-vmss000001-attachment) I0119 02:07:07.963656 1 azuredisk_v2.go:578] "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-19T02:07:07Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="fa141e41-979d-11ed-be91-002248822c79" I0119 02:07:07.978253 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-29517166-vmss000001/status 200 OK in 14 milliseconds I0119 02:07:07.978579 1 azuredisk_v2.go:565] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="fa141e41-979d-11ed-be91-002248822c79" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=15005282 I0119 02:07:10.809350 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:07:10.809506 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:07:10.817689 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:07:10.818830 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:07:10.820990 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:07:10.883475 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:07:15.185083 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I0119 02:07:15.185108 1 utils.go:79] GRPC request: {} I0119 02:07:15.185149 1 utils.go:85] GRPC response: {"ready":{"value":true}} I0119 02:07:37.965136 1 azuredisk_v2.go:578] "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-19T02:07:37Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="0bf5fd03-979e-11ed-be91-002248822c79" I0119 02:07:37.974515 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-29517166-vmss000001/status 200 OK in 9 milliseconds I0119 02:07:37.974747 1 azuredisk_v2.go:565] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="0bf5fd03-979e-11ed-be91-002248822c79" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=9634022 I0119 02:07:40.809950 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:07:40.810066 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:07:40.818428 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:07:40.819615 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:07:40.821788 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:07:40.884202 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:07:45.185077 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I0119 02:07:45.185106 1 utils.go:79] GRPC request: {} I0119 02:07:45.185151 1 utils.go:85] GRPC response: {"ready":{"value":true}} I0119 02:08:07.963706 1 azuredisk_v2.go:578] "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-19T02:08:07Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="1dd767cc-979e-11ed-be91-002248822c79" I0119 02:08:07.969687 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-29517166-vmss000001/status 200 OK in 5 milliseconds I0119 02:08:07.970274 1 azuredisk_v2.go:565] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="1dd767cc-979e-11ed-be91-002248822c79" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=6375667 I0119 02:08:10.810762 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:08:10.810883 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:08:10.819178 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:08:10.820290 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:08:10.822480 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:08:10.884715 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:08:13.819936 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:08:13.820020 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="f889043b-979d-11ed-be91-002248822c79" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-b7a7b01e-0c32-4fcf-b260-77fda1a7ba63" "latency"=68445706163 I0119 02:08:13.820095 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="f889043b-979d-11ed-be91-002248822c79" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=68445861065 I0119 02:08:14.877619 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities I0119 02:08:14.877646 1 utils.go:79] GRPC request: {} I0119 02:08:14.877695 1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]} I0119 02:08:14.878549 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities ... skipping 52 lines ... I0119 02:08:21.261660 1 nodeserver_v2.go:257] NodeUnstageVolume: unmounting /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-b7a7b01e-0c32-4fcf-b260-77fda1a7ba63/globalmount I0119 02:08:21.261690 1 mount_helper_common.go:99] "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-b7a7b01e-0c32-4fcf-b260-77fda1a7ba63/globalmount" is a mountpoint, unmounting I0119 02:08:21.261707 1 mount_linux.go:294] Unmounting /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-b7a7b01e-0c32-4fcf-b260-77fda1a7ba63/globalmount W0119 02:08:21.277702 1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-b7a7b01e-0c32-4fcf-b260-77fda1a7ba63/globalmount" is not a mountpoint, deleting I0119 02:08:21.277763 1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-b7a7b01e-0c32-4fcf-b260-77fda1a7ba63/globalmount successfully I0119 02:08:21.277783 1 utils.go:85] GRPC response: {} I0119 02:08:24.069586 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:08:24.069655 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="21f6b4d8-979e-11ed-be91-002248822c79" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-2749889f-4b80-4f8a-baec-6799fe90ac65" "latency"=9189900975 I0119 02:08:24.069676 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="21f6b4d8-979e-11ed-be91-002248822c79" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9190001875 I0119 02:08:24.665297 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdd under /dev/disk/azure/scsi1/ I0119 02:08:24.665347 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType I0119 02:08:24.665360 1 utils.go:85] GRPC response: {} I0119 02:08:24.671877 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities ... skipping 73 lines ... I0119 02:09:15.966712 1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-5f4c484c-d14a-4d6f-b8b2-537fbbf0b09b/globalmount","volume_capability":{"AccessType":{"Mount":{"fs_type":"xfs"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-5f4c484c-d14a-4d6f-b8b2-537fbbf0b09b","csi.storage.k8s.io/pvc/name":"test.csi.azure.compbf9c","csi.storage.k8s.io/pvc/namespace":"multivolume-9217","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1674093937787-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-iei3lcrr/providers/Microsoft.Compute/disks/pvc-5f4c484c-d14a-4d6f-b8b2-537fbbf0b09b"} I0119 02:09:15.966861 1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-5f4c484c-d14a-4d6f-b8b2-537fbbf0b09b-k8s-agentpool1-29517166-vmss000001-attachment) I0119 02:09:37.963291 1 azuredisk_v2.go:578] "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-19T02:09:37Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="537c40a9-979e-11ed-be91-002248822c79" I0119 02:09:37.970605 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-29517166-vmss000001/status 200 OK in 7 milliseconds I0119 02:09:37.971210 1 azuredisk_v2.go:565] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="537c40a9-979e-11ed-be91-002248822c79" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=7946091 I0119 02:09:40.813623 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:09:40.813738 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:09:40.813773 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:09:40.820952 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:09:40.823116 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:09:40.824255 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:09:40.886512 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:09:45.185165 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I0119 02:09:45.185190 1 utils.go:79] GRPC request: {} I0119 02:09:45.185239 1 utils.go:85] GRPC response: {"ready":{"value":true}} I0119 02:09:51.444615 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:09:51.444675 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="465fde7d-979e-11ed-be91-002248822c79" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-5f4c484c-d14a-4d6f-b8b2-537fbbf0b09b" "latency"=35477756498 I0119 02:09:51.444698 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="465fde7d-979e-11ed-be91-002248822c79" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=35477851400 I0119 02:09:51.445753 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:09:51.445816 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="465fce5b-979e-11ed-be91-002248822c79" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-295b2ad2-4b71-453f-8341-7741a43cde91" "latency"=35479286720 I0119 02:09:51.445839 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="465fce5b-979e-11ed-be91-002248822c79" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=35479407122 I0119 02:09:51.828437 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities I0119 02:09:51.828458 1 utils.go:79] GRPC request: {} I0119 02:09:51.828506 1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]} I0119 02:09:51.831772 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities ... skipping 87 lines ... I0119 02:09:59.732044 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities I0119 02:09:59.732063 1 utils.go:79] GRPC request: {} I0119 02:09:59.732093 1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]} I0119 02:09:59.733063 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetVolumeStats I0119 02:09:59.733084 1 utils.go:79] GRPC request: {"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-iei3lcrr/providers/Microsoft.Compute/disks/pvc-295b2ad2-4b71-453f-8341-7741a43cde91","volume_path":"/var/lib/kubelet/pods/e53ea653-f353-4089-9bee-e08f9c1c4357/volumes/kubernetes.io~csi/pvc-295b2ad2-4b71-453f-8341-7741a43cde91/mount"} I0119 02:09:59.733162 1 utils.go:85] GRPC response: {"usage":[{"available":5286936576,"total":5358223360,"unit":1,"used":71286784},{"available":2621436,"total":2621440,"unit":2,"used":4}]} I0119 02:10:01.730604 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:10:01.730672 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="5bc0acb1-979e-11ed-be91-002248822c79" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-3953852e-5d09-4b69-aeac-18f10ea101ae" "latency"=9897168810 I0119 02:10:01.730696 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="5bc0acb1-979e-11ed-be91-002248822c79" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9897271811 I0119 02:10:01.736590 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:10:01.736665 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="5c0d0071-979e-11ed-be91-002248822c79" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-2749889f-4b80-4f8a-baec-6799fe90ac65" "latency"=9402950726 I0119 02:10:01.736701 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="5c0d0071-979e-11ed-be91-002248822c79" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9403054427 I0119 02:10:02.355385 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sde under /dev/disk/azure/scsi1/ I0119 02:10:02.355388 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun3 by sdf under /dev/disk/azure/scsi1/ I0119 02:10:02.355439 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType I0119 02:10:02.355459 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun3. perfProfile none accountType ... skipping 247 lines ... I0119 02:10:33.008133 1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-5f4c484c-d14a-4d6f-b8b2-537fbbf0b09b/globalmount successfully I0119 02:10:33.008146 1 utils.go:85] GRPC response: {} I0119 02:10:37.963600 1 azuredisk_v2.go:578] "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-19T02:10:37Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="773f9165-979e-11ed-be91-002248822c79" I0119 02:10:37.968493 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-29517166-vmss000001/status 200 OK in 4 milliseconds I0119 02:10:37.968737 1 azuredisk_v2.go:565] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="773f9165-979e-11ed-be91-002248822c79" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=5197750 I0119 02:10:40.816217 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:10:40.816351 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:10:40.822595 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:10:40.823682 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:10:40.824787 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:10:40.887038 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:10:45.185234 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I0119 02:10:45.185258 1 utils.go:79] GRPC request: {} I0119 02:10:45.185295 1 utils.go:85] GRPC response: {"ready":{"value":true}} I0119 02:10:49.818565 1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: Watch close - *v1beta2.AzVolume total 96 items received I0119 02:10:49.820359 1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumes?allowWatchBookmarks=true&resourceVersion=5058&timeout=6m0s&timeoutSeconds=360&watch=true 200 OK in 1 milliseconds I0119 02:10:52.971815 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:10:52.971887 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="6aa87887-979e-11ed-be91-002248822c79" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-ab32d138-b163-4210-969f-9df85033cdb3" "latency"=36131158244 I0119 02:10:52.971915 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="6aa87887-979e-11ed-be91-002248822c79" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=36131284945 I0119 02:10:53.565309 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun4 by sdg under /dev/disk/azure/scsi1/ I0119 02:10:53.565358 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun4. perfProfile none accountType I0119 02:10:53.565369 1 utils.go:85] GRPC response: {} I0119 02:10:53.574207 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities ... skipping 88 lines ... I0119 02:11:48.931564 1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]} I0119 02:11:48.932260 1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume I0119 02:11:48.932275 1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-b698debb-c272-445f-b637-ac08dc754195/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-b698debb-c272-445f-b637-ac08dc754195","csi.storage.k8s.io/pvc/name":"test.csi.azure.comx5gb6","csi.storage.k8s.io/pvc/namespace":"fsgroupchangepolicy-8032","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1674093937787-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-iei3lcrr/providers/Microsoft.Compute/disks/pvc-b698debb-c272-445f-b637-ac08dc754195"} I0119 02:11:48.932469 1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-b698debb-c272-445f-b637-ac08dc754195-k8s-agentpool1-29517166-vmss000001-attachment) I0119 02:11:52.822723 1 reflector.go:559] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: Watch close - *v1.CustomResourceDefinition total 0 items received I0119 02:11:52.825181 1 round_trippers.go:553] GET https://10.0.0.1:443/apis/apiextensions.k8s.io/v1/customresourcedefinitions?allowWatchBookmarks=true&resourceVersion=2543&timeout=5m36s&timeoutSeconds=336&watch=true 200 OK in 2 milliseconds I0119 02:11:56.900889 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:11:56.900979 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="a18c918a-979e-11ed-be91-002248822c79" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-b698debb-c272-445f-b637-ac08dc754195" "latency"=7968429334 I0119 02:11:56.901015 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="a18c918a-979e-11ed-be91-002248822c79" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=7968550435 I0119 02:11:58.595212 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/ I0119 02:11:58.595269 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType I0119 02:11:58.595309 1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-b698debb-c272-445f-b637-ac08dc754195/globalmount with mount options([]) I0119 02:11:58.595327 1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0]) ... skipping 131 lines ... I0119 02:12:26.173312 1 nodeserver_v2.go:257] NodeUnstageVolume: unmounting /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-b698debb-c272-445f-b637-ac08dc754195/globalmount I0119 02:12:26.173350 1 mount_helper_common.go:99] "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-b698debb-c272-445f-b637-ac08dc754195/globalmount" is a mountpoint, unmounting I0119 02:12:26.173360 1 mount_linux.go:294] Unmounting /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-b698debb-c272-445f-b637-ac08dc754195/globalmount W0119 02:12:26.188447 1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-b698debb-c272-445f-b637-ac08dc754195/globalmount" is not a mountpoint, deleting I0119 02:12:26.188509 1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-b698debb-c272-445f-b637-ac08dc754195/globalmount successfully I0119 02:12:26.188520 1 utils.go:85] GRPC response: {} I0119 02:12:29.593000 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:12:29.593092 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="b41673d3-979e-11ed-be91-002248822c79" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-1460c1ff-edba-42e4-b308-6c882564674b" "latency"=9557908970 I0119 02:12:29.593117 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="b41673d3-979e-11ed-be91-002248822c79" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9558036172 I0119 02:12:30.208358 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdd under /dev/disk/azure/scsi1/ I0119 02:12:30.208405 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType I0119 02:12:30.208609 1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun1 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-1460c1ff-edba-42e4-b308-6c882564674b/globalmount with mount options([]) I0119 02:12:30.208628 1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun1" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun1]) ... skipping 69 lines ... I0119 02:12:40.575004 1 utils.go:79] GRPC request: {} I0119 02:12:40.575038 1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]} I0119 02:12:40.575810 1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume I0119 02:12:40.575826 1 utils.go:79] GRPC request: {"publish_context":{"LUN":"2"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-da7ad20b-e1f4-4037-bd0f-67e616879e07/globalmount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-da7ad20b-e1f4-4037-bd0f-67e616879e07","csi.storage.k8s.io/pvc/name":"test.csi.azure.com94p9h","csi.storage.k8s.io/pvc/namespace":"multivolume-1667","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1674093937787-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-iei3lcrr/providers/Microsoft.Compute/disks/pvc-da7ad20b-e1f4-4037-bd0f-67e616879e07"} I0119 02:12:40.575994 1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-da7ad20b-e1f4-4037-bd0f-67e616879e07-k8s-agentpool1-29517166-vmss000001-attachment) I0119 02:12:40.822598 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:12:40.822721 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:12:40.822749 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:12:40.823785 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:12:40.824875 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:12:40.825960 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:12:40.888153 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:12:45.185678 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I0119 02:12:45.185698 1 utils.go:79] GRPC request: {} I0119 02:12:45.185740 1 utils.go:85] GRPC response: {"ready":{"value":true}} I0119 02:13:07.963284 1 azuredisk_v2.go:578] "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-19T02:13:07Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="d0a7b575-979e-11ed-be91-002248822c79" I0119 02:13:07.975104 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-29517166-vmss000001/status 200 OK in 11 milliseconds I0119 02:13:07.975475 1 azuredisk_v2.go:565] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="d0a7b575-979e-11ed-be91-002248822c79" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=12239817 I0119 02:13:10.823522 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:13:10.823646 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:13:10.823661 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:13:10.824721 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:13:10.825808 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:13:10.826894 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:13:10.889146 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:13:15.185163 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I0119 02:13:15.185185 1 utils.go:79] GRPC request: {} I0119 02:13:15.185233 1 utils.go:85] GRPC response: {"ready":{"value":true}} I0119 02:13:17.141991 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:13:17.142061 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="c054bea9-979e-11ed-be91-002248822c79" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-da7ad20b-e1f4-4037-bd0f-67e616879e07" "latency"=36566005286 I0119 02:13:17.142084 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="c054bea9-979e-11ed-be91-002248822c79" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=36566105287 I0119 02:13:17.150189 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:13:17.150248 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="c054936e-979e-11ed-be91-002248822c79" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-da1b00f3-5874-46e2-90ff-f463da550578" "latency"=36575280074 I0119 02:13:17.150268 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="c054936e-979e-11ed-be91-002248822c79" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=36575399276 I0119 02:13:17.732308 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun3 by sdc under /dev/disk/azure/scsi1/ I0119 02:13:17.732309 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun2 by sdd under /dev/disk/azure/scsi1/ I0119 02:13:17.732362 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun3. perfProfile none accountType I0119 02:13:17.732378 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun2. perfProfile none accountType ... skipping 120 lines ... I0119 02:13:31.850718 1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-da1b00f3-5874-46e2-90ff-f463da550578/globalmount successfully I0119 02:13:31.850733 1 utils.go:85] GRPC response: {} I0119 02:13:37.965152 1 azuredisk_v2.go:578] "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-19T02:13:37Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="e289a13e-979e-11ed-be91-002248822c79" I0119 02:13:37.971542 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-29517166-vmss000001/status 200 OK in 6 milliseconds I0119 02:13:37.972107 1 azuredisk_v2.go:565] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="e289a13e-979e-11ed-be91-002248822c79" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=7005470 I0119 02:13:40.824190 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:13:40.824303 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:13:40.824320 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:13:40.825259 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:13:40.826356 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:13:40.827436 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:13:40.889687 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:13:42.474567 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:13:42.474639 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="d6ab6518-979e-11ed-be91-002248822c79" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-86b77127-60db-4745-b51c-d8d3bcfc3938" "latency"=24420828064 I0119 02:13:42.474663 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="d6ab6518-979e-11ed-be91-002248822c79" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=24420939764 I0119 02:13:42.478936 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:13:42.478996 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="d900968f-979e-11ed-be91-002248822c79" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-3c40689a-c445-4fc0-857a-ca8ec95a985d" "latency"=20511413402 I0119 02:13:42.479029 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="d900968f-979e-11ed-be91-002248822c79" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=20511534604 I0119 02:13:43.088414 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdf under /dev/disk/azure/scsi1/ I0119 02:13:43.088473 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType I0119 02:13:43.088516 1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun1 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-3c40689a-c445-4fc0-857a-ca8ec95a985d/globalmount with mount options([]) I0119 02:13:43.088529 1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun1" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun1]) ... skipping 109 lines ... I0119 02:13:59.733363 1 utils.go:79] GRPC request: {"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-iei3lcrr/providers/Microsoft.Compute/disks/pvc-86b77127-60db-4745-b51c-d8d3bcfc3938","volume_path":"/var/lib/kubelet/pods/8c51dcaf-5d43-4e64-9489-3dbd58b9e265/volumes/kubernetes.io~csi/pvc-86b77127-60db-4745-b51c-d8d3bcfc3938/mount"} I0119 02:13:59.733451 1 utils.go:85] GRPC response: {"usage":[{"available":5179580416,"total":5196382208,"unit":1,"used":24576},{"available":327669,"total":327680,"unit":2,"used":11}]} I0119 02:14:07.964563 1 azuredisk_v2.go:578] "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-19T02:14:07Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="f46b2c3e-979e-11ed-be91-002248822c79" I0119 02:14:07.984057 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-29517166-vmss000001/status 200 OK in 19 milliseconds I0119 02:14:07.984276 1 azuredisk_v2.go:565] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="f46b2c3e-979e-11ed-be91-002248822c79" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=19773203 I0119 02:14:10.825259 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:14:10.825373 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:14:10.825397 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:14:10.825513 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:14:10.826607 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:14:10.827690 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:14:10.889910 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:14:15.185634 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I0119 02:14:15.185661 1 utils.go:79] GRPC request: {} I0119 02:14:15.185707 1 utils.go:85] GRPC response: {"ready":{"value":true}} I0119 02:14:37.964570 1 azuredisk_v2.go:578] "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-19T02:14:37Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="064cd04a-979f-11ed-be91-002248822c79" I0119 02:14:37.970151 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-29517166-vmss000001/status 200 OK in 5 milliseconds I0119 02:14:37.971394 1 azuredisk_v2.go:565] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="064cd04a-979f-11ed-be91-002248822c79" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=6870052 I0119 02:14:40.825646 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:14:40.825734 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:14:40.825732 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:14:40.825757 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:14:40.826966 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:14:40.828050 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:14:40.890306 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:14:44.090737 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:14:44.090806 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="e7a6deb6-979e-11ed-be91-002248822c79" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-6338796d-185e-4c9a-a6c3-e3d4496b6d84" "latency"=57545387171 I0119 02:14:44.090831 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="e7a6deb6-979e-11ed-be91-002248822c79" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=57545495072 I0119 02:14:44.093981 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:14:44.094053 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="e7c5b6d3-979e-11ed-be91-002248822c79" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-caf4d040-a20b-42e3-bd47-23095f505e7b" "latency"=57346493913 I0119 02:14:44.094095 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="e7c5b6d3-979e-11ed-be91-002248822c79" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=57346609915 I0119 02:14:44.698296 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun4 by sdd under /dev/disk/azure/scsi1/ I0119 02:14:44.698344 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun4. perfProfile none accountType I0119 02:14:44.698384 1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun4 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6338796d-185e-4c9a-a6c3-e3d4496b6d84/globalmount with mount options([nouuid]) I0119 02:14:44.698403 1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun4" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun4]) ... skipping 64 lines ... I0119 02:14:53.839651 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities I0119 02:14:53.839668 1 utils.go:79] GRPC request: {} I0119 02:14:53.839700 1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]} I0119 02:14:53.840440 1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume I0119 02:14:53.840454 1 utils.go:79] GRPC request: {"publish_context":{"LUN":"1"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-f1793e44-de7f-4ef6-9a12-2879db7bf678","volume_capability":{"AccessType":{"Block":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-f1793e44-de7f-4ef6-9a12-2879db7bf678","csi.storage.k8s.io/pvc/name":"pvc-q5rsf","csi.storage.k8s.io/pvc/namespace":"provisioning-8201","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1674093937787-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-iei3lcrr/providers/Microsoft.Compute/disks/pvc-f1793e44-de7f-4ef6-9a12-2879db7bf678"} I0119 02:14:53.840613 1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-f1793e44-de7f-4ef6-9a12-2879db7bf678-k8s-agentpool1-29517166-vmss000001-attachment) I0119 02:14:59.733165 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:14:59.733234 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="0fc34fe8-979f-11ed-be91-002248822c79" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-f1793e44-de7f-4ef6-9a12-2879db7bf678" "latency"=5892548004 I0119 02:14:59.733264 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="0fc34fe8-979f-11ed-be91-002248822c79" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=5892661705 I0119 02:15:00.390117 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdf under /dev/disk/azure/scsi1/ I0119 02:15:00.390162 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType I0119 02:15:00.390175 1 utils.go:85] GRPC response: {} I0119 02:15:00.395087 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities ... skipping 146 lines ... I0119 02:16:37.969525 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-29517166-vmss000001/status 200 OK in 5 milliseconds I0119 02:16:37.969774 1 azuredisk_v2.go:565] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="4dd33466-979f-11ed-be91-002248822c79" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=6231963 I0119 02:16:40.831014 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:16:40.831089 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:16:40.831105 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:16:40.831174 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:16:40.831311 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:16:40.831382 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:16:40.891412 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:16:45.185692 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I0119 02:16:45.185717 1 utils.go:79] GRPC request: {} I0119 02:16:45.185767 1 utils.go:85] GRPC response: {"ready":{"value":true}} I0119 02:16:49.819897 1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: Watch close - *v1beta2.AzVolume total 180 items received I0119 02:16:49.822678 1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumes?allowWatchBookmarks=true&resourceVersion=9439&timeout=8m6s&timeoutSeconds=486&watch=true 200 OK in 2 milliseconds I0119 02:16:52.594816 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:16:52.594935 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="46aad032-979f-11ed-be91-002248822c79" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-9cf44a14-862a-43ea-b75f-c2e7d9e676e2" "latency"=26640094867 I0119 02:16:52.595002 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="46aad032-979f-11ed-be91-002248822c79" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=26640243768 I0119 02:16:52.595375 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:16:52.595432 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="46aacd5a-979f-11ed-be91-002248822c79" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-ed3848af-046d-48ff-acf1-8d3d335f9e87" "latency"=26640688472 I0119 02:16:52.595473 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="46aacd5a-979f-11ed-be91-002248822c79" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=26640806173 I0119 02:16:53.188280 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun2 by sdd under /dev/disk/azure/scsi1/ I0119 02:16:53.188334 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun2. perfProfile none accountType I0119 02:16:53.188344 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdc under /dev/disk/azure/scsi1/ I0119 02:16:53.188348 1 utils.go:85] GRPC response: {} ... skipping 77 lines ... W0119 02:17:07.742410 1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-ed3848af-046d-48ff-acf1-8d3d335f9e87" is not a mountpoint, deleting I0119 02:17:07.742471 1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-ed3848af-046d-48ff-acf1-8d3d335f9e87 successfully W0119 02:17:07.742474 1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-9cf44a14-862a-43ea-b75f-c2e7d9e676e2" is not a mountpoint, deleting I0119 02:17:07.742480 1 utils.go:85] GRPC response: {} I0119 02:17:07.742535 1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-9cf44a14-862a-43ea-b75f-c2e7d9e676e2 successfully I0119 02:17:07.742554 1 utils.go:85] GRPC response: {} I0119 02:17:07.895150 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:17:07.895222 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="58d806f8-979f-11ed-be91-002248822c79" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-d2f050dd-5ba5-4c0d-ba73-6b0e46e54b8a" "latency"=11445086296 I0119 02:17:07.895251 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="58d806f8-979f-11ed-be91-002248822c79" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=11445214797 I0119 02:17:07.963961 1 azuredisk_v2.go:578] "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-19T02:17:07Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="5fb4e797-979f-11ed-be91-002248822c79" I0119 02:17:07.969216 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-29517166-vmss000001/status 200 OK in 5 milliseconds I0119 02:17:07.969468 1 azuredisk_v2.go:565] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="5fb4e797-979f-11ed-be91-002248822c79" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=5542853 I0119 02:17:08.498010 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sde under /dev/disk/azure/scsi1/ ... skipping 167 lines ... I0119 02:18:18.519938 1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-7d8237a4-f85f-40c3-a328-ae27db8d8607/globalmount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-7d8237a4-f85f-40c3-a328-ae27db8d8607","csi.storage.k8s.io/pvc/name":"test.csi.azure.compfc6n","csi.storage.k8s.io/pvc/namespace":"multivolume-968","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1674093937787-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-iei3lcrr/providers/Microsoft.Compute/disks/pvc-7d8237a4-f85f-40c3-a328-ae27db8d8607"} I0119 02:18:18.520037 1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-4d3e1a2c-d8ba-4c91-8f8e-7cbe8951e2dd-k8s-agentpool1-29517166-vmss000001-attachment) I0119 02:18:18.520090 1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-7d8237a4-f85f-40c3-a328-ae27db8d8607-k8s-agentpool1-29517166-vmss000001-attachment) I0119 02:18:37.963981 1 azuredisk_v2.go:578] "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-19T02:18:37Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="9559d12b-979f-11ed-be91-002248822c79" I0119 02:18:37.971075 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-29517166-vmss000001/status 200 OK in 6 milliseconds I0119 02:18:37.971266 1 azuredisk_v2.go:565] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="9559d12b-979f-11ed-be91-002248822c79" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=7328371 I0119 02:18:38.811796 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:18:38.811881 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="89c2ec05-979f-11ed-be91-002248822c79" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-7d8237a4-f85f-40c3-a328-ae27db8d8607" "latency"=20291715282 I0119 02:18:38.811922 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="89c2ec05-979f-11ed-be91-002248822c79" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=20291838084 I0119 02:18:38.819280 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:18:38.819366 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="89c2e99a-979f-11ed-be91-002248822c79" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-4d3e1a2c-d8ba-4c91-8f8e-7cbe8951e2dd" "latency"=20299266156 I0119 02:18:38.819401 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="89c2e99a-979f-11ed-be91-002248822c79" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=20299385556 I0119 02:18:39.400486 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdd under /dev/disk/azure/scsi1/ I0119 02:18:39.400540 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType I0119 02:18:39.400575 1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun1 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-4d3e1a2c-d8ba-4c91-8f8e-7cbe8951e2dd/globalmount with mount options([]) I0119 02:18:39.400587 1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun1" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun1]) ... skipping 70 lines ... I0119 02:18:40.383032 1 utils.go:79] GRPC request: {"publish_context":{"LUN":"2"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-ddcc725e-dc48-442b-ac7e-fc3144daf546/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-ddcc725e-dc48-442b-ac7e-fc3144daf546","csi.storage.k8s.io/pvc/name":"inline-volume-tester2-9b2b5-my-volume-0","csi.storage.k8s.io/pvc/namespace":"ephemeral-4718","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1674093937787-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-iei3lcrr/providers/Microsoft.Compute/disks/pvc-ddcc725e-dc48-442b-ac7e-fc3144daf546"} I0119 02:18:40.383185 1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-ddcc725e-dc48-442b-ac7e-fc3144daf546-k8s-agentpool1-29517166-vmss000001-attachment) I0119 02:18:40.834882 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:18:40.835113 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:18:40.835136 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:18:40.835145 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:18:40.835247 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:18:40.893548 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:18:45.185284 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I0119 02:18:45.185309 1 utils.go:79] GRPC request: {} I0119 02:18:45.185352 1 utils.go:85] GRPC response: {"ready":{"value":true}} I0119 02:18:52.555652 1 utils.go:78] GRPC call: /csi.v1.Node/NodeUnpublishVolume I0119 02:18:52.555689 1 utils.go:79] GRPC request: {"target_path":"/var/lib/kubelet/pods/7b59a1d9-378c-4445-ba80-8f02ddfbfab7/volumes/kubernetes.io~csi/pvc-7d8237a4-f85f-40c3-a328-ae27db8d8607/mount","volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-iei3lcrr/providers/Microsoft.Compute/disks/pvc-7d8237a4-f85f-40c3-a328-ae27db8d8607"} ... skipping 39 lines ... I0119 02:19:08.819443 1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: Watch close - *v1beta2.AzDriverNode total 67 items received I0119 02:19:08.821517 1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azdrivernodes?allowWatchBookmarks=true&resourceVersion=10986&timeout=5m42s&timeoutSeconds=342&watch=true 200 OK in 1 milliseconds I0119 02:19:10.836669 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:19:10.836744 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:19:10.836768 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:19:10.836786 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:19:10.836842 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:19:10.894065 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:19:14.306156 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:19:14.306227 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="96caf6cf-979f-11ed-be91-002248822c79" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-ddcc725e-dc48-442b-ac7e-fc3144daf546" "latency"=33922974665 I0119 02:19:14.306252 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="96caf6cf-979f-11ed-be91-002248822c79" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=33923087065 I0119 02:19:14.909275 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun2 by sde under /dev/disk/azure/scsi1/ I0119 02:19:14.909320 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun2. perfProfile none accountType I0119 02:19:14.909358 1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun2 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-ddcc725e-dc48-442b-ac7e-fc3144daf546/globalmount with mount options([]) I0119 02:19:14.909375 1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun2" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun2]) ... skipping 84 lines ... I0119 02:20:10.707572 1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume I0119 02:20:10.707585 1 utils.go:79] GRPC request: {"publish_context":{"LUN":"4"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-ae53b3d5-cd92-44b2-9e43-e3ae283bbf80/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-ae53b3d5-cd92-44b2-9e43-e3ae283bbf80","csi.storage.k8s.io/pvc/name":"test.csi.azure.comt2v4z","csi.storage.k8s.io/pvc/namespace":"multivolume-2422","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1674093937787-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-iei3lcrr/providers/Microsoft.Compute/disks/pvc-ae53b3d5-cd92-44b2-9e43-e3ae283bbf80"} I0119 02:20:10.707739 1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-ae53b3d5-cd92-44b2-9e43-e3ae283bbf80-k8s-agentpool1-29517166-vmss000001-attachment) I0119 02:20:10.839896 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:20:10.839961 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:20:10.839981 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:20:10.840060 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:20:10.840093 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:20:10.840097 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:20:10.895269 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:20:15.185229 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I0119 02:20:15.185255 1 utils.go:79] GRPC request: {} I0119 02:20:15.185305 1 utils.go:85] GRPC response: {"ready":{"value":true}} I0119 02:20:37.962974 1 azuredisk_v2.go:578] "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-19T02:20:37Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="dce03651-979f-11ed-be91-002248822c79" I0119 02:20:37.969483 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-29517166-vmss000001/status 200 OK in 6 milliseconds I0119 02:20:37.971441 1 azuredisk_v2.go:565] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="dce03651-979f-11ed-be91-002248822c79" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=8499350 I0119 02:20:40.842387 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:20:40.842428 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:20:40.842489 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:20:40.842510 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:20:40.842604 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:20:40.842650 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:20:40.895817 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:20:45.185595 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I0119 02:20:45.185622 1 utils.go:79] GRPC request: {} I0119 02:20:45.185663 1 utils.go:85] GRPC response: {"ready":{"value":true}} I0119 02:21:07.964570 1 azuredisk_v2.go:578] "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-19T02:21:07Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="eec2174b-979f-11ed-be91-002248822c79" I0119 02:21:07.974949 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-29517166-vmss000001/status 200 OK in 10 milliseconds I0119 02:21:07.975691 1 azuredisk_v2.go:565] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="eec2174b-979f-11ed-be91-002248822c79" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=11154797 I0119 02:21:10.842692 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:21:10.842742 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:21:10.842776 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:21:10.842803 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:21:10.842816 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:21:10.842891 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:21:10.896114 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:21:15.185740 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I0119 02:21:15.185759 1 utils.go:79] GRPC request: {} I0119 02:21:15.185796 1 utils.go:85] GRPC response: {"ready":{"value":true}} I0119 02:21:25.818032 1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: Watch close - *v1beta2.AzVolumeAttachment total 340 items received I0119 02:21:25.820556 1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumeattachments?allowWatchBookmarks=true&resourceVersion=11879&timeout=7m49s&timeoutSeconds=469&watch=true 200 OK in 2 milliseconds I0119 02:21:37.963259 1 azuredisk_v2.go:578] "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-19T02:21:37Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="00a38781-97a0-11ed-be91-002248822c79" I0119 02:21:37.969819 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-29517166-vmss000001/status 200 OK in 6 milliseconds I0119 02:21:37.969963 1 azuredisk_v2.go:565] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="00a38781-97a0-11ed-be91-002248822c79" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=6754408 I0119 02:21:40.844362 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:21:40.844459 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:21:40.844495 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:21:40.844522 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:21:40.844584 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:21:40.844621 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:21:40.896775 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:21:45.185907 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I0119 02:21:45.185932 1 utils.go:79] GRPC request: {} I0119 02:21:45.185981 1 utils.go:85] GRPC response: {"ready":{"value":true}} I0119 02:21:52.105646 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:21:52.105711 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="cca1660b-979f-11ed-be91-002248822c79" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-ae53b3d5-cd92-44b2-9e43-e3ae283bbf80" "latency"=101397914844 I0119 02:21:52.105737 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="cca1660b-979f-11ed-be91-002248822c79" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=101398009146 I0119 02:21:52.115313 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:21:52.115367 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="cca12593-979f-11ed-be91-002248822c79" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-8a79af9e-176f-4ba8-9546-3d1ce477dbba" "latency"=101409202427 I0119 02:21:52.115393 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="cca12593-979f-11ed-be91-002248822c79" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=101409314628 I0119 02:21:52.695043 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun3 by sdc under /dev/disk/azure/scsi1/ I0119 02:21:52.695093 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun3. perfProfile none accountType I0119 02:21:52.695105 1 utils.go:85] GRPC response: {} I0119 02:21:52.699414 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities ... skipping 236 lines ... I0119 02:24:07.972994 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-29517166-vmss000001/status 200 OK in 8 milliseconds I0119 02:24:07.973479 1 azuredisk_v2.go:565] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="5a0be142-97a0-11ed-be91-002248822c79" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=9159109 I0119 02:24:10.849165 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:24:10.849221 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:24:10.849239 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:24:10.849279 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:24:10.849344 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:24:10.898655 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:24:15.185303 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I0119 02:24:15.185324 1 utils.go:79] GRPC request: {} I0119 02:24:15.185369 1 utils.go:85] GRPC response: {"ready":{"value":true}} I0119 02:24:19.988070 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:24:19.988141 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="58a8b524-97a0-11ed-be91-002248822c79" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-34d33a50-afb3-4038-b041-5307f70e60d9" "latency"=14351406318 I0119 02:24:19.988163 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="58a8b524-97a0-11ed-be91-002248822c79" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=14351514319 I0119 02:24:21.664987 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/ I0119 02:24:21.665055 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType I0119 02:24:21.665068 1 utils.go:85] GRPC response: {} I0119 02:24:21.669266 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities ... skipping 57 lines ... I0119 02:25:27.105205 1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume I0119 02:25:27.105218 1 utils.go:79] GRPC request: {"publish_context":{"LUN":"1"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-bde92d85-04ce-4ddc-87e9-64f2e59b7cc7/globalmount","volume_capability":{"AccessType":{"Mount":{"fs_type":"xfs"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-bde92d85-04ce-4ddc-87e9-64f2e59b7cc7","csi.storage.k8s.io/pvc/name":"test.csi.azure.comlxw5g","csi.storage.k8s.io/pvc/namespace":"multivolume-8831","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1674093937787-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-iei3lcrr/providers/Microsoft.Compute/disks/pvc-bde92d85-04ce-4ddc-87e9-64f2e59b7cc7"} I0119 02:25:27.105372 1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-bde92d85-04ce-4ddc-87e9-64f2e59b7cc7-k8s-agentpool1-29517166-vmss000001-attachment) I0119 02:25:37.964143 1 azuredisk_v2.go:578] "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-19T02:25:37Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="8fb0c1b6-97a0-11ed-be91-002248822c79" I0119 02:25:37.974688 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-29517166-vmss000001/status 200 OK in 10 milliseconds I0119 02:25:37.977094 1 azuredisk_v2.go:565] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="8fb0c1b6-97a0-11ed-be91-002248822c79" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=12992457 I0119 02:25:39.601723 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:25:39.601802 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="8937d838-97a0-11ed-be91-002248822c79" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-bde92d85-04ce-4ddc-87e9-64f2e59b7cc7" "latency"=12496353538 I0119 02:25:39.601825 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="8937d838-97a0-11ed-be91-002248822c79" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=12496478039 I0119 02:25:40.200593 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdd under /dev/disk/azure/scsi1/ I0119 02:25:40.200643 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType I0119 02:25:40.200675 1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun1 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-bde92d85-04ce-4ddc-87e9-64f2e59b7cc7/globalmount with mount options([nouuid]) I0119 02:25:40.200690 1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun1" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun1]) ... skipping 127 lines ... I0119 02:26:39.591656 1 utils.go:79] GRPC request: {"publish_context":{"LUN":"1"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6d2fb453-339c-4a6b-9cc6-74317426ad51/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-6d2fb453-339c-4a6b-9cc6-74317426ad51","csi.storage.k8s.io/pvc/name":"pvc-lkgkn","csi.storage.k8s.io/pvc/namespace":"provisioning-8257","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1674093937787-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-iei3lcrr/providers/Microsoft.Compute/disks/pvc-6d2fb453-339c-4a6b-9cc6-74317426ad51"} I0119 02:26:39.591827 1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-6d2fb453-339c-4a6b-9cc6-74317426ad51-k8s-agentpool1-29517166-vmss000001-attachment) I0119 02:26:40.853112 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:26:40.853137 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:26:40.853144 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:26:40.853155 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:26:40.853315 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:26:40.853352 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:26:40.900745 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:26:45.185298 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I0119 02:26:45.185324 1 utils.go:79] GRPC request: {} I0119 02:26:45.185375 1 utils.go:85] GRPC response: {"ready":{"value":true}} I0119 02:26:54.743201 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:26:54.743272 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="b46c6660-97a0-11ed-be91-002248822c79" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-6d2fb453-339c-4a6b-9cc6-74317426ad51" "latency"=15151384439 I0119 02:26:54.743295 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="b46c6660-97a0-11ed-be91-002248822c79" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=15151491839 I0119 02:26:54.748142 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:26:54.748202 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="b46c56a2-97a0-11ed-be91-002248822c79" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-5cdc2327-4b8b-4a75-b49f-5f7cb92ca7db" "latency"=15156708365 I0119 02:26:54.748228 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="b46c56a2-97a0-11ed-be91-002248822c79" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=15156829566 I0119 02:26:55.352163 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun2 by sde under /dev/disk/azure/scsi1/ I0119 02:26:55.352214 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun2. perfProfile none accountType I0119 02:26:55.352242 1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun2 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-5cdc2327-4b8b-4a75-b49f-5f7cb92ca7db/globalmount with mount options([]) I0119 02:26:55.352251 1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun2" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun2]) ... skipping 190 lines ... I0119 02:29:07.974573 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-29517166-vmss000001/status 200 OK in 9 milliseconds I0119 02:29:07.974784 1 azuredisk_v2.go:565] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="0cdc49a1-97a1-11ed-be91-002248822c79" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=10212207 I0119 02:29:10.860576 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:29:10.860603 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:29:10.860622 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:29:10.860611 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:29:10.860726 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:29:10.903937 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:29:14.822017 1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: Watch close - *v1beta2.AzVolumeAttachment total 162 items received I0119 02:29:14.823378 1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumeattachments?allowWatchBookmarks=true&resourceVersion=15340&timeout=5m46s&timeoutSeconds=346&watch=true 200 OK in 1 milliseconds I0119 02:29:15.185392 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I0119 02:29:15.185412 1 utils.go:79] GRPC request: {} I0119 02:29:15.185448 1 utils.go:85] GRPC response: {"ready":{"value":true}} I0119 02:29:37.963897 1 azuredisk_v2.go:578] "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-19T02:29:37Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="1ebdcfcb-97a1-11ed-be91-002248822c79" I0119 02:29:37.980538 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-29517166-vmss000001/status 200 OK in 16 milliseconds I0119 02:29:37.980705 1 azuredisk_v2.go:565] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="1ebdcfcb-97a1-11ed-be91-002248822c79" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=16854275 I0119 02:29:40.861094 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:29:40.861146 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:29:40.861244 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:29:40.861273 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:29:40.861301 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:29:40.904497 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:29:45.186299 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I0119 02:29:45.186319 1 utils.go:79] GRPC request: {} I0119 02:29:45.186365 1 utils.go:85] GRPC response: {"ready":{"value":true}} I0119 02:29:47.624202 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:29:47.624290 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="fcbc4510-97a0-11ed-be91-002248822c79" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-8902c4ec-a02c-4a3f-a96c-04d72070c956" "latency"=66713009472 I0119 02:29:47.624328 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="fcbc4510-97a0-11ed-be91-002248822c79" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=66713123874 I0119 02:29:49.363314 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdc under /dev/disk/azure/scsi1/ I0119 02:29:49.363355 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType I0119 02:29:49.363379 1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun1 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8902c4ec-a02c-4a3f-a96c-04d72070c956/globalmount with mount options([]) I0119 02:29:49.363387 1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun1" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun1]) ... skipping 40 lines ... I0119 02:30:07.971537 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-29517166-vmss000001/status 200 OK in 6 milliseconds I0119 02:30:07.971717 1 azuredisk_v2.go:565] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="309f87e7-97a1-11ed-be91-002248822c79" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=7333275 I0119 02:30:10.862261 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:30:10.862292 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:30:10.862305 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:30:10.862330 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:30:10.862471 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:30:10.904680 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:30:15.185244 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I0119 02:30:15.185267 1 utils.go:79] GRPC request: {} I0119 02:30:15.185314 1 utils.go:85] GRPC response: {"ready":{"value":true}} I0119 02:30:24.932701 1 utils.go:78] GRPC call: /csi.v1.Node/NodeUnpublishVolume I0119 02:30:24.932737 1 utils.go:79] GRPC request: {"target_path":"/var/lib/kubelet/pods/99bc0ead-8e1f-4327-bbd8-aae1eac6ba3d/volumes/kubernetes.io~csi/pvc-8902c4ec-a02c-4a3f-a96c-04d72070c956/mount","volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-iei3lcrr/providers/Microsoft.Compute/disks/pvc-8902c4ec-a02c-4a3f-a96c-04d72070c956"} ... skipping 18 lines ... I0119 02:30:37.970570 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-29517166-vmss000001/status 200 OK in 6 milliseconds I0119 02:30:37.970733 1 azuredisk_v2.go:565] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="42810f8f-97a1-11ed-be91-002248822c79" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=7051272 I0119 02:30:40.862645 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:30:40.862676 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:30:40.862725 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:30:40.862731 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:30:40.862820 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:30:40.905156 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:30:45.185542 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I0119 02:30:45.185569 1 utils.go:79] GRPC request: {} I0119 02:30:45.185615 1 utils.go:85] GRPC response: {"ready":{"value":true}} I0119 02:31:07.965085 1 azuredisk_v2.go:578] "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-19T02:31:07Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="5462e73f-97a1-11ed-be91-002248822c79" I0119 02:31:07.979863 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-29517166-vmss000001/status 200 OK in 14 milliseconds I0119 02:31:07.980516 1 azuredisk_v2.go:565] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="5462e73f-97a1-11ed-be91-002248822c79" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=15475441 I0119 02:31:10.863467 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:31:10.863556 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:31:10.863579 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:31:10.863562 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:31:10.863706 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:31:10.906030 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:31:15.185156 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I0119 02:31:15.185184 1 utils.go:79] GRPC request: {} I0119 02:31:15.185232 1 utils.go:85] GRPC response: {"ready":{"value":true}} I0119 02:31:17.833139 1 reflector.go:559] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: Watch close - *v1.CustomResourceDefinition total 1 items received I0119 02:31:17.838544 1 round_trippers.go:553] GET https://10.0.0.1:443/apis/apiextensions.k8s.io/v1/customresourcedefinitions?allowWatchBookmarks=true&resourceVersion=16255&timeout=6m17s&timeoutSeconds=377&watch=true 200 OK in 5 milliseconds ... skipping 3 lines ... I0119 02:31:37.973429 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-29517166-vmss000001/status 200 OK in 9 milliseconds I0119 02:31:37.973940 1 azuredisk_v2.go:565] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="66445a51-97a1-11ed-be91-002248822c79" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=10133518 I0119 02:31:40.863667 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:31:40.863716 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:31:40.863789 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:31:40.863790 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:31:40.863907 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:31:40.907098 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:31:41.827689 1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: Watch close - *v1beta2.AzDriverNode total 63 items received I0119 02:31:41.833899 1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azdrivernodes?allowWatchBookmarks=true&resourceVersion=16447&timeout=7m35s&timeoutSeconds=455&watch=true 200 OK in 6 milliseconds I0119 02:31:45.185340 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I0119 02:31:45.185365 1 utils.go:79] GRPC request: {} I0119 02:31:45.185412 1 utils.go:85] GRPC response: {"ready":{"value":true}} E0119 02:31:50.799953 1 conditionwaiter.go:50] "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [context canceled]" "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="26654109-97a1-11ed-be91-002248822c79" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-b60a43d5-32ea-4dc5-94e9-b57828b77226" "latency"=119994575411 E0119 02:31:50.800036 1 crdprovisioner.go:743] "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [context canceled]" "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="26654109-97a1-11ed-be91-002248822c79" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=119994768413 E0119 02:31:50.800056 1 utils.go:83] GRPC error: rpc error: code = Internal desc = failed to wait for volume (/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-iei3lcrr/providers/Microsoft.Compute/disks/pvc-b60a43d5-32ea-4dc5-94e9-b57828b77226) to be attached to node (k8s-agentpool1-29517166-vmss000001): context canceled I0119 02:31:51.320832 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities I0119 02:31:51.320858 1 utils.go:79] GRPC request: {} I0119 02:31:51.320906 1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]} I0119 02:31:51.331355 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities I0119 02:31:51.331373 1 utils.go:79] GRPC request: {} I0119 02:31:51.331403 1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]} ... skipping 9 lines ... I0119 02:32:07.978484 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-29517166-vmss000001/status 200 OK in 12 milliseconds I0119 02:32:07.979445 1 azuredisk_v2.go:565] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="78262f67-97a1-11ed-be91-002248822c79" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=14350536 I0119 02:32:10.865219 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:32:10.865257 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:32:10.865294 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:32:10.865245 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:32:10.865365 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:32:10.907705 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:32:15.185481 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I0119 02:32:15.185507 1 utils.go:79] GRPC request: {} I0119 02:32:15.185554 1 utils.go:85] GRPC response: {"ready":{"value":true}} I0119 02:32:37.963246 1 azuredisk_v2.go:578] "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-19T02:32:37Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="8a078871-97a1-11ed-be91-002248822c79" I0119 02:32:37.970936 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-29517166-vmss000001/status 200 OK in 7 milliseconds I0119 02:32:37.972065 1 azuredisk_v2.go:565] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="8a078871-97a1-11ed-be91-002248822c79" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=8867687 I0119 02:32:40.866997 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:32:40.867076 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:32:40.867120 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:32:40.867164 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:32:40.867213 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:32:40.908457 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:32:45.185921 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I0119 02:32:45.185946 1 utils.go:79] GRPC request: {} I0119 02:32:45.185987 1 utils.go:85] GRPC response: {"ready":{"value":true}} I0119 02:33:07.965169 1 azuredisk_v2.go:578] "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-19T02:33:07Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="9be97680-97a1-11ed-be91-002248822c79" I0119 02:33:07.974882 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-29517166-vmss000001/status 200 OK in 9 milliseconds I0119 02:33:07.975548 1 azuredisk_v2.go:565] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="9be97680-97a1-11ed-be91-002248822c79" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=10417898 I0119 02:33:10.868778 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:33:10.868810 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:33:10.868882 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:33:10.868894 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:33:10.868896 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:33:10.909315 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:33:15.185584 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I0119 02:33:15.185612 1 utils.go:79] GRPC request: {} I0119 02:33:15.185659 1 utils.go:85] GRPC response: {"ready":{"value":true}} I0119 02:33:37.963678 1 azuredisk_v2.go:578] "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-19T02:33:37Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="adcadf9a-97a1-11ed-be91-002248822c79" I0119 02:33:37.969659 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-29517166-vmss000001/status 200 OK in 5 milliseconds I0119 02:33:37.970433 1 azuredisk_v2.go:565] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="adcadf9a-97a1-11ed-be91-002248822c79" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=6796566 I0119 02:33:40.869219 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:33:40.869302 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:33:40.869308 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:33:40.869346 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:33:40.869434 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:33:40.909692 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:33:45.185316 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I0119 02:33:45.185336 1 utils.go:79] GRPC request: {} I0119 02:33:45.185375 1 utils.go:85] GRPC response: {"ready":{"value":true}} E0119 02:33:51.321012 1 conditionwaiter.go:50] "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [context canceled]" "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="6e3c54e3-97a1-11ed-be91-002248822c79" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-b60a43d5-32ea-4dc5-94e9-b57828b77226" "latency"=119987877216 E0119 02:33:51.321086 1 crdprovisioner.go:743] "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [context canceled]" "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="6e3c54e3-97a1-11ed-be91-002248822c79" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=119988072717 E0119 02:33:51.321106 1 utils.go:83] GRPC error: rpc error: code = Internal desc = failed to wait for volume (/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-iei3lcrr/providers/Microsoft.Compute/disks/pvc-b60a43d5-32ea-4dc5-94e9-b57828b77226) to be attached to node (k8s-agentpool1-29517166-vmss000001): context canceled I0119 02:33:52.375663 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities I0119 02:33:52.375686 1 utils.go:79] GRPC request: {} I0119 02:33:52.375724 1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]} I0119 02:33:52.379127 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities I0119 02:33:52.379145 1 utils.go:79] GRPC request: {} I0119 02:33:52.379181 1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]} ... skipping 7 lines ... I0119 02:34:07.974424 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-29517166-vmss000001/status 200 OK in 9 milliseconds I0119 02:34:07.974587 1 azuredisk_v2.go:565] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="bfacbca4-97a1-11ed-be91-002248822c79" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=9467592 I0119 02:34:10.869412 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:34:10.869433 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:34:10.869560 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:34:10.869578 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:34:10.869706 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:34:10.909796 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:34:15.185748 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I0119 02:34:15.185770 1 utils.go:79] GRPC request: {} I0119 02:34:15.185819 1 utils.go:85] GRPC response: {"ready":{"value":true}} I0119 02:34:37.964579 1 azuredisk_v2.go:578] "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-19T02:34:37Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="d18e489c-97a1-11ed-be91-002248822c79" I0119 02:34:37.975445 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-29517166-vmss000001/status 200 OK in 10 milliseconds I0119 02:34:37.975692 1 azuredisk_v2.go:565] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="d18e489c-97a1-11ed-be91-002248822c79" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=11162207 I0119 02:34:40.871536 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:34:40.871572 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:34:40.871584 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:34:40.871618 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:34:40.871666 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:34:40.909920 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:34:45.185337 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I0119 02:34:45.185358 1 utils.go:79] GRPC request: {} I0119 02:34:45.185397 1 utils.go:85] GRPC response: {"ready":{"value":true}} I0119 02:34:48.975378 1 conditionwatcher.go:173] condition result: succeeded: false, error: rpc error: code = Unknown desc = context deadline exceeded E0119 02:34:48.975515 1 conditionwaiter.go:50] "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [rpc error: code = Unknown desc = context deadline exceeded]" "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="b662bfcc-97a1-11ed-be91-002248822c79" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-b60a43d5-32ea-4dc5-94e9-b57828b77226" "latency"=56594655151 E0119 02:34:48.975567 1 crdprovisioner.go:743] "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [rpc error: code = Unknown desc = context deadline exceeded]" "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="b662bfcc-97a1-11ed-be91-002248822c79" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=56594824952 E0119 02:34:48.975583 1 utils.go:83] GRPC error: rpc error: code = Internal desc = failed to wait for volume (/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-iei3lcrr/providers/Microsoft.Compute/disks/pvc-b60a43d5-32ea-4dc5-94e9-b57828b77226) to be attached to node (k8s-agentpool1-29517166-vmss000001): rpc error: code = Unknown desc = context deadline exceeded I0119 02:34:50.986283 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities I0119 02:34:50.986310 1 utils.go:79] GRPC request: {} I0119 02:34:50.986356 1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]} I0119 02:34:50.989839 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities I0119 02:34:50.989856 1 utils.go:79] GRPC request: {} I0119 02:34:50.989883 1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]} ... skipping 2 lines ... I0119 02:34:50.990640 1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]} I0119 02:34:50.991374 1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume I0119 02:34:50.991389 1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-b60a43d5-32ea-4dc5-94e9-b57828b77226/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-b60a43d5-32ea-4dc5-94e9-b57828b77226","csi.storage.k8s.io/pvc/name":"test.csi.azure.comwmhgd","csi.storage.k8s.io/pvc/namespace":"volume-1076","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1674093937787-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-iei3lcrr/providers/Microsoft.Compute/disks/pvc-b60a43d5-32ea-4dc5-94e9-b57828b77226"} I0119 02:34:50.998370 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-b60a43d5-32ea-4dc5-94e9-b57828b77226-k8s-agentpool1-29517166-vmss000001-attachment/status 200 OK in 6 milliseconds I0119 02:34:50.998589 1 azure_disk_utils.go:1103] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/request-id"="d9520e28-97a1-11ed-be91-002248822c79" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "latency"=6954868 "number_of_net_retry"=0 "number_of_retry"=0 I0119 02:34:50.998606 1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-b60a43d5-32ea-4dc5-94e9-b57828b77226-k8s-agentpool1-29517166-vmss000001-attachment) I0119 02:34:50.999360 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:34:51.004999 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:34:51.039027 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:34:51.891017 1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: Watch close - *v1beta2.AzDriverNode total 19 items received I0119 02:34:51.894781 1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dk8s-agentpool1-29517166-vmss000001&resourceVersion=17282&timeout=8m25s&timeoutSeconds=505&watch=true 200 OK in 3 milliseconds I0119 02:34:52.050697 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:34:52.050823 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="d9520e28-97a1-11ed-be91-002248822c79" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-b60a43d5-32ea-4dc5-94e9-b57828b77226" "latency"=1052098422 I0119 02:34:52.050868 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="d9520e28-97a1-11ed-be91-002248822c79" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=1059276392 I0119 02:34:52.639056 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdd under /dev/disk/azure/scsi1/ I0119 02:34:52.639108 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType I0119 02:34:52.639148 1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-b60a43d5-32ea-4dc5-94e9-b57828b77226/globalmount with mount options([]) I0119 02:34:52.639165 1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0]) ... skipping 69 lines ... I0119 02:35:39.850412 1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-4562cdae-40b7-45fb-8a07-75afe7e9b752","volume_capability":{"AccessType":{"Block":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-4562cdae-40b7-45fb-8a07-75afe7e9b752","csi.storage.k8s.io/pvc/name":"pvc-tm8xq","csi.storage.k8s.io/pvc/namespace":"provisioning-2626","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1674093937787-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-iei3lcrr/providers/Microsoft.Compute/disks/pvc-4562cdae-40b7-45fb-8a07-75afe7e9b752"} I0119 02:35:39.850562 1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-4562cdae-40b7-45fb-8a07-75afe7e9b752-k8s-agentpool1-29517166-vmss000001-attachment) I0119 02:35:40.872669 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:35:40.872708 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:35:40.872799 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:35:40.872828 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:35:40.872902 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:35:40.911093 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:35:45.184987 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I0119 02:35:45.185015 1 utils.go:79] GRPC request: {} I0119 02:35:45.185084 1 utils.go:85] GRPC response: {"ready":{"value":true}} I0119 02:35:49.072137 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:35:49.072215 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="f6715766-97a1-11ed-be91-002248822c79" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-4562cdae-40b7-45fb-8a07-75afe7e9b752" "latency"=9221573067 I0119 02:35:49.072248 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="f6715766-97a1-11ed-be91-002248822c79" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9221696468 I0119 02:35:50.783700 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/ I0119 02:35:50.783749 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType I0119 02:35:50.783762 1 utils.go:85] GRPC response: {} I0119 02:35:50.788590 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities ... skipping 63 lines ... I0119 02:36:47.431133 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities I0119 02:36:47.431149 1 utils.go:79] GRPC request: {} I0119 02:36:47.431177 1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]} I0119 02:36:47.431845 1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume I0119 02:36:47.431859 1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8512c96c-52c9-47a5-b805-c1a2164b9779/globalmount","volume_capability":{"AccessType":{"Mount":{"fs_type":"xfs"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-8512c96c-52c9-47a5-b805-c1a2164b9779","csi.storage.k8s.io/pvc/name":"test.csi.azure.comzxdj8","csi.storage.k8s.io/pvc/namespace":"volume-9217","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1674093937787-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-iei3lcrr/providers/Microsoft.Compute/disks/pvc-8512c96c-52c9-47a5-b805-c1a2164b9779"} I0119 02:36:47.432054 1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-8512c96c-52c9-47a5-b805-c1a2164b9779-k8s-agentpool1-29517166-vmss000001-attachment) I0119 02:37:02.389113 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:37:02.389201 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="1eb974e9-97a2-11ed-be91-002248822c79" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-8512c96c-52c9-47a5-b805-c1a2164b9779" "latency"=14957059983 I0119 02:37:02.389234 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="1eb974e9-97a2-11ed-be91-002248822c79" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=14957204486 I0119 02:37:04.079635 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/ I0119 02:37:04.079690 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType I0119 02:37:04.079724 1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8512c96c-52c9-47a5-b805-c1a2164b9779/globalmount with mount options([nouuid]) I0119 02:37:04.079740 1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0]) ... skipping 72 lines ... I0119 02:37:37.969818 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-29517166-vmss000001/status 200 OK in 5 milliseconds I0119 02:37:37.970049 1 azuredisk_v2.go:565] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="3cd8109f-97a2-11ed-be91-002248822c79" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=5766404 I0119 02:37:40.876910 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:37:40.876941 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:37:40.876998 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:37:40.877058 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:37:40.877114 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:37:40.912430 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:37:45.185680 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I0119 02:37:45.185707 1 utils.go:79] GRPC request: {} I0119 02:37:45.185753 1 utils.go:85] GRPC response: {"ready":{"value":true}} I0119 02:38:07.965190 1 azuredisk_v2.go:578] "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-19T02:38:07Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="4eb9d4ce-97a2-11ed-be91-002248822c79" I0119 02:38:07.975466 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-29517166-vmss000001/status 200 OK in 10 milliseconds I0119 02:38:07.975674 1 azuredisk_v2.go:565] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="4eb9d4ce-97a2-11ed-be91-002248822c79" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=10548959 I0119 02:38:10.879765 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:38:10.879802 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:38:10.879873 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:38:10.879887 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:38:10.879896 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:38:10.913331 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:38:15.184928 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I0119 02:38:15.184955 1 utils.go:79] GRPC request: {} I0119 02:38:15.185005 1 utils.go:85] GRPC response: {"ready":{"value":true}} I0119 02:38:23.646927 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:38:23.647004 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="3493b417-97a2-11ed-be91-002248822c79" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-c1059538-b4f0-4bd1-8857-5cc7423f2b65" "latency"=59552428897 I0119 02:38:23.647030 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="3493b417-97a2-11ed-be91-002248822c79" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=59552551899 I0119 02:38:25.356725 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdc under /dev/disk/azure/scsi1/ I0119 02:38:25.356774 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType I0119 02:38:25.356802 1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun1 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-c1059538-b4f0-4bd1-8857-5cc7423f2b65/globalmount with mount options([]) I0119 02:38:25.356811 1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun1" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun1]) ... skipping 76 lines ... I0119 02:39:07.970354 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-29517166-vmss000001/status 200 OK in 6 milliseconds I0119 02:39:07.970579 1 azuredisk_v2.go:565] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="727cee73-97a2-11ed-be91-002248822c79" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=6593696 I0119 02:39:10.881509 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:39:10.881579 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:39:10.881601 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:39:10.881616 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:39:10.881718 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:39:10.914975 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:39:15.185021 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I0119 02:39:15.185087 1 utils.go:79] GRPC request: {} I0119 02:39:15.185139 1 utils.go:85] GRPC response: {"ready":{"value":true}} I0119 02:39:16.834680 1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: Watch close - *v1beta2.AzDriverNode total 70 items received I0119 02:39:16.838504 1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azdrivernodes?allowWatchBookmarks=true&resourceVersion=20146&timeout=9m8s&timeoutSeconds=548&watch=true 200 OK in 3 milliseconds ... skipping 3 lines ... I0119 02:39:37.975615 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-29517166-vmss000001/status 200 OK in 10 milliseconds I0119 02:39:37.975753 1 azuredisk_v2.go:565] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="845ead1a-97a2-11ed-be91-002248822c79" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=11060466 I0119 02:39:40.882022 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:39:40.882125 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:39:40.882179 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:39:40.882307 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:39:40.882424 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:39:40.915610 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:39:45.185228 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I0119 02:39:45.185252 1 utils.go:79] GRPC request: {} I0119 02:39:45.185288 1 utils.go:85] GRPC response: {"ready":{"value":true}} I0119 02:39:51.061345 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:39:51.061405 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="7107e60b-97a2-11ed-be91-002248822c79" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-8509c51d-0aca-4a01-a6cd-f3c733583298" "latency"=45542047007 I0119 02:39:51.061428 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="7107e60b-97a2-11ed-be91-002248822c79" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=45542157308 I0119 02:39:52.160240 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities I0119 02:39:52.160262 1 utils.go:79] GRPC request: {} I0119 02:39:52.160306 1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]} I0119 02:39:52.162353 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities ... skipping 53 lines ... I0119 02:40:07.973589 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-29517166-vmss000001/status 200 OK in 9 milliseconds I0119 02:40:07.974044 1 azuredisk_v2.go:565] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="96404096-97a2-11ed-be91-002248822c79" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=9740711 I0119 02:40:10.884955 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:40:10.885109 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:40:10.885184 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:40:10.885200 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:40:10.885264 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:40:10.885296 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:40:10.916567 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:40:15.185466 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I0119 02:40:15.185487 1 utils.go:79] GRPC request: {} I0119 02:40:15.185529 1 utils.go:85] GRPC response: {"ready":{"value":true}} I0119 02:40:21.812958 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:40:21.813020 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="8cd538cc-97a2-11ed-be91-002248822c79" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-d5beef35-dd4c-4e3f-a489-1226c0107fd2" "latency"=29649569360 I0119 02:40:21.813077 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="8cd538cc-97a2-11ed-be91-002248822c79" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=29649709362 I0119 02:40:21.815067 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:40:21.815123 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="8cf3be70-97a2-11ed-be91-002248822c79" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-869c29e9-e060-42a3-bbc3-7bb9aec7a2b3" "latency"=29451645684 I0119 02:40:21.815151 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="8cf3be70-97a2-11ed-be91-002248822c79" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=29451760685 I0119 02:40:22.415614 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun2 by sde under /dev/disk/azure/scsi1/ I0119 02:40:22.415627 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdd under /dev/disk/azure/scsi1/ I0119 02:40:22.415662 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun2. perfProfile none accountType I0119 02:40:22.415667 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType ... skipping 59 lines ... I0119 02:40:28.340588 1 nodeserver_v2.go:257] NodeUnstageVolume: unmounting /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8509c51d-0aca-4a01-a6cd-f3c733583298/globalmount I0119 02:40:28.340614 1 mount_helper_common.go:99] "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8509c51d-0aca-4a01-a6cd-f3c733583298/globalmount" is a mountpoint, unmounting I0119 02:40:28.340628 1 mount_linux.go:294] Unmounting /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8509c51d-0aca-4a01-a6cd-f3c733583298/globalmount W0119 02:40:28.356557 1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8509c51d-0aca-4a01-a6cd-f3c733583298/globalmount" is not a mountpoint, deleting I0119 02:40:28.356612 1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8509c51d-0aca-4a01-a6cd-f3c733583298/globalmount successfully I0119 02:40:28.356625 1 utils.go:85] GRPC response: {} I0119 02:40:37.049254 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:40:37.049323 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="9fa04da2-97a2-11ed-be91-002248822c79" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-be1e4442-0d18-404d-b36c-8089b08cbe32" "latency"=13355971511 I0119 02:40:37.049345 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="9fa04da2-97a2-11ed-be91-002248822c79" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=13356082512 I0119 02:40:37.053891 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:40:37.053949 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="9fbeef0a-97a2-11ed-be91-002248822c79" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-9a960e5e-a3d4-416b-a791-4a8e35348ee6" "latency"=13159868899 I0119 02:40:37.053971 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="9fbeef0a-97a2-11ed-be91-002248822c79" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=13159969301 I0119 02:40:37.645432 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun4 by sdf under /dev/disk/azure/scsi1/ I0119 02:40:37.645478 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun4. perfProfile none accountType I0119 02:40:37.645490 1 utils.go:85] GRPC response: {} I0119 02:40:37.645592 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun3 by sdg under /dev/disk/azure/scsi1/ ... skipping 172 lines ... I0119 02:43:19.977552 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities I0119 02:43:19.977567 1 utils.go:79] GRPC request: {} I0119 02:43:19.977600 1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]} I0119 02:43:19.978300 1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume I0119 02:43:19.978313 1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-440f5d97-1dce-423e-9bd6-736c5272f264/globalmount","volume_capability":{"AccessType":{"Mount":{"fs_type":"xfs"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-440f5d97-1dce-423e-9bd6-736c5272f264","csi.storage.k8s.io/pvc/name":"test.csi.azure.comzjpcq","csi.storage.k8s.io/pvc/namespace":"multivolume-4184","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1674093937787-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-iei3lcrr/providers/Microsoft.Compute/disks/pvc-440f5d97-1dce-423e-9bd6-736c5272f264"} I0119 02:43:19.978505 1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-440f5d97-1dce-423e-9bd6-736c5272f264-k8s-agentpool1-29517166-vmss000001-attachment) I0119 02:43:28.653184 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:43:28.653247 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="08b34b14-97a3-11ed-be91-002248822c79" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-440f5d97-1dce-423e-9bd6-736c5272f264" "latency"=8674669075 I0119 02:43:28.653269 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="08b34b14-97a3-11ed-be91-002248822c79" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=8674785775 I0119 02:43:29.255045 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/ I0119 02:43:29.255096 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType I0119 02:43:29.255125 1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-440f5d97-1dce-423e-9bd6-736c5272f264/globalmount with mount options([nouuid]) I0119 02:43:29.255136 1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0]) ... skipping 121 lines ... I0119 02:44:07.982114 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-29517166-vmss000001/status 200 OK in 17 milliseconds I0119 02:44:07.982480 1 azuredisk_v2.go:565] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="254d5845-97a3-11ed-be91-002248822c79" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=18168884 I0119 02:44:10.898315 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:44:10.898429 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:44:10.898448 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:44:10.898437 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:44:10.898534 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:44:10.920815 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:44:15.185357 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I0119 02:44:15.185386 1 utils.go:79] GRPC request: {} I0119 02:44:15.185434 1 utils.go:85] GRPC response: {"ready":{"value":true}} I0119 02:44:15.933686 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:44:15.933772 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="2113111f-97a3-11ed-be91-002248822c79" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-e0e7c410-d6b1-4282-8840-de0fba9489d5" "latency"=15062185310 I0119 02:44:15.933810 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="2113111f-97a3-11ed-be91-002248822c79" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=15062337912 I0119 02:44:16.520401 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdd under /dev/disk/azure/scsi1/ I0119 02:44:16.520452 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType I0119 02:44:16.520485 1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun1 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-e0e7c410-d6b1-4282-8840-de0fba9489d5/globalmount with mount options([nouuid]) I0119 02:44:16.520498 1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun1" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun1]) ... skipping 95 lines ... I0119 02:45:32.615872 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities I0119 02:45:32.615887 1 utils.go:79] GRPC request: {} I0119 02:45:32.615919 1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]} I0119 02:45:32.616719 1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume I0119 02:45:32.616738 1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-85c2856a-cf1f-4e99-b4a0-d890eba87abc/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-85c2856a-cf1f-4e99-b4a0-d890eba87abc","csi.storage.k8s.io/pvc/name":"restored-pvc-tester-nsfkr-my-volume","csi.storage.k8s.io/pvc/namespace":"snapshotting-1841","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1674093937787-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-iei3lcrr/providers/Microsoft.Compute/disks/pvc-85c2856a-cf1f-4e99-b4a0-d890eba87abc"} I0119 02:45:32.616934 1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-85c2856a-cf1f-4e99-b4a0-d890eba87abc-k8s-agentpool1-29517166-vmss000001-attachment) I0119 02:45:36.818664 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:45:36.818752 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="57c24fd3-97a3-11ed-be91-002248822c79" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-85c2856a-cf1f-4e99-b4a0-d890eba87abc" "latency"=4201727254 I0119 02:45:36.818779 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="57c24fd3-97a3-11ed-be91-002248822c79" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=4201861555 I0119 02:45:37.845562 1 reflector.go:559] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: Watch close - *v1.CustomResourceDefinition total 6 items received I0119 02:45:37.850110 1 round_trippers.go:553] GET https://10.0.0.1:443/apis/apiextensions.k8s.io/v1/customresourcedefinitions?allowWatchBookmarks=true&resourceVersion=20948&timeout=9m16s&timeoutSeconds=556&watch=true 200 OK in 4 milliseconds I0119 02:45:37.963428 1 azuredisk_v2.go:578] "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-19T02:45:37Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="5af21d53-97a3-11ed-be91-002248822c79" I0119 02:45:37.969691 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-29517166-vmss000001/status 200 OK in 6 milliseconds ... skipping 46 lines ... I0119 02:45:40.973705 1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume I0119 02:45:40.973720 1 utils.go:79] GRPC request: {"publish_context":{"LUN":"1"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-62398535-c59c-4669-aa53-4a0739a3ad50/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-62398535-c59c-4669-aa53-4a0739a3ad50","csi.storage.k8s.io/pvc/name":"test.csi.azure.comzd48c","csi.storage.k8s.io/pvc/namespace":"snapshotting-9690","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1674093937787-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-iei3lcrr/providers/Microsoft.Compute/disks/pvc-62398535-c59c-4669-aa53-4a0739a3ad50"} I0119 02:45:40.973910 1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-62398535-c59c-4669-aa53-4a0739a3ad50-k8s-agentpool1-29517166-vmss000001-attachment) I0119 02:45:45.185912 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I0119 02:45:45.185938 1 utils.go:79] GRPC request: {} I0119 02:45:45.185986 1 utils.go:85] GRPC response: {"ready":{"value":true}} I0119 02:45:51.009517 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:45:51.009594 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="5cbd7c27-97a3-11ed-be91-002248822c79" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-62398535-c59c-4669-aa53-4a0739a3ad50" "latency"=10035615068 I0119 02:45:51.009631 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="5cbd7c27-97a3-11ed-be91-002248822c79" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=10035733269 I0119 02:45:51.628400 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdd under /dev/disk/azure/scsi1/ I0119 02:45:51.628456 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType I0119 02:45:51.628485 1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun1 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-62398535-c59c-4669-aa53-4a0739a3ad50/globalmount with mount options([]) I0119 02:45:51.628500 1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun1" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun1]) ... skipping 63 lines ... I0119 02:46:03.328672 1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-bc939303-0a2f-4000-a3c8-a93860b91074-k8s-agentpool1-29517166-vmss000001-attachment) I0119 02:46:07.963215 1 azuredisk_v2.go:578] "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-19T02:46:07Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="6cd3b8bd-97a3-11ed-be91-002248822c79" I0119 02:46:07.975611 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-29517166-vmss000001/status 200 OK in 12 milliseconds I0119 02:46:07.975816 1 azuredisk_v2.go:565] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="6cd3b8bd-97a3-11ed-be91-002248822c79" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=12649334 I0119 02:46:10.903752 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:46:10.903895 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:46:10.903968 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:46:10.903975 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:46:10.904008 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:46:10.923404 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:46:15.185310 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I0119 02:46:15.185340 1 utils.go:79] GRPC request: {} I0119 02:46:15.185383 1 utils.go:85] GRPC response: {"ready":{"value":true}} I0119 02:46:23.012725 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:46:23.012813 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="6a108cd1-97a3-11ed-be91-002248822c79" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-bc939303-0a2f-4000-a3c8-a93860b91074" "latency"=19684047719 I0119 02:46:23.012855 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="6a108cd1-97a3-11ed-be91-002248822c79" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=19684199221 I0119 02:46:23.607986 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun2 by sde under /dev/disk/azure/scsi1/ I0119 02:46:23.608044 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun2. perfProfile none accountType I0119 02:46:23.608084 1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun2 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-bc939303-0a2f-4000-a3c8-a93860b91074/globalmount with mount options([]) I0119 02:46:23.608102 1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun2" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun2]) ... skipping 106 lines ... I0119 02:47:55.606923 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities I0119 02:47:55.606942 1 utils.go:79] GRPC request: {} I0119 02:47:55.606980 1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]} I0119 02:47:55.607796 1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume I0119 02:47:55.607811 1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-d2cc607e-eeb0-4910-953e-da5c936183e6/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-d2cc607e-eeb0-4910-953e-da5c936183e6","csi.storage.k8s.io/pvc/name":"test.csi.azure.comljbg5","csi.storage.k8s.io/pvc/namespace":"snapshotting-6034","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1674093937787-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-iei3lcrr/providers/Microsoft.Compute/disks/pvc-d2cc607e-eeb0-4910-953e-da5c936183e6"} I0119 02:47:55.607983 1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-d2cc607e-eeb0-4910-953e-da5c936183e6-k8s-agentpool1-29517166-vmss000001-attachment) I0119 02:47:59.191948 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:47:59.192018 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="acfd03aa-97a3-11ed-be91-002248822c79" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-d2cc607e-eeb0-4910-953e-da5c936183e6" "latency"=3583953737 I0119 02:47:59.192052 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="acfd03aa-97a3-11ed-be91-002248822c79" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=3584080137 I0119 02:48:00.882631 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/ I0119 02:48:00.882690 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType I0119 02:48:00.882732 1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-d2cc607e-eeb0-4910-953e-da5c936183e6/globalmount with mount options([]) I0119 02:48:00.882755 1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0]) ... skipping 60 lines ... I0119 02:48:07.984859 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-29517166-vmss000001/status 200 OK in 20 milliseconds I0119 02:48:07.985640 1 azuredisk_v2.go:565] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="b45a6f07-97a3-11ed-be91-002248822c79" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=21379604 I0119 02:48:10.910515 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:48:10.910571 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:48:10.910487 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:48:10.910603 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:48:10.910666 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:48:10.925876 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:48:15.187151 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I0119 02:48:15.187177 1 utils.go:79] GRPC request: {} I0119 02:48:15.187238 1 utils.go:85] GRPC response: {"ready":{"value":true}} I0119 02:48:15.777767 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:48:15.777850 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="b03d1932-97a3-11ed-be91-002248822c79" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-ff0c3c20-3611-4d61-bd57-17ac092a3f1d" "latency"=14716642599 I0119 02:48:15.777885 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="b03d1932-97a3-11ed-be91-002248822c79" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=14716767200 I0119 02:48:16.161716 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities I0119 02:48:16.161745 1 utils.go:79] GRPC request: {} I0119 02:48:16.161793 1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]} I0119 02:48:16.162572 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities ... skipping 45 lines ... I0119 02:48:37.974800 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-29517166-vmss000001/status 200 OK in 9 milliseconds I0119 02:48:37.975070 1 azuredisk_v2.go:565] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="c63c1ec3-97a3-11ed-be91-002248822c79" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=10479304 I0119 02:48:40.911366 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:48:40.911456 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:48:40.911516 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:48:40.911570 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:48:40.911701 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:48:40.926915 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:48:41.219139 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:48:41.219208 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="b93d8dc5-97a3-11ed-be91-002248822c79" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-66f5f690-224e-4d3c-b4f1-57a8145c8f7d" "latency"=25055505265 I0119 02:48:41.219238 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="b93d8dc5-97a3-11ed-be91-002248822c79" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=25055645367 I0119 02:48:41.818244 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun2 by sdc under /dev/disk/azure/scsi1/ I0119 02:48:41.818294 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun2. perfProfile none accountType I0119 02:48:41.818310 1 utils.go:85] GRPC response: {} I0119 02:48:41.822119 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities ... skipping 87 lines ... I0119 02:50:07.975664 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-29517166-vmss000001/status 200 OK in 10 milliseconds I0119 02:50:07.975964 1 azuredisk_v2.go:565] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="fbe108d0-97a3-11ed-be91-002248822c79" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=11313710 I0119 02:50:10.917223 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:50:10.917299 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:50:10.917394 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:50:10.917427 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:50:10.917561 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:50:10.928769 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:50:14.262844 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:50:14.262908 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="f0203097-97a3-11ed-be91-002248822c79" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-9d9961c0-d9ec-49b4-a68d-b2a68fcca2b8" "latency"=26016967198 I0119 02:50:14.262930 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="f0203097-97a3-11ed-be91-002248822c79" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=26017092399 I0119 02:50:15.185363 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I0119 02:50:15.185391 1 utils.go:79] GRPC request: {} I0119 02:50:15.185436 1 utils.go:85] GRPC response: {"ready":{"value":true}} I0119 02:50:15.999810 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/ ... skipping 44 lines ... I0119 02:50:39.456514 1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume I0119 02:50:39.456532 1 utils.go:79] GRPC request: {"publish_context":{"LUN":"1"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-eb46b192-5475-4bdb-85cd-8d6c4b8e2fb5/globalmount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-eb46b192-5475-4bdb-85cd-8d6c4b8e2fb5","csi.storage.k8s.io/pvc/name":"test.csi.azure.comtlgc4","csi.storage.k8s.io/pvc/namespace":"volume-4020","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1674093937787-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-iei3lcrr/providers/Microsoft.Compute/disks/pvc-eb46b192-5475-4bdb-85cd-8d6c4b8e2fb5"} I0119 02:50:39.456688 1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-eb46b192-5475-4bdb-85cd-8d6c4b8e2fb5-k8s-agentpool1-29517166-vmss000001-attachment) I0119 02:50:40.918198 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:50:40.918290 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:50:40.918385 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:50:40.918470 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:50:40.918547 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:50:40.929780 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:50:45.186110 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I0119 02:50:45.186134 1 utils.go:79] GRPC request: {} I0119 02:50:45.186182 1 utils.go:85] GRPC response: {"ready":{"value":true}} I0119 02:51:01.065016 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:51:01.065103 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="0ea657ae-97a4-11ed-be91-002248822c79" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-eb46b192-5475-4bdb-85cd-8d6c4b8e2fb5" "latency"=21608344352 I0119 02:51:01.065127 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="0ea657ae-97a4-11ed-be91-002248822c79" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=21608457353 I0119 02:51:02.826534 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdc under /dev/disk/azure/scsi1/ I0119 02:51:02.826583 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType I0119 02:51:02.826619 1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun1 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-eb46b192-5475-4bdb-85cd-8d6c4b8e2fb5/globalmount with mount options([]) I0119 02:51:02.826630 1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun1" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun1]) ... skipping 66 lines ... I0119 02:51:09.672569 1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-4f2c37ac-8825-40bb-a55f-dc9dd2622455","volume_capability":{"AccessType":{"Block":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-4f2c37ac-8825-40bb-a55f-dc9dd2622455","csi.storage.k8s.io/pvc/name":"pvc-qwlnt","csi.storage.k8s.io/pvc/namespace":"provisioning-745","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1674093937787-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-iei3lcrr/providers/Microsoft.Compute/disks/pvc-4f2c37ac-8825-40bb-a55f-dc9dd2622455"} I0119 02:51:09.672700 1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-4f2c37ac-8825-40bb-a55f-dc9dd2622455-k8s-agentpool1-29517166-vmss000001-attachment) I0119 02:51:10.918657 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:51:10.918746 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:51:10.918813 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:51:10.918826 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:51:10.919015 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:51:10.919067 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:51:10.930155 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:51:15.185154 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I0119 02:51:15.185176 1 utils.go:79] GRPC request: {} I0119 02:51:15.185214 1 utils.go:85] GRPC response: {"ready":{"value":true}} I0119 02:51:20.213617 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:51:20.213681 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="20a8e59f-97a4-11ed-be91-002248822c79" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-55fcbf1e-e8a1-4e65-8eee-890ae020006d" "latency"=10541180174 I0119 02:51:20.213703 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="20a8e59f-97a4-11ed-be91-002248822c79" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=10541305775 I0119 02:51:20.218473 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:51:20.218520 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="20a8f067-97a4-11ed-be91-002248822c79" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-4f2c37ac-8825-40bb-a55f-dc9dd2622455" "latency"=10545775318 I0119 02:51:20.218541 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="20a8f067-97a4-11ed-be91-002248822c79" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=10545866719 I0119 02:51:20.816661 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun2 by sdd under /dev/disk/azure/scsi1/ I0119 02:51:20.816708 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun2. perfProfile none accountType I0119 02:51:20.816722 1 utils.go:85] GRPC response: {} I0119 02:51:20.816830 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sde under /dev/disk/azure/scsi1/ ... skipping 92 lines ... I0119 02:52:10.939555 1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume I0119 02:52:10.939572 1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-accf5611-c609-40f8-bbf2-08f6dca11872/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-accf5611-c609-40f8-bbf2-08f6dca11872","csi.storage.k8s.io/pvc/name":"test.csi.azure.comcxzf5","csi.storage.k8s.io/pvc/namespace":"snapshotting-8130","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1674093937787-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-iei3lcrr/providers/Microsoft.Compute/disks/pvc-accf5611-c609-40f8-bbf2-08f6dca11872"} I0119 02:52:10.939732 1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-accf5611-c609-40f8-bbf2-08f6dca11872-k8s-agentpool1-29517166-vmss000001-attachment) I0119 02:52:15.185522 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I0119 02:52:15.185540 1 utils.go:79] GRPC request: {} I0119 02:52:15.185575 1 utils.go:85] GRPC response: {"ready":{"value":true}} I0119 02:52:21.817800 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:52:21.817878 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="452d8c1c-97a4-11ed-be91-002248822c79" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-accf5611-c609-40f8-bbf2-08f6dca11872" "latency"=10878072240 I0119 02:52:21.817921 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="452d8c1c-97a4-11ed-be91-002248822c79" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=10878198940 I0119 02:52:23.586922 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/ I0119 02:52:23.586965 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType I0119 02:52:23.586990 1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-accf5611-c609-40f8-bbf2-08f6dca11872/globalmount with mount options([]) I0119 02:52:23.586998 1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0]) ... skipping 64 lines ... I0119 02:52:50.125134 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities I0119 02:52:50.125150 1 utils.go:79] GRPC request: {} I0119 02:52:50.125173 1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]} I0119 02:52:50.125930 1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume I0119 02:52:50.125947 1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-accf5611-c609-40f8-bbf2-08f6dca11872/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-accf5611-c609-40f8-bbf2-08f6dca11872","csi.storage.k8s.io/pvc/name":"test.csi.azure.comcxzf5","csi.storage.k8s.io/pvc/namespace":"snapshotting-8130","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1674093937787-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-iei3lcrr/providers/Microsoft.Compute/disks/pvc-accf5611-c609-40f8-bbf2-08f6dca11872"} I0119 02:52:50.126144 1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-accf5611-c609-40f8-bbf2-08f6dca11872-k8s-agentpool1-29517166-vmss000001-attachment) I0119 02:53:00.274593 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:53:00.274671 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="5c88eb49-97a4-11ed-be91-002248822c79" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-accf5611-c609-40f8-bbf2-08f6dca11872" "latency"=10148454149 I0119 02:53:00.274732 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="5c88eb49-97a4-11ed-be91-002248822c79" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=10148571049 I0119 02:53:02.003749 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/ I0119 02:53:02.003804 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType I0119 02:53:02.003834 1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-accf5611-c609-40f8-bbf2-08f6dca11872/globalmount with mount options([]) I0119 02:53:02.003843 1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0]) ... skipping 70 lines ... I0119 02:53:37.972167 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-29517166-vmss000001/status 200 OK in 7 milliseconds I0119 02:53:37.972882 1 azuredisk_v2.go:565] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="790c7aec-97a4-11ed-be91-002248822c79" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=8340388 I0119 02:53:40.922266 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:53:40.922342 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:53:40.922353 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:53:40.922380 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:53:40.922412 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:53:40.932644 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:53:45.186026 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I0119 02:53:45.186050 1 utils.go:79] GRPC request: {} I0119 02:53:45.186089 1 utils.go:85] GRPC response: {"ready":{"value":true}} I0119 02:53:48.718634 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:53:48.718710 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="75bfe6ba-97a4-11ed-be91-002248822c79" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-3c0678e4-d398-404d-8392-1aaa4b107c1b" "latency"=16289105691 I0119 02:53:48.718749 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="75bfe6ba-97a4-11ed-be91-002248822c79" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=16289246293 I0119 02:53:50.516014 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdc under /dev/disk/azure/scsi1/ I0119 02:53:50.516072 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType I0119 02:53:50.516106 1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun1 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-3c0678e4-d398-404d-8392-1aaa4b107c1b/globalmount with mount options([]) I0119 02:53:50.516120 1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun1" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun1]) ... skipping 71 lines ... I0119 02:54:37.972040 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-29517166-vmss000001/status 200 OK in 7 milliseconds I0119 02:54:37.972228 1 azuredisk_v2.go:565] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="9ccfbb47-97a4-11ed-be91-002248822c79" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=7828540 I0119 02:54:40.924144 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:54:40.924295 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:54:40.924341 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:54:40.924410 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:54:40.924502 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:54:40.934300 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:54:45.185739 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I0119 02:54:45.185766 1 utils.go:79] GRPC request: {} I0119 02:54:45.185816 1 utils.go:85] GRPC response: {"ready":{"value":true}} I0119 02:54:48.841378 1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: Watch close - *v1beta2.AzDriverNode total 60 items received I0119 02:54:48.844453 1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azdrivernodes?allowWatchBookmarks=true&resourceVersion=28368&timeout=8m2s&timeoutSeconds=482&watch=true 200 OK in 2 milliseconds ... skipping 2 lines ... I0119 02:54:54.841408 1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: Watch close - *v1beta2.AzVolume total 189 items received I0119 02:54:54.845437 1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumes?allowWatchBookmarks=true&resourceVersion=28340&timeout=6m28s&timeoutSeconds=388&watch=true 200 OK in 3 milliseconds I0119 02:55:07.964378 1 azuredisk_v2.go:578] "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-19T02:55:07Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="aeb15bbe-97a4-11ed-be91-002248822c79" I0119 02:55:07.971880 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-29517166-vmss000001/status 200 OK in 7 milliseconds I0119 02:55:07.972225 1 azuredisk_v2.go:565] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="aeb15bbe-97a4-11ed-be91-002248822c79" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=7894527 I0119 02:55:10.926501 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:55:10.926582 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:55:10.926581 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:55:10.926593 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:55:10.926618 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:55:10.935067 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:55:15.185596 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I0119 02:55:15.185619 1 utils.go:79] GRPC request: {} ... skipping 2 lines ... I0119 02:55:37.972117 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-29517166-vmss000001/status 200 OK in 6 milliseconds I0119 02:55:37.972264 1 azuredisk_v2.go:565] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="c0931d61-97a4-11ed-be91-002248822c79" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=7152818 I0119 02:55:40.929282 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:55:40.929309 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:55:40.929276 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:55:40.929392 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:55:40.929369 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:55:40.935823 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:55:45.185634 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I0119 02:55:45.185660 1 utils.go:79] GRPC request: {} I0119 02:55:45.185708 1 utils.go:85] GRPC response: {"ready":{"value":true}} I0119 02:56:07.964810 1 azuredisk_v2.go:578] "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-19T02:56:07Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="d274b2b9-97a4-11ed-be91-002248822c79" I0119 02:56:07.971312 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-29517166-vmss000001/status 200 OK in 6 milliseconds I0119 02:56:07.971711 1 azuredisk_v2.go:565] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="d274b2b9-97a4-11ed-be91-002248822c79" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=6948906 I0119 02:56:10.931687 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:56:10.931754 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:56:10.931778 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:56:10.931811 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:56:10.931764 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:56:10.936084 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync E0119 02:56:13.207009 1 conditionwaiter.go:50] "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [context canceled]" "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="8e0eba2f-97a4-11ed-be91-002248822c79" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-9af0f507-796b-45fe-b638-aa6b2d536fff" "latency"=119995406790 E0119 02:56:13.207129 1 crdprovisioner.go:743] "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [context canceled]" "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="8e0eba2f-97a4-11ed-be91-002248822c79" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=119995702794 E0119 02:56:13.207146 1 utils.go:83] GRPC error: rpc error: code = Internal desc = failed to wait for volume (/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-iei3lcrr/providers/Microsoft.Compute/disks/pvc-9af0f507-796b-45fe-b638-aa6b2d536fff) to be attached to node (k8s-agentpool1-29517166-vmss000001): context canceled I0119 02:56:13.794460 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities I0119 02:56:13.794485 1 utils.go:79] GRPC request: {} I0119 02:56:13.794531 1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]} I0119 02:56:13.798078 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities I0119 02:56:13.798102 1 utils.go:79] GRPC request: {} I0119 02:56:13.798143 1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]} I0119 02:56:13.798790 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities I0119 02:56:13.798809 1 utils.go:79] GRPC request: {} I0119 02:56:13.798836 1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]} I0119 02:56:13.799562 1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume I0119 02:56:13.799578 1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-9af0f507-796b-45fe-b638-aa6b2d536fff/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-9af0f507-796b-45fe-b638-aa6b2d536fff","csi.storage.k8s.io/pvc/name":"pvc-bnksh","csi.storage.k8s.io/pvc/namespace":"provisioning-7897","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1674093937787-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-iei3lcrr/providers/Microsoft.Compute/disks/pvc-9af0f507-796b-45fe-b638-aa6b2d536fff"} I0119 02:56:13.799752 1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-9af0f507-796b-45fe-b638-aa6b2d536fff-k8s-agentpool1-29517166-vmss000001-attachment) I0119 02:56:14.999825 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:56:14.999891 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="d5ef0b96-97a4-11ed-be91-002248822c79" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-9af0f507-796b-45fe-b638-aa6b2d536fff" "latency"=1200082083 I0119 02:56:14.999913 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="d5ef0b96-97a4-11ed-be91-002248822c79" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=1200182385 I0119 02:56:15.185914 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I0119 02:56:15.185943 1 utils.go:79] GRPC request: {} I0119 02:56:15.185991 1 utils.go:85] GRPC response: {"ready":{"value":true}} I0119 02:56:16.723656 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/ ... skipping 86 lines ... I0119 02:57:38.834304 1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: Watch close - *v1beta2.AzVolumeAttachment total 166 items received I0119 02:57:38.838088 1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumeattachments?allowWatchBookmarks=true&resourceVersion=29248&timeout=7m30s&timeoutSeconds=450&watch=true 200 OK in 3 milliseconds I0119 02:57:40.937207 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:57:40.937245 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:57:40.937263 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:57:40.937199 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:57:40.937287 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:57:40.937226 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:57:44.964939 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:57:44.965010 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="07b3e088-97a5-11ed-be91-002248822c79" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-0bb7c0f5-406d-41dc-9704-c94cffe3c5d9" "latency"=7666855892 I0119 02:57:44.965057 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="07b3e088-97a5-11ed-be91-002248822c79" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=7666985795 I0119 02:57:45.185397 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I0119 02:57:45.185419 1 utils.go:79] GRPC request: {} I0119 02:57:45.185459 1 utils.go:85] GRPC response: {"ready":{"value":true}} I0119 02:57:46.688657 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/ ... skipping 110 lines ... I0119 02:58:07.963810 1 azuredisk_v2.go:578] "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-19T02:58:07Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="19fb17f0-97a5-11ed-be91-002248822c79" I0119 02:58:07.973151 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-29517166-vmss000001/status 200 OK in 9 milliseconds I0119 02:58:07.973341 1 azuredisk_v2.go:565] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="19fb17f0-97a5-11ed-be91-002248822c79" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=9573131 I0119 02:58:10.940355 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:58:10.940436 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:58:10.940474 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:58:10.940513 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0119 02:58:10.940534 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0119 02:58:10.940556 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0119 02:58:14.320500 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:58:14.320565 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="19f40ed6-97a5-11ed-be91-002248822c79" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-fc5d9aa0-a895-4142-a6c9-1bfa9991d1cf" "latency"=6402829242 I0119 02:58:14.320588 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="19f40ed6-97a5-11ed-be91-002248822c79" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=6402935944 I0119 02:58:14.918996 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdd under /dev/disk/azure/scsi1/ I0119 02:58:14.919052 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType I0119 02:58:14.919093 1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun1 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-fc5d9aa0-a895-4142-a6c9-1bfa9991d1cf/globalmount with mount options([]) I0119 02:58:14.919110 1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun1" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun1]) ... skipping 98 lines ... I0119 02:59:18.844870 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities I0119 02:59:18.844889 1 utils.go:79] GRPC request: {} I0119 02:59:18.844918 1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]} I0119 02:59:18.845687 1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume I0119 02:59:18.845704 1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-1caac5d6-fd5f-4d44-882b-786f34d360a1/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-1caac5d6-fd5f-4d44-882b-786f34d360a1","csi.storage.k8s.io/pvc/name":"pvc-azuredisk","csi.storage.k8s.io/pvc/namespace":"default","requestedsizegib":"10","skuName":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1674093937787-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-iei3lcrr/providers/Microsoft.Compute/disks/pvc-1caac5d6-fd5f-4d44-882b-786f34d360a1"} I0119 02:59:18.845868 1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-1caac5d6-fd5f-4d44-882b-786f34d360a1-k8s-agentpool1-29517166-vmss000001-attachment) I0119 02:59:30.127409 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 02:59:30.127494 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="443ad7a9-97a5-11ed-be91-002248822c79" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-1caac5d6-fd5f-4d44-882b-786f34d360a1" "latency"=11281534047 I0119 02:59:30.127528 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="443ad7a9-97a5-11ed-be91-002248822c79" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=11281668749 I0119 02:59:31.792239 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/ I0119 02:59:31.792287 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType StandardSSD_LRS I0119 02:59:31.792316 1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-1caac5d6-fd5f-4d44-882b-786f34d360a1/globalmount with mount options([]) I0119 02:59:31.792324 1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0]) ... skipping 94 lines ... I0119 03:01:25.905858 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities I0119 03:01:25.905882 1 utils.go:79] GRPC request: {} I0119 03:01:25.905925 1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]} I0119 03:01:25.906690 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetVolumeStats I0119 03:01:25.906706 1 utils.go:79] GRPC request: {"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-iei3lcrr/providers/Microsoft.Compute/disks/pvc-1caac5d6-fd5f-4d44-882b-786f34d360a1","volume_path":"/var/lib/kubelet/pods/8b963894-8d3d-4fbc-80e1-56f5a735b455/volumes/kubernetes.io~csi/pvc-1caac5d6-fd5f-4d44-882b-786f34d360a1/mount"} I0119 03:01:25.906770 1 utils.go:85] GRPC response: {"usage":[{"available":10447216640,"total":10464022528,"unit":1,"used":28672},{"available":655348,"total":655360,"unit":2,"used":12}]} I0119 03:01:27.557957 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0119 03:01:27.558027 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-29517166-vmss000001" "disk.csi.azure.com/request-id"="8d434ce9-97a5-11ed-be91-002248822c79" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-703b4caf-6213-4ef0-b4ab-2047d8d1d45f" "latency"=6182964179 I0119 03:01:27.558050 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="8d434ce9-97a5-11ed-be91-002248822c79" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=6183091180 I0119 03:01:28.162106 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdd under /dev/disk/azure/scsi1/ I0119 03:01:28.162154 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType StandardSSD_LRS I0119 03:01:28.162183 1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun1 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-703b4caf-6213-4ef0-b4ab-2047d8d1d45f/globalmount with mount options([]) I0119 03:01:28.162191 1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun1" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun1]) ... skipping 449 lines ... cloudprovider_azure_op_duration_seconds_bucket{request="azuredisk_csi_driver_controller_unpublish_volume",resource_group="kubetest-iei3lcrr",source="test.csi.azure.com",subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e",le="300"} 211 cloudprovider_azure_op_duration_seconds_bucket{request="azuredisk_csi_driver_controller_unpublish_volume",resource_group="kubetest-iei3lcrr",source="test.csi.azure.com",subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e",le="600"} 216 cloudprovider_azure_op_duration_seconds_bucket{request="azuredisk_csi_driver_controller_unpublish_volume",resource_group="kubetest-iei3lcrr",source="test.csi.azure.com",subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e",le="1200"} 216 cloudprovider_azure_op_duration_seconds_bucket{request="azuredisk_csi_driver_controller_unpublish_volume",resource_group="kubetest-iei3lcrr",source="test.csi.azure.com",subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e",le="+Inf"} 216 cloudprovider_azure_op_duration_seconds_sum{request="azuredisk_csi_driver_controller_unpublish_volume",resource_group="kubetest-iei3lcrr",source="test.csi.azure.com",subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e"} 9699.481177821997 cloudprovider_azure_op_duration_seconds_count{request="azuredisk_csi_driver_controller_unpublish_volume",resource_group="kubetest-iei3lcrr",source="test.csi.azure.com",subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e"} 216 # HELP cloudprovider_azure_op_failure_count [ALPHA] Number of failed Azure service operations # TYPE cloudprovider_azure_op_failure_count counter cloudprovider_azure_op_failure_count{request="azuredisk_csi_driver_attach_volume_latency",resource_group="kubetest-iei3lcrr",source="test.csi.azure.com",subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e"} 2 cloudprovider_azure_op_failure_count{request="azuredisk_csi_driver_controller_unpublish_volume",resource_group="kubetest-iei3lcrr",source="test.csi.azure.com",subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e"} 5 # HELP disabled_metric_total [ALPHA] The count of disabled metrics. # TYPE disabled_metric_total counter disabled_metric_total 0 ... skipping 68 lines ... # HELP go_gc_heap_objects_objects Number of objects, live or unswept, occupying heap memory. # TYPE go_gc_heap_objects_objects gauge go_gc_heap_objects_objects 80096 # HELP go_gc_heap_tiny_allocs_objects_total Count of small allocations that are packed together into blocks. These allocations are counted separately from other allocations because each individual allocation is not tracked by the runtime, only their block. Each block is already accounted for in allocs-by-size and frees-by-size. # TYPE go_gc_heap_tiny_allocs_objects_total counter go_gc_heap_tiny_allocs_objects_total 482879 # HELP go_gc_limiter_last_enabled_gc_cycle GC cycle the last time the GC CPU limiter was enabled. This metric is useful for diagnosing the root cause of an out-of-memory error, because the limiter trades memory for CPU time when the GC's CPU time gets too high. This is most likely to occur with use of SetMemoryLimit. The first GC cycle is cycle 1, so a value of 0 indicates that it was never enabled. # TYPE go_gc_limiter_last_enabled_gc_cycle gauge go_gc_limiter_last_enabled_gc_cycle 0 # HELP go_gc_pauses_seconds Distribution individual GC-related stop-the-world pause latencies. # TYPE go_gc_pauses_seconds histogram go_gc_pauses_seconds_bucket{le="9.999999999999999e-10"} 0 go_gc_pauses_seconds_bucket{le="9.999999999999999e-09"} 0 ... skipping 175 lines ... # HELP process_virtual_memory_max_bytes Maximum amount of virtual memory available in bytes. # TYPE process_virtual_memory_max_bytes gauge process_virtual_memory_max_bytes 1.8446744073709552e+19 # HELP registered_metric_total [ALPHA] The count of registered metrics broken by stability level and deprecation version. # TYPE registered_metric_total counter registered_metric_total{deprecated_version="",stability_level="ALPHA"} 16 make: *** [Makefile:341: e2e-test] Error 1 2023/01/19 03:02:59 process.go:155: Step 'make e2e-test' finished in 1h10m7.020224343s 2023/01/19 03:02:59 aksengine_helpers.go:425: downloading /root/tmp1374176058/log-dump.sh from https://raw.githubusercontent.com/kubernetes-sigs/cloud-provider-azure/master/hack/log-dump/log-dump.sh 2023/01/19 03:02:59 util.go:70: curl https://raw.githubusercontent.com/kubernetes-sigs/cloud-provider-azure/master/hack/log-dump/log-dump.sh 2023/01/19 03:02:59 process.go:153: Running: chmod +x /root/tmp1374176058/log-dump.sh 2023/01/19 03:02:59 process.go:155: Step 'chmod +x /root/tmp1374176058/log-dump.sh' finished in 3.503072ms 2023/01/19 03:02:59 aksengine_helpers.go:425: downloading /root/tmp1374176058/log-dump-daemonset.yaml from https://raw.githubusercontent.com/kubernetes-sigs/cloud-provider-azure/master/hack/log-dump/log-dump-daemonset.yaml ... skipping 76 lines ... ssh key file /root/.ssh/id_rsa does not exist. Exiting. 2023/01/19 03:03:59 process.go:155: Step 'bash -c /root/tmp1374176058/win-ci-logs-collector.sh kubetest-iei3lcrr.westeurope.cloudapp.azure.com /root/tmp1374176058 /root/.ssh/id_rsa' finished in 3.773961ms 2023/01/19 03:03:59 aksengine.go:1141: Deleting resource group: kubetest-iei3lcrr. 2023/01/19 03:11:10 process.go:96: Saved XML output to /logs/artifacts/junit_runner.xml. 2023/01/19 03:11:10 process.go:153: Running: bash -c . hack/lib/version.sh && KUBE_ROOT=. kube::version::get_version_vars && echo "${KUBE_GIT_VERSION-}" 2023/01/19 03:11:12 process.go:155: Step 'bash -c . hack/lib/version.sh && KUBE_ROOT=. kube::version::get_version_vars && echo "${KUBE_GIT_VERSION-}"' finished in 1.619532998s 2023/01/19 03:11:12 main.go:328: Something went wrong: encountered 1 errors: [error during make e2e-test: exit status 2] + EXIT_VALUE=1 + set +o xtrace Cleaning up after docker in docker. ================================================================================ Cleaning up after docker 72900636fcc9 ... skipping 4 lines ...