pull-azuredisk-csi-driver-e2e-capz-windows-2019-mainv2 #1615394037500481536 Results

Error lines from build-log.txt

Raw build-log.txt
... skipping 789 lines ...
certificate.cert-manager.io "selfsigned-cert" deleted
# Create secret for AzureClusterIdentity
./hack/create-identity-secret.sh
make[2]: Entering directory '/home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure'
make[2]: Nothing to be done for 'kubectl'.
make[2]: Leaving directory '/home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure'
Error from server (NotFound): secrets "cluster-identity-secret" not found
secret/cluster-identity-secret created
secret/cluster-identity-secret labeled
# Create customized cloud provider configs
./hack/create-custom-cloud-provider-config.sh
make[2]: Entering directory '/home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure'
make[2]: Nothing to be done for 'kubectl'.
... skipping 1123 lines ...
                    type: string
                type: object
                oneOf:
                - required: ["persistentVolumeClaimName"]
                - required: ["volumeSnapshotContentName"]
              volumeSnapshotClassName:
                description: 'VolumeSnapshotClassName is the name of the VolumeSnapshotClass requested by the VolumeSnapshot. VolumeSnapshotClassName may be left nil to indicate that the default SnapshotClass should be used. A given cluster may have multiple default Volume SnapshotClasses: one default per CSI Driver. If a VolumeSnapshot does not specify a SnapshotClass, VolumeSnapshotSource will be checked to figure out what the associated CSI Driver is, and the default VolumeSnapshotClass associated with that CSI Driver will be used. If more than one VolumeSnapshotClass exist for a given CSI Driver and more than one have been marked as default, CreateSnapshot will fail and generate an event. Empty string is not allowed for this field.'
                type: string
            required:
            - source
            type: object
          status:
            description: status represents the current information of a snapshot. Consumers must verify binding between VolumeSnapshot and VolumeSnapshotContent objects is successful (by validating that both VolumeSnapshot and VolumeSnapshotContent point at each other) before using this object.
... skipping 2 lines ...
                description: 'boundVolumeSnapshotContentName is the name of the VolumeSnapshotContent object to which this VolumeSnapshot object intends to bind to. If not specified, it indicates that the VolumeSnapshot object has not been successfully bound to a VolumeSnapshotContent object yet. NOTE: To avoid possible security issues, consumers must verify binding between VolumeSnapshot and VolumeSnapshotContent objects is successful (by validating that both VolumeSnapshot and VolumeSnapshotContent point at each other) before using this object.'
                type: string
              creationTime:
                description: creationTime is the timestamp when the point-in-time snapshot is taken by the underlying storage system. In dynamic snapshot creation case, this field will be filled in by the snapshot controller with the "creation_time" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "creation_time" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. If not specified, it may indicate that the creation time of the snapshot is unknown.
                format: date-time
                type: string
              error:
                description: error is the last observed error during snapshot creation, if any. This field could be helpful to upper level controllers(i.e., application controller) to decide whether they should continue on waiting for the snapshot to be created based on the type of error reported. The snapshot controller will keep retrying when an error occurs during the snapshot creation. Upon success, this error field will be cleared.
                properties:
                  message:
                    description: 'message is a string detailing the encountered error during snapshot creation if specified. NOTE: message may be logged, and it should not contain sensitive information.'
                    type: string
                  time:
                    description: time is the timestamp when the error was encountered.
                    format: date-time
                    type: string
                type: object
              readyToUse:
                description: readyToUse indicates if the snapshot is ready to be used to restore a volume. In dynamic snapshot creation case, this field will be filled in by the snapshot controller with the "ready_to_use" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "ready_to_use" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it, otherwise, this field will be set to "True". If not specified, it means the readiness of a snapshot is unknown.
                type: boolean
              restoreSize:
                type: string
                description: restoreSize represents the minimum size of volume required to create a volume from this snapshot. In dynamic snapshot creation case, this field will be filled in by the snapshot controller with the "size_bytes" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "size_bytes" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. When restoring a volume from this snapshot, the size of the volume MUST NOT be smaller than the restoreSize if it is specified, otherwise the restoration will fail. If not specified, it indicates that the size is unknown.
                pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
                x-kubernetes-int-or-string: true
            type: object
        required:
        - spec
        type: object
... skipping 60 lines ...
                    type: string
                  volumeSnapshotContentName:
                    description: volumeSnapshotContentName specifies the name of a pre-existing VolumeSnapshotContent object representing an existing volume snapshot. This field should be set if the snapshot already exists and only needs a representation in Kubernetes. This field is immutable.
                    type: string
                type: object
              volumeSnapshotClassName:
                description: 'VolumeSnapshotClassName is the name of the VolumeSnapshotClass requested by the VolumeSnapshot. VolumeSnapshotClassName may be left nil to indicate that the default SnapshotClass should be used. A given cluster may have multiple default Volume SnapshotClasses: one default per CSI Driver. If a VolumeSnapshot does not specify a SnapshotClass, VolumeSnapshotSource will be checked to figure out what the associated CSI Driver is, and the default VolumeSnapshotClass associated with that CSI Driver will be used. If more than one VolumeSnapshotClass exist for a given CSI Driver and more than one have been marked as default, CreateSnapshot will fail and generate an event. Empty string is not allowed for this field.'
                type: string
            required:
            - source
            type: object
          status:
            description: status represents the current information of a snapshot. Consumers must verify binding between VolumeSnapshot and VolumeSnapshotContent objects is successful (by validating that both VolumeSnapshot and VolumeSnapshotContent point at each other) before using this object.
... skipping 2 lines ...
                description: 'boundVolumeSnapshotContentName is the name of the VolumeSnapshotContent object to which this VolumeSnapshot object intends to bind to. If not specified, it indicates that the VolumeSnapshot object has not been successfully bound to a VolumeSnapshotContent object yet. NOTE: To avoid possible security issues, consumers must verify binding between VolumeSnapshot and VolumeSnapshotContent objects is successful (by validating that both VolumeSnapshot and VolumeSnapshotContent point at each other) before using this object.'
                type: string
              creationTime:
                description: creationTime is the timestamp when the point-in-time snapshot is taken by the underlying storage system. In dynamic snapshot creation case, this field will be filled in by the snapshot controller with the "creation_time" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "creation_time" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. If not specified, it may indicate that the creation time of the snapshot is unknown.
                format: date-time
                type: string
              error:
                description: error is the last observed error during snapshot creation, if any. This field could be helpful to upper level controllers(i.e., application controller) to decide whether they should continue on waiting for the snapshot to be created based on the type of error reported. The snapshot controller will keep retrying when an error occurs during the snapshot creation. Upon success, this error field will be cleared.
                properties:
                  message:
                    description: 'message is a string detailing the encountered error during snapshot creation if specified. NOTE: message may be logged, and it should not contain sensitive information.'
                    type: string
                  time:
                    description: time is the timestamp when the error was encountered.
                    format: date-time
                    type: string
                type: object
              readyToUse:
                description: readyToUse indicates if the snapshot is ready to be used to restore a volume. In dynamic snapshot creation case, this field will be filled in by the snapshot controller with the "ready_to_use" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "ready_to_use" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it, otherwise, this field will be set to "True". If not specified, it means the readiness of a snapshot is unknown.
                type: boolean
              restoreSize:
                type: string
                description: restoreSize represents the minimum size of volume required to create a volume from this snapshot. In dynamic snapshot creation case, this field will be filled in by the snapshot controller with the "size_bytes" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "size_bytes" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. When restoring a volume from this snapshot, the size of the volume MUST NOT be smaller than the restoreSize if it is specified, otherwise the restoration will fail. If not specified, it indicates that the size is unknown.
                pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
                x-kubernetes-int-or-string: true
            type: object
        required:
        - spec
        type: object
... skipping 254 lines ...
            description: status represents the current information of a snapshot.
            properties:
              creationTime:
                description: creationTime is the timestamp when the point-in-time snapshot is taken by the underlying storage system. In dynamic snapshot creation case, this field will be filled in by the CSI snapshotter sidecar with the "creation_time" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "creation_time" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. If not specified, it indicates the creation time is unknown. The format of this field is a Unix nanoseconds time encoded as an int64. On Unix, the command `date +%s%N` returns the current time in nanoseconds since 1970-01-01 00:00:00 UTC.
                format: int64
                type: integer
              error:
                description: error is the last observed error during snapshot creation, if any. Upon success after retry, this error field will be cleared.
                properties:
                  message:
                    description: 'message is a string detailing the encountered error during snapshot creation if specified. NOTE: message may be logged, and it should not contain sensitive information.'
                    type: string
                  time:
                    description: time is the timestamp when the error was encountered.
                    format: date-time
                    type: string
                type: object
              readyToUse:
                description: readyToUse indicates if a snapshot is ready to be used to restore a volume. In dynamic snapshot creation case, this field will be filled in by the CSI snapshotter sidecar with the "ready_to_use" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "ready_to_use" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it, otherwise, this field will be set to "True". If not specified, it means the readiness of a snapshot is unknown.
                type: boolean
              restoreSize:
                description: restoreSize represents the complete size of the snapshot in bytes. In dynamic snapshot creation case, this field will be filled in by the CSI snapshotter sidecar with the "size_bytes" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "size_bytes" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. When restoring a volume from this snapshot, the size of the volume MUST NOT be smaller than the restoreSize if it is specified, otherwise the restoration will fail. If not specified, it indicates that the size is unknown.
                format: int64
                minimum: 0
                type: integer
              snapshotHandle:
                description: snapshotHandle is the CSI "snapshot_id" of a snapshot on the underlying storage system. If not specified, it indicates that dynamic snapshot creation has either failed or it is still in progress.
                type: string
            type: object
        required:
        - spec
        type: object
    served: true
... skipping 108 lines ...
            description: status represents the current information of a snapshot.
            properties:
              creationTime:
                description: creationTime is the timestamp when the point-in-time snapshot is taken by the underlying storage system. In dynamic snapshot creation case, this field will be filled in by the CSI snapshotter sidecar with the "creation_time" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "creation_time" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. If not specified, it indicates the creation time is unknown. The format of this field is a Unix nanoseconds time encoded as an int64. On Unix, the command `date +%s%N` returns the current time in nanoseconds since 1970-01-01 00:00:00 UTC.
                format: int64
                type: integer
              error:
                description: error is the last observed error during snapshot creation, if any. Upon success after retry, this error field will be cleared.
                properties:
                  message:
                    description: 'message is a string detailing the encountered error during snapshot creation if specified. NOTE: message may be logged, and it should not contain sensitive information.'
                    type: string
                  time:
                    description: time is the timestamp when the error was encountered.
                    format: date-time
                    type: string
                type: object
              readyToUse:
                description: readyToUse indicates if a snapshot is ready to be used to restore a volume. In dynamic snapshot creation case, this field will be filled in by the CSI snapshotter sidecar with the "ready_to_use" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "ready_to_use" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it, otherwise, this field will be set to "True". If not specified, it means the readiness of a snapshot is unknown.
                type: boolean
              restoreSize:
                description: restoreSize represents the complete size of the snapshot in bytes. In dynamic snapshot creation case, this field will be filled in by the CSI snapshotter sidecar with the "size_bytes" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "size_bytes" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. When restoring a volume from this snapshot, the size of the volume MUST NOT be smaller than the restoreSize if it is specified, otherwise the restoration will fail. If not specified, it indicates that the size is unknown.
                format: int64
                minimum: 0
                type: integer
              snapshotHandle:
                description: snapshotHandle is the CSI "snapshot_id" of a snapshot on the underlying storage system. If not specified, it indicates that dynamic snapshot creation has either failed or it is still in progress.
                type: string
            type: object
        required:
        - spec
        type: object
    served: true
... skipping 359 lines ...
            - volumeName
            - volume_context
            - volume_id
            type: object
          status:
            description: status represents the current state of AzVolumeAttachment.
              includes error, state, and attachment status Required
            properties:
              detail:
                description: Status summarizes the current attachment state of the
                  volume attachment Nil Status indicates that the volume has not yet
                  been attached to the node
                properties:
... skipping 7 lines ...
                  role:
                    description: The current attachment role.
                    type: string
                required:
                - role
                type: object
              error:
                description: Error occurred during attach/detach of volume
                properties:
                  code:
                    type: string
                  message:
                    type: string
                  parameters:
... skipping 90 lines ...
            - volumeName
            - volume_context
            - volume_id
            type: object
          status:
            description: status represents the current state of AzVolumeAttachment.
              includes error, state, and attachment status
            properties:
              annotation:
                additionalProperties:
                  type: string
                description: Annotations contains additional resource information
                  to guide driver actions
... skipping 13 lines ...
                  role:
                    description: The current attachment role.
                    type: string
                required:
                - role
                type: object
              error:
                description: Error occurred during attach/detach of volume
                properties:
                  code:
                    type: string
                  message:
                    type: string
                  parameters:
... skipping 169 lines ...
            - maxMountReplicaCount
            - volumeCapability
            - volumeName
            type: object
          status:
            description: status represents the current state of AzVolume. includes
              error, state, and volume status
            properties:
              detail:
                description: Current status detail of the AzVolume Nil detail indicates
                  that the volume has not been created
                properties:
                  accessible_topology:
... skipping 28 lines ...
                    type: string
                required:
                - capacity_bytes
                - node_expansion_required
                - volume_id
                type: object
              error:
                description: Error occurred during creation/deletion of volume
                properties:
                  code:
                    type: string
                  message:
                    type: string
                  parameters:
... skipping 154 lines ...
            - maxMountReplicaCount
            - volumeCapability
            - volumeName
            type: object
          status:
            description: status represents the current state of AzVolume. includes
              error, state, and volume status
            properties:
              annotation:
                additionalProperties:
                  type: string
                description: Annotations contains additional resource information
                  to guide driver actions
... skipping 34 lines ...
                    type: string
                required:
                - capacity_bytes
                - node_expansion_required
                - volume_id
                type: object
              error:
                description: Error occurred during creation/deletion of volume
                properties:
                  code:
                    type: string
                  message:
                    type: string
                  parameters:
... skipping 831 lines ...
          image: "mcr.microsoft.com/oss/kubernetes-csi/csi-resizer:v1.5.0"
          args:
            - "-csi-address=$(ADDRESS)"
            - "-v=2"
            - "-leader-election"
            - "--leader-election-namespace=kube-system"
            - '-handle-volume-inuse-error=false'
            - '-feature-gates=RecoverVolumeExpansionFailure=true'
            - "-timeout=240s"
          env:
            - name: ADDRESS
              value: /csi/csi.sock
          volumeMounts:
... skipping 283 lines ...
Jan 17 17:38:04.615: INFO: >>> kubeConfig: /home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure/kubeconfig
[1mSTEP:[0m Building a namespace api object, basename azuredisk [38;5;243m01/17/23 17:38:04.617[0m
[1mSTEP:[0m Waiting for a default service account to be provisioned in namespace [38;5;243m01/17/23 17:38:04.985[0m
[1mSTEP:[0m Waiting for kube-root-ca.crt to be provisioned in namespace [38;5;243m01/17/23 17:38:05.095[0m
I0117 17:38:05.208330   43786 azuredisk_driver.go:52] Using azure disk driver: disk.csi.azure.com
[1mSTEP:[0m deploying the pod [38;5;243m01/17/23 17:38:05.287[0m
[1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/17/23 17:38:05.352[0m
Jan 17 17:38:05.353: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-v78b4" in namespace "azuredisk-7230" to be "Succeeded or Failed"
Jan 17 17:38:05.409: INFO: Pod "azuredisk-volume-tester-v78b4": Phase="Pending", Reason="", readiness=false. Elapsed: 56.067702ms
Jan 17 17:38:07.467: INFO: Pod "azuredisk-volume-tester-v78b4": Phase="Pending", Reason="", readiness=false. Elapsed: 2.114217004s
Jan 17 17:38:09.474: INFO: Pod "azuredisk-volume-tester-v78b4": Phase="Pending", Reason="", readiness=false. Elapsed: 4.121577512s
Jan 17 17:38:11.467: INFO: Pod "azuredisk-volume-tester-v78b4": Phase="Pending", Reason="", readiness=false. Elapsed: 6.11450939s
Jan 17 17:38:13.467: INFO: Pod "azuredisk-volume-tester-v78b4": Phase="Pending", Reason="", readiness=false. Elapsed: 8.114688707s
Jan 17 17:38:15.467: INFO: Pod "azuredisk-volume-tester-v78b4": Phase="Pending", Reason="", readiness=false. Elapsed: 10.114023068s
... skipping 110 lines ...
Jan 17 17:41:57.471: INFO: Pod "azuredisk-volume-tester-v78b4": Phase="Running", Reason="", readiness=true. Elapsed: 3m52.11841781s
Jan 17 17:41:59.472: INFO: Pod "azuredisk-volume-tester-v78b4": Phase="Running", Reason="", readiness=true. Elapsed: 3m54.11899189s
Jan 17 17:42:01.472: INFO: Pod "azuredisk-volume-tester-v78b4": Phase="Running", Reason="", readiness=false. Elapsed: 3m56.119756926s
Jan 17 17:42:03.471: INFO: Pod "azuredisk-volume-tester-v78b4": Phase="Running", Reason="", readiness=false. Elapsed: 3m58.118664776s
Jan 17 17:42:05.470: INFO: Pod "azuredisk-volume-tester-v78b4": Phase="Succeeded", Reason="", readiness=false. Elapsed: 4m0.117570634s
[1mSTEP:[0m Saw pod success [38;5;243m01/17/23 17:42:05.47[0m
Jan 17 17:42:05.470: INFO: Pod "azuredisk-volume-tester-v78b4" satisfied condition "Succeeded or Failed"
Jan 17 17:42:05.471: INFO: deleting Pod "azuredisk-7230"/"azuredisk-volume-tester-v78b4"
Jan 17 17:42:05.564: INFO: Pod azuredisk-volume-tester-v78b4 has the following logs: hello world

[1mSTEP:[0m Deleting pod azuredisk-volume-tester-v78b4 in namespace azuredisk-7230 [38;5;243m01/17/23 17:42:05.564[0m
[1mSTEP:[0m Destroying namespace "azuredisk-7230" for this suite. [38;5;243m01/17/23 17:42:05.636[0m
[38;5;243m------------------------------[0m
... skipping 10 lines ...
    Jan 17 17:38:04.615: INFO: >>> kubeConfig: /home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure/kubeconfig
    [1mSTEP:[0m Building a namespace api object, basename azuredisk [38;5;243m01/17/23 17:38:04.617[0m
    [1mSTEP:[0m Waiting for a default service account to be provisioned in namespace [38;5;243m01/17/23 17:38:04.985[0m
    [1mSTEP:[0m Waiting for kube-root-ca.crt to be provisioned in namespace [38;5;243m01/17/23 17:38:05.095[0m
    I0117 17:38:05.208330   43786 azuredisk_driver.go:52] Using azure disk driver: disk.csi.azure.com
    [1mSTEP:[0m deploying the pod [38;5;243m01/17/23 17:38:05.287[0m
    [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/17/23 17:38:05.352[0m
    Jan 17 17:38:05.353: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-v78b4" in namespace "azuredisk-7230" to be "Succeeded or Failed"
    Jan 17 17:38:05.409: INFO: Pod "azuredisk-volume-tester-v78b4": Phase="Pending", Reason="", readiness=false. Elapsed: 56.067702ms
    Jan 17 17:38:07.467: INFO: Pod "azuredisk-volume-tester-v78b4": Phase="Pending", Reason="", readiness=false. Elapsed: 2.114217004s
    Jan 17 17:38:09.474: INFO: Pod "azuredisk-volume-tester-v78b4": Phase="Pending", Reason="", readiness=false. Elapsed: 4.121577512s
    Jan 17 17:38:11.467: INFO: Pod "azuredisk-volume-tester-v78b4": Phase="Pending", Reason="", readiness=false. Elapsed: 6.11450939s
    Jan 17 17:38:13.467: INFO: Pod "azuredisk-volume-tester-v78b4": Phase="Pending", Reason="", readiness=false. Elapsed: 8.114688707s
    Jan 17 17:38:15.467: INFO: Pod "azuredisk-volume-tester-v78b4": Phase="Pending", Reason="", readiness=false. Elapsed: 10.114023068s
... skipping 110 lines ...
    Jan 17 17:41:57.471: INFO: Pod "azuredisk-volume-tester-v78b4": Phase="Running", Reason="", readiness=true. Elapsed: 3m52.11841781s
    Jan 17 17:41:59.472: INFO: Pod "azuredisk-volume-tester-v78b4": Phase="Running", Reason="", readiness=true. Elapsed: 3m54.11899189s
    Jan 17 17:42:01.472: INFO: Pod "azuredisk-volume-tester-v78b4": Phase="Running", Reason="", readiness=false. Elapsed: 3m56.119756926s
    Jan 17 17:42:03.471: INFO: Pod "azuredisk-volume-tester-v78b4": Phase="Running", Reason="", readiness=false. Elapsed: 3m58.118664776s
    Jan 17 17:42:05.470: INFO: Pod "azuredisk-volume-tester-v78b4": Phase="Succeeded", Reason="", readiness=false. Elapsed: 4m0.117570634s
    [1mSTEP:[0m Saw pod success [38;5;243m01/17/23 17:42:05.47[0m
    Jan 17 17:42:05.470: INFO: Pod "azuredisk-volume-tester-v78b4" satisfied condition "Succeeded or Failed"
    Jan 17 17:42:05.471: INFO: deleting Pod "azuredisk-7230"/"azuredisk-volume-tester-v78b4"
    Jan 17 17:42:05.564: INFO: Pod azuredisk-volume-tester-v78b4 has the following logs: hello world

    [1mSTEP:[0m Deleting pod azuredisk-volume-tester-v78b4 in namespace azuredisk-7230 [38;5;243m01/17/23 17:42:05.564[0m
    [1mSTEP:[0m Destroying namespace "azuredisk-7230" for this suite. [38;5;243m01/17/23 17:42:05.636[0m
  [38;5;243m<< End Captured GinkgoWriter Output[0m
... skipping 17 lines ...
Jan 17 17:42:08.359: INFO: PersistentVolumeClaim pvc-r2p9b found but phase is Pending instead of Bound.
Jan 17 17:42:10.421: INFO: PersistentVolumeClaim pvc-r2p9b found and phase=Bound (4.184034508s)
[1mSTEP:[0m checking the PVC [38;5;243m01/17/23 17:42:10.421[0m
[1mSTEP:[0m validating provisioned PV [38;5;243m01/17/23 17:42:10.479[0m
[1mSTEP:[0m checking the PV [38;5;243m01/17/23 17:42:10.536[0m
[1mSTEP:[0m deploying the pod [38;5;243m01/17/23 17:42:10.617[0m
[1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/17/23 17:42:10.68[0m
Jan 17 17:42:10.681: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-gfpwf" in namespace "azuredisk-1410" to be "Succeeded or Failed"
Jan 17 17:42:10.751: INFO: Pod "azuredisk-volume-tester-gfpwf": Phase="Pending", Reason="", readiness=false. Elapsed: 70.209281ms
Jan 17 17:42:12.808: INFO: Pod "azuredisk-volume-tester-gfpwf": Phase="Pending", Reason="", readiness=false. Elapsed: 2.127738735s
Jan 17 17:42:14.810: INFO: Pod "azuredisk-volume-tester-gfpwf": Phase="Pending", Reason="", readiness=false. Elapsed: 4.128987231s
Jan 17 17:42:16.809: INFO: Pod "azuredisk-volume-tester-gfpwf": Phase="Pending", Reason="", readiness=false. Elapsed: 6.127919728s
Jan 17 17:42:18.810: INFO: Pod "azuredisk-volume-tester-gfpwf": Phase="Pending", Reason="", readiness=false. Elapsed: 8.129251772s
Jan 17 17:42:20.810: INFO: Pod "azuredisk-volume-tester-gfpwf": Phase="Pending", Reason="", readiness=false. Elapsed: 10.129584261s
... skipping 22 lines ...
Jan 17 17:43:06.822: INFO: Pod "azuredisk-volume-tester-gfpwf": Phase="Running", Reason="", readiness=true. Elapsed: 56.140941634s
Jan 17 17:43:08.813: INFO: Pod "azuredisk-volume-tester-gfpwf": Phase="Running", Reason="", readiness=true. Elapsed: 58.132672426s
Jan 17 17:43:10.813: INFO: Pod "azuredisk-volume-tester-gfpwf": Phase="Running", Reason="", readiness=true. Elapsed: 1m0.132239733s
Jan 17 17:43:12.814: INFO: Pod "azuredisk-volume-tester-gfpwf": Phase="Running", Reason="", readiness=false. Elapsed: 1m2.133297547s
Jan 17 17:43:14.812: INFO: Pod "azuredisk-volume-tester-gfpwf": Phase="Succeeded", Reason="", readiness=false. Elapsed: 1m4.131695551s
[1mSTEP:[0m Saw pod success [38;5;243m01/17/23 17:43:14.812[0m
Jan 17 17:43:14.813: INFO: Pod "azuredisk-volume-tester-gfpwf" satisfied condition "Succeeded or Failed"
Jan 17 17:43:14.813: INFO: deleting Pod "azuredisk-1410"/"azuredisk-volume-tester-gfpwf"
Jan 17 17:43:14.889: INFO: Pod azuredisk-volume-tester-gfpwf has the following logs: hello world

[1mSTEP:[0m Deleting pod azuredisk-volume-tester-gfpwf in namespace azuredisk-1410 [38;5;243m01/17/23 17:43:14.889[0m
Jan 17 17:43:14.961: INFO: deleting PVC "azuredisk-1410"/"pvc-r2p9b"
Jan 17 17:43:14.961: INFO: Deleting PersistentVolumeClaim "pvc-r2p9b"
... skipping 38 lines ...
    Jan 17 17:42:08.359: INFO: PersistentVolumeClaim pvc-r2p9b found but phase is Pending instead of Bound.
    Jan 17 17:42:10.421: INFO: PersistentVolumeClaim pvc-r2p9b found and phase=Bound (4.184034508s)
    [1mSTEP:[0m checking the PVC [38;5;243m01/17/23 17:42:10.421[0m
    [1mSTEP:[0m validating provisioned PV [38;5;243m01/17/23 17:42:10.479[0m
    [1mSTEP:[0m checking the PV [38;5;243m01/17/23 17:42:10.536[0m
    [1mSTEP:[0m deploying the pod [38;5;243m01/17/23 17:42:10.617[0m
    [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/17/23 17:42:10.68[0m
    Jan 17 17:42:10.681: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-gfpwf" in namespace "azuredisk-1410" to be "Succeeded or Failed"
    Jan 17 17:42:10.751: INFO: Pod "azuredisk-volume-tester-gfpwf": Phase="Pending", Reason="", readiness=false. Elapsed: 70.209281ms
    Jan 17 17:42:12.808: INFO: Pod "azuredisk-volume-tester-gfpwf": Phase="Pending", Reason="", readiness=false. Elapsed: 2.127738735s
    Jan 17 17:42:14.810: INFO: Pod "azuredisk-volume-tester-gfpwf": Phase="Pending", Reason="", readiness=false. Elapsed: 4.128987231s
    Jan 17 17:42:16.809: INFO: Pod "azuredisk-volume-tester-gfpwf": Phase="Pending", Reason="", readiness=false. Elapsed: 6.127919728s
    Jan 17 17:42:18.810: INFO: Pod "azuredisk-volume-tester-gfpwf": Phase="Pending", Reason="", readiness=false. Elapsed: 8.129251772s
    Jan 17 17:42:20.810: INFO: Pod "azuredisk-volume-tester-gfpwf": Phase="Pending", Reason="", readiness=false. Elapsed: 10.129584261s
... skipping 22 lines ...
    Jan 17 17:43:06.822: INFO: Pod "azuredisk-volume-tester-gfpwf": Phase="Running", Reason="", readiness=true. Elapsed: 56.140941634s
    Jan 17 17:43:08.813: INFO: Pod "azuredisk-volume-tester-gfpwf": Phase="Running", Reason="", readiness=true. Elapsed: 58.132672426s
    Jan 17 17:43:10.813: INFO: Pod "azuredisk-volume-tester-gfpwf": Phase="Running", Reason="", readiness=true. Elapsed: 1m0.132239733s
    Jan 17 17:43:12.814: INFO: Pod "azuredisk-volume-tester-gfpwf": Phase="Running", Reason="", readiness=false. Elapsed: 1m2.133297547s
    Jan 17 17:43:14.812: INFO: Pod "azuredisk-volume-tester-gfpwf": Phase="Succeeded", Reason="", readiness=false. Elapsed: 1m4.131695551s
    [1mSTEP:[0m Saw pod success [38;5;243m01/17/23 17:43:14.812[0m
    Jan 17 17:43:14.813: INFO: Pod "azuredisk-volume-tester-gfpwf" satisfied condition "Succeeded or Failed"
    Jan 17 17:43:14.813: INFO: deleting Pod "azuredisk-1410"/"azuredisk-volume-tester-gfpwf"
    Jan 17 17:43:14.889: INFO: Pod azuredisk-volume-tester-gfpwf has the following logs: hello world

    [1mSTEP:[0m Deleting pod azuredisk-volume-tester-gfpwf in namespace azuredisk-1410 [38;5;243m01/17/23 17:43:14.889[0m
    Jan 17 17:43:14.961: INFO: deleting PVC "azuredisk-1410"/"pvc-r2p9b"
    Jan 17 17:43:14.961: INFO: Deleting PersistentVolumeClaim "pvc-r2p9b"
... skipping 195 lines ...
Jan 17 17:50:27.555: INFO: PersistentVolumeClaim pvc-x552l found and phase=Bound (4.172985803s)
[1mSTEP:[0m checking the PVC [38;5;243m01/17/23 17:50:27.555[0m
[1mSTEP:[0m validating provisioned PV [38;5;243m01/17/23 17:50:27.612[0m
[1mSTEP:[0m checking the PV [38;5;243m01/17/23 17:50:27.668[0m
I0117 17:50:27.669036   43786 resource_setup.go:89] adding PV (pvc-27b77e3b-ca11-4dc4-a4cf-8160e32ca0a3) to pod ()
[1mSTEP:[0m deploying the pod [38;5;243m01/17/23 17:50:27.749[0m
[1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/17/23 17:50:27.814[0m
Jan 17 17:50:27.814: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-c62nf" in namespace "azuredisk-1183" to be "Succeeded or Failed"
Jan 17 17:50:27.877: INFO: Pod "azuredisk-volume-tester-c62nf": Phase="Pending", Reason="", readiness=false. Elapsed: 62.380536ms
Jan 17 17:50:29.937: INFO: Pod "azuredisk-volume-tester-c62nf": Phase="Pending", Reason="", readiness=false. Elapsed: 2.122872337s
Jan 17 17:50:31.937: INFO: Pod "azuredisk-volume-tester-c62nf": Phase="Pending", Reason="", readiness=false. Elapsed: 4.123143892s
Jan 17 17:50:33.939: INFO: Pod "azuredisk-volume-tester-c62nf": Phase="Pending", Reason="", readiness=false. Elapsed: 6.124881083s
Jan 17 17:50:35.947: INFO: Pod "azuredisk-volume-tester-c62nf": Phase="Pending", Reason="", readiness=false. Elapsed: 8.132614338s
Jan 17 17:50:37.939: INFO: Pod "azuredisk-volume-tester-c62nf": Phase="Pending", Reason="", readiness=false. Elapsed: 10.124318857s
... skipping 29 lines ...
Jan 17 17:51:37.938: INFO: Pod "azuredisk-volume-tester-c62nf": Phase="Running", Reason="", readiness=true. Elapsed: 1m10.123750479s
Jan 17 17:51:39.937: INFO: Pod "azuredisk-volume-tester-c62nf": Phase="Running", Reason="", readiness=false. Elapsed: 1m12.122986488s
Jan 17 17:51:41.939: INFO: Pod "azuredisk-volume-tester-c62nf": Phase="Running", Reason="", readiness=false. Elapsed: 1m14.124599777s
Jan 17 17:51:43.937: INFO: Pod "azuredisk-volume-tester-c62nf": Phase="Running", Reason="", readiness=false. Elapsed: 1m16.123022993s
Jan 17 17:51:45.940: INFO: Pod "azuredisk-volume-tester-c62nf": Phase="Succeeded", Reason="", readiness=false. Elapsed: 1m18.125573507s
[1mSTEP:[0m Saw pod success [38;5;243m01/17/23 17:51:45.94[0m
Jan 17 17:51:45.940: INFO: Pod "azuredisk-volume-tester-c62nf" satisfied condition "Succeeded or Failed"
Jan 17 17:51:45.940: INFO: deleting Pod "azuredisk-1183"/"azuredisk-volume-tester-c62nf"
Jan 17 17:51:46.041: INFO: Pod azuredisk-volume-tester-c62nf has the following logs: hello world

[1mSTEP:[0m Deleting pod azuredisk-volume-tester-c62nf in namespace azuredisk-1183 [38;5;243m01/17/23 17:51:46.041[0m
Jan 17 17:51:46.119: INFO: deleting PVC "azuredisk-1183"/"pvc-x552l"
Jan 17 17:51:46.119: INFO: Deleting PersistentVolumeClaim "pvc-x552l"
... skipping 85 lines ...
    Jan 17 17:50:27.555: INFO: PersistentVolumeClaim pvc-x552l found and phase=Bound (4.172985803s)
    [1mSTEP:[0m checking the PVC [38;5;243m01/17/23 17:50:27.555[0m
    [1mSTEP:[0m validating provisioned PV [38;5;243m01/17/23 17:50:27.612[0m
    [1mSTEP:[0m checking the PV [38;5;243m01/17/23 17:50:27.668[0m
    I0117 17:50:27.669036   43786 resource_setup.go:89] adding PV (pvc-27b77e3b-ca11-4dc4-a4cf-8160e32ca0a3) to pod ()
    [1mSTEP:[0m deploying the pod [38;5;243m01/17/23 17:50:27.749[0m
    [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/17/23 17:50:27.814[0m
    Jan 17 17:50:27.814: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-c62nf" in namespace "azuredisk-1183" to be "Succeeded or Failed"
    Jan 17 17:50:27.877: INFO: Pod "azuredisk-volume-tester-c62nf": Phase="Pending", Reason="", readiness=false. Elapsed: 62.380536ms
    Jan 17 17:50:29.937: INFO: Pod "azuredisk-volume-tester-c62nf": Phase="Pending", Reason="", readiness=false. Elapsed: 2.122872337s
    Jan 17 17:50:31.937: INFO: Pod "azuredisk-volume-tester-c62nf": Phase="Pending", Reason="", readiness=false. Elapsed: 4.123143892s
    Jan 17 17:50:33.939: INFO: Pod "azuredisk-volume-tester-c62nf": Phase="Pending", Reason="", readiness=false. Elapsed: 6.124881083s
    Jan 17 17:50:35.947: INFO: Pod "azuredisk-volume-tester-c62nf": Phase="Pending", Reason="", readiness=false. Elapsed: 8.132614338s
    Jan 17 17:50:37.939: INFO: Pod "azuredisk-volume-tester-c62nf": Phase="Pending", Reason="", readiness=false. Elapsed: 10.124318857s
... skipping 29 lines ...
    Jan 17 17:51:37.938: INFO: Pod "azuredisk-volume-tester-c62nf": Phase="Running", Reason="", readiness=true. Elapsed: 1m10.123750479s
    Jan 17 17:51:39.937: INFO: Pod "azuredisk-volume-tester-c62nf": Phase="Running", Reason="", readiness=false. Elapsed: 1m12.122986488s
    Jan 17 17:51:41.939: INFO: Pod "azuredisk-volume-tester-c62nf": Phase="Running", Reason="", readiness=false. Elapsed: 1m14.124599777s
    Jan 17 17:51:43.937: INFO: Pod "azuredisk-volume-tester-c62nf": Phase="Running", Reason="", readiness=false. Elapsed: 1m16.123022993s
    Jan 17 17:51:45.940: INFO: Pod "azuredisk-volume-tester-c62nf": Phase="Succeeded", Reason="", readiness=false. Elapsed: 1m18.125573507s
    [1mSTEP:[0m Saw pod success [38;5;243m01/17/23 17:51:45.94[0m
    Jan 17 17:51:45.940: INFO: Pod "azuredisk-volume-tester-c62nf" satisfied condition "Succeeded or Failed"
    Jan 17 17:51:45.940: INFO: deleting Pod "azuredisk-1183"/"azuredisk-volume-tester-c62nf"
    Jan 17 17:51:46.041: INFO: Pod azuredisk-volume-tester-c62nf has the following logs: hello world

    [1mSTEP:[0m Deleting pod azuredisk-volume-tester-c62nf in namespace azuredisk-1183 [38;5;243m01/17/23 17:51:46.041[0m
    Jan 17 17:51:46.119: INFO: deleting PVC "azuredisk-1183"/"pvc-x552l"
    Jan 17 17:51:46.119: INFO: Deleting PersistentVolumeClaim "pvc-x552l"
... skipping 895 lines ...
Jan 17 18:06:59.552: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [pvc-p65s5] to have phase Bound
Jan 17 18:06:59.608: INFO: PersistentVolumeClaim pvc-p65s5 found and phase=Bound (56.263948ms)
[1mSTEP:[0m checking the PVC [38;5;243m01/17/23 18:06:59.608[0m
[1mSTEP:[0m validating provisioned PV [38;5;243m01/17/23 18:06:59.665[0m
[1mSTEP:[0m checking the PV [38;5;243m01/17/23 18:06:59.724[0m
[1mSTEP:[0m deploying the pod [38;5;243m01/17/23 18:06:59.724[0m
[1mSTEP:[0m checking that the pod's command exits with an error [38;5;243m01/17/23 18:06:59.792[0m
Jan 17 18:06:59.792: INFO: Waiting up to 10m0s for pod "azuredisk-volume-tester-vd65s" in namespace "azuredisk-8377" to be "Error status code"
Jan 17 18:06:59.848: INFO: Pod "azuredisk-volume-tester-vd65s": Phase="Pending", Reason="", readiness=false. Elapsed: 56.275864ms
Jan 17 18:07:01.907: INFO: Pod "azuredisk-volume-tester-vd65s": Phase="Pending", Reason="", readiness=false. Elapsed: 2.114612203s
Jan 17 18:07:03.908: INFO: Pod "azuredisk-volume-tester-vd65s": Phase="Pending", Reason="", readiness=false. Elapsed: 4.115493109s
Jan 17 18:07:05.908: INFO: Pod "azuredisk-volume-tester-vd65s": Phase="Pending", Reason="", readiness=false. Elapsed: 6.115431998s
Jan 17 18:07:07.909: INFO: Pod "azuredisk-volume-tester-vd65s": Phase="Pending", Reason="", readiness=false. Elapsed: 8.116698549s
Jan 17 18:07:09.907: INFO: Pod "azuredisk-volume-tester-vd65s": Phase="Pending", Reason="", readiness=false. Elapsed: 10.114887108s
... skipping 16 lines ...
Jan 17 18:07:43.910: INFO: Pod "azuredisk-volume-tester-vd65s": Phase="Pending", Reason="", readiness=false. Elapsed: 44.117891214s
Jan 17 18:07:45.914: INFO: Pod "azuredisk-volume-tester-vd65s": Phase="Running", Reason="", readiness=true. Elapsed: 46.12183633s
Jan 17 18:07:47.911: INFO: Pod "azuredisk-volume-tester-vd65s": Phase="Running", Reason="", readiness=true. Elapsed: 48.118421835s
Jan 17 18:07:49.911: INFO: Pod "azuredisk-volume-tester-vd65s": Phase="Running", Reason="", readiness=true. Elapsed: 50.118630353s
Jan 17 18:07:51.909: INFO: Pod "azuredisk-volume-tester-vd65s": Phase="Running", Reason="", readiness=true. Elapsed: 52.116390223s
Jan 17 18:07:53.910: INFO: Pod "azuredisk-volume-tester-vd65s": Phase="Running", Reason="", readiness=false. Elapsed: 54.117992109s
Jan 17 18:07:55.911: INFO: Pod "azuredisk-volume-tester-vd65s": Phase="Failed", Reason="", readiness=false. Elapsed: 56.118351987s
[1mSTEP:[0m Saw pod failure [38;5;243m01/17/23 18:07:55.911[0m
Jan 17 18:07:55.911: INFO: Pod "azuredisk-volume-tester-vd65s" satisfied condition "Error status code"
[1mSTEP:[0m checking that pod logs contain expected message [38;5;243m01/17/23 18:07:55.911[0m
Jan 17 18:07:56.010: INFO: deleting Pod "azuredisk-8377"/"azuredisk-volume-tester-vd65s"
Jan 17 18:07:56.088: INFO: Pod azuredisk-volume-tester-vd65s has the following logs: Out-File : Access to the path 'C:\mnt\test-1\data.txt' is denied.
At line:1 char:22
+ ... cho 'hello world' | Out-File -FilePath C:\mnt\test-1\data.txt; Get-Co ...
+                         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
... skipping 55 lines ...
    Jan 17 18:06:59.552: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [pvc-p65s5] to have phase Bound
    Jan 17 18:06:59.608: INFO: PersistentVolumeClaim pvc-p65s5 found and phase=Bound (56.263948ms)
    [1mSTEP:[0m checking the PVC [38;5;243m01/17/23 18:06:59.608[0m
    [1mSTEP:[0m validating provisioned PV [38;5;243m01/17/23 18:06:59.665[0m
    [1mSTEP:[0m checking the PV [38;5;243m01/17/23 18:06:59.724[0m
    [1mSTEP:[0m deploying the pod [38;5;243m01/17/23 18:06:59.724[0m
    [1mSTEP:[0m checking that the pod's command exits with an error [38;5;243m01/17/23 18:06:59.792[0m
    Jan 17 18:06:59.792: INFO: Waiting up to 10m0s for pod "azuredisk-volume-tester-vd65s" in namespace "azuredisk-8377" to be "Error status code"
    Jan 17 18:06:59.848: INFO: Pod "azuredisk-volume-tester-vd65s": Phase="Pending", Reason="", readiness=false. Elapsed: 56.275864ms
    Jan 17 18:07:01.907: INFO: Pod "azuredisk-volume-tester-vd65s": Phase="Pending", Reason="", readiness=false. Elapsed: 2.114612203s
    Jan 17 18:07:03.908: INFO: Pod "azuredisk-volume-tester-vd65s": Phase="Pending", Reason="", readiness=false. Elapsed: 4.115493109s
    Jan 17 18:07:05.908: INFO: Pod "azuredisk-volume-tester-vd65s": Phase="Pending", Reason="", readiness=false. Elapsed: 6.115431998s
    Jan 17 18:07:07.909: INFO: Pod "azuredisk-volume-tester-vd65s": Phase="Pending", Reason="", readiness=false. Elapsed: 8.116698549s
    Jan 17 18:07:09.907: INFO: Pod "azuredisk-volume-tester-vd65s": Phase="Pending", Reason="", readiness=false. Elapsed: 10.114887108s
... skipping 16 lines ...
    Jan 17 18:07:43.910: INFO: Pod "azuredisk-volume-tester-vd65s": Phase="Pending", Reason="", readiness=false. Elapsed: 44.117891214s
    Jan 17 18:07:45.914: INFO: Pod "azuredisk-volume-tester-vd65s": Phase="Running", Reason="", readiness=true. Elapsed: 46.12183633s
    Jan 17 18:07:47.911: INFO: Pod "azuredisk-volume-tester-vd65s": Phase="Running", Reason="", readiness=true. Elapsed: 48.118421835s
    Jan 17 18:07:49.911: INFO: Pod "azuredisk-volume-tester-vd65s": Phase="Running", Reason="", readiness=true. Elapsed: 50.118630353s
    Jan 17 18:07:51.909: INFO: Pod "azuredisk-volume-tester-vd65s": Phase="Running", Reason="", readiness=true. Elapsed: 52.116390223s
    Jan 17 18:07:53.910: INFO: Pod "azuredisk-volume-tester-vd65s": Phase="Running", Reason="", readiness=false. Elapsed: 54.117992109s
    Jan 17 18:07:55.911: INFO: Pod "azuredisk-volume-tester-vd65s": Phase="Failed", Reason="", readiness=false. Elapsed: 56.118351987s
    [1mSTEP:[0m Saw pod failure [38;5;243m01/17/23 18:07:55.911[0m
    Jan 17 18:07:55.911: INFO: Pod "azuredisk-volume-tester-vd65s" satisfied condition "Error status code"
    [1mSTEP:[0m checking that pod logs contain expected message [38;5;243m01/17/23 18:07:55.911[0m
    Jan 17 18:07:56.010: INFO: deleting Pod "azuredisk-8377"/"azuredisk-volume-tester-vd65s"
    Jan 17 18:07:56.088: INFO: Pod azuredisk-volume-tester-vd65s has the following logs: Out-File : Access to the path 'C:\mnt\test-1\data.txt' is denied.
    At line:1 char:22
    + ... cho 'hello world' | Out-File -FilePath C:\mnt\test-1\data.txt; Get-Co ...
    +                         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
... skipping 84 lines ...
Jan 17 18:09:05.181: INFO: PersistentVolumeClaim pvc-ljrfz found and phase=Bound (56.278241ms)
[1mSTEP:[0m checking the PVC [38;5;243m01/17/23 18:09:05.181[0m
[1mSTEP:[0m validating provisioned PV [38;5;243m01/17/23 18:09:05.24[0m
[1mSTEP:[0m checking the PV [38;5;243m01/17/23 18:09:05.298[0m
[1mSTEP:[0m attaching disk to node#0 [38;5;243m01/17/23 18:09:05.371[0m
[1mSTEP:[0m deploying the pod [38;5;243m01/17/23 18:09:12.965[0m
[1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/17/23 18:09:13.03[0m
Jan 17 18:09:13.030: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-6ljm9" in namespace "azuredisk-4923" to be "Succeeded or Failed"
Jan 17 18:09:13.090: INFO: Pod "azuredisk-volume-tester-6ljm9": Phase="Pending", Reason="", readiness=false. Elapsed: 59.751645ms
Jan 17 18:09:15.147: INFO: Pod "azuredisk-volume-tester-6ljm9": Phase="Pending", Reason="", readiness=false. Elapsed: 2.116775157s
Jan 17 18:09:17.149: INFO: Pod "azuredisk-volume-tester-6ljm9": Phase="Pending", Reason="", readiness=false. Elapsed: 4.118248844s
Jan 17 18:09:19.158: INFO: Pod "azuredisk-volume-tester-6ljm9": Phase="Pending", Reason="", readiness=false. Elapsed: 6.127928739s
Jan 17 18:09:21.150: INFO: Pod "azuredisk-volume-tester-6ljm9": Phase="Pending", Reason="", readiness=false. Elapsed: 8.119344474s
Jan 17 18:09:23.151: INFO: Pod "azuredisk-volume-tester-6ljm9": Phase="Pending", Reason="", readiness=false. Elapsed: 10.120159778s
... skipping 29 lines ...
Jan 17 18:10:23.152: INFO: Pod "azuredisk-volume-tester-6ljm9": Phase="Running", Reason="", readiness=true. Elapsed: 1m10.121102139s
Jan 17 18:10:25.153: INFO: Pod "azuredisk-volume-tester-6ljm9": Phase="Running", Reason="", readiness=false. Elapsed: 1m12.122000755s
Jan 17 18:10:27.152: INFO: Pod "azuredisk-volume-tester-6ljm9": Phase="Running", Reason="", readiness=false. Elapsed: 1m14.121787658s
Jan 17 18:10:29.151: INFO: Pod "azuredisk-volume-tester-6ljm9": Phase="Running", Reason="", readiness=false. Elapsed: 1m16.120066887s
Jan 17 18:10:31.151: INFO: Pod "azuredisk-volume-tester-6ljm9": Phase="Succeeded", Reason="", readiness=false. Elapsed: 1m18.120547933s
[1mSTEP:[0m Saw pod success [38;5;243m01/17/23 18:10:31.151[0m
Jan 17 18:10:31.151: INFO: Pod "azuredisk-volume-tester-6ljm9" satisfied condition "Succeeded or Failed"
Jan 17 18:10:31.151: INFO: deleting Pod "azuredisk-4923"/"azuredisk-volume-tester-6ljm9"
Jan 17 18:10:31.244: INFO: Pod azuredisk-volume-tester-6ljm9 has the following logs: hello world

[1mSTEP:[0m Deleting pod azuredisk-volume-tester-6ljm9 in namespace azuredisk-4923 [38;5;243m01/17/23 18:10:31.244[0m
Jan 17 18:10:31.449: INFO: deleting PVC "azuredisk-4923"/"pvc-ljrfz"
Jan 17 18:10:31.449: INFO: Deleting PersistentVolumeClaim "pvc-ljrfz"
... skipping 38 lines ...
    Jan 17 18:09:05.181: INFO: PersistentVolumeClaim pvc-ljrfz found and phase=Bound (56.278241ms)
    [1mSTEP:[0m checking the PVC [38;5;243m01/17/23 18:09:05.181[0m
    [1mSTEP:[0m validating provisioned PV [38;5;243m01/17/23 18:09:05.24[0m
    [1mSTEP:[0m checking the PV [38;5;243m01/17/23 18:09:05.298[0m
    [1mSTEP:[0m attaching disk to node#0 [38;5;243m01/17/23 18:09:05.371[0m
    [1mSTEP:[0m deploying the pod [38;5;243m01/17/23 18:09:12.965[0m
    [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/17/23 18:09:13.03[0m
    Jan 17 18:09:13.030: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-6ljm9" in namespace "azuredisk-4923" to be "Succeeded or Failed"
    Jan 17 18:09:13.090: INFO: Pod "azuredisk-volume-tester-6ljm9": Phase="Pending", Reason="", readiness=false. Elapsed: 59.751645ms
    Jan 17 18:09:15.147: INFO: Pod "azuredisk-volume-tester-6ljm9": Phase="Pending", Reason="", readiness=false. Elapsed: 2.116775157s
    Jan 17 18:09:17.149: INFO: Pod "azuredisk-volume-tester-6ljm9": Phase="Pending", Reason="", readiness=false. Elapsed: 4.118248844s
    Jan 17 18:09:19.158: INFO: Pod "azuredisk-volume-tester-6ljm9": Phase="Pending", Reason="", readiness=false. Elapsed: 6.127928739s
    Jan 17 18:09:21.150: INFO: Pod "azuredisk-volume-tester-6ljm9": Phase="Pending", Reason="", readiness=false. Elapsed: 8.119344474s
    Jan 17 18:09:23.151: INFO: Pod "azuredisk-volume-tester-6ljm9": Phase="Pending", Reason="", readiness=false. Elapsed: 10.120159778s
... skipping 29 lines ...
    Jan 17 18:10:23.152: INFO: Pod "azuredisk-volume-tester-6ljm9": Phase="Running", Reason="", readiness=true. Elapsed: 1m10.121102139s
    Jan 17 18:10:25.153: INFO: Pod "azuredisk-volume-tester-6ljm9": Phase="Running", Reason="", readiness=false. Elapsed: 1m12.122000755s
    Jan 17 18:10:27.152: INFO: Pod "azuredisk-volume-tester-6ljm9": Phase="Running", Reason="", readiness=false. Elapsed: 1m14.121787658s
    Jan 17 18:10:29.151: INFO: Pod "azuredisk-volume-tester-6ljm9": Phase="Running", Reason="", readiness=false. Elapsed: 1m16.120066887s
    Jan 17 18:10:31.151: INFO: Pod "azuredisk-volume-tester-6ljm9": Phase="Succeeded", Reason="", readiness=false. Elapsed: 1m18.120547933s
    [1mSTEP:[0m Saw pod success [38;5;243m01/17/23 18:10:31.151[0m
    Jan 17 18:10:31.151: INFO: Pod "azuredisk-volume-tester-6ljm9" satisfied condition "Succeeded or Failed"
    Jan 17 18:10:31.151: INFO: deleting Pod "azuredisk-4923"/"azuredisk-volume-tester-6ljm9"
    Jan 17 18:10:31.244: INFO: Pod azuredisk-volume-tester-6ljm9 has the following logs: hello world

    [1mSTEP:[0m Deleting pod azuredisk-volume-tester-6ljm9 in namespace azuredisk-4923 [38;5;243m01/17/23 18:10:31.244[0m
    Jan 17 18:10:31.449: INFO: deleting PVC "azuredisk-4923"/"pvc-ljrfz"
    Jan 17 18:10:31.449: INFO: Deleting PersistentVolumeClaim "pvc-ljrfz"
... skipping 151 lines ...
  [38;5;243m<< End Captured GinkgoWriter Output[0m

  [38;5;14mtest case not supported by Windows clusters[0m
  [38;5;14mIn [1m[It][0m[38;5;14m at: [1m/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/utils/testutil/env_utils.go:26[0m

  [1mThere were additional failures detected after the initial failure:[0m
    [38;5;9m[FAILED][0m
    [38;5;9mpre-provisioning disk got deleted with the deletion of pv: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {
      "error": {

        "code": "NotFound",
        "message": "Disk single-shared-disk is not found."
      }
    }[0m
    [38;5;9mIn [1m[AfterEach][0m[38;5;9m at: [1m/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/pre_provisioning_test.go:490[0m
[38;5;243m------------------------------[0m
... skipping 27 lines ...
  [38;5;243m<< End Captured GinkgoWriter Output[0m

  [38;5;14mtest case not supported by Windows clusters[0m
  [38;5;14mIn [1m[It][0m[38;5;14m at: [1m/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/utils/testutil/env_utils.go:26[0m

  [1mThere were additional failures detected after the initial failure:[0m
    [38;5;9m[FAILED][0m
    [38;5;9mpre-provisioning disk got deleted with the deletion of pv: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {
      "error": {

        "code": "NotFound",
        "message": "Disk single-shared-disk is not found."
      }
    }[0m
    [38;5;9mIn [1m[AfterEach][0m[38;5;9m at: [1m/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/pre_provisioning_test.go:490[0m
[38;5;243m------------------------------[0m
... skipping 27 lines ...
  [38;5;243m<< End Captured GinkgoWriter Output[0m

  [38;5;14mtest case is only available for csi driver[0m
  [38;5;14mIn [1m[It][0m[38;5;14m at: [1m/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/pre_provisioning_test.go:395[0m

  [1mThere were additional failures detected after the initial failure:[0m
    [38;5;9m[FAILED][0m
    [38;5;9mpre-provisioning disk got deleted with the deletion of pv: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {
      "error": {

        "code": "NotFound",
        "message": "Disk single-shared-disk is not found."
      }
    }[0m
    [38;5;9mIn [1m[AfterEach][0m[38;5;9m at: [1m/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/pre_provisioning_test.go:490[0m
[38;5;243m------------------------------[0m
... skipping 51 lines ...
[1mSTEP:[0m checking the PVC [38;5;243m01/17/23 18:11:35.683[0m
[1mSTEP:[0m validating provisioned PV [38;5;243m01/17/23 18:11:35.743[0m
[1mSTEP:[0m checking the PV [38;5;243m01/17/23 18:11:35.801[0m
[1mSTEP:[0m setting up the pod [38;5;243m01/17/23 18:11:35.802[0m
I0117 18:11:35.802393   43786 resource_setup.go:62] adding PV (pvc-2b4ca4aa-d4bc-4662-85c6-8fb4d165bfad) to pod ()
[1mSTEP:[0m deploying the pod [38;5;243m01/17/23 18:11:35.802[0m
[1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/17/23 18:11:35.868[0m
Jan 17 18:11:35.868: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-j8vtj" in namespace "azuredisk-9580" to be "Succeeded or Failed"
Jan 17 18:11:35.926: INFO: Pod "azuredisk-volume-tester-j8vtj": Phase="Pending", Reason="", readiness=false. Elapsed: 57.95153ms
Jan 17 18:11:37.984: INFO: Pod "azuredisk-volume-tester-j8vtj": Phase="Pending", Reason="", readiness=false. Elapsed: 2.116176231s
Jan 17 18:11:39.985: INFO: Pod "azuredisk-volume-tester-j8vtj": Phase="Pending", Reason="", readiness=false. Elapsed: 4.116749628s
Jan 17 18:11:41.985: INFO: Pod "azuredisk-volume-tester-j8vtj": Phase="Pending", Reason="", readiness=false. Elapsed: 6.116909719s
Jan 17 18:11:43.986: INFO: Pod "azuredisk-volume-tester-j8vtj": Phase="Pending", Reason="", readiness=false. Elapsed: 8.117980996s
Jan 17 18:11:45.987: INFO: Pod "azuredisk-volume-tester-j8vtj": Phase="Pending", Reason="", readiness=false. Elapsed: 10.11912584s
... skipping 19 lines ...
Jan 17 18:12:25.987: INFO: Pod "azuredisk-volume-tester-j8vtj": Phase="Running", Reason="", readiness=true. Elapsed: 50.119346316s
Jan 17 18:12:27.994: INFO: Pod "azuredisk-volume-tester-j8vtj": Phase="Running", Reason="", readiness=false. Elapsed: 52.126024601s
Jan 17 18:12:29.988: INFO: Pod "azuredisk-volume-tester-j8vtj": Phase="Running", Reason="", readiness=false. Elapsed: 54.119979032s
Jan 17 18:12:31.989: INFO: Pod "azuredisk-volume-tester-j8vtj": Phase="Running", Reason="", readiness=false. Elapsed: 56.120712038s
Jan 17 18:12:33.988: INFO: Pod "azuredisk-volume-tester-j8vtj": Phase="Succeeded", Reason="", readiness=false. Elapsed: 58.120009423s
[1mSTEP:[0m Saw pod success [38;5;243m01/17/23 18:12:33.988[0m
Jan 17 18:12:33.988: INFO: Pod "azuredisk-volume-tester-j8vtj" satisfied condition "Succeeded or Failed"
Jan 17 18:12:33.988: INFO: deleting Pod "azuredisk-9580"/"azuredisk-volume-tester-j8vtj"
Jan 17 18:12:34.068: INFO: Pod azuredisk-volume-tester-j8vtj has the following logs: hello world

[1mSTEP:[0m Deleting pod azuredisk-volume-tester-j8vtj in namespace azuredisk-9580 [38;5;243m01/17/23 18:12:34.069[0m
Jan 17 18:12:34.143: INFO: deleting PVC "azuredisk-9580"/"pvc-bnskg"
Jan 17 18:12:34.143: INFO: Deleting PersistentVolumeClaim "pvc-bnskg"
... skipping 40 lines ...
    [1mSTEP:[0m checking the PVC [38;5;243m01/17/23 18:11:35.683[0m
    [1mSTEP:[0m validating provisioned PV [38;5;243m01/17/23 18:11:35.743[0m
    [1mSTEP:[0m checking the PV [38;5;243m01/17/23 18:11:35.801[0m
    [1mSTEP:[0m setting up the pod [38;5;243m01/17/23 18:11:35.802[0m
    I0117 18:11:35.802393   43786 resource_setup.go:62] adding PV (pvc-2b4ca4aa-d4bc-4662-85c6-8fb4d165bfad) to pod ()
    [1mSTEP:[0m deploying the pod [38;5;243m01/17/23 18:11:35.802[0m
    [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/17/23 18:11:35.868[0m
    Jan 17 18:11:35.868: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-j8vtj" in namespace "azuredisk-9580" to be "Succeeded or Failed"
    Jan 17 18:11:35.926: INFO: Pod "azuredisk-volume-tester-j8vtj": Phase="Pending", Reason="", readiness=false. Elapsed: 57.95153ms
    Jan 17 18:11:37.984: INFO: Pod "azuredisk-volume-tester-j8vtj": Phase="Pending", Reason="", readiness=false. Elapsed: 2.116176231s
    Jan 17 18:11:39.985: INFO: Pod "azuredisk-volume-tester-j8vtj": Phase="Pending", Reason="", readiness=false. Elapsed: 4.116749628s
    Jan 17 18:11:41.985: INFO: Pod "azuredisk-volume-tester-j8vtj": Phase="Pending", Reason="", readiness=false. Elapsed: 6.116909719s
    Jan 17 18:11:43.986: INFO: Pod "azuredisk-volume-tester-j8vtj": Phase="Pending", Reason="", readiness=false. Elapsed: 8.117980996s
    Jan 17 18:11:45.987: INFO: Pod "azuredisk-volume-tester-j8vtj": Phase="Pending", Reason="", readiness=false. Elapsed: 10.11912584s
... skipping 19 lines ...
    Jan 17 18:12:25.987: INFO: Pod "azuredisk-volume-tester-j8vtj": Phase="Running", Reason="", readiness=true. Elapsed: 50.119346316s
    Jan 17 18:12:27.994: INFO: Pod "azuredisk-volume-tester-j8vtj": Phase="Running", Reason="", readiness=false. Elapsed: 52.126024601s
    Jan 17 18:12:29.988: INFO: Pod "azuredisk-volume-tester-j8vtj": Phase="Running", Reason="", readiness=false. Elapsed: 54.119979032s
    Jan 17 18:12:31.989: INFO: Pod "azuredisk-volume-tester-j8vtj": Phase="Running", Reason="", readiness=false. Elapsed: 56.120712038s
    Jan 17 18:12:33.988: INFO: Pod "azuredisk-volume-tester-j8vtj": Phase="Succeeded", Reason="", readiness=false. Elapsed: 58.120009423s
    [1mSTEP:[0m Saw pod success [38;5;243m01/17/23 18:12:33.988[0m
    Jan 17 18:12:33.988: INFO: Pod "azuredisk-volume-tester-j8vtj" satisfied condition "Succeeded or Failed"
    Jan 17 18:12:33.988: INFO: deleting Pod "azuredisk-9580"/"azuredisk-volume-tester-j8vtj"
    Jan 17 18:12:34.068: INFO: Pod azuredisk-volume-tester-j8vtj has the following logs: hello world

    [1mSTEP:[0m Deleting pod azuredisk-volume-tester-j8vtj in namespace azuredisk-9580 [38;5;243m01/17/23 18:12:34.069[0m
    Jan 17 18:12:34.143: INFO: deleting PVC "azuredisk-9580"/"pvc-bnskg"
    Jan 17 18:12:34.143: INFO: Deleting PersistentVolumeClaim "pvc-bnskg"
... skipping 37 lines ...
Jan 17 18:13:18.134: INFO: PersistentVolumeClaim pvc-wt6l8 found but phase is Pending instead of Bound.
Jan 17 18:13:20.195: INFO: PersistentVolumeClaim pvc-wt6l8 found and phase=Bound (4.177143933s)
[1mSTEP:[0m checking the PVC [38;5;243m01/17/23 18:13:20.195[0m
[1mSTEP:[0m validating provisioned PV [38;5;243m01/17/23 18:13:20.254[0m
[1mSTEP:[0m checking the PV [38;5;243m01/17/23 18:13:20.311[0m
[1mSTEP:[0m deploying the pod [38;5;243m01/17/23 18:13:20.312[0m
[1mSTEP:[0m checking that the pods command exits with no error [38;5;243m01/17/23 18:13:20.374[0m
Jan 17 18:13:20.374: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-nmsfx" in namespace "azuredisk-4923" to be "Succeeded or Failed"
Jan 17 18:13:20.431: INFO: Pod "azuredisk-volume-tester-nmsfx": Phase="Pending", Reason="", readiness=false. Elapsed: 57.230392ms
Jan 17 18:13:22.490: INFO: Pod "azuredisk-volume-tester-nmsfx": Phase="Pending", Reason="", readiness=false. Elapsed: 2.116174066s
Jan 17 18:13:24.490: INFO: Pod "azuredisk-volume-tester-nmsfx": Phase="Pending", Reason="", readiness=false. Elapsed: 4.116255297s
Jan 17 18:13:26.490: INFO: Pod "azuredisk-volume-tester-nmsfx": Phase="Pending", Reason="", readiness=false. Elapsed: 6.115828151s
Jan 17 18:13:28.492: INFO: Pod "azuredisk-volume-tester-nmsfx": Phase="Pending", Reason="", readiness=false. Elapsed: 8.117704835s
Jan 17 18:13:30.490: INFO: Pod "azuredisk-volume-tester-nmsfx": Phase="Pending", Reason="", readiness=false. Elapsed: 10.116100778s
... skipping 22 lines ...
Jan 17 18:14:16.493: INFO: Pod "azuredisk-volume-tester-nmsfx": Phase="Running", Reason="", readiness=true. Elapsed: 56.118416253s
Jan 17 18:14:18.493: INFO: Pod "azuredisk-volume-tester-nmsfx": Phase="Running", Reason="", readiness=true. Elapsed: 58.118559431s
Jan 17 18:14:20.492: INFO: Pod "azuredisk-volume-tester-nmsfx": Phase="Running", Reason="", readiness=false. Elapsed: 1m0.118094134s
Jan 17 18:14:22.492: INFO: Pod "azuredisk-volume-tester-nmsfx": Phase="Running", Reason="", readiness=false. Elapsed: 1m2.11812949s
Jan 17 18:14:24.497: INFO: Pod "azuredisk-volume-tester-nmsfx": Phase="Succeeded", Reason="", readiness=false. Elapsed: 1m4.123014176s
[1mSTEP:[0m Saw pod success [38;5;243m01/17/23 18:14:24.497[0m
Jan 17 18:14:24.497: INFO: Pod "azuredisk-volume-tester-nmsfx" satisfied condition "Succeeded or Failed"
Jan 17 18:14:24.497: INFO: deleting Pod "azuredisk-4923"/"azuredisk-volume-tester-nmsfx"
Jan 17 18:14:24.590: INFO: Pod azuredisk-volume-tester-nmsfx has the following logs: hello world

[1mSTEP:[0m Deleting pod azuredisk-volume-tester-nmsfx in namespace azuredisk-4923 [38;5;243m01/17/23 18:14:24.59[0m
Jan 17 18:14:24.665: INFO: deleting PVC "azuredisk-4923"/"pvc-wt6l8"
Jan 17 18:14:24.665: INFO: Deleting PersistentVolumeClaim "pvc-wt6l8"
... skipping 38 lines ...
    Jan 17 18:13:18.134: INFO: PersistentVolumeClaim pvc-wt6l8 found but phase is Pending instead of Bound.
    Jan 17 18:13:20.195: INFO: PersistentVolumeClaim pvc-wt6l8 found and phase=Bound (4.177143933s)
    [1mSTEP:[0m checking the PVC [38;5;243m01/17/23 18:13:20.195[0m
    [1mSTEP:[0m validating provisioned PV [38;5;243m01/17/23 18:13:20.254[0m
    [1mSTEP:[0m checking the PV [38;5;243m01/17/23 18:13:20.311[0m
    [1mSTEP:[0m deploying the pod [38;5;243m01/17/23 18:13:20.312[0m
    [1mSTEP:[0m checking that the pods command exits with no error [38;5;243m01/17/23 18:13:20.374[0m
    Jan 17 18:13:20.374: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-nmsfx" in namespace "azuredisk-4923" to be "Succeeded or Failed"
    Jan 17 18:13:20.431: INFO: Pod "azuredisk-volume-tester-nmsfx": Phase="Pending", Reason="", readiness=false. Elapsed: 57.230392ms
    Jan 17 18:13:22.490: INFO: Pod "azuredisk-volume-tester-nmsfx": Phase="Pending", Reason="", readiness=false. Elapsed: 2.116174066s
    Jan 17 18:13:24.490: INFO: Pod "azuredisk-volume-tester-nmsfx": Phase="Pending", Reason="", readiness=false. Elapsed: 4.116255297s
    Jan 17 18:13:26.490: INFO: Pod "azuredisk-volume-tester-nmsfx": Phase="Pending", Reason="", readiness=false. Elapsed: 6.115828151s
    Jan 17 18:13:28.492: INFO: Pod "azuredisk-volume-tester-nmsfx": Phase="Pending", Reason="", readiness=false. Elapsed: 8.117704835s
    Jan 17 18:13:30.490: INFO: Pod "azuredisk-volume-tester-nmsfx": Phase="Pending", Reason="", readiness=false. Elapsed: 10.116100778s
... skipping 22 lines ...
    Jan 17 18:14:16.493: INFO: Pod "azuredisk-volume-tester-nmsfx": Phase="Running", Reason="", readiness=true. Elapsed: 56.118416253s
    Jan 17 18:14:18.493: INFO: Pod "azuredisk-volume-tester-nmsfx": Phase="Running", Reason="", readiness=true. Elapsed: 58.118559431s
    Jan 17 18:14:20.492: INFO: Pod "azuredisk-volume-tester-nmsfx": Phase="Running", Reason="", readiness=false. Elapsed: 1m0.118094134s
    Jan 17 18:14:22.492: INFO: Pod "azuredisk-volume-tester-nmsfx": Phase="Running", Reason="", readiness=false. Elapsed: 1m2.11812949s
    Jan 17 18:14:24.497: INFO: Pod "azuredisk-volume-tester-nmsfx": Phase="Succeeded", Reason="", readiness=false. Elapsed: 1m4.123014176s
    [1mSTEP:[0m Saw pod success [38;5;243m01/17/23 18:14:24.497[0m
    Jan 17 18:14:24.497: INFO: Pod "azuredisk-volume-tester-nmsfx" satisfied condition "Succeeded or Failed"
    Jan 17 18:14:24.497: INFO: deleting Pod "azuredisk-4923"/"azuredisk-volume-tester-nmsfx"
    Jan 17 18:14:24.590: INFO: Pod azuredisk-volume-tester-nmsfx has the following logs: hello world

    [1mSTEP:[0m Deleting pod azuredisk-volume-tester-nmsfx in namespace azuredisk-4923 [38;5;243m01/17/23 18:14:24.59[0m
    Jan 17 18:14:24.665: INFO: deleting PVC "azuredisk-4923"/"pvc-wt6l8"
    Jan 17 18:14:24.665: INFO: Deleting PersistentVolumeClaim "pvc-wt6l8"
... skipping 39 lines ...
[1mSTEP:[0m checking the PVC [38;5;243m01/17/23 18:15:10.706[0m
[1mSTEP:[0m validating provisioned PV [38;5;243m01/17/23 18:15:10.764[0m
[1mSTEP:[0m checking the PV [38;5;243m01/17/23 18:15:10.822[0m
[1mSTEP:[0m setting up the pod [38;5;243m01/17/23 18:15:10.822[0m
I0117 18:15:10.822881   43786 resource_setup.go:62] adding PV (pvc-3a39daf3-2bee-46b8-a17a-2a0cf65b0c10) to pod ()
[1mSTEP:[0m deploying the pod [38;5;243m01/17/23 18:15:10.822[0m
[1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/17/23 18:15:10.888[0m
Jan 17 18:15:10.889: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-wk85n" in namespace "azuredisk-6646" to be "Succeeded or Failed"
Jan 17 18:15:10.947: INFO: Pod "azuredisk-volume-tester-wk85n": Phase="Pending", Reason="", readiness=false. Elapsed: 58.336886ms
Jan 17 18:15:13.006: INFO: Pod "azuredisk-volume-tester-wk85n": Phase="Pending", Reason="", readiness=false. Elapsed: 2.117765402s
Jan 17 18:15:15.006: INFO: Pod "azuredisk-volume-tester-wk85n": Phase="Pending", Reason="", readiness=false. Elapsed: 4.117534006s
Jan 17 18:15:17.007: INFO: Pod "azuredisk-volume-tester-wk85n": Phase="Pending", Reason="", readiness=false. Elapsed: 6.118336878s
Jan 17 18:15:19.007: INFO: Pod "azuredisk-volume-tester-wk85n": Phase="Pending", Reason="", readiness=false. Elapsed: 8.117901588s
Jan 17 18:15:21.006: INFO: Pod "azuredisk-volume-tester-wk85n": Phase="Pending", Reason="", readiness=false. Elapsed: 10.117583612s
... skipping 22 lines ...
Jan 17 18:16:07.008: INFO: Pod "azuredisk-volume-tester-wk85n": Phase="Running", Reason="", readiness=true. Elapsed: 56.119016906s
Jan 17 18:16:09.009: INFO: Pod "azuredisk-volume-tester-wk85n": Phase="Running", Reason="", readiness=false. Elapsed: 58.120511093s
Jan 17 18:16:11.010: INFO: Pod "azuredisk-volume-tester-wk85n": Phase="Running", Reason="", readiness=false. Elapsed: 1m0.121314448s
Jan 17 18:16:13.008: INFO: Pod "azuredisk-volume-tester-wk85n": Phase="Running", Reason="", readiness=false. Elapsed: 1m2.119507414s
Jan 17 18:16:15.008: INFO: Pod "azuredisk-volume-tester-wk85n": Phase="Succeeded", Reason="", readiness=false. Elapsed: 1m4.119074792s
[1mSTEP:[0m Saw pod success [38;5;243m01/17/23 18:16:15.008[0m
Jan 17 18:16:15.008: INFO: Pod "azuredisk-volume-tester-wk85n" satisfied condition "Succeeded or Failed"
Jan 17 18:16:15.008: INFO: deleting Pod "azuredisk-6646"/"azuredisk-volume-tester-wk85n"
Jan 17 18:16:15.096: INFO: Pod azuredisk-volume-tester-wk85n has the following logs: hello world

[1mSTEP:[0m Deleting pod azuredisk-volume-tester-wk85n in namespace azuredisk-6646 [38;5;243m01/17/23 18:16:15.096[0m
Jan 17 18:16:15.169: INFO: deleting PVC "azuredisk-6646"/"pvc-xvq4c"
Jan 17 18:16:15.169: INFO: Deleting PersistentVolumeClaim "pvc-xvq4c"
... skipping 40 lines ...
    [1mSTEP:[0m checking the PVC [38;5;243m01/17/23 18:15:10.706[0m
    [1mSTEP:[0m validating provisioned PV [38;5;243m01/17/23 18:15:10.764[0m
    [1mSTEP:[0m checking the PV [38;5;243m01/17/23 18:15:10.822[0m
    [1mSTEP:[0m setting up the pod [38;5;243m01/17/23 18:15:10.822[0m
    I0117 18:15:10.822881   43786 resource_setup.go:62] adding PV (pvc-3a39daf3-2bee-46b8-a17a-2a0cf65b0c10) to pod ()
    [1mSTEP:[0m deploying the pod [38;5;243m01/17/23 18:15:10.822[0m
    [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/17/23 18:15:10.888[0m
    Jan 17 18:15:10.889: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-wk85n" in namespace "azuredisk-6646" to be "Succeeded or Failed"
    Jan 17 18:15:10.947: INFO: Pod "azuredisk-volume-tester-wk85n": Phase="Pending", Reason="", readiness=false. Elapsed: 58.336886ms
    Jan 17 18:15:13.006: INFO: Pod "azuredisk-volume-tester-wk85n": Phase="Pending", Reason="", readiness=false. Elapsed: 2.117765402s
    Jan 17 18:15:15.006: INFO: Pod "azuredisk-volume-tester-wk85n": Phase="Pending", Reason="", readiness=false. Elapsed: 4.117534006s
    Jan 17 18:15:17.007: INFO: Pod "azuredisk-volume-tester-wk85n": Phase="Pending", Reason="", readiness=false. Elapsed: 6.118336878s
    Jan 17 18:15:19.007: INFO: Pod "azuredisk-volume-tester-wk85n": Phase="Pending", Reason="", readiness=false. Elapsed: 8.117901588s
    Jan 17 18:15:21.006: INFO: Pod "azuredisk-volume-tester-wk85n": Phase="Pending", Reason="", readiness=false. Elapsed: 10.117583612s
... skipping 22 lines ...
    Jan 17 18:16:07.008: INFO: Pod "azuredisk-volume-tester-wk85n": Phase="Running", Reason="", readiness=true. Elapsed: 56.119016906s
    Jan 17 18:16:09.009: INFO: Pod "azuredisk-volume-tester-wk85n": Phase="Running", Reason="", readiness=false. Elapsed: 58.120511093s
    Jan 17 18:16:11.010: INFO: Pod "azuredisk-volume-tester-wk85n": Phase="Running", Reason="", readiness=false. Elapsed: 1m0.121314448s
    Jan 17 18:16:13.008: INFO: Pod "azuredisk-volume-tester-wk85n": Phase="Running", Reason="", readiness=false. Elapsed: 1m2.119507414s
    Jan 17 18:16:15.008: INFO: Pod "azuredisk-volume-tester-wk85n": Phase="Succeeded", Reason="", readiness=false. Elapsed: 1m4.119074792s
    [1mSTEP:[0m Saw pod success [38;5;243m01/17/23 18:16:15.008[0m
    Jan 17 18:16:15.008: INFO: Pod "azuredisk-volume-tester-wk85n" satisfied condition "Succeeded or Failed"
    Jan 17 18:16:15.008: INFO: deleting Pod "azuredisk-6646"/"azuredisk-volume-tester-wk85n"
    Jan 17 18:16:15.096: INFO: Pod azuredisk-volume-tester-wk85n has the following logs: hello world

    [1mSTEP:[0m Deleting pod azuredisk-volume-tester-wk85n in namespace azuredisk-6646 [38;5;243m01/17/23 18:16:15.096[0m
    Jan 17 18:16:15.169: INFO: deleting PVC "azuredisk-6646"/"pvc-xvq4c"
    Jan 17 18:16:15.169: INFO: Deleting PersistentVolumeClaim "pvc-xvq4c"
... skipping 39 lines ...
[1mSTEP:[0m checking the PVC [38;5;243m01/17/23 18:17:01.266[0m
[1mSTEP:[0m validating provisioned PV [38;5;243m01/17/23 18:17:01.326[0m
[1mSTEP:[0m checking the PV [38;5;243m01/17/23 18:17:01.384[0m
[1mSTEP:[0m setting up the pod [38;5;243m01/17/23 18:17:01.385[0m
I0117 18:17:01.385174   43786 resource_setup.go:62] adding PV (pvc-c8d7feb0-2992-4126-9eb2-aa09b520ed8c) to pod ()
[1mSTEP:[0m deploying the pod [38;5;243m01/17/23 18:17:01.385[0m
[1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/17/23 18:17:01.451[0m
Jan 17 18:17:01.452: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-g8htv" in namespace "azuredisk-3610" to be "Succeeded or Failed"
Jan 17 18:17:01.510: INFO: Pod "azuredisk-volume-tester-g8htv": Phase="Pending", Reason="", readiness=false. Elapsed: 58.296305ms
Jan 17 18:17:03.571: INFO: Pod "azuredisk-volume-tester-g8htv": Phase="Pending", Reason="", readiness=false. Elapsed: 2.119460247s
Jan 17 18:17:05.570: INFO: Pod "azuredisk-volume-tester-g8htv": Phase="Pending", Reason="", readiness=false. Elapsed: 4.118192701s
Jan 17 18:17:07.569: INFO: Pod "azuredisk-volume-tester-g8htv": Phase="Pending", Reason="", readiness=false. Elapsed: 6.11761802s
Jan 17 18:17:09.569: INFO: Pod "azuredisk-volume-tester-g8htv": Phase="Pending", Reason="", readiness=false. Elapsed: 8.117296421s
Jan 17 18:17:11.570: INFO: Pod "azuredisk-volume-tester-g8htv": Phase="Pending", Reason="", readiness=false. Elapsed: 10.118578061s
... skipping 22 lines ...
Jan 17 18:17:57.573: INFO: Pod "azuredisk-volume-tester-g8htv": Phase="Running", Reason="", readiness=true. Elapsed: 56.121346685s
Jan 17 18:17:59.571: INFO: Pod "azuredisk-volume-tester-g8htv": Phase="Running", Reason="", readiness=true. Elapsed: 58.119599952s
Jan 17 18:18:01.574: INFO: Pod "azuredisk-volume-tester-g8htv": Phase="Running", Reason="", readiness=false. Elapsed: 1m0.122013726s
Jan 17 18:18:03.577: INFO: Pod "azuredisk-volume-tester-g8htv": Phase="Running", Reason="", readiness=false. Elapsed: 1m2.125634247s
Jan 17 18:18:05.572: INFO: Pod "azuredisk-volume-tester-g8htv": Phase="Succeeded", Reason="", readiness=false. Elapsed: 1m4.12075624s
[1mSTEP:[0m Saw pod success [38;5;243m01/17/23 18:18:05.573[0m
Jan 17 18:18:05.573: INFO: Pod "azuredisk-volume-tester-g8htv" satisfied condition "Succeeded or Failed"
Jan 17 18:18:05.573: INFO: deleting Pod "azuredisk-3610"/"azuredisk-volume-tester-g8htv"
Jan 17 18:18:05.658: INFO: Pod azuredisk-volume-tester-g8htv has the following logs: hello world

[1mSTEP:[0m Deleting pod azuredisk-volume-tester-g8htv in namespace azuredisk-3610 [38;5;243m01/17/23 18:18:05.658[0m
Jan 17 18:18:05.734: INFO: deleting PVC "azuredisk-3610"/"pvc-7ftlq"
Jan 17 18:18:05.734: INFO: Deleting PersistentVolumeClaim "pvc-7ftlq"
... skipping 40 lines ...
    [1mSTEP:[0m checking the PVC [38;5;243m01/17/23 18:17:01.266[0m
    [1mSTEP:[0m validating provisioned PV [38;5;243m01/17/23 18:17:01.326[0m
    [1mSTEP:[0m checking the PV [38;5;243m01/17/23 18:17:01.384[0m
    [1mSTEP:[0m setting up the pod [38;5;243m01/17/23 18:17:01.385[0m
    I0117 18:17:01.385174   43786 resource_setup.go:62] adding PV (pvc-c8d7feb0-2992-4126-9eb2-aa09b520ed8c) to pod ()
    [1mSTEP:[0m deploying the pod [38;5;243m01/17/23 18:17:01.385[0m
    [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/17/23 18:17:01.451[0m
    Jan 17 18:17:01.452: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-g8htv" in namespace "azuredisk-3610" to be "Succeeded or Failed"
    Jan 17 18:17:01.510: INFO: Pod "azuredisk-volume-tester-g8htv": Phase="Pending", Reason="", readiness=false. Elapsed: 58.296305ms
    Jan 17 18:17:03.571: INFO: Pod "azuredisk-volume-tester-g8htv": Phase="Pending", Reason="", readiness=false. Elapsed: 2.119460247s
    Jan 17 18:17:05.570: INFO: Pod "azuredisk-volume-tester-g8htv": Phase="Pending", Reason="", readiness=false. Elapsed: 4.118192701s
    Jan 17 18:17:07.569: INFO: Pod "azuredisk-volume-tester-g8htv": Phase="Pending", Reason="", readiness=false. Elapsed: 6.11761802s
    Jan 17 18:17:09.569: INFO: Pod "azuredisk-volume-tester-g8htv": Phase="Pending", Reason="", readiness=false. Elapsed: 8.117296421s
    Jan 17 18:17:11.570: INFO: Pod "azuredisk-volume-tester-g8htv": Phase="Pending", Reason="", readiness=false. Elapsed: 10.118578061s
... skipping 22 lines ...
    Jan 17 18:17:57.573: INFO: Pod "azuredisk-volume-tester-g8htv": Phase="Running", Reason="", readiness=true. Elapsed: 56.121346685s
    Jan 17 18:17:59.571: INFO: Pod "azuredisk-volume-tester-g8htv": Phase="Running", Reason="", readiness=true. Elapsed: 58.119599952s
    Jan 17 18:18:01.574: INFO: Pod "azuredisk-volume-tester-g8htv": Phase="Running", Reason="", readiness=false. Elapsed: 1m0.122013726s
    Jan 17 18:18:03.577: INFO: Pod "azuredisk-volume-tester-g8htv": Phase="Running", Reason="", readiness=false. Elapsed: 1m2.125634247s
    Jan 17 18:18:05.572: INFO: Pod "azuredisk-volume-tester-g8htv": Phase="Succeeded", Reason="", readiness=false. Elapsed: 1m4.12075624s
    [1mSTEP:[0m Saw pod success [38;5;243m01/17/23 18:18:05.573[0m
    Jan 17 18:18:05.573: INFO: Pod "azuredisk-volume-tester-g8htv" satisfied condition "Succeeded or Failed"
    Jan 17 18:18:05.573: INFO: deleting Pod "azuredisk-3610"/"azuredisk-volume-tester-g8htv"
    Jan 17 18:18:05.658: INFO: Pod azuredisk-volume-tester-g8htv has the following logs: hello world

    [1mSTEP:[0m Deleting pod azuredisk-volume-tester-g8htv in namespace azuredisk-3610 [38;5;243m01/17/23 18:18:05.658[0m
    Jan 17 18:18:05.734: INFO: deleting PVC "azuredisk-3610"/"pvc-7ftlq"
    Jan 17 18:18:05.734: INFO: Deleting PersistentVolumeClaim "pvc-7ftlq"
... skipping 111 lines ...
[1mSTEP:[0m checking the PVC [38;5;243m01/17/23 18:18:53.975[0m
[1mSTEP:[0m validating provisioned PV [38;5;243m01/17/23 18:18:54.034[0m
[1mSTEP:[0m checking the PV [38;5;243m01/17/23 18:18:54.092[0m
[1mSTEP:[0m setting up the pod [38;5;243m01/17/23 18:18:54.093[0m
I0117 18:18:54.093654   43786 resource_setup.go:62] adding PV (pvc-d2c00bf2-e84b-4a07-a00e-a27857b4b321) to pod ()
[1mSTEP:[0m deploying the pod [38;5;243m01/17/23 18:18:54.093[0m
[1mSTEP:[0m checking that the pod's command exits with an error [38;5;243m01/17/23 18:18:54.156[0m
Jan 17 18:18:54.157: INFO: Waiting up to 10m0s for pod "azuredisk-volume-tester-vx7h2" in namespace "azuredisk-622" to be "Error status code"
Jan 17 18:18:54.215: INFO: Pod "azuredisk-volume-tester-vx7h2": Phase="Pending", Reason="", readiness=false. Elapsed: 58.568312ms
Jan 17 18:18:56.273: INFO: Pod "azuredisk-volume-tester-vx7h2": Phase="Pending", Reason="", readiness=false. Elapsed: 2.116706224s
Jan 17 18:18:58.275: INFO: Pod "azuredisk-volume-tester-vx7h2": Phase="Pending", Reason="", readiness=false. Elapsed: 4.11832993s
Jan 17 18:19:00.275: INFO: Pod "azuredisk-volume-tester-vx7h2": Phase="Pending", Reason="", readiness=false. Elapsed: 6.118378441s
Jan 17 18:19:02.277: INFO: Pod "azuredisk-volume-tester-vx7h2": Phase="Pending", Reason="", readiness=false. Elapsed: 8.119860289s
Jan 17 18:19:04.277: INFO: Pod "azuredisk-volume-tester-vx7h2": Phase="Pending", Reason="", readiness=false. Elapsed: 10.120013018s
... skipping 31 lines ...
Jan 17 18:20:08.283: INFO: Pod "azuredisk-volume-tester-vx7h2": Phase="Running", Reason="", readiness=true. Elapsed: 1m14.126239326s
Jan 17 18:20:10.277: INFO: Pod "azuredisk-volume-tester-vx7h2": Phase="Running", Reason="", readiness=true. Elapsed: 1m16.120271095s
Jan 17 18:20:12.276: INFO: Pod "azuredisk-volume-tester-vx7h2": Phase="Running", Reason="", readiness=true. Elapsed: 1m18.119543475s
Jan 17 18:20:14.276: INFO: Pod "azuredisk-volume-tester-vx7h2": Phase="Running", Reason="", readiness=false. Elapsed: 1m20.119538208s
Jan 17 18:20:16.277: INFO: Pod "azuredisk-volume-tester-vx7h2": Phase="Running", Reason="", readiness=false. Elapsed: 1m22.120285644s
Jan 17 18:20:18.277: INFO: Pod "azuredisk-volume-tester-vx7h2": Phase="Running", Reason="", readiness=false. Elapsed: 1m24.119770021s
Jan 17 18:20:20.276: INFO: Pod "azuredisk-volume-tester-vx7h2": Phase="Failed", Reason="", readiness=false. Elapsed: 1m26.119008821s
[1mSTEP:[0m Saw pod failure [38;5;243m01/17/23 18:20:20.276[0m
Jan 17 18:20:20.276: INFO: Pod "azuredisk-volume-tester-vx7h2" satisfied condition "Error status code"
[1mSTEP:[0m checking that pod logs contain expected message [38;5;243m01/17/23 18:20:20.276[0m
Jan 17 18:20:20.374: INFO: deleting Pod "azuredisk-622"/"azuredisk-volume-tester-vx7h2"
Jan 17 18:20:20.438: INFO: Pod azuredisk-volume-tester-vx7h2 has the following logs: out-file : Access to the path 'C:\mnt\test-1\data' is denied.
At line:1 char:1
+ echo $null >> C:\mnt\test-1\data
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
... skipping 49 lines ...
    [1mSTEP:[0m checking the PVC [38;5;243m01/17/23 18:18:53.975[0m
    [1mSTEP:[0m validating provisioned PV [38;5;243m01/17/23 18:18:54.034[0m
    [1mSTEP:[0m checking the PV [38;5;243m01/17/23 18:18:54.092[0m
    [1mSTEP:[0m setting up the pod [38;5;243m01/17/23 18:18:54.093[0m
    I0117 18:18:54.093654   43786 resource_setup.go:62] adding PV (pvc-d2c00bf2-e84b-4a07-a00e-a27857b4b321) to pod ()
    [1mSTEP:[0m deploying the pod [38;5;243m01/17/23 18:18:54.093[0m
    [1mSTEP:[0m checking that the pod's command exits with an error [38;5;243m01/17/23 18:18:54.156[0m
    Jan 17 18:18:54.157: INFO: Waiting up to 10m0s for pod "azuredisk-volume-tester-vx7h2" in namespace "azuredisk-622" to be "Error status code"
    Jan 17 18:18:54.215: INFO: Pod "azuredisk-volume-tester-vx7h2": Phase="Pending", Reason="", readiness=false. Elapsed: 58.568312ms
    Jan 17 18:18:56.273: INFO: Pod "azuredisk-volume-tester-vx7h2": Phase="Pending", Reason="", readiness=false. Elapsed: 2.116706224s
    Jan 17 18:18:58.275: INFO: Pod "azuredisk-volume-tester-vx7h2": Phase="Pending", Reason="", readiness=false. Elapsed: 4.11832993s
    Jan 17 18:19:00.275: INFO: Pod "azuredisk-volume-tester-vx7h2": Phase="Pending", Reason="", readiness=false. Elapsed: 6.118378441s
    Jan 17 18:19:02.277: INFO: Pod "azuredisk-volume-tester-vx7h2": Phase="Pending", Reason="", readiness=false. Elapsed: 8.119860289s
    Jan 17 18:19:04.277: INFO: Pod "azuredisk-volume-tester-vx7h2": Phase="Pending", Reason="", readiness=false. Elapsed: 10.120013018s
... skipping 31 lines ...
    Jan 17 18:20:08.283: INFO: Pod "azuredisk-volume-tester-vx7h2": Phase="Running", Reason="", readiness=true. Elapsed: 1m14.126239326s
    Jan 17 18:20:10.277: INFO: Pod "azuredisk-volume-tester-vx7h2": Phase="Running", Reason="", readiness=true. Elapsed: 1m16.120271095s
    Jan 17 18:20:12.276: INFO: Pod "azuredisk-volume-tester-vx7h2": Phase="Running", Reason="", readiness=true. Elapsed: 1m18.119543475s
    Jan 17 18:20:14.276: INFO: Pod "azuredisk-volume-tester-vx7h2": Phase="Running", Reason="", readiness=false. Elapsed: 1m20.119538208s
    Jan 17 18:20:16.277: INFO: Pod "azuredisk-volume-tester-vx7h2": Phase="Running", Reason="", readiness=false. Elapsed: 1m22.120285644s
    Jan 17 18:20:18.277: INFO: Pod "azuredisk-volume-tester-vx7h2": Phase="Running", Reason="", readiness=false. Elapsed: 1m24.119770021s
    Jan 17 18:20:20.276: INFO: Pod "azuredisk-volume-tester-vx7h2": Phase="Failed", Reason="", readiness=false. Elapsed: 1m26.119008821s
    [1mSTEP:[0m Saw pod failure [38;5;243m01/17/23 18:20:20.276[0m
    Jan 17 18:20:20.276: INFO: Pod "azuredisk-volume-tester-vx7h2" satisfied condition "Error status code"
    [1mSTEP:[0m checking that pod logs contain expected message [38;5;243m01/17/23 18:20:20.276[0m
    Jan 17 18:20:20.374: INFO: deleting Pod "azuredisk-622"/"azuredisk-volume-tester-vx7h2"
    Jan 17 18:20:20.438: INFO: Pod azuredisk-volume-tester-vx7h2 has the following logs: out-file : Access to the path 'C:\mnt\test-1\data' is denied.
    At line:1 char:1
    + echo $null >> C:\mnt\test-1\data
    + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
... skipping 485 lines ...
Jan 17 18:26:45.432: INFO: deployment status: v1.DeploymentStatus{ObservedGeneration:1, Replicas:1, UpdatedReplicas:1, ReadyReplicas:0, AvailableReplicas:0, UnavailableReplicas:1, Conditions:[]v1.DeploymentCondition{v1.DeploymentCondition{Type:"Available", Status:"False", LastUpdateTime:time.Date(2023, time.January, 17, 18, 26, 3, 0, time.Local), LastTransitionTime:time.Date(2023, time.January, 17, 18, 26, 3, 0, time.Local), Reason:"MinimumReplicasUnavailable", Message:"Deployment does not have minimum availability."}, v1.DeploymentCondition{Type:"Progressing", Status:"True", LastUpdateTime:time.Date(2023, time.January, 17, 18, 26, 3, 0, time.Local), LastTransitionTime:time.Date(2023, time.January, 17, 18, 26, 3, 0, time.Local), Reason:"ReplicaSetUpdated", Message:"ReplicaSet \"azuredisk-volume-tester-ts4nm-68d69dbb6c\" is progressing."}}, CollisionCount:(*int32)(nil)}
Jan 17 18:26:47.432: INFO: deployment status: v1.DeploymentStatus{ObservedGeneration:1, Replicas:1, UpdatedReplicas:1, ReadyReplicas:0, AvailableReplicas:0, UnavailableReplicas:1, Conditions:[]v1.DeploymentCondition{v1.DeploymentCondition{Type:"Available", Status:"False", LastUpdateTime:time.Date(2023, time.January, 17, 18, 26, 3, 0, time.Local), LastTransitionTime:time.Date(2023, time.January, 17, 18, 26, 3, 0, time.Local), Reason:"MinimumReplicasUnavailable", Message:"Deployment does not have minimum availability."}, v1.DeploymentCondition{Type:"Progressing", Status:"True", LastUpdateTime:time.Date(2023, time.January, 17, 18, 26, 3, 0, time.Local), LastTransitionTime:time.Date(2023, time.January, 17, 18, 26, 3, 0, time.Local), Reason:"ReplicaSetUpdated", Message:"ReplicaSet \"azuredisk-volume-tester-ts4nm-68d69dbb6c\" is progressing."}}, CollisionCount:(*int32)(nil)}
[1mSTEP:[0m checking that the pod is running [38;5;243m01/17/23 18:26:49.551[0m
[1mSTEP:[0m check pod exec [38;5;243m01/17/23 18:26:49.736[0m
Jan 17 18:26:49.737: INFO: Running '/usr/local/bin/kubectl --kubeconfig=/home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure/kubeconfig --namespace=azuredisk-8905 exec azuredisk-volume-tester-ts4nm-68d69dbb6c-dwsjd -- cmd /c type C:\mnt\test-1\data.txt'
Jan 17 18:26:50.591: INFO: rc: 1
Jan 17 18:26:50.591: INFO: Error waiting for output "hello world\r\n" in pod "azuredisk-volume-tester-ts4nm-68d69dbb6c-dwsjd": error running /usr/local/bin/kubectl --kubeconfig=/home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure/kubeconfig --namespace=azuredisk-8905 exec azuredisk-volume-tester-ts4nm-68d69dbb6c-dwsjd -- cmd /c type C:\mnt\test-1\data.txt:
Command stdout:

stderr:
The system cannot find the file specified.
command terminated with exit code 1

error:
exit status 1.
Jan 17 18:26:52.592: INFO: Running '/usr/local/bin/kubectl --kubeconfig=/home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure/kubeconfig --namespace=azuredisk-8905 exec azuredisk-volume-tester-ts4nm-68d69dbb6c-dwsjd -- cmd /c type C:\mnt\test-1\data.txt'
Jan 17 18:26:53.426: INFO: rc: 1
Jan 17 18:26:53.426: INFO: Error waiting for output "hello world\r\n" in pod "azuredisk-volume-tester-ts4nm-68d69dbb6c-dwsjd": error running /usr/local/bin/kubectl --kubeconfig=/home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure/kubeconfig --namespace=azuredisk-8905 exec azuredisk-volume-tester-ts4nm-68d69dbb6c-dwsjd -- cmd /c type C:\mnt\test-1\data.txt:
Command stdout:

stderr:
The system cannot find the file specified.
command terminated with exit code 1

error:
exit status 1.
Jan 17 18:26:54.592: INFO: Running '/usr/local/bin/kubectl --kubeconfig=/home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure/kubeconfig --namespace=azuredisk-8905 exec azuredisk-volume-tester-ts4nm-68d69dbb6c-dwsjd -- cmd /c type C:\mnt\test-1\data.txt'
Jan 17 18:26:55.394: INFO: stderr: ""
Jan 17 18:26:55.394: INFO: stdout: "hello world\r\n"
[1mSTEP:[0m deleting the pod for deployment [38;5;243m01/17/23 18:26:55.394[0m
Jan 17 18:26:55.394: INFO: Deleting pod "azuredisk-volume-tester-ts4nm-68d69dbb6c-dwsjd" in namespace "azuredisk-8905"
... skipping 89 lines ...
    Jan 17 18:26:45.432: INFO: deployment status: v1.DeploymentStatus{ObservedGeneration:1, Replicas:1, UpdatedReplicas:1, ReadyReplicas:0, AvailableReplicas:0, UnavailableReplicas:1, Conditions:[]v1.DeploymentCondition{v1.DeploymentCondition{Type:"Available", Status:"False", LastUpdateTime:time.Date(2023, time.January, 17, 18, 26, 3, 0, time.Local), LastTransitionTime:time.Date(2023, time.January, 17, 18, 26, 3, 0, time.Local), Reason:"MinimumReplicasUnavailable", Message:"Deployment does not have minimum availability."}, v1.DeploymentCondition{Type:"Progressing", Status:"True", LastUpdateTime:time.Date(2023, time.January, 17, 18, 26, 3, 0, time.Local), LastTransitionTime:time.Date(2023, time.January, 17, 18, 26, 3, 0, time.Local), Reason:"ReplicaSetUpdated", Message:"ReplicaSet \"azuredisk-volume-tester-ts4nm-68d69dbb6c\" is progressing."}}, CollisionCount:(*int32)(nil)}
    Jan 17 18:26:47.432: INFO: deployment status: v1.DeploymentStatus{ObservedGeneration:1, Replicas:1, UpdatedReplicas:1, ReadyReplicas:0, AvailableReplicas:0, UnavailableReplicas:1, Conditions:[]v1.DeploymentCondition{v1.DeploymentCondition{Type:"Available", Status:"False", LastUpdateTime:time.Date(2023, time.January, 17, 18, 26, 3, 0, time.Local), LastTransitionTime:time.Date(2023, time.January, 17, 18, 26, 3, 0, time.Local), Reason:"MinimumReplicasUnavailable", Message:"Deployment does not have minimum availability."}, v1.DeploymentCondition{Type:"Progressing", Status:"True", LastUpdateTime:time.Date(2023, time.January, 17, 18, 26, 3, 0, time.Local), LastTransitionTime:time.Date(2023, time.January, 17, 18, 26, 3, 0, time.Local), Reason:"ReplicaSetUpdated", Message:"ReplicaSet \"azuredisk-volume-tester-ts4nm-68d69dbb6c\" is progressing."}}, CollisionCount:(*int32)(nil)}
    [1mSTEP:[0m checking that the pod is running [38;5;243m01/17/23 18:26:49.551[0m
    [1mSTEP:[0m check pod exec [38;5;243m01/17/23 18:26:49.736[0m
    Jan 17 18:26:49.737: INFO: Running '/usr/local/bin/kubectl --kubeconfig=/home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure/kubeconfig --namespace=azuredisk-8905 exec azuredisk-volume-tester-ts4nm-68d69dbb6c-dwsjd -- cmd /c type C:\mnt\test-1\data.txt'
    Jan 17 18:26:50.591: INFO: rc: 1
    Jan 17 18:26:50.591: INFO: Error waiting for output "hello world\r\n" in pod "azuredisk-volume-tester-ts4nm-68d69dbb6c-dwsjd": error running /usr/local/bin/kubectl --kubeconfig=/home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure/kubeconfig --namespace=azuredisk-8905 exec azuredisk-volume-tester-ts4nm-68d69dbb6c-dwsjd -- cmd /c type C:\mnt\test-1\data.txt:
    Command stdout:

    stderr:
    The system cannot find the file specified.
    command terminated with exit code 1

    error:
    exit status 1.
    Jan 17 18:26:52.592: INFO: Running '/usr/local/bin/kubectl --kubeconfig=/home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure/kubeconfig --namespace=azuredisk-8905 exec azuredisk-volume-tester-ts4nm-68d69dbb6c-dwsjd -- cmd /c type C:\mnt\test-1\data.txt'
    Jan 17 18:26:53.426: INFO: rc: 1
    Jan 17 18:26:53.426: INFO: Error waiting for output "hello world\r\n" in pod "azuredisk-volume-tester-ts4nm-68d69dbb6c-dwsjd": error running /usr/local/bin/kubectl --kubeconfig=/home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure/kubeconfig --namespace=azuredisk-8905 exec azuredisk-volume-tester-ts4nm-68d69dbb6c-dwsjd -- cmd /c type C:\mnt\test-1\data.txt:
    Command stdout:

    stderr:
    The system cannot find the file specified.
    command terminated with exit code 1

    error:
    exit status 1.
    Jan 17 18:26:54.592: INFO: Running '/usr/local/bin/kubectl --kubeconfig=/home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure/kubeconfig --namespace=azuredisk-8905 exec azuredisk-volume-tester-ts4nm-68d69dbb6c-dwsjd -- cmd /c type C:\mnt\test-1\data.txt'
    Jan 17 18:26:55.394: INFO: stderr: ""
    Jan 17 18:26:55.394: INFO: stdout: "hello world\r\n"
    [1mSTEP:[0m deleting the pod for deployment [38;5;243m01/17/23 18:26:55.394[0m
    Jan 17 18:26:55.394: INFO: Deleting pod "azuredisk-volume-tester-ts4nm-68d69dbb6c-dwsjd" in namespace "azuredisk-8905"
... skipping 318 lines ...
[1mSTEP:[0m checking the PVC [38;5;243m01/17/23 18:29:03.986[0m
[1mSTEP:[0m validating provisioned PV [38;5;243m01/17/23 18:29:04.044[0m
[1mSTEP:[0m checking the PV [38;5;243m01/17/23 18:29:04.102[0m
[1mSTEP:[0m setting up the pod [38;5;243m01/17/23 18:29:04.102[0m
I0117 18:29:04.102538   43786 resource_setup.go:62] adding PV (pvc-aa8b1e04-c805-49e9-9244-69ad325fdb0c) to pod ()
[1mSTEP:[0m deploying the pod [38;5;243m01/17/23 18:29:04.102[0m
[1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/17/23 18:29:04.178[0m
Jan 17 18:29:04.179: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-p62gh" in namespace "azuredisk-8071" to be "Succeeded or Failed"
Jan 17 18:29:04.238: INFO: Pod "azuredisk-volume-tester-p62gh": Phase="Pending", Reason="", readiness=false. Elapsed: 58.888574ms
Jan 17 18:29:06.299: INFO: Pod "azuredisk-volume-tester-p62gh": Phase="Pending", Reason="", readiness=false. Elapsed: 2.120156823s
Jan 17 18:29:08.302: INFO: Pod "azuredisk-volume-tester-p62gh": Phase="Pending", Reason="", readiness=false. Elapsed: 4.122898421s
Jan 17 18:29:10.299: INFO: Pod "azuredisk-volume-tester-p62gh": Phase="Pending", Reason="", readiness=false. Elapsed: 6.120496355s
Jan 17 18:29:12.300: INFO: Pod "azuredisk-volume-tester-p62gh": Phase="Pending", Reason="", readiness=false. Elapsed: 8.121124536s
Jan 17 18:29:14.298: INFO: Pod "azuredisk-volume-tester-p62gh": Phase="Pending", Reason="", readiness=false. Elapsed: 10.1194179s
... skipping 29 lines ...
Jan 17 18:30:14.300: INFO: Pod "azuredisk-volume-tester-p62gh": Phase="Running", Reason="", readiness=false. Elapsed: 1m10.121759552s
Jan 17 18:30:16.299: INFO: Pod "azuredisk-volume-tester-p62gh": Phase="Running", Reason="", readiness=false. Elapsed: 1m12.120182383s
Jan 17 18:30:18.301: INFO: Pod "azuredisk-volume-tester-p62gh": Phase="Running", Reason="", readiness=false. Elapsed: 1m14.121851267s
Jan 17 18:30:20.299: INFO: Pod "azuredisk-volume-tester-p62gh": Phase="Running", Reason="", readiness=false. Elapsed: 1m16.120601935s
Jan 17 18:30:22.299: INFO: Pod "azuredisk-volume-tester-p62gh": Phase="Succeeded", Reason="", readiness=false. Elapsed: 1m18.120043129s
[1mSTEP:[0m Saw pod success [38;5;243m01/17/23 18:30:22.299[0m
Jan 17 18:30:22.300: INFO: Pod "azuredisk-volume-tester-p62gh" satisfied condition "Succeeded or Failed"
Jan 17 18:30:22.301: INFO: deleting Pod "azuredisk-8071"/"azuredisk-volume-tester-p62gh"
Jan 17 18:30:22.386: INFO: Pod azuredisk-volume-tester-p62gh has the following logs: hello world
hello world
hello world

[1mSTEP:[0m Deleting pod azuredisk-volume-tester-p62gh in namespace azuredisk-8071 [38;5;243m01/17/23 18:30:22.387[0m
... skipping 96 lines ...
    [1mSTEP:[0m checking the PVC [38;5;243m01/17/23 18:29:03.986[0m
    [1mSTEP:[0m validating provisioned PV [38;5;243m01/17/23 18:29:04.044[0m
    [1mSTEP:[0m checking the PV [38;5;243m01/17/23 18:29:04.102[0m
    [1mSTEP:[0m setting up the pod [38;5;243m01/17/23 18:29:04.102[0m
    I0117 18:29:04.102538   43786 resource_setup.go:62] adding PV (pvc-aa8b1e04-c805-49e9-9244-69ad325fdb0c) to pod ()
    [1mSTEP:[0m deploying the pod [38;5;243m01/17/23 18:29:04.102[0m
    [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/17/23 18:29:04.178[0m
    Jan 17 18:29:04.179: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-p62gh" in namespace "azuredisk-8071" to be "Succeeded or Failed"
    Jan 17 18:29:04.238: INFO: Pod "azuredisk-volume-tester-p62gh": Phase="Pending", Reason="", readiness=false. Elapsed: 58.888574ms
    Jan 17 18:29:06.299: INFO: Pod "azuredisk-volume-tester-p62gh": Phase="Pending", Reason="", readiness=false. Elapsed: 2.120156823s
    Jan 17 18:29:08.302: INFO: Pod "azuredisk-volume-tester-p62gh": Phase="Pending", Reason="", readiness=false. Elapsed: 4.122898421s
    Jan 17 18:29:10.299: INFO: Pod "azuredisk-volume-tester-p62gh": Phase="Pending", Reason="", readiness=false. Elapsed: 6.120496355s
    Jan 17 18:29:12.300: INFO: Pod "azuredisk-volume-tester-p62gh": Phase="Pending", Reason="", readiness=false. Elapsed: 8.121124536s
    Jan 17 18:29:14.298: INFO: Pod "azuredisk-volume-tester-p62gh": Phase="Pending", Reason="", readiness=false. Elapsed: 10.1194179s
... skipping 29 lines ...
    Jan 17 18:30:14.300: INFO: Pod "azuredisk-volume-tester-p62gh": Phase="Running", Reason="", readiness=false. Elapsed: 1m10.121759552s
    Jan 17 18:30:16.299: INFO: Pod "azuredisk-volume-tester-p62gh": Phase="Running", Reason="", readiness=false. Elapsed: 1m12.120182383s
    Jan 17 18:30:18.301: INFO: Pod "azuredisk-volume-tester-p62gh": Phase="Running", Reason="", readiness=false. Elapsed: 1m14.121851267s
    Jan 17 18:30:20.299: INFO: Pod "azuredisk-volume-tester-p62gh": Phase="Running", Reason="", readiness=false. Elapsed: 1m16.120601935s
    Jan 17 18:30:22.299: INFO: Pod "azuredisk-volume-tester-p62gh": Phase="Succeeded", Reason="", readiness=false. Elapsed: 1m18.120043129s
    [1mSTEP:[0m Saw pod success [38;5;243m01/17/23 18:30:22.299[0m
    Jan 17 18:30:22.300: INFO: Pod "azuredisk-volume-tester-p62gh" satisfied condition "Succeeded or Failed"
    Jan 17 18:30:22.301: INFO: deleting Pod "azuredisk-8071"/"azuredisk-volume-tester-p62gh"
    Jan 17 18:30:22.386: INFO: Pod azuredisk-volume-tester-p62gh has the following logs: hello world
    hello world
    hello world

    [1mSTEP:[0m Deleting pod azuredisk-volume-tester-p62gh in namespace azuredisk-8071 [38;5;243m01/17/23 18:30:22.387[0m
... skipping 101 lines ...
Jan 17 18:31:58.984: INFO: PersistentVolumeClaim pvc-w9tmr found but phase is Pending instead of Bound.
Jan 17 18:32:01.042: INFO: PersistentVolumeClaim pvc-w9tmr found and phase=Bound (4.177031362s)
[1mSTEP:[0m checking the PVC [38;5;243m01/17/23 18:32:01.042[0m
[1mSTEP:[0m validating provisioned PV [38;5;243m01/17/23 18:32:01.1[0m
[1mSTEP:[0m checking the PV [38;5;243m01/17/23 18:32:01.159[0m
[1mSTEP:[0m deploying the pod [38;5;243m01/17/23 18:32:01.161[0m
[1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/17/23 18:32:01.229[0m
Jan 17 18:32:01.230: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-r468r" in namespace "azuredisk-5936" to be "Succeeded or Failed"
Jan 17 18:32:01.287: INFO: Pod "azuredisk-volume-tester-r468r": Phase="Pending", Reason="", readiness=false. Elapsed: 57.041248ms
Jan 17 18:32:03.347: INFO: Pod "azuredisk-volume-tester-r468r": Phase="Pending", Reason="", readiness=false. Elapsed: 2.117285345s
Jan 17 18:32:05.346: INFO: Pod "azuredisk-volume-tester-r468r": Phase="Pending", Reason="", readiness=false. Elapsed: 4.116231364s
Jan 17 18:32:07.345: INFO: Pod "azuredisk-volume-tester-r468r": Phase="Pending", Reason="", readiness=false. Elapsed: 6.114946519s
Jan 17 18:32:09.347: INFO: Pod "azuredisk-volume-tester-r468r": Phase="Pending", Reason="", readiness=false. Elapsed: 8.116822482s
Jan 17 18:32:11.349: INFO: Pod "azuredisk-volume-tester-r468r": Phase="Pending", Reason="", readiness=false. Elapsed: 10.118972346s
... skipping 22 lines ...
Jan 17 18:32:57.348: INFO: Pod "azuredisk-volume-tester-r468r": Phase="Running", Reason="", readiness=true. Elapsed: 56.118106464s
Jan 17 18:32:59.352: INFO: Pod "azuredisk-volume-tester-r468r": Phase="Running", Reason="", readiness=true. Elapsed: 58.122687543s
Jan 17 18:33:01.348: INFO: Pod "azuredisk-volume-tester-r468r": Phase="Running", Reason="", readiness=false. Elapsed: 1m0.118661599s
Jan 17 18:33:03.348: INFO: Pod "azuredisk-volume-tester-r468r": Phase="Running", Reason="", readiness=false. Elapsed: 1m2.117830011s
Jan 17 18:33:05.348: INFO: Pod "azuredisk-volume-tester-r468r": Phase="Succeeded", Reason="", readiness=false. Elapsed: 1m4.118732711s
[1mSTEP:[0m Saw pod success [38;5;243m01/17/23 18:33:05.349[0m
Jan 17 18:33:05.349: INFO: Pod "azuredisk-volume-tester-r468r" satisfied condition "Succeeded or Failed"
Jan 17 18:33:05.407: INFO: deleting Pod "azuredisk-5936"/"azuredisk-volume-tester-r468r"
Jan 17 18:33:05.491: INFO: Pod azuredisk-volume-tester-r468r has the following logs: 
[1mSTEP:[0m Deleting pod azuredisk-volume-tester-r468r in namespace azuredisk-5936 [38;5;243m01/17/23 18:33:05.491[0m
[1mSTEP:[0m waiting for disk to detach from node [38;5;243m01/17/23 18:33:05.571[0m
[1mSTEP:[0m Checking Prow test resource group [38;5;243m01/17/23 18:33:33.686[0m
[1mSTEP:[0m Prow test resource group: capz-wto256 [38;5;243m01/17/23 18:33:33.695[0m
... skipping 16 lines ...
[1mSTEP:[0m checking the PVC [38;5;243m01/17/23 18:33:54.447[0m
[1mSTEP:[0m validating provisioned PV [38;5;243m01/17/23 18:33:54.504[0m
[1mSTEP:[0m checking the PV [38;5;243m01/17/23 18:33:54.561[0m
[1mSTEP:[0m setting up the pod [38;5;243m01/17/23 18:33:54.562[0m
I0117 18:33:54.563093   43786 resource_setup.go:62] adding PV (pvc-feaf0839-af46-49b1-a7b5-5cd6e4df7d3f) to pod ()
[1mSTEP:[0m deploying a pod with a volume restored from the snapshot [38;5;243m01/17/23 18:33:54.563[0m
[1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/17/23 18:33:54.626[0m
Jan 17 18:33:54.626: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-pcrgg" in namespace "azuredisk-5936" to be "Succeeded or Failed"
Jan 17 18:33:54.683: INFO: Pod "azuredisk-volume-tester-pcrgg": Phase="Pending", Reason="", readiness=false. Elapsed: 56.617585ms
Jan 17 18:33:56.741: INFO: Pod "azuredisk-volume-tester-pcrgg": Phase="Pending", Reason="", readiness=false. Elapsed: 2.11512589s
Jan 17 18:33:58.741: INFO: Pod "azuredisk-volume-tester-pcrgg": Phase="Pending", Reason="", readiness=false. Elapsed: 4.114512266s
Jan 17 18:34:00.741: INFO: Pod "azuredisk-volume-tester-pcrgg": Phase="Pending", Reason="", readiness=false. Elapsed: 6.114317425s
Jan 17 18:34:02.741: INFO: Pod "azuredisk-volume-tester-pcrgg": Phase="Pending", Reason="", readiness=false. Elapsed: 8.114376934s
Jan 17 18:34:04.742: INFO: Pod "azuredisk-volume-tester-pcrgg": Phase="Pending", Reason="", readiness=false. Elapsed: 10.115816754s
... skipping 14 lines ...
Jan 17 18:34:34.743: INFO: Pod "azuredisk-volume-tester-pcrgg": Phase="Running", Reason="", readiness=true. Elapsed: 40.116386387s
Jan 17 18:34:36.744: INFO: Pod "azuredisk-volume-tester-pcrgg": Phase="Running", Reason="", readiness=false. Elapsed: 42.11804017s
Jan 17 18:34:38.745: INFO: Pod "azuredisk-volume-tester-pcrgg": Phase="Running", Reason="", readiness=false. Elapsed: 44.118334218s
Jan 17 18:34:40.743: INFO: Pod "azuredisk-volume-tester-pcrgg": Phase="Running", Reason="", readiness=false. Elapsed: 46.117047167s
Jan 17 18:34:42.744: INFO: Pod "azuredisk-volume-tester-pcrgg": Phase="Succeeded", Reason="", readiness=false. Elapsed: 48.11733046s
[1mSTEP:[0m Saw pod success [38;5;243m01/17/23 18:34:42.744[0m
Jan 17 18:34:42.744: INFO: Pod "azuredisk-volume-tester-pcrgg" satisfied condition "Succeeded or Failed"
Jan 17 18:34:42.744: INFO: deleting Pod "azuredisk-5936"/"azuredisk-volume-tester-pcrgg"
Jan 17 18:34:42.827: INFO: Pod azuredisk-volume-tester-pcrgg has the following logs: hello world

[1mSTEP:[0m Deleting pod azuredisk-volume-tester-pcrgg in namespace azuredisk-5936 [38;5;243m01/17/23 18:34:42.828[0m
Jan 17 18:34:42.903: INFO: deleting PVC "azuredisk-5936"/"pvc-dllmb"
Jan 17 18:34:42.903: INFO: Deleting PersistentVolumeClaim "pvc-dllmb"
... skipping 12 lines ...
Jan 17 18:35:23.567: INFO: Claim "pvc-dllmb" in namespace "azuredisk-5936" doesn't exist in the system
Jan 17 18:35:23.568: INFO: deleting StorageClass azuredisk-5936-disk.csi.azure.com-dynamic-sc-68sjm
[1mSTEP:[0m deleting a VolumeSnapshot volume-snapshot-c4q7n [38;5;243m01/17/23 18:35:23.63[0m
Jan 17 18:35:23.702: INFO: skip deleting VolumeSnapshotClass azuredisk-5936-disk.csi.azure.com-dynamic-sc-9xb8h
Jan 17 18:35:23.702: INFO: Deleting resource group azuredisk-csi-driver-test-72b29980-9695-11ed-b9f0-5a32c2634674
Jan 17 18:36:25.897: INFO: deleting Pod "azuredisk-5936"/"azuredisk-volume-tester-r468r"
Jan 17 18:36:25.955: INFO: Error getting logs for pod azuredisk-volume-tester-r468r: the server could not find the requested resource (get pods azuredisk-volume-tester-r468r)
[1mSTEP:[0m Deleting pod azuredisk-volume-tester-r468r in namespace azuredisk-5936 [38;5;243m01/17/23 18:36:25.955[0m
Jan 17 18:36:26.016: INFO: deleting PVC "azuredisk-5936"/"pvc-w9tmr"
Jan 17 18:36:26.016: INFO: Deleting PersistentVolumeClaim "pvc-w9tmr"
[1mSTEP:[0m waiting for claim's PV "pvc-52ec6b4d-35d2-4c79-bf70-b4241935f95e" to be deleted [38;5;243m01/17/23 18:36:26.079[0m
Jan 17 18:36:26.079: INFO: Waiting up to 10m0s for PersistentVolume pvc-52ec6b4d-35d2-4c79-bf70-b4241935f95e to get deleted
Jan 17 18:36:26.135: INFO: PersistentVolume pvc-52ec6b4d-35d2-4c79-bf70-b4241935f95e found and phase=Released (55.773556ms)
... skipping 29 lines ...
    Jan 17 18:31:58.984: INFO: PersistentVolumeClaim pvc-w9tmr found but phase is Pending instead of Bound.
    Jan 17 18:32:01.042: INFO: PersistentVolumeClaim pvc-w9tmr found and phase=Bound (4.177031362s)
    [1mSTEP:[0m checking the PVC [38;5;243m01/17/23 18:32:01.042[0m
    [1mSTEP:[0m validating provisioned PV [38;5;243m01/17/23 18:32:01.1[0m
    [1mSTEP:[0m checking the PV [38;5;243m01/17/23 18:32:01.159[0m
    [1mSTEP:[0m deploying the pod [38;5;243m01/17/23 18:32:01.161[0m
    [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/17/23 18:32:01.229[0m
    Jan 17 18:32:01.230: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-r468r" in namespace "azuredisk-5936" to be "Succeeded or Failed"
    Jan 17 18:32:01.287: INFO: Pod "azuredisk-volume-tester-r468r": Phase="Pending", Reason="", readiness=false. Elapsed: 57.041248ms
    Jan 17 18:32:03.347: INFO: Pod "azuredisk-volume-tester-r468r": Phase="Pending", Reason="", readiness=false. Elapsed: 2.117285345s
    Jan 17 18:32:05.346: INFO: Pod "azuredisk-volume-tester-r468r": Phase="Pending", Reason="", readiness=false. Elapsed: 4.116231364s
    Jan 17 18:32:07.345: INFO: Pod "azuredisk-volume-tester-r468r": Phase="Pending", Reason="", readiness=false. Elapsed: 6.114946519s
    Jan 17 18:32:09.347: INFO: Pod "azuredisk-volume-tester-r468r": Phase="Pending", Reason="", readiness=false. Elapsed: 8.116822482s
    Jan 17 18:32:11.349: INFO: Pod "azuredisk-volume-tester-r468r": Phase="Pending", Reason="", readiness=false. Elapsed: 10.118972346s
... skipping 22 lines ...
    Jan 17 18:32:57.348: INFO: Pod "azuredisk-volume-tester-r468r": Phase="Running", Reason="", readiness=true. Elapsed: 56.118106464s
    Jan 17 18:32:59.352: INFO: Pod "azuredisk-volume-tester-r468r": Phase="Running", Reason="", readiness=true. Elapsed: 58.122687543s
    Jan 17 18:33:01.348: INFO: Pod "azuredisk-volume-tester-r468r": Phase="Running", Reason="", readiness=false. Elapsed: 1m0.118661599s
    Jan 17 18:33:03.348: INFO: Pod "azuredisk-volume-tester-r468r": Phase="Running", Reason="", readiness=false. Elapsed: 1m2.117830011s
    Jan 17 18:33:05.348: INFO: Pod "azuredisk-volume-tester-r468r": Phase="Succeeded", Reason="", readiness=false. Elapsed: 1m4.118732711s
    [1mSTEP:[0m Saw pod success [38;5;243m01/17/23 18:33:05.349[0m
    Jan 17 18:33:05.349: INFO: Pod "azuredisk-volume-tester-r468r" satisfied condition "Succeeded or Failed"
    Jan 17 18:33:05.407: INFO: deleting Pod "azuredisk-5936"/"azuredisk-volume-tester-r468r"
    Jan 17 18:33:05.491: INFO: Pod azuredisk-volume-tester-r468r has the following logs: 
    [1mSTEP:[0m Deleting pod azuredisk-volume-tester-r468r in namespace azuredisk-5936 [38;5;243m01/17/23 18:33:05.491[0m
    [1mSTEP:[0m waiting for disk to detach from node [38;5;243m01/17/23 18:33:05.571[0m
    [1mSTEP:[0m Checking Prow test resource group [38;5;243m01/17/23 18:33:33.686[0m
    [1mSTEP:[0m Prow test resource group: capz-wto256 [38;5;243m01/17/23 18:33:33.695[0m
... skipping 16 lines ...
    [1mSTEP:[0m checking the PVC [38;5;243m01/17/23 18:33:54.447[0m
    [1mSTEP:[0m validating provisioned PV [38;5;243m01/17/23 18:33:54.504[0m
    [1mSTEP:[0m checking the PV [38;5;243m01/17/23 18:33:54.561[0m
    [1mSTEP:[0m setting up the pod [38;5;243m01/17/23 18:33:54.562[0m
    I0117 18:33:54.563093   43786 resource_setup.go:62] adding PV (pvc-feaf0839-af46-49b1-a7b5-5cd6e4df7d3f) to pod ()
    [1mSTEP:[0m deploying a pod with a volume restored from the snapshot [38;5;243m01/17/23 18:33:54.563[0m
    [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/17/23 18:33:54.626[0m
    Jan 17 18:33:54.626: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-pcrgg" in namespace "azuredisk-5936" to be "Succeeded or Failed"
    Jan 17 18:33:54.683: INFO: Pod "azuredisk-volume-tester-pcrgg": Phase="Pending", Reason="", readiness=false. Elapsed: 56.617585ms
    Jan 17 18:33:56.741: INFO: Pod "azuredisk-volume-tester-pcrgg": Phase="Pending", Reason="", readiness=false. Elapsed: 2.11512589s
    Jan 17 18:33:58.741: INFO: Pod "azuredisk-volume-tester-pcrgg": Phase="Pending", Reason="", readiness=false. Elapsed: 4.114512266s
    Jan 17 18:34:00.741: INFO: Pod "azuredisk-volume-tester-pcrgg": Phase="Pending", Reason="", readiness=false. Elapsed: 6.114317425s
    Jan 17 18:34:02.741: INFO: Pod "azuredisk-volume-tester-pcrgg": Phase="Pending", Reason="", readiness=false. Elapsed: 8.114376934s
    Jan 17 18:34:04.742: INFO: Pod "azuredisk-volume-tester-pcrgg": Phase="Pending", Reason="", readiness=false. Elapsed: 10.115816754s
... skipping 14 lines ...
    Jan 17 18:34:34.743: INFO: Pod "azuredisk-volume-tester-pcrgg": Phase="Running", Reason="", readiness=true. Elapsed: 40.116386387s
    Jan 17 18:34:36.744: INFO: Pod "azuredisk-volume-tester-pcrgg": Phase="Running", Reason="", readiness=false. Elapsed: 42.11804017s
    Jan 17 18:34:38.745: INFO: Pod "azuredisk-volume-tester-pcrgg": Phase="Running", Reason="", readiness=false. Elapsed: 44.118334218s
    Jan 17 18:34:40.743: INFO: Pod "azuredisk-volume-tester-pcrgg": Phase="Running", Reason="", readiness=false. Elapsed: 46.117047167s
    Jan 17 18:34:42.744: INFO: Pod "azuredisk-volume-tester-pcrgg": Phase="Succeeded", Reason="", readiness=false. Elapsed: 48.11733046s
    [1mSTEP:[0m Saw pod success [38;5;243m01/17/23 18:34:42.744[0m
    Jan 17 18:34:42.744: INFO: Pod "azuredisk-volume-tester-pcrgg" satisfied condition "Succeeded or Failed"
    Jan 17 18:34:42.744: INFO: deleting Pod "azuredisk-5936"/"azuredisk-volume-tester-pcrgg"
    Jan 17 18:34:42.827: INFO: Pod azuredisk-volume-tester-pcrgg has the following logs: hello world

    [1mSTEP:[0m Deleting pod azuredisk-volume-tester-pcrgg in namespace azuredisk-5936 [38;5;243m01/17/23 18:34:42.828[0m
    Jan 17 18:34:42.903: INFO: deleting PVC "azuredisk-5936"/"pvc-dllmb"
    Jan 17 18:34:42.903: INFO: Deleting PersistentVolumeClaim "pvc-dllmb"
... skipping 12 lines ...
    Jan 17 18:35:23.567: INFO: Claim "pvc-dllmb" in namespace "azuredisk-5936" doesn't exist in the system
    Jan 17 18:35:23.568: INFO: deleting StorageClass azuredisk-5936-disk.csi.azure.com-dynamic-sc-68sjm
    [1mSTEP:[0m deleting a VolumeSnapshot volume-snapshot-c4q7n [38;5;243m01/17/23 18:35:23.63[0m
    Jan 17 18:35:23.702: INFO: skip deleting VolumeSnapshotClass azuredisk-5936-disk.csi.azure.com-dynamic-sc-9xb8h
    Jan 17 18:35:23.702: INFO: Deleting resource group azuredisk-csi-driver-test-72b29980-9695-11ed-b9f0-5a32c2634674
    Jan 17 18:36:25.897: INFO: deleting Pod "azuredisk-5936"/"azuredisk-volume-tester-r468r"
    Jan 17 18:36:25.955: INFO: Error getting logs for pod azuredisk-volume-tester-r468r: the server could not find the requested resource (get pods azuredisk-volume-tester-r468r)
    [1mSTEP:[0m Deleting pod azuredisk-volume-tester-r468r in namespace azuredisk-5936 [38;5;243m01/17/23 18:36:25.955[0m
    Jan 17 18:36:26.016: INFO: deleting PVC "azuredisk-5936"/"pvc-w9tmr"
    Jan 17 18:36:26.016: INFO: Deleting PersistentVolumeClaim "pvc-w9tmr"
    [1mSTEP:[0m waiting for claim's PV "pvc-52ec6b4d-35d2-4c79-bf70-b4241935f95e" to be deleted [38;5;243m01/17/23 18:36:26.079[0m
    Jan 17 18:36:26.079: INFO: Waiting up to 10m0s for PersistentVolume pvc-52ec6b4d-35d2-4c79-bf70-b4241935f95e to get deleted
    Jan 17 18:36:26.135: INFO: PersistentVolume pvc-52ec6b4d-35d2-4c79-bf70-b4241935f95e found and phase=Released (55.773556ms)
... skipping 28 lines ...
Jan 17 18:36:39.582: INFO: PersistentVolumeClaim pvc-dpzgl found but phase is Pending instead of Bound.
Jan 17 18:36:41.639: INFO: PersistentVolumeClaim pvc-dpzgl found and phase=Bound (4.170310125s)
[1mSTEP:[0m checking the PVC [38;5;243m01/17/23 18:36:41.639[0m
[1mSTEP:[0m validating provisioned PV [38;5;243m01/17/23 18:36:41.696[0m
[1mSTEP:[0m checking the PV [38;5;243m01/17/23 18:36:41.753[0m
[1mSTEP:[0m deploying the pod [38;5;243m01/17/23 18:36:41.753[0m
[1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/17/23 18:36:41.815[0m
Jan 17 18:36:41.815: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-j9zlf" in namespace "azuredisk-1120" to be "Succeeded or Failed"
Jan 17 18:36:41.879: INFO: Pod "azuredisk-volume-tester-j9zlf": Phase="Pending", Reason="", readiness=false. Elapsed: 63.488344ms
Jan 17 18:36:43.940: INFO: Pod "azuredisk-volume-tester-j9zlf": Phase="Pending", Reason="", readiness=false. Elapsed: 2.124335003s
Jan 17 18:36:45.942: INFO: Pod "azuredisk-volume-tester-j9zlf": Phase="Pending", Reason="", readiness=false. Elapsed: 4.126701615s
Jan 17 18:36:47.939: INFO: Pod "azuredisk-volume-tester-j9zlf": Phase="Pending", Reason="", readiness=false. Elapsed: 6.124112708s
Jan 17 18:36:49.938: INFO: Pod "azuredisk-volume-tester-j9zlf": Phase="Pending", Reason="", readiness=false. Elapsed: 8.122608312s
Jan 17 18:36:51.937: INFO: Pod "azuredisk-volume-tester-j9zlf": Phase="Pending", Reason="", readiness=false. Elapsed: 10.122095325s
... skipping 22 lines ...
Jan 17 18:37:37.941: INFO: Pod "azuredisk-volume-tester-j9zlf": Phase="Running", Reason="", readiness=true. Elapsed: 56.12613592s
Jan 17 18:37:39.940: INFO: Pod "azuredisk-volume-tester-j9zlf": Phase="Running", Reason="", readiness=false. Elapsed: 58.125014293s
Jan 17 18:37:41.941: INFO: Pod "azuredisk-volume-tester-j9zlf": Phase="Running", Reason="", readiness=false. Elapsed: 1m0.125962158s
Jan 17 18:37:43.941: INFO: Pod "azuredisk-volume-tester-j9zlf": Phase="Running", Reason="", readiness=false. Elapsed: 1m2.125904773s
Jan 17 18:37:45.944: INFO: Pod "azuredisk-volume-tester-j9zlf": Phase="Succeeded", Reason="", readiness=false. Elapsed: 1m4.128377963s
[1mSTEP:[0m Saw pod success [38;5;243m01/17/23 18:37:45.944[0m
Jan 17 18:37:45.944: INFO: Pod "azuredisk-volume-tester-j9zlf" satisfied condition "Succeeded or Failed"
Jan 17 18:37:46.002: INFO: deleting Pod "azuredisk-1120"/"azuredisk-volume-tester-j9zlf"
Jan 17 18:37:46.078: INFO: Pod azuredisk-volume-tester-j9zlf has the following logs: 
[1mSTEP:[0m Deleting pod azuredisk-volume-tester-j9zlf in namespace azuredisk-1120 [38;5;243m01/17/23 18:37:46.079[0m
[1mSTEP:[0m waiting for disk to detach from node [38;5;243m01/17/23 18:37:46.16[0m
[1mSTEP:[0m Checking Prow test resource group [38;5;243m01/17/23 18:38:16.275[0m
[1mSTEP:[0m Prow test resource group: capz-wto256 [38;5;243m01/17/23 18:38:16.275[0m
... skipping 42 lines ...
[1mSTEP:[0m validating provisioned PV [38;5;243m01/17/23 18:39:17.091[0m
[1mSTEP:[0m checking the PV [38;5;243m01/17/23 18:39:17.148[0m
[1mSTEP:[0m setting up the pod [38;5;243m01/17/23 18:39:17.148[0m
I0117 18:39:17.149067   43786 resource_setup.go:62] adding PV (pvc-a6ced79f-41ba-4074-bd5d-b9c75f2d7a3c) to pod ()
[1mSTEP:[0m Set pod anti-affinity to make sure two pods are scheduled on different nodes [38;5;243m01/17/23 18:39:17.149[0m
[1mSTEP:[0m deploying a pod with a volume restored from the snapshot [38;5;243m01/17/23 18:39:17.149[0m
[1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/17/23 18:39:17.21[0m
Jan 17 18:39:17.211: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-642xt" in namespace "azuredisk-1120" to be "Succeeded or Failed"
Jan 17 18:39:17.276: INFO: Pod "azuredisk-volume-tester-642xt": Phase="Pending", Reason="", readiness=false. Elapsed: 64.972215ms
Jan 17 18:39:19.339: INFO: Pod "azuredisk-volume-tester-642xt": Phase="Pending", Reason="", readiness=false. Elapsed: 2.12755347s
Jan 17 18:39:21.339: INFO: Pod "azuredisk-volume-tester-642xt": Phase="Pending", Reason="", readiness=false. Elapsed: 4.128292981s
Jan 17 18:39:23.340: INFO: Pod "azuredisk-volume-tester-642xt": Phase="Pending", Reason="", readiness=false. Elapsed: 6.128634572s
Jan 17 18:39:25.341: INFO: Pod "azuredisk-volume-tester-642xt": Phase="Pending", Reason="", readiness=false. Elapsed: 8.129871965s
Jan 17 18:39:27.338: INFO: Pod "azuredisk-volume-tester-642xt": Phase="Pending", Reason="", readiness=false. Elapsed: 10.127401412s
... skipping 96 lines ...
Jan 17 18:42:41.339: INFO: Pod "azuredisk-volume-tester-642xt": Phase="Running", Reason="", readiness=true. Elapsed: 3m24.127630997s
Jan 17 18:42:43.339: INFO: Pod "azuredisk-volume-tester-642xt": Phase="Running", Reason="", readiness=true. Elapsed: 3m26.1278925s
Jan 17 18:42:45.338: INFO: Pod "azuredisk-volume-tester-642xt": Phase="Running", Reason="", readiness=false. Elapsed: 3m28.127331707s
Jan 17 18:42:47.339: INFO: Pod "azuredisk-volume-tester-642xt": Phase="Running", Reason="", readiness=false. Elapsed: 3m30.127718962s
Jan 17 18:42:49.339: INFO: Pod "azuredisk-volume-tester-642xt": Phase="Succeeded", Reason="", readiness=false. Elapsed: 3m32.128378357s
[1mSTEP:[0m Saw pod success [38;5;243m01/17/23 18:42:49.34[0m
Jan 17 18:42:49.340: INFO: Pod "azuredisk-volume-tester-642xt" satisfied condition "Succeeded or Failed"
Jan 17 18:42:49.340: INFO: deleting Pod "azuredisk-1120"/"azuredisk-volume-tester-642xt"
Jan 17 18:42:49.425: INFO: Pod azuredisk-volume-tester-642xt has the following logs: hello world

[1mSTEP:[0m Deleting pod azuredisk-volume-tester-642xt in namespace azuredisk-1120 [38;5;243m01/17/23 18:42:49.425[0m
Jan 17 18:42:49.503: INFO: deleting PVC "azuredisk-1120"/"pvc-j5h6n"
Jan 17 18:42:49.503: INFO: Deleting PersistentVolumeClaim "pvc-j5h6n"
... skipping 15 lines ...
Jan 17 18:43:30.337: INFO: Pod azuredisk-volume-tester-b5dpb has the following logs: 
[1mSTEP:[0m Deleting pod azuredisk-volume-tester-b5dpb in namespace azuredisk-1120 [38;5;243m01/17/23 18:43:30.337[0m
[1mSTEP:[0m deleting a VolumeSnapshot volume-snapshot-kczgd [38;5;243m01/17/23 18:43:30.405[0m
Jan 17 18:43:30.469: INFO: skip deleting VolumeSnapshotClass azuredisk-1120-disk.csi.azure.com-dynamic-sc-lv4vd
Jan 17 18:43:30.469: INFO: Deleting resource group azuredisk-csi-driver-test-1b20f3ef-9696-11ed-b9f0-5a32c2634674
Jan 17 18:44:47.663: INFO: deleting Pod "azuredisk-1120"/"azuredisk-volume-tester-j9zlf"
Jan 17 18:44:47.725: INFO: Error getting logs for pod azuredisk-volume-tester-j9zlf: the server could not find the requested resource (get pods azuredisk-volume-tester-j9zlf)
[1mSTEP:[0m Deleting pod azuredisk-volume-tester-j9zlf in namespace azuredisk-1120 [38;5;243m01/17/23 18:44:47.725[0m
Jan 17 18:44:47.791: INFO: deleting PVC "azuredisk-1120"/"pvc-dpzgl"
Jan 17 18:44:47.791: INFO: Deleting PersistentVolumeClaim "pvc-dpzgl"
[1mSTEP:[0m waiting for claim's PV "pvc-fd92305e-1233-4e13-9901-1b8cfe378f45" to be deleted [38;5;243m01/17/23 18:44:47.86[0m
Jan 17 18:44:47.861: INFO: Waiting up to 10m0s for PersistentVolume pvc-fd92305e-1233-4e13-9901-1b8cfe378f45 to get deleted
Jan 17 18:44:47.922: INFO: PersistentVolume pvc-fd92305e-1233-4e13-9901-1b8cfe378f45 found and phase=Released (61.712271ms)
... skipping 29 lines ...
    Jan 17 18:36:39.582: INFO: PersistentVolumeClaim pvc-dpzgl found but phase is Pending instead of Bound.
    Jan 17 18:36:41.639: INFO: PersistentVolumeClaim pvc-dpzgl found and phase=Bound (4.170310125s)
    [1mSTEP:[0m checking the PVC [38;5;243m01/17/23 18:36:41.639[0m
    [1mSTEP:[0m validating provisioned PV [38;5;243m01/17/23 18:36:41.696[0m
    [1mSTEP:[0m checking the PV [38;5;243m01/17/23 18:36:41.753[0m
    [1mSTEP:[0m deploying the pod [38;5;243m01/17/23 18:36:41.753[0m
    [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/17/23 18:36:41.815[0m
    Jan 17 18:36:41.815: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-j9zlf" in namespace "azuredisk-1120" to be "Succeeded or Failed"
    Jan 17 18:36:41.879: INFO: Pod "azuredisk-volume-tester-j9zlf": Phase="Pending", Reason="", readiness=false. Elapsed: 63.488344ms
    Jan 17 18:36:43.940: INFO: Pod "azuredisk-volume-tester-j9zlf": Phase="Pending", Reason="", readiness=false. Elapsed: 2.124335003s
    Jan 17 18:36:45.942: INFO: Pod "azuredisk-volume-tester-j9zlf": Phase="Pending", Reason="", readiness=false. Elapsed: 4.126701615s
    Jan 17 18:36:47.939: INFO: Pod "azuredisk-volume-tester-j9zlf": Phase="Pending", Reason="", readiness=false. Elapsed: 6.124112708s
    Jan 17 18:36:49.938: INFO: Pod "azuredisk-volume-tester-j9zlf": Phase="Pending", Reason="", readiness=false. Elapsed: 8.122608312s
    Jan 17 18:36:51.937: INFO: Pod "azuredisk-volume-tester-j9zlf": Phase="Pending", Reason="", readiness=false. Elapsed: 10.122095325s
... skipping 22 lines ...
    Jan 17 18:37:37.941: INFO: Pod "azuredisk-volume-tester-j9zlf": Phase="Running", Reason="", readiness=true. Elapsed: 56.12613592s
    Jan 17 18:37:39.940: INFO: Pod "azuredisk-volume-tester-j9zlf": Phase="Running", Reason="", readiness=false. Elapsed: 58.125014293s
    Jan 17 18:37:41.941: INFO: Pod "azuredisk-volume-tester-j9zlf": Phase="Running", Reason="", readiness=false. Elapsed: 1m0.125962158s
    Jan 17 18:37:43.941: INFO: Pod "azuredisk-volume-tester-j9zlf": Phase="Running", Reason="", readiness=false. Elapsed: 1m2.125904773s
    Jan 17 18:37:45.944: INFO: Pod "azuredisk-volume-tester-j9zlf": Phase="Succeeded", Reason="", readiness=false. Elapsed: 1m4.128377963s
    [1mSTEP:[0m Saw pod success [38;5;243m01/17/23 18:37:45.944[0m
    Jan 17 18:37:45.944: INFO: Pod "azuredisk-volume-tester-j9zlf" satisfied condition "Succeeded or Failed"
    Jan 17 18:37:46.002: INFO: deleting Pod "azuredisk-1120"/"azuredisk-volume-tester-j9zlf"
    Jan 17 18:37:46.078: INFO: Pod azuredisk-volume-tester-j9zlf has the following logs: 
    [1mSTEP:[0m Deleting pod azuredisk-volume-tester-j9zlf in namespace azuredisk-1120 [38;5;243m01/17/23 18:37:46.079[0m
    [1mSTEP:[0m waiting for disk to detach from node [38;5;243m01/17/23 18:37:46.16[0m
    [1mSTEP:[0m Checking Prow test resource group [38;5;243m01/17/23 18:38:16.275[0m
    [1mSTEP:[0m Prow test resource group: capz-wto256 [38;5;243m01/17/23 18:38:16.275[0m
... skipping 42 lines ...
    [1mSTEP:[0m validating provisioned PV [38;5;243m01/17/23 18:39:17.091[0m
    [1mSTEP:[0m checking the PV [38;5;243m01/17/23 18:39:17.148[0m
    [1mSTEP:[0m setting up the pod [38;5;243m01/17/23 18:39:17.148[0m
    I0117 18:39:17.149067   43786 resource_setup.go:62] adding PV (pvc-a6ced79f-41ba-4074-bd5d-b9c75f2d7a3c) to pod ()
    [1mSTEP:[0m Set pod anti-affinity to make sure two pods are scheduled on different nodes [38;5;243m01/17/23 18:39:17.149[0m
    [1mSTEP:[0m deploying a pod with a volume restored from the snapshot [38;5;243m01/17/23 18:39:17.149[0m
    [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/17/23 18:39:17.21[0m
    Jan 17 18:39:17.211: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-642xt" in namespace "azuredisk-1120" to be "Succeeded or Failed"
    Jan 17 18:39:17.276: INFO: Pod "azuredisk-volume-tester-642xt": Phase="Pending", Reason="", readiness=false. Elapsed: 64.972215ms
    Jan 17 18:39:19.339: INFO: Pod "azuredisk-volume-tester-642xt": Phase="Pending", Reason="", readiness=false. Elapsed: 2.12755347s
    Jan 17 18:39:21.339: INFO: Pod "azuredisk-volume-tester-642xt": Phase="Pending", Reason="", readiness=false. Elapsed: 4.128292981s
    Jan 17 18:39:23.340: INFO: Pod "azuredisk-volume-tester-642xt": Phase="Pending", Reason="", readiness=false. Elapsed: 6.128634572s
    Jan 17 18:39:25.341: INFO: Pod "azuredisk-volume-tester-642xt": Phase="Pending", Reason="", readiness=false. Elapsed: 8.129871965s
    Jan 17 18:39:27.338: INFO: Pod "azuredisk-volume-tester-642xt": Phase="Pending", Reason="", readiness=false. Elapsed: 10.127401412s
... skipping 96 lines ...
    Jan 17 18:42:41.339: INFO: Pod "azuredisk-volume-tester-642xt": Phase="Running", Reason="", readiness=true. Elapsed: 3m24.127630997s
    Jan 17 18:42:43.339: INFO: Pod "azuredisk-volume-tester-642xt": Phase="Running", Reason="", readiness=true. Elapsed: 3m26.1278925s
    Jan 17 18:42:45.338: INFO: Pod "azuredisk-volume-tester-642xt": Phase="Running", Reason="", readiness=false. Elapsed: 3m28.127331707s
    Jan 17 18:42:47.339: INFO: Pod "azuredisk-volume-tester-642xt": Phase="Running", Reason="", readiness=false. Elapsed: 3m30.127718962s
    Jan 17 18:42:49.339: INFO: Pod "azuredisk-volume-tester-642xt": Phase="Succeeded", Reason="", readiness=false. Elapsed: 3m32.128378357s
    [1mSTEP:[0m Saw pod success [38;5;243m01/17/23 18:42:49.34[0m
    Jan 17 18:42:49.340: INFO: Pod "azuredisk-volume-tester-642xt" satisfied condition "Succeeded or Failed"
    Jan 17 18:42:49.340: INFO: deleting Pod "azuredisk-1120"/"azuredisk-volume-tester-642xt"
    Jan 17 18:42:49.425: INFO: Pod azuredisk-volume-tester-642xt has the following logs: hello world

    [1mSTEP:[0m Deleting pod azuredisk-volume-tester-642xt in namespace azuredisk-1120 [38;5;243m01/17/23 18:42:49.425[0m
    Jan 17 18:42:49.503: INFO: deleting PVC "azuredisk-1120"/"pvc-j5h6n"
    Jan 17 18:42:49.503: INFO: Deleting PersistentVolumeClaim "pvc-j5h6n"
... skipping 15 lines ...
    Jan 17 18:43:30.337: INFO: Pod azuredisk-volume-tester-b5dpb has the following logs: 
    [1mSTEP:[0m Deleting pod azuredisk-volume-tester-b5dpb in namespace azuredisk-1120 [38;5;243m01/17/23 18:43:30.337[0m
    [1mSTEP:[0m deleting a VolumeSnapshot volume-snapshot-kczgd [38;5;243m01/17/23 18:43:30.405[0m
    Jan 17 18:43:30.469: INFO: skip deleting VolumeSnapshotClass azuredisk-1120-disk.csi.azure.com-dynamic-sc-lv4vd
    Jan 17 18:43:30.469: INFO: Deleting resource group azuredisk-csi-driver-test-1b20f3ef-9696-11ed-b9f0-5a32c2634674
    Jan 17 18:44:47.663: INFO: deleting Pod "azuredisk-1120"/"azuredisk-volume-tester-j9zlf"
    Jan 17 18:44:47.725: INFO: Error getting logs for pod azuredisk-volume-tester-j9zlf: the server could not find the requested resource (get pods azuredisk-volume-tester-j9zlf)
    [1mSTEP:[0m Deleting pod azuredisk-volume-tester-j9zlf in namespace azuredisk-1120 [38;5;243m01/17/23 18:44:47.725[0m
    Jan 17 18:44:47.791: INFO: deleting PVC "azuredisk-1120"/"pvc-dpzgl"
    Jan 17 18:44:47.791: INFO: Deleting PersistentVolumeClaim "pvc-dpzgl"
    [1mSTEP:[0m waiting for claim's PV "pvc-fd92305e-1233-4e13-9901-1b8cfe378f45" to be deleted [38;5;243m01/17/23 18:44:47.86[0m
    Jan 17 18:44:47.861: INFO: Waiting up to 10m0s for PersistentVolume pvc-fd92305e-1233-4e13-9901-1b8cfe378f45 to get deleted
    Jan 17 18:44:47.922: INFO: PersistentVolume pvc-fd92305e-1233-4e13-9901-1b8cfe378f45 found and phase=Released (61.712271ms)
... skipping 55 lines ...
Jan 17 18:45:12.866: INFO: PersistentVolumeClaim pvc-kdff7 found and phase=Bound (4.178442118s)
[1mSTEP:[0m checking the PVC [38;5;243m01/17/23 18:45:12.866[0m
[1mSTEP:[0m validating provisioned PV [38;5;243m01/17/23 18:45:12.927[0m
[1mSTEP:[0m checking the PV [38;5;243m01/17/23 18:45:12.988[0m
I0117 18:45:12.989775   43786 resource_setup.go:89] adding PV (pvc-91229a00-6f8e-4615-8de3-8a315518c670) to pod ()
[1mSTEP:[0m deploying the pod [38;5;243m01/17/23 18:45:12.99[0m
[1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/17/23 18:45:13.057[0m
Jan 17 18:45:13.057: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-t9psw" in namespace "azuredisk-2437" to be "Succeeded or Failed"
Jan 17 18:45:13.119: INFO: Pod "azuredisk-volume-tester-t9psw": Phase="Pending", Reason="", readiness=false. Elapsed: 61.703371ms
Jan 17 18:45:15.180: INFO: Pod "azuredisk-volume-tester-t9psw": Phase="Pending", Reason="", readiness=false. Elapsed: 2.122995573s
Jan 17 18:45:17.183: INFO: Pod "azuredisk-volume-tester-t9psw": Phase="Pending", Reason="", readiness=false. Elapsed: 4.126067766s
Jan 17 18:45:19.181: INFO: Pod "azuredisk-volume-tester-t9psw": Phase="Pending", Reason="", readiness=false. Elapsed: 6.123710589s
Jan 17 18:45:21.182: INFO: Pod "azuredisk-volume-tester-t9psw": Phase="Pending", Reason="", readiness=false. Elapsed: 8.12509602s
Jan 17 18:45:23.180: INFO: Pod "azuredisk-volume-tester-t9psw": Phase="Pending", Reason="", readiness=false. Elapsed: 10.123203875s
... skipping 19 lines ...
Jan 17 18:46:03.191: INFO: Pod "azuredisk-volume-tester-t9psw": Phase="Running", Reason="", readiness=true. Elapsed: 50.133755562s
Jan 17 18:46:05.180: INFO: Pod "azuredisk-volume-tester-t9psw": Phase="Running", Reason="", readiness=true. Elapsed: 52.122787032s
Jan 17 18:46:07.182: INFO: Pod "azuredisk-volume-tester-t9psw": Phase="Running", Reason="", readiness=false. Elapsed: 54.124370316s
Jan 17 18:46:09.182: INFO: Pod "azuredisk-volume-tester-t9psw": Phase="Running", Reason="", readiness=false. Elapsed: 56.124825112s
Jan 17 18:46:11.183: INFO: Pod "azuredisk-volume-tester-t9psw": Phase="Succeeded", Reason="", readiness=false. Elapsed: 58.125977125s
[1mSTEP:[0m Saw pod success [38;5;243m01/17/23 18:46:11.186[0m
Jan 17 18:46:11.187: INFO: Pod "azuredisk-volume-tester-t9psw" satisfied condition "Succeeded or Failed"
Jan 17 18:46:11.187: INFO: deleting Pod "azuredisk-2437"/"azuredisk-volume-tester-t9psw"
Jan 17 18:46:11.268: INFO: Pod azuredisk-volume-tester-t9psw has the following logs: hello world

[1mSTEP:[0m Deleting pod azuredisk-volume-tester-t9psw in namespace azuredisk-2437 [38;5;243m01/17/23 18:46:11.268[0m
Jan 17 18:46:11.346: INFO: deleting PVC "azuredisk-2437"/"pvc-kdff7"
Jan 17 18:46:11.346: INFO: Deleting PersistentVolumeClaim "pvc-kdff7"
... skipping 85 lines ...
    Jan 17 18:45:12.866: INFO: PersistentVolumeClaim pvc-kdff7 found and phase=Bound (4.178442118s)
    [1mSTEP:[0m checking the PVC [38;5;243m01/17/23 18:45:12.866[0m
    [1mSTEP:[0m validating provisioned PV [38;5;243m01/17/23 18:45:12.927[0m
    [1mSTEP:[0m checking the PV [38;5;243m01/17/23 18:45:12.988[0m
    I0117 18:45:12.989775   43786 resource_setup.go:89] adding PV (pvc-91229a00-6f8e-4615-8de3-8a315518c670) to pod ()
    [1mSTEP:[0m deploying the pod [38;5;243m01/17/23 18:45:12.99[0m
    [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/17/23 18:45:13.057[0m
    Jan 17 18:45:13.057: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-t9psw" in namespace "azuredisk-2437" to be "Succeeded or Failed"
    Jan 17 18:45:13.119: INFO: Pod "azuredisk-volume-tester-t9psw": Phase="Pending", Reason="", readiness=false. Elapsed: 61.703371ms
    Jan 17 18:45:15.180: INFO: Pod "azuredisk-volume-tester-t9psw": Phase="Pending", Reason="", readiness=false. Elapsed: 2.122995573s
    Jan 17 18:45:17.183: INFO: Pod "azuredisk-volume-tester-t9psw": Phase="Pending", Reason="", readiness=false. Elapsed: 4.126067766s
    Jan 17 18:45:19.181: INFO: Pod "azuredisk-volume-tester-t9psw": Phase="Pending", Reason="", readiness=false. Elapsed: 6.123710589s
    Jan 17 18:45:21.182: INFO: Pod "azuredisk-volume-tester-t9psw": Phase="Pending", Reason="", readiness=false. Elapsed: 8.12509602s
    Jan 17 18:45:23.180: INFO: Pod "azuredisk-volume-tester-t9psw": Phase="Pending", Reason="", readiness=false. Elapsed: 10.123203875s
... skipping 19 lines ...
    Jan 17 18:46:03.191: INFO: Pod "azuredisk-volume-tester-t9psw": Phase="Running", Reason="", readiness=true. Elapsed: 50.133755562s
    Jan 17 18:46:05.180: INFO: Pod "azuredisk-volume-tester-t9psw": Phase="Running", Reason="", readiness=true. Elapsed: 52.122787032s
    Jan 17 18:46:07.182: INFO: Pod "azuredisk-volume-tester-t9psw": Phase="Running", Reason="", readiness=false. Elapsed: 54.124370316s
    Jan 17 18:46:09.182: INFO: Pod "azuredisk-volume-tester-t9psw": Phase="Running", Reason="", readiness=false. Elapsed: 56.124825112s
    Jan 17 18:46:11.183: INFO: Pod "azuredisk-volume-tester-t9psw": Phase="Succeeded", Reason="", readiness=false. Elapsed: 58.125977125s
    [1mSTEP:[0m Saw pod success [38;5;243m01/17/23 18:46:11.186[0m
    Jan 17 18:46:11.187: INFO: Pod "azuredisk-volume-tester-t9psw" satisfied condition "Succeeded or Failed"
    Jan 17 18:46:11.187: INFO: deleting Pod "azuredisk-2437"/"azuredisk-volume-tester-t9psw"
    Jan 17 18:46:11.268: INFO: Pod azuredisk-volume-tester-t9psw has the following logs: hello world

    [1mSTEP:[0m Deleting pod azuredisk-volume-tester-t9psw in namespace azuredisk-2437 [38;5;243m01/17/23 18:46:11.268[0m
    Jan 17 18:46:11.346: INFO: deleting PVC "azuredisk-2437"/"pvc-kdff7"
    Jan 17 18:46:11.346: INFO: Deleting PersistentVolumeClaim "pvc-kdff7"
... skipping 652 lines ...
I0117 18:56:39.906588   43786 dynamically_provisioned_pod_delete.go:101] volumes: [{Name:test-volume-1 VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:&PersistentVolumeClaimVolumeSource{ClaimName:pvc-h4x6n,ReadOnly:false,} RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}]
[1mSTEP:[0m begin to delete the pod [38;5;243m01/17/23 18:56:39.906[0m
Jan 17 18:56:39.906: INFO: deleting Pod "azuredisk-517"/"azuredisk-volume-tester-gvk4l"
Jan 17 18:56:39.985: INFO: Pod azuredisk-volume-tester-gvk4l has the following logs: 
[1mSTEP:[0m Deleting pod azuredisk-volume-tester-gvk4l in namespace azuredisk-517 [38;5;243m01/17/23 18:56:39.986[0m
Jan 17 18:57:20.173: INFO: deleting Pod "azuredisk-517"/"azuredisk-volume-tester-gvk4l"
Jan 17 18:57:20.231: INFO: Error getting logs for pod azuredisk-volume-tester-gvk4l: the server could not find the requested resource (get pods azuredisk-volume-tester-gvk4l)
[1mSTEP:[0m Deleting pod azuredisk-volume-tester-gvk4l in namespace azuredisk-517 [38;5;243m01/17/23 18:57:20.231[0m
Jan 17 18:57:20.288: INFO: deleting PVC "azuredisk-517"/"pvc-h4x6n"
Jan 17 18:57:20.288: INFO: Deleting PersistentVolumeClaim "pvc-h4x6n"
[1mSTEP:[0m waiting for claim's PV "pvc-bf8b0e3d-bc24-4084-ae3d-94238448b7fc" to be deleted [38;5;243m01/17/23 18:57:20.352[0m
Jan 17 18:57:20.352: INFO: Waiting up to 10m0s for PersistentVolume pvc-bf8b0e3d-bc24-4084-ae3d-94238448b7fc to get deleted
Jan 17 18:57:20.409: INFO: PersistentVolume pvc-bf8b0e3d-bc24-4084-ae3d-94238448b7fc found and phase=Released (56.974357ms)
... skipping 67 lines ...
    I0117 18:56:39.906588   43786 dynamically_provisioned_pod_delete.go:101] volumes: [{Name:test-volume-1 VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:&PersistentVolumeClaimVolumeSource{ClaimName:pvc-h4x6n,ReadOnly:false,} RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}]
    [1mSTEP:[0m begin to delete the pod [38;5;243m01/17/23 18:56:39.906[0m
    Jan 17 18:56:39.906: INFO: deleting Pod "azuredisk-517"/"azuredisk-volume-tester-gvk4l"
    Jan 17 18:56:39.985: INFO: Pod azuredisk-volume-tester-gvk4l has the following logs: 
    [1mSTEP:[0m Deleting pod azuredisk-volume-tester-gvk4l in namespace azuredisk-517 [38;5;243m01/17/23 18:56:39.986[0m
    Jan 17 18:57:20.173: INFO: deleting Pod "azuredisk-517"/"azuredisk-volume-tester-gvk4l"
    Jan 17 18:57:20.231: INFO: Error getting logs for pod azuredisk-volume-tester-gvk4l: the server could not find the requested resource (get pods azuredisk-volume-tester-gvk4l)
    [1mSTEP:[0m Deleting pod azuredisk-volume-tester-gvk4l in namespace azuredisk-517 [38;5;243m01/17/23 18:57:20.231[0m
    Jan 17 18:57:20.288: INFO: deleting PVC "azuredisk-517"/"pvc-h4x6n"
    Jan 17 18:57:20.288: INFO: Deleting PersistentVolumeClaim "pvc-h4x6n"
    [1mSTEP:[0m waiting for claim's PV "pvc-bf8b0e3d-bc24-4084-ae3d-94238448b7fc" to be deleted [38;5;243m01/17/23 18:57:20.352[0m
    Jan 17 18:57:20.352: INFO: Waiting up to 10m0s for PersistentVolume pvc-bf8b0e3d-bc24-4084-ae3d-94238448b7fc to get deleted
    Jan 17 18:57:20.409: INFO: PersistentVolume pvc-bf8b0e3d-bc24-4084-ae3d-94238448b7fc found and phase=Released (56.974357ms)
... skipping 61 lines ...
I0117 18:58:15.510664   43786 dynamically_provisioned_pod_delete.go:101] volumes: [{Name:test-volume-1 VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:&PersistentVolumeClaimVolumeSource{ClaimName:pvc-hl555,ReadOnly:false,} RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}]
[1mSTEP:[0m begin to delete the pod [38;5;243m01/17/23 18:58:15.51[0m
Jan 17 18:58:15.511: INFO: deleting Pod "azuredisk-3722"/"azuredisk-volume-tester-k9jgm"
Jan 17 18:58:15.589: INFO: Pod azuredisk-volume-tester-k9jgm has the following logs: 
[1mSTEP:[0m Deleting pod azuredisk-volume-tester-k9jgm in namespace azuredisk-3722 [38;5;243m01/17/23 18:58:15.589[0m
Jan 17 18:58:25.839: INFO: deleting Pod "azuredisk-3722"/"azuredisk-volume-tester-k9jgm"
Jan 17 18:58:25.895: INFO: Error getting logs for pod azuredisk-volume-tester-k9jgm: the server could not find the requested resource (get pods azuredisk-volume-tester-k9jgm)
[1mSTEP:[0m Deleting pod azuredisk-volume-tester-k9jgm in namespace azuredisk-3722 [38;5;243m01/17/23 18:58:25.895[0m
Jan 17 18:58:25.959: INFO: deleting PVC "azuredisk-3722"/"pvc-hl555"
Jan 17 18:58:25.959: INFO: Deleting PersistentVolumeClaim "pvc-hl555"
[1mSTEP:[0m waiting for claim's PV "pvc-33a0c090-d23d-43ea-8b14-34b5b18ace85" to be deleted [38;5;243m01/17/23 18:58:26.026[0m
Jan 17 18:58:26.027: INFO: Waiting up to 10m0s for PersistentVolume pvc-33a0c090-d23d-43ea-8b14-34b5b18ace85 to get deleted
Jan 17 18:58:26.084: INFO: PersistentVolume pvc-33a0c090-d23d-43ea-8b14-34b5b18ace85 found and phase=Released (56.223168ms)
... skipping 65 lines ...
    I0117 18:58:15.510664   43786 dynamically_provisioned_pod_delete.go:101] volumes: [{Name:test-volume-1 VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:&PersistentVolumeClaimVolumeSource{ClaimName:pvc-hl555,ReadOnly:false,} RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}]
    [1mSTEP:[0m begin to delete the pod [38;5;243m01/17/23 18:58:15.51[0m
    Jan 17 18:58:15.511: INFO: deleting Pod "azuredisk-3722"/"azuredisk-volume-tester-k9jgm"
    Jan 17 18:58:15.589: INFO: Pod azuredisk-volume-tester-k9jgm has the following logs: 
    [1mSTEP:[0m Deleting pod azuredisk-volume-tester-k9jgm in namespace azuredisk-3722 [38;5;243m01/17/23 18:58:15.589[0m
    Jan 17 18:58:25.839: INFO: deleting Pod "azuredisk-3722"/"azuredisk-volume-tester-k9jgm"
    Jan 17 18:58:25.895: INFO: Error getting logs for pod azuredisk-volume-tester-k9jgm: the server could not find the requested resource (get pods azuredisk-volume-tester-k9jgm)
    [1mSTEP:[0m Deleting pod azuredisk-volume-tester-k9jgm in namespace azuredisk-3722 [38;5;243m01/17/23 18:58:25.895[0m
    Jan 17 18:58:25.959: INFO: deleting PVC "azuredisk-3722"/"pvc-hl555"
    Jan 17 18:58:25.959: INFO: Deleting PersistentVolumeClaim "pvc-hl555"
    [1mSTEP:[0m waiting for claim's PV "pvc-33a0c090-d23d-43ea-8b14-34b5b18ace85" to be deleted [38;5;243m01/17/23 18:58:26.026[0m
    Jan 17 18:58:26.027: INFO: Waiting up to 10m0s for PersistentVolume pvc-33a0c090-d23d-43ea-8b14-34b5b18ace85 to get deleted
    Jan 17 18:58:26.084: INFO: PersistentVolume pvc-33a0c090-d23d-43ea-8b14-34b5b18ace85 found and phase=Released (56.223168ms)
... skipping 50 lines ...
I0117 18:59:34.836124   43786 statefulset.go:175] 1/1 replicas in the StatefulSet are ready
[1mSTEP:[0m checking that the pod is running [38;5;243m01/17/23 18:59:34.952[0m
I0117 18:59:35.009315   43786 statefulset.go:175] 1/1 replicas in the StatefulSet are ready
[1mSTEP:[0m check pod exec [38;5;243m01/17/23 18:59:35.009[0m
Jan 17 18:59:35.009: INFO: Running '/usr/local/bin/kubectl --kubeconfig=/home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure/kubeconfig --namespace=azuredisk-4739 exec azuredisk-volume-tester-sprtv-0 -- cmd /c type C:\mnt\test-1\data.txt'
Jan 17 18:59:35.884: INFO: rc: 1
Jan 17 18:59:35.884: INFO: Error waiting for output "hello world\r\n" in pod "azuredisk-volume-tester-sprtv-0": error running /usr/local/bin/kubectl --kubeconfig=/home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure/kubeconfig --namespace=azuredisk-4739 exec azuredisk-volume-tester-sprtv-0 -- cmd /c type C:\mnt\test-1\data.txt:
Command stdout:

stderr:
The system cannot find the file specified.
command terminated with exit code 1

error:
exit status 1.
Jan 17 18:59:37.885: INFO: Running '/usr/local/bin/kubectl --kubeconfig=/home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure/kubeconfig --namespace=azuredisk-4739 exec azuredisk-volume-tester-sprtv-0 -- cmd /c type C:\mnt\test-1\data.txt'
Jan 17 18:59:38.649: INFO: stderr: ""
Jan 17 18:59:38.649: INFO: stdout: "hello world\r\n"
[1mSTEP:[0m deleting the pod for statefulset [38;5;243m01/17/23 18:59:38.649[0m
Jan 17 18:59:38.649: INFO: Deleting pod "azuredisk-volume-tester-sprtv-0" in namespace "azuredisk-4739"
... skipping 62 lines ...
    I0117 18:59:34.836124   43786 statefulset.go:175] 1/1 replicas in the StatefulSet are ready
    [1mSTEP:[0m checking that the pod is running [38;5;243m01/17/23 18:59:34.952[0m
    I0117 18:59:35.009315   43786 statefulset.go:175] 1/1 replicas in the StatefulSet are ready
    [1mSTEP:[0m check pod exec [38;5;243m01/17/23 18:59:35.009[0m
    Jan 17 18:59:35.009: INFO: Running '/usr/local/bin/kubectl --kubeconfig=/home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure/kubeconfig --namespace=azuredisk-4739 exec azuredisk-volume-tester-sprtv-0 -- cmd /c type C:\mnt\test-1\data.txt'
    Jan 17 18:59:35.884: INFO: rc: 1
    Jan 17 18:59:35.884: INFO: Error waiting for output "hello world\r\n" in pod "azuredisk-volume-tester-sprtv-0": error running /usr/local/bin/kubectl --kubeconfig=/home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure/kubeconfig --namespace=azuredisk-4739 exec azuredisk-volume-tester-sprtv-0 -- cmd /c type C:\mnt\test-1\data.txt:
    Command stdout:

    stderr:
    The system cannot find the file specified.
    command terminated with exit code 1

    error:
    exit status 1.
    Jan 17 18:59:37.885: INFO: Running '/usr/local/bin/kubectl --kubeconfig=/home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure/kubeconfig --namespace=azuredisk-4739 exec azuredisk-volume-tester-sprtv-0 -- cmd /c type C:\mnt\test-1\data.txt'
    Jan 17 18:59:38.649: INFO: stderr: ""
    Jan 17 18:59:38.649: INFO: stdout: "hello world\r\n"
    [1mSTEP:[0m deleting the pod for statefulset [38;5;243m01/17/23 18:59:38.649[0m
    Jan 17 18:59:38.649: INFO: Deleting pod "azuredisk-volume-tester-sprtv-0" in namespace "azuredisk-4739"
... skipping 795 lines ...
I0117 19:16:37.627400   43786 dynamically_provisioned_pod_affinity.go:145] found 1 AzVolumeAttachments for volume (pvc-37c0c80a-58bd-419a-9ef9-f3805fdc51d8)
I0117 19:16:39.629205   43786 dynamically_provisioned_pod_affinity.go:145] found 1 AzVolumeAttachments for volume (pvc-37c0c80a-58bd-419a-9ef9-f3805fdc51d8)
I0117 19:16:41.629294   43786 dynamically_provisioned_pod_affinity.go:145] found 1 AzVolumeAttachments for volume (pvc-37c0c80a-58bd-419a-9ef9-f3805fdc51d8)
I0117 19:16:43.632869   43786 dynamically_provisioned_pod_affinity.go:145] found 1 AzVolumeAttachments for volume (pvc-37c0c80a-58bd-419a-9ef9-f3805fdc51d8)
I0117 19:16:45.628915   43786 dynamically_provisioned_pod_affinity.go:145] found 1 AzVolumeAttachments for volume (pvc-37c0c80a-58bd-419a-9ef9-f3805fdc51d8)
I0117 19:16:45.691085   43786 dynamically_provisioned_pod_affinity.go:145] found 1 AzVolumeAttachments for volume (pvc-37c0c80a-58bd-419a-9ef9-f3805fdc51d8)
Jan 17 19:16:45.691: INFO: Unexpected error: 
    <*errors.errorString | 0xc000586fc0>: {
        s: "timed out waiting for the condition",
    }
Jan 17 19:16:45.691: FAIL: timed out waiting for the condition

Full Stack Trace
sigs.k8s.io/azuredisk-csi-driver/test/e2e/testsuites.(*PodAffinity).Run(0xc000e39340, {0x26fb380, 0xc000e1a4e0}, 0xb?, {0x220ec17, 0x20})
	/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/testsuites/dynamically_provisioned_pod_affinity.go:168 +0x1e9
sigs.k8s.io/azuredisk-csi-driver/test/e2e.(*dynamicProvisioningTestSuite).defineTests.func31()
	/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning_test.go:1347 +0xcca
... skipping 36 lines ...
Jan 17 19:18:02.256: INFO: Waiting up to 5m0s for PersistentVolumeClaim pvc-ph4m6 to be removed
Jan 17 19:18:02.313: INFO: Claim "pvc-ph4m6" in namespace "azuredisk-1632" doesn't exist in the system
Jan 17 19:18:02.313: INFO: deleting StorageClass azuredisk-1632-disk.csi.azure.com-dynamic-sc-f6pd4
[1mSTEP:[0m dump namespace information after failure [38;5;243m01/17/23 19:18:02.378[0m
[1mSTEP:[0m Destroying namespace "azuredisk-1632" for this suite. [38;5;243m01/17/23 19:18:02.379[0m
[38;5;243m------------------------------[0m
[38;5;9m• [FAILED] [797.642 seconds][0m
Dynamic Provisioning
[38;5;243m/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning_test.go:41[0m
  [single-az]
  [38;5;243m/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning_test.go:45[0m
    [38;5;9m[1m[It] Should create replicas on node with matching pod affinity[0m
    [38;5;243m/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning_test.go:1275[0m
... skipping 400 lines ...
    I0117 19:16:37.627400   43786 dynamically_provisioned_pod_affinity.go:145] found 1 AzVolumeAttachments for volume (pvc-37c0c80a-58bd-419a-9ef9-f3805fdc51d8)
    I0117 19:16:39.629205   43786 dynamically_provisioned_pod_affinity.go:145] found 1 AzVolumeAttachments for volume (pvc-37c0c80a-58bd-419a-9ef9-f3805fdc51d8)
    I0117 19:16:41.629294   43786 dynamically_provisioned_pod_affinity.go:145] found 1 AzVolumeAttachments for volume (pvc-37c0c80a-58bd-419a-9ef9-f3805fdc51d8)
    I0117 19:16:43.632869   43786 dynamically_provisioned_pod_affinity.go:145] found 1 AzVolumeAttachments for volume (pvc-37c0c80a-58bd-419a-9ef9-f3805fdc51d8)
    I0117 19:16:45.628915   43786 dynamically_provisioned_pod_affinity.go:145] found 1 AzVolumeAttachments for volume (pvc-37c0c80a-58bd-419a-9ef9-f3805fdc51d8)
    I0117 19:16:45.691085   43786 dynamically_provisioned_pod_affinity.go:145] found 1 AzVolumeAttachments for volume (pvc-37c0c80a-58bd-419a-9ef9-f3805fdc51d8)
    Jan 17 19:16:45.691: INFO: Unexpected error: 
        <*errors.errorString | 0xc000586fc0>: {
            s: "timed out waiting for the condition",
        }
    Jan 17 19:16:45.691: FAIL: timed out waiting for the condition

    Full Stack Trace
    sigs.k8s.io/azuredisk-csi-driver/test/e2e/testsuites.(*PodAffinity).Run(0xc000e39340, {0x26fb380, 0xc000e1a4e0}, 0xb?, {0x220ec17, 0x20})
    	/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/testsuites/dynamically_provisioned_pod_affinity.go:168 +0x1e9
    sigs.k8s.io/azuredisk-csi-driver/test/e2e.(*dynamicProvisioningTestSuite).defineTests.func31()
    	/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning_test.go:1347 +0xcca
... skipping 45 lines ...

  [1mThere were additional failures detected after the initial failure:[0m
    [38;5;13m[PANICKED][0m
    [38;5;13mTest Panicked[0m
    [38;5;13mIn [1m[DeferCleanup (Each)][0m[38;5;13m at: [1m/usr/local/go/src/runtime/panic.go:260[0m

    [38;5;13mruntime error: invalid memory address or nil pointer dereference[0m

    [38;5;13mFull Stack Trace[0m
      k8s.io/kubernetes/test/e2e/framework.(*Framework).dumpNamespaceInfo.func1()
      	/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:274 +0x5c
      k8s.io/kubernetes/test/e2e/framework.(*Framework).dumpNamespaceInfo(0xc0006e43c0)
      	/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:271 +0x179
... skipping 369 lines ...
Jan 17 19:21:47.673: INFO: deployment status: v1.DeploymentStatus{ObservedGeneration:1, Replicas:1, UpdatedReplicas:1, ReadyReplicas:0, AvailableReplicas:0, UnavailableReplicas:1, Conditions:[]v1.DeploymentCondition{v1.DeploymentCondition{Type:"Available", Status:"False", LastUpdateTime:time.Date(2023, time.January, 17, 19, 21, 7, 0, time.Local), LastTransitionTime:time.Date(2023, time.January, 17, 19, 21, 7, 0, time.Local), Reason:"MinimumReplicasUnavailable", Message:"Deployment does not have minimum availability."}, v1.DeploymentCondition{Type:"Progressing", Status:"True", LastUpdateTime:time.Date(2023, time.January, 17, 19, 21, 7, 0, time.Local), LastTransitionTime:time.Date(2023, time.January, 17, 19, 21, 7, 0, time.Local), Reason:"ReplicaSetUpdated", Message:"ReplicaSet \"azuredisk-volume-tester-btfbq-6d49ccc84f\" is progressing."}}, CollisionCount:(*int32)(nil)}
Jan 17 19:21:49.674: INFO: deployment status: v1.DeploymentStatus{ObservedGeneration:1, Replicas:1, UpdatedReplicas:1, ReadyReplicas:0, AvailableReplicas:0, UnavailableReplicas:1, Conditions:[]v1.DeploymentCondition{v1.DeploymentCondition{Type:"Available", Status:"False", LastUpdateTime:time.Date(2023, time.January, 17, 19, 21, 7, 0, time.Local), LastTransitionTime:time.Date(2023, time.January, 17, 19, 21, 7, 0, time.Local), Reason:"MinimumReplicasUnavailable", Message:"Deployment does not have minimum availability."}, v1.DeploymentCondition{Type:"Progressing", Status:"True", LastUpdateTime:time.Date(2023, time.January, 17, 19, 21, 7, 0, time.Local), LastTransitionTime:time.Date(2023, time.January, 17, 19, 21, 7, 0, time.Local), Reason:"ReplicaSetUpdated", Message:"ReplicaSet \"azuredisk-volume-tester-btfbq-6d49ccc84f\" is progressing."}}, CollisionCount:(*int32)(nil)}
[1mSTEP:[0m checking that the pod is running [38;5;243m01/17/23 19:21:51.79[0m
[1mSTEP:[0m check pod exec [38;5;243m01/17/23 19:21:51.966[0m
Jan 17 19:21:51.966: INFO: Running '/usr/local/bin/kubectl --kubeconfig=/home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure/kubeconfig --namespace=azuredisk-6257 exec azuredisk-volume-tester-btfbq-6d49ccc84f-6szsq -- cmd /c type C:\mnt\test-1\data.txt'
Jan 17 19:21:52.841: INFO: rc: 1
Jan 17 19:21:52.841: INFO: Error waiting for output "hello world\r\n" in pod "azuredisk-volume-tester-btfbq-6d49ccc84f-6szsq": error running /usr/local/bin/kubectl --kubeconfig=/home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure/kubeconfig --namespace=azuredisk-6257 exec azuredisk-volume-tester-btfbq-6d49ccc84f-6szsq -- cmd /c type C:\mnt\test-1\data.txt:
Command stdout:

stderr:
The system cannot find the file specified.
command terminated with exit code 1

error:
exit status 1.
Jan 17 19:21:54.842: INFO: Running '/usr/local/bin/kubectl --kubeconfig=/home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure/kubeconfig --namespace=azuredisk-6257 exec azuredisk-volume-tester-btfbq-6d49ccc84f-6szsq -- cmd /c type C:\mnt\test-1\data.txt'
Jan 17 19:21:55.727: INFO: rc: 1
Jan 17 19:21:55.727: INFO: Error waiting for output "hello world\r\n" in pod "azuredisk-volume-tester-btfbq-6d49ccc84f-6szsq": error running /usr/local/bin/kubectl --kubeconfig=/home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure/kubeconfig --namespace=azuredisk-6257 exec azuredisk-volume-tester-btfbq-6d49ccc84f-6szsq -- cmd /c type C:\mnt\test-1\data.txt:
Command stdout:

stderr:
The system cannot find the file specified.
command terminated with exit code 1

error:
exit status 1.
Jan 17 19:21:56.841: INFO: Running '/usr/local/bin/kubectl --kubeconfig=/home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure/kubeconfig --namespace=azuredisk-6257 exec azuredisk-volume-tester-btfbq-6d49ccc84f-6szsq -- cmd /c type C:\mnt\test-1\data.txt'
Jan 17 19:21:57.725: INFO: stderr: ""
Jan 17 19:21:57.725: INFO: stdout: "hello world\r\n"
[1mSTEP:[0m cordoning node 0 [38;5;243m01/17/23 19:21:57.725[0m
[1mSTEP:[0m deleting the pod for deployment [38;5;243m01/17/23 19:21:57.867[0m
... skipping 84 lines ...
    Jan 17 19:21:47.673: INFO: deployment status: v1.DeploymentStatus{ObservedGeneration:1, Replicas:1, UpdatedReplicas:1, ReadyReplicas:0, AvailableReplicas:0, UnavailableReplicas:1, Conditions:[]v1.DeploymentCondition{v1.DeploymentCondition{Type:"Available", Status:"False", LastUpdateTime:time.Date(2023, time.January, 17, 19, 21, 7, 0, time.Local), LastTransitionTime:time.Date(2023, time.January, 17, 19, 21, 7, 0, time.Local), Reason:"MinimumReplicasUnavailable", Message:"Deployment does not have minimum availability."}, v1.DeploymentCondition{Type:"Progressing", Status:"True", LastUpdateTime:time.Date(2023, time.January, 17, 19, 21, 7, 0, time.Local), LastTransitionTime:time.Date(2023, time.January, 17, 19, 21, 7, 0, time.Local), Reason:"ReplicaSetUpdated", Message:"ReplicaSet \"azuredisk-volume-tester-btfbq-6d49ccc84f\" is progressing."}}, CollisionCount:(*int32)(nil)}
    Jan 17 19:21:49.674: INFO: deployment status: v1.DeploymentStatus{ObservedGeneration:1, Replicas:1, UpdatedReplicas:1, ReadyReplicas:0, AvailableReplicas:0, UnavailableReplicas:1, Conditions:[]v1.DeploymentCondition{v1.DeploymentCondition{Type:"Available", Status:"False", LastUpdateTime:time.Date(2023, time.January, 17, 19, 21, 7, 0, time.Local), LastTransitionTime:time.Date(2023, time.January, 17, 19, 21, 7, 0, time.Local), Reason:"MinimumReplicasUnavailable", Message:"Deployment does not have minimum availability."}, v1.DeploymentCondition{Type:"Progressing", Status:"True", LastUpdateTime:time.Date(2023, time.January, 17, 19, 21, 7, 0, time.Local), LastTransitionTime:time.Date(2023, time.January, 17, 19, 21, 7, 0, time.Local), Reason:"ReplicaSetUpdated", Message:"ReplicaSet \"azuredisk-volume-tester-btfbq-6d49ccc84f\" is progressing."}}, CollisionCount:(*int32)(nil)}
    [1mSTEP:[0m checking that the pod is running [38;5;243m01/17/23 19:21:51.79[0m
    [1mSTEP:[0m check pod exec [38;5;243m01/17/23 19:21:51.966[0m
    Jan 17 19:21:51.966: INFO: Running '/usr/local/bin/kubectl --kubeconfig=/home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure/kubeconfig --namespace=azuredisk-6257 exec azuredisk-volume-tester-btfbq-6d49ccc84f-6szsq -- cmd /c type C:\mnt\test-1\data.txt'
    Jan 17 19:21:52.841: INFO: rc: 1
    Jan 17 19:21:52.841: INFO: Error waiting for output "hello world\r\n" in pod "azuredisk-volume-tester-btfbq-6d49ccc84f-6szsq": error running /usr/local/bin/kubectl --kubeconfig=/home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure/kubeconfig --namespace=azuredisk-6257 exec azuredisk-volume-tester-btfbq-6d49ccc84f-6szsq -- cmd /c type C:\mnt\test-1\data.txt:
    Command stdout:

    stderr:
    The system cannot find the file specified.
    command terminated with exit code 1

    error:
    exit status 1.
    Jan 17 19:21:54.842: INFO: Running '/usr/local/bin/kubectl --kubeconfig=/home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure/kubeconfig --namespace=azuredisk-6257 exec azuredisk-volume-tester-btfbq-6d49ccc84f-6szsq -- cmd /c type C:\mnt\test-1\data.txt'
    Jan 17 19:21:55.727: INFO: rc: 1
    Jan 17 19:21:55.727: INFO: Error waiting for output "hello world\r\n" in pod "azuredisk-volume-tester-btfbq-6d49ccc84f-6szsq": error running /usr/local/bin/kubectl --kubeconfig=/home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure/kubeconfig --namespace=azuredisk-6257 exec azuredisk-volume-tester-btfbq-6d49ccc84f-6szsq -- cmd /c type C:\mnt\test-1\data.txt:
    Command stdout:

    stderr:
    The system cannot find the file specified.
    command terminated with exit code 1

    error:
    exit status 1.
    Jan 17 19:21:56.841: INFO: Running '/usr/local/bin/kubectl --kubeconfig=/home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure/kubeconfig --namespace=azuredisk-6257 exec azuredisk-volume-tester-btfbq-6d49ccc84f-6szsq -- cmd /c type C:\mnt\test-1\data.txt'
    Jan 17 19:21:57.725: INFO: stderr: ""
    Jan 17 19:21:57.725: INFO: stdout: "hello world\r\n"
    [1mSTEP:[0m cordoning node 0 [38;5;243m01/17/23 19:21:57.725[0m
    [1mSTEP:[0m deleting the pod for deployment [38;5;243m01/17/23 19:21:57.867[0m
... skipping 83 lines ...
Jan 17 19:23:49.241: INFO: deployment status: v1.DeploymentStatus{ObservedGeneration:1, Replicas:1, UpdatedReplicas:1, ReadyReplicas:0, AvailableReplicas:0, UnavailableReplicas:1, Conditions:[]v1.DeploymentCondition{v1.DeploymentCondition{Type:"Available", Status:"False", LastUpdateTime:time.Date(2023, time.January, 17, 19, 23, 9, 0, time.Local), LastTransitionTime:time.Date(2023, time.January, 17, 19, 23, 9, 0, time.Local), Reason:"MinimumReplicasUnavailable", Message:"Deployment does not have minimum availability."}, v1.DeploymentCondition{Type:"Progressing", Status:"True", LastUpdateTime:time.Date(2023, time.January, 17, 19, 23, 9, 0, time.Local), LastTransitionTime:time.Date(2023, time.January, 17, 19, 23, 9, 0, time.Local), Reason:"ReplicaSetUpdated", Message:"ReplicaSet \"azuredisk-volume-tester-8m7qb-c8c74869d\" is progressing."}}, CollisionCount:(*int32)(nil)}
Jan 17 19:23:51.238: INFO: deployment status: v1.DeploymentStatus{ObservedGeneration:1, Replicas:1, UpdatedReplicas:1, ReadyReplicas:0, AvailableReplicas:0, UnavailableReplicas:1, Conditions:[]v1.DeploymentCondition{v1.DeploymentCondition{Type:"Available", Status:"False", LastUpdateTime:time.Date(2023, time.January, 17, 19, 23, 9, 0, time.Local), LastTransitionTime:time.Date(2023, time.January, 17, 19, 23, 9, 0, time.Local), Reason:"MinimumReplicasUnavailable", Message:"Deployment does not have minimum availability."}, v1.DeploymentCondition{Type:"Progressing", Status:"True", LastUpdateTime:time.Date(2023, time.January, 17, 19, 23, 9, 0, time.Local), LastTransitionTime:time.Date(2023, time.January, 17, 19, 23, 9, 0, time.Local), Reason:"ReplicaSetUpdated", Message:"ReplicaSet \"azuredisk-volume-tester-8m7qb-c8c74869d\" is progressing."}}, CollisionCount:(*int32)(nil)}
[1mSTEP:[0m checking that the pod is running [38;5;243m01/17/23 19:23:53.361[0m
[1mSTEP:[0m check pod exec [38;5;243m01/17/23 19:23:53.545[0m
Jan 17 19:23:53.545: INFO: Running '/usr/local/bin/kubectl --kubeconfig=/home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure/kubeconfig --namespace=azuredisk-205 exec azuredisk-volume-tester-8m7qb-c8c74869d-zgf7z -- cmd /c type C:\mnt\test-1\data.txt'
Jan 17 19:23:54.872: INFO: rc: 1
Jan 17 19:23:54.873: INFO: Error waiting for output "hello world\r\n" in pod "azuredisk-volume-tester-8m7qb-c8c74869d-zgf7z": error running /usr/local/bin/kubectl --kubeconfig=/home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure/kubeconfig --namespace=azuredisk-205 exec azuredisk-volume-tester-8m7qb-c8c74869d-zgf7z -- cmd /c type C:\mnt\test-1\data.txt:
Command stdout:

stderr:
The system cannot find the file specified.
command terminated with exit code 1

error:
exit status 1.
Jan 17 19:23:56.874: INFO: Running '/usr/local/bin/kubectl --kubeconfig=/home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure/kubeconfig --namespace=azuredisk-205 exec azuredisk-volume-tester-8m7qb-c8c74869d-zgf7z -- cmd /c type C:\mnt\test-1\data.txt'
Jan 17 19:23:58.062: INFO: stderr: ""
Jan 17 19:23:58.062: INFO: stdout: "hello world\r\n"
[1mSTEP:[0m cordoning node 0 [38;5;243m01/17/23 19:23:58.062[0m
[1mSTEP:[0m deleting the pod for deployment [38;5;243m01/17/23 19:23:58.209[0m
... skipping 83 lines ...
    Jan 17 19:23:49.241: INFO: deployment status: v1.DeploymentStatus{ObservedGeneration:1, Replicas:1, UpdatedReplicas:1, ReadyReplicas:0, AvailableReplicas:0, UnavailableReplicas:1, Conditions:[]v1.DeploymentCondition{v1.DeploymentCondition{Type:"Available", Status:"False", LastUpdateTime:time.Date(2023, time.January, 17, 19, 23, 9, 0, time.Local), LastTransitionTime:time.Date(2023, time.January, 17, 19, 23, 9, 0, time.Local), Reason:"MinimumReplicasUnavailable", Message:"Deployment does not have minimum availability."}, v1.DeploymentCondition{Type:"Progressing", Status:"True", LastUpdateTime:time.Date(2023, time.January, 17, 19, 23, 9, 0, time.Local), LastTransitionTime:time.Date(2023, time.January, 17, 19, 23, 9, 0, time.Local), Reason:"ReplicaSetUpdated", Message:"ReplicaSet \"azuredisk-volume-tester-8m7qb-c8c74869d\" is progressing."}}, CollisionCount:(*int32)(nil)}
    Jan 17 19:23:51.238: INFO: deployment status: v1.DeploymentStatus{ObservedGeneration:1, Replicas:1, UpdatedReplicas:1, ReadyReplicas:0, AvailableReplicas:0, UnavailableReplicas:1, Conditions:[]v1.DeploymentCondition{v1.DeploymentCondition{Type:"Available", Status:"False", LastUpdateTime:time.Date(2023, time.January, 17, 19, 23, 9, 0, time.Local), LastTransitionTime:time.Date(2023, time.January, 17, 19, 23, 9, 0, time.Local), Reason:"MinimumReplicasUnavailable", Message:"Deployment does not have minimum availability."}, v1.DeploymentCondition{Type:"Progressing", Status:"True", LastUpdateTime:time.Date(2023, time.January, 17, 19, 23, 9, 0, time.Local), LastTransitionTime:time.Date(2023, time.January, 17, 19, 23, 9, 0, time.Local), Reason:"ReplicaSetUpdated", Message:"ReplicaSet \"azuredisk-volume-tester-8m7qb-c8c74869d\" is progressing."}}, CollisionCount:(*int32)(nil)}
    [1mSTEP:[0m checking that the pod is running [38;5;243m01/17/23 19:23:53.361[0m
    [1mSTEP:[0m check pod exec [38;5;243m01/17/23 19:23:53.545[0m
    Jan 17 19:23:53.545: INFO: Running '/usr/local/bin/kubectl --kubeconfig=/home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure/kubeconfig --namespace=azuredisk-205 exec azuredisk-volume-tester-8m7qb-c8c74869d-zgf7z -- cmd /c type C:\mnt\test-1\data.txt'
    Jan 17 19:23:54.872: INFO: rc: 1
    Jan 17 19:23:54.873: INFO: Error waiting for output "hello world\r\n" in pod "azuredisk-volume-tester-8m7qb-c8c74869d-zgf7z": error running /usr/local/bin/kubectl --kubeconfig=/home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure/kubeconfig --namespace=azuredisk-205 exec azuredisk-volume-tester-8m7qb-c8c74869d-zgf7z -- cmd /c type C:\mnt\test-1\data.txt:
    Command stdout:

    stderr:
    The system cannot find the file specified.
    command terminated with exit code 1

    error:
    exit status 1.
    Jan 17 19:23:56.874: INFO: Running '/usr/local/bin/kubectl --kubeconfig=/home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure/kubeconfig --namespace=azuredisk-205 exec azuredisk-volume-tester-8m7qb-c8c74869d-zgf7z -- cmd /c type C:\mnt\test-1\data.txt'
    Jan 17 19:23:58.062: INFO: stderr: ""
    Jan 17 19:23:58.062: INFO: stdout: "hello world\r\n"
    [1mSTEP:[0m cordoning node 0 [38;5;243m01/17/23 19:23:58.062[0m
    [1mSTEP:[0m deleting the pod for deployment [38;5;243m01/17/23 19:23:58.209[0m
... skipping 138 lines ...
  [38;5;243m<< End Captured GinkgoWriter Output[0m

  [38;5;14mtest case not supported by Windows clusters[0m
  [38;5;14mIn [1m[It][0m[38;5;14m at: [1m/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/utils/testutil/env_utils.go:26[0m
[38;5;243m------------------------------[0m
[0mDynamic Provisioning [38;5;243m[single-az][0m
  [1mshould check failed replica attachments are recreated after space is made from a volume detaching.[0m
  [38;5;243m/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning_test.go:1778[0m
[1mSTEP:[0m Creating a kubernetes client [38;5;243m01/17/23 19:25:00.014[0m
Jan 17 19:25:00.014: INFO: >>> kubeConfig: /home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure/kubeconfig
[1mSTEP:[0m Building a namespace api object, basename azuredisk [38;5;243m01/17/23 19:25:00.016[0m
[1mSTEP:[0m Waiting for a default service account to be provisioned in namespace [38;5;243m01/17/23 19:25:00.196[0m
[1mSTEP:[0m Waiting for kube-root-ca.crt to be provisioned in namespace [38;5;243m01/17/23 19:25:00.31[0m
... skipping 6 lines ...
[38;5;243m------------------------------[0m
[38;5;14mS [SKIPPED] [1.481 seconds][0m
Dynamic Provisioning
[38;5;243m/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning_test.go:41[0m
  [single-az]
  [38;5;243m/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning_test.go:45[0m
    [38;5;14m[1m[It] should check failed replica attachments are recreated after space is made from a volume detaching.[0m
    [38;5;243m/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning_test.go:1778[0m

  [38;5;243mBegin Captured GinkgoWriter Output >>[0m
    [1mSTEP:[0m Creating a kubernetes client [38;5;243m01/17/23 19:25:00.014[0m
    Jan 17 19:25:00.014: INFO: >>> kubeConfig: /home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure/kubeconfig
    [1mSTEP:[0m Building a namespace api object, basename azuredisk [38;5;243m01/17/23 19:25:00.016[0m
... skipping 129 lines ...
I0117 17:37:55.551404       1 round_trippers.go:553] GET https://10.96.0.1:443/api/v1/nodes?allowWatchBookmarks=true&resourceVersion=3837&timeout=9m6s&timeoutSeconds=546&watch=true 200 OK in 30 milliseconds
I0117 17:37:55.649031       1 round_trippers.go:553] GET https://10.96.0.1:443/apis/apiextensions.k8s.io/v1/customresourcedefinitions?allowWatchBookmarks=true&resourceVersion=3503&timeout=9m11s&timeoutSeconds=551&watch=true 200 OK in 1 milliseconds
I0117 17:37:55.664199       1 shared_informer.go:303] caches populated
I0117 17:37:55.664357       1 azuredisk_v2.go:225] driver userAgent: disk.csi.azure.com/latest-v2-9ef068a8cb36a997d4ea04b90c05c6f92a488a19 e2e-test
I0117 17:37:55.664383       1 azure_disk_utils.go:564] reading cloud config from secret kube-system/azure-cloud-provider
I0117 17:37:55.666445       1 round_trippers.go:553] GET https://10.96.0.1:443/api/v1/namespaces/kube-system/secrets/azure-cloud-provider 404 Not Found in 1 milliseconds
I0117 17:37:55.666917       1 azure_disk_utils.go:571] InitializeCloudFromSecret: failed to get cloud config from secret kube-system/azure-cloud-provider: failed to get secret kube-system/azure-cloud-provider: secrets "azure-cloud-provider" not found
I0117 17:37:55.666939       1 azure_disk_utils.go:576] could not read cloud config from secret kube-system/azure-cloud-provider
I0117 17:37:55.666946       1 azure_disk_utils.go:586] use default AZURE_CREDENTIAL_FILE env var: /etc/kubernetes/azure.json
I0117 17:37:55.666985       1 azure_disk_utils.go:594] read cloud config from file: /etc/kubernetes/azure.json successfully
I0117 17:37:55.667571       1 azure_auth.go:253] Using AzurePublicCloud environment
I0117 17:37:55.667628       1 azure_auth.go:138] azure: using client_id+client_secret to retrieve access token
I0117 17:37:55.667707       1 azure_diskclient.go:68] Azure DisksClient using API version: 2022-03-02
... skipping 9 lines ...
I0117 17:37:55.979123       1 azure_backoff.go:101] VirtualMachinesClient.List(capz-wto256) success
I0117 17:37:55.979140       1 azure_vmss_cache.go:400] Node  has joined the cluster since the last VM cache refresh in NonVmssUniformNodesEntry, refreshing the cache
I0117 17:37:55.979149       1 azure_vmss_cache.go:327] refresh the cache of NonVmssUniformNodesCache in rg map[capz-wto256:{}]
I0117 17:37:56.069869       1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 13854
I0117 17:37:56.070874       1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=0.091679622 request="vm_list" resource_group="capz-wto256" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0117 17:37:56.070889       1 azure_backoff.go:101] VirtualMachinesClient.List(capz-wto256) success
E0117 17:37:56.070925       1 azuredisk_v2.go:272] Failed to get node info. Error: NewNodeInfo: Failed to get instance type from Azure cloud provider, nodeName: , error: not a vmss instance
I0117 17:37:56.071069       1 mount_linux.go:208] Detected OS without systemd
I0117 17:37:56.071088       1 driver.go:81] Enabling controller service capability: CREATE_DELETE_VOLUME
I0117 17:37:56.071094       1 driver.go:81] Enabling controller service capability: PUBLISH_UNPUBLISH_VOLUME
I0117 17:37:56.071098       1 driver.go:81] Enabling controller service capability: CREATE_DELETE_SNAPSHOT
I0117 17:37:56.071103       1 driver.go:81] Enabling controller service capability: CLONE_VOLUME
I0117 17:37:56.071107       1 driver.go:81] Enabling controller service capability: EXPAND_VOLUME
... skipping 434 lines ...
I0117 17:37:56.350633       1 shared_state.go:490]  "msg"="Storing pod calico-node-v5rh5 and claim [] to podToClaimsMap map." "disk.csi.azure.com/request-id"="ad5ef4e7-968d-11ed-ae72-002248b69c13"
I0117 17:37:56.350716       1 pod.go:91]  "msg"="Creating replicas for pod calico-system/calico-node-v5rh5." "disk.csi.azure.com/request-id"="ad5ef4e7-968d-11ed-ae72-002248b69c13" "disk.csi/azure.com/pod-name"="calico-system/calico-node-v5rh5"
I0117 17:37:56.350856       1 shared_state.go:314]  "msg"="Getting requested volumes for pod (calico-system/calico-node-v5rh5)." "disk.csi.azure.com/request-id"="ad5ef4e7-968d-11ed-ae72-002248b69c13" "disk.csi/azure.com/pod-name"="calico-system/calico-node-v5rh5"
I0117 17:37:56.350927       1 pod.go:99]  "msg"="Pod calico-system/calico-node-v5rh5 has 0 volumes. Volumes: []" "disk.csi.azure.com/request-id"="ad5ef4e7-968d-11ed-ae72-002248b69c13" "disk.csi/azure.com/pod-name"="calico-system/calico-node-v5rh5"
I0117 17:37:56.351007       1 pod.go:89]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="ad5ef4e7-968d-11ed-ae72-002248b69c13" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcilePod).createReplicas" "disk.csi/azure.com/pod-name"="calico-system/calico-node-v5rh5" "latency"=292001
I0117 17:37:56.351036       1 shared_state.go:420]  "msg"="Adding pod csi-azuredisk-controller-7f6f8d54b5-t5fv2 to shared map with keyName kube-system/csi-azuredisk-controller-7f6f8d54b5-t5fv2." "disk.csi.azure.com/request-id"="ad5ef4e7-968d-11ed-ae72-002248b69c13"
I0117 17:37:56.351525       1 shared_state.go:426]  "msg"="Pod spec of pod csi-azuredisk-controller-7f6f8d54b5-t5fv2 is: {Volumes:[{Name:socket-dir VolumeSource:{HostPath:nil EmptyDir:&EmptyDirVolumeSource{Medium:,SizeLimit:<nil>,} GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:azure-cred VolumeSource:{HostPath:&HostPathVolumeSource{Path:/etc/kubernetes/,Type:*DirectoryOrCreate,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:csi-azuredisk-controller-config VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:&ConfigMapVolumeSource{LocalObjectReference:LocalObjectReference{Name:csi-azuredisk-controller-config,},Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,} VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:kube-api-access-2ppxh VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:&ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,} PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:csi-provisioner Image:mcr.microsoft.com/oss/kubernetes-csi/csi-provisioner:v3.2.0 Command:[] Args:[--feature-gates=Topology=true --csi-address=$(ADDRESS) --v=2 --timeout=30s --leader-election --leader-election-namespace=kube-system --worker-threads=100 --extra-create-metadata=true --strict-topology=true --kube-api-qps=50 --kube-api-burst=100] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:ADDRESS Value:/csi/csi.sock ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:524288000 scale:0} d:{Dec:<nil>} s:500Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}] Claims:[]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-2ppxh ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:csi-attacher Image:mcr.microsoft.com/oss/kubernetes-csi/csi-attacher:v3.5.0 Command:[] Args:[-v=2 -csi-address=$(ADDRESS) -timeout=600s -leader-election --leader-election-namespace=kube-system -worker-threads=500 -kube-api-qps=50 -kube-api-burst=100] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:ADDRESS Value:/csi/csi.sock ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:524288000 scale:0} d:{Dec:<nil>} s:500Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}] Claims:[]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-2ppxh ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:csi-snapshotter Image:mcr.microsoft.com/oss/kubernetes-csi/csi-snapshotter:v5.0.1 Command:[] Args:[-csi-address=$(ADDRESS) -leader-election --leader-election-namespace=kube-system -v=2] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:ADDRESS Value:/csi/csi.sock ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:104857600 scale:0} d:{Dec:<nil>} s:100Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}] Claims:[]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-2ppxh ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:csi-resizer Image:mcr.microsoft.com/oss/kubernetes-csi/csi-resizer:v1.5.0 Command:[] Args:[-csi-address=$(ADDRESS) -v=2 -leader-election --leader-election-namespace=kube-system -handle-volume-inuse-error=false -feature-gates=RecoverVolumeExpansionFailure=true -timeout=240s] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:ADDRESS Value:/csi/csi.sock ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:524288000 scale:0} d:{Dec:<nil>} s:500Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}] Claims:[]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-2ppxh ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:liveness-probe Image:mcr.microsoft.com/oss/kubernetes-csi/livenessprobe:v2.7.0 Command:[] Args:[--csi-address=/csi/csi.sock --probe-timeout=3s --health-port=29602 --v=2] WorkingDir: Ports:[] EnvFrom:[] Env:[] Resources:{Limits:map[memory:{i:{value:104857600 scale:0} d:{Dec:<nil>} s:100Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}] Claims:[]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-2ppxh ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:azuredisk Image:capzci.azurecr.io/azuredisk-csi:latest-v2-9ef068a8cb36a997d4ea04b90c05c6f92a488a19 Command:[] Args:[--v=6 --config=/etc/csi-azuredisk-controller/config.yaml] WorkingDir: Ports:[{Name:healthz HostPort:29602 ContainerPort:29602 Protocol:TCP HostIP:} {Name:metrics HostPort:29604 ContainerPort:29604 Protocol:TCP HostIP:}] EnvFrom:[] Env:[{Name:AZURE_CREDENTIAL_FILE Value: ValueFrom:&EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:&ConfigMapKeySelector{LocalObjectReference:LocalObjectReference{Name:azure-cred-file,},Key:path,Optional:*true,},SecretKeyRef:nil,}} {Name:CSI_ENDPOINT Value:unix:///csi/csi.sock ValueFrom:nil} {Name:AZURE_GO_SDK_LOG_LEVEL Value: ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:524288000 scale:0} d:{Dec:<nil>} s:500Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}] Claims:[]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:azure-cred ReadOnly:false MountPath:/etc/kubernetes/ SubPath: MountPropagation:<nil> SubPathExpr:} {Name:csi-azuredisk-controller-config ReadOnly:false MountPath:/etc/csi-azuredisk-controller SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-2ppxh ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/healthz,Port:{1 0 healthz},Host:,Scheme:HTTP,HTTPHeaders:[]HTTPHeader{},},TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:30,TimeoutSeconds:10,PeriodSeconds:30,SuccessThreshold:1,FailureThreshold:5,TerminationGracePeriodSeconds:nil,} ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:Always SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy:Always TerminationGracePeriodSeconds:0xc00072fbd0 ActiveDeadlineSeconds:<nil> DNSPolicy:ClusterFirst NodeSelector:map[kubernetes.io/os:linux] ServiceAccountName:csi-azuredisk-controller-sa DeprecatedServiceAccount:csi-azuredisk-controller-sa AutomountServiceAccountToken:<nil> NodeName:capz-wto256-control-plane-b5m52 HostNetwork:true HostPID:false HostIPC:false ShareProcessNamespace:<nil> SecurityContext:&PodSecurityContext{SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,SupplementalGroups:[],FSGroup:nil,RunAsGroup:nil,Sysctls:[]Sysctl{},WindowsOptions:nil,FSGroupChangePolicy:nil,SeccompProfile:nil,} ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName:default-scheduler Tolerations:[{Key:node-role.kubernetes.io/master Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>} {Key:node-role.kubernetes.io/controlplane Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>} {Key:node-role.kubernetes.io/control-plane Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>} {Key:node.kubernetes.io/not-ready Operator:Exists Value: Effect:NoExecute TolerationSeconds:0xc00072fbd8} {Key:node.kubernetes.io/unreachable Operator:Exists Value: Effect:NoExecute TolerationSeconds:0xc00072fbe0}] HostAliases:[] PriorityClassName:system-cluster-critical Priority:0xc00072fbe8 DNSConfig:nil ReadinessGates:[] RuntimeClassName:<nil> EnableServiceLinks:0xc00072fbec PreemptionPolicy:0xc000a4d200 Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN:<nil> OS:nil HostUsers:<nil> SchedulingGates:[] ResourceClaims:[]}. With volumes: [{Name:socket-dir VolumeSource:{HostPath:nil EmptyDir:&EmptyDirVolumeSource{Medium:,SizeLimit:<nil>,} GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:azure-cred VolumeSource:{HostPath:&HostPathVolumeSource{Path:/etc/kubernetes/,Type:*DirectoryOrCreate,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:csi-azuredisk-controller-config VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:&ConfigMapVolumeSource{LocalObjectReference:LocalObjectReference{Name:csi-azuredisk-controller-config,},Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,} VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:kube-api-access-2ppxh VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:&ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,} PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}]" "disk.csi.azure.com/request-id"="ad5ef4e7-968d-11ed-ae72-002248b69c13"
I0117 17:37:56.351940       1 shared_state.go:464]  "msg"="Pod csi-azuredisk-controller-7f6f8d54b5-t5fv2: Skipping Volume {socket-dir {nil &EmptyDirVolumeSource{Medium:,SizeLimit:<nil>,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}}. No persistent volume exists." "disk.csi.azure.com/request-id"="ad5ef4e7-968d-11ed-ae72-002248b69c13"
I0117 17:37:56.352145       1 shared_state.go:464]  "msg"="Pod csi-azuredisk-controller-7f6f8d54b5-t5fv2: Skipping Volume {azure-cred {&HostPathVolumeSource{Path:/etc/kubernetes/,Type:*DirectoryOrCreate,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}}. No persistent volume exists." "disk.csi.azure.com/request-id"="ad5ef4e7-968d-11ed-ae72-002248b69c13"
I0117 17:37:56.352320       1 shared_state.go:464]  "msg"="Pod csi-azuredisk-controller-7f6f8d54b5-t5fv2: Skipping Volume {csi-azuredisk-controller-config {nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil &ConfigMapVolumeSource{LocalObjectReference:LocalObjectReference{Name:csi-azuredisk-controller-config,},Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,} nil nil nil nil nil nil nil nil nil nil}}. No persistent volume exists." "disk.csi.azure.com/request-id"="ad5ef4e7-968d-11ed-ae72-002248b69c13"
I0117 17:37:56.352527       1 shared_state.go:464]  "msg"="Pod csi-azuredisk-controller-7f6f8d54b5-t5fv2: Skipping Volume {kube-api-access-2ppxh {nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil &ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,} nil nil nil nil nil}}. No persistent volume exists." "disk.csi.azure.com/request-id"="ad5ef4e7-968d-11ed-ae72-002248b69c13"
I0117 17:37:56.352734       1 shared_state.go:490]  "msg"="Storing pod csi-azuredisk-controller-7f6f8d54b5-t5fv2 and claim [] to podToClaimsMap map." "disk.csi.azure.com/request-id"="ad5ef4e7-968d-11ed-ae72-002248b69c13"
I0117 17:37:56.353007       1 pod.go:91]  "msg"="Creating replicas for pod kube-system/csi-azuredisk-controller-7f6f8d54b5-t5fv2." "disk.csi.azure.com/request-id"="ad5ef4e7-968d-11ed-ae72-002248b69c13" "disk.csi/azure.com/pod-name"="kube-system/csi-azuredisk-controller-7f6f8d54b5-t5fv2"
... skipping 91 lines ...
I0117 17:37:56.373728       1 round_trippers.go:553] PUT https://10.96.0.1:443/apis/apiextensions.k8s.io/v1/customresourcedefinitions/azdrivernodes.disk.csi.azure.com/status 200 OK in 9 milliseconds
I0117 17:37:56.377622       1 round_trippers.go:553] GET https://10.96.0.1:443/apis/apiextensions.k8s.io/v1/customresourcedefinitions/azvolumes.disk.csi.azure.com 200 OK in 2 milliseconds
I0117 17:37:56.389489       1 round_trippers.go:553] PUT https://10.96.0.1:443/apis/apiextensions.k8s.io/v1/customresourcedefinitions/azvolumes.disk.csi.azure.com/status 200 OK in 10 milliseconds
I0117 17:37:56.395824       1 round_trippers.go:553] GET https://10.96.0.1:443/apis/apiextensions.k8s.io/v1/customresourcedefinitions/azvolumeattachments.disk.csi.azure.com 200 OK in 4 milliseconds
I0117 17:37:56.408220       1 round_trippers.go:553] PUT https://10.96.0.1:443/apis/apiextensions.k8s.io/v1/customresourcedefinitions/azvolumeattachments.disk.csi.azure.com/status 200 OK in 10 milliseconds
I0117 17:37:56.409103       1 azuredisk_v2.go:435]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="ad5ef4e7-968d-11ed-ae72-002248b69c13" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).StartControllersAndDieOnExit.func1" "latency"=250027323
I0117 17:37:56.415721       1 node.go:90] AzDiskControllerManager "msg"="Node is now available. Will requeue failed replica creation requests." "controller"="node" "disk.csi.azure.com/node-name"="capz-wto2-xzbdk" "namespace"="azure-disk-csi" "partition"="csi-azuredisk-controller"
I0117 17:37:56.415894       1 shared_state.go:1205]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="ad8626d6-968d-11ed-ae72-002248b69c13" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*SharedState).tryCreateFailedReplicas" "latency"=22500
I0117 17:37:56.416089       1 azure_disk_utils.go:1048]  "msg"="Getting AzVolumeAttachment list for node (capz-wto256-control-plane-b5m52)" 
I0117 17:37:56.416258       1 azure_disk_utils.go:1068]  "msg"="Label selector is: disk.csi.azure.com/node-name=capz-wto256-control-plane-b5m52." 
I0117 17:37:56.416294       1 shared_state.go:1745] Added node(capz-wto256-control-plane-b5m52) to availableAttachmentsMap with capacity: 4
I0117 17:37:56.416401       1 node.go:90] AzDiskControllerManager "msg"="Node is now available. Will requeue failed replica creation requests." "controller"="node" "disk.csi.azure.com/node-name"="capz-wto256-control-plane-b5m52" "namespace"="azure-disk-csi" "partition"="csi-azuredisk-controller"
I0117 17:37:56.416509       1 shared_state.go:1205]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="ad863c39-968d-11ed-ae72-002248b69c13" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*SharedState).tryCreateFailedReplicas" "latency"=6800
I0117 17:37:56.416100       1 azure_disk_utils.go:1048]  "msg"="Getting AzVolumeAttachment list for node (capz-wto2-qsbsf)" 
I0117 17:37:56.416555       1 azure_disk_utils.go:1068]  "msg"="Label selector is: disk.csi.azure.com/node-name=capz-wto2-qsbsf." 
I0117 17:37:56.416577       1 shared_state.go:1745] Added node(capz-wto2-qsbsf) to availableAttachmentsMap with capacity: 8
I0117 17:37:56.416631       1 node.go:90] AzDiskControllerManager "msg"="Node is now available. Will requeue failed replica creation requests." "controller"="node" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "namespace"="azure-disk-csi" "partition"="csi-azuredisk-controller"
I0117 17:37:56.416705       1 shared_state.go:1205]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="ad864628-968d-11ed-ae72-002248b69c13" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*SharedState).tryCreateFailedReplicas" "latency"=7100
I0117 17:37:56.415971       1 azure_disk_utils.go:1048]  "msg"="Getting AzVolumeAttachment list for node (capz-wto2-brk9b)" 
I0117 17:37:56.416819       1 azure_disk_utils.go:1068]  "msg"="Label selector is: disk.csi.azure.com/node-name=capz-wto2-brk9b." 
I0117 17:37:56.416916       1 shared_state.go:1745] Added node(capz-wto2-brk9b) to availableAttachmentsMap with capacity: 8
I0117 17:37:56.416955       1 node.go:90] AzDiskControllerManager "msg"="Node is now available. Will requeue failed replica creation requests." "controller"="node" "disk.csi.azure.com/node-name"="capz-wto2-brk9b" "namespace"="azure-disk-csi" "partition"="csi-azuredisk-controller"
I0117 17:37:56.417018       1 shared_state.go:1205]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="ad8651de-968d-11ed-ae72-002248b69c13" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*SharedState).tryCreateFailedReplicas" "latency"=6600
I0117 17:37:56.462592       1 shared_state.go:420]  "msg"="Adding pod calico-typha-556bd6bcbd-m5fv4 to shared map with keyName calico-system/calico-typha-556bd6bcbd-m5fv4." 
I0117 17:37:56.463224       1 shared_state.go:426]  "msg"="Pod spec of pod calico-typha-556bd6bcbd-m5fv4 is: {Volumes:[{Name:tigera-ca-bundle VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:&ConfigMapVolumeSource{LocalObjectReference:LocalObjectReference{Name:tigera-ca-bundle,},Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,} VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:typha-certs VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:&SecretVolumeSource{SecretName:typha-certs,Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,} NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:kube-api-access-cxq9g VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:&ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,} PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:calico-typha Image:docker.io/calico/typha:v3.25.0 Command:[] Args:[] WorkingDir: Ports:[{Name:calico-typha HostPort:5473 ContainerPort:5473 Protocol:TCP HostIP:}] EnvFrom:[] Env:[{Name:TYPHA_LOGSEVERITYSCREEN Value:info ValueFrom:nil} {Name:TYPHA_LOGFILEPATH Value:none ValueFrom:nil} {Name:TYPHA_LOGSEVERITYSYS Value:none ValueFrom:nil} {Name:TYPHA_CONNECTIONREBALANCINGMODE Value:kubernetes ValueFrom:nil} {Name:TYPHA_DATASTORETYPE Value:kubernetes ValueFrom:nil} {Name:TYPHA_HEALTHENABLED Value:true ValueFrom:nil} {Name:TYPHA_HEALTHPORT Value:9098 ValueFrom:nil} {Name:TYPHA_K8SNAMESPACE Value:calico-system ValueFrom:nil} {Name:TYPHA_CAFILE Value:/etc/pki/tls/certs/tigera-ca-bundle.crt ValueFrom:nil} {Name:TYPHA_SERVERCERTFILE Value:/typha-certs/tls.crt ValueFrom:nil} {Name:TYPHA_SERVERKEYFILE Value:/typha-certs/tls.key ValueFrom:nil} {Name:TYPHA_FIPSMODEENABLED Value:false ValueFrom:nil} {Name:TYPHA_SHUTDOWNTIMEOUTSECS Value:300 ValueFrom:nil} {Name:TYPHA_CLIENTCN Value:typha-client ValueFrom:nil} {Name:KUBERNETES_SERVICE_HOST Value:10.96.0.1 ValueFrom:nil} {Name:KUBERNETES_SERVICE_PORT Value:443 ValueFrom:nil}] Resources:{Limits:map[] Requests:map[] Claims:[]} VolumeMounts:[{Name:tigera-ca-bundle ReadOnly:true MountPath:/etc/pki/tls/certs/ SubPath: MountPropagation:<nil> SubPathExpr:} {Name:typha-certs ReadOnly:true MountPath:/typha-certs SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-cxq9g ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/liveness,Port:{0 9098 },Host:localhost,Scheme:HTTP,HTTPHeaders:[]HTTPHeader{},},TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:0,TimeoutSeconds:10,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:3,TerminationGracePeriodSeconds:nil,} ReadinessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/readiness,Port:{0 9098 },Host:localhost,Scheme:HTTP,HTTPHeaders:[]HTTPHeader{},},TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:0,TimeoutSeconds:10,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:3,TerminationGracePeriodSeconds:nil,} StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy:Always TerminationGracePeriodSeconds:0xc0015c6878 ActiveDeadlineSeconds:<nil> DNSPolicy:ClusterFirst NodeSelector:map[kubernetes.io/os:linux] ServiceAccountName:calico-typha DeprecatedServiceAccount:calico-typha AutomountServiceAccountToken:<nil> NodeName:capz-wto256-control-plane-b5m52 HostNetwork:true HostPID:false HostIPC:false ShareProcessNamespace:<nil> SecurityContext:&PodSecurityContext{SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,SupplementalGroups:[],FSGroup:nil,RunAsGroup:nil,Sysctls:[]Sysctl{},WindowsOptions:nil,FSGroupChangePolicy:nil,SeccompProfile:nil,} ImagePullSecrets:[] Hostname: Subdomain: Affinity:&Affinity{NodeAffinity:nil,PodAffinity:nil,PodAntiAffinity:&PodAntiAffinity{RequiredDuringSchedulingIgnoredDuringExecution:[]PodAffinityTerm{},PreferredDuringSchedulingIgnoredDuringExecution:[]WeightedPodAffinityTerm{WeightedPodAffinityTerm{Weight:1,PodAffinityTerm:PodAffinityTerm{LabelSelector:&v1.LabelSelector{MatchLabels:map[string]string{},MatchExpressions:[]LabelSelectorRequirement{LabelSelectorRequirement{Key:k8s-app,Operator:In,Values:[calico-typha],},},},Namespaces:[],TopologyKey:topology.kubernetes.io/zone,NamespaceSelector:nil,},},},},} SchedulerName:default-scheduler Tolerations:[{Key:CriticalAddonsOnly Operator:Exists Value: Effect: TolerationSeconds:<nil>} {Key: Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>} {Key: Operator:Exists Value: Effect:NoExecute TolerationSeconds:<nil>}] HostAliases:[] PriorityClassName:system-cluster-critical Priority:0xc0015c6880 DNSConfig:nil ReadinessGates:[] RuntimeClassName:<nil> EnableServiceLinks:0xc0015c6884 PreemptionPolicy:0xc0016131e0 Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN:<nil> OS:nil HostUsers:<nil> SchedulingGates:[] ResourceClaims:[]}. With volumes: [{Name:tigera-ca-bundle VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:&ConfigMapVolumeSource{LocalObjectReference:LocalObjectReference{Name:tigera-ca-bundle,},Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,} VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:typha-certs VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:&SecretVolumeSource{SecretName:typha-certs,Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,} NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:kube-api-access-cxq9g VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:&ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,} PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}]" 
I0117 17:37:56.463356       1 shared_state.go:464]  "msg"="Pod calico-typha-556bd6bcbd-m5fv4: Skipping Volume {tigera-ca-bundle {nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil &ConfigMapVolumeSource{LocalObjectReference:LocalObjectReference{Name:tigera-ca-bundle,},Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,} nil nil nil nil nil nil nil nil nil nil}}. No persistent volume exists." 
I0117 17:37:56.463422       1 shared_state.go:464]  "msg"="Pod calico-typha-556bd6bcbd-m5fv4: Skipping Volume {typha-certs {nil nil nil nil nil &SecretVolumeSource{SecretName:typha-certs,Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}}. No persistent volume exists." 
I0117 17:37:56.463493       1 shared_state.go:464]  "msg"="Pod calico-typha-556bd6bcbd-m5fv4: Skipping Volume {kube-api-access-cxq9g {nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil &ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,} nil nil nil nil nil}}. No persistent volume exists." 
... skipping 245 lines ...
I0117 17:37:58.465297       1 shared_state.go:490]  "msg"="Storing pod calico-node-v5rh5 and claim [] to podToClaimsMap map." 
I0117 17:37:58.465355       1 pod.go:91]  "msg"="Creating replicas for pod calico-system/calico-node-v5rh5." "disk.csi.azure.com/request-id"="aebedfad-968d-11ed-ae72-002248b69c13" "disk.csi/azure.com/pod-name"="calico-system/calico-node-v5rh5"
I0117 17:37:58.465461       1 shared_state.go:314]  "msg"="Getting requested volumes for pod (calico-system/calico-node-v5rh5)." "disk.csi.azure.com/request-id"="aebedfad-968d-11ed-ae72-002248b69c13" "disk.csi/azure.com/pod-name"="calico-system/calico-node-v5rh5"
I0117 17:37:58.465497       1 pod.go:99]  "msg"="Pod calico-system/calico-node-v5rh5 has 0 volumes. Volumes: []" "disk.csi.azure.com/request-id"="aebedfad-968d-11ed-ae72-002248b69c13" "disk.csi/azure.com/pod-name"="calico-system/calico-node-v5rh5"
I0117 17:37:58.465525       1 pod.go:89]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="aebedfad-968d-11ed-ae72-002248b69c13" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcilePod).createReplicas" "disk.csi/azure.com/pod-name"="calico-system/calico-node-v5rh5" "latency"=183401
I0117 17:37:58.562320       1 shared_state.go:420]  "msg"="Adding pod csi-azuredisk-controller-7f6f8d54b5-t5fv2 to shared map with keyName kube-system/csi-azuredisk-controller-7f6f8d54b5-t5fv2." 
I0117 17:37:58.563053       1 shared_state.go:426]  "msg"="Pod spec of pod csi-azuredisk-controller-7f6f8d54b5-t5fv2 is: {Volumes:[{Name:socket-dir VolumeSource:{HostPath:nil EmptyDir:&EmptyDirVolumeSource{Medium:,SizeLimit:<nil>,} GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:azure-cred VolumeSource:{HostPath:&HostPathVolumeSource{Path:/etc/kubernetes/,Type:*DirectoryOrCreate,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:csi-azuredisk-controller-config VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:&ConfigMapVolumeSource{LocalObjectReference:LocalObjectReference{Name:csi-azuredisk-controller-config,},Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,} VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:kube-api-access-2ppxh VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:&ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,} PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:csi-provisioner Image:mcr.microsoft.com/oss/kubernetes-csi/csi-provisioner:v3.2.0 Command:[] Args:[--feature-gates=Topology=true --csi-address=$(ADDRESS) --v=2 --timeout=30s --leader-election --leader-election-namespace=kube-system --worker-threads=100 --extra-create-metadata=true --strict-topology=true --kube-api-qps=50 --kube-api-burst=100] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:ADDRESS Value:/csi/csi.sock ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:524288000 scale:0} d:{Dec:<nil>} s:500Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}] Claims:[]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-2ppxh ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:csi-attacher Image:mcr.microsoft.com/oss/kubernetes-csi/csi-attacher:v3.5.0 Command:[] Args:[-v=2 -csi-address=$(ADDRESS) -timeout=600s -leader-election --leader-election-namespace=kube-system -worker-threads=500 -kube-api-qps=50 -kube-api-burst=100] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:ADDRESS Value:/csi/csi.sock ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:524288000 scale:0} d:{Dec:<nil>} s:500Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}] Claims:[]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-2ppxh ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:csi-snapshotter Image:mcr.microsoft.com/oss/kubernetes-csi/csi-snapshotter:v5.0.1 Command:[] Args:[-csi-address=$(ADDRESS) -leader-election --leader-election-namespace=kube-system -v=2] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:ADDRESS Value:/csi/csi.sock ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:104857600 scale:0} d:{Dec:<nil>} s:100Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}] Claims:[]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-2ppxh ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:csi-resizer Image:mcr.microsoft.com/oss/kubernetes-csi/csi-resizer:v1.5.0 Command:[] Args:[-csi-address=$(ADDRESS) -v=2 -leader-election --leader-election-namespace=kube-system -handle-volume-inuse-error=false -feature-gates=RecoverVolumeExpansionFailure=true -timeout=240s] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:ADDRESS Value:/csi/csi.sock ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:524288000 scale:0} d:{Dec:<nil>} s:500Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}] Claims:[]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-2ppxh ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:liveness-probe Image:mcr.microsoft.com/oss/kubernetes-csi/livenessprobe:v2.7.0 Command:[] Args:[--csi-address=/csi/csi.sock --probe-timeout=3s --health-port=29602 --v=2] WorkingDir: Ports:[] EnvFrom:[] Env:[] Resources:{Limits:map[memory:{i:{value:104857600 scale:0} d:{Dec:<nil>} s:100Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}] Claims:[]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-2ppxh ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:azuredisk Image:capzci.azurecr.io/azuredisk-csi:latest-v2-9ef068a8cb36a997d4ea04b90c05c6f92a488a19 Command:[] Args:[--v=6 --config=/etc/csi-azuredisk-controller/config.yaml] WorkingDir: Ports:[{Name:healthz HostPort:29602 ContainerPort:29602 Protocol:TCP HostIP:} {Name:metrics HostPort:29604 ContainerPort:29604 Protocol:TCP HostIP:}] EnvFrom:[] Env:[{Name:AZURE_CREDENTIAL_FILE Value: ValueFrom:&EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:&ConfigMapKeySelector{LocalObjectReference:LocalObjectReference{Name:azure-cred-file,},Key:path,Optional:*true,},SecretKeyRef:nil,}} {Name:CSI_ENDPOINT Value:unix:///csi/csi.sock ValueFrom:nil} {Name:AZURE_GO_SDK_LOG_LEVEL Value: ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:524288000 scale:0} d:{Dec:<nil>} s:500Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}] Claims:[]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:azure-cred ReadOnly:false MountPath:/etc/kubernetes/ SubPath: MountPropagation:<nil> SubPathExpr:} {Name:csi-azuredisk-controller-config ReadOnly:false MountPath:/etc/csi-azuredisk-controller SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-2ppxh ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/healthz,Port:{1 0 healthz},Host:,Scheme:HTTP,HTTPHeaders:[]HTTPHeader{},},TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:30,TimeoutSeconds:10,PeriodSeconds:30,SuccessThreshold:1,FailureThreshold:5,TerminationGracePeriodSeconds:nil,} ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:Always SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy:Always TerminationGracePeriodSeconds:0xc000c40c98 ActiveDeadlineSeconds:<nil> DNSPolicy:ClusterFirst NodeSelector:map[kubernetes.io/os:linux] ServiceAccountName:csi-azuredisk-controller-sa DeprecatedServiceAccount:csi-azuredisk-controller-sa AutomountServiceAccountToken:<nil> NodeName:capz-wto256-control-plane-b5m52 HostNetwork:true HostPID:false HostIPC:false ShareProcessNamespace:<nil> SecurityContext:&PodSecurityContext{SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,SupplementalGroups:[],FSGroup:nil,RunAsGroup:nil,Sysctls:[]Sysctl{},WindowsOptions:nil,FSGroupChangePolicy:nil,SeccompProfile:nil,} ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName:default-scheduler Tolerations:[{Key:node-role.kubernetes.io/master Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>} {Key:node-role.kubernetes.io/controlplane Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>} {Key:node-role.kubernetes.io/control-plane Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>} {Key:node.kubernetes.io/not-ready Operator:Exists Value: Effect:NoExecute TolerationSeconds:0xc000c40ca0} {Key:node.kubernetes.io/unreachable Operator:Exists Value: Effect:NoExecute TolerationSeconds:0xc000c40ca8}] HostAliases:[] PriorityClassName:system-cluster-critical Priority:0xc000c40cb0 DNSConfig:nil ReadinessGates:[] RuntimeClassName:<nil> EnableServiceLinks:0xc000c40cb4 PreemptionPolicy:0xc000fb6340 Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN:<nil> OS:nil HostUsers:<nil> SchedulingGates:[] ResourceClaims:[]}. With volumes: [{Name:socket-dir VolumeSource:{HostPath:nil EmptyDir:&EmptyDirVolumeSource{Medium:,SizeLimit:<nil>,} GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:azure-cred VolumeSource:{HostPath:&HostPathVolumeSource{Path:/etc/kubernetes/,Type:*DirectoryOrCreate,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:csi-azuredisk-controller-config VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:&ConfigMapVolumeSource{LocalObjectReference:LocalObjectReference{Name:csi-azuredisk-controller-config,},Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,} VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:kube-api-access-2ppxh VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:&ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,} PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}]" 
I0117 17:37:58.563118       1 shared_state.go:464]  "msg"="Pod csi-azuredisk-controller-7f6f8d54b5-t5fv2: Skipping Volume {socket-dir {nil &EmptyDirVolumeSource{Medium:,SizeLimit:<nil>,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}}. No persistent volume exists." 
I0117 17:37:58.563145       1 shared_state.go:464]  "msg"="Pod csi-azuredisk-controller-7f6f8d54b5-t5fv2: Skipping Volume {azure-cred {&HostPathVolumeSource{Path:/etc/kubernetes/,Type:*DirectoryOrCreate,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}}. No persistent volume exists." 
I0117 17:37:58.563192       1 shared_state.go:464]  "msg"="Pod csi-azuredisk-controller-7f6f8d54b5-t5fv2: Skipping Volume {csi-azuredisk-controller-config {nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil &ConfigMapVolumeSource{LocalObjectReference:LocalObjectReference{Name:csi-azuredisk-controller-config,},Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,} nil nil nil nil nil nil nil nil nil nil}}. No persistent volume exists." 
I0117 17:37:58.563231       1 shared_state.go:464]  "msg"="Pod csi-azuredisk-controller-7f6f8d54b5-t5fv2: Skipping Volume {kube-api-access-2ppxh {nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil &ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,} nil nil nil nil nil}}. No persistent volume exists." 
I0117 17:37:58.563258       1 shared_state.go:490]  "msg"="Storing pod csi-azuredisk-controller-7f6f8d54b5-t5fv2 and claim [] to podToClaimsMap map." 
I0117 17:37:58.563287       1 pod.go:91]  "msg"="Creating replicas for pod kube-system/csi-azuredisk-controller-7f6f8d54b5-t5fv2." "disk.csi.azure.com/request-id"="aecdd1c9-968d-11ed-ae72-002248b69c13" "disk.csi/azure.com/pod-name"="kube-system/csi-azuredisk-controller-7f6f8d54b5-t5fv2"
... skipping 578 lines ...
I0117 17:42:06.246274       1 utils.go:78] GRPC call: /csi.v1.Controller/CreateVolume
I0117 17:42:06.246304       1 utils.go:79] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.disk.csi.azure.com/zone":""}}],"requisite":[{"segments":{"topology.disk.csi.azure.com/zone":""}}]},"capacity_range":{"required_bytes":10737418240},"name":"pvc-f412592d-476e-45c8-ba51-9201946b22da","parameters":{"csi.storage.k8s.io/pv/name":"pvc-f412592d-476e-45c8-ba51-9201946b22da","csi.storage.k8s.io/pvc/name":"pvc-r2p9b","csi.storage.k8s.io/pvc/namespace":"azuredisk-1410","skuname":"StandardSSD_LRS"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]}
I0117 17:42:06.247323       1 crdprovisioner.go:308]  "msg"="Creating AzVolume CRI" "csi.storage.k8s.io/pv/name"="pvc-f412592d-476e-45c8-ba51-9201946b22da" "disk.csi.azure.com/request-id"="426f65df-968e-11ed-ae72-002248b69c13" "disk.csi.azure.com/volume-name"="pvc-f412592d-476e-45c8-ba51-9201946b22da"
I0117 17:42:06.263492       1 round_trippers.go:553] POST https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes 201 Created in 15 milliseconds
I0117 17:42:06.264003       1 crdprovisioner.go:316]  "msg"="Successfully created AzVolume CRI" "csi.storage.k8s.io/pv/name"="pvc-f412592d-476e-45c8-ba51-9201946b22da" "disk.csi.azure.com/request-id"="426f65df-968e-11ed-ae72-002248b69c13" "disk.csi.azure.com/volume-name"="pvc-f412592d-476e-45c8-ba51-9201946b22da"
I0117 17:42:06.264127       1 conditionwatcher.go:99] Adding a condition function for azvolume (pvc-f412592d-476e-45c8-ba51-9201946b22da)
I0117 17:42:06.264464       1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0117 17:42:06.275336       1 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-f412592d-476e-45c8-ba51-9201946b22da/status 200 OK in 9 milliseconds
I0117 17:42:06.276052       1 azure_disk_utils.go:1103]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate" "csi.storage.k8s.io/pv/name"="pvc-f412592d-476e-45c8-ba51-9201946b22da" "disk.csi.azure.com/request-id"="426f65df-968e-11ed-ae72-002248b69c13" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-f412592d-476e-45c8-ba51-9201946b22da" "latency"=10773642 "number_of_net_retry"=0 "number_of_retry"=0
I0117 17:42:06.276173       1 azvolume.go:159]  "msg"="Creating Volume..." "csi.storage.k8s.io/pv/name"="pvc-f412592d-476e-45c8-ba51-9201946b22da" "disk.csi.azure.com/request-id"="426f65df-968e-11ed-ae72-002248b69c13" "disk.csi.azure.com/volume-name"="pvc-f412592d-476e-45c8-ba51-9201946b22da"
I0117 17:42:06.277629       1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0117 17:42:06.306674       1 util.go:124] Send.sendRequest got response with ContentLength 247, StatusCode 404 and responseBody length 247
I0117 17:42:06.306708       1 azure_diskclient.go:139] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-wto256/providers/Microsoft.Compute/disks/pvc-f412592d-476e-45c8-ba51-9201946b22da, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-f412592d-476e-45c8-ba51-9201946b22da' under resource group 'capz-wto256' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I0117 17:42:06.306971       1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=0.029967718 request="disks_get" resource_group="capz-wto256" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code="ResourceNotFound"
I0117 17:42:06.307002       1 cloudprovisioner.go:246] begin to create disk(pvc-f412592d-476e-45c8-ba51-9201946b22da) account type(StandardSSD_LRS) rg(capz-wto256) location() size(10) selectedAvailabilityZone() maxShares(0)
I0117 17:42:06.350407       1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 10850
I0117 17:42:06.350910       1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=0.043777172 request="disks_get" resource_group="capz-wto256" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0117 17:42:06.351007       1 azure_managedDiskController.go:94] azureDisk - creating new managed Name:pvc-f412592d-476e-45c8-ba51-9201946b22da StorageAccountType:StandardSSD_LRS Size:10
I0117 17:42:08.069690       1 round_trippers.go:553] GET https://10.96.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 3 milliseconds
... skipping 2 lines ...
I0117 17:42:08.758865       1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=2.407574854 request="disks_create_or_update" resource_group="capz-wto256" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0117 17:42:08.781955       1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1187
I0117 17:42:08.782322       1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=0.023375592 request="disks_get" resource_group="capz-wto256" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0117 17:42:08.782460       1 azure_managedDiskController.go:280] azureDisk - created new MD Name:pvc-f412592d-476e-45c8-ba51-9201946b22da StorageAccountType:StandardSSD_LRS Size:10
I0117 17:42:08.782592       1 cloudprovisioner.go:320]  "msg"="create disk(pvc-f412592d-476e-45c8-ba51-9201946b22da) account type(StandardSSD_LRS) rg(capz-wto256) location() size(10) tags(map[kubernetes.io-created-for-pv-name:pvc-f412592d-476e-45c8-ba51-9201946b22da kubernetes.io-created-for-pvc-name:pvc-r2p9b kubernetes.io-created-for-pvc-namespace:azuredisk-1410]) successfully" "csi.storage.k8s.io/pv/name"="pvc-f412592d-476e-45c8-ba51-9201946b22da" "disk.csi.azure.com/request-id"="426f65df-968e-11ed-ae72-002248b69c13" "disk.csi.azure.com/volume-name"="pvc-f412592d-476e-45c8-ba51-9201946b22da"
I0117 17:42:08.782635       1 cloudprovisioner.go:142]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-f412592d-476e-45c8-ba51-9201946b22da" "disk.csi.azure.com/request-id"="426f65df-968e-11ed-ae72-002248b69c13" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-f412592d-476e-45c8-ba51-9201946b22da" "latency"=2505897140
I0117 17:42:08.799561       1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 17:42:08.799626       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).CreateVolume" "csi.storage.k8s.io/pv/name"="pvc-f412592d-476e-45c8-ba51-9201946b22da" "disk.csi.azure.com/request-id"="426f65df-968e-11ed-ae72-002248b69c13" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-f412592d-476e-45c8-ba51-9201946b22da" "latency"=2535268255
I0117 17:42:08.799663       1 crdprovisioner.go:228]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-f412592d-476e-45c8-ba51-9201946b22da" "disk.csi.azure.com/request-id"="426f65df-968e-11ed-ae72-002248b69c13" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-f412592d-476e-45c8-ba51-9201946b22da" "latency"=2552379522
I0117 17:42:08.799703       1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=2.552717024 request="azuredisk_csi_driver_controller_create_volume" resource_group="capz-wto256" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-wto256/providers/Microsoft.Compute/disks/pvc-f412592d-476e-45c8-ba51-9201946b22da" result_code="succeeded"
I0117 17:42:08.799716       1 utils.go:85] GRPC response: {"volume":{"accessible_topology":[{"segments":{"topology.disk.csi.azure.com/zone":""}}],"capacity_bytes":10737418240,"content_source":{"Type":{"Volume":{}}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-f412592d-476e-45c8-ba51-9201946b22da","csi.storage.k8s.io/pvc/name":"pvc-r2p9b","csi.storage.k8s.io/pvc/namespace":"azuredisk-1410","requestedsizegib":"10","skuname":"StandardSSD_LRS"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-wto256/providers/Microsoft.Compute/disks/pvc-f412592d-476e-45c8-ba51-9201946b22da"}}
I0117 17:42:08.800082       1 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-f412592d-476e-45c8-ba51-9201946b22da/status 200 OK in 17 milliseconds
I0117 17:42:08.800565       1 azure_disk_utils.go:1103]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "csi.storage.k8s.io/pv/name"="pvc-f412592d-476e-45c8-ba51-9201946b22da" "disk.csi.azure.com/request-id"="426f65df-968e-11ed-ae72-002248b69c13" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-f412592d-476e-45c8-ba51-9201946b22da" "latency"=17880571 "number_of_net_retry"=0 "number_of_retry"=0
... skipping 13 lines ...
I0117 17:42:10.765110       1 azure_disk_utils.go:1068]  "msg"="Label selector is: disk.csi.azure.com/node-name=capz-wto2-qsbsf,disk.csi.azure.com/volume-name!=pvc-f412592d-476e-45c8-ba51-9201946b22da." "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="4520bc1c-968e-11ed-ae72-002248b69c13" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-f412592d-476e-45c8-ba51-9201946b22da"
I0117 17:42:10.773165       1 round_trippers.go:553] POST https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments 201 Created in 7 milliseconds
I0117 17:42:10.775725       1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-f412592d-476e-45c8-ba51-9201946b22da-capz-wto2-qsbsf-attachment)
I0117 17:42:10.789548       1 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-f412592d-476e-45c8-ba51-9201946b22da-capz-wto2-qsbsf-attachment/status 200 OK in 11 milliseconds
I0117 17:42:10.790012       1 azure_disk_utils.go:1103]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="4520bc1c-968e-11ed-ae72-002248b69c13" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-f412592d-476e-45c8-ba51-9201946b22da" "latency"=12250047 "number_of_net_retry"=0 "number_of_retry"=0
I0117 17:42:10.790379       1 attach_detach.go:192]  "msg"="Attaching volume" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="4520bc1c-968e-11ed-ae72-002248b69c13" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-f412592d-476e-45c8-ba51-9201946b22da"
I0117 17:42:10.791300       1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0117 17:42:10.816047       1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1187
I0117 17:42:10.816421       1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=0.024599996 request="disks_get" resource_group="capz-wto256" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0117 17:42:10.911865       1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 5887
I0117 17:42:10.913022       1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=0.095978872 request="vm_get" resource_group="capz-wto256" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0117 17:42:10.913525       1 cloudprovisioner.go:405]  "msg"="GetDiskLun returned: -1. Initiating attaching volume \"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-wto256/providers/Microsoft.Compute/disks/pvc-f412592d-476e-45c8-ba51-9201946b22da\" to node \"capz-wto2-qsbsf\"." "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="4520bc1c-968e-11ed-ae72-002248b69c13" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-f412592d-476e-45c8-ba51-9201946b22da"
I0117 17:42:10.913699       1 cloudprovisioner.go:424]  "msg"="Trying to attach volume \"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-wto256/providers/Microsoft.Compute/disks/pvc-f412592d-476e-45c8-ba51-9201946b22da\" to node \"capz-wto2-qsbsf\"." "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="4520bc1c-968e-11ed-ae72-002248b69c13" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-f412592d-476e-45c8-ba51-9201946b22da"
I0117 17:42:11.914520       1 batch.go:224] "cloud-provider-azure: Delayed processing of batch due to start delay" type="batch" operation="attach_disk" key="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e|capz-wto256|capz-wto2-qsbsf" delay="1s"
I0117 17:42:11.915352       1 azure_controller_standard.go:103] azureDisk - update(capz-wto256): vm(capz-wto2-qsbsf) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-wto256/providers/Microsoft.Compute/disks/pvc-f412592d-476e-45c8-ba51-9201946b22da:AttachDiskOptions{diskName: "pvc-f412592d-476e-45c8-ba51-9201946b22da", lun: 0}])
I0117 17:42:11.932413       1 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-f412592d-476e-45c8-ba51-9201946b22da-capz-wto2-qsbsf-attachment/status 200 OK in 17 milliseconds
I0117 17:42:11.933235       1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 17:42:11.933308       1 azure_disk_utils.go:1103]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3.4" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="4520bc1c-968e-11ed-ae72-002248b69c13" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-f412592d-476e-45c8-ba51-9201946b22da" "latency"=18517272 "number_of_net_retry"=0 "number_of_retry"=0
I0117 17:42:11.933676       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="4520bc1c-968e-11ed-ae72-002248b69c13" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-f412592d-476e-45c8-ba51-9201946b22da" "latency"=1155443578
I0117 17:42:11.933764       1 crdprovisioner.go:727]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="4520bc1c-968e-11ed-ae72-002248b69c13" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLun" "disk.csi.azure.com/volume-name"="pvc-f412592d-476e-45c8-ba51-9201946b22da" "latency"=1158010788
I0117 17:42:11.933824       1 crdprovisioner.go:486]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="4520bc1c-968e-11ed-ae72-002248b69c13" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).PublishVolume" "disk.csi.azure.com/volume-name"="pvc-f412592d-476e-45c8-ba51-9201946b22da" "latency"=1168874730
I0117 17:42:11.933944       1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=1.169300631 request="azuredisk_csi_driver_controller_publish_volume" resource_group="capz-wto256" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-wto256/providers/Microsoft.Compute/disks/pvc-f412592d-476e-45c8-ba51-9201946b22da" node="capz-wto2-qsbsf" result_code="succeeded"
I0117 17:42:11.933963       1 utils.go:85] GRPC response: {"publish_context":{"LUN":"0"}}
... skipping 36 lines ...
I0117 17:42:22.285743       1 azure_disk_utils.go:1165]  "msg"="Skip update. No update needed." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).updateVolumeAttachmentWithResult" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="4520bc1c-968e-11ed-ae72-002248b69c13" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-f412592d-476e-45c8-ba51-9201946b22da"
I0117 17:42:22.285791       1 azure_disk_utils.go:1103]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).updateVolumeAttachmentWithResult" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="4520bc1c-968e-11ed-ae72-002248b69c13" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-f412592d-476e-45c8-ba51-9201946b22da" "latency"=100400 "number_of_net_retry"=0 "number_of_retry"=0
I0117 17:42:22.298668       1 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-f412592d-476e-45c8-ba51-9201946b22da-capz-wto2-qsbsf-attachment/status 200 OK in 12 milliseconds
I0117 17:42:22.299053       1 azure_disk_utils.go:1103]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3.4" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="4520bc1c-968e-11ed-ae72-002248b69c13" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-f412592d-476e-45c8-ba51-9201946b22da" "latency"=13220051 "number_of_net_retry"=0 "number_of_retry"=0
I0117 17:42:22.299851       1 attach_detach.go:198]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="4520bc1c-968e-11ed-ae72-002248b69c13" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3" "disk.csi.azure.com/volume-name"="pvc-f412592d-476e-45c8-ba51-9201946b22da" "latency"=11508148726
I0117 17:42:22.300031       1 workflow.go:155]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="4520bc1c-968e-11ed-ae72-002248b69c13" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach" "disk.csi.azure.com/volume-name"="pvc-f412592d-476e-45c8-ba51-9201946b22da" "latency"=11522301781
I0117 17:42:22.299427       1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 17:42:22.299529       1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 17:42:22.300383       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="45d5fc64-968e-11ed-ae72-002248b69c13" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-f412592d-476e-45c8-ba51-9201946b22da" "latency"=10347354228
I0117 17:42:22.300516       1 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).ControllerPublishVolume.func2" "disk.csi.azure.com/request-id"="45d5fc64-968e-11ed-ae72-002248b69c13" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=10347731929
I0117 17:42:22.300679       1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=10.348858533 request="azuredisk_csi_driver_attach_volume_latency" resource_group="capz-wto256" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-wto256/providers/Microsoft.Compute/disks/pvc-f412592d-476e-45c8-ba51-9201946b22da" node="capz-wto2-qsbsf" result_code="succeeded"
I0117 17:42:22.300714       1 controllerserver_v2.go:264]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="45d5fc64-968e-11ed-ae72-002248b69c13" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).ControllerPublishVolume.func2" "latency"=10347946029
I0117 17:42:22.300387       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="45d325fc-968e-11ed-ae72-002248b69c13" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-f412592d-476e-45c8-ba51-9201946b22da" "latency"=10366062500
I0117 17:42:22.300829       1 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).ControllerPublishVolume.func2" "disk.csi.azure.com/request-id"="45d325fc-968e-11ed-ae72-002248b69c13" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=10366640602
... skipping 130 lines ...
I0117 17:43:23.343594       1 utils.go:79] GRPC request: {"node_id":"capz-wto2-qsbsf","volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-wto256/providers/Microsoft.Compute/disks/pvc-f412592d-476e-45c8-ba51-9201946b22da"}
I0117 17:43:23.344359       1 crdprovisioner.go:927]  "msg"="Requesting AzVolumeAttachment (pvc-f412592d-476e-45c8-ba51-9201946b22da-capz-wto2-qsbsf-attachment) detachment" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="4520bc1c-968e-11ed-ae72-002248b69c13" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-f412592d-476e-45c8-ba51-9201946b22da"
I0117 17:43:23.355048       1 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-f412592d-476e-45c8-ba51-9201946b22da-capz-wto2-qsbsf-attachment/status 200 OK in 10 milliseconds
I0117 17:43:23.355544       1 azure_disk_utils.go:1103]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).detachVolume" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="4520bc1c-968e-11ed-ae72-002248b69c13" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-f412592d-476e-45c8-ba51-9201946b22da" "latency"=11126681 "number_of_net_retry"=0 "number_of_retry"=0
I0117 17:43:23.355793       1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-f412592d-476e-45c8-ba51-9201946b22da-capz-wto2-qsbsf-attachment)
I0117 17:43:23.356770       1 replica.go:157]  "msg"="Garbage collection of AzVolumeAttachments for AzVolume (pvc-f412592d-476e-45c8-ba51-9201946b22da) scheduled in 5m0s." "disk.csi.azure.com/request-id"="70655e90-968e-11ed-ae72-002248b69c13" "disk.csi.azure.com/volume-name"="pvc-f412592d-476e-45c8-ba51-9201946b22da"
I0117 17:43:23.358573       1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0117 17:43:23.371024       1 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-f412592d-476e-45c8-ba51-9201946b22da-capz-wto2-qsbsf-attachment/status 200 OK in 12 milliseconds
I0117 17:43:23.372929       1 azure_disk_utils.go:1103]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="4520bc1c-968e-11ed-ae72-002248b69c13" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-f412592d-476e-45c8-ba51-9201946b22da" "latency"=15017574 "number_of_net_retry"=0 "number_of_retry"=0
I0117 17:43:23.373174       1 attach_detach.go:340]  "msg"="Detaching volume" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="4520bc1c-968e-11ed-ae72-002248b69c13" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-f412592d-476e-45c8-ba51-9201946b22da"
I0117 17:43:23.373447       1 cloudprovisioner.go:481]  "msg"="Trying to detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-wto256/providers/Microsoft.Compute/disks/pvc-f412592d-476e-45c8-ba51-9201946b22da from node capz-wto2-qsbsf" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="4520bc1c-968e-11ed-ae72-002248b69c13" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-f412592d-476e-45c8-ba51-9201946b22da"
I0117 17:43:23.374885       1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0117 17:43:24.373949       1 batch.go:224] "cloud-provider-azure: Delayed processing of batch due to start delay" type="batch" operation="detach_disk" key="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e|capz-wto256|capz-wto2-qsbsf" delay="1s"
I0117 17:43:24.374032       1 azure_controller_common.go:461] azuredisk - trying to detach disks from node capz-wto2-qsbsf: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-wto256/providers/Microsoft.Compute/disks/pvc-f412592d-476e-45c8-ba51-9201946b22da:pvc-f412592d-476e-45c8-ba51-9201946b22da]
I0117 17:43:24.436353       1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 6844
I0117 17:43:24.437226       1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=0.063119291 request="vm_get" resource_group="capz-wto256" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0117 17:43:24.437257       1 azure_controller_standard.go:180] azureDisk - detach disk: name pvc-f412592d-476e-45c8-ba51-9201946b22da uri /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-wto256/providers/Microsoft.Compute/disks/pvc-f412592d-476e-45c8-ba51-9201946b22da
I0117 17:43:24.437265       1 azure_controller_standard.go:210] azureDisk - update(capz-wto256): vm(capz-wto2-qsbsf) - detach disk list(capz-wto2-qsbsf)%!(EXTRA map[string]string=map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-wto256/providers/Microsoft.Compute/disks/pvc-f412592d-476e-45c8-ba51-9201946b22da:pvc-f412592d-476e-45c8-ba51-9201946b22da])
I0117 17:43:24.618046       1 round_trippers.go:553] GET https://10.96.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 6 milliseconds
I0117 17:43:24.626186       1 round_trippers.go:553] PUT https://10.96.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 7 milliseconds
I0117 17:43:24.626402       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0117 17:43:25.510511       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 17:43:25.510515       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 17:43:25.510527       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 17:43:25.510792       1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0117 17:43:25.655463       1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync
I0117 17:43:26.632857       1 round_trippers.go:553] GET https://10.96.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 5 milliseconds
I0117 17:43:26.644125       1 round_trippers.go:553] PUT https://10.96.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 10 milliseconds
I0117 17:43:26.644463       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0117 17:43:28.650190       1 round_trippers.go:553] GET https://10.96.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 5 milliseconds
I0117 17:43:28.663325       1 round_trippers.go:553] PUT https://10.96.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 12 milliseconds
... skipping 24 lines ...
I0117 17:43:39.933113       1 azure_controller_common.go:434] azureDisk - detach disk(pvc-f412592d-476e-45c8-ba51-9201946b22da, /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-wto256/providers/Microsoft.Compute/disks/pvc-f412592d-476e-45c8-ba51-9201946b22da) succeeded
I0117 17:43:39.933197       1 cloudprovisioner.go:491]  "msg"="detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-wto256/providers/Microsoft.Compute/disks/pvc-f412592d-476e-45c8-ba51-9201946b22da from node capz-wto2-qsbsf successfully" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="4520bc1c-968e-11ed-ae72-002248b69c13" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-f412592d-476e-45c8-ba51-9201946b22da"
I0117 17:43:39.933302       1 cloudprovisioner.go:471]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="4520bc1c-968e-11ed-ae72-002248b69c13" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).UnpublishVolume" "disk.csi.azure.com/volume-name"="pvc-f412592d-476e-45c8-ba51-9201946b22da" "latency"=16559868109
I0117 17:43:39.957279       1 round_trippers.go:553] DELETE https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-f412592d-476e-45c8-ba51-9201946b22da-capz-wto2-qsbsf-attachment 200 OK in 23 milliseconds
I0117 17:43:39.957579       1 attach_detach.go:346]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="4520bc1c-968e-11ed-ae72-002248b69c13" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach.func3" "disk.csi.azure.com/volume-name"="pvc-f412592d-476e-45c8-ba51-9201946b22da" "latency"=16584172578
I0117 17:43:39.957645       1 workflow.go:155]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="4520bc1c-968e-11ed-ae72-002248b69c13" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach" "disk.csi.azure.com/volume-name"="pvc-f412592d-476e-45c8-ba51-9201946b22da" "latency"=16599753152
I0117 17:43:39.958212       1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0117 17:43:39.968496       1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 17:43:39.969819       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForDetach" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="4520bc1c-968e-11ed-ae72-002248b69c13" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-f412592d-476e-45c8-ba51-9201946b22da" "latency"=16613796284
I0117 17:43:39.970304       1 crdprovisioner.go:938]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).detachVolume" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="4520bc1c-968e-11ed-ae72-002248b69c13" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForDetach" "disk.csi.azure.com/volume-name"="pvc-f412592d-476e-45c8-ba51-9201946b22da" "latency"=16614530485
I0117 17:43:39.970758       1 crdprovisioner.go:836]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="70637946-968e-11ed-ae72-002248b69c13" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).UnpublishVolume" "disk.csi.azure.com/volume-name"="pvc-f412592d-476e-45c8-ba51-9201946b22da" "latency"=16626420067
I0117 17:43:39.971065       1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=16.626670866 request="azuredisk_csi_driver_controller_unpublish_volume" resource_group="capz-wto256" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-wto256/providers/Microsoft.Compute/disks/pvc-f412592d-476e-45c8-ba51-9201946b22da" node="capz-wto2-qsbsf" result_code="succeeded"
I0117 17:43:39.971096       1 utils.go:85] GRPC response: {}
I0117 17:43:39.970696       1 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-f412592d-476e-45c8-ba51-9201946b22da-capz-wto2-qsbsf-attachment 200 OK in 12 milliseconds
... skipping 20 lines ...
I0117 17:43:46.049761       1 utils.go:79] GRPC request: {"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-wto256/providers/Microsoft.Compute/disks/pvc-f412592d-476e-45c8-ba51-9201946b22da"}
I0117 17:43:46.049858       1 controllerserver_v2.go:201] deleting disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-wto256/providers/Microsoft.Compute/disks/pvc-f412592d-476e-45c8-ba51-9201946b22da)
I0117 17:43:46.065153       1 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-f412592d-476e-45c8-ba51-9201946b22da/status 200 OK in 14 milliseconds
I0117 17:43:46.065767       1 azure_disk_utils.go:1103]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).DeleteVolume" "csi.storage.k8s.io/pv/name"="pvc-f412592d-476e-45c8-ba51-9201946b22da" "disk.csi.azure.com/request-id"="7dec1312-968e-11ed-ae72-002248b69c13" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-f412592d-476e-45c8-ba51-9201946b22da" "latency"=15752760 "number_of_net_retry"=0 "number_of_retry"=0
I0117 17:43:46.065781       1 conditionwatcher.go:99] Adding a condition function for azvolume (pvc-f412592d-476e-45c8-ba51-9201946b22da)
I0117 17:43:46.079435       1 round_trippers.go:553] DELETE https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-f412592d-476e-45c8-ba51-9201946b22da 200 OK in 13 milliseconds
I0117 17:43:46.080572       1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0117 17:43:46.093892       1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0117 17:43:46.093917       1 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-f412592d-476e-45c8-ba51-9201946b22da/status 200 OK in 12 milliseconds
I0117 17:43:46.094229       1 azure_disk_utils.go:1103]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerDelete" "csi.storage.k8s.io/pv/name"="pvc-f412592d-476e-45c8-ba51-9201946b22da" "disk.csi.azure.com/request-id"="426f65df-968e-11ed-ae72-002248b69c13" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-f412592d-476e-45c8-ba51-9201946b22da" "latency"=13007158 "number_of_net_retry"=0 "number_of_retry"=0
I0117 17:43:46.094252       1 azvolume.go:247]  "msg"="Deleting Volume..." "csi.storage.k8s.io/pv/name"="pvc-f412592d-476e-45c8-ba51-9201946b22da" "disk.csi.azure.com/request-id"="426f65df-968e-11ed-ae72-002248b69c13" "disk.csi.azure.com/volume-name"="pvc-f412592d-476e-45c8-ba51-9201946b22da"
I0117 17:43:46.094360       1 shared_state.go:1038]  "msg"="AzVolumeAttachment clean up requested by azvolume-controller for AzVolume (pvc-f412592d-476e-45c8-ba51-9201946b22da)" "csi.storage.k8s.io/pv/name"="pvc-f412592d-476e-45c8-ba51-9201946b22da" "disk.csi.azure.com/request-id"="426f65df-968e-11ed-ae72-002248b69c13" "disk.csi.azure.com/volume-name"="pvc-f412592d-476e-45c8-ba51-9201946b22da"
I0117 17:43:46.094417       1 azure_disk_utils.go:1039]  "msg"="Getting AzVolumeAttachment list for volume (pvc-f412592d-476e-45c8-ba51-9201946b22da)" "csi.storage.k8s.io/pv/name"="pvc-f412592d-476e-45c8-ba51-9201946b22da" "disk.csi.azure.com/request-id"="426f65df-968e-11ed-ae72-002248b69c13" "disk.csi.azure.com/volume-name"="pvc-f412592d-476e-45c8-ba51-9201946b22da"
I0117 17:43:46.094464       1 azure_disk_utils.go:1068]  "msg"="Label selector is: disk.csi.azure.com/volume-name=pvc-f412592d-476e-45c8-ba51-9201946b22da." "csi.storage.k8s.io/pv/name"="pvc-f412592d-476e-45c8-ba51-9201946b22da" "disk.csi.azure.com/request-id"="426f65df-968e-11ed-ae72-002248b69c13" "disk.csi.azure.com/volume-name"="pvc-f412592d-476e-45c8-ba51-9201946b22da"
... skipping 9 lines ...
I0117 17:43:50.821856       1 round_trippers.go:553] GET https://10.96.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 2 milliseconds
I0117 17:43:50.827684       1 round_trippers.go:553] PUT https://10.96.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 5 milliseconds
I0117 17:43:50.828183       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0117 17:43:51.370928       1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=5.248046431 request="disks_delete" resource_group="capz-wto256" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0117 17:43:51.370963       1 azure_managedDiskController.go:317] azureDisk - deleted a managed disk: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-wto256/providers/Microsoft.Compute/disks/pvc-f412592d-476e-45c8-ba51-9201946b22da
I0117 17:43:51.371022       1 cloudprovisioner.go:337]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-f412592d-476e-45c8-ba51-9201946b22da" "disk.csi.azure.com/request-id"="426f65df-968e-11ed-ae72-002248b69c13" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).DeleteVolume" "disk.csi.azure.com/volume-name"="pvc-f412592d-476e-45c8-ba51-9201946b22da" "latency"=5276446556
I0117 17:43:51.387286       1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 17:43:51.387950       1 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-f412592d-476e-45c8-ba51-9201946b22da 200 OK in 16 milliseconds
I0117 17:43:51.388437       1 azure_disk_utils.go:1103]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerDelete.func4" "csi.storage.k8s.io/pv/name"="pvc-f412592d-476e-45c8-ba51-9201946b22da" "disk.csi.azure.com/request-id"="426f65df-968e-11ed-ae72-002248b69c13" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-f412592d-476e-45c8-ba51-9201946b22da" "latency"=17085376 "number_of_net_retry"=0 "number_of_retry"=0
I0117 17:43:51.388484       1 azvolume.go:257]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-f412592d-476e-45c8-ba51-9201946b22da" "disk.csi.azure.com/request-id"="426f65df-968e-11ed-ae72-002248b69c13" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerDelete.func4" "disk.csi.azure.com/volume-name"="pvc-f412592d-476e-45c8-ba51-9201946b22da" "latency"=5294202434
I0117 17:43:51.388522       1 workflow.go:155]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-f412592d-476e-45c8-ba51-9201946b22da" "disk.csi.azure.com/request-id"="426f65df-968e-11ed-ae72-002248b69c13" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerDelete" "disk.csi.azure.com/volume-name"="pvc-f412592d-476e-45c8-ba51-9201946b22da" "latency"=5307357592
I0117 17:43:51.388557       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).DeleteVolume" "csi.storage.k8s.io/pv/name"="pvc-f412592d-476e-45c8-ba51-9201946b22da" "disk.csi.azure.com/request-id"="7dec1312-968e-11ed-ae72-002248b69c13" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-f412592d-476e-45c8-ba51-9201946b22da" "latency"=5307257191
I0117 17:43:51.388586       1 crdprovisioner.go:375]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-f412592d-476e-45c8-ba51-9201946b22da" "disk.csi.azure.com/request-id"="7dec1312-968e-11ed-ae72-002248b69c13" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).DeleteVolume" "disk.csi.azure.com/volume-name"="pvc-f412592d-476e-45c8-ba51-9201946b22da" "latency"=5338647820
... skipping 20 lines ...
I0117 17:43:56.522962       1 utils.go:78] GRPC call: /csi.v1.Controller/CreateVolume
I0117 17:43:56.524880       1 utils.go:79] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.disk.csi.azure.com/zone":""}}],"requisite":[{"segments":{"topology.disk.csi.azure.com/zone":""}}]},"capacity_range":{"required_bytes":10737418240},"name":"pvc-3c15cd13-4c71-427d-ab77-b4fc98315eef","parameters":{"csi.storage.k8s.io/pv/name":"pvc-3c15cd13-4c71-427d-ab77-b4fc98315eef","csi.storage.k8s.io/pvc/name":"test-volume-1-azuredisk-volume-tester-h8jrt-0","csi.storage.k8s.io/pvc/namespace":"azuredisk-3523","skuname":"StandardSSD_LRS"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]}
I0117 17:43:56.525555       1 crdprovisioner.go:308]  "msg"="Creating AzVolume CRI" "csi.storage.k8s.io/pv/name"="pvc-3c15cd13-4c71-427d-ab77-b4fc98315eef" "disk.csi.azure.com/request-id"="842a84a9-968e-11ed-ae72-002248b69c13" "disk.csi.azure.com/volume-name"="pvc-3c15cd13-4c71-427d-ab77-b4fc98315eef"
I0117 17:43:56.547323       1 round_trippers.go:553] POST https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes 201 Created in 21 milliseconds
I0117 17:43:56.550102       1 crdprovisioner.go:316]  "msg"="Successfully created AzVolume CRI" "csi.storage.k8s.io/pv/name"="pvc-3c15cd13-4c71-427d-ab77-b4fc98315eef" "disk.csi.azure.com/request-id"="842a84a9-968e-11ed-ae72-002248b69c13" "disk.csi.azure.com/volume-name"="pvc-3c15cd13-4c71-427d-ab77-b4fc98315eef"
I0117 17:43:56.550558       1 conditionwatcher.go:99] Adding a condition function for azvolume (pvc-3c15cd13-4c71-427d-ab77-b4fc98315eef)
I0117 17:43:56.569241       1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0117 17:43:56.570482       1 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-3c15cd13-4c71-427d-ab77-b4fc98315eef/status 200 OK in 18 milliseconds
I0117 17:43:56.571141       1 azure_disk_utils.go:1103]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate" "csi.storage.k8s.io/pv/name"="pvc-3c15cd13-4c71-427d-ab77-b4fc98315eef" "disk.csi.azure.com/request-id"="842a84a9-968e-11ed-ae72-002248b69c13" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-3c15cd13-4c71-427d-ab77-b4fc98315eef" "latency"=19587686 "number_of_net_retry"=0 "number_of_retry"=0
I0117 17:43:56.571221       1 azvolume.go:159]  "msg"="Creating Volume..." "csi.storage.k8s.io/pv/name"="pvc-3c15cd13-4c71-427d-ab77-b4fc98315eef" "disk.csi.azure.com/request-id"="842a84a9-968e-11ed-ae72-002248b69c13" "disk.csi.azure.com/volume-name"="pvc-3c15cd13-4c71-427d-ab77-b4fc98315eef"
I0117 17:43:56.619926       1 util.go:124] Send.sendRequest got response with ContentLength 247, StatusCode 404 and responseBody length 247
I0117 17:43:56.619961       1 azure_diskclient.go:139] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-wto256/providers/Microsoft.Compute/disks/pvc-3c15cd13-4c71-427d-ab77-b4fc98315eef, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-3c15cd13-4c71-427d-ab77-b4fc98315eef' under resource group 'capz-wto256' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I0117 17:43:56.620172       1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=0.048539813 request="disks_get" resource_group="capz-wto256" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code="ResourceNotFound"
I0117 17:43:56.620278       1 cloudprovisioner.go:246] begin to create disk(pvc-3c15cd13-4c71-427d-ab77-b4fc98315eef) account type(StandardSSD_LRS) rg(capz-wto256) location() size(10) selectedAvailabilityZone() maxShares(0)
I0117 17:43:56.807405       1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 10850
I0117 17:43:56.807905       1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=0.187496024 request="disks_get" resource_group="capz-wto256" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0117 17:43:56.808063       1 azure_managedDiskController.go:94] azureDisk - creating new managed Name:pvc-3c15cd13-4c71-427d-ab77-b4fc98315eef StorageAccountType:StandardSSD_LRS Size:10
I0117 17:43:56.859881       1 round_trippers.go:553] GET https://10.96.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 5 milliseconds
... skipping 12 lines ...
I0117 17:43:59.114177       1 cloudprovisioner.go:320]  "msg"="create disk(pvc-3c15cd13-4c71-427d-ab77-b4fc98315eef) account type(StandardSSD_LRS) rg(capz-wto256) location() size(10) tags(map[kubernetes.io-created-for-pv-name:pvc-3c15cd13-4c71-427d-ab77-b4fc98315eef kubernetes.io-created-for-pvc-name:test-volume-1-azuredisk-volume-tester-h8jrt-0 kubernetes.io-created-for-pvc-namespace:azuredisk-3523]) successfully" "csi.storage.k8s.io/pv/name"="pvc-3c15cd13-4c71-427d-ab77-b4fc98315eef" "disk.csi.azure.com/request-id"="842a84a9-968e-11ed-ae72-002248b69c13" "disk.csi.azure.com/volume-name"="pvc-3c15cd13-4c71-427d-ab77-b4fc98315eef"
I0117 17:43:59.114432       1 cloudprovisioner.go:142]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-3c15cd13-4c71-427d-ab77-b4fc98315eef" "disk.csi.azure.com/request-id"="842a84a9-968e-11ed-ae72-002248b69c13" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-3c15cd13-4c71-427d-ab77-b4fc98315eef" "latency"=2542920675
I0117 17:43:59.130755       1 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-3c15cd13-4c71-427d-ab77-b4fc98315eef/status 200 OK in 15 milliseconds
I0117 17:43:59.131195       1 azure_disk_utils.go:1103]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "csi.storage.k8s.io/pv/name"="pvc-3c15cd13-4c71-427d-ab77-b4fc98315eef" "disk.csi.azure.com/request-id"="842a84a9-968e-11ed-ae72-002248b69c13" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-3c15cd13-4c71-427d-ab77-b4fc98315eef" "latency"=16684873 "number_of_net_retry"=0 "number_of_retry"=0
I0117 17:43:59.131957       1 azvolume.go:166]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-3c15cd13-4c71-427d-ab77-b4fc98315eef" "disk.csi.azure.com/request-id"="842a84a9-968e-11ed-ae72-002248b69c13" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "disk.csi.azure.com/volume-name"="pvc-3c15cd13-4c71-427d-ab77-b4fc98315eef" "latency"=2559979250
I0117 17:43:59.132102       1 workflow.go:155]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-3c15cd13-4c71-427d-ab77-b4fc98315eef" "disk.csi.azure.com/request-id"="842a84a9-968e-11ed-ae72-002248b69c13" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate" "disk.csi.azure.com/volume-name"="pvc-3c15cd13-4c71-427d-ab77-b4fc98315eef" "latency"=2580608341
I0117 17:43:59.132863       1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 17:43:59.133559       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).CreateVolume" "csi.storage.k8s.io/pv/name"="pvc-3c15cd13-4c71-427d-ab77-b4fc98315eef" "disk.csi.azure.com/request-id"="842a84a9-968e-11ed-ae72-002248b69c13" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-3c15cd13-4c71-427d-ab77-b4fc98315eef" "latency"=2582736650
I0117 17:43:59.133697       1 crdprovisioner.go:228]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-3c15cd13-4c71-427d-ab77-b4fc98315eef" "disk.csi.azure.com/request-id"="842a84a9-968e-11ed-ae72-002248b69c13" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-3c15cd13-4c71-427d-ab77-b4fc98315eef" "latency"=2608192062
I0117 17:43:59.133744       1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=2.608304562 request="azuredisk_csi_driver_controller_create_volume" resource_group="capz-wto256" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-wto256/providers/Microsoft.Compute/disks/pvc-3c15cd13-4c71-427d-ab77-b4fc98315eef" result_code="succeeded"
I0117 17:43:59.133756       1 utils.go:85] GRPC response: {"volume":{"accessible_topology":[{"segments":{"topology.disk.csi.azure.com/zone":""}}],"capacity_bytes":10737418240,"content_source":{"Type":{"Volume":{}}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-3c15cd13-4c71-427d-ab77-b4fc98315eef","csi.storage.k8s.io/pvc/name":"test-volume-1-azuredisk-volume-tester-h8jrt-0","csi.storage.k8s.io/pvc/namespace":"azuredisk-3523","requestedsizegib":"10","skuname":"StandardSSD_LRS"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-wto256/providers/Microsoft.Compute/disks/pvc-3c15cd13-4c71-427d-ab77-b4fc98315eef"}}
I0117 17:44:00.499999       1 utils.go:78] GRPC call: /csi.v1.Controller/ControllerPublishVolume
I0117 17:44:00.500027       1 utils.go:79] GRPC request: {"node_id":"capz-wto2-qsbsf","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-3c15cd13-4c71-427d-ab77-b4fc98315eef","csi.storage.k8s.io/pvc/name":"test-volume-1-azuredisk-volume-tester-h8jrt-0","csi.storage.k8s.io/pvc/namespace":"azuredisk-3523","requestedsizegib":"10","skuname":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1673977076937-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-wto256/providers/Microsoft.Compute/disks/pvc-3c15cd13-4c71-427d-ab77-b4fc98315eef"}
I0117 17:44:00.500670       1 azure_disk_utils.go:1068]  "msg"="Label selector is: disk.csi.azure.com/node-name=capz-wto2-qsbsf,disk.csi.azure.com/volume-name!=pvc-3c15cd13-4c71-427d-ab77-b4fc98315eef." "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="86890ec7-968e-11ed-ae72-002248b69c13" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-3c15cd13-4c71-427d-ab77-b4fc98315eef"
I0117 17:44:00.509459       1 round_trippers.go:553] POST https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments 201 Created in 8 milliseconds
I0117 17:44:00.509869       1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-3c15cd13-4c71-427d-ab77-b4fc98315eef-capz-wto2-qsbsf-attachment)
I0117 17:44:00.518792       1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0117 17:44:00.518854       1 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-3c15cd13-4c71-427d-ab77-b4fc98315eef-capz-wto2-qsbsf-attachment/status 200 OK in 7 milliseconds
I0117 17:44:00.519587       1 azure_disk_utils.go:1103]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="86890ec7-968e-11ed-ae72-002248b69c13" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-3c15cd13-4c71-427d-ab77-b4fc98315eef" "latency"=8871139 "number_of_net_retry"=0 "number_of_retry"=0
I0117 17:44:00.519617       1 attach_detach.go:192]  "msg"="Attaching volume" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="86890ec7-968e-11ed-ae72-002248b69c13" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-3c15cd13-4c71-427d-ab77-b4fc98315eef"
I0117 17:44:00.577656       1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1223
I0117 17:44:00.578101       1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=0.058306457 request="disks_get" resource_group="capz-wto256" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0117 17:44:00.578280       1 cloudprovisioner.go:405]  "msg"="GetDiskLun returned: -1. Initiating attaching volume \"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-wto256/providers/Microsoft.Compute/disks/pvc-3c15cd13-4c71-427d-ab77-b4fc98315eef\" to node \"capz-wto2-qsbsf\"." "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="86890ec7-968e-11ed-ae72-002248b69c13" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-3c15cd13-4c71-427d-ab77-b4fc98315eef"
I0117 17:44:00.578386       1 cloudprovisioner.go:424]  "msg"="Trying to attach volume \"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-wto256/providers/Microsoft.Compute/disks/pvc-3c15cd13-4c71-427d-ab77-b4fc98315eef\" to node \"capz-wto2-qsbsf\"." "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="86890ec7-968e-11ed-ae72-002248b69c13" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-3c15cd13-4c71-427d-ab77-b4fc98315eef"
I0117 17:44:00.882049       1 round_trippers.go:553] GET https://10.96.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 3 milliseconds
I0117 17:44:00.888958       1 round_trippers.go:553] PUT https://10.96.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 6 milliseconds
I0117 17:44:00.889229       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0117 17:44:01.579283       1 batch.go:224] "cloud-provider-azure: Delayed processing of batch due to start delay" type="batch" operation="attach_disk" key="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e|capz-wto256|capz-wto2-qsbsf" delay="1s"
I0117 17:44:01.579392       1 azure_controller_standard.go:103] azureDisk - update(capz-wto256): vm(capz-wto2-qsbsf) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-wto256/providers/Microsoft.Compute/disks/pvc-3c15cd13-4c71-427d-ab77-b4fc98315eef:AttachDiskOptions{diskName: "pvc-3c15cd13-4c71-427d-ab77-b4fc98315eef", lun: 0}])
I0117 17:44:01.592593       1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 17:44:01.592861       1 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-3c15cd13-4c71-427d-ab77-b4fc98315eef-capz-wto2-qsbsf-attachment/status 200 OK in 12 milliseconds
I0117 17:44:01.593220       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="86890ec7-968e-11ed-ae72-002248b69c13" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-3c15cd13-4c71-427d-ab77-b4fc98315eef" "latency"=1083082758
I0117 17:44:01.593394       1 crdprovisioner.go:727]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="86890ec7-968e-11ed-ae72-002248b69c13" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLun" "disk.csi.azure.com/volume-name"="pvc-3c15cd13-4c71-427d-ab77-b4fc98315eef" "latency"=1083410159
I0117 17:44:01.593596       1 crdprovisioner.go:486]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="86890ec7-968e-11ed-ae72-002248b69c13" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).PublishVolume" "disk.csi.azure.com/volume-name"="pvc-3c15cd13-4c71-427d-ab77-b4fc98315eef" "latency"=1092915401
I0117 17:44:01.593688       1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=1.093253502 request="azuredisk_csi_driver_controller_publish_volume" resource_group="capz-wto256" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-wto256/providers/Microsoft.Compute/disks/pvc-3c15cd13-4c71-427d-ab77-b4fc98315eef" node="capz-wto2-qsbsf" result_code="succeeded"
I0117 17:44:01.593746       1 utils.go:85] GRPC response: {"publish_context":{"LUN":"0"}}
... skipping 28 lines ...
I0117 17:44:11.988756       1 azure_disk_utils.go:1165]  "msg"="Skip update. No update needed." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).updateVolumeAttachmentWithResult" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="86890ec7-968e-11ed-ae72-002248b69c13" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-3c15cd13-4c71-427d-ab77-b4fc98315eef"
I0117 17:44:11.988803       1 azure_disk_utils.go:1103]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).updateVolumeAttachmentWithResult" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="86890ec7-968e-11ed-ae72-002248b69c13" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-3c15cd13-4c71-427d-ab77-b4fc98315eef" "latency"=140600 "number_of_net_retry"=0 "number_of_retry"=0
I0117 17:44:12.008426       1 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-3c15cd13-4c71-427d-ab77-b4fc98315eef-capz-wto2-qsbsf-attachment/status 200 OK in 19 milliseconds
I0117 17:44:12.009354       1 azure_disk_utils.go:1103]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3.4" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="86890ec7-968e-11ed-ae72-002248b69c13" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-3c15cd13-4c71-427d-ab77-b4fc98315eef" "latency"=20457432 "number_of_net_retry"=0 "number_of_retry"=0
I0117 17:44:12.009617       1 attach_detach.go:198]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="86890ec7-968e-11ed-ae72-002248b69c13" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3" "disk.csi.azure.com/volume-name"="pvc-3c15cd13-4c71-427d-ab77-b4fc98315eef" "latency"=11489944516
I0117 17:44:12.009788       1 workflow.go:155]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="86890ec7-968e-11ed-ae72-002248b69c13" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach" "disk.csi.azure.com/volume-name"="pvc-3c15cd13-4c71-427d-ab77-b4fc98315eef" "latency"=11499102655
I0117 17:44:12.009949       1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 17:44:12.010060       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="872fe548-968e-11ed-ae72-002248b69c13" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-3c15cd13-4c71-427d-ab77-b4fc98315eef" "latency"=10415742795
I0117 17:44:12.010139       1 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).ControllerPublishVolume.func2" "disk.csi.azure.com/request-id"="872fe548-968e-11ed-ae72-002248b69c13" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=10416136897
I0117 17:44:12.010243       1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=11.509851701 request="azuredisk_csi_driver_attach_volume_latency" resource_group="capz-wto256" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-wto256/providers/Microsoft.Compute/disks/pvc-3c15cd13-4c71-427d-ab77-b4fc98315eef" node="capz-wto2-qsbsf" result_code="succeeded"
I0117 17:44:12.010304       1 controllerserver_v2.go:264]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="872fe548-968e-11ed-ae72-002248b69c13" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).ControllerPublishVolume.func2" "latency"=10416376198
I0117 17:44:12.954378       1 round_trippers.go:553] GET https://10.96.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 1 milliseconds
I0117 17:44:12.960928       1 round_trippers.go:553] PUT https://10.96.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 6 milliseconds
... skipping 21658 lines ...
I0117 19:26:05.885173       1 utils.go:78] GRPC call: /csi.v1.Controller/CreateVolume
I0117 19:26:05.885329       1 utils.go:79] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.disk.csi.azure.com/zone":""}}],"requisite":[{"segments":{"topology.disk.csi.azure.com/zone":""}}]},"capacity_range":{"required_bytes":107374182400},"name":"pvc-70cf148f-ed9f-4422-ab28-c1f0a0a69b27","parameters":{"csi.storage.k8s.io/pv/name":"pvc-70cf148f-ed9f-4422-ab28-c1f0a0a69b27","csi.storage.k8s.io/pvc/name":"persistent-storage-statefulset-azuredisk-win-0","csi.storage.k8s.io/pvc/namespace":"default","skuName":"StandardSSD_LRS"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]}
I0117 19:26:05.885973       1 crdprovisioner.go:308]  "msg"="Creating AzVolume CRI" "csi.storage.k8s.io/pv/name"="pvc-70cf148f-ed9f-4422-ab28-c1f0a0a69b27" "disk.csi.azure.com/request-id"="c98cb028-969c-11ed-ae72-002248b69c13" "disk.csi.azure.com/volume-name"="pvc-70cf148f-ed9f-4422-ab28-c1f0a0a69b27"
I0117 19:26:05.906309       1 round_trippers.go:553] POST https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes 201 Created in 20 milliseconds
I0117 19:26:05.913989       1 crdprovisioner.go:316]  "msg"="Successfully created AzVolume CRI" "csi.storage.k8s.io/pv/name"="pvc-70cf148f-ed9f-4422-ab28-c1f0a0a69b27" "disk.csi.azure.com/request-id"="c98cb028-969c-11ed-ae72-002248b69c13" "disk.csi.azure.com/volume-name"="pvc-70cf148f-ed9f-4422-ab28-c1f0a0a69b27"
I0117 19:26:05.914009       1 conditionwatcher.go:99] Adding a condition function for azvolume (pvc-70cf148f-ed9f-4422-ab28-c1f0a0a69b27)
I0117 19:26:05.918630       1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0117 19:26:05.919460       1 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-70cf148f-ed9f-4422-ab28-c1f0a0a69b27/status 200 OK in 13 milliseconds
I0117 19:26:05.919829       1 azure_disk_utils.go:1103]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate" "csi.storage.k8s.io/pv/name"="pvc-70cf148f-ed9f-4422-ab28-c1f0a0a69b27" "disk.csi.azure.com/request-id"="c98cb028-969c-11ed-ae72-002248b69c13" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-70cf148f-ed9f-4422-ab28-c1f0a0a69b27" "latency"=14214541 "number_of_net_retry"=0 "number_of_retry"=0
I0117 19:26:05.920038       1 azvolume.go:159]  "msg"="Creating Volume..." "csi.storage.k8s.io/pv/name"="pvc-70cf148f-ed9f-4422-ab28-c1f0a0a69b27" "disk.csi.azure.com/request-id"="c98cb028-969c-11ed-ae72-002248b69c13" "disk.csi.azure.com/volume-name"="pvc-70cf148f-ed9f-4422-ab28-c1f0a0a69b27"
I0117 19:26:05.936422       1 util.go:124] Send.sendRequest got response with ContentLength 247, StatusCode 404 and responseBody length 247
I0117 19:26:05.936498       1 azure_diskclient.go:139] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-wto256/providers/Microsoft.Compute/disks/pvc-70cf148f-ed9f-4422-ab28-c1f0a0a69b27, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-70cf148f-ed9f-4422-ab28-c1f0a0a69b27' under resource group 'capz-wto256' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I0117 19:26:05.936640       1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=0.016298047 request="disks_get" resource_group="capz-wto256" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code="ResourceNotFound"
I0117 19:26:05.936661       1 cloudprovisioner.go:246] begin to create disk(pvc-70cf148f-ed9f-4422-ab28-c1f0a0a69b27) account type(StandardSSD_LRS) rg(capz-wto256) location() size(100) selectedAvailabilityZone() maxShares(0)
I0117 19:26:05.975534       1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 12333
I0117 19:26:05.975963       1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=0.039263613 request="disks_get" resource_group="capz-wto256" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0117 19:26:05.976004       1 azure_managedDiskController.go:94] azureDisk - creating new managed Name:pvc-70cf148f-ed9f-4422-ab28-c1f0a0a69b27 StorageAccountType:StandardSSD_LRS Size:100
I0117 19:26:06.837941       1 round_trippers.go:553] GET https://10.96.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 8 milliseconds
... skipping 2 lines ...
I0117 19:26:08.291901       1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=2.315825086 request="disks_create_or_update" resource_group="capz-wto256" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0117 19:26:08.327346       1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1219
I0117 19:26:08.327738       1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=0.035781603 request="disks_get" resource_group="capz-wto256" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0117 19:26:08.327771       1 azure_managedDiskController.go:280] azureDisk - created new MD Name:pvc-70cf148f-ed9f-4422-ab28-c1f0a0a69b27 StorageAccountType:StandardSSD_LRS Size:100
I0117 19:26:08.327876       1 cloudprovisioner.go:320]  "msg"="create disk(pvc-70cf148f-ed9f-4422-ab28-c1f0a0a69b27) account type(StandardSSD_LRS) rg(capz-wto256) location() size(100) tags(map[kubernetes.io-created-for-pv-name:pvc-70cf148f-ed9f-4422-ab28-c1f0a0a69b27 kubernetes.io-created-for-pvc-name:persistent-storage-statefulset-azuredisk-win-0 kubernetes.io-created-for-pvc-namespace:default]) successfully" "csi.storage.k8s.io/pv/name"="pvc-70cf148f-ed9f-4422-ab28-c1f0a0a69b27" "disk.csi.azure.com/request-id"="c98cb028-969c-11ed-ae72-002248b69c13" "disk.csi.azure.com/volume-name"="pvc-70cf148f-ed9f-4422-ab28-c1f0a0a69b27"
I0117 19:26:08.327990       1 cloudprovisioner.go:142]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-70cf148f-ed9f-4422-ab28-c1f0a0a69b27" "disk.csi.azure.com/request-id"="c98cb028-969c-11ed-ae72-002248b69c13" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-70cf148f-ed9f-4422-ab28-c1f0a0a69b27" "latency"=2407675452
I0117 19:26:08.343272       1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 19:26:08.343852       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).CreateVolume" "csi.storage.k8s.io/pv/name"="pvc-70cf148f-ed9f-4422-ab28-c1f0a0a69b27" "disk.csi.azure.com/request-id"="c98cb028-969c-11ed-ae72-002248b69c13" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-70cf148f-ed9f-4422-ab28-c1f0a0a69b27" "latency"=2429761215
I0117 19:26:08.344724       1 crdprovisioner.go:228]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-70cf148f-ed9f-4422-ab28-c1f0a0a69b27" "disk.csi.azure.com/request-id"="c98cb028-969c-11ed-ae72-002248b69c13" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-70cf148f-ed9f-4422-ab28-c1f0a0a69b27" "latency"=2458839599
I0117 19:26:08.345468       1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=2.4589701 request="azuredisk_csi_driver_controller_create_volume" resource_group="capz-wto256" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-wto256/providers/Microsoft.Compute/disks/pvc-70cf148f-ed9f-4422-ab28-c1f0a0a69b27" result_code="succeeded"
I0117 19:26:08.345499       1 utils.go:85] GRPC response: {"volume":{"accessible_topology":[{"segments":{"topology.disk.csi.azure.com/zone":""}}],"capacity_bytes":107374182400,"content_source":{"Type":{"Volume":{}}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-70cf148f-ed9f-4422-ab28-c1f0a0a69b27","csi.storage.k8s.io/pvc/name":"persistent-storage-statefulset-azuredisk-win-0","csi.storage.k8s.io/pvc/namespace":"default","requestedsizegib":"100","skuName":"StandardSSD_LRS"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-wto256/providers/Microsoft.Compute/disks/pvc-70cf148f-ed9f-4422-ab28-c1f0a0a69b27"}}
I0117 19:26:08.346871       1 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-70cf148f-ed9f-4422-ab28-c1f0a0a69b27/status 200 OK in 18 milliseconds
I0117 19:26:08.347396       1 azure_disk_utils.go:1103]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "csi.storage.k8s.io/pv/name"="pvc-70cf148f-ed9f-4422-ab28-c1f0a0a69b27" "disk.csi.azure.com/request-id"="c98cb028-969c-11ed-ae72-002248b69c13" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-70cf148f-ed9f-4422-ab28-c1f0a0a69b27" "latency"=19224355 "number_of_net_retry"=0 "number_of_retry"=0
... skipping 5 lines ...
I0117 19:26:08.925533       1 utils.go:78] GRPC call: /csi.v1.Controller/ControllerPublishVolume
I0117 19:26:08.926240       1 utils.go:79] GRPC request: {"node_id":"capz-wto2-xzbdk","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-70cf148f-ed9f-4422-ab28-c1f0a0a69b27","csi.storage.k8s.io/pvc/name":"persistent-storage-statefulset-azuredisk-win-0","csi.storage.k8s.io/pvc/namespace":"default","requestedsizegib":"100","skuName":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1673977076937-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-wto256/providers/Microsoft.Compute/disks/pvc-70cf148f-ed9f-4422-ab28-c1f0a0a69b27"}
I0117 19:26:08.926655       1 azure_disk_utils.go:1068]  "msg"="Label selector is: disk.csi.azure.com/node-name=capz-wto2-xzbdk,disk.csi.azure.com/volume-name!=pvc-70cf148f-ed9f-4422-ab28-c1f0a0a69b27." "disk.csi.azure.com/node-name"="capz-wto2-xzbdk" "disk.csi.azure.com/request-id"="cb5ca6ce-969c-11ed-ae72-002248b69c13" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-70cf148f-ed9f-4422-ab28-c1f0a0a69b27"
I0117 19:26:08.936570       1 round_trippers.go:553] POST https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments 201 Created in 9 milliseconds
I0117 19:26:08.936956       1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-70cf148f-ed9f-4422-ab28-c1f0a0a69b27-capz-wto2-xzbdk-attachment)
I0117 19:26:08.944789       1 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-70cf148f-ed9f-4422-ab28-c1f0a0a69b27-capz-wto2-xzbdk-attachment/status 200 OK in 6 milliseconds
I0117 19:26:08.945908       1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0117 19:26:08.946258       1 azure_disk_utils.go:1103]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach" "disk.csi.azure.com/node-name"="capz-wto2-xzbdk" "disk.csi.azure.com/request-id"="cb5ca6ce-969c-11ed-ae72-002248b69c13" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-70cf148f-ed9f-4422-ab28-c1f0a0a69b27" "latency"=8474225 "number_of_net_retry"=0 "number_of_retry"=0
I0117 19:26:08.946295       1 attach_detach.go:192]  "msg"="Attaching volume" "disk.csi.azure.com/node-name"="capz-wto2-xzbdk" "disk.csi.azure.com/request-id"="cb5ca6ce-969c-11ed-ae72-002248b69c13" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-70cf148f-ed9f-4422-ab28-c1f0a0a69b27"
I0117 19:26:08.967837       1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1219
I0117 19:26:08.968185       1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=0.021630563 request="disks_get" resource_group="capz-wto256" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0117 19:26:09.047612       1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 5886
I0117 19:26:09.048348       1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=0.080057831 request="vm_get" resource_group="capz-wto256" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0117 19:26:09.048517       1 cloudprovisioner.go:405]  "msg"="GetDiskLun returned: -1. Initiating attaching volume \"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-wto256/providers/Microsoft.Compute/disks/pvc-70cf148f-ed9f-4422-ab28-c1f0a0a69b27\" to node \"capz-wto2-xzbdk\"." "disk.csi.azure.com/node-name"="capz-wto2-xzbdk" "disk.csi.azure.com/request-id"="cb5ca6ce-969c-11ed-ae72-002248b69c13" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-70cf148f-ed9f-4422-ab28-c1f0a0a69b27"
I0117 19:26:09.048568       1 cloudprovisioner.go:424]  "msg"="Trying to attach volume \"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-wto256/providers/Microsoft.Compute/disks/pvc-70cf148f-ed9f-4422-ab28-c1f0a0a69b27\" to node \"capz-wto2-xzbdk\"." "disk.csi.azure.com/node-name"="capz-wto2-xzbdk" "disk.csi.azure.com/request-id"="cb5ca6ce-969c-11ed-ae72-002248b69c13" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-70cf148f-ed9f-4422-ab28-c1f0a0a69b27"
I0117 19:26:10.049002       1 batch.go:224] "cloud-provider-azure: Delayed processing of batch due to start delay" type="batch" operation="attach_disk" key="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e|capz-wto256|capz-wto2-xzbdk" delay="1s"
I0117 19:26:10.049130       1 azure_controller_standard.go:103] azureDisk - update(capz-wto256): vm(capz-wto2-xzbdk) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-wto256/providers/Microsoft.Compute/disks/pvc-70cf148f-ed9f-4422-ab28-c1f0a0a69b27:AttachDiskOptions{diskName: "pvc-70cf148f-ed9f-4422-ab28-c1f0a0a69b27", lun: 0}])
I0117 19:26:10.063789       1 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-70cf148f-ed9f-4422-ab28-c1f0a0a69b27-capz-wto2-xzbdk-attachment/status 200 OK in 13 milliseconds
I0117 19:26:10.064592       1 azure_disk_utils.go:1103]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3.4" "disk.csi.azure.com/node-name"="capz-wto2-xzbdk" "disk.csi.azure.com/request-id"="cb5ca6ce-969c-11ed-ae72-002248b69c13" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-70cf148f-ed9f-4422-ab28-c1f0a0a69b27" "latency"=15094143 "number_of_net_retry"=0 "number_of_retry"=0
I0117 19:26:10.066132       1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 19:26:10.066462       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-wto2-xzbdk" "disk.csi.azure.com/request-id"="cb5ca6ce-969c-11ed-ae72-002248b69c13" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-70cf148f-ed9f-4422-ab28-c1f0a0a69b27" "latency"=1128774257
I0117 19:26:10.066510       1 crdprovisioner.go:727]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="capz-wto2-xzbdk" "disk.csi.azure.com/request-id"="cb5ca6ce-969c-11ed-ae72-002248b69c13" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLun" "disk.csi.azure.com/volume-name"="pvc-70cf148f-ed9f-4422-ab28-c1f0a0a69b27" "latency"=1129564360
I0117 19:26:10.066535       1 crdprovisioner.go:486]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="capz-wto2-xzbdk" "disk.csi.azure.com/request-id"="cb5ca6ce-969c-11ed-ae72-002248b69c13" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).PublishVolume" "disk.csi.azure.com/volume-name"="pvc-70cf148f-ed9f-4422-ab28-c1f0a0a69b27" "latency"=1140035590
I0117 19:26:10.066575       1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=1.140166391 request="azuredisk_csi_driver_controller_publish_volume" resource_group="capz-wto256" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-wto256/providers/Microsoft.Compute/disks/pvc-70cf148f-ed9f-4422-ab28-c1f0a0a69b27" node="capz-wto2-xzbdk" result_code="succeeded"
I0117 19:26:10.066586       1 utils.go:85] GRPC response: {"publish_context":{"LUN":"0"}}
I0117 19:26:10.066879       1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-70cf148f-ed9f-4422-ab28-c1f0a0a69b27-capz-wto2-xzbdk-attachment)
... skipping 28 lines ...
I0117 19:26:20.386670       1 azure_controller_standard.go:272] updateCache(capz-wto2-xzbdk) successfully
I0117 19:26:20.386801       1 cloudprovisioner.go:448]  "msg"="attach operation successful: volume \"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-wto256/providers/Microsoft.Compute/disks/pvc-70cf148f-ed9f-4422-ab28-c1f0a0a69b27\" attached to node \"capz-wto2-xzbdk\"." "disk.csi.azure.com/node-name"="capz-wto2-xzbdk" "disk.csi.azure.com/request-id"="cb5ca6ce-969c-11ed-ae72-002248b69c13" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-70cf148f-ed9f-4422-ab28-c1f0a0a69b27"
I0117 19:26:20.386941       1 cloudprovisioner.go:440]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="capz-wto2-xzbdk" "disk.csi.azure.com/request-id"="cb5ca6ce-969c-11ed-ae72-002248b69c13" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).PublishVolume.func3" "disk.csi.azure.com/volume-name"="pvc-70cf148f-ed9f-4422-ab28-c1f0a0a69b27" "latency"=11338210763
I0117 19:26:20.386985       1 workflow.go:155]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="capz-wto2-xzbdk" "disk.csi.azure.com/request-id"="cb5ca6ce-969c-11ed-ae72-002248b69c13" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).PublishVolume" "disk.csi.azure.com/volume-name"="pvc-70cf148f-ed9f-4422-ab28-c1f0a0a69b27" "latency"=11440506258
I0117 19:26:20.387147       1 azure_disk_utils.go:1165]  "msg"="Skip update. No update needed." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).updateVolumeAttachmentWithResult" "disk.csi.azure.com/node-name"="capz-wto2-xzbdk" "disk.csi.azure.com/request-id"="cb5ca6ce-969c-11ed-ae72-002248b69c13" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-70cf148f-ed9f-4422-ab28-c1f0a0a69b27"
I0117 19:26:20.387199       1 azure_disk_utils.go:1103]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).updateVolumeAttachmentWithResult" "disk.csi.azure.com/node-name"="capz-wto2-xzbdk" "disk.csi.azure.com/request-id"="cb5ca6ce-969c-11ed-ae72-002248b69c13" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-70cf148f-ed9f-4422-ab28-c1f0a0a69b27" "latency"=132001 "number_of_net_retry"=0 "number_of_retry"=0
I0117 19:26:20.405674       1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 19:26:20.406146       1 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-70cf148f-ed9f-4422-ab28-c1f0a0a69b27-capz-wto2-xzbdk-attachment/status 200 OK in 18 milliseconds
I0117 19:26:20.406442       1 azure_disk_utils.go:1103]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3.4" "disk.csi.azure.com/node-name"="capz-wto2-xzbdk" "disk.csi.azure.com/request-id"="cb5ca6ce-969c-11ed-ae72-002248b69c13" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-70cf148f-ed9f-4422-ab28-c1f0a0a69b27" "latency"=19181555 "number_of_net_retry"=0 "number_of_retry"=0
I0117 19:26:20.406484       1 attach_detach.go:198]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="capz-wto2-xzbdk" "disk.csi.azure.com/request-id"="cb5ca6ce-969c-11ed-ae72-002248b69c13" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3" "disk.csi.azure.com/volume-name"="pvc-70cf148f-ed9f-4422-ab28-c1f0a0a69b27" "latency"=11460033114
I0117 19:26:20.406507       1 workflow.go:155]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="capz-wto2-xzbdk" "disk.csi.azure.com/request-id"="cb5ca6ce-969c-11ed-ae72-002248b69c13" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach" "disk.csi.azure.com/volume-name"="pvc-70cf148f-ed9f-4422-ab28-c1f0a0a69b27" "latency"=11468771539
I0117 19:26:20.406888       1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 19:26:20.406937       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-wto2-xzbdk" "disk.csi.azure.com/request-id"="cc0aa772-969c-11ed-ae72-002248b69c13" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-70cf148f-ed9f-4422-ab28-c1f0a0a69b27" "latency"=10339987681
I0117 19:26:20.406964       1 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).ControllerPublishVolume.func2" "disk.csi.azure.com/request-id"="cc0aa772-969c-11ed-ae72-002248b69c13" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=10340113381
I0117 19:26:20.406991       1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=11.480587273 request="azuredisk_csi_driver_attach_volume_latency" resource_group="capz-wto256" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-wto256/providers/Microsoft.Compute/disks/pvc-70cf148f-ed9f-4422-ab28-c1f0a0a69b27" node="capz-wto2-xzbdk" result_code="succeeded"
I0117 19:26:20.407027       1 controllerserver_v2.go:264]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="cc0aa772-969c-11ed-ae72-002248b69c13" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).ControllerPublishVolume.func2" "latency"=10340187881
I0117 19:26:20.407150       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-wto2-xzbdk" "disk.csi.azure.com/request-id"="cc0c1184-969c-11ed-ae72-002248b69c13" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-70cf148f-ed9f-4422-ab28-c1f0a0a69b27" "latency"=10329334651
I0117 19:26:20.407178       1 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).ControllerPublishVolume.func2" "disk.csi.azure.com/request-id"="cc0c1184-969c-11ed-ae72-002248b69c13" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=10331060956
... skipping 141 lines ...
I0117 17:38:00.429809    1360 round_trippers.go:553] GET https://10.96.0.1:443/apis/apiextensions.k8s.io/v1/customresourcedefinitions?limit=500&resourceVersion=0 200 OK in 44 milliseconds
I0117 17:38:00.473241    1360 round_trippers.go:553] GET https://10.96.0.1:443/apis/apiextensions.k8s.io/v1/customresourcedefinitions?allowWatchBookmarks=true&resourceVersion=3503&timeout=7m0s&timeoutSeconds=420&watch=true 200 OK in 0 milliseconds
I0117 17:38:00.487289    1360 shared_informer.go:303] caches populated
I0117 17:38:00.487289    1360 azuredisk_v2.go:225] driver userAgent: disk.csi.azure.com/latest-v2-9ef068a8cb36a997d4ea04b90c05c6f92a488a19 e2e-test
I0117 17:38:00.487289    1360 azure_disk_utils.go:564] reading cloud config from secret kube-system/azure-cloud-provider
I0117 17:38:00.491825    1360 round_trippers.go:553] GET https://10.96.0.1:443/api/v1/namespaces/kube-system/secrets/azure-cloud-provider 404 Not Found in 1 milliseconds
I0117 17:38:00.493263    1360 azure_disk_utils.go:571] InitializeCloudFromSecret: failed to get cloud config from secret kube-system/azure-cloud-provider: failed to get secret kube-system/azure-cloud-provider: secrets "azure-cloud-provider" not found
I0117 17:38:00.493316    1360 azure_disk_utils.go:576] could not read cloud config from secret kube-system/azure-cloud-provider
I0117 17:38:00.493373    1360 azure_disk_utils.go:586] use default AZURE_CREDENTIAL_FILE env var: C:\k\azure.json
I0117 17:38:00.493596    1360 azure_disk_utils.go:594] read cloud config from file: C:\k\azure.json successfully
I0117 17:38:00.497217    1360 azure_auth.go:253] Using AzurePublicCloud environment
I0117 17:38:00.497217    1360 azure_auth.go:138] azure: using client_id+client_secret to retrieve access token
I0117 17:38:00.504113    1360 azure_diskclient.go:68] Azure DisksClient using API version: 2022-03-02
... skipping 22 lines ...
I0117 17:38:00.652340    1360 azuredisk_v2.go:578]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-17T17:38:00Z" "ReadyForVolumeAllocation"=false "StatusMessage"="Driver node initializing." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="b00b0d89-968d-11ed-80d9-002248bca231"
I0117 17:38:00.655269    1360 round_trippers.go:553] POST https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes 201 Created in 1 milliseconds
I0117 17:38:00.672664    1360 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-wto2-qsbsf/status 200 OK in 10 milliseconds
I0117 17:38:00.674043    1360 azuredisk_v2.go:554]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="b00b0d89-968d-11ed-80d9-002248bca231" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).registerAzDriverNode" "latency"=31589600
I0117 17:38:00.674671    1360 azuredisk_v2.go:578]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-17T17:38:00Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="b00fe815-968d-11ed-80d9-002248bca231"
I0117 17:38:00.679182    1360 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-wto2-qsbsf/status 409 Conflict in 4 milliseconds
E0117 17:38:00.679182    1360 azuredisk_v2.go:613]  "msg"="Failed to update AzDriverNode status after creation" "error"="Operation cannot be fulfilled on azdrivernodes.disk.csi.azure.com \"capz-wto2-qsbsf\": the object has been modified; please apply your changes to the latest version and try again" "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="b00fe815-968d-11ed-80d9-002248bca231"
E0117 17:38:00.679182    1360 azuredisk_v2.go:565]  "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [Operation cannot be fulfilled on azdrivernodes.disk.csi.azure.com \"capz-wto2-qsbsf\": the object has been modified; please apply your changes to the latest version and try again]" "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="b00fe815-968d-11ed-80d9-002248bca231" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=5080800
I0117 17:38:00.679182    1360 azuredisk_v2.go:528] Starting heartbeat loop with initial delay 4.301s and frequency 30s
I0117 17:38:00.678378    1360 server.go:117] Listening for connections on address: &net.UnixAddr{Name:"C:\\var\\lib\\kubelet\\plugins\\disk.csi.azure.com\\csi.sock", Net:"unix"}
I0117 17:38:01.003358    1360 utils.go:78] GRPC call: /csi.v1.Identity/GetPluginInfo
I0117 17:38:01.003358    1360 utils.go:79] GRPC request: {}
I0117 17:38:01.009884    1360 utils.go:85] GRPC response: {"name":"disk.csi.azure.com","vendor_version":"latest-v2-9ef068a8cb36a997d4ea04b90c05c6f92a488a19"}
I0117 17:38:01.149364    1360 utils.go:78] GRPC call: /csi.v1.Identity/GetPluginInfo
I0117 17:38:01.149364    1360 utils.go:79] GRPC request: {}
I0117 17:38:01.149364    1360 utils.go:85] GRPC response: {"name":"disk.csi.azure.com","vendor_version":"latest-v2-9ef068a8cb36a997d4ea04b90c05c6f92a488a19"}
I0117 17:38:02.018327    1360 utils.go:78] GRPC call: /csi.v1.Node/NodeGetInfo
I0117 17:38:02.018389    1360 utils.go:79] GRPC request: {}
I0117 17:38:02.042509    1360 azure_instance_metadata.go:140] Warning: failed to get loadbalancer metadata: failure of getting loadbalancer metadata with response "404 Not Found"
I0117 17:38:02.042509    1360 nodeserver_v2.go:438] NodeGetInfo: nodeName(capz-wto2-qsbsf), VM Size(Standard_D4s_v3)
I0117 17:38:02.042509    1360 nodeserver.go:409] got a matching size in getMaxDataDiskCount, VM Size: STANDARD_D4S_V3, MaxDataDiskCount: 8
I0117 17:38:02.042509    1360 utils.go:85] GRPC response: {"accessible_topology":{"segments":{"topology.disk.csi.azure.com/zone":""}},"max_volumes_per_node":8,"node_id":"capz-wto2-qsbsf"}
I0117 17:38:30.424826    1360 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 17:38:30.424826    1360 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 17:38:30.424826    1360 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
... skipping 88 lines ...
I0117 17:42:12.588161    1360 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0117 17:42:12.588161    1360 utils.go:79] GRPC request: {}
I0117 17:42:12.588161    1360 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I0117 17:42:12.591125    1360 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0117 17:42:12.591125    1360 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-f412592d-476e-45c8-ba51-9201946b22da\\globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-f412592d-476e-45c8-ba51-9201946b22da","csi.storage.k8s.io/pvc/name":"pvc-r2p9b","csi.storage.k8s.io/pvc/namespace":"azuredisk-1410","requestedsizegib":"10","skuname":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1673977076937-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-wto256/providers/Microsoft.Compute/disks/pvc-f412592d-476e-45c8-ba51-9201946b22da"}
I0117 17:42:12.592850    1360 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-f412592d-476e-45c8-ba51-9201946b22da-capz-wto2-qsbsf-attachment)
I0117 17:42:22.283691    1360 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 17:42:22.284350    1360 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="463788d2-968e-11ed-80d9-002248bca231" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-f412592d-476e-45c8-ba51-9201946b22da" "latency"=9691078100
I0117 17:42:22.284522    1360 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="463788d2-968e-11ed-80d9-002248bca231" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9692047200
I0117 17:42:28.169078    1360 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I0117 17:42:28.169078    1360 utils.go:79] GRPC request: {}
I0117 17:42:28.169078    1360 utils.go:85] GRPC response: {"ready":{"value":true}}
I0117 17:42:30.497673    1360 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
... skipping 78 lines ...
I0117 17:44:04.152128    1360 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0117 17:44:04.152176    1360 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-3c15cd13-4c71-427d-ab77-b4fc98315eef\\globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-3c15cd13-4c71-427d-ab77-b4fc98315eef","csi.storage.k8s.io/pvc/name":"test-volume-1-azuredisk-volume-tester-h8jrt-0","csi.storage.k8s.io/pvc/namespace":"azuredisk-3523","requestedsizegib":"10","skuname":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1673977076937-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-wto256/providers/Microsoft.Compute/disks/pvc-3c15cd13-4c71-427d-ab77-b4fc98315eef"}
I0117 17:44:04.152176    1360 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-3c15cd13-4c71-427d-ab77-b4fc98315eef-capz-wto2-qsbsf-attachment)
I0117 17:44:05.022545    1360 azuredisk_v2.go:578]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-17T17:44:05Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="893af42e-968e-11ed-80d9-002248bca231"
I0117 17:44:05.037050    1360 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-wto2-qsbsf/status 200 OK in 11 milliseconds
I0117 17:44:05.041361    1360 azuredisk_v2.go:565]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="893af42e-968e-11ed-80d9-002248bca231" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=19557800
I0117 17:44:11.991310    1360 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 17:44:11.992035    1360 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="88b64261-968e-11ed-80d9-002248bca231" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-3c15cd13-4c71-427d-ab77-b4fc98315eef" "latency"=7839358800
I0117 17:44:11.992092    1360 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="88b64261-968e-11ed-80d9-002248bca231" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=7839544200
I0117 17:44:16.804351    1360 nodeserver_v2.go:211] NodeStageVolume: formatting 10 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-3c15cd13-4c71-427d-ab77-b4fc98315eef\globalmount with mount options([])
I0117 17:44:28.167096    1360 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I0117 17:44:28.167096    1360 utils.go:79] GRPC request: {}
I0117 17:44:28.167096    1360 utils.go:85] GRPC response: {"ready":{"value":true}}
... skipping 144 lines ...
I0117 17:48:50.899029    1360 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0117 17:48:50.899589    1360 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-3c15cd13-4c71-427d-ab77-b4fc98315eef\\globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-3c15cd13-4c71-427d-ab77-b4fc98315eef","csi.storage.k8s.io/pvc/name":"test-volume-1-azuredisk-volume-tester-h8jrt-0","csi.storage.k8s.io/pvc/namespace":"azuredisk-3523","requestedsizegib":"10","skuname":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1673977076937-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-wto256/providers/Microsoft.Compute/disks/pvc-3c15cd13-4c71-427d-ab77-b4fc98315eef"}
I0117 17:48:50.900193    1360 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-3c15cd13-4c71-427d-ab77-b4fc98315eef-capz-wto2-qsbsf-attachment)
I0117 17:48:58.185880    1360 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I0117 17:48:58.185928    1360 utils.go:79] GRPC request: {}
I0117 17:48:58.186072    1360 utils.go:85] GRPC response: {"ready":{"value":true}}
I0117 17:48:58.849589    1360 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 17:48:58.850141    1360 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="33a07a34-968f-11ed-80d9-002248bca231" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-3c15cd13-4c71-427d-ab77-b4fc98315eef" "latency"=7948764900
I0117 17:48:58.850196    1360 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="33a07a34-968f-11ed-80d9-002248bca231" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=7949762000
I0117 17:49:00.598487    1360 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 17:49:00.598487    1360 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 17:49:00.598487    1360 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 17:49:00.711021    1360 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync
... skipping 38 lines ...
I0117 17:49:30.734305    1360 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync
I0117 17:49:30.811874    1360 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 17:49:33.022351    1360 nodeserver_v2.go:262] NodeUnstageVolume: unmount \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-3c15cd13-4c71-427d-ab77-b4fc98315eef\globalmount successfully
I0117 17:49:33.022351    1360 utils.go:85] GRPC response: {}
I0117 17:49:35.035319    1360 azuredisk_v2.go:578]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-17T17:49:35Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="4def0054-968f-11ed-80d9-002248bca231"
I0117 17:49:54.008186    1360 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-wto2-qsbsf/status  in 18971 milliseconds
W0117 17:49:54.010165    1360 reflector.go:347] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: watch of *v1beta2.AzVolume ended with: an error on the server ("unable to decode an event from the watch stream: read tcp 10.1.0.5:34612->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.") has prevented the request from succeeding
W0117 17:49:54.011037    1360 reflector.go:347] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: watch of *v1beta2.AzDriverNode ended with: an error on the server ("unable to decode an event from the watch stream: read tcp 10.1.0.5:34612->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.") has prevented the request from succeeding
W0117 17:49:54.011214    1360 reflector.go:347] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: watch of *v1beta2.AzDriverNode ended with: an error on the server ("unable to decode an event from the watch stream: read tcp 10.1.0.5:34612->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.") has prevented the request from succeeding
W0117 17:49:54.011259    1360 reflector.go:347] k8s.io/client-go/informers/factory.go:150: watch of *v1.Node ended with: an error on the server ("unable to decode an event from the watch stream: read tcp 10.1.0.5:34612->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.") has prevented the request from succeeding
E0117 17:49:54.011259    1360 azuredisk_v2.go:613]  "msg"="Failed to update AzDriverNode status after creation" "error"="Put \"https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-wto2-qsbsf/status\": read tcp 10.1.0.5:34612->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="4def0054-968f-11ed-80d9-002248bca231"
W0117 17:49:54.011259    1360 reflector.go:347] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: watch of *v1.CustomResourceDefinition ended with: an error on the server ("unable to decode an event from the watch stream: read tcp 10.1.0.5:34612->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.") has prevented the request from succeeding
W0117 17:49:54.011259    1360 reflector.go:347] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: watch of *v1beta2.AzVolumeAttachment ended with: an error on the server ("unable to decode an event from the watch stream: read tcp 10.1.0.5:34612->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.") has prevented the request from succeeding
E0117 17:49:54.011259    1360 azuredisk_v2.go:565]  "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [Put \"https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-wto2-qsbsf/status\": read tcp 10.1.0.5:34612->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.]" "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="4def0054-968f-11ed-80d9-002248bca231" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=18974996600
I0117 17:49:54.859055    1360 reflector.go:257] Listing and watching *v1beta2.AzVolume from sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132
I0117 17:49:54.882150    1360 round_trippers.go:553] GET https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumes?resourceVersion=5579 200 OK in 22 milliseconds
I0117 17:49:54.884198    1360 round_trippers.go:553] GET https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumes?allowWatchBookmarks=true&resourceVersion=5579&timeout=7m37s&timeoutSeconds=457&watch=true 200 OK in 1 milliseconds
I0117 17:49:54.978277    1360 reflector.go:257] Listing and watching *v1beta2.AzDriverNode from sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132
I0117 17:49:54.978989    1360 round_trippers.go:553] GET https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes?fieldSelector=metadata.name%3Dcapz-wto2-qsbsf&resourceVersion=6960 200 OK in 0 milliseconds
I0117 17:49:54.982985    1360 round_trippers.go:553] GET https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dcapz-wto2-qsbsf&resourceVersion=7187&timeout=5m57s&timeoutSeconds=357&watch=true 200 OK in 2 milliseconds
... skipping 59 lines ...
I0117 17:50:32.675042    1360 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0117 17:50:32.675042    1360 utils.go:79] GRPC request: {"publish_context":{"LUN":"2"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-2e02fb74-1485-418d-a324-48d3678de946\\globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-2e02fb74-1485-418d-a324-48d3678de946","csi.storage.k8s.io/pvc/name":"pvc-48t5t","csi.storage.k8s.io/pvc/namespace":"azuredisk-1183","requestedsizegib":"10","skuname":"Standard_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1673977076937-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-wto256/providers/Microsoft.Compute/disks/pvc-2e02fb74-1485-418d-a324-48d3678de946"}
I0117 17:50:32.675474    1360 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-2e02fb74-1485-418d-a324-48d3678de946-capz-wto2-qsbsf-attachment)
I0117 17:50:35.029497    1360 azuredisk_v2.go:578]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-17T17:50:35Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="71b15c5a-968f-11ed-80d9-002248bca231"
I0117 17:50:35.039615    1360 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-wto2-qsbsf/status 200 OK in 9 milliseconds
I0117 17:50:35.044009    1360 azuredisk_v2.go:565]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="71b15c5a-968f-11ed-80d9-002248bca231" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=14802600
I0117 17:50:45.026785    1360 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 17:50:45.027853    1360 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="7049de0d-968f-11ed-80d9-002248bca231" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-b3e2d788-a90e-4162-9eab-518aad65d728" "latency"=12353428400
I0117 17:50:45.027931    1360 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="7049de0d-968f-11ed-80d9-002248bca231" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=12353920100
I0117 17:50:45.028404    1360 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 17:50:45.028404    1360 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="704954e4-968f-11ed-80d9-002248bca231" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-27b77e3b-ca11-4dc4-a4cf-8160e32ca0a3" "latency"=12356653600
I0117 17:50:45.028404    1360 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="704954e4-968f-11ed-80d9-002248bca231" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=12357981100
I0117 17:50:45.028404    1360 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 17:50:45.028951    1360 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="704a317b-968f-11ed-80d9-002248bca231" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-2e02fb74-1485-418d-a324-48d3678de946" "latency"=12352225700
I0117 17:50:45.029077    1360 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="704a317b-968f-11ed-80d9-002248bca231" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=12352971400
I0117 17:50:52.199099    1360 nodeserver_v2.go:211] NodeStageVolume: formatting 10 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-b3e2d788-a90e-4162-9eab-518aad65d728\globalmount with mount options([])
I0117 17:50:52.213340    1360 nodeserver_v2.go:211] NodeStageVolume: formatting 11 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-27b77e3b-ca11-4dc4-a4cf-8160e32ca0a3\globalmount with mount options([])
I0117 17:50:52.287830    1360 nodeserver_v2.go:211] NodeStageVolume: formatting 12 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-2e02fb74-1485-418d-a324-48d3678de946\globalmount with mount options([])
I0117 17:50:54.900669    1360 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
... skipping 207 lines ...
I0117 17:52:58.199647    1360 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I0117 17:52:58.199647    1360 utils.go:79] GRPC request: {}
I0117 17:52:58.199802    1360 utils.go:85] GRPC response: {"ready":{"value":true}}
I0117 17:53:05.047786    1360 azuredisk_v2.go:578]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-17T17:53:05Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="cb1c5b4d-968f-11ed-80d9-002248bca231"
I0117 17:53:05.068325    1360 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-wto2-qsbsf/status 200 OK in 18 milliseconds
I0117 17:53:05.070900    1360 azuredisk_v2.go:565]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="cb1c5b4d-968f-11ed-80d9-002248bca231" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=23186600
I0117 17:53:08.434408    1360 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 17:53:08.434482    1360 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="c60014c8-968f-11ed-80d9-002248bca231" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-45a31262-d9a6-4b41-8d72-f700c20207c5" "latency"=11960140200
I0117 17:53:08.434482    1360 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="c60014c8-968f-11ed-80d9-002248bca231" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=11960284800
I0117 17:53:08.446102    1360 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 17:53:08.446102    1360 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="c67a33f6-968f-11ed-80d9-002248bca231" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-08f8802e-c0b6-4d80-a0d3-41b0a0eb3db6" "latency"=11170964000
I0117 17:53:08.446102    1360 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="c67a33f6-968f-11ed-80d9-002248bca231" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=11171608400
I0117 17:53:08.447213    1360 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 17:53:08.448330    1360 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="c5e0ec02-968f-11ed-80d9-002248bca231" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-656970b4-b120-4453-86ff-b851ec91e19e" "latency"=12178067500
I0117 17:53:08.448330    1360 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="c5e0ec02-968f-11ed-80d9-002248bca231" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=12178326600
I0117 17:53:08.448989    1360 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 17:53:08.448989    1360 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 17:53:08.448989    1360 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="c61e9bba-968f-11ed-80d9-002248bca231" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-576d59ec-a236-4b80-91b8-97e2d0bf679f" "latency"=11774738700
I0117 17:53:08.448989    1360 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="c61e9bba-968f-11ed-80d9-002248bca231" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=11774738700
I0117 17:53:08.448989    1360 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 17:53:08.448989    1360 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="c63d434b-968f-11ed-80d9-002248bca231" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-41c0b27f-c446-4282-9ed7-782a236636e3" "latency"=11573712700
I0117 17:53:08.448989    1360 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="c63d434b-968f-11ed-80d9-002248bca231" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=11573851300
I0117 17:53:08.448989    1360 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="c65c263a-968f-11ed-80d9-002248bca231" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-d222704b-b8b7-487a-901d-2119f3facb50" "latency"=11371444200
I0117 17:53:08.448989    1360 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="c65c263a-968f-11ed-80d9-002248bca231" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=11371444200
I0117 17:53:20.068779    1360 nodeserver_v2.go:211] NodeStageVolume: formatting 11 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-d222704b-b8b7-487a-901d-2119f3facb50\globalmount with mount options([])
I0117 17:53:20.191326    1360 nodeserver_v2.go:211] NodeStageVolume: formatting 15 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-45a31262-d9a6-4b41-8d72-f700c20207c5\globalmount with mount options([])
... skipping 282 lines ...
I0117 17:54:57.065991    1360 utils.go:79] GRPC request: {}
I0117 17:54:57.066061    1360 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I0117 17:54:57.067944    1360 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0117 17:54:57.068012    1360 utils.go:79] GRPC request: {"publish_context":{"LUN":"5"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-d222704b-b8b7-487a-901d-2119f3facb50\\globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-d222704b-b8b7-487a-901d-2119f3facb50","csi.storage.k8s.io/pvc/name":"test-volume-2-azuredisk-volume-tester-gwhkb-0","csi.storage.k8s.io/pvc/namespace":"azuredisk-9557","requestedsizegib":"10","skuname":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1673977076937-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-wto256/providers/Microsoft.Compute/disks/pvc-d222704b-b8b7-487a-901d-2119f3facb50"}
I0117 17:54:57.070910    1360 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0117 17:54:57.070910    1360 utils.go:79] GRPC request: {"publish_context":{"LUN":"1"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-41c0b27f-c446-4282-9ed7-782a236636e3\\globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-41c0b27f-c446-4282-9ed7-782a236636e3","csi.storage.k8s.io/pvc/name":"test-volume-2-azuredisk-volume-tester-2dzkl-0","csi.storage.k8s.io/pvc/namespace":"azuredisk-9557","requestedsizegib":"10","skuname":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1673977076937-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-wto256/providers/Microsoft.Compute/disks/pvc-41c0b27f-c446-4282-9ed7-782a236636e3"}
E0117 17:54:57.328066    1360 utils.go:83] GRPC error: GetVolumeIDFromMount(c:\var\lib\kubelet\pods\cb074b2a-9314-4b26-83c9-8668cd3f25fd\volumes\kubernetes.io~csi\pvc-08f8802e-c0b6-4d80-a0d3-41b0a0eb3db6\mount) failed with error: rpc error: code = Unknown desc = error getting the volume for the mount c:\var\lib\kubelet\pods\cb074b2a-9314-4b26-83c9-8668cd3f25fd\volumes\kubernetes.io~csi\pvc-08f8802e-c0b6-4d80-a0d3-41b0a0eb3db6\mount, internal error error getting volume from mount. cmd: (Get-Item -Path Get-Item : Cannot find path 'C:\var\lib\kubelet\pods\cb074b2a-9314-4b26-83c9-8668cd3f25fd\volumes\kubernetes.io~csi\pvc
-08f8802e-c0b6-4d80-a0d3-41b0a0eb3db6\mount' because it does not exist.
At line:1 char:2
+ (Get-Item -Path c:\var\lib\kubelet\pods\cb074b2a-9314-4b26-83c9-8668c ...
+  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : ObjectNotFound: (C:\var\lib\kube...0a0eb3db6\mount:String) [Get-Item], ItemNotFoundExcep 
   tion
... skipping 37 lines ...
+ ... d : PathNotFound,Microsoft.PowerShell.Commands.GetItemCommand).Target
+                                                                  ~
Unexpected token ')' in expression or statement.
    + CategoryInfo          : ParserError: (:) [], ParentContainsErrorRecordException
    + FullyQualifiedErrorId : MissingEndParenthesisInExpression
 
, error: exit status 1, response: <nil>
I0117 17:54:57.331102    1360 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0117 17:54:57.331102    1360 utils.go:79] GRPC request: {}
I0117 17:54:57.331193    1360 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I0117 17:54:57.333703    1360 utils.go:78] GRPC call: /csi.v1.Node/NodeGetVolumeStats
I0117 17:54:57.333766    1360 utils.go:79] GRPC request: {"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-wto256/providers/Microsoft.Compute/disks/pvc-656970b4-b120-4453-86ff-b851ec91e19e","volume_path":"c:\\var\\lib\\kubelet\\pods\\cb074b2a-9314-4b26-83c9-8668cd3f25fd\\volumes\\kubernetes.io~csi\\pvc-656970b4-b120-4453-86ff-b851ec91e19e\\mount"}
I0117 17:54:58.214631    1360 utils.go:78] GRPC call: /csi.v1.Identity/Probe
... skipping 9 lines ...
I0117 17:54:58.911710    1360 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I0117 17:54:58.915830    1360 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0117 17:54:58.915830    1360 utils.go:79] GRPC request: {}
I0117 17:54:58.916065    1360 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I0117 17:54:58.919174    1360 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0117 17:54:58.919174    1360 utils.go:79] GRPC request: {"publish_context":{"LUN":"2"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-656970b4-b120-4453-86ff-b851ec91e19e\\globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-656970b4-b120-4453-86ff-b851ec91e19e","csi.storage.k8s.io/pvc/name":"test-volume-3-azuredisk-volume-tester-gwhkb-0","csi.storage.k8s.io/pvc/namespace":"azuredisk-9557","requestedsizegib":"10","skuname":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1673977076937-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-wto256/providers/Microsoft.Compute/disks/pvc-656970b4-b120-4453-86ff-b851ec91e19e"}
E0117 17:54:59.298949    1360 utils.go:83] GRPC error: GetVolumeIDFromMount(c:\var\lib\kubelet\pods\cb074b2a-9314-4b26-83c9-8668cd3f25fd\volumes\kubernetes.io~csi\pvc-656970b4-b120-4453-86ff-b851ec91e19e\mount) failed with error: rpc error: code = Unknown desc = error getting the volume for the mount c:\var\lib\kubelet\pods\cb074b2a-9314-4b26-83c9-8668cd3f25fd\volumes\kubernetes.io~csi\pvc-656970b4-b120-4453-86ff-b851ec91e19e\mount, internal error error getting volume from mount. cmd: (Get-Item -Path Get-Item : Cannot find path 'C:\var\lib\kubelet\pods\cb074b2a-9314-4b26-83c9-8668cd3f25fd\volumes\kubernetes.io~csi\pvc
-656970b4-b120-4453-86ff-b851ec91e19e\mount' because it does not exist.
At line:1 char:2
+ (Get-Item -Path c:\var\lib\kubelet\pods\cb074b2a-9314-4b26-83c9-8668c ...
+  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : ObjectNotFound: (C:\var\lib\kube...1ec91e19e\mount:String) [Get-Item], ItemNotFoundExcep 
   tion
... skipping 37 lines ...
+ ... d : PathNotFound,Microsoft.PowerShell.Commands.GetItemCommand).Target
+                                                                  ~
Unexpected token ')' in expression or statement.
    + CategoryInfo          : ParserError: (:) [], ParentContainsErrorRecordException
    + FullyQualifiedErrorId : MissingEndParenthesisInExpression
 
, error: exit status 1, response: <nil>
I0117 17:55:05.059399    1360 azuredisk_v2.go:578]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-17T17:55:05Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="12a49be8-9690-11ed-80d9-002248bca231"
I0117 17:55:05.074078    1360 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-wto2-qsbsf/status 200 OK in 13 milliseconds
I0117 17:55:05.074078    1360 azuredisk_v2.go:565]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="12a49be8-9690-11ed-80d9-002248bca231" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=15264400
I0117 17:55:08.570434    1360 nodeserver_v2.go:211] NodeStageVolume: formatting 15 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-45a31262-d9a6-4b41-8d72-f700c20207c5\globalmount with mount options([])
I0117 17:55:09.450282    1360 nodeserver_v2.go:211] NodeStageVolume: formatting 12 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-576d59ec-a236-4b80-91b8-97e2d0bf679f\globalmount with mount options([])
I0117 17:55:09.749154    1360 nodeserver_v2.go:211] NodeStageVolume: formatting 14 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-08f8802e-c0b6-4d80-a0d3-41b0a0eb3db6\globalmount with mount options([])
... skipping 832 lines ...
I0117 18:00:35.588349    1360 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0117 18:00:35.588349    1360 utils.go:79] GRPC request: {}
I0117 18:00:35.588493    1360 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I0117 18:00:35.590390    1360 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0117 18:00:35.590390    1360 utils.go:79] GRPC request: {"publish_context":{"LUN":"2"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-475ea1ef-cc82-4b61-8e1f-c4768b3ab903\\globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"cachingmode":"None","csi.storage.k8s.io/pv/name":"pvc-475ea1ef-cc82-4b61-8e1f-c4768b3ab903","csi.storage.k8s.io/pvc/name":"test-volume-1-azuredisk-volume-tester-whr5p-0","csi.storage.k8s.io/pvc/namespace":"azuredisk-4631","maxShares":"2","requestedsizegib":"256","skuname":"Premium_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1673977076937-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-wto256/providers/Microsoft.Compute/disks/pvc-475ea1ef-cc82-4b61-8e1f-c4768b3ab903"}
I0117 18:00:35.590390    1360 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-475ea1ef-cc82-4b61-8e1f-c4768b3ab903-capz-wto2-qsbsf-attachment)
I0117 18:00:40.713054    1360 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 18:00:40.713586    1360 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="d7a7b9a0-9690-11ed-80d9-002248bca231" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-475ea1ef-cc82-4b61-8e1f-c4768b3ab903" "latency"=5122339500
I0117 18:00:40.713689    1360 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="d7a7b9a0-9690-11ed-80d9-002248bca231" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=5123085500
I0117 18:00:40.725084    1360 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 18:00:40.725230    1360 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="d2027e29-9690-11ed-80d9-002248bca231" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-c9c12100-9c1e-48fa-908b-dbaf9c449e81" "latency"=14605672000
I0117 18:00:40.725300    1360 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="d2027e29-9690-11ed-80d9-002248bca231" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=14606002500
I0117 18:00:40.726118    1360 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 18:00:40.726313    1360 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="d21f6292-9690-11ed-80d9-002248bca231" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-f42109be-211d-4ce4-b103-4174c4d72665" "latency"=14417479100
I0117 18:00:40.726430    1360 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="d21f6292-9690-11ed-80d9-002248bca231" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=14417797400
I0117 18:00:41.560901    1360 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0117 18:00:41.560901    1360 utils.go:79] GRPC request: {}
I0117 18:00:41.560901    1360 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I0117 18:00:41.570191    1360 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
... skipping 5 lines ...
I0117 18:00:41.573977    1360 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0117 18:00:41.573977    1360 utils.go:79] GRPC request: {"publish_context":{"LUN":"3"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-2086398c-1b3a-45fb-8d91-04e14e8df6c7\\globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"cachingmode":"None","csi.storage.k8s.io/pv/name":"pvc-2086398c-1b3a-45fb-8d91-04e14e8df6c7","csi.storage.k8s.io/pvc/name":"test-volume-2-azuredisk-volume-tester-whr5p-0","csi.storage.k8s.io/pvc/namespace":"azuredisk-4631","maxShares":"2","requestedsizegib":"256","skuname":"Premium_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1673977076937-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-wto256/providers/Microsoft.Compute/disks/pvc-2086398c-1b3a-45fb-8d91-04e14e8df6c7"}
I0117 18:00:41.574521    1360 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-2086398c-1b3a-45fb-8d91-04e14e8df6c7-capz-wto2-qsbsf-attachment)
I0117 18:00:47.790735    1360 nodeserver_v2.go:211] NodeStageVolume: formatting 10 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-c9c12100-9c1e-48fa-908b-dbaf9c449e81\globalmount with mount options([])
I0117 18:00:47.790735    1360 nodeserver_v2.go:211] NodeStageVolume: formatting 12 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-f42109be-211d-4ce4-b103-4174c4d72665\globalmount with mount options([])
I0117 18:00:48.097785    1360 nodeserver_v2.go:211] NodeStageVolume: formatting 11 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-475ea1ef-cc82-4b61-8e1f-c4768b3ab903\globalmount with mount options([])
I0117 18:00:51.070893    1360 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 18:00:51.070893    1360 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="db38bf81-9690-11ed-80d9-002248bca231" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-2086398c-1b3a-45fb-8d91-04e14e8df6c7" "latency"=9496032900
I0117 18:00:51.070893    1360 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="db38bf81-9690-11ed-80d9-002248bca231" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9496756200
I0117 18:00:55.146170    1360 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 18:00:55.284308    1360 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 18:00:55.457587    1360 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 18:00:55.469831    1360 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
... skipping 850 lines ...
I0117 18:07:03.068437    1360 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0117 18:07:03.068437    1360 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\azuredisk-8377-disk.csi.azure.com-preprovisioned-pv-ddcrg\\globalmount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-wto256/providers/Microsoft.Compute/disks/pre-provisioned-read-only"}
I0117 18:07:03.069088    1360 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pre-provisioned-read-only-capz-wto2-qsbsf-attachment)
I0117 18:07:05.086741    1360 azuredisk_v2.go:578]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-17T18:07:05Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="bfd02388-9691-11ed-80d9-002248bca231"
I0117 18:07:05.096709    1360 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-wto2-qsbsf/status 200 OK in 8 milliseconds
I0117 18:07:05.100134    1360 azuredisk_v2.go:565]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="bfd02388-9691-11ed-80d9-002248bca231" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=13477100
I0117 18:07:11.387228    1360 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 18:07:11.387228    1360 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="be9c43b5-9691-11ed-80d9-002248bca231" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pre-provisioned-read-only" "latency"=8317789100
I0117 18:07:11.388285    1360 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="be9c43b5-9691-11ed-80d9-002248bca231" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=8318748900
I0117 18:07:16.044342    1360 nodeserver_v2.go:211] NodeStageVolume: formatting 10 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\azuredisk-8377-disk.csi.azure.com-preprovisioned-pv-ddcrg\globalmount with mount options([])
I0117 18:07:25.391492    1360 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 18:07:25.432343    1360 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 18:07:25.798328    1360 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
... skipping 92 lines ...
I0117 18:09:16.920183    1360 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (reattach-disk-multiple-nodes-capz-wto2-qsbsf-attachment)
I0117 18:09:25.410355    1360 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 18:09:25.462020    1360 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 18:09:25.828797    1360 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 18:09:25.975721    1360 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync
I0117 18:09:26.068459    1360 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 18:09:26.068459    1360 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0117 18:09:28.240430    1360 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I0117 18:09:28.240430    1360 utils.go:79] GRPC request: {}
I0117 18:09:28.240573    1360 utils.go:85] GRPC response: {"ready":{"value":true}}
I0117 18:09:30.694738    1360 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0117 18:09:35.080552    1360 azuredisk_v2.go:578]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-17T18:09:35Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="19375f24-9692-11ed-80d9-002248bca231"
I0117 18:09:35.102397    1360 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-wto2-qsbsf/status 200 OK in 21 milliseconds
I0117 18:09:35.103227    1360 azuredisk_v2.go:565]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="19375f24-9692-11ed-80d9-002248bca231" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=22882000
I0117 18:09:41.272216    1360 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 18:09:41.272495    1360 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="0e6453e5-9692-11ed-80d9-002248bca231" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="reattach-disk-multiple-nodes" "latency"=24351277600
I0117 18:09:41.272495    1360 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="0e6453e5-9692-11ed-80d9-002248bca231" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=24351348600
I0117 18:09:46.192950    1360 nodeserver_v2.go:211] NodeStageVolume: formatting 10 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\azuredisk-4923-disk.csi.azure.com-preprovisioned-pv-qclq7\globalmount with mount options([])
I0117 18:09:55.418547    1360 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 18:09:55.480897    1360 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 18:09:55.842752    1360 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
... skipping 78 lines ...
I0117 18:11:39.727480    1360 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0117 18:11:39.727480    1360 utils.go:79] GRPC request: {}
I0117 18:11:39.727480    1360 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I0117 18:11:39.730669    1360 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0117 18:11:39.730669    1360 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-2b4ca4aa-d4bc-4662-85c6-8fb4d165bfad\\globalmount","volume_capability":{"AccessType":{"Mount":{"mount_flags":["barrier=1","acl"]}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-2b4ca4aa-d4bc-4662-85c6-8fb4d165bfad","csi.storage.k8s.io/pvc/name":"pvc-bnskg","csi.storage.k8s.io/pvc/namespace":"azuredisk-9580","requestedsizegib":"10","skuname":"Standard_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1673977076937-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-wto256/providers/Microsoft.Compute/disks/pvc-2b4ca4aa-d4bc-4662-85c6-8fb4d165bfad"}
I0117 18:11:39.731314    1360 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-2b4ca4aa-d4bc-4662-85c6-8fb4d165bfad-capz-wto2-qsbsf-attachment)
I0117 18:11:47.452333    1360 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 18:11:47.452896    1360 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="63839677-9692-11ed-80d9-002248bca231" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-2b4ca4aa-d4bc-4662-85c6-8fb4d165bfad" "latency"=7720753200
I0117 18:11:47.453020    1360 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="63839677-9692-11ed-80d9-002248bca231" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=7721393500
I0117 18:11:52.101855    1360 nodeserver_v2.go:211] NodeStageVolume: formatting 10 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-2b4ca4aa-d4bc-4662-85c6-8fb4d165bfad\globalmount with mount options([barrier=1 acl])
I0117 18:11:55.454717    1360 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 18:11:55.511912    1360 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 18:11:55.881094    1360 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
... skipping 74 lines ...
I0117 18:13:22.260078    1360 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-6aee15f1-7a60-4c6f-9ce7-313e87415635-capz-wto2-qsbsf-attachment)
I0117 18:13:25.483447    1360 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 18:13:25.534595    1360 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 18:13:25.899135    1360 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 18:13:26.043192    1360 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync
I0117 18:13:26.125616    1360 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 18:13:26.125754    1360 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0117 18:13:28.267284    1360 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I0117 18:13:28.267284    1360 utils.go:79] GRPC request: {}
I0117 18:13:28.267883    1360 utils.go:85] GRPC response: {"ready":{"value":true}}
I0117 18:13:31.922936    1360 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 18:13:31.923208    1360 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="a0a03c4e-9692-11ed-80d9-002248bca231" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-6aee15f1-7a60-4c6f-9ce7-313e87415635" "latency"=9662807000
I0117 18:13:31.923338    1360 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="a0a03c4e-9692-11ed-80d9-002248bca231" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9662977100
I0117 18:13:35.087187    1360 azuredisk_v2.go:578]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-17T18:13:35Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="a8456ca5-9692-11ed-80d9-002248bca231"
I0117 18:13:35.097739    1360 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-wto2-qsbsf/status 200 OK in 10 milliseconds
I0117 18:13:35.103787    1360 azuredisk_v2.go:565]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="a8456ca5-9692-11ed-80d9-002248bca231" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=16202500
I0117 18:13:36.923894    1360 nodeserver_v2.go:211] NodeStageVolume: formatting 10 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-6aee15f1-7a60-4c6f-9ce7-313e87415635\globalmount with mount options([])
... skipping 77 lines ...
I0117 18:15:14.537958    1360 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-3a39daf3-2bee-46b8-a17a-2a0cf65b0c10-capz-wto2-qsbsf-attachment)
I0117 18:15:25.533290    1360 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 18:15:25.582555    1360 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 18:15:25.946789    1360 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 18:15:26.081981    1360 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync
I0117 18:15:26.155519    1360 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 18:15:26.155623    1360 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0117 18:15:27.451348    1360 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 18:15:27.451475    1360 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="e38c7b00-9692-11ed-80d9-002248bca231" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-3a39daf3-2bee-46b8-a17a-2a0cf65b0c10" "latency"=12912949100
I0117 18:15:27.451562    1360 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="e38c7b00-9692-11ed-80d9-002248bca231" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=12913115500
I0117 18:15:28.255338    1360 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I0117 18:15:28.255381    1360 utils.go:79] GRPC request: {}
I0117 18:15:28.255440    1360 utils.go:85] GRPC response: {"ready":{"value":true}}
I0117 18:15:32.390348    1360 nodeserver_v2.go:211] NodeStageVolume: formatting 10 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-3a39daf3-2bee-46b8-a17a-2a0cf65b0c10\globalmount with mount options([])
... skipping 75 lines ...
I0117 18:17:06.250178    1360 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0117 18:17:06.250178    1360 utils.go:79] GRPC request: {}
I0117 18:17:06.250178    1360 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I0117 18:17:06.253263    1360 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0117 18:17:06.253406    1360 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-c8d7feb0-2992-4126-9eb2-aa09b520ed8c\\globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-c8d7feb0-2992-4126-9eb2-aa09b520ed8c","csi.storage.k8s.io/pvc/name":"pvc-7ftlq","csi.storage.k8s.io/pvc/namespace":"azuredisk-3610","device-setting/queue/nomerges":"0","device-setting/queue/read_ahead_kb":"8","enableAsyncAttach":"false","enableBursting":"true","enableasyncattach":"false","perfProfile":"Advanced","requestedsizegib":"1024","skuname":"Premium_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1673977076937-8081-disk.csi.azure.com","userAgent":"azuredisk-e2e-test"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-wto256/providers/Microsoft.Compute/disks/pvc-c8d7feb0-2992-4126-9eb2-aa09b520ed8c"}
I0117 18:17:06.253486    1360 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-c8d7feb0-2992-4126-9eb2-aa09b520ed8c-capz-wto2-qsbsf-attachment)
I0117 18:17:13.034583    1360 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 18:17:13.044908    1360 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="2622ead4-9693-11ed-80d9-002248bca231" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-c8d7feb0-2992-4126-9eb2-aa09b520ed8c" "latency"=6790922100
I0117 18:17:13.047828    1360 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="2622ead4-9693-11ed-80d9-002248bca231" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=6791572000
I0117 18:17:17.904337    1360 nodeserver_v2.go:211] NodeStageVolume: formatting 10 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-c8d7feb0-2992-4126-9eb2-aa09b520ed8c\globalmount with mount options([])
I0117 18:17:25.580612    1360 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 18:17:25.622581    1360 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 18:17:25.993284    1360 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
... skipping 90 lines ...
I0117 18:19:05.143764    1360 azuredisk_v2.go:565]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="6cfd1e76-9693-11ed-80d9-002248bca231" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=19758800
I0117 18:19:25.617762    1360 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 18:19:25.687032    1360 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 18:19:26.042659    1360 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 18:19:26.142469    1360 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync
I0117 18:19:26.230371    1360 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 18:19:26.230565    1360 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0117 18:19:28.260579    1360 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I0117 18:19:28.260579    1360 utils.go:79] GRPC request: {}
I0117 18:19:28.260729    1360 utils.go:85] GRPC response: {"ready":{"value":true}}
I0117 18:19:30.744040    1360 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 18:19:30.744040    1360 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="6900371c-9693-11ed-80d9-002248bca231" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-d2c00bf2-e84b-4a07-a00e-a27857b4b321" "latency"=32309593500
I0117 18:19:30.744677    1360 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="6900371c-9693-11ed-80d9-002248bca231" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=32310169100
I0117 18:19:35.096298    1360 azuredisk_v2.go:578]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-17T18:19:35Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="7eda8724-9693-11ed-80d9-002248bca231"
I0117 18:19:35.108453    1360 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-wto2-qsbsf/status 200 OK in 12 milliseconds
I0117 18:19:35.109039    1360 azuredisk_v2.go:565]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="7eda8724-9693-11ed-80d9-002248bca231" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=12740700
I0117 18:19:35.585984    1360 nodeserver_v2.go:211] NodeStageVolume: formatting 10 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-d2c00bf2-e84b-4a07-a00e-a27857b4b321\globalmount with mount options([])
... skipping 68 lines ...
I0117 18:21:10.460881    1360 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0117 18:21:10.460936    1360 utils.go:79] GRPC request: {}
I0117 18:21:10.460976    1360 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I0117 18:21:10.465507    1360 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0117 18:21:10.466038    1360 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-4a318787-f6f0-4910-975a-77b8f4c8586d\\globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-4a318787-f6f0-4910-975a-77b8f4c8586d","csi.storage.k8s.io/pvc/name":"pvc-tzhfn","csi.storage.k8s.io/pvc/namespace":"azuredisk-7130","requestedsizegib":"10","skuname":"StandardSSD_ZRS","storage.kubernetes.io/csiProvisionerIdentity":"1673977076937-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-wto256/providers/Microsoft.Compute/disks/pvc-4a318787-f6f0-4910-975a-77b8f4c8586d"}
I0117 18:21:10.466365    1360 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-4a318787-f6f0-4910-975a-77b8f4c8586d-capz-wto2-qsbsf-attachment)
I0117 18:21:18.236690    1360 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 18:21:18.236838    1360 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="b7b2d864-9693-11ed-80d9-002248bca231" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-4a318787-f6f0-4910-975a-77b8f4c8586d" "latency"=7769453000
I0117 18:21:18.236838    1360 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="b7b2d864-9693-11ed-80d9-002248bca231" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=7769522200
I0117 18:21:23.577697    1360 nodeserver_v2.go:211] NodeStageVolume: formatting 10 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-4a318787-f6f0-4910-975a-77b8f4c8586d\globalmount with mount options([])
I0117 18:21:25.662909    1360 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 18:21:25.713969    1360 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 18:21:26.123226    1360 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
... skipping 50 lines ...
I0117 18:22:01.400573    1360 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-817558fb-88f0-4ffe-8584-6b21015e8fd3-capz-wto2-qsbsf-attachment)
I0117 18:22:05.108465    1360 azuredisk_v2.go:578]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-17T18:22:05Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="d844916a-9693-11ed-80d9-002248bca231"
I0117 18:22:05.126821    1360 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-wto2-qsbsf/status 200 OK in 12 milliseconds
I0117 18:22:05.127122    1360 azuredisk_v2.go:565]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="d844916a-9693-11ed-80d9-002248bca231" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=18656200
I0117 18:22:07.912827    1360 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: Watch close - *v1beta2.AzVolume total 39 items received
I0117 18:22:07.916889    1360 round_trippers.go:553] GET https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumes?allowWatchBookmarks=true&resourceVersion=17463&timeout=9m36s&timeoutSeconds=576&watch=true 200 OK in 4 milliseconds
I0117 18:22:08.834176    1360 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 18:22:08.834176    1360 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="d60ec9e2-9693-11ed-80d9-002248bca231" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-817558fb-88f0-4ffe-8584-6b21015e8fd3" "latency"=7433317100
I0117 18:22:08.834723    1360 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="d60ec9e2-9693-11ed-80d9-002248bca231" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=7433317100
I0117 18:22:13.366957    1360 nodeserver_v2.go:211] NodeStageVolume: formatting 13 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-817558fb-88f0-4ffe-8584-6b21015e8fd3\globalmount with mount options([])
I0117 18:22:25.693090    1360 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 18:22:25.734493    1360 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 18:22:26.140842    1360 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
... skipping 45 lines ...
I0117 18:22:55.442235    1360 safe_mounter_windows.go:356] GetVolumeStats(c:\var\lib\kubelet\pods\d144da99-755b-4b48-81b7-24557e062ac0\volumes\kubernetes.io~csi\pvc-817558fb-88f0-4ffe-8584-6b21015e8fd3\mount) returned volumeID(\\?\Volume{79c1b372-7dde-4423-98ec-8bfb01d0ddd1}\)
I0117 18:22:55.707436    1360 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 18:22:55.743744    1360 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 18:22:56.154816    1360 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 18:22:56.212066    1360 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync
I0117 18:22:56.314857    1360 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 18:22:56.314857    1360 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0117 18:22:57.803480    1360 utils.go:85] GRPC response: {"usage":[{"available":10677248000,"total":10719588352,"unit":1,"used":42340352}]}
I0117 18:22:58.279124    1360 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I0117 18:22:58.279124    1360 utils.go:79] GRPC request: {}
I0117 18:22:58.279244    1360 utils.go:85] GRPC response: {"ready":{"value":true}}
I0117 18:23:04.530975    1360 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 18:23:04.531540    1360 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="f465f35b-9693-11ed-80d9-002248bca231" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-fa6b2129-f427-41fa-b32b-1050edcf7321" "latency"=12226253800
I0117 18:23:04.531612    1360 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="f465f35b-9693-11ed-80d9-002248bca231" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=12227428000
I0117 18:23:05.118157    1360 azuredisk_v2.go:578]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-17T18:23:05Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="fc094d25-9693-11ed-80d9-002248bca231"
I0117 18:23:05.132456    1360 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-wto2-qsbsf/status 200 OK in 11 milliseconds
I0117 18:23:05.133121    1360 azuredisk_v2.go:565]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="fc094d25-9693-11ed-80d9-002248bca231" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=14950600
I0117 18:23:09.464633    1360 nodeserver_v2.go:211] NodeStageVolume: formatting 16 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-fa6b2129-f427-41fa-b32b-1050edcf7321\globalmount with mount options([])
... skipping 143 lines ...
I0117 18:26:06.376647    1360 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0117 18:26:06.376712    1360 utils.go:79] GRPC request: {}
I0117 18:26:06.376763    1360 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I0117 18:26:06.378808    1360 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0117 18:26:06.378808    1360 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-f7203c18-8691-47c7-a57b-fe54574417e1\\globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-f7203c18-8691-47c7-a57b-fe54574417e1","csi.storage.k8s.io/pvc/name":"pvc-h8zpb","csi.storage.k8s.io/pvc/namespace":"azuredisk-8905","requestedsizegib":"10","skuname":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1673977076937-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-wto256/providers/Microsoft.Compute/disks/pvc-f7203c18-8691-47c7-a57b-fe54574417e1"}
I0117 18:26:06.379840    1360 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-f7203c18-8691-47c7-a57b-fe54574417e1-capz-wto2-qsbsf-attachment)
I0117 18:26:14.970756    1360 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 18:26:14.970756    1360 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="6813a3e6-9694-11ed-80d9-002248bca231" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-f7203c18-8691-47c7-a57b-fe54574417e1" "latency"=8590516400
I0117 18:26:14.973383    1360 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="6813a3e6-9694-11ed-80d9-002248bca231" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=8590703100
I0117 18:26:20.306864    1360 nodeserver_v2.go:211] NodeStageVolume: formatting 10 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-f7203c18-8691-47c7-a57b-fe54574417e1\globalmount with mount options([])
I0117 18:26:25.795778    1360 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 18:26:25.795778    1360 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 18:26:26.230560    1360 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
... skipping 154 lines ...
I0117 18:29:10.020405    1360 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-aa8b1e04-c805-49e9-9244-69ad325fdb0c-capz-wto2-qsbsf-attachment)
I0117 18:29:10.020405    1360 utils.go:79] GRPC request: {"publish_context":{"LUN":"1"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-fe65b1f7-5234-424e-929f-d07dc7186da3\\globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-fe65b1f7-5234-424e-929f-d07dc7186da3","csi.storage.k8s.io/pvc/name":"pvc-85hz9","csi.storage.k8s.io/pvc/namespace":"azuredisk-8071","requestedsizegib":"10","skuname":"StandardSSD_ZRS","storage.kubernetes.io/csiProvisionerIdentity":"1673977076937-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-wto256/providers/Microsoft.Compute/disks/pvc-fe65b1f7-5234-424e-929f-d07dc7186da3"}
I0117 18:29:10.021039    1360 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-fe65b1f7-5234-424e-929f-d07dc7186da3-capz-wto2-qsbsf-attachment)
I0117 18:29:10.021855    1360 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0117 18:29:10.021855    1360 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-7307fb5a-df6d-4b56-9d03-beadd2352ccb\\globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-7307fb5a-df6d-4b56-9d03-beadd2352ccb","csi.storage.k8s.io/pvc/name":"pvc-j59s7","csi.storage.k8s.io/pvc/namespace":"azuredisk-8071","requestedsizegib":"10","skuname":"StandardSSD_ZRS","storage.kubernetes.io/csiProvisionerIdentity":"1673977076937-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-wto256/providers/Microsoft.Compute/disks/pvc-7307fb5a-df6d-4b56-9d03-beadd2352ccb"}
I0117 18:29:10.021855    1360 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-7307fb5a-df6d-4b56-9d03-beadd2352ccb-capz-wto2-qsbsf-attachment)
I0117 18:29:20.979234    1360 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 18:29:20.979381    1360 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="d58934be-9694-11ed-80d9-002248bca231" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-7307fb5a-df6d-4b56-9d03-beadd2352ccb" "latency"=10956997400
I0117 18:29:20.979491    1360 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="d58934be-9694-11ed-80d9-002248bca231" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=10957142100
I0117 18:29:20.986358    1360 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 18:29:20.986358    1360 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="d588fc16-9694-11ed-80d9-002248bca231" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-aa8b1e04-c805-49e9-9244-69ad325fdb0c" "latency"=10964929700
I0117 18:29:20.986358    1360 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="d588fc16-9694-11ed-80d9-002248bca231" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=10965513700
I0117 18:29:20.992602    1360 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 18:29:20.992602    1360 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="d58914d7-9694-11ed-80d9-002248bca231" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-fe65b1f7-5234-424e-929f-d07dc7186da3" "latency"=10971123200
I0117 18:29:20.992602    1360 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="d58914d7-9694-11ed-80d9-002248bca231" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=10971123200
I0117 18:29:25.834621    1360 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 18:29:25.834895    1360 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 18:29:26.300118    1360 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 18:29:26.352619    1360 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync
... skipping 164 lines ...
I0117 18:32:03.638764    1360 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0117 18:32:03.638764    1360 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-52ec6b4d-35d2-4c79-bf70-b4241935f95e\\globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-52ec6b4d-35d2-4c79-bf70-b4241935f95e","csi.storage.k8s.io/pvc/name":"pvc-w9tmr","csi.storage.k8s.io/pvc/namespace":"azuredisk-5936","requestedsizegib":"10","skuname":"StandardSSD_ZRS","storage.kubernetes.io/csiProvisionerIdentity":"1673977076937-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-wto256/providers/Microsoft.Compute/disks/pvc-52ec6b4d-35d2-4c79-bf70-b4241935f95e"}
I0117 18:32:03.638764    1360 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-52ec6b4d-35d2-4c79-bf70-b4241935f95e-capz-wto2-qsbsf-attachment)
I0117 18:32:05.134779    1360 azuredisk_v2.go:578]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-17T18:32:05Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="3de94b35-9695-11ed-80d9-002248bca231"
I0117 18:32:05.150457    1360 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-wto2-qsbsf/status 200 OK in 14 milliseconds
I0117 18:32:05.150859    1360 azuredisk_v2.go:565]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="3de94b35-9695-11ed-80d9-002248bca231" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=16183600
I0117 18:32:18.025328    1360 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 18:32:18.025843    1360 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="3d050b3d-9695-11ed-80d9-002248bca231" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-52ec6b4d-35d2-4c79-bf70-b4241935f95e" "latency"=14386099500
I0117 18:32:18.026066    1360 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="3d050b3d-9695-11ed-80d9-002248bca231" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=14386719500
I0117 18:32:23.021285    1360 nodeserver_v2.go:211] NodeStageVolume: formatting 10 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-52ec6b4d-35d2-4c79-bf70-b4241935f95e\globalmount with mount options([])
I0117 18:32:25.889676    1360 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 18:32:25.890012    1360 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 18:32:26.347238    1360 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
... skipping 83 lines ...
I0117 18:33:58.297110    1360 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I0117 18:33:58.297110    1360 utils.go:79] GRPC request: {}
I0117 18:33:58.297280    1360 utils.go:85] GRPC response: {"ready":{"value":true}}
I0117 18:34:05.146370    1360 azuredisk_v2.go:578]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-17T18:34:05Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="85719f7c-9695-11ed-80d9-002248bca231"
I0117 18:34:05.157155    1360 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-wto2-qsbsf/status 200 OK in 9 milliseconds
I0117 18:34:05.164855    1360 azuredisk_v2.go:565]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="85719f7c-9695-11ed-80d9-002248bca231" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=18462100
I0117 18:34:06.121439    1360 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 18:34:06.122334    1360 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="80a602b0-9695-11ed-80d9-002248bca231" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-feaf0839-af46-49b1-a7b5-5cd6e4df7d3f" "latency"=9020094000
I0117 18:34:06.122609    1360 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="80a602b0-9695-11ed-80d9-002248bca231" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9021263400
I0117 18:34:10.938306    1360 nodeserver_v2.go:211] NodeStageVolume: formatting 10 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-feaf0839-af46-49b1-a7b5-5cd6e4df7d3f\globalmount with mount options([])
I0117 18:34:23.097203    1360 nodeserver_v2.go:215] NodeStageVolume: format 10 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-feaf0839-af46-49b1-a7b5-5cd6e4df7d3f\globalmount successfully.
I0117 18:34:23.097203    1360 utils.go:85] GRPC response: {}
I0117 18:34:23.104532    1360 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
... skipping 98 lines ...
I0117 18:36:50.900262    1360 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-fd92305e-1233-4e13-9901-1b8cfe378f45-capz-wto2-qsbsf-attachment)
I0117 18:36:55.988303    1360 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 18:36:55.988704    1360 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 18:36:56.439130    1360 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 18:36:56.456141    1360 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync
I0117 18:36:56.583257    1360 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 18:36:56.583257    1360 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0117 18:36:58.299284    1360 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I0117 18:36:58.299354    1360 utils.go:79] GRPC request: {}
I0117 18:36:58.299462    1360 utils.go:85] GRPC response: {"ready":{"value":true}}
I0117 18:36:58.673673    1360 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 18:36:58.673673    1360 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="e83dab7f-9695-11ed-80d9-002248bca231" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-fd92305e-1233-4e13-9901-1b8cfe378f45" "latency"=7773125400
I0117 18:36:58.673673    1360 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="e83dab7f-9695-11ed-80d9-002248bca231" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=7773125400
I0117 18:37:03.474253    1360 nodeserver_v2.go:211] NodeStageVolume: formatting 10 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-fd92305e-1233-4e13-9901-1b8cfe378f45\globalmount with mount options([])
I0117 18:37:05.150962    1360 azuredisk_v2.go:578]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-17T18:37:05Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="f0bc095a-9695-11ed-80d9-002248bca231"
I0117 18:37:05.164230    1360 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-wto2-qsbsf/status 200 OK in 9 milliseconds
I0117 18:37:05.164979    1360 azuredisk_v2.go:565]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="f0bc095a-9695-11ed-80d9-002248bca231" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=14630300
... skipping 68 lines ...
I0117 18:38:34.502465    1360 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0117 18:38:34.502465    1360 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-fd92305e-1233-4e13-9901-1b8cfe378f45\\globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-fd92305e-1233-4e13-9901-1b8cfe378f45","csi.storage.k8s.io/pvc/name":"pvc-dpzgl","csi.storage.k8s.io/pvc/namespace":"azuredisk-1120","requestedsizegib":"10","skuname":"StandardSSD_ZRS","storage.kubernetes.io/csiProvisionerIdentity":"1673977076937-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-wto256/providers/Microsoft.Compute/disks/pvc-fd92305e-1233-4e13-9901-1b8cfe378f45"}
I0117 18:38:34.503273    1360 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-fd92305e-1233-4e13-9901-1b8cfe378f45-capz-wto2-qsbsf-attachment)
I0117 18:38:35.155089    1360 azuredisk_v2.go:578]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-17T18:38:35Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="2661b1ae-9696-11ed-80d9-002248bca231"
I0117 18:38:35.165553    1360 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-wto2-qsbsf/status 200 OK in 9 milliseconds
I0117 18:38:35.166854    1360 azuredisk_v2.go:565]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="2661b1ae-9696-11ed-80d9-002248bca231" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=12001300
I0117 18:38:49.189574    1360 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 18:38:49.189574    1360 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="25fe37ff-9696-11ed-80d9-002248bca231" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-fd92305e-1233-4e13-9901-1b8cfe378f45" "latency"=14685076100
I0117 18:38:49.189574    1360 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="25fe37ff-9696-11ed-80d9-002248bca231" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=14685181700
I0117 18:38:53.949675    1360 nodeserver_v2.go:211] NodeStageVolume: formatting 10 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-fd92305e-1233-4e13-9901-1b8cfe378f45\globalmount with mount options([])
I0117 18:38:56.023433    1360 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 18:38:56.023487    1360 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 18:38:56.476721    1360 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
... skipping 263 lines ...
I0117 18:47:25.750668    1360 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-536e90b4-6184-47b2-9e43-cbadc4af5b18-capz-wto2-qsbsf-attachment)
I0117 18:47:26.151956    1360 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 18:47:26.151956    1360 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 18:47:26.600139    1360 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 18:47:26.615847    1360 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync
I0117 18:47:26.859455    1360 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 18:47:26.859455    1360 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0117 18:47:28.334678    1360 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I0117 18:47:28.334759    1360 utils.go:79] GRPC request: {}
I0117 18:47:28.334884    1360 utils.go:85] GRPC response: {"ready":{"value":true}}
I0117 18:47:30.726980    1360 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 18:47:30.727668    1360 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="62a4253d-9697-11ed-80d9-002248bca231" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-536e90b4-6184-47b2-9e43-cbadc4af5b18" "latency"=4976684200
I0117 18:47:30.727821    1360 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="62a4253d-9697-11ed-80d9-002248bca231" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=4976934500
I0117 18:47:35.193540    1360 azuredisk_v2.go:578]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-17T18:47:35Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="6845030a-9697-11ed-80d9-002248bca231"
I0117 18:47:35.206972    1360 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-wto2-qsbsf/status 200 OK in 13 milliseconds
I0117 18:47:35.208189    1360 azuredisk_v2.go:565]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="6845030a-9697-11ed-80d9-002248bca231" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=14716400
I0117 18:47:35.548111    1360 nodeserver_v2.go:211] NodeStageVolume: formatting 10 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-536e90b4-6184-47b2-9e43-cbadc4af5b18\globalmount with mount options([])
... skipping 33 lines ...
I0117 18:48:13.093559    1360 utils.go:78] GRPC call: /csi.v1.Node/NodeUnstageVolume
I0117 18:48:13.093624    1360 utils.go:79] GRPC request: {"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-536e90b4-6184-47b2-9e43-cbadc4af5b18\\globalmount","volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-wto256/providers/Microsoft.Compute/disks/pvc-536e90b4-6184-47b2-9e43-cbadc4af5b18"}
I0117 18:48:13.093773    1360 nodeserver_v2.go:257] NodeUnstageVolume: unmounting \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-536e90b4-6184-47b2-9e43-cbadc4af5b18\globalmount
I0117 18:48:15.999194    1360 nodeserver_v2.go:262] NodeUnstageVolume: unmount \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-536e90b4-6184-47b2-9e43-cbadc4af5b18\globalmount successfully
I0117 18:48:15.999311    1360 utils.go:85] GRPC response: {}
I0117 18:48:24.198668    1360 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-wto2-qsbsf/status  in 19015 milliseconds
W0117 18:48:24.198668    1360 reflector.go:347] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: watch of *v1beta2.AzDriverNode ended with: an error on the server ("unable to decode an event from the watch stream: read tcp 10.1.0.5:35288->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.") has prevented the request from succeeding
W0117 18:48:24.198668    1360 reflector.go:347] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: watch of *v1beta2.AzDriverNode ended with: an error on the server ("unable to decode an event from the watch stream: read tcp 10.1.0.5:35288->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.") has prevented the request from succeeding
W0117 18:48:24.198668    1360 reflector.go:347] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: watch of *v1beta2.AzVolumeAttachment ended with: an error on the server ("unable to decode an event from the watch stream: read tcp 10.1.0.5:35288->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.") has prevented the request from succeeding
E0117 18:48:24.198668    1360 azuredisk_v2.go:613]  "msg"="Failed to update AzDriverNode status after creation" "error"="Put \"https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-wto2-qsbsf/status\": read tcp 10.1.0.5:35288->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="7a2467bc-9697-11ed-80d9-002248bca231"
W0117 18:48:24.198668    1360 reflector.go:347] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: watch of *v1beta2.AzVolume ended with: an error on the server ("unable to decode an event from the watch stream: read tcp 10.1.0.5:35288->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.") has prevented the request from succeeding
W0117 18:48:24.198668    1360 reflector.go:347] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: watch of *v1.CustomResourceDefinition ended with: an error on the server ("unable to decode an event from the watch stream: read tcp 10.1.0.5:35288->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.") has prevented the request from succeeding
W0117 18:48:24.198668    1360 reflector.go:347] k8s.io/client-go/informers/factory.go:150: watch of *v1.Node ended with: an error on the server ("unable to decode an event from the watch stream: read tcp 10.1.0.5:35288->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.") has prevented the request from succeeding
E0117 18:48:24.198668    1360 azuredisk_v2.go:565]  "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [Put \"https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-wto2-qsbsf/status\": read tcp 10.1.0.5:35288->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.]" "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="7a2467bc-9697-11ed-80d9-002248bca231" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=19019205800
I0117 18:48:25.073566    1360 reflector.go:257] Listing and watching *v1beta2.AzVolumeAttachment from sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132
I0117 18:48:25.088916    1360 round_trippers.go:553] GET https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumeattachments?resourceVersion=25235 200 OK in 14 milliseconds
I0117 18:48:25.089971    1360 round_trippers.go:553] GET https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumeattachments?allowWatchBookmarks=true&resourceVersion=25459&timeout=8m1s&timeoutSeconds=481&watch=true 200 OK in 0 milliseconds
I0117 18:48:25.307120    1360 reflector.go:257] Listing and watching *v1beta2.AzVolume from sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132
I0117 18:48:25.311048    1360 round_trippers.go:553] GET https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumes?resourceVersion=25136 200 OK in 3 milliseconds
I0117 18:48:25.313696    1360 round_trippers.go:553] GET https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumes?allowWatchBookmarks=true&resourceVersion=25136&timeout=5m48s&timeoutSeconds=348&watch=true 200 OK in 0 milliseconds
... skipping 68 lines ...
I0117 18:50:40.097317    1360 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0117 18:50:40.097317    1360 utils.go:79] GRPC request: {}
I0117 18:50:40.097317    1360 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I0117 18:50:40.099212    1360 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0117 18:50:40.099272    1360 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-536e90b4-6184-47b2-9e43-cbadc4af5b18\\globalmount","volume_capability":{"AccessType":{"Mount":{"fs_type":"btrfs"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-536e90b4-6184-47b2-9e43-cbadc4af5b18","csi.storage.k8s.io/pvc/name":"test-volume-1-azuredisk-volume-tester-mmkz8-0","csi.storage.k8s.io/pvc/namespace":"azuredisk-4400","fsType":"btrfs","requestedsizegib":"10","skuname":"StandardSSD_ZRS","storage.kubernetes.io/csiProvisionerIdentity":"1673977076937-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-wto256/providers/Microsoft.Compute/disks/pvc-536e90b4-6184-47b2-9e43-cbadc4af5b18"}
I0117 18:50:40.099737    1360 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-536e90b4-6184-47b2-9e43-cbadc4af5b18-capz-wto2-qsbsf-attachment)
I0117 18:50:49.574565    1360 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 18:50:49.575121    1360 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="d67b740d-9697-11ed-80d9-002248bca231" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-536e90b4-6184-47b2-9e43-cbadc4af5b18" "latency"=9474496600
I0117 18:50:49.575175    1360 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="d67b740d-9697-11ed-80d9-002248bca231" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9475279800
I0117 18:50:54.619035    1360 nodeserver_v2.go:211] NodeStageVolume: formatting 10 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-536e90b4-6184-47b2-9e43-cbadc4af5b18\globalmount with mount options([])
I0117 18:50:55.148670    1360 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 18:50:55.364615    1360 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 18:50:55.498564    1360 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
... skipping 90 lines ...
I0117 18:52:31.372958    1360 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0117 18:52:31.373024    1360 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-27878023-7744-4a1e-872b-e5385d1e30a2\\globalmount","volume_capability":{"AccessType":{"Mount":{"mount_flags":["barrier=1","acl"]}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-27878023-7744-4a1e-872b-e5385d1e30a2","csi.storage.k8s.io/pvc/name":"pvc-wrsnd","csi.storage.k8s.io/pvc/namespace":"azuredisk-1808","requestedsizegib":"10","skuname":"StandardSSD_ZRS","storage.kubernetes.io/csiProvisionerIdentity":"1673977076937-8081-disk.csi.azure.com","tags":"disk=test"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-wto256/providers/Microsoft.Compute/disks/pvc-27878023-7744-4a1e-872b-e5385d1e30a2"}
I0117 18:52:31.373467    1360 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-27878023-7744-4a1e-872b-e5385d1e30a2-capz-wto2-qsbsf-attachment)
I0117 18:52:35.196299    1360 azuredisk_v2.go:578]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-17T18:52:35Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="1b15c27c-9698-11ed-80d9-002248bca231"
I0117 18:52:35.207209    1360 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-wto2-qsbsf/status 200 OK in 10 milliseconds
I0117 18:52:35.209134    1360 azuredisk_v2.go:565]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="1b15c27c-9698-11ed-80d9-002248bca231" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=13166100
I0117 18:52:35.688943    1360 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 18:52:35.698994    1360 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="18ce7a36-9698-11ed-80d9-002248bca231" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-27878023-7744-4a1e-872b-e5385d1e30a2" "latency"=4325300800
I0117 18:52:35.699180    1360 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="18ce7a36-9698-11ed-80d9-002248bca231" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=4325647900
I0117 18:52:40.597106    1360 nodeserver_v2.go:211] NodeStageVolume: formatting 10 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-27878023-7744-4a1e-872b-e5385d1e30a2\globalmount with mount options([barrier=1 acl])
I0117 18:52:55.198505    1360 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 18:52:55.420341    1360 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 18:52:55.546622    1360 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync
... skipping 69 lines ...
I0117 18:54:04.466330    1360 round_trippers.go:553] GET https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/azdrivernodes?allowWatchBookmarks=true&resourceVersion=27079&timeout=8m17s&timeoutSeconds=497&watch=true 200 OK in 3 milliseconds
I0117 18:54:05.186321    1360 azuredisk_v2.go:578]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-17T18:54:05Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="50b932ab-9698-11ed-80d9-002248bca231"
I0117 18:54:05.196847    1360 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-wto2-qsbsf/status 200 OK in 9 milliseconds
I0117 18:54:05.197965    1360 azuredisk_v2.go:565]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="50b932ab-9698-11ed-80d9-002248bca231" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=11515100
I0117 18:54:13.310695    1360 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: Watch close - *v1beta2.AzVolume total 22 items received
I0117 18:54:13.316237    1360 round_trippers.go:553] GET https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumes?allowWatchBookmarks=true&resourceVersion=27139&timeout=7m3s&timeoutSeconds=423&watch=true 200 OK in 5 milliseconds
I0117 18:54:14.115953    1360 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 18:54:14.116097    1360 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="5015a29b-9698-11ed-80d9-002248bca231" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-4fdb34b9-a907-4238-8709-425ed082bc72" "latency"=10001079400
I0117 18:54:14.116187    1360 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="5015a29b-9698-11ed-80d9-002248bca231" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=10001512200
I0117 18:54:18.941196    1360 nodeserver_v2.go:211] NodeStageVolume: formatting 10 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-4fdb34b9-a907-4238-8709-425ed082bc72\globalmount with mount options([barrier=1 acl])
I0117 18:54:25.226057    1360 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 18:54:25.441843    1360 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 18:54:25.567074    1360 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync
... skipping 39 lines ...
I0117 18:54:55.316603    1360 utils.go:79] GRPC request: {"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-4fdb34b9-a907-4238-8709-425ed082bc72\\globalmount","volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-wto256/providers/Microsoft.Compute/disks/pvc-4fdb34b9-a907-4238-8709-425ed082bc72"}
I0117 18:54:55.317218    1360 nodeserver_v2.go:257] NodeUnstageVolume: unmounting \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-4fdb34b9-a907-4238-8709-425ed082bc72\globalmount
I0117 18:54:55.449026    1360 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 18:54:55.569648    1360 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync
I0117 18:54:55.591067    1360 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 18:54:55.879791    1360 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
E0117 18:54:56.199315    1360 utils.go:83] GRPC error: GetVolumeIDFromMount(c:\var\lib\kubelet\pods\c5668f7b-fa22-4080-beac-8870f8fa61bb\volumes\kubernetes.io~csi\pvc-4fdb34b9-a907-4238-8709-425ed082bc72\mount) failed with error: rpc error: code = Unknown desc = error getting the volume for the mount c:\var\lib\kubelet\pods\c5668f7b-fa22-4080-beac-8870f8fa61bb\volumes\kubernetes.io~csi\pvc-4fdb34b9-a907-4238-8709-425ed082bc72\mount, internal error error getting volume from mount. cmd: (Get-Item -Path Get-Item : Cannot find path 'C:\var\lib\kubelet\pods\c5668f7b-fa22-4080-beac-8870f8fa61bb\volumes\kubernetes.io~csi\pvc
-4fdb34b9-a907-4238-8709-425ed082bc72\mount' because it does not exist.
At line:1 char:2
+ (Get-Item -Path c:\var\lib\kubelet\pods\c5668f7b-fa22-4080-beac-8870f ...
+  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : ObjectNotFound: (C:\var\lib\kube...ed082bc72\mount:String) [Get-Item], ItemNotFoundExcep 
   tion
... skipping 37 lines ...
+ ... d : PathNotFound,Microsoft.PowerShell.Commands.GetItemCommand).Target
+                                                                  ~
Unexpected token ')' in expression or statement.
    + CategoryInfo          : ParserError: (:) [], ParentContainsErrorRecordException
    + FullyQualifiedErrorId : MissingEndParenthesisInExpression
 
, error: exit status 1, response: <nil>
I0117 18:54:58.253586    1360 nodeserver_v2.go:262] NodeUnstageVolume: unmount \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-4fdb34b9-a907-4238-8709-425ed082bc72\globalmount successfully
I0117 18:54:58.253586    1360 utils.go:85] GRPC response: {}
I0117 18:54:58.352075    1360 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I0117 18:54:58.352075    1360 utils.go:79] GRPC request: {}
I0117 18:54:58.352075    1360 utils.go:85] GRPC response: {"ready":{"value":true}}
I0117 18:55:05.205795    1360 azuredisk_v2.go:578]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-17T18:55:05Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="747f715f-9698-11ed-80d9-002248bca231"
... skipping 27 lines ...
I0117 18:55:56.051788    1360 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0117 18:55:56.051788    1360 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-bf8b0e3d-bc24-4084-ae3d-94238448b7fc\\globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"cachingmode":"None","csi.storage.k8s.io/pv/name":"pvc-bf8b0e3d-bc24-4084-ae3d-94238448b7fc","csi.storage.k8s.io/pvc/name":"pvc-h4x6n","csi.storage.k8s.io/pvc/namespace":"azuredisk-517","maxshares":"1","requestedsizegib":"10","skuname":"Premium_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1673977076937-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-wto256/providers/Microsoft.Compute/disks/pvc-bf8b0e3d-bc24-4084-ae3d-94238448b7fc"}
I0117 18:55:56.053071    1360 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-bf8b0e3d-bc24-4084-ae3d-94238448b7fc-capz-wto2-qsbsf-attachment)
I0117 18:55:58.359939    1360 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I0117 18:55:58.360009    1360 utils.go:79] GRPC request: {}
I0117 18:55:58.360138    1360 utils.go:85] GRPC response: {"ready":{"value":true}}
I0117 18:56:02.187089    1360 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 18:56:02.187325    1360 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="92ce1a07-9698-11ed-80d9-002248bca231" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-bf8b0e3d-bc24-4084-ae3d-94238448b7fc" "latency"=6133274400
I0117 18:56:02.187384    1360 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="92ce1a07-9698-11ed-80d9-002248bca231" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=6133925100
I0117 18:56:05.201478    1360 azuredisk_v2.go:578]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-17T18:56:05Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="98420713-9698-11ed-80d9-002248bca231"
I0117 18:56:05.219297    1360 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-wto2-qsbsf/status 200 OK in 16 milliseconds
I0117 18:56:05.219297    1360 azuredisk_v2.go:565]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="98420713-9698-11ed-80d9-002248bca231" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=18014100
I0117 18:56:05.758814    1360 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: Watch close - *v1beta2.AzDriverNode total 25 items received
... skipping 166 lines ...
I0117 19:01:42.849389    1360 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0117 19:01:42.849389    1360 utils.go:79] GRPC request: {}
I0117 19:01:42.849389    1360 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I0117 19:01:42.851847    1360 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0117 19:01:42.851847    1360 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-6bc44c07-9782-4672-b136-d12720431f98\\globalmount","volume_capability":{"AccessType":{"Mount":{"mount_flags":["barrier=1","acl"]}},"access_mode":{"mode":7}},"volume_context":{"cachingmode":"None","csi.storage.k8s.io/pv/name":"pvc-6bc44c07-9782-4672-b136-d12720431f98","csi.storage.k8s.io/pvc/name":"pvc-2dths","csi.storage.k8s.io/pvc/namespace":"azuredisk-9710","maxshares":"2","requestedsizegib":"10","skuname":"Premium_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1673977076937-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-wto256/providers/Microsoft.Compute/disks/pvc-6bc44c07-9782-4672-b136-d12720431f98"}
I0117 19:01:42.852179    1360 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-6bc44c07-9782-4672-b136-d12720431f98-capz-wto2-qsbsf-attachment)
I0117 19:01:49.465460    1360 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 19:01:49.466203    1360 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="61837741-9699-11ed-80d9-002248bca231" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-6bc44c07-9782-4672-b136-d12720431f98" "latency"=6612934700
I0117 19:01:49.466539    1360 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="61837741-9699-11ed-80d9-002248bca231" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=6613856800
I0117 19:01:54.212895    1360 nodeserver_v2.go:211] NodeStageVolume: formatting 10 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-6bc44c07-9782-4672-b136-d12720431f98\globalmount with mount options([barrier=1 acl])
I0117 19:01:55.381076    1360 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 19:01:55.608461    1360 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 19:01:55.670028    1360 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync
... skipping 42 lines ...
I0117 19:02:30.849800    1360 utils.go:78] GRPC call: /csi.v1.Node/NodeUnstageVolume
I0117 19:02:30.849849    1360 utils.go:79] GRPC request: {"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-6bc44c07-9782-4672-b136-d12720431f98\\globalmount","volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-wto256/providers/Microsoft.Compute/disks/pvc-6bc44c07-9782-4672-b136-d12720431f98"}
I0117 19:02:30.849898    1360 nodeserver_v2.go:257] NodeUnstageVolume: unmounting \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-6bc44c07-9782-4672-b136-d12720431f98\globalmount
I0117 19:02:33.617569    1360 nodeserver_v2.go:262] NodeUnstageVolume: unmount \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-6bc44c07-9782-4672-b136-d12720431f98\globalmount successfully
I0117 19:02:33.617569    1360 utils.go:85] GRPC response: {}
I0117 19:02:35.227627    1360 azuredisk_v2.go:578]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-17T19:02:35Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="80bb5337-9699-11ed-80d9-002248bca231"
W0117 19:02:54.197608    1360 reflector.go:347] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: watch of *v1beta2.AzDriverNode ended with: an error on the server ("unable to decode an event from the watch stream: read tcp 10.1.0.5:40930->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.") has prevented the request from succeeding
W0117 19:02:54.197608    1360 reflector.go:347] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: watch of *v1.CustomResourceDefinition ended with: an error on the server ("unable to decode an event from the watch stream: read tcp 10.1.0.5:40930->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.") has prevented the request from succeeding
I0117 19:02:54.197608    1360 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-wto2-qsbsf/status  in 18969 milliseconds
W0117 19:02:54.197608    1360 reflector.go:347] k8s.io/client-go/informers/factory.go:150: watch of *v1.Node ended with: an error on the server ("unable to decode an event from the watch stream: read tcp 10.1.0.5:40930->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.") has prevented the request from succeeding
W0117 19:02:54.197608    1360 reflector.go:347] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: watch of *v1beta2.AzVolumeAttachment ended with: an error on the server ("unable to decode an event from the watch stream: read tcp 10.1.0.5:40930->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.") has prevented the request from succeeding
W0117 19:02:54.197608    1360 reflector.go:347] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: watch of *v1beta2.AzVolume ended with: an error on the server ("unable to decode an event from the watch stream: read tcp 10.1.0.5:40930->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.") has prevented the request from succeeding
W0117 19:02:54.197608    1360 reflector.go:347] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: watch of *v1beta2.AzDriverNode ended with: an error on the server ("unable to decode an event from the watch stream: read tcp 10.1.0.5:40930->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.") has prevented the request from succeeding
E0117 19:02:54.197608    1360 azuredisk_v2.go:613]  "msg"="Failed to update AzDriverNode status after creation" "error"="Put \"https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-wto2-qsbsf/status\": read tcp 10.1.0.5:40930->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="80bb5337-9699-11ed-80d9-002248bca231"
E0117 19:02:54.197608    1360 azuredisk_v2.go:565]  "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [Put \"https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-wto2-qsbsf/status\": read tcp 10.1.0.5:40930->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.]" "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="80bb5337-9699-11ed-80d9-002248bca231" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=18969293700
I0117 19:02:55.022783    1360 reflector.go:257] Listing and watching *v1.CustomResourceDefinition from k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132
I0117 19:02:55.048233    1360 reflector.go:257] Listing and watching *v1beta2.AzVolume from sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132
I0117 19:02:55.048233    1360 round_trippers.go:553] GET https://10.96.0.1:443/apis/apiextensions.k8s.io/v1/customresourcedefinitions?resourceVersion=28045 200 OK in 24 milliseconds
I0117 19:02:55.054299    1360 round_trippers.go:553] GET https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumes?resourceVersion=29504 200 OK in 6 milliseconds
I0117 19:02:55.059408    1360 round_trippers.go:553] GET https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumes?allowWatchBookmarks=true&resourceVersion=29835&timeout=6m26s&timeoutSeconds=386&watch=true 200 OK in 1 milliseconds
I0117 19:02:55.101441    1360 round_trippers.go:553] GET https://10.96.0.1:443/apis/apiextensions.k8s.io/v1/customresourcedefinitions?allowWatchBookmarks=true&resourceVersion=28045&timeout=6m41s&timeoutSeconds=401&watch=true 200 OK in 0 milliseconds
... skipping 29 lines ...
I0117 19:03:15.326611    1360 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0117 19:03:15.326611    1360 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-3dfb1469-aa9d-4341-a5d7-a6a5652f8829\\globalmount","volume_capability":{"AccessType":{"Mount":{"mount_flags":["barrier=1","acl"]}},"access_mode":{"mode":7}},"volume_context":{"cachingmode":"None","csi.storage.k8s.io/pv/name":"pvc-3dfb1469-aa9d-4341-a5d7-a6a5652f8829","csi.storage.k8s.io/pvc/name":"pvc-d24vn","csi.storage.k8s.io/pvc/namespace":"azuredisk-9632","maxshares":"2","requestedsizegib":"10","skuname":"Premium_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1673977076937-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-wto256/providers/Microsoft.Compute/disks/pvc-3dfb1469-aa9d-4341-a5d7-a6a5652f8829"}
I0117 19:03:15.327617    1360 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-3dfb1469-aa9d-4341-a5d7-a6a5652f8829-capz-wto2-qsbsf-attachment)
I0117 19:03:25.061146    1360 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 19:03:25.123213    1360 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync
I0117 19:03:25.274896    1360 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 19:03:25.274896    1360 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0117 19:03:25.558377    1360 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 19:03:25.621095    1360 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 19:03:27.928696    1360 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 19:03:27.928696    1360 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="98a20688-9699-11ed-80d9-002248bca231" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-3dfb1469-aa9d-4341-a5d7-a6a5652f8829" "latency"=12600423700
I0117 19:03:27.928696    1360 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="98a20688-9699-11ed-80d9-002248bca231" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=12601074300
I0117 19:03:28.376568    1360 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I0117 19:03:28.376568    1360 utils.go:79] GRPC request: {}
I0117 19:03:28.376568    1360 utils.go:85] GRPC response: {"ready":{"value":true}}
I0117 19:03:32.769094    1360 nodeserver_v2.go:211] NodeStageVolume: formatting 10 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-3dfb1469-aa9d-4341-a5d7-a6a5652f8829\globalmount with mount options([barrier=1 acl])
... skipping 393 lines ...
I0117 19:18:55.381636    1360 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I0117 19:18:55.383633    1360 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0117 19:18:55.383633    1360 utils.go:79] GRPC request: {"publish_context":{"LUN":"1"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-38c31576-b711-4773-9a26-fafa65b97de8\\globalmount","volume_capability":{"AccessType":{"Mount":{"mount_flags":["barrier=1","acl"]}},"access_mode":{"mode":7}},"volume_context":{"cachingmode":"None","csi.storage.k8s.io/pv/name":"pvc-38c31576-b711-4773-9a26-fafa65b97de8","csi.storage.k8s.io/pvc/name":"pvc-vxhqb","csi.storage.k8s.io/pvc/namespace":"azuredisk-5314","maxshares":"2","requestedsizegib":"10","skuname":"Premium_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1673977076937-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-wto256/providers/Microsoft.Compute/disks/pvc-38c31576-b711-4773-9a26-fafa65b97de8"}
I0117 19:18:55.384385    1360 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-38c31576-b711-4773-9a26-fafa65b97de8-capz-wto2-qsbsf-attachment)
I0117 19:18:55.428431    1360 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync
I0117 19:18:55.580678    1360 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 19:18:55.580991    1360 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0117 19:18:55.883262    1360 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 19:18:55.993149    1360 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 19:18:58.421555    1360 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I0117 19:18:58.421612    1360 utils.go:79] GRPC request: {}
I0117 19:18:58.421705    1360 utils.go:85] GRPC response: {"ready":{"value":true}}
I0117 19:19:05.018337    1360 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 19:19:05.018337    1360 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="c8f35e9d-969b-11ed-80d9-002248bca231" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-38c31576-b711-4773-9a26-fafa65b97de8" "latency"=9633541300
I0117 19:19:05.019007    1360 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="c8f35e9d-969b-11ed-80d9-002248bca231" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9634175100
I0117 19:19:05.265832    1360 azuredisk_v2.go:578]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-17T19:19:05Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="ced725c8-969b-11ed-80d9-002248bca231"
I0117 19:19:05.277397    1360 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-wto2-qsbsf/status 200 OK in 10 milliseconds
I0117 19:19:05.277397    1360 azuredisk_v2.go:565]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="ced725c8-969b-11ed-80d9-002248bca231" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=11687700
I0117 19:19:09.807777    1360 nodeserver_v2.go:211] NodeStageVolume: formatting 11 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-38c31576-b711-4773-9a26-fafa65b97de8\globalmount with mount options([barrier=1 acl])
... skipping 44 lines ...
I0117 19:19:56.008246    1360 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 19:19:58.420277    1360 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I0117 19:19:58.420277    1360 utils.go:79] GRPC request: {}
I0117 19:19:58.420277    1360 utils.go:85] GRPC response: {"ready":{"value":true}}
I0117 19:20:05.264260    1360 azuredisk_v2.go:578]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-17T19:20:05Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="f29a2de1-969b-11ed-80d9-002248bca231"
I0117 19:20:20.333752    1360 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-wto2-qsbsf/status  in 15067 milliseconds
W0117 19:20:20.333793    1360 reflector.go:347] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: watch of *v1beta2.AzDriverNode ended with: an error on the server ("unable to decode an event from the watch stream: http2: client connection lost") has prevented the request from succeeding
E0117 19:20:20.333793    1360 azuredisk_v2.go:613]  "msg"="Failed to update AzDriverNode status after creation" "error"="Put \"https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-wto2-qsbsf/status\": http2: client connection lost" "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="f29a2de1-969b-11ed-80d9-002248bca231"
W0117 19:20:20.333793    1360 reflector.go:347] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: watch of *v1beta2.AzVolumeAttachment ended with: an error on the server ("unable to decode an event from the watch stream: http2: client connection lost") has prevented the request from succeeding
W0117 19:20:20.333793    1360 reflector.go:347] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: watch of *v1beta2.AzVolume ended with: an error on the server ("unable to decode an event from the watch stream: http2: client connection lost") has prevented the request from succeeding
E0117 19:20:20.333793    1360 azuredisk_v2.go:565]  "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [Put \"https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-wto2-qsbsf/status\": http2: client connection lost]" "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="f29a2de1-969b-11ed-80d9-002248bca231" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=15069072300
W0117 19:20:20.333793    1360 reflector.go:347] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: watch of *v1beta2.AzDriverNode ended with: an error on the server ("unable to decode an event from the watch stream: http2: client connection lost") has prevented the request from succeeding
W0117 19:20:20.333793    1360 reflector.go:347] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: watch of *v1.CustomResourceDefinition ended with: an error on the server ("unable to decode an event from the watch stream: http2: client connection lost") has prevented the request from succeeding
W0117 19:20:20.333793    1360 reflector.go:347] k8s.io/client-go/informers/factory.go:150: watch of *v1.Node ended with: an error on the server ("unable to decode an event from the watch stream: http2: client connection lost") has prevented the request from succeeding
I0117 19:20:21.158908    1360 reflector.go:257] Listing and watching *v1beta2.AzDriverNode from sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132
I0117 19:20:21.171262    1360 round_trippers.go:553] GET https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/azdrivernodes?resourceVersion=34573 200 OK in 11 milliseconds
I0117 19:20:21.177284    1360 round_trippers.go:553] GET https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/azdrivernodes?allowWatchBookmarks=true&resourceVersion=34814&timeout=5m29s&timeoutSeconds=329&watch=true 200 OK in 2 milliseconds
I0117 19:20:21.346671    1360 reflector.go:257] Listing and watching *v1.Node from k8s.io/client-go/informers/factory.go:150
I0117 19:20:21.351153    1360 round_trippers.go:553] GET https://10.96.0.1:443/api/v1/nodes?resourceVersion=34400 200 OK in 3 milliseconds
I0117 19:20:21.355335    1360 round_trippers.go:553] GET https://10.96.0.1:443/api/v1/nodes?allowWatchBookmarks=true&resourceVersion=34772&timeout=9m7s&timeoutSeconds=547&watch=true 200 OK in 1 milliseconds
... skipping 37 lines ...
I0117 19:21:09.534488    1360 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0117 19:21:09.534488    1360 utils.go:79] GRPC request: {}
I0117 19:21:09.535018    1360 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I0117 19:21:09.536758    1360 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0117 19:21:09.536758    1360 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-4863675b-c2dc-43eb-ad90-b8a36d33bf07\\globalmount","volume_capability":{"AccessType":{"Mount":{"mount_flags":["barrier=1","acl"]}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-4863675b-c2dc-43eb-ad90-b8a36d33bf07","csi.storage.k8s.io/pvc/name":"pvc-8fphq","csi.storage.k8s.io/pvc/namespace":"azuredisk-6257","requestedsizegib":"10","skuname":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1673977076937-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-wto256/providers/Microsoft.Compute/disks/pvc-4863675b-c2dc-43eb-ad90-b8a36d33bf07"}
I0117 19:21:09.538422    1360 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-4863675b-c2dc-43eb-ad90-b8a36d33bf07-capz-wto2-qsbsf-attachment)
I0117 19:21:19.086885    1360 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 19:21:19.086885    1360 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="18e9a2ca-969c-11ed-80d9-002248bca231" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-4863675b-c2dc-43eb-ad90-b8a36d33bf07" "latency"=9548109300
I0117 19:21:19.088454    1360 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="18e9a2ca-969c-11ed-80d9-002248bca231" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9548258900
I0117 19:21:21.195678    1360 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 19:21:21.406410    1360 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 19:21:21.834806    1360 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 19:21:21.916441    1360 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
... skipping 63 lines ...
I0117 19:22:15.049373    1360 nodeserver_v2.go:369] NodeUnpublishVolume: unmounting volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-wto256/providers/Microsoft.Compute/disks/pvc-4863675b-c2dc-43eb-ad90-b8a36d33bf07 on c:\var\lib\kubelet\pods\ad734b2b-b918-4cbe-ac5b-145aafd99c05\volumes\kubernetes.io~csi\pvc-4863675b-c2dc-43eb-ad90-b8a36d33bf07\mount
I0117 19:22:18.163986    1360 nodeserver_v2.go:375] NodeUnpublishVolume: unmount volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-wto256/providers/Microsoft.Compute/disks/pvc-4863675b-c2dc-43eb-ad90-b8a36d33bf07 on c:\var\lib\kubelet\pods\ad734b2b-b918-4cbe-ac5b-145aafd99c05\volumes\kubernetes.io~csi\pvc-4863675b-c2dc-43eb-ad90-b8a36d33bf07\mount successfully
I0117 19:22:18.164788    1360 utils.go:85] GRPC response: {}
I0117 19:22:18.694907    1360 utils.go:78] GRPC call: /csi.v1.Node/NodeUnpublishVolume
I0117 19:22:18.695196    1360 utils.go:79] GRPC request: {"target_path":"c:\\var\\lib\\kubelet\\pods\\ad734b2b-b918-4cbe-ac5b-145aafd99c05\\volumes\\kubernetes.io~csi\\pvc-4863675b-c2dc-43eb-ad90-b8a36d33bf07\\mount","volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-wto256/providers/Microsoft.Compute/disks/pvc-4863675b-c2dc-43eb-ad90-b8a36d33bf07"}
I0117 19:22:18.695196    1360 nodeserver_v2.go:369] NodeUnpublishVolume: unmounting volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-wto256/providers/Microsoft.Compute/disks/pvc-4863675b-c2dc-43eb-ad90-b8a36d33bf07 on c:\var\lib\kubelet\pods\ad734b2b-b918-4cbe-ac5b-145aafd99c05\volumes\kubernetes.io~csi\pvc-4863675b-c2dc-43eb-ad90-b8a36d33bf07\mount
W0117 19:22:19.537724    1360 safe_mounter_windows.go:113] GetVolumeIDFromTargetPath(c:\var\lib\kubelet\pods\ad734b2b-b918-4cbe-ac5b-145aafd99c05\volumes\kubernetes.io~csi\pvc-4863675b-c2dc-43eb-ad90-b8a36d33bf07\mount) failed with error: rpc error: code = Unknown desc = error getting the volume for the mount c:\var\lib\kubelet\pods\ad734b2b-b918-4cbe-ac5b-145aafd99c05\volumes\kubernetes.io~csi\pvc-4863675b-c2dc-43eb-ad90-b8a36d33bf07\mount, internal error error getting volume from mount. cmd: (Get-Item -Path Get-Item : Cannot find path 'C:\var\lib\kubelet\pods\ad734b2b-b918-4cbe-ac5b-145aafd99c05\volumes\kubernetes.io~csi\pvc
-4863675b-c2dc-43eb-ad90-b8a36d33bf07\mount' because it does not exist.
At line:1 char:2
+ (Get-Item -Path c:\var\lib\kubelet\pods\ad734b2b-b918-4cbe-ac5b-145aa ...
+  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : ObjectNotFound: (C:\var\lib\kube...36d33bf07\mount:String) [Get-Item], ItemNotFoundExcep 
   tion
... skipping 37 lines ...
+ ... d : PathNotFound,Microsoft.PowerShell.Commands.GetItemCommand).Target
+                                                                  ~
Unexpected token ')' in expression or statement.
    + CategoryInfo          : ParserError: (:) [], ParentContainsErrorRecordException
    + FullyQualifiedErrorId : MissingEndParenthesisInExpression
 
, error: exit status 1, response: <nil>
I0117 19:22:19.538782    1360 nodeserver_v2.go:375] NodeUnpublishVolume: unmount volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-wto256/providers/Microsoft.Compute/disks/pvc-4863675b-c2dc-43eb-ad90-b8a36d33bf07 on c:\var\lib\kubelet\pods\ad734b2b-b918-4cbe-ac5b-145aafd99c05\volumes\kubernetes.io~csi\pvc-4863675b-c2dc-43eb-ad90-b8a36d33bf07\mount successfully
I0117 19:22:19.538834    1360 utils.go:85] GRPC response: {}
I0117 19:22:19.562779    1360 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0117 19:22:19.562779    1360 utils.go:79] GRPC request: {}
I0117 19:22:19.562779    1360 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I0117 19:22:19.564036    1360 utils.go:78] GRPC call: /csi.v1.Node/NodeUnstageVolume
... skipping 80 lines ...
I0117 19:25:13.840281    1360 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-ec142684-4e44-4610-9f4d-c16f20fae958\\globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-ec142684-4e44-4610-9f4d-c16f20fae958","csi.storage.k8s.io/pvc/name":"pvc-azuredisk","csi.storage.k8s.io/pvc/namespace":"default","requestedsizegib":"10","skuName":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1673977076937-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-wto256/providers/Microsoft.Compute/disks/pvc-ec142684-4e44-4610-9f4d-c16f20fae958"}
I0117 19:25:13.840740    1360 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-ec142684-4e44-4610-9f4d-c16f20fae958-capz-wto2-qsbsf-attachment)
I0117 19:25:21.273793    1360 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 19:25:21.460592    1360 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 19:25:21.917467    1360 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 19:25:21.980262    1360 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 19:25:21.980581    1360 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0117 19:25:22.037867    1360 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync
I0117 19:25:23.536156    1360 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 19:25:23.536156    1360 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-wto2-qsbsf" "disk.csi.azure.com/request-id"="aa87396b-969c-11ed-80d9-002248bca231" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-ec142684-4e44-4610-9f4d-c16f20fae958" "latency"=9694981100
I0117 19:25:23.536156    1360 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="aa87396b-969c-11ed-80d9-002248bca231" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9695152600
I0117 19:25:28.434139    1360 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I0117 19:25:28.434139    1360 utils.go:79] GRPC request: {}
I0117 19:25:28.434139    1360 utils.go:85] GRPC response: {"ready":{"value":true}}
I0117 19:25:31.127909    1360 nodeserver_v2.go:211] NodeStageVolume: formatting 10 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-ec142684-4e44-4610-9f4d-c16f20fae958\globalmount with mount options([])
... skipping 104 lines ...
I0117 17:38:05.146810    3820 round_trippers.go:553] GET https://10.96.0.1:443/apis/apiextensions.k8s.io/v1/customresourcedefinitions?limit=500&resourceVersion=0 200 OK in 40 milliseconds
I0117 17:38:05.176791    3820 round_trippers.go:553] GET https://10.96.0.1:443/apis/apiextensions.k8s.io/v1/customresourcedefinitions?allowWatchBookmarks=true&resourceVersion=3503&timeout=8m7s&timeoutSeconds=487&watch=true 200 OK in 0 milliseconds
I0117 17:38:05.216170    3820 shared_informer.go:303] caches populated
I0117 17:38:05.216170    3820 azuredisk_v2.go:225] driver userAgent: disk.csi.azure.com/latest-v2-9ef068a8cb36a997d4ea04b90c05c6f92a488a19 e2e-test
I0117 17:38:05.216259    3820 azure_disk_utils.go:564] reading cloud config from secret kube-system/azure-cloud-provider
I0117 17:38:05.222017    3820 round_trippers.go:553] GET https://10.96.0.1:443/api/v1/namespaces/kube-system/secrets/azure-cloud-provider 404 Not Found in 3 milliseconds
I0117 17:38:05.226355    3820 azure_disk_utils.go:571] InitializeCloudFromSecret: failed to get cloud config from secret kube-system/azure-cloud-provider: failed to get secret kube-system/azure-cloud-provider: secrets "azure-cloud-provider" not found
I0117 17:38:05.226355    3820 azure_disk_utils.go:576] could not read cloud config from secret kube-system/azure-cloud-provider
I0117 17:38:05.226355    3820 azure_disk_utils.go:586] use default AZURE_CREDENTIAL_FILE env var: C:\k\azure.json
I0117 17:38:05.226355    3820 azure_disk_utils.go:594] read cloud config from file: C:\k\azure.json successfully
I0117 17:38:05.228308    3820 azure_auth.go:253] Using AzurePublicCloud environment
I0117 17:38:05.228308    3820 azure_auth.go:138] azure: using client_id+client_secret to retrieve access token
I0117 17:38:05.231883    3820 azure_diskclient.go:68] Azure DisksClient using API version: 2022-03-02
... skipping 30 lines ...
I0117 17:38:05.395951    3820 azuredisk_v2.go:528] Starting heartbeat loop with initial delay 8.064s and frequency 30s
I0117 17:38:05.704772    3820 utils.go:78] GRPC call: /csi.v1.Identity/GetPluginInfo
I0117 17:38:05.704826    3820 utils.go:79] GRPC request: {}
I0117 17:38:05.707452    3820 utils.go:85] GRPC response: {"name":"disk.csi.azure.com","vendor_version":"latest-v2-9ef068a8cb36a997d4ea04b90c05c6f92a488a19"}
I0117 17:38:06.254480    3820 utils.go:78] GRPC call: /csi.v1.Node/NodeGetInfo
I0117 17:38:06.254480    3820 utils.go:79] GRPC request: {}
I0117 17:38:06.273579    3820 azure_instance_metadata.go:140] Warning: failed to get loadbalancer metadata: failure of getting loadbalancer metadata with response "404 Not Found"
I0117 17:38:06.273579    3820 nodeserver_v2.go:438] NodeGetInfo: nodeName(capz-wto2-brk9b), VM Size(Standard_D4s_v3)
I0117 17:38:06.273579    3820 nodeserver.go:409] got a matching size in getMaxDataDiskCount, VM Size: STANDARD_D4S_V3, MaxDataDiskCount: 8
I0117 17:38:06.273579    3820 utils.go:85] GRPC response: {"accessible_topology":{"segments":{"topology.disk.csi.azure.com/zone":""}},"max_volumes_per_node":8,"node_id":"capz-wto2-brk9b"}
I0117 17:38:06.318347    3820 utils.go:78] GRPC call: /csi.v1.Identity/GetPluginInfo
I0117 17:38:06.318347    3820 utils.go:79] GRPC request: {}
I0117 17:38:06.318347    3820 utils.go:85] GRPC response: {"name":"disk.csi.azure.com","vendor_version":"latest-v2-9ef068a8cb36a997d4ea04b90c05c6f92a488a19"}
... skipping 2597 lines ...
I0117 17:37:54.567953    5412 round_trippers.go:553] GET https://10.96.0.1:443/apis/apiextensions.k8s.io/v1/customresourcedefinitions?limit=500&resourceVersion=0 200 OK in 884 milliseconds
I0117 17:37:54.608257    5412 round_trippers.go:553] GET https://10.96.0.1:443/apis/apiextensions.k8s.io/v1/customresourcedefinitions?allowWatchBookmarks=true&resourceVersion=3503&timeout=7m17s&timeoutSeconds=437&watch=true 200 OK in 5 milliseconds
I0117 17:37:54.687879    5412 shared_informer.go:303] caches populated
I0117 17:37:54.687879    5412 azuredisk_v2.go:225] driver userAgent: disk.csi.azure.com/latest-v2-9ef068a8cb36a997d4ea04b90c05c6f92a488a19 e2e-test
I0117 17:37:54.687879    5412 azure_disk_utils.go:564] reading cloud config from secret kube-system/azure-cloud-provider
I0117 17:37:54.725253    5412 round_trippers.go:553] GET https://10.96.0.1:443/api/v1/namespaces/kube-system/secrets/azure-cloud-provider 404 Not Found in 35 milliseconds
I0117 17:37:54.725780    5412 azure_disk_utils.go:571] InitializeCloudFromSecret: failed to get cloud config from secret kube-system/azure-cloud-provider: failed to get secret kube-system/azure-cloud-provider: secrets "azure-cloud-provider" not found
I0117 17:37:54.725780    5412 azure_disk_utils.go:576] could not read cloud config from secret kube-system/azure-cloud-provider
I0117 17:37:54.725780    5412 azure_disk_utils.go:586] use default AZURE_CREDENTIAL_FILE env var: C:\k\azure.json
I0117 17:37:54.725780    5412 azure_disk_utils.go:594] read cloud config from file: C:\k\azure.json successfully
I0117 17:37:54.727269    5412 azure_auth.go:253] Using AzurePublicCloud environment
I0117 17:37:54.727768    5412 azure_auth.go:138] azure: using client_id+client_secret to retrieve access token
I0117 17:37:54.731262    5412 azure_diskclient.go:68] Azure DisksClient using API version: 2022-03-02
... skipping 33 lines ...
I0117 17:37:55.376603    5412 utils.go:85] GRPC response: {"name":"disk.csi.azure.com","vendor_version":"latest-v2-9ef068a8cb36a997d4ea04b90c05c6f92a488a19"}
I0117 17:37:55.402099    5412 utils.go:78] GRPC call: /csi.v1.Identity/GetPluginInfo
I0117 17:37:55.402099    5412 utils.go:79] GRPC request: {}
I0117 17:37:55.402099    5412 utils.go:85] GRPC response: {"name":"disk.csi.azure.com","vendor_version":"latest-v2-9ef068a8cb36a997d4ea04b90c05c6f92a488a19"}
I0117 17:37:56.040192    5412 utils.go:78] GRPC call: /csi.v1.Node/NodeGetInfo
I0117 17:37:56.040192    5412 utils.go:79] GRPC request: {}
I0117 17:37:56.057673    5412 azure_instance_metadata.go:140] Warning: failed to get loadbalancer metadata: failure of getting loadbalancer metadata with response "404 Not Found"
I0117 17:37:56.058171    5412 nodeserver_v2.go:438] NodeGetInfo: nodeName(capz-wto2-xzbdk), VM Size(Standard_D4s_v3)
I0117 17:37:56.058171    5412 nodeserver.go:409] got a matching size in getMaxDataDiskCount, VM Size: STANDARD_D4S_V3, MaxDataDiskCount: 8
I0117 17:37:56.058171    5412 utils.go:85] GRPC response: {"accessible_topology":{"segments":{"topology.disk.csi.azure.com/zone":""}},"max_volumes_per_node":8,"node_id":"capz-wto2-xzbdk"}
I0117 17:38:23.718065    5412 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 17:38:23.718065    5412 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 17:38:23.718065    5412 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
... skipping 1443 lines ...
I0117 18:39:19.468266    5412 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0117 18:39:19.468266    5412 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-a6ced79f-41ba-4074-bd5d-b9c75f2d7a3c\\globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-a6ced79f-41ba-4074-bd5d-b9c75f2d7a3c","csi.storage.k8s.io/pvc/name":"pvc-j5h6n","csi.storage.k8s.io/pvc/namespace":"azuredisk-1120","requestedsizegib":"10","skuname":"StandardSSD_ZRS","storage.kubernetes.io/csiProvisionerIdentity":"1673977076937-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-wto256/providers/Microsoft.Compute/disks/pvc-a6ced79f-41ba-4074-bd5d-b9c75f2d7a3c"}
I0117 18:39:19.468774    5412 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-a6ced79f-41ba-4074-bd5d-b9c75f2d7a3c-capz-wto2-xzbdk-attachment)
I0117 18:39:20.488798    5412 azuredisk_v2.go:578]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-17T18:39:20Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-wto2-xzbdk" "disk.csi.azure.com/request-id"="416711b4-9696-11ed-86d9-002248c1a32d"
I0117 18:39:20.500734    5412 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-wto2-xzbdk/status 200 OK in 11 milliseconds
I0117 18:39:20.500734    5412 azuredisk_v2.go:565]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-wto2-xzbdk" "disk.csi.azure.com/request-id"="416711b4-9696-11ed-86d9-002248c1a32d" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=11935300
I0117 18:39:24.002317    5412 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 18:39:24.002317    5412 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-wto2-xzbdk" "disk.csi.azure.com/request-id"="40cb592d-9696-11ed-86d9-002248c1a32d" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-a6ced79f-41ba-4074-bd5d-b9c75f2d7a3c" "latency"=4533533200
I0117 18:39:24.002317    5412 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="40cb592d-9696-11ed-86d9-002248c1a32d" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=4534040700
I0117 18:39:24.647706    5412 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 18:39:24.647706    5412 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 18:39:24.647706    5412 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 18:39:25.530030    5412 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync
... skipping 204 lines ...
I0117 18:45:20.489866    5412 azuredisk_v2.go:578]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-17T18:45:20Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-wto2-xzbdk" "disk.csi.azure.com/request-id"="17fadd9e-9697-11ed-86db-002248c1a32d"
I0117 18:45:20.500116    5412 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-wto2-xzbdk/status 200 OK in 10 milliseconds
I0117 18:45:20.500116    5412 azuredisk_v2.go:565]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-wto2-xzbdk" "disk.csi.azure.com/request-id"="17fadd9e-9697-11ed-86db-002248c1a32d" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=10295600
I0117 18:45:24.734744    5412 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 18:45:24.734744    5412 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 18:45:24.734744    5412 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 18:45:24.734744    5412 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0117 18:45:24.734744    5412 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0117 18:45:24.734744    5412 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0117 18:45:24.870067    5412 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 18:45:24.870067    5412 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-wto2-xzbdk" "disk.csi.azure.com/request-id"="161aff9b-9697-11ed-86db-002248c1a32d" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-91229a00-6f8e-4615-8de3-8a315518c670" "latency"=7525089700
I0117 18:45:24.870067    5412 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="161aff9b-9697-11ed-86db-002248c1a32d" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=7525089700
I0117 18:45:24.882071    5412 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 18:45:24.882071    5412 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-wto2-xzbdk" "disk.csi.azure.com/request-id"="161aff9b-9697-11ed-86d9-002248c1a32d" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-5bff7770-0299-43b7-b41e-e259c7bba48f" "latency"=7537094000
I0117 18:45:24.882071    5412 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="161aff9b-9697-11ed-86d9-002248c1a32d" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=7537094000
I0117 18:45:24.884567    5412 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 18:45:24.884567    5412 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-wto2-xzbdk" "disk.csi.azure.com/request-id"="161aff9b-9697-11ed-86da-002248c1a32d" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-13af6280-6d58-408c-aed8-d29007c071ff" "latency"=7539590100
I0117 18:45:24.884567    5412 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="161aff9b-9697-11ed-86da-002248c1a32d" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=7539590100
I0117 18:45:25.586740    5412 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync
I0117 18:45:25.745036    5412 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 18:45:27.838072    5412 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I0117 18:45:27.838072    5412 utils.go:79] GRPC request: {}
... skipping 433 lines ...
I0117 18:58:25.736338    5412 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync
I0117 18:58:25.906231    5412 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 18:58:27.836279    5412 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I0117 18:58:27.836279    5412 utils.go:79] GRPC request: {}
I0117 18:58:27.836279    5412 utils.go:85] GRPC response: {"ready":{"value":true}}
I0117 18:58:39.519219    5412 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-wto2-xzbdk/status  in 19018 milliseconds
W0117 18:58:39.520456    5412 reflector.go:347] k8s.io/client-go/informers/factory.go:150: watch of *v1.Node ended with: an error on the server ("unable to decode an event from the watch stream: read tcp 10.1.0.4:34573->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.") has prevented the request from succeeding
W0117 18:58:39.520944    5412 reflector.go:347] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: watch of *v1beta2.AzDriverNode ended with: an error on the server ("unable to decode an event from the watch stream: read tcp 10.1.0.4:34573->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.") has prevented the request from succeeding
W0117 18:58:39.520944    5412 reflector.go:347] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: watch of *v1beta2.AzVolume ended with: an error on the server ("unable to decode an event from the watch stream: read tcp 10.1.0.4:34573->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.") has prevented the request from succeeding
E0117 18:58:39.520944    5412 azuredisk_v2.go:613]  "msg"="Failed to update AzDriverNode status after creation" "error"="Put \"https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-wto2-xzbdk/status\": read tcp 10.1.0.4:34573->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-wto2-xzbdk" "disk.csi.azure.com/request-id"="e8e7142b-9698-11ed-86db-002248c1a32d"
W0117 18:58:39.520944    5412 reflector.go:347] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: watch of *v1beta2.AzVolumeAttachment ended with: an error on the server ("unable to decode an event from the watch stream: read tcp 10.1.0.4:34573->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.") has prevented the request from succeeding
W0117 18:58:39.520944    5412 reflector.go:347] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: watch of *v1.CustomResourceDefinition ended with: an error on the server ("unable to decode an event from the watch stream: read tcp 10.1.0.4:34573->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.") has prevented the request from succeeding
W0117 18:58:39.520944    5412 reflector.go:347] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: watch of *v1beta2.AzDriverNode ended with: an error on the server ("unable to decode an event from the watch stream: read tcp 10.1.0.4:34573->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.") has prevented the request from succeeding
E0117 18:58:39.520944    5412 azuredisk_v2.go:565]  "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [Put \"https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-wto2-xzbdk/status\": read tcp 10.1.0.4:34573->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.]" "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-wto2-xzbdk" "disk.csi.azure.com/request-id"="e8e7142b-9698-11ed-86db-002248c1a32d" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=19020217500
I0117 18:58:40.807883    5412 reflector.go:257] Listing and watching *v1beta2.AzVolume from sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132
I0117 18:58:40.818799    5412 round_trippers.go:553] GET https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumes?resourceVersion=28220 200 OK in 10 milliseconds
I0117 18:58:40.820945    5412 round_trippers.go:553] GET https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumes?allowWatchBookmarks=true&resourceVersion=28510&timeout=7m29s&timeoutSeconds=449&watch=true 200 OK in 1 milliseconds
I0117 18:58:40.837945    5412 reflector.go:257] Listing and watching *v1.Node from k8s.io/client-go/informers/factory.go:150
I0117 18:58:40.837945    5412 reflector.go:257] Listing and watching *v1.CustomResourceDefinition from k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132
I0117 18:58:40.846446    5412 round_trippers.go:553] GET https://10.96.0.1:443/api/v1/nodes?resourceVersion=28297 200 OK in 7 milliseconds
... skipping 258 lines ...
I0117 19:04:53.560560    5412 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0117 19:04:53.560560    5412 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-c8bcc5c7-0592-4e0c-ae67-3dd7a693dd92\\globalmount","volume_capability":{"AccessType":{"Mount":{"mount_flags":["barrier=1","acl"]}},"access_mode":{"mode":7}},"volume_context":{"cachingmode":"None","csi.storage.k8s.io/pv/name":"pvc-c8bcc5c7-0592-4e0c-ae67-3dd7a693dd92","csi.storage.k8s.io/pvc/name":"pvc-ph4m6","csi.storage.k8s.io/pvc/namespace":"azuredisk-1632","maxshares":"2","requestedsizegib":"10","skuname":"Premium_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1673977076937-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-wto256/providers/Microsoft.Compute/disks/pvc-c8bcc5c7-0592-4e0c-ae67-3dd7a693dd92"}
I0117 19:04:53.560560    5412 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-c8bcc5c7-0592-4e0c-ae67-3dd7a693dd92-capz-wto2-xzbdk-attachment)
I0117 19:04:57.834391    5412 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I0117 19:04:57.834391    5412 utils.go:79] GRPC request: {}
I0117 19:04:57.834391    5412 utils.go:85] GRPC response: {"ready":{"value":true}}
I0117 19:05:02.042741    5412 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 19:05:02.042741    5412 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-wto2-xzbdk" "disk.csi.azure.com/request-id"="d32f4164-9699-11ed-86db-002248c1a32d" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-c8bcc5c7-0592-4e0c-ae67-3dd7a693dd92" "latency"=8482163900
I0117 19:05:02.042741    5412 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="d32f4164-9699-11ed-86db-002248c1a32d" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=8482163900
I0117 19:05:06.094246    5412 nodeserver_v2.go:211] NodeStageVolume: formatting 10 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-c8bcc5c7-0592-4e0c-ae67-3dd7a693dd92\globalmount with mount options([barrier=1 acl])
I0117 19:05:10.958815    5412 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 19:05:10.958815    5412 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 19:05:11.005264    5412 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync
... skipping 41 lines ...
I0117 19:05:44.996194    5412 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I0117 19:05:44.997210    5412 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0117 19:05:44.997210    5412 utils.go:79] GRPC request: {}
I0117 19:05:44.997210    5412 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I0117 19:05:44.998220    5412 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0117 19:05:44.998220    5412 utils.go:79] GRPC request: {"publish_context":{"LUN":"1"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-37c0c80a-58bd-419a-9ef9-f3805fdc51d8\\globalmount","volume_capability":{"AccessType":{"Mount":{"mount_flags":["barrier=1","acl"]}},"access_mode":{"mode":7}},"volume_context":{"cachingmode":"None","csi.storage.k8s.io/pv/name":"pvc-37c0c80a-58bd-419a-9ef9-f3805fdc51d8","csi.storage.k8s.io/pvc/name":"pvc-vlc2v","csi.storage.k8s.io/pvc/namespace":"azuredisk-1632","maxshares":"2","requestedsizegib":"10","skuname":"Premium_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1673977076937-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-wto256/providers/Microsoft.Compute/disks/pvc-37c0c80a-58bd-419a-9ef9-f3805fdc51d8"}
E0117 19:05:44.998704    5412 utils.go:83] GRPC error: rpc error: code = Internal desc = failed to get current attachment for volume (/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-wto256/providers/Microsoft.Compute/disks/pvc-37c0c80a-58bd-419a-9ef9-f3805fdc51d8) and node (capz-wto2-xzbdk): azvolumeattachment.disk.csi.azure.com "pvc-37c0c80a-58bd-419a-9ef9-f3805fdc51d8-capz-wto2-xzbdk-attachment" not found
I0117 19:05:45.523404    5412 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0117 19:05:45.523404    5412 utils.go:79] GRPC request: {}
I0117 19:05:45.523404    5412 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I0117 19:05:45.527915    5412 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0117 19:05:45.527915    5412 utils.go:79] GRPC request: {}
I0117 19:05:45.527915    5412 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I0117 19:05:45.528919    5412 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0117 19:05:45.528919    5412 utils.go:79] GRPC request: {}
I0117 19:05:45.528919    5412 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I0117 19:05:45.529899    5412 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0117 19:05:45.529899    5412 utils.go:79] GRPC request: {"publish_context":{"LUN":"1"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-37c0c80a-58bd-419a-9ef9-f3805fdc51d8\\globalmount","volume_capability":{"AccessType":{"Mount":{"mount_flags":["barrier=1","acl"]}},"access_mode":{"mode":7}},"volume_context":{"cachingmode":"None","csi.storage.k8s.io/pv/name":"pvc-37c0c80a-58bd-419a-9ef9-f3805fdc51d8","csi.storage.k8s.io/pvc/name":"pvc-vlc2v","csi.storage.k8s.io/pvc/namespace":"azuredisk-1632","maxshares":"2","requestedsizegib":"10","skuname":"Premium_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1673977076937-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-wto256/providers/Microsoft.Compute/disks/pvc-37c0c80a-58bd-419a-9ef9-f3805fdc51d8"}
E0117 19:05:45.529899    5412 utils.go:83] GRPC error: rpc error: code = Internal desc = failed to get current attachment for volume (/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-wto256/providers/Microsoft.Compute/disks/pvc-37c0c80a-58bd-419a-9ef9-f3805fdc51d8) and node (capz-wto2-xzbdk): azvolumeattachment.disk.csi.azure.com "pvc-37c0c80a-58bd-419a-9ef9-f3805fdc51d8-capz-wto2-xzbdk-attachment" not found
I0117 19:05:46.591317    5412 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0117 19:05:46.591317    5412 utils.go:79] GRPC request: {}
I0117 19:05:46.591317    5412 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I0117 19:05:46.595315    5412 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0117 19:05:46.595315    5412 utils.go:79] GRPC request: {}
I0117 19:05:46.595816    5412 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I0117 19:05:46.596327    5412 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0117 19:05:46.596327    5412 utils.go:79] GRPC request: {}
I0117 19:05:46.596815    5412 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I0117 19:05:46.597814    5412 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0117 19:05:46.597814    5412 utils.go:79] GRPC request: {"publish_context":{"LUN":"1"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-37c0c80a-58bd-419a-9ef9-f3805fdc51d8\\globalmount","volume_capability":{"AccessType":{"Mount":{"mount_flags":["barrier=1","acl"]}},"access_mode":{"mode":7}},"volume_context":{"cachingmode":"None","csi.storage.k8s.io/pv/name":"pvc-37c0c80a-58bd-419a-9ef9-f3805fdc51d8","csi.storage.k8s.io/pvc/name":"pvc-vlc2v","csi.storage.k8s.io/pvc/namespace":"azuredisk-1632","maxshares":"2","requestedsizegib":"10","skuname":"Premium_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1673977076937-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-wto256/providers/Microsoft.Compute/disks/pvc-37c0c80a-58bd-419a-9ef9-f3805fdc51d8"}
E0117 19:05:46.598320    5412 utils.go:83] GRPC error: rpc error: code = Internal desc = failed to get current attachment for volume (/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-wto256/providers/Microsoft.Compute/disks/pvc-37c0c80a-58bd-419a-9ef9-f3805fdc51d8) and node (capz-wto2-xzbdk): azvolumeattachment.disk.csi.azure.com "pvc-37c0c80a-58bd-419a-9ef9-f3805fdc51d8-capz-wto2-xzbdk-attachment" not found
I0117 19:05:48.625186    5412 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0117 19:05:48.625186    5412 utils.go:79] GRPC request: {}
I0117 19:05:48.625186    5412 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I0117 19:05:48.630218    5412 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0117 19:05:48.630218    5412 utils.go:79] GRPC request: {}
I0117 19:05:48.630218    5412 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I0117 19:05:48.631203    5412 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0117 19:05:48.631203    5412 utils.go:79] GRPC request: {}
I0117 19:05:48.631686    5412 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I0117 19:05:48.632696    5412 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0117 19:05:48.632696    5412 utils.go:79] GRPC request: {"publish_context":{"LUN":"1"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-37c0c80a-58bd-419a-9ef9-f3805fdc51d8\\globalmount","volume_capability":{"AccessType":{"Mount":{"mount_flags":["barrier=1","acl"]}},"access_mode":{"mode":7}},"volume_context":{"cachingmode":"None","csi.storage.k8s.io/pv/name":"pvc-37c0c80a-58bd-419a-9ef9-f3805fdc51d8","csi.storage.k8s.io/pvc/name":"pvc-vlc2v","csi.storage.k8s.io/pvc/namespace":"azuredisk-1632","maxshares":"2","requestedsizegib":"10","skuname":"Premium_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1673977076937-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-wto256/providers/Microsoft.Compute/disks/pvc-37c0c80a-58bd-419a-9ef9-f3805fdc51d8"}
E0117 19:05:48.632696    5412 utils.go:83] GRPC error: rpc error: code = Internal desc = failed to get current attachment for volume (/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-wto256/providers/Microsoft.Compute/disks/pvc-37c0c80a-58bd-419a-9ef9-f3805fdc51d8) and node (capz-wto2-xzbdk): azvolumeattachment.disk.csi.azure.com "pvc-37c0c80a-58bd-419a-9ef9-f3805fdc51d8-capz-wto2-xzbdk-attachment" not found
I0117 19:05:50.502766    5412 azuredisk_v2.go:578]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-17T19:05:50Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-wto2-xzbdk" "disk.csi.azure.com/request-id"="f51ff252-9699-11ed-86db-002248c1a32d"
I0117 19:05:52.698434    5412 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0117 19:05:52.698434    5412 utils.go:79] GRPC request: {}
I0117 19:05:52.698434    5412 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I0117 19:05:52.706486    5412 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0117 19:05:52.706486    5412 utils.go:79] GRPC request: {}
I0117 19:05:52.706486    5412 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I0117 19:05:52.707438    5412 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0117 19:05:52.707438    5412 utils.go:79] GRPC request: {}
I0117 19:05:52.707934    5412 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I0117 19:05:52.708935    5412 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0117 19:05:52.708935    5412 utils.go:79] GRPC request: {"publish_context":{"LUN":"1"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-37c0c80a-58bd-419a-9ef9-f3805fdc51d8\\globalmount","volume_capability":{"AccessType":{"Mount":{"mount_flags":["barrier=1","acl"]}},"access_mode":{"mode":7}},"volume_context":{"cachingmode":"None","csi.storage.k8s.io/pv/name":"pvc-37c0c80a-58bd-419a-9ef9-f3805fdc51d8","csi.storage.k8s.io/pvc/name":"pvc-vlc2v","csi.storage.k8s.io/pvc/namespace":"azuredisk-1632","maxshares":"2","requestedsizegib":"10","skuname":"Premium_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1673977076937-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-wto256/providers/Microsoft.Compute/disks/pvc-37c0c80a-58bd-419a-9ef9-f3805fdc51d8"}
E0117 19:05:52.709446    5412 utils.go:83] GRPC error: rpc error: code = Internal desc = failed to get current attachment for volume (/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-wto256/providers/Microsoft.Compute/disks/pvc-37c0c80a-58bd-419a-9ef9-f3805fdc51d8) and node (capz-wto2-xzbdk): azvolumeattachment.disk.csi.azure.com "pvc-37c0c80a-58bd-419a-9ef9-f3805fdc51d8-capz-wto2-xzbdk-attachment" not found
I0117 19:05:57.833442    5412 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I0117 19:05:57.833442    5412 utils.go:79] GRPC request: {}
I0117 19:05:57.833442    5412 utils.go:85] GRPC response: {"ready":{"value":true}}
I0117 19:06:00.736917    5412 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0117 19:06:00.736917    5412 utils.go:79] GRPC request: {}
I0117 19:06:00.736917    5412 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
... skipping 2 lines ...
I0117 19:06:00.741938    5412 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I0117 19:06:00.743925    5412 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0117 19:06:00.743925    5412 utils.go:79] GRPC request: {}
I0117 19:06:00.743925    5412 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I0117 19:06:00.745425    5412 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0117 19:06:00.745425    5412 utils.go:79] GRPC request: {"publish_context":{"LUN":"1"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-37c0c80a-58bd-419a-9ef9-f3805fdc51d8\\globalmount","volume_capability":{"AccessType":{"Mount":{"mount_flags":["barrier=1","acl"]}},"access_mode":{"mode":7}},"volume_context":{"cachingmode":"None","csi.storage.k8s.io/pv/name":"pvc-37c0c80a-58bd-419a-9ef9-f3805fdc51d8","csi.storage.k8s.io/pvc/name":"pvc-vlc2v","csi.storage.k8s.io/pvc/namespace":"azuredisk-1632","maxshares":"2","requestedsizegib":"10","skuname":"Premium_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1673977076937-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-wto256/providers/Microsoft.Compute/disks/pvc-37c0c80a-58bd-419a-9ef9-f3805fdc51d8"}
E0117 19:06:00.745929    5412 utils.go:83] GRPC error: rpc error: code = Internal desc = failed to get current attachment for volume (/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-wto256/providers/Microsoft.Compute/disks/pvc-37c0c80a-58bd-419a-9ef9-f3805fdc51d8) and node (capz-wto2-xzbdk): azvolumeattachment.disk.csi.azure.com "pvc-37c0c80a-58bd-419a-9ef9-f3805fdc51d8-capz-wto2-xzbdk-attachment" not found
W0117 19:06:05.519462    5412 reflector.go:347] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: watch of *v1beta2.AzDriverNode ended with: an error on the server ("unable to decode an event from the watch stream: http2: client connection lost") has prevented the request from succeeding
W0117 19:06:05.519462    5412 reflector.go:347] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: watch of *v1.CustomResourceDefinition ended with: an error on the server ("unable to decode an event from the watch stream: http2: client connection lost") has prevented the request from succeeding
W0117 19:06:05.519462    5412 reflector.go:347] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: watch of *v1beta2.AzVolumeAttachment ended with: an error on the server ("unable to decode an event from the watch stream: http2: client connection lost") has prevented the request from succeeding
W0117 19:06:05.519462    5412 reflector.go:347] k8s.io/client-go/informers/factory.go:150: watch of *v1.Node ended with: an error on the server ("unable to decode an event from the watch stream: http2: client connection lost") has prevented the request from succeeding
I0117 19:06:05.519462    5412 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-wto2-xzbdk/status  in 15016 milliseconds
W0117 19:06:05.519462    5412 reflector.go:347] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: watch of *v1beta2.AzVolume ended with: an error on the server ("unable to decode an event from the watch stream: http2: client connection lost") has prevented the request from succeeding
W0117 19:06:05.519462    5412 reflector.go:347] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: watch of *v1beta2.AzDriverNode ended with: an error on the server ("unable to decode an event from the watch stream: http2: client connection lost") has prevented the request from succeeding
E0117 19:06:05.519462    5412 azuredisk_v2.go:613]  "msg"="Failed to update AzDriverNode status after creation" "error"="Put \"https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-wto2-xzbdk/status\": http2: client connection lost" "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-wto2-xzbdk" "disk.csi.azure.com/request-id"="f51ff252-9699-11ed-86db-002248c1a32d"
E0117 19:06:05.519462    5412 azuredisk_v2.go:565]  "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [Put \"https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-wto2-xzbdk/status\": http2: client connection lost]" "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-wto2-xzbdk" "disk.csi.azure.com/request-id"="f51ff252-9699-11ed-86db-002248c1a32d" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=15016665700
I0117 19:06:06.366214    5412 reflector.go:257] Listing and watching *v1.CustomResourceDefinition from k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132
I0117 19:06:06.398214    5412 round_trippers.go:553] GET https://10.96.0.1:443/apis/apiextensions.k8s.io/v1/customresourcedefinitions?resourceVersion=28045 200 OK in 31 milliseconds
I0117 19:06:06.429714    5412 round_trippers.go:553] GET https://10.96.0.1:443/apis/apiextensions.k8s.io/v1/customresourcedefinitions?allowWatchBookmarks=true&resourceVersion=28045&timeout=5m53s&timeoutSeconds=353&watch=true 200 OK in 1 milliseconds
I0117 19:06:06.496929    5412 reflector.go:257] Listing and watching *v1beta2.AzDriverNode from sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132
I0117 19:06:06.499214    5412 round_trippers.go:553] GET https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes?fieldSelector=metadata.name%3Dcapz-wto2-xzbdk&resourceVersion=30691 200 OK in 2 milliseconds
I0117 19:06:06.500715    5412 round_trippers.go:553] GET https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dcapz-wto2-xzbdk&resourceVersion=30929&timeout=6m2s&timeoutSeconds=362&watch=true 200 OK in 0 milliseconds
... skipping 457 lines ...
I0117 19:18:11.086419    5412 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0117 19:18:11.086419    5412 utils.go:79] GRPC request: {}
I0117 19:18:11.086419    5412 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I0117 19:18:11.087431    5412 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0117 19:18:11.087431    5412 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-0ad7ecd7-c284-4f8f-a5ca-f07797b6ed55\\globalmount","volume_capability":{"AccessType":{"Mount":{"mount_flags":["barrier=1","acl"]}},"access_mode":{"mode":7}},"volume_context":{"cachingmode":"None","csi.storage.k8s.io/pv/name":"pvc-0ad7ecd7-c284-4f8f-a5ca-f07797b6ed55","csi.storage.k8s.io/pvc/name":"pvc-lbxpz","csi.storage.k8s.io/pvc/namespace":"azuredisk-5314","maxshares":"2","requestedsizegib":"10","skuname":"Premium_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1673977076937-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-wto256/providers/Microsoft.Compute/disks/pvc-0ad7ecd7-c284-4f8f-a5ca-f07797b6ed55"}
I0117 19:18:11.087918    5412 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-0ad7ecd7-c284-4f8f-a5ca-f07797b6ed55-capz-wto2-xzbdk-attachment)
I0117 19:18:15.596025    5412 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 19:18:15.596025    5412 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-wto2-xzbdk" "disk.csi.azure.com/request-id"="ae8c45cf-969b-11ed-86db-002248c1a32d" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-0ad7ecd7-c284-4f8f-a5ca-f07797b6ed55" "latency"=4508096100
I0117 19:18:15.596025    5412 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="ae8c45cf-969b-11ed-86db-002248c1a32d" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=4508096100
I0117 19:18:19.516530    5412 nodeserver_v2.go:211] NodeStageVolume: formatting 10 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-0ad7ecd7-c284-4f8f-a5ca-f07797b6ed55\globalmount with mount options([barrier=1 acl])
I0117 19:18:20.498047    5412 azuredisk_v2.go:578]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-17T19:18:20Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-wto2-xzbdk" "disk.csi.azure.com/request-id"="b42824f9-969b-11ed-86db-002248c1a32d"
I0117 19:18:20.511537    5412 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-wto2-xzbdk/status 200 OK in 13 milliseconds
I0117 19:18:20.511537    5412 azuredisk_v2.go:565]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-wto2-xzbdk" "disk.csi.azure.com/request-id"="b42824f9-969b-11ed-86db-002248c1a32d" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=13489800
... skipping 167 lines ...
I0117 19:23:16.721709    5412 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0117 19:23:16.721709    5412 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-284ddd2a-63e4-4cb2-9724-ef0436e9a1a1\\globalmount","volume_capability":{"AccessType":{"Mount":{"mount_flags":["barrier=1","acl"]}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-284ddd2a-63e4-4cb2-9724-ef0436e9a1a1","csi.storage.k8s.io/pvc/name":"pvc-2g5bn","csi.storage.k8s.io/pvc/namespace":"azuredisk-205","requestedsizegib":"10","skuname":"StandardSSD_ZRS","storage.kubernetes.io/csiProvisionerIdentity":"1673977076937-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-wto256/providers/Microsoft.Compute/disks/pvc-284ddd2a-63e4-4cb2-9724-ef0436e9a1a1"}
I0117 19:23:16.721709    5412 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-284ddd2a-63e4-4cb2-9724-ef0436e9a1a1-capz-wto2-xzbdk-attachment)
I0117 19:23:20.492332    5412 azuredisk_v2.go:578]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-17T19:23:20Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-wto2-xzbdk" "disk.csi.azure.com/request-id"="66f79a39-969c-11ed-86db-002248c1a32d"
I0117 19:23:20.504589    5412 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-wto2-xzbdk/status 200 OK in 12 milliseconds
I0117 19:23:20.504589    5412 azuredisk_v2.go:565]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-wto2-xzbdk" "disk.csi.azure.com/request-id"="66f79a39-969c-11ed-86db-002248c1a32d" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=12500200
I0117 19:23:20.727583    5412 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 19:23:20.727583    5412 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-wto2-xzbdk" "disk.csi.azure.com/request-id"="64b849c8-969c-11ed-86db-002248c1a32d" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-284ddd2a-63e4-4cb2-9724-ef0436e9a1a1" "latency"=4005368400
I0117 19:23:20.727583    5412 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="64b849c8-969c-11ed-86db-002248c1a32d" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=4005868000
I0117 19:23:26.868942    5412 nodeserver_v2.go:211] NodeStageVolume: formatting 10 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-284ddd2a-63e4-4cb2-9724-ef0436e9a1a1\globalmount with mount options([barrier=1 acl])
I0117 19:23:27.840453    5412 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I0117 19:23:27.840453    5412 utils.go:79] GRPC request: {}
I0117 19:23:27.840453    5412 utils.go:85] GRPC response: {"ready":{"value":true}}
... skipping 118 lines ...
I0117 19:26:10.780810    5412 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0117 19:26:10.780810    5412 utils.go:79] GRPC request: {}
I0117 19:26:10.780810    5412 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I0117 19:26:10.781798    5412 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0117 19:26:10.781798    5412 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-70cf148f-ed9f-4422-ab28-c1f0a0a69b27\\globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-70cf148f-ed9f-4422-ab28-c1f0a0a69b27","csi.storage.k8s.io/pvc/name":"persistent-storage-statefulset-azuredisk-win-0","csi.storage.k8s.io/pvc/namespace":"default","requestedsizegib":"100","skuName":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1673977076937-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-wto256/providers/Microsoft.Compute/disks/pvc-70cf148f-ed9f-4422-ab28-c1f0a0a69b27"}
I0117 19:26:10.782304    5412 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-70cf148f-ed9f-4422-ab28-c1f0a0a69b27-capz-wto2-xzbdk-attachment)
I0117 19:26:20.405377    5412 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 19:26:20.405377    5412 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-wto2-xzbdk" "disk.csi.azure.com/request-id"="cc77d3c6-969c-11ed-86db-002248c1a32d" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-70cf148f-ed9f-4422-ab28-c1f0a0a69b27" "latency"=9623048500
I0117 19:26:20.405377    5412 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="cc77d3c6-969c-11ed-86db-002248c1a32d" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9623048500
I0117 19:26:20.495384    5412 azuredisk_v2.go:578]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-17T19:26:20Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-wto2-xzbdk" "disk.csi.azure.com/request-id"="d241ecf2-969c-11ed-86db-002248c1a32d"
I0117 19:26:20.507386    5412 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-wto2-xzbdk/status 200 OK in 12 milliseconds
I0117 19:26:20.507890    5412 azuredisk_v2.go:565]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-wto2-xzbdk" "disk.csi.azure.com/request-id"="d241ecf2-969c-11ed-86db-002248c1a32d" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=12001900
I0117 19:26:24.648400    5412 nodeserver_v2.go:211] NodeStageVolume: formatting 10 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-70cf148f-ed9f-4422-ab28-c1f0a0a69b27\globalmount with mount options([])
... skipping 572 lines ...
# HELP go_gc_heap_objects_objects Number of objects, live or unswept, occupying heap memory.
# TYPE go_gc_heap_objects_objects gauge
go_gc_heap_objects_objects 93392
# HELP go_gc_heap_tiny_allocs_objects_total Count of small allocations that are packed together into blocks. These allocations are counted separately from other allocations because each individual allocation is not tracked by the runtime, only their block. Each block is already accounted for in allocs-by-size and frees-by-size.
# TYPE go_gc_heap_tiny_allocs_objects_total counter
go_gc_heap_tiny_allocs_objects_total 326255
# HELP go_gc_limiter_last_enabled_gc_cycle GC cycle the last time the GC CPU limiter was enabled. This metric is useful for diagnosing the root cause of an out-of-memory error, because the limiter trades memory for CPU time when the GC's CPU time gets too high. This is most likely to occur with use of SetMemoryLimit. The first GC cycle is cycle 1, so a value of 0 indicates that it was never enabled.
# TYPE go_gc_limiter_last_enabled_gc_cycle gauge
go_gc_limiter_last_enabled_gc_cycle 0
# HELP go_gc_pauses_seconds Distribution individual GC-related stop-the-world pause latencies.
# TYPE go_gc_pauses_seconds histogram
go_gc_pauses_seconds_bucket{le="9.999999999999999e-10"} 0
go_gc_pauses_seconds_bucket{le="9.999999999999999e-09"} 0
... skipping 211 lines ...
[ReportAfterSuite] Autogenerated ReportAfterSuite for --junit-report
[38;5;243mautogenerated by Ginkgo[0m
[38;5;243m------------------------------[0m


[38;5;9m[1mSummarizing 1 Failure:[0m
  [38;5;9m[FAIL][0m [0mDynamic Provisioning [38;5;243m[single-az] [0m[38;5;9m[1m[It] Should create replicas on node with matching pod affinity[0m[0m
  [38;5;243m/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/testsuites/dynamically_provisioned_pod_affinity.go:168[0m

[38;5;9m[1mRan 37 of 108 Specs in 6676.588 seconds[0m
[38;5;9m[1mFAIL![0m -- [38;5;10m[1m36 Passed[0m | [38;5;9m[1m1 Failed[0m | [38;5;11m[1m0 Pending[0m | [38;5;14m[1m71 Skipped[0m
--- FAIL: TestE2E (6676.69s)
FAIL
FAIL	sigs.k8s.io/azuredisk-csi-driver/test/e2e	6676.980s
FAIL
make: *** [Makefile:341: e2e-test] Error 1
NAME                              STATUS   ROLES                  AGE    VERSION                           INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                         KERNEL-VERSION     CONTAINER-RUNTIME
capz-wto2-brk9b                   Ready    <none>                 125m   v1.23.16-rc.0.61+c8e964281208da   10.1.0.6      <none>        Windows Server 2019 Datacenter   10.0.17763.3770    containerd://1.6.2
capz-wto2-qsbsf                   Ready    <none>                 124m   v1.23.16-rc.0.61+c8e964281208da   10.1.0.5      <none>        Windows Server 2019 Datacenter   10.0.17763.3770    containerd://1.6.2
capz-wto2-xzbdk                   Ready    <none>                 117m   v1.23.16-rc.0.61+c8e964281208da   10.1.0.4      <none>        Windows Server 2019 Datacenter   10.0.17763.3770    containerd://1.6.2
capz-wto256-control-plane-b5m52   Ready    control-plane,master   129m   v1.23.16-rc.0.61+c8e964281208da   10.0.0.4      <none>        Ubuntu 18.04.6 LTS               5.4.0-1098-azure   containerd://1.6.2
NAMESPACE          NAME                                                      READY   STATUS    RESTARTS       AGE    IP                NODE                              NOMINATED NODE   READINESS GATES
... skipping 36 lines ...

  Jan 17 19:33:59.814: INFO: Collecting logs for Windows node capz-wto2-xzbdk in cluster capz-wto256 in namespace default

  Jan 17 19:35:27.856: INFO: Attempting to copy file /c:/crashdumps.tar on node capz-wto2-xzbdk to /logs/artifacts/clusters/capz-wto256/machines/capz-wto256-md-win-6bff88f697-4x9vf/crashdumps.tar
  Jan 17 19:35:30.268: INFO: Collecting boot logs for AzureMachine capz-wto256-md-win-xzbdk

Failed to get logs for Machine capz-wto256-md-win-6bff88f697-4x9vf, Cluster default/capz-wto256: running command "$p = 'c:\localdumps' ; if (Test-Path $p) { tar.exe -cvzf c:\crashdumps.tar $p *>&1 | %{ Write-Output "$_"} } else { Write-Host "No crash dumps found at $p" }": Process exited with status 1
  Jan 17 19:35:31.642: INFO: Collecting logs for Windows node capz-wto2-brk9b in cluster capz-wto256 in namespace default

  Jan 17 19:38:04.941: INFO: Attempting to copy file /c:/crashdumps.tar on node capz-wto2-brk9b to /logs/artifacts/clusters/capz-wto256/machines/capz-wto256-md-win-6bff88f697-nc69c/crashdumps.tar
  Jan 17 19:38:07.444: INFO: Collecting boot logs for AzureMachine capz-wto256-md-win-brk9b

Failed to get logs for Machine capz-wto256-md-win-6bff88f697-nc69c, Cluster default/capz-wto256: [running command "Get-Content "C:\\cni.log"": Process exited with status 1, running command "$p = 'c:\localdumps' ; if (Test-Path $p) { tar.exe -cvzf c:\crashdumps.tar $p *>&1 | %{ Write-Output "$_"} } else { Write-Host "No crash dumps found at $p" }": Process exited with status 1]
  Jan 17 19:38:08.932: INFO: Collecting logs for Windows node capz-wto2-qsbsf in cluster capz-wto256 in namespace default

  Jan 17 19:39:58.602: INFO: Attempting to copy file /c:/crashdumps.tar on node capz-wto2-qsbsf to /logs/artifacts/clusters/capz-wto256/machines/capz-wto256-md-win-6bff88f697-xzzvh/crashdumps.tar
  Jan 17 19:40:01.107: INFO: Collecting boot logs for AzureMachine capz-wto256-md-win-qsbsf

Failed to get logs for Machine capz-wto256-md-win-6bff88f697-xzzvh, Cluster default/capz-wto256: running command "$p = 'c:\localdumps' ; if (Test-Path $p) { tar.exe -cvzf c:\crashdumps.tar $p *>&1 | %{ Write-Output "$_"} } else { Write-Host "No crash dumps found at $p" }": Process exited with status 1
  Jan 17 19:40:02.341: INFO: Dumping workload cluster default/capz-wto256 kube-system pod logs
  Jan 17 19:40:02.979: INFO: Creating log watcher for controller calico-apiserver/calico-apiserver-74bccfd679-7tpf6, container calico-apiserver
  Jan 17 19:40:02.979: INFO: Collecting events for Pod calico-apiserver/calico-apiserver-74bccfd679-7tpf6
  Jan 17 19:40:02.982: INFO: Creating log watcher for controller calico-apiserver/calico-apiserver-74bccfd679-7xd8s, container calico-apiserver
  Jan 17 19:40:02.983: INFO: Collecting events for Pod calico-apiserver/calico-apiserver-74bccfd679-7xd8s
  Jan 17 19:40:03.048: INFO: Creating log watcher for controller calico-system/calico-kube-controllers-fb49b9cf7-fj7gq, container calico-kube-controllers
... skipping 49 lines ...
  Jan 17 19:40:03.179: INFO: Collecting events for Pod kube-system/metrics-server-7bdcf69694-mxn7x
  Jan 17 19:40:03.179: INFO: Collecting events for Pod kube-system/containerd-logger-pmn89
  Jan 17 19:40:03.325: INFO: Fetching kube-system pod logs took 983.94809ms
  Jan 17 19:40:03.325: INFO: Dumping workload cluster default/capz-wto256 Azure activity log
  Jan 17 19:40:03.326: INFO: Creating log watcher for controller tigera-operator/tigera-operator-6bbf97c9cf-shlf4, container tigera-operator
  Jan 17 19:40:03.326: INFO: Collecting events for Pod tigera-operator/tigera-operator-6bbf97c9cf-shlf4
  Jan 17 19:40:11.342: INFO: Got error while iterating over activity logs for resource group capz-wto256: insights.ActivityLogsClient#listNextResults: Failure responding to next results request: StatusCode=404 -- Original Error: autorest/azure: error response cannot be parsed: {"<!DOCTYPE html PUBLIC \"-//W3C//DTD XHTML 1.0 Strict//EN\" \"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd\">\r\n<html xmlns=\"http://www.w3.org/1999/xhtml\">\r\n<head>\r\n<meta http-equiv=\"Content-Type\" content=\"text/html; charset=iso-8859-1\"/>\r\n<title>404 - File or directory not found.</title>\r\n<style type=\"text/css\">\r\n<!--\r\nbody{margin:0;font-size:.7em;font-family:Verdana, Arial, Helvetica, sans-serif;background:#EEEEEE;}\r\nfieldset{padding:0 15px 10px 15px;} \r\nh1{font-size:2.4em;margin:0;color:#FFF;}\r\nh2{font-si" '\x00' '\x00'} error: invalid character '<' looking for beginning of value
  Jan 17 19:40:11.342: INFO: Fetching activity logs took 8.017676403s
================ REDACTING LOGS ================
All sensitive variables are redacted
cluster.cluster.x-k8s.io "capz-wto256" deleted
/home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure/hack/tools/bin/kind-v0.17.0 delete cluster --name=capz || true
Deleting cluster "capz" ...
... skipping 12 lines ...
PR	hccheng72: [V2] fix: only count replica attachments w/t deletion timestamp from cache client
Result	FAILURE
Tests	1 failed / 38 succeeded
Started	2023-01-17 17:02
Elapsed	2h46m
Revision	23b9a8b4a23ed46f3c0622bc10b1ef0956cdbc9e
Refs	1691
kubernetes-sigs/azuredisk-csi-driver PR #1691 pull-azuredisk-csi-driver-e2e-capz-windows-2019-mainv2 #1615394037500481536

Test Failures

AzureDisk CSI Driver End-to-End Tests [It] Dynamic Provisioning [single-az] Should create replicas on node with matching pod affinity 13m17s

Passed

Skipped

Error lines from build-log.txt
