pull-azuredisk-csi-driver-e2e-capz-windows-2019-mainv2 #1615439868979056640 Results

Error lines from build-log.txt

Raw build-log.txt
... skipping 790 lines ...
certificate.cert-manager.io "selfsigned-cert" deleted
# Create secret for AzureClusterIdentity
./hack/create-identity-secret.sh
make[2]: Entering directory '/home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure'
make[2]: Nothing to be done for 'kubectl'.
make[2]: Leaving directory '/home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure'
Error from server (NotFound): secrets "cluster-identity-secret" not found
secret/cluster-identity-secret created
secret/cluster-identity-secret labeled
# Create customized cloud provider configs
./hack/create-custom-cloud-provider-config.sh
make[2]: Entering directory '/home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure'
make[2]: Nothing to be done for 'kubectl'.
... skipping 134 lines ...
timeout --foreground 600 bash -c "while ! /home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure/hack/tools/bin/kubectl-v1.22.4 --kubeconfig=./kubeconfig get nodes | grep control-plane; do sleep 1; done"
Unable to connect to the server: dial tcp 51.138.176.173:6443: i/o timeout
capz-ri100d-control-plane-8n9zj   NotReady   <none>   0s    v1.23.16-rc.0.61+c8e964281208da
run "/home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure/hack/tools/bin/kubectl-v1.22.4 --kubeconfig=./kubeconfig ..." to work with the new target cluster
make[1]: Leaving directory '/home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure'
namespace/calico-system created
Error from server (NotFound): configmaps "kubeadm-config" not found
NAME             DATA   AGE
kubeadm-config   1      4s
configmap/kubeadm-config created
Installing Calico CNI via helm
Cluster CIDR is IPv4
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure/kubeconfig
... skipping 976 lines ...
                    type: string
                type: object
                oneOf:
                - required: ["persistentVolumeClaimName"]
                - required: ["volumeSnapshotContentName"]
              volumeSnapshotClassName:
                description: 'VolumeSnapshotClassName is the name of the VolumeSnapshotClass requested by the VolumeSnapshot. VolumeSnapshotClassName may be left nil to indicate that the default SnapshotClass should be used. A given cluster may have multiple default Volume SnapshotClasses: one default per CSI Driver. If a VolumeSnapshot does not specify a SnapshotClass, VolumeSnapshotSource will be checked to figure out what the associated CSI Driver is, and the default VolumeSnapshotClass associated with that CSI Driver will be used. If more than one VolumeSnapshotClass exist for a given CSI Driver and more than one have been marked as default, CreateSnapshot will fail and generate an event. Empty string is not allowed for this field.'
                type: string
            required:
            - source
            type: object
          status:
            description: status represents the current information of a snapshot. Consumers must verify binding between VolumeSnapshot and VolumeSnapshotContent objects is successful (by validating that both VolumeSnapshot and VolumeSnapshotContent point at each other) before using this object.
... skipping 2 lines ...
                description: 'boundVolumeSnapshotContentName is the name of the VolumeSnapshotContent object to which this VolumeSnapshot object intends to bind to. If not specified, it indicates that the VolumeSnapshot object has not been successfully bound to a VolumeSnapshotContent object yet. NOTE: To avoid possible security issues, consumers must verify binding between VolumeSnapshot and VolumeSnapshotContent objects is successful (by validating that both VolumeSnapshot and VolumeSnapshotContent point at each other) before using this object.'
                type: string
              creationTime:
                description: creationTime is the timestamp when the point-in-time snapshot is taken by the underlying storage system. In dynamic snapshot creation case, this field will be filled in by the snapshot controller with the "creation_time" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "creation_time" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. If not specified, it may indicate that the creation time of the snapshot is unknown.
                format: date-time
                type: string
              error:
                description: error is the last observed error during snapshot creation, if any. This field could be helpful to upper level controllers(i.e., application controller) to decide whether they should continue on waiting for the snapshot to be created based on the type of error reported. The snapshot controller will keep retrying when an error occurs during the snapshot creation. Upon success, this error field will be cleared.
                properties:
                  message:
                    description: 'message is a string detailing the encountered error during snapshot creation if specified. NOTE: message may be logged, and it should not contain sensitive information.'
                    type: string
                  time:
                    description: time is the timestamp when the error was encountered.
                    format: date-time
                    type: string
                type: object
              readyToUse:
                description: readyToUse indicates if the snapshot is ready to be used to restore a volume. In dynamic snapshot creation case, this field will be filled in by the snapshot controller with the "ready_to_use" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "ready_to_use" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it, otherwise, this field will be set to "True". If not specified, it means the readiness of a snapshot is unknown.
                type: boolean
              restoreSize:
                type: string
                description: restoreSize represents the minimum size of volume required to create a volume from this snapshot. In dynamic snapshot creation case, this field will be filled in by the snapshot controller with the "size_bytes" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "size_bytes" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. When restoring a volume from this snapshot, the size of the volume MUST NOT be smaller than the restoreSize if it is specified, otherwise the restoration will fail. If not specified, it indicates that the size is unknown.
                pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
                x-kubernetes-int-or-string: true
            type: object
        required:
        - spec
        type: object
... skipping 60 lines ...
                    type: string
                  volumeSnapshotContentName:
                    description: volumeSnapshotContentName specifies the name of a pre-existing VolumeSnapshotContent object representing an existing volume snapshot. This field should be set if the snapshot already exists and only needs a representation in Kubernetes. This field is immutable.
                    type: string
                type: object
              volumeSnapshotClassName:
                description: 'VolumeSnapshotClassName is the name of the VolumeSnapshotClass requested by the VolumeSnapshot. VolumeSnapshotClassName may be left nil to indicate that the default SnapshotClass should be used. A given cluster may have multiple default Volume SnapshotClasses: one default per CSI Driver. If a VolumeSnapshot does not specify a SnapshotClass, VolumeSnapshotSource will be checked to figure out what the associated CSI Driver is, and the default VolumeSnapshotClass associated with that CSI Driver will be used. If more than one VolumeSnapshotClass exist for a given CSI Driver and more than one have been marked as default, CreateSnapshot will fail and generate an event. Empty string is not allowed for this field.'
                type: string
            required:
            - source
            type: object
          status:
            description: status represents the current information of a snapshot. Consumers must verify binding between VolumeSnapshot and VolumeSnapshotContent objects is successful (by validating that both VolumeSnapshot and VolumeSnapshotContent point at each other) before using this object.
... skipping 2 lines ...
                description: 'boundVolumeSnapshotContentName is the name of the VolumeSnapshotContent object to which this VolumeSnapshot object intends to bind to. If not specified, it indicates that the VolumeSnapshot object has not been successfully bound to a VolumeSnapshotContent object yet. NOTE: To avoid possible security issues, consumers must verify binding between VolumeSnapshot and VolumeSnapshotContent objects is successful (by validating that both VolumeSnapshot and VolumeSnapshotContent point at each other) before using this object.'
                type: string
              creationTime:
                description: creationTime is the timestamp when the point-in-time snapshot is taken by the underlying storage system. In dynamic snapshot creation case, this field will be filled in by the snapshot controller with the "creation_time" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "creation_time" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. If not specified, it may indicate that the creation time of the snapshot is unknown.
                format: date-time
                type: string
              error:
                description: error is the last observed error during snapshot creation, if any. This field could be helpful to upper level controllers(i.e., application controller) to decide whether they should continue on waiting for the snapshot to be created based on the type of error reported. The snapshot controller will keep retrying when an error occurs during the snapshot creation. Upon success, this error field will be cleared.
                properties:
                  message:
                    description: 'message is a string detailing the encountered error during snapshot creation if specified. NOTE: message may be logged, and it should not contain sensitive information.'
                    type: string
                  time:
                    description: time is the timestamp when the error was encountered.
                    format: date-time
                    type: string
                type: object
              readyToUse:
                description: readyToUse indicates if the snapshot is ready to be used to restore a volume. In dynamic snapshot creation case, this field will be filled in by the snapshot controller with the "ready_to_use" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "ready_to_use" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it, otherwise, this field will be set to "True". If not specified, it means the readiness of a snapshot is unknown.
                type: boolean
              restoreSize:
                type: string
                description: restoreSize represents the minimum size of volume required to create a volume from this snapshot. In dynamic snapshot creation case, this field will be filled in by the snapshot controller with the "size_bytes" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "size_bytes" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. When restoring a volume from this snapshot, the size of the volume MUST NOT be smaller than the restoreSize if it is specified, otherwise the restoration will fail. If not specified, it indicates that the size is unknown.
                pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
                x-kubernetes-int-or-string: true
            type: object
        required:
        - spec
        type: object
... skipping 254 lines ...
            description: status represents the current information of a snapshot.
            properties:
              creationTime:
                description: creationTime is the timestamp when the point-in-time snapshot is taken by the underlying storage system. In dynamic snapshot creation case, this field will be filled in by the CSI snapshotter sidecar with the "creation_time" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "creation_time" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. If not specified, it indicates the creation time is unknown. The format of this field is a Unix nanoseconds time encoded as an int64. On Unix, the command `date +%s%N` returns the current time in nanoseconds since 1970-01-01 00:00:00 UTC.
                format: int64
                type: integer
              error:
                description: error is the last observed error during snapshot creation, if any. Upon success after retry, this error field will be cleared.
                properties:
                  message:
                    description: 'message is a string detailing the encountered error during snapshot creation if specified. NOTE: message may be logged, and it should not contain sensitive information.'
                    type: string
                  time:
                    description: time is the timestamp when the error was encountered.
                    format: date-time
                    type: string
                type: object
              readyToUse:
                description: readyToUse indicates if a snapshot is ready to be used to restore a volume. In dynamic snapshot creation case, this field will be filled in by the CSI snapshotter sidecar with the "ready_to_use" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "ready_to_use" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it, otherwise, this field will be set to "True". If not specified, it means the readiness of a snapshot is unknown.
                type: boolean
              restoreSize:
                description: restoreSize represents the complete size of the snapshot in bytes. In dynamic snapshot creation case, this field will be filled in by the CSI snapshotter sidecar with the "size_bytes" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "size_bytes" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. When restoring a volume from this snapshot, the size of the volume MUST NOT be smaller than the restoreSize if it is specified, otherwise the restoration will fail. If not specified, it indicates that the size is unknown.
                format: int64
                minimum: 0
                type: integer
              snapshotHandle:
                description: snapshotHandle is the CSI "snapshot_id" of a snapshot on the underlying storage system. If not specified, it indicates that dynamic snapshot creation has either failed or it is still in progress.
                type: string
            type: object
        required:
        - spec
        type: object
    served: true
... skipping 108 lines ...
            description: status represents the current information of a snapshot.
            properties:
              creationTime:
                description: creationTime is the timestamp when the point-in-time snapshot is taken by the underlying storage system. In dynamic snapshot creation case, this field will be filled in by the CSI snapshotter sidecar with the "creation_time" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "creation_time" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. If not specified, it indicates the creation time is unknown. The format of this field is a Unix nanoseconds time encoded as an int64. On Unix, the command `date +%s%N` returns the current time in nanoseconds since 1970-01-01 00:00:00 UTC.
                format: int64
                type: integer
              error:
                description: error is the last observed error during snapshot creation, if any. Upon success after retry, this error field will be cleared.
                properties:
                  message:
                    description: 'message is a string detailing the encountered error during snapshot creation if specified. NOTE: message may be logged, and it should not contain sensitive information.'
                    type: string
                  time:
                    description: time is the timestamp when the error was encountered.
                    format: date-time
                    type: string
                type: object
              readyToUse:
                description: readyToUse indicates if a snapshot is ready to be used to restore a volume. In dynamic snapshot creation case, this field will be filled in by the CSI snapshotter sidecar with the "ready_to_use" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "ready_to_use" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it, otherwise, this field will be set to "True". If not specified, it means the readiness of a snapshot is unknown.
                type: boolean
              restoreSize:
                description: restoreSize represents the complete size of the snapshot in bytes. In dynamic snapshot creation case, this field will be filled in by the CSI snapshotter sidecar with the "size_bytes" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "size_bytes" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. When restoring a volume from this snapshot, the size of the volume MUST NOT be smaller than the restoreSize if it is specified, otherwise the restoration will fail. If not specified, it indicates that the size is unknown.
                format: int64
                minimum: 0
                type: integer
              snapshotHandle:
                description: snapshotHandle is the CSI "snapshot_id" of a snapshot on the underlying storage system. If not specified, it indicates that dynamic snapshot creation has either failed or it is still in progress.
                type: string
            type: object
        required:
        - spec
        type: object
    served: true
... skipping 359 lines ...
            - volumeName
            - volume_context
            - volume_id
            type: object
          status:
            description: status represents the current state of AzVolumeAttachment.
              includes error, state, and attachment status Required
            properties:
              detail:
                description: Status summarizes the current attachment state of the
                  volume attachment Nil Status indicates that the volume has not yet
                  been attached to the node
                properties:
... skipping 7 lines ...
                  role:
                    description: The current attachment role.
                    type: string
                required:
                - role
                type: object
              error:
                description: Error occurred during attach/detach of volume
                properties:
                  code:
                    type: string
                  message:
                    type: string
                  parameters:
... skipping 90 lines ...
            - volumeName
            - volume_context
            - volume_id
            type: object
          status:
            description: status represents the current state of AzVolumeAttachment.
              includes error, state, and attachment status
            properties:
              annotation:
                additionalProperties:
                  type: string
                description: Annotations contains additional resource information
                  to guide driver actions
... skipping 13 lines ...
                  role:
                    description: The current attachment role.
                    type: string
                required:
                - role
                type: object
              error:
                description: Error occurred during attach/detach of volume
                properties:
                  code:
                    type: string
                  message:
                    type: string
                  parameters:
... skipping 169 lines ...
            - maxMountReplicaCount
            - volumeCapability
            - volumeName
            type: object
          status:
            description: status represents the current state of AzVolume. includes
              error, state, and volume status
            properties:
              detail:
                description: Current status detail of the AzVolume Nil detail indicates
                  that the volume has not been created
                properties:
                  accessible_topology:
... skipping 28 lines ...
                    type: string
                required:
                - capacity_bytes
                - node_expansion_required
                - volume_id
                type: object
              error:
                description: Error occurred during creation/deletion of volume
                properties:
                  code:
                    type: string
                  message:
                    type: string
                  parameters:
... skipping 154 lines ...
            - maxMountReplicaCount
            - volumeCapability
            - volumeName
            type: object
          status:
            description: status represents the current state of AzVolume. includes
              error, state, and volume status
            properties:
              annotation:
                additionalProperties:
                  type: string
                description: Annotations contains additional resource information
                  to guide driver actions
... skipping 34 lines ...
                    type: string
                required:
                - capacity_bytes
                - node_expansion_required
                - volume_id
                type: object
              error:
                description: Error occurred during creation/deletion of volume
                properties:
                  code:
                    type: string
                  message:
                    type: string
                  parameters:
... skipping 831 lines ...
          image: "mcr.microsoft.com/oss/kubernetes-csi/csi-resizer:v1.5.0"
          args:
            - "-csi-address=$(ADDRESS)"
            - "-v=2"
            - "-leader-election"
            - "--leader-election-namespace=kube-system"
            - '-handle-volume-inuse-error=false'
            - '-feature-gates=RecoverVolumeExpansionFailure=true'
            - "-timeout=240s"
          env:
            - name: ADDRESS
              value: /csi/csi.sock
          volumeMounts:
... skipping 283 lines ...
Jan 17 20:25:57.293: INFO: >>> kubeConfig: /home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure/kubeconfig
[1mSTEP:[0m Building a namespace api object, basename azuredisk [38;5;243m01/17/23 20:25:57.295[0m
[1mSTEP:[0m Waiting for a default service account to be provisioned in namespace [38;5;243m01/17/23 20:25:57.86[0m
[1mSTEP:[0m Waiting for kube-root-ca.crt to be provisioned in namespace [38;5;243m01/17/23 20:25:58.065[0m
I0117 20:25:58.273808   41861 azuredisk_driver.go:52] Using azure disk driver: disk.csi.azure.com
[1mSTEP:[0m deploying the pod [38;5;243m01/17/23 20:25:58.42[0m
[1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/17/23 20:25:58.53[0m
Jan 17 20:25:58.531: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-n48xd" in namespace "azuredisk-267" to be "Succeeded or Failed"
Jan 17 20:25:58.634: INFO: Pod "azuredisk-volume-tester-n48xd": Phase="Pending", Reason="", readiness=false. Elapsed: 103.395082ms
Jan 17 20:26:00.739: INFO: Pod "azuredisk-volume-tester-n48xd": Phase="Pending", Reason="", readiness=false. Elapsed: 2.208390659s
Jan 17 20:26:02.740: INFO: Pod "azuredisk-volume-tester-n48xd": Phase="Pending", Reason="", readiness=false. Elapsed: 4.208995372s
Jan 17 20:26:04.741: INFO: Pod "azuredisk-volume-tester-n48xd": Phase="Pending", Reason="", readiness=false. Elapsed: 6.210137583s
Jan 17 20:26:06.739: INFO: Pod "azuredisk-volume-tester-n48xd": Phase="Pending", Reason="", readiness=false. Elapsed: 8.20830213s
Jan 17 20:26:08.738: INFO: Pod "azuredisk-volume-tester-n48xd": Phase="Pending", Reason="", readiness=false. Elapsed: 10.207770638s
... skipping 84 lines ...
Jan 17 20:28:58.747: INFO: Pod "azuredisk-volume-tester-n48xd": Phase="Running", Reason="", readiness=true. Elapsed: 3m0.216341985s
Jan 17 20:29:00.748: INFO: Pod "azuredisk-volume-tester-n48xd": Phase="Running", Reason="", readiness=true. Elapsed: 3m2.216936456s
Jan 17 20:29:02.746: INFO: Pod "azuredisk-volume-tester-n48xd": Phase="Running", Reason="", readiness=true. Elapsed: 3m4.215201629s
Jan 17 20:29:04.746: INFO: Pod "azuredisk-volume-tester-n48xd": Phase="Running", Reason="", readiness=false. Elapsed: 3m6.215264309s
Jan 17 20:29:06.747: INFO: Pod "azuredisk-volume-tester-n48xd": Phase="Succeeded", Reason="", readiness=false. Elapsed: 3m8.215988782s
[1mSTEP:[0m Saw pod success [38;5;243m01/17/23 20:29:06.747[0m
Jan 17 20:29:06.747: INFO: Pod "azuredisk-volume-tester-n48xd" satisfied condition "Succeeded or Failed"
Jan 17 20:29:06.747: INFO: deleting Pod "azuredisk-267"/"azuredisk-volume-tester-n48xd"
Jan 17 20:29:06.893: INFO: Pod azuredisk-volume-tester-n48xd has the following logs: hello world

[1mSTEP:[0m Deleting pod azuredisk-volume-tester-n48xd in namespace azuredisk-267 [38;5;243m01/17/23 20:29:06.893[0m
[1mSTEP:[0m Destroying namespace "azuredisk-267" for this suite. [38;5;243m01/17/23 20:29:07.022[0m
[38;5;243m------------------------------[0m
... skipping 10 lines ...
    Jan 17 20:25:57.293: INFO: >>> kubeConfig: /home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure/kubeconfig
    [1mSTEP:[0m Building a namespace api object, basename azuredisk [38;5;243m01/17/23 20:25:57.295[0m
    [1mSTEP:[0m Waiting for a default service account to be provisioned in namespace [38;5;243m01/17/23 20:25:57.86[0m
    [1mSTEP:[0m Waiting for kube-root-ca.crt to be provisioned in namespace [38;5;243m01/17/23 20:25:58.065[0m
    I0117 20:25:58.273808   41861 azuredisk_driver.go:52] Using azure disk driver: disk.csi.azure.com
    [1mSTEP:[0m deploying the pod [38;5;243m01/17/23 20:25:58.42[0m
    [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/17/23 20:25:58.53[0m
    Jan 17 20:25:58.531: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-n48xd" in namespace "azuredisk-267" to be "Succeeded or Failed"
    Jan 17 20:25:58.634: INFO: Pod "azuredisk-volume-tester-n48xd": Phase="Pending", Reason="", readiness=false. Elapsed: 103.395082ms
    Jan 17 20:26:00.739: INFO: Pod "azuredisk-volume-tester-n48xd": Phase="Pending", Reason="", readiness=false. Elapsed: 2.208390659s
    Jan 17 20:26:02.740: INFO: Pod "azuredisk-volume-tester-n48xd": Phase="Pending", Reason="", readiness=false. Elapsed: 4.208995372s
    Jan 17 20:26:04.741: INFO: Pod "azuredisk-volume-tester-n48xd": Phase="Pending", Reason="", readiness=false. Elapsed: 6.210137583s
    Jan 17 20:26:06.739: INFO: Pod "azuredisk-volume-tester-n48xd": Phase="Pending", Reason="", readiness=false. Elapsed: 8.20830213s
    Jan 17 20:26:08.738: INFO: Pod "azuredisk-volume-tester-n48xd": Phase="Pending", Reason="", readiness=false. Elapsed: 10.207770638s
... skipping 84 lines ...
    Jan 17 20:28:58.747: INFO: Pod "azuredisk-volume-tester-n48xd": Phase="Running", Reason="", readiness=true. Elapsed: 3m0.216341985s
    Jan 17 20:29:00.748: INFO: Pod "azuredisk-volume-tester-n48xd": Phase="Running", Reason="", readiness=true. Elapsed: 3m2.216936456s
    Jan 17 20:29:02.746: INFO: Pod "azuredisk-volume-tester-n48xd": Phase="Running", Reason="", readiness=true. Elapsed: 3m4.215201629s
    Jan 17 20:29:04.746: INFO: Pod "azuredisk-volume-tester-n48xd": Phase="Running", Reason="", readiness=false. Elapsed: 3m6.215264309s
    Jan 17 20:29:06.747: INFO: Pod "azuredisk-volume-tester-n48xd": Phase="Succeeded", Reason="", readiness=false. Elapsed: 3m8.215988782s
    [1mSTEP:[0m Saw pod success [38;5;243m01/17/23 20:29:06.747[0m
    Jan 17 20:29:06.747: INFO: Pod "azuredisk-volume-tester-n48xd" satisfied condition "Succeeded or Failed"
    Jan 17 20:29:06.747: INFO: deleting Pod "azuredisk-267"/"azuredisk-volume-tester-n48xd"
    Jan 17 20:29:06.893: INFO: Pod azuredisk-volume-tester-n48xd has the following logs: hello world

    [1mSTEP:[0m Deleting pod azuredisk-volume-tester-n48xd in namespace azuredisk-267 [38;5;243m01/17/23 20:29:06.893[0m
    [1mSTEP:[0m Destroying namespace "azuredisk-267" for this suite. [38;5;243m01/17/23 20:29:07.022[0m
  [38;5;243m<< End Captured GinkgoWriter Output[0m
... skipping 17 lines ...
Jan 17 20:29:10.302: INFO: PersistentVolumeClaim pvc-6tkv4 found but phase is Pending instead of Bound.
Jan 17 20:29:12.407: INFO: PersistentVolumeClaim pvc-6tkv4 found and phase=Bound (4.319308783s)
[1mSTEP:[0m checking the PVC [38;5;243m01/17/23 20:29:12.407[0m
[1mSTEP:[0m validating provisioned PV [38;5;243m01/17/23 20:29:12.511[0m
[1mSTEP:[0m checking the PV [38;5;243m01/17/23 20:29:12.615[0m
[1mSTEP:[0m deploying the pod [38;5;243m01/17/23 20:29:12.763[0m
[1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/17/23 20:29:12.871[0m
Jan 17 20:29:12.872: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-jpfnv" in namespace "azuredisk-1538" to be "Succeeded or Failed"
Jan 17 20:29:12.976: INFO: Pod "azuredisk-volume-tester-jpfnv": Phase="Pending", Reason="", readiness=false. Elapsed: 104.680356ms
Jan 17 20:29:15.082: INFO: Pod "azuredisk-volume-tester-jpfnv": Phase="Pending", Reason="", readiness=false. Elapsed: 2.210145191s
Jan 17 20:29:17.080: INFO: Pod "azuredisk-volume-tester-jpfnv": Phase="Pending", Reason="", readiness=false. Elapsed: 4.208186833s
Jan 17 20:29:19.084: INFO: Pod "azuredisk-volume-tester-jpfnv": Phase="Pending", Reason="", readiness=false. Elapsed: 6.212577317s
Jan 17 20:29:21.081: INFO: Pod "azuredisk-volume-tester-jpfnv": Phase="Pending", Reason="", readiness=false. Elapsed: 8.209008274s
Jan 17 20:29:23.081: INFO: Pod "azuredisk-volume-tester-jpfnv": Phase="Pending", Reason="", readiness=false. Elapsed: 10.209121348s
... skipping 19 lines ...
Jan 17 20:30:03.088: INFO: Pod "azuredisk-volume-tester-jpfnv": Phase="Running", Reason="", readiness=true. Elapsed: 50.216803534s
Jan 17 20:30:05.088: INFO: Pod "azuredisk-volume-tester-jpfnv": Phase="Running", Reason="", readiness=false. Elapsed: 52.216368998s
Jan 17 20:30:07.088: INFO: Pod "azuredisk-volume-tester-jpfnv": Phase="Running", Reason="", readiness=false. Elapsed: 54.216831692s
Jan 17 20:30:09.087: INFO: Pod "azuredisk-volume-tester-jpfnv": Phase="Running", Reason="", readiness=false. Elapsed: 56.215903379s
Jan 17 20:30:11.088: INFO: Pod "azuredisk-volume-tester-jpfnv": Phase="Succeeded", Reason="", readiness=false. Elapsed: 58.216008711s
[1mSTEP:[0m Saw pod success [38;5;243m01/17/23 20:30:11.088[0m
Jan 17 20:30:11.088: INFO: Pod "azuredisk-volume-tester-jpfnv" satisfied condition "Succeeded or Failed"
Jan 17 20:30:11.088: INFO: deleting Pod "azuredisk-1538"/"azuredisk-volume-tester-jpfnv"
Jan 17 20:30:11.202: INFO: Pod azuredisk-volume-tester-jpfnv has the following logs: hello world

[1mSTEP:[0m Deleting pod azuredisk-volume-tester-jpfnv in namespace azuredisk-1538 [38;5;243m01/17/23 20:30:11.202[0m
Jan 17 20:30:11.425: INFO: deleting PVC "azuredisk-1538"/"pvc-6tkv4"
Jan 17 20:30:11.425: INFO: Deleting PersistentVolumeClaim "pvc-6tkv4"
... skipping 38 lines ...
    Jan 17 20:29:10.302: INFO: PersistentVolumeClaim pvc-6tkv4 found but phase is Pending instead of Bound.
    Jan 17 20:29:12.407: INFO: PersistentVolumeClaim pvc-6tkv4 found and phase=Bound (4.319308783s)
    [1mSTEP:[0m checking the PVC [38;5;243m01/17/23 20:29:12.407[0m
    [1mSTEP:[0m validating provisioned PV [38;5;243m01/17/23 20:29:12.511[0m
    [1mSTEP:[0m checking the PV [38;5;243m01/17/23 20:29:12.615[0m
    [1mSTEP:[0m deploying the pod [38;5;243m01/17/23 20:29:12.763[0m
    [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/17/23 20:29:12.871[0m
    Jan 17 20:29:12.872: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-jpfnv" in namespace "azuredisk-1538" to be "Succeeded or Failed"
    Jan 17 20:29:12.976: INFO: Pod "azuredisk-volume-tester-jpfnv": Phase="Pending", Reason="", readiness=false. Elapsed: 104.680356ms
    Jan 17 20:29:15.082: INFO: Pod "azuredisk-volume-tester-jpfnv": Phase="Pending", Reason="", readiness=false. Elapsed: 2.210145191s
    Jan 17 20:29:17.080: INFO: Pod "azuredisk-volume-tester-jpfnv": Phase="Pending", Reason="", readiness=false. Elapsed: 4.208186833s
    Jan 17 20:29:19.084: INFO: Pod "azuredisk-volume-tester-jpfnv": Phase="Pending", Reason="", readiness=false. Elapsed: 6.212577317s
    Jan 17 20:29:21.081: INFO: Pod "azuredisk-volume-tester-jpfnv": Phase="Pending", Reason="", readiness=false. Elapsed: 8.209008274s
    Jan 17 20:29:23.081: INFO: Pod "azuredisk-volume-tester-jpfnv": Phase="Pending", Reason="", readiness=false. Elapsed: 10.209121348s
... skipping 19 lines ...
    Jan 17 20:30:03.088: INFO: Pod "azuredisk-volume-tester-jpfnv": Phase="Running", Reason="", readiness=true. Elapsed: 50.216803534s
    Jan 17 20:30:05.088: INFO: Pod "azuredisk-volume-tester-jpfnv": Phase="Running", Reason="", readiness=false. Elapsed: 52.216368998s
    Jan 17 20:30:07.088: INFO: Pod "azuredisk-volume-tester-jpfnv": Phase="Running", Reason="", readiness=false. Elapsed: 54.216831692s
    Jan 17 20:30:09.087: INFO: Pod "azuredisk-volume-tester-jpfnv": Phase="Running", Reason="", readiness=false. Elapsed: 56.215903379s
    Jan 17 20:30:11.088: INFO: Pod "azuredisk-volume-tester-jpfnv": Phase="Succeeded", Reason="", readiness=false. Elapsed: 58.216008711s
    [1mSTEP:[0m Saw pod success [38;5;243m01/17/23 20:30:11.088[0m
    Jan 17 20:30:11.088: INFO: Pod "azuredisk-volume-tester-jpfnv" satisfied condition "Succeeded or Failed"
    Jan 17 20:30:11.088: INFO: deleting Pod "azuredisk-1538"/"azuredisk-volume-tester-jpfnv"
    Jan 17 20:30:11.202: INFO: Pod azuredisk-volume-tester-jpfnv has the following logs: hello world

    [1mSTEP:[0m Deleting pod azuredisk-volume-tester-jpfnv in namespace azuredisk-1538 [38;5;243m01/17/23 20:30:11.202[0m
    Jan 17 20:30:11.425: INFO: deleting PVC "azuredisk-1538"/"pvc-6tkv4"
    Jan 17 20:30:11.425: INFO: Deleting PersistentVolumeClaim "pvc-6tkv4"
... skipping 221 lines ...
Jan 17 20:37:53.430: INFO: PersistentVolumeClaim pvc-m5sl8 found and phase=Bound (4.311345657s)
[1mSTEP:[0m checking the PVC [38;5;243m01/17/23 20:37:53.431[0m
[1mSTEP:[0m validating provisioned PV [38;5;243m01/17/23 20:37:53.534[0m
[1mSTEP:[0m checking the PV [38;5;243m01/17/23 20:37:53.637[0m
I0117 20:37:53.637440   41861 resource_setup.go:89] adding PV (pvc-5aea05f2-b729-4024-867d-387eaab33883) to pod ()
[1mSTEP:[0m deploying the pod [38;5;243m01/17/23 20:37:53.783[0m
[1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/17/23 20:37:53.895[0m
Jan 17 20:37:53.895: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-dxvjg" in namespace "azuredisk-9092" to be "Succeeded or Failed"
Jan 17 20:37:54.029: INFO: Pod "azuredisk-volume-tester-dxvjg": Phase="Pending", Reason="", readiness=false. Elapsed: 134.602608ms
Jan 17 20:37:56.140: INFO: Pod "azuredisk-volume-tester-dxvjg": Phase="Pending", Reason="", readiness=false. Elapsed: 2.245138191s
Jan 17 20:37:58.141: INFO: Pod "azuredisk-volume-tester-dxvjg": Phase="Pending", Reason="", readiness=false. Elapsed: 4.246402952s
Jan 17 20:38:00.141: INFO: Pod "azuredisk-volume-tester-dxvjg": Phase="Pending", Reason="", readiness=false. Elapsed: 6.246325057s
Jan 17 20:38:02.141: INFO: Pod "azuredisk-volume-tester-dxvjg": Phase="Pending", Reason="", readiness=false. Elapsed: 8.246035325s
Jan 17 20:38:04.149: INFO: Pod "azuredisk-volume-tester-dxvjg": Phase="Pending", Reason="", readiness=false. Elapsed: 10.254241703s
... skipping 23 lines ...
Jan 17 20:38:52.139: INFO: Pod "azuredisk-volume-tester-dxvjg": Phase="Running", Reason="", readiness=true. Elapsed: 58.244373109s
Jan 17 20:38:54.141: INFO: Pod "azuredisk-volume-tester-dxvjg": Phase="Running", Reason="", readiness=false. Elapsed: 1m0.246627424s
Jan 17 20:38:56.142: INFO: Pod "azuredisk-volume-tester-dxvjg": Phase="Running", Reason="", readiness=false. Elapsed: 1m2.246906126s
Jan 17 20:38:58.141: INFO: Pod "azuredisk-volume-tester-dxvjg": Phase="Running", Reason="", readiness=false. Elapsed: 1m4.245817521s
Jan 17 20:39:00.141: INFO: Pod "azuredisk-volume-tester-dxvjg": Phase="Succeeded", Reason="", readiness=false. Elapsed: 1m6.24636614s
[1mSTEP:[0m Saw pod success [38;5;243m01/17/23 20:39:00.141[0m
Jan 17 20:39:00.141: INFO: Pod "azuredisk-volume-tester-dxvjg" satisfied condition "Succeeded or Failed"
Jan 17 20:39:00.142: INFO: deleting Pod "azuredisk-9092"/"azuredisk-volume-tester-dxvjg"
Jan 17 20:39:00.277: INFO: Pod azuredisk-volume-tester-dxvjg has the following logs: hello world

[1mSTEP:[0m Deleting pod azuredisk-volume-tester-dxvjg in namespace azuredisk-9092 [38;5;243m01/17/23 20:39:00.277[0m
Jan 17 20:39:00.407: INFO: deleting PVC "azuredisk-9092"/"pvc-m5sl8"
Jan 17 20:39:00.407: INFO: Deleting PersistentVolumeClaim "pvc-m5sl8"
... skipping 85 lines ...
    Jan 17 20:37:53.430: INFO: PersistentVolumeClaim pvc-m5sl8 found and phase=Bound (4.311345657s)
    [1mSTEP:[0m checking the PVC [38;5;243m01/17/23 20:37:53.431[0m
    [1mSTEP:[0m validating provisioned PV [38;5;243m01/17/23 20:37:53.534[0m
    [1mSTEP:[0m checking the PV [38;5;243m01/17/23 20:37:53.637[0m
    I0117 20:37:53.637440   41861 resource_setup.go:89] adding PV (pvc-5aea05f2-b729-4024-867d-387eaab33883) to pod ()
    [1mSTEP:[0m deploying the pod [38;5;243m01/17/23 20:37:53.783[0m
    [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/17/23 20:37:53.895[0m
    Jan 17 20:37:53.895: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-dxvjg" in namespace "azuredisk-9092" to be "Succeeded or Failed"
    Jan 17 20:37:54.029: INFO: Pod "azuredisk-volume-tester-dxvjg": Phase="Pending", Reason="", readiness=false. Elapsed: 134.602608ms
    Jan 17 20:37:56.140: INFO: Pod "azuredisk-volume-tester-dxvjg": Phase="Pending", Reason="", readiness=false. Elapsed: 2.245138191s
    Jan 17 20:37:58.141: INFO: Pod "azuredisk-volume-tester-dxvjg": Phase="Pending", Reason="", readiness=false. Elapsed: 4.246402952s
    Jan 17 20:38:00.141: INFO: Pod "azuredisk-volume-tester-dxvjg": Phase="Pending", Reason="", readiness=false. Elapsed: 6.246325057s
    Jan 17 20:38:02.141: INFO: Pod "azuredisk-volume-tester-dxvjg": Phase="Pending", Reason="", readiness=false. Elapsed: 8.246035325s
    Jan 17 20:38:04.149: INFO: Pod "azuredisk-volume-tester-dxvjg": Phase="Pending", Reason="", readiness=false. Elapsed: 10.254241703s
... skipping 23 lines ...
    Jan 17 20:38:52.139: INFO: Pod "azuredisk-volume-tester-dxvjg": Phase="Running", Reason="", readiness=true. Elapsed: 58.244373109s
    Jan 17 20:38:54.141: INFO: Pod "azuredisk-volume-tester-dxvjg": Phase="Running", Reason="", readiness=false. Elapsed: 1m0.246627424s
    Jan 17 20:38:56.142: INFO: Pod "azuredisk-volume-tester-dxvjg": Phase="Running", Reason="", readiness=false. Elapsed: 1m2.246906126s
    Jan 17 20:38:58.141: INFO: Pod "azuredisk-volume-tester-dxvjg": Phase="Running", Reason="", readiness=false. Elapsed: 1m4.245817521s
    Jan 17 20:39:00.141: INFO: Pod "azuredisk-volume-tester-dxvjg": Phase="Succeeded", Reason="", readiness=false. Elapsed: 1m6.24636614s
    [1mSTEP:[0m Saw pod success [38;5;243m01/17/23 20:39:00.141[0m
    Jan 17 20:39:00.141: INFO: Pod "azuredisk-volume-tester-dxvjg" satisfied condition "Succeeded or Failed"
    Jan 17 20:39:00.142: INFO: deleting Pod "azuredisk-9092"/"azuredisk-volume-tester-dxvjg"
    Jan 17 20:39:00.277: INFO: Pod azuredisk-volume-tester-dxvjg has the following logs: hello world

    [1mSTEP:[0m Deleting pod azuredisk-volume-tester-dxvjg in namespace azuredisk-9092 [38;5;243m01/17/23 20:39:00.277[0m
    Jan 17 20:39:00.407: INFO: deleting PVC "azuredisk-9092"/"pvc-m5sl8"
    Jan 17 20:39:00.407: INFO: Deleting PersistentVolumeClaim "pvc-m5sl8"
... skipping 737 lines ...
[1mSTEP:[0m checking the PVC [38;5;243m01/17/23 20:53:02.44[0m
[1mSTEP:[0m validating provisioned PV [38;5;243m01/17/23 20:53:02.544[0m
[1mSTEP:[0m checking the PV [38;5;243m01/17/23 20:53:02.648[0m
[1mSTEP:[0m setting up the pod [38;5;243m01/17/23 20:53:02.649[0m
I0117 20:53:02.649773   41861 resource_setup.go:62] adding PV (pvc-d125a20e-7c64-4017-bd41-6edcd24569ac) to pod ()
[1mSTEP:[0m deploying the pod [38;5;243m01/17/23 20:53:02.649[0m
[1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/17/23 20:53:02.759[0m
Jan 17 20:53:02.759: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-f28vs" in namespace "azuredisk-6615" to be "Succeeded or Failed"
Jan 17 20:53:02.864: INFO: Pod "azuredisk-volume-tester-f28vs": Phase="Pending", Reason="", readiness=false. Elapsed: 104.428883ms
Jan 17 20:53:04.969: INFO: Pod "azuredisk-volume-tester-f28vs": Phase="Pending", Reason="", readiness=false. Elapsed: 2.209936636s
Jan 17 20:53:06.969: INFO: Pod "azuredisk-volume-tester-f28vs": Phase="Pending", Reason="", readiness=false. Elapsed: 4.210027854s
Jan 17 20:53:08.969: INFO: Pod "azuredisk-volume-tester-f28vs": Phase="Pending", Reason="", readiness=false. Elapsed: 6.210033774s
Jan 17 20:53:10.972: INFO: Pod "azuredisk-volume-tester-f28vs": Phase="Pending", Reason="", readiness=false. Elapsed: 8.212529974s
Jan 17 20:53:12.982: INFO: Pod "azuredisk-volume-tester-f28vs": Phase="Pending", Reason="", readiness=false. Elapsed: 10.222356533s
... skipping 16 lines ...
Jan 17 20:53:46.974: INFO: Pod "azuredisk-volume-tester-f28vs": Phase="Running", Reason="", readiness=true. Elapsed: 44.214763344s
Jan 17 20:53:48.975: INFO: Pod "azuredisk-volume-tester-f28vs": Phase="Running", Reason="", readiness=true. Elapsed: 46.215567523s
Jan 17 20:53:50.979: INFO: Pod "azuredisk-volume-tester-f28vs": Phase="Running", Reason="", readiness=false. Elapsed: 48.219518269s
Jan 17 20:53:52.973: INFO: Pod "azuredisk-volume-tester-f28vs": Phase="Running", Reason="", readiness=false. Elapsed: 50.214053918s
Jan 17 20:53:54.974: INFO: Pod "azuredisk-volume-tester-f28vs": Phase="Succeeded", Reason="", readiness=false. Elapsed: 52.214420818s
[1mSTEP:[0m Saw pod success [38;5;243m01/17/23 20:53:54.974[0m
Jan 17 20:53:54.974: INFO: Pod "azuredisk-volume-tester-f28vs" satisfied condition "Succeeded or Failed"
Jan 17 20:53:54.974: INFO: deleting Pod "azuredisk-6615"/"azuredisk-volume-tester-f28vs"
Jan 17 20:53:55.109: INFO: Pod azuredisk-volume-tester-f28vs has the following logs: hello world

[1mSTEP:[0m Deleting pod azuredisk-volume-tester-f28vs in namespace azuredisk-6615 [38;5;243m01/17/23 20:53:55.109[0m
Jan 17 20:53:55.237: INFO: deleting PVC "azuredisk-6615"/"pvc-jp2x5"
Jan 17 20:53:55.237: INFO: Deleting PersistentVolumeClaim "pvc-jp2x5"
... skipping 40 lines ...
    [1mSTEP:[0m checking the PVC [38;5;243m01/17/23 20:53:02.44[0m
    [1mSTEP:[0m validating provisioned PV [38;5;243m01/17/23 20:53:02.544[0m
    [1mSTEP:[0m checking the PV [38;5;243m01/17/23 20:53:02.648[0m
    [1mSTEP:[0m setting up the pod [38;5;243m01/17/23 20:53:02.649[0m
    I0117 20:53:02.649773   41861 resource_setup.go:62] adding PV (pvc-d125a20e-7c64-4017-bd41-6edcd24569ac) to pod ()
    [1mSTEP:[0m deploying the pod [38;5;243m01/17/23 20:53:02.649[0m
    [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/17/23 20:53:02.759[0m
    Jan 17 20:53:02.759: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-f28vs" in namespace "azuredisk-6615" to be "Succeeded or Failed"
    Jan 17 20:53:02.864: INFO: Pod "azuredisk-volume-tester-f28vs": Phase="Pending", Reason="", readiness=false. Elapsed: 104.428883ms
    Jan 17 20:53:04.969: INFO: Pod "azuredisk-volume-tester-f28vs": Phase="Pending", Reason="", readiness=false. Elapsed: 2.209936636s
    Jan 17 20:53:06.969: INFO: Pod "azuredisk-volume-tester-f28vs": Phase="Pending", Reason="", readiness=false. Elapsed: 4.210027854s
    Jan 17 20:53:08.969: INFO: Pod "azuredisk-volume-tester-f28vs": Phase="Pending", Reason="", readiness=false. Elapsed: 6.210033774s
    Jan 17 20:53:10.972: INFO: Pod "azuredisk-volume-tester-f28vs": Phase="Pending", Reason="", readiness=false. Elapsed: 8.212529974s
    Jan 17 20:53:12.982: INFO: Pod "azuredisk-volume-tester-f28vs": Phase="Pending", Reason="", readiness=false. Elapsed: 10.222356533s
... skipping 16 lines ...
    Jan 17 20:53:46.974: INFO: Pod "azuredisk-volume-tester-f28vs": Phase="Running", Reason="", readiness=true. Elapsed: 44.214763344s
    Jan 17 20:53:48.975: INFO: Pod "azuredisk-volume-tester-f28vs": Phase="Running", Reason="", readiness=true. Elapsed: 46.215567523s
    Jan 17 20:53:50.979: INFO: Pod "azuredisk-volume-tester-f28vs": Phase="Running", Reason="", readiness=false. Elapsed: 48.219518269s
    Jan 17 20:53:52.973: INFO: Pod "azuredisk-volume-tester-f28vs": Phase="Running", Reason="", readiness=false. Elapsed: 50.214053918s
    Jan 17 20:53:54.974: INFO: Pod "azuredisk-volume-tester-f28vs": Phase="Succeeded", Reason="", readiness=false. Elapsed: 52.214420818s
    [1mSTEP:[0m Saw pod success [38;5;243m01/17/23 20:53:54.974[0m
    Jan 17 20:53:54.974: INFO: Pod "azuredisk-volume-tester-f28vs" satisfied condition "Succeeded or Failed"
    Jan 17 20:53:54.974: INFO: deleting Pod "azuredisk-6615"/"azuredisk-volume-tester-f28vs"
    Jan 17 20:53:55.109: INFO: Pod azuredisk-volume-tester-f28vs has the following logs: hello world

    [1mSTEP:[0m Deleting pod azuredisk-volume-tester-f28vs in namespace azuredisk-6615 [38;5;243m01/17/23 20:53:55.109[0m
    Jan 17 20:53:55.237: INFO: deleting PVC "azuredisk-6615"/"pvc-jp2x5"
    Jan 17 20:53:55.237: INFO: Deleting PersistentVolumeClaim "pvc-jp2x5"
... skipping 37 lines ...
Jan 17 20:54:40.516: INFO: PersistentVolumeClaim pvc-t4t66 found but phase is Pending instead of Bound.
Jan 17 20:54:42.622: INFO: PersistentVolumeClaim pvc-t4t66 found and phase=Bound (4.314326163s)
[1mSTEP:[0m checking the PVC [38;5;243m01/17/23 20:54:42.622[0m
[1mSTEP:[0m validating provisioned PV [38;5;243m01/17/23 20:54:42.726[0m
[1mSTEP:[0m checking the PV [38;5;243m01/17/23 20:54:42.831[0m
[1mSTEP:[0m deploying the pod [38;5;243m01/17/23 20:54:42.831[0m
[1mSTEP:[0m checking that the pods command exits with no error [38;5;243m01/17/23 20:54:42.943[0m
Jan 17 20:54:42.943: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-q9w4f" in namespace "azuredisk-4970" to be "Succeeded or Failed"
Jan 17 20:54:43.053: INFO: Pod "azuredisk-volume-tester-q9w4f": Phase="Pending", Reason="", readiness=false. Elapsed: 109.961768ms
Jan 17 20:54:45.159: INFO: Pod "azuredisk-volume-tester-q9w4f": Phase="Pending", Reason="", readiness=false. Elapsed: 2.215421294s
Jan 17 20:54:47.161: INFO: Pod "azuredisk-volume-tester-q9w4f": Phase="Pending", Reason="", readiness=false. Elapsed: 4.217674137s
Jan 17 20:54:49.158: INFO: Pod "azuredisk-volume-tester-q9w4f": Phase="Pending", Reason="", readiness=false. Elapsed: 6.215179119s
Jan 17 20:54:51.160: INFO: Pod "azuredisk-volume-tester-q9w4f": Phase="Pending", Reason="", readiness=false. Elapsed: 8.216413984s
Jan 17 20:54:53.158: INFO: Pod "azuredisk-volume-tester-q9w4f": Phase="Pending", Reason="", readiness=false. Elapsed: 10.215068426s
... skipping 15 lines ...
Jan 17 20:55:25.164: INFO: Pod "azuredisk-volume-tester-q9w4f": Phase="Running", Reason="", readiness=true. Elapsed: 42.220934222s
Jan 17 20:55:27.162: INFO: Pod "azuredisk-volume-tester-q9w4f": Phase="Running", Reason="", readiness=true. Elapsed: 44.218945721s
Jan 17 20:55:29.162: INFO: Pod "azuredisk-volume-tester-q9w4f": Phase="Running", Reason="", readiness=false. Elapsed: 46.218834221s
Jan 17 20:55:31.164: INFO: Pod "azuredisk-volume-tester-q9w4f": Phase="Running", Reason="", readiness=false. Elapsed: 48.221146572s
Jan 17 20:55:33.162: INFO: Pod "azuredisk-volume-tester-q9w4f": Phase="Succeeded", Reason="", readiness=false. Elapsed: 50.21920613s
[1mSTEP:[0m Saw pod success [38;5;243m01/17/23 20:55:33.162[0m
Jan 17 20:55:33.163: INFO: Pod "azuredisk-volume-tester-q9w4f" satisfied condition "Succeeded or Failed"
Jan 17 20:55:33.163: INFO: deleting Pod "azuredisk-4970"/"azuredisk-volume-tester-q9w4f"
Jan 17 20:55:33.286: INFO: Pod azuredisk-volume-tester-q9w4f has the following logs: hello world

[1mSTEP:[0m Deleting pod azuredisk-volume-tester-q9w4f in namespace azuredisk-4970 [38;5;243m01/17/23 20:55:33.286[0m
Jan 17 20:55:33.511: INFO: deleting PVC "azuredisk-4970"/"pvc-t4t66"
Jan 17 20:55:33.511: INFO: Deleting PersistentVolumeClaim "pvc-t4t66"
... skipping 41 lines ...
    Jan 17 20:54:40.516: INFO: PersistentVolumeClaim pvc-t4t66 found but phase is Pending instead of Bound.
    Jan 17 20:54:42.622: INFO: PersistentVolumeClaim pvc-t4t66 found and phase=Bound (4.314326163s)
    [1mSTEP:[0m checking the PVC [38;5;243m01/17/23 20:54:42.622[0m
    [1mSTEP:[0m validating provisioned PV [38;5;243m01/17/23 20:54:42.726[0m
    [1mSTEP:[0m checking the PV [38;5;243m01/17/23 20:54:42.831[0m
    [1mSTEP:[0m deploying the pod [38;5;243m01/17/23 20:54:42.831[0m
    [1mSTEP:[0m checking that the pods command exits with no error [38;5;243m01/17/23 20:54:42.943[0m
    Jan 17 20:54:42.943: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-q9w4f" in namespace "azuredisk-4970" to be "Succeeded or Failed"
    Jan 17 20:54:43.053: INFO: Pod "azuredisk-volume-tester-q9w4f": Phase="Pending", Reason="", readiness=false. Elapsed: 109.961768ms
    Jan 17 20:54:45.159: INFO: Pod "azuredisk-volume-tester-q9w4f": Phase="Pending", Reason="", readiness=false. Elapsed: 2.215421294s
    Jan 17 20:54:47.161: INFO: Pod "azuredisk-volume-tester-q9w4f": Phase="Pending", Reason="", readiness=false. Elapsed: 4.217674137s
    Jan 17 20:54:49.158: INFO: Pod "azuredisk-volume-tester-q9w4f": Phase="Pending", Reason="", readiness=false. Elapsed: 6.215179119s
    Jan 17 20:54:51.160: INFO: Pod "azuredisk-volume-tester-q9w4f": Phase="Pending", Reason="", readiness=false. Elapsed: 8.216413984s
    Jan 17 20:54:53.158: INFO: Pod "azuredisk-volume-tester-q9w4f": Phase="Pending", Reason="", readiness=false. Elapsed: 10.215068426s
... skipping 15 lines ...
    Jan 17 20:55:25.164: INFO: Pod "azuredisk-volume-tester-q9w4f": Phase="Running", Reason="", readiness=true. Elapsed: 42.220934222s
    Jan 17 20:55:27.162: INFO: Pod "azuredisk-volume-tester-q9w4f": Phase="Running", Reason="", readiness=true. Elapsed: 44.218945721s
    Jan 17 20:55:29.162: INFO: Pod "azuredisk-volume-tester-q9w4f": Phase="Running", Reason="", readiness=false. Elapsed: 46.218834221s
    Jan 17 20:55:31.164: INFO: Pod "azuredisk-volume-tester-q9w4f": Phase="Running", Reason="", readiness=false. Elapsed: 48.221146572s
    Jan 17 20:55:33.162: INFO: Pod "azuredisk-volume-tester-q9w4f": Phase="Succeeded", Reason="", readiness=false. Elapsed: 50.21920613s
    [1mSTEP:[0m Saw pod success [38;5;243m01/17/23 20:55:33.162[0m
    Jan 17 20:55:33.163: INFO: Pod "azuredisk-volume-tester-q9w4f" satisfied condition "Succeeded or Failed"
    Jan 17 20:55:33.163: INFO: deleting Pod "azuredisk-4970"/"azuredisk-volume-tester-q9w4f"
    Jan 17 20:55:33.286: INFO: Pod azuredisk-volume-tester-q9w4f has the following logs: hello world

    [1mSTEP:[0m Deleting pod azuredisk-volume-tester-q9w4f in namespace azuredisk-4970 [38;5;243m01/17/23 20:55:33.286[0m
    Jan 17 20:55:33.511: INFO: deleting PVC "azuredisk-4970"/"pvc-t4t66"
    Jan 17 20:55:33.511: INFO: Deleting PersistentVolumeClaim "pvc-t4t66"
... skipping 42 lines ...
[1mSTEP:[0m checking the PVC [38;5;243m01/17/23 20:56:36.217[0m
[1mSTEP:[0m validating provisioned PV [38;5;243m01/17/23 20:56:36.324[0m
[1mSTEP:[0m checking the PV [38;5;243m01/17/23 20:56:36.428[0m
[1mSTEP:[0m setting up the pod [38;5;243m01/17/23 20:56:36.428[0m
I0117 20:56:36.428780   41861 resource_setup.go:62] adding PV (pvc-dc1acede-46e2-48e8-9a2c-a5abfa872f05) to pod ()
[1mSTEP:[0m deploying the pod [38;5;243m01/17/23 20:56:36.428[0m
[1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/17/23 20:56:36.542[0m
Jan 17 20:56:36.542: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-hghsv" in namespace "azuredisk-6530" to be "Succeeded or Failed"
Jan 17 20:56:36.648: INFO: Pod "azuredisk-volume-tester-hghsv": Phase="Pending", Reason="", readiness=false. Elapsed: 105.622987ms
Jan 17 20:56:38.762: INFO: Pod "azuredisk-volume-tester-hghsv": Phase="Pending", Reason="", readiness=false. Elapsed: 2.220015632s
Jan 17 20:56:40.755: INFO: Pod "azuredisk-volume-tester-hghsv": Phase="Pending", Reason="", readiness=false. Elapsed: 4.21203318s
Jan 17 20:56:42.752: INFO: Pod "azuredisk-volume-tester-hghsv": Phase="Pending", Reason="", readiness=false. Elapsed: 6.209666416s
Jan 17 20:56:44.754: INFO: Pod "azuredisk-volume-tester-hghsv": Phase="Pending", Reason="", readiness=false. Elapsed: 8.211729531s
Jan 17 20:56:46.754: INFO: Pod "azuredisk-volume-tester-hghsv": Phase="Pending", Reason="", readiness=false. Elapsed: 10.211215103s
... skipping 17 lines ...
Jan 17 20:57:22.759: INFO: Pod "azuredisk-volume-tester-hghsv": Phase="Running", Reason="", readiness=true. Elapsed: 46.21650546s
Jan 17 20:57:24.756: INFO: Pod "azuredisk-volume-tester-hghsv": Phase="Running", Reason="", readiness=false. Elapsed: 48.213972463s
Jan 17 20:57:26.758: INFO: Pod "azuredisk-volume-tester-hghsv": Phase="Running", Reason="", readiness=false. Elapsed: 50.215918383s
Jan 17 20:57:28.758: INFO: Pod "azuredisk-volume-tester-hghsv": Phase="Running", Reason="", readiness=false. Elapsed: 52.215998801s
Jan 17 20:57:30.758: INFO: Pod "azuredisk-volume-tester-hghsv": Phase="Succeeded", Reason="", readiness=false. Elapsed: 54.215275244s
[1mSTEP:[0m Saw pod success [38;5;243m01/17/23 20:57:30.758[0m
Jan 17 20:57:30.758: INFO: Pod "azuredisk-volume-tester-hghsv" satisfied condition "Succeeded or Failed"
Jan 17 20:57:30.758: INFO: deleting Pod "azuredisk-6530"/"azuredisk-volume-tester-hghsv"
Jan 17 20:57:30.881: INFO: Pod azuredisk-volume-tester-hghsv has the following logs: hello world

[1mSTEP:[0m Deleting pod azuredisk-volume-tester-hghsv in namespace azuredisk-6530 [38;5;243m01/17/23 20:57:30.881[0m
Jan 17 20:57:31.009: INFO: deleting PVC "azuredisk-6530"/"pvc-ls478"
Jan 17 20:57:31.009: INFO: Deleting PersistentVolumeClaim "pvc-ls478"
... skipping 40 lines ...
    [1mSTEP:[0m checking the PVC [38;5;243m01/17/23 20:56:36.217[0m
    [1mSTEP:[0m validating provisioned PV [38;5;243m01/17/23 20:56:36.324[0m
    [1mSTEP:[0m checking the PV [38;5;243m01/17/23 20:56:36.428[0m
    [1mSTEP:[0m setting up the pod [38;5;243m01/17/23 20:56:36.428[0m
    I0117 20:56:36.428780   41861 resource_setup.go:62] adding PV (pvc-dc1acede-46e2-48e8-9a2c-a5abfa872f05) to pod ()
    [1mSTEP:[0m deploying the pod [38;5;243m01/17/23 20:56:36.428[0m
    [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/17/23 20:56:36.542[0m
    Jan 17 20:56:36.542: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-hghsv" in namespace "azuredisk-6530" to be "Succeeded or Failed"
    Jan 17 20:56:36.648: INFO: Pod "azuredisk-volume-tester-hghsv": Phase="Pending", Reason="", readiness=false. Elapsed: 105.622987ms
    Jan 17 20:56:38.762: INFO: Pod "azuredisk-volume-tester-hghsv": Phase="Pending", Reason="", readiness=false. Elapsed: 2.220015632s
    Jan 17 20:56:40.755: INFO: Pod "azuredisk-volume-tester-hghsv": Phase="Pending", Reason="", readiness=false. Elapsed: 4.21203318s
    Jan 17 20:56:42.752: INFO: Pod "azuredisk-volume-tester-hghsv": Phase="Pending", Reason="", readiness=false. Elapsed: 6.209666416s
    Jan 17 20:56:44.754: INFO: Pod "azuredisk-volume-tester-hghsv": Phase="Pending", Reason="", readiness=false. Elapsed: 8.211729531s
    Jan 17 20:56:46.754: INFO: Pod "azuredisk-volume-tester-hghsv": Phase="Pending", Reason="", readiness=false. Elapsed: 10.211215103s
... skipping 17 lines ...
    Jan 17 20:57:22.759: INFO: Pod "azuredisk-volume-tester-hghsv": Phase="Running", Reason="", readiness=true. Elapsed: 46.21650546s
    Jan 17 20:57:24.756: INFO: Pod "azuredisk-volume-tester-hghsv": Phase="Running", Reason="", readiness=false. Elapsed: 48.213972463s
    Jan 17 20:57:26.758: INFO: Pod "azuredisk-volume-tester-hghsv": Phase="Running", Reason="", readiness=false. Elapsed: 50.215918383s
    Jan 17 20:57:28.758: INFO: Pod "azuredisk-volume-tester-hghsv": Phase="Running", Reason="", readiness=false. Elapsed: 52.215998801s
    Jan 17 20:57:30.758: INFO: Pod "azuredisk-volume-tester-hghsv": Phase="Succeeded", Reason="", readiness=false. Elapsed: 54.215275244s
    [1mSTEP:[0m Saw pod success [38;5;243m01/17/23 20:57:30.758[0m
    Jan 17 20:57:30.758: INFO: Pod "azuredisk-volume-tester-hghsv" satisfied condition "Succeeded or Failed"
    Jan 17 20:57:30.758: INFO: deleting Pod "azuredisk-6530"/"azuredisk-volume-tester-hghsv"
    Jan 17 20:57:30.881: INFO: Pod azuredisk-volume-tester-hghsv has the following logs: hello world

    [1mSTEP:[0m Deleting pod azuredisk-volume-tester-hghsv in namespace azuredisk-6530 [38;5;243m01/17/23 20:57:30.881[0m
    Jan 17 20:57:31.009: INFO: deleting PVC "azuredisk-6530"/"pvc-ls478"
    Jan 17 20:57:31.009: INFO: Deleting PersistentVolumeClaim "pvc-ls478"
... skipping 39 lines ...
[1mSTEP:[0m checking the PVC [38;5;243m01/17/23 20:58:18.399[0m
[1mSTEP:[0m validating provisioned PV [38;5;243m01/17/23 20:58:18.503[0m
[1mSTEP:[0m checking the PV [38;5;243m01/17/23 20:58:18.608[0m
[1mSTEP:[0m setting up the pod [38;5;243m01/17/23 20:58:18.608[0m
I0117 20:58:18.608336   41861 resource_setup.go:62] adding PV (pvc-12d12ae2-b6a2-4170-a917-98abe1860bf7) to pod ()
[1mSTEP:[0m deploying the pod [38;5;243m01/17/23 20:58:18.608[0m
[1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/17/23 20:58:18.72[0m
Jan 17 20:58:18.720: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-xpbrq" in namespace "azuredisk-6036" to be "Succeeded or Failed"
Jan 17 20:58:18.823: INFO: Pod "azuredisk-volume-tester-xpbrq": Phase="Pending", Reason="", readiness=false. Elapsed: 103.495205ms
Jan 17 20:58:20.931: INFO: Pod "azuredisk-volume-tester-xpbrq": Phase="Pending", Reason="", readiness=false. Elapsed: 2.211057414s
Jan 17 20:58:22.928: INFO: Pod "azuredisk-volume-tester-xpbrq": Phase="Pending", Reason="", readiness=false. Elapsed: 4.208551074s
Jan 17 20:58:24.928: INFO: Pod "azuredisk-volume-tester-xpbrq": Phase="Pending", Reason="", readiness=false. Elapsed: 6.208488338s
Jan 17 20:58:26.929: INFO: Pod "azuredisk-volume-tester-xpbrq": Phase="Pending", Reason="", readiness=false. Elapsed: 8.209192884s
Jan 17 20:58:28.931: INFO: Pod "azuredisk-volume-tester-xpbrq": Phase="Pending", Reason="", readiness=false. Elapsed: 10.21106591s
... skipping 16 lines ...
Jan 17 20:59:02.934: INFO: Pod "azuredisk-volume-tester-xpbrq": Phase="Running", Reason="", readiness=true. Elapsed: 44.214232584s
Jan 17 20:59:04.933: INFO: Pod "azuredisk-volume-tester-xpbrq": Phase="Running", Reason="", readiness=true. Elapsed: 46.212855314s
Jan 17 20:59:06.934: INFO: Pod "azuredisk-volume-tester-xpbrq": Phase="Running", Reason="", readiness=false. Elapsed: 48.214621864s
Jan 17 20:59:08.937: INFO: Pod "azuredisk-volume-tester-xpbrq": Phase="Running", Reason="", readiness=false. Elapsed: 50.216988593s
Jan 17 20:59:10.932: INFO: Pod "azuredisk-volume-tester-xpbrq": Phase="Succeeded", Reason="", readiness=false. Elapsed: 52.2123807s
[1mSTEP:[0m Saw pod success [38;5;243m01/17/23 20:59:10.932[0m
Jan 17 20:59:10.932: INFO: Pod "azuredisk-volume-tester-xpbrq" satisfied condition "Succeeded or Failed"
Jan 17 20:59:10.932: INFO: deleting Pod "azuredisk-6036"/"azuredisk-volume-tester-xpbrq"
Jan 17 20:59:11.060: INFO: Pod azuredisk-volume-tester-xpbrq has the following logs: hello world

[1mSTEP:[0m Deleting pod azuredisk-volume-tester-xpbrq in namespace azuredisk-6036 [38;5;243m01/17/23 20:59:11.06[0m
Jan 17 20:59:11.263: INFO: deleting PVC "azuredisk-6036"/"pvc-vsk92"
Jan 17 20:59:11.264: INFO: Deleting PersistentVolumeClaim "pvc-vsk92"
... skipping 40 lines ...
    [1mSTEP:[0m checking the PVC [38;5;243m01/17/23 20:58:18.399[0m
    [1mSTEP:[0m validating provisioned PV [38;5;243m01/17/23 20:58:18.503[0m
    [1mSTEP:[0m checking the PV [38;5;243m01/17/23 20:58:18.608[0m
    [1mSTEP:[0m setting up the pod [38;5;243m01/17/23 20:58:18.608[0m
    I0117 20:58:18.608336   41861 resource_setup.go:62] adding PV (pvc-12d12ae2-b6a2-4170-a917-98abe1860bf7) to pod ()
    [1mSTEP:[0m deploying the pod [38;5;243m01/17/23 20:58:18.608[0m
    [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/17/23 20:58:18.72[0m
    Jan 17 20:58:18.720: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-xpbrq" in namespace "azuredisk-6036" to be "Succeeded or Failed"
    Jan 17 20:58:18.823: INFO: Pod "azuredisk-volume-tester-xpbrq": Phase="Pending", Reason="", readiness=false. Elapsed: 103.495205ms
    Jan 17 20:58:20.931: INFO: Pod "azuredisk-volume-tester-xpbrq": Phase="Pending", Reason="", readiness=false. Elapsed: 2.211057414s
    Jan 17 20:58:22.928: INFO: Pod "azuredisk-volume-tester-xpbrq": Phase="Pending", Reason="", readiness=false. Elapsed: 4.208551074s
    Jan 17 20:58:24.928: INFO: Pod "azuredisk-volume-tester-xpbrq": Phase="Pending", Reason="", readiness=false. Elapsed: 6.208488338s
    Jan 17 20:58:26.929: INFO: Pod "azuredisk-volume-tester-xpbrq": Phase="Pending", Reason="", readiness=false. Elapsed: 8.209192884s
    Jan 17 20:58:28.931: INFO: Pod "azuredisk-volume-tester-xpbrq": Phase="Pending", Reason="", readiness=false. Elapsed: 10.21106591s
... skipping 16 lines ...
    Jan 17 20:59:02.934: INFO: Pod "azuredisk-volume-tester-xpbrq": Phase="Running", Reason="", readiness=true. Elapsed: 44.214232584s
    Jan 17 20:59:04.933: INFO: Pod "azuredisk-volume-tester-xpbrq": Phase="Running", Reason="", readiness=true. Elapsed: 46.212855314s
    Jan 17 20:59:06.934: INFO: Pod "azuredisk-volume-tester-xpbrq": Phase="Running", Reason="", readiness=false. Elapsed: 48.214621864s
    Jan 17 20:59:08.937: INFO: Pod "azuredisk-volume-tester-xpbrq": Phase="Running", Reason="", readiness=false. Elapsed: 50.216988593s
    Jan 17 20:59:10.932: INFO: Pod "azuredisk-volume-tester-xpbrq": Phase="Succeeded", Reason="", readiness=false. Elapsed: 52.2123807s
    [1mSTEP:[0m Saw pod success [38;5;243m01/17/23 20:59:10.932[0m
    Jan 17 20:59:10.932: INFO: Pod "azuredisk-volume-tester-xpbrq" satisfied condition "Succeeded or Failed"
    Jan 17 20:59:10.932: INFO: deleting Pod "azuredisk-6036"/"azuredisk-volume-tester-xpbrq"
    Jan 17 20:59:11.060: INFO: Pod azuredisk-volume-tester-xpbrq has the following logs: hello world

    [1mSTEP:[0m Deleting pod azuredisk-volume-tester-xpbrq in namespace azuredisk-6036 [38;5;243m01/17/23 20:59:11.06[0m
    Jan 17 20:59:11.263: INFO: deleting PVC "azuredisk-6036"/"pvc-vsk92"
    Jan 17 20:59:11.264: INFO: Deleting PersistentVolumeClaim "pvc-vsk92"
... skipping 111 lines ...
[1mSTEP:[0m checking the PVC [38;5;243m01/17/23 21:00:01.83[0m
[1mSTEP:[0m validating provisioned PV [38;5;243m01/17/23 21:00:01.935[0m
[1mSTEP:[0m checking the PV [38;5;243m01/17/23 21:00:02.038[0m
[1mSTEP:[0m setting up the pod [38;5;243m01/17/23 21:00:02.039[0m
I0117 21:00:02.039139   41861 resource_setup.go:62] adding PV (pvc-3e929c22-2b53-4acd-b131-0b489234b929) to pod ()
[1mSTEP:[0m deploying the pod [38;5;243m01/17/23 21:00:02.039[0m
[1mSTEP:[0m checking that the pod's command exits with an error [38;5;243m01/17/23 21:00:02.151[0m
Jan 17 21:00:02.151: INFO: Waiting up to 10m0s for pod "azuredisk-volume-tester-965ft" in namespace "azuredisk-5474" to be "Error status code"
Jan 17 21:00:02.255: INFO: Pod "azuredisk-volume-tester-965ft": Phase="Pending", Reason="", readiness=false. Elapsed: 103.930214ms
Jan 17 21:00:04.371: INFO: Pod "azuredisk-volume-tester-965ft": Phase="Pending", Reason="", readiness=false. Elapsed: 2.21990384s
Jan 17 21:00:06.359: INFO: Pod "azuredisk-volume-tester-965ft": Phase="Pending", Reason="", readiness=false. Elapsed: 4.207966819s
Jan 17 21:00:08.362: INFO: Pod "azuredisk-volume-tester-965ft": Phase="Pending", Reason="", readiness=false. Elapsed: 6.210340405s
Jan 17 21:00:10.360: INFO: Pod "azuredisk-volume-tester-965ft": Phase="Pending", Reason="", readiness=false. Elapsed: 8.208598776s
Jan 17 21:00:12.361: INFO: Pod "azuredisk-volume-tester-965ft": Phase="Pending", Reason="", readiness=false. Elapsed: 10.209347183s
... skipping 27 lines ...
Jan 17 21:01:08.370: INFO: Pod "azuredisk-volume-tester-965ft": Phase="Running", Reason="", readiness=true. Elapsed: 1m6.219129127s
Jan 17 21:01:10.364: INFO: Pod "azuredisk-volume-tester-965ft": Phase="Running", Reason="", readiness=true. Elapsed: 1m8.213146023s
Jan 17 21:01:12.365: INFO: Pod "azuredisk-volume-tester-965ft": Phase="Running", Reason="", readiness=true. Elapsed: 1m10.21385144s
Jan 17 21:01:14.368: INFO: Pod "azuredisk-volume-tester-965ft": Phase="Running", Reason="", readiness=true. Elapsed: 1m12.216396722s
Jan 17 21:01:16.366: INFO: Pod "azuredisk-volume-tester-965ft": Phase="Running", Reason="", readiness=false. Elapsed: 1m14.214666446s
Jan 17 21:01:18.366: INFO: Pod "azuredisk-volume-tester-965ft": Phase="Running", Reason="", readiness=false. Elapsed: 1m16.214757076s
Jan 17 21:01:20.366: INFO: Pod "azuredisk-volume-tester-965ft": Phase="Failed", Reason="", readiness=false. Elapsed: 1m18.214320738s
[1mSTEP:[0m Saw pod failure [38;5;243m01/17/23 21:01:20.366[0m
Jan 17 21:01:20.366: INFO: Pod "azuredisk-volume-tester-965ft" satisfied condition "Error status code"
[1mSTEP:[0m checking that pod logs contain expected message [38;5;243m01/17/23 21:01:20.366[0m
Jan 17 21:01:20.493: INFO: deleting Pod "azuredisk-5474"/"azuredisk-volume-tester-965ft"
Jan 17 21:01:20.610: INFO: Pod azuredisk-volume-tester-965ft has the following logs: out-file : Access to the path 'C:\mnt\test-1\data' is denied.
At line:1 char:1
+ echo $null >> C:\mnt\test-1\data
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
... skipping 49 lines ...
    [1mSTEP:[0m checking the PVC [38;5;243m01/17/23 21:00:01.83[0m
    [1mSTEP:[0m validating provisioned PV [38;5;243m01/17/23 21:00:01.935[0m
    [1mSTEP:[0m checking the PV [38;5;243m01/17/23 21:00:02.038[0m
    [1mSTEP:[0m setting up the pod [38;5;243m01/17/23 21:00:02.039[0m
    I0117 21:00:02.039139   41861 resource_setup.go:62] adding PV (pvc-3e929c22-2b53-4acd-b131-0b489234b929) to pod ()
    [1mSTEP:[0m deploying the pod [38;5;243m01/17/23 21:00:02.039[0m
    [1mSTEP:[0m checking that the pod's command exits with an error [38;5;243m01/17/23 21:00:02.151[0m
    Jan 17 21:00:02.151: INFO: Waiting up to 10m0s for pod "azuredisk-volume-tester-965ft" in namespace "azuredisk-5474" to be "Error status code"
    Jan 17 21:00:02.255: INFO: Pod "azuredisk-volume-tester-965ft": Phase="Pending", Reason="", readiness=false. Elapsed: 103.930214ms
    Jan 17 21:00:04.371: INFO: Pod "azuredisk-volume-tester-965ft": Phase="Pending", Reason="", readiness=false. Elapsed: 2.21990384s
    Jan 17 21:00:06.359: INFO: Pod "azuredisk-volume-tester-965ft": Phase="Pending", Reason="", readiness=false. Elapsed: 4.207966819s
    Jan 17 21:00:08.362: INFO: Pod "azuredisk-volume-tester-965ft": Phase="Pending", Reason="", readiness=false. Elapsed: 6.210340405s
    Jan 17 21:00:10.360: INFO: Pod "azuredisk-volume-tester-965ft": Phase="Pending", Reason="", readiness=false. Elapsed: 8.208598776s
    Jan 17 21:00:12.361: INFO: Pod "azuredisk-volume-tester-965ft": Phase="Pending", Reason="", readiness=false. Elapsed: 10.209347183s
... skipping 27 lines ...
    Jan 17 21:01:08.370: INFO: Pod "azuredisk-volume-tester-965ft": Phase="Running", Reason="", readiness=true. Elapsed: 1m6.219129127s
    Jan 17 21:01:10.364: INFO: Pod "azuredisk-volume-tester-965ft": Phase="Running", Reason="", readiness=true. Elapsed: 1m8.213146023s
    Jan 17 21:01:12.365: INFO: Pod "azuredisk-volume-tester-965ft": Phase="Running", Reason="", readiness=true. Elapsed: 1m10.21385144s
    Jan 17 21:01:14.368: INFO: Pod "azuredisk-volume-tester-965ft": Phase="Running", Reason="", readiness=true. Elapsed: 1m12.216396722s
    Jan 17 21:01:16.366: INFO: Pod "azuredisk-volume-tester-965ft": Phase="Running", Reason="", readiness=false. Elapsed: 1m14.214666446s
    Jan 17 21:01:18.366: INFO: Pod "azuredisk-volume-tester-965ft": Phase="Running", Reason="", readiness=false. Elapsed: 1m16.214757076s
    Jan 17 21:01:20.366: INFO: Pod "azuredisk-volume-tester-965ft": Phase="Failed", Reason="", readiness=false. Elapsed: 1m18.214320738s
    [1mSTEP:[0m Saw pod failure [38;5;243m01/17/23 21:01:20.366[0m
    Jan 17 21:01:20.366: INFO: Pod "azuredisk-volume-tester-965ft" satisfied condition "Error status code"
    [1mSTEP:[0m checking that pod logs contain expected message [38;5;243m01/17/23 21:01:20.366[0m
    Jan 17 21:01:20.493: INFO: deleting Pod "azuredisk-5474"/"azuredisk-volume-tester-965ft"
    Jan 17 21:01:20.610: INFO: Pod azuredisk-volume-tester-965ft has the following logs: out-file : Access to the path 'C:\mnt\test-1\data' is denied.
    At line:1 char:1
    + echo $null >> C:\mnt\test-1\data
    + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
... skipping 489 lines ...
Jan 17 21:07:51.422: INFO: deployment status: v1.DeploymentStatus{ObservedGeneration:1, Replicas:1, UpdatedReplicas:1, ReadyReplicas:0, AvailableReplicas:0, UnavailableReplicas:1, Conditions:[]v1.DeploymentCondition{v1.DeploymentCondition{Type:"Available", Status:"False", LastUpdateTime:time.Date(2023, time.January, 17, 21, 7, 17, 0, time.Local), LastTransitionTime:time.Date(2023, time.January, 17, 21, 7, 17, 0, time.Local), Reason:"MinimumReplicasUnavailable", Message:"Deployment does not have minimum availability."}, v1.DeploymentCondition{Type:"Progressing", Status:"True", LastUpdateTime:time.Date(2023, time.January, 17, 21, 7, 17, 0, time.Local), LastTransitionTime:time.Date(2023, time.January, 17, 21, 7, 17, 0, time.Local), Reason:"ReplicaSetUpdated", Message:"ReplicaSet \"azuredisk-volume-tester-rjsfq-849b58d585\" is progressing."}}, CollisionCount:(*int32)(nil)}
Jan 17 21:07:53.424: INFO: deployment status: v1.DeploymentStatus{ObservedGeneration:1, Replicas:1, UpdatedReplicas:1, ReadyReplicas:0, AvailableReplicas:0, UnavailableReplicas:1, Conditions:[]v1.DeploymentCondition{v1.DeploymentCondition{Type:"Available", Status:"False", LastUpdateTime:time.Date(2023, time.January, 17, 21, 7, 17, 0, time.Local), LastTransitionTime:time.Date(2023, time.January, 17, 21, 7, 17, 0, time.Local), Reason:"MinimumReplicasUnavailable", Message:"Deployment does not have minimum availability."}, v1.DeploymentCondition{Type:"Progressing", Status:"True", LastUpdateTime:time.Date(2023, time.January, 17, 21, 7, 17, 0, time.Local), LastTransitionTime:time.Date(2023, time.January, 17, 21, 7, 17, 0, time.Local), Reason:"ReplicaSetUpdated", Message:"ReplicaSet \"azuredisk-volume-tester-rjsfq-849b58d585\" is progressing."}}, CollisionCount:(*int32)(nil)}
[1mSTEP:[0m checking that the pod is running [38;5;243m01/17/23 21:07:55.636[0m
[1mSTEP:[0m check pod exec [38;5;243m01/17/23 21:07:55.952[0m
Jan 17 21:07:55.952: INFO: Running '/usr/local/bin/kubectl --kubeconfig=/home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure/kubeconfig --namespace=azuredisk-1183 exec azuredisk-volume-tester-rjsfq-849b58d585-p8p7f -- cmd /c type C:\mnt\test-1\data.txt'
Jan 17 21:07:57.120: INFO: rc: 1
Jan 17 21:07:57.120: INFO: Error waiting for output "hello world\r\n" in pod "azuredisk-volume-tester-rjsfq-849b58d585-p8p7f": error running /usr/local/bin/kubectl --kubeconfig=/home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure/kubeconfig --namespace=azuredisk-1183 exec azuredisk-volume-tester-rjsfq-849b58d585-p8p7f -- cmd /c type C:\mnt\test-1\data.txt:
Command stdout:

stderr:
The system cannot find the file specified.
command terminated with exit code 1

error:
exit status 1.
Jan 17 21:07:59.121: INFO: Running '/usr/local/bin/kubectl --kubeconfig=/home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure/kubeconfig --namespace=azuredisk-1183 exec azuredisk-volume-tester-rjsfq-849b58d585-p8p7f -- cmd /c type C:\mnt\test-1\data.txt'
Jan 17 21:08:00.255: INFO: stderr: ""
Jan 17 21:08:00.255: INFO: stdout: "hello world\r\n"
[1mSTEP:[0m deleting the pod for deployment [38;5;243m01/17/23 21:08:00.255[0m
Jan 17 21:08:00.255: INFO: Deleting pod "azuredisk-volume-tester-rjsfq-849b58d585-p8p7f" in namespace "azuredisk-1183"
... skipping 79 lines ...
    Jan 17 21:07:51.422: INFO: deployment status: v1.DeploymentStatus{ObservedGeneration:1, Replicas:1, UpdatedReplicas:1, ReadyReplicas:0, AvailableReplicas:0, UnavailableReplicas:1, Conditions:[]v1.DeploymentCondition{v1.DeploymentCondition{Type:"Available", Status:"False", LastUpdateTime:time.Date(2023, time.January, 17, 21, 7, 17, 0, time.Local), LastTransitionTime:time.Date(2023, time.January, 17, 21, 7, 17, 0, time.Local), Reason:"MinimumReplicasUnavailable", Message:"Deployment does not have minimum availability."}, v1.DeploymentCondition{Type:"Progressing", Status:"True", LastUpdateTime:time.Date(2023, time.January, 17, 21, 7, 17, 0, time.Local), LastTransitionTime:time.Date(2023, time.January, 17, 21, 7, 17, 0, time.Local), Reason:"ReplicaSetUpdated", Message:"ReplicaSet \"azuredisk-volume-tester-rjsfq-849b58d585\" is progressing."}}, CollisionCount:(*int32)(nil)}
    Jan 17 21:07:53.424: INFO: deployment status: v1.DeploymentStatus{ObservedGeneration:1, Replicas:1, UpdatedReplicas:1, ReadyReplicas:0, AvailableReplicas:0, UnavailableReplicas:1, Conditions:[]v1.DeploymentCondition{v1.DeploymentCondition{Type:"Available", Status:"False", LastUpdateTime:time.Date(2023, time.January, 17, 21, 7, 17, 0, time.Local), LastTransitionTime:time.Date(2023, time.January, 17, 21, 7, 17, 0, time.Local), Reason:"MinimumReplicasUnavailable", Message:"Deployment does not have minimum availability."}, v1.DeploymentCondition{Type:"Progressing", Status:"True", LastUpdateTime:time.Date(2023, time.January, 17, 21, 7, 17, 0, time.Local), LastTransitionTime:time.Date(2023, time.January, 17, 21, 7, 17, 0, time.Local), Reason:"ReplicaSetUpdated", Message:"ReplicaSet \"azuredisk-volume-tester-rjsfq-849b58d585\" is progressing."}}, CollisionCount:(*int32)(nil)}
    [1mSTEP:[0m checking that the pod is running [38;5;243m01/17/23 21:07:55.636[0m
    [1mSTEP:[0m check pod exec [38;5;243m01/17/23 21:07:55.952[0m
    Jan 17 21:07:55.952: INFO: Running '/usr/local/bin/kubectl --kubeconfig=/home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure/kubeconfig --namespace=azuredisk-1183 exec azuredisk-volume-tester-rjsfq-849b58d585-p8p7f -- cmd /c type C:\mnt\test-1\data.txt'
    Jan 17 21:07:57.120: INFO: rc: 1
    Jan 17 21:07:57.120: INFO: Error waiting for output "hello world\r\n" in pod "azuredisk-volume-tester-rjsfq-849b58d585-p8p7f": error running /usr/local/bin/kubectl --kubeconfig=/home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure/kubeconfig --namespace=azuredisk-1183 exec azuredisk-volume-tester-rjsfq-849b58d585-p8p7f -- cmd /c type C:\mnt\test-1\data.txt:
    Command stdout:

    stderr:
    The system cannot find the file specified.
    command terminated with exit code 1

    error:
    exit status 1.
    Jan 17 21:07:59.121: INFO: Running '/usr/local/bin/kubectl --kubeconfig=/home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure/kubeconfig --namespace=azuredisk-1183 exec azuredisk-volume-tester-rjsfq-849b58d585-p8p7f -- cmd /c type C:\mnt\test-1\data.txt'
    Jan 17 21:08:00.255: INFO: stderr: ""
    Jan 17 21:08:00.255: INFO: stdout: "hello world\r\n"
    [1mSTEP:[0m deleting the pod for deployment [38;5;243m01/17/23 21:08:00.255[0m
    Jan 17 21:08:00.255: INFO: Deleting pod "azuredisk-volume-tester-rjsfq-849b58d585-p8p7f" in namespace "azuredisk-1183"
... skipping 312 lines ...
[1mSTEP:[0m checking the PVC [38;5;243m01/17/23 21:09:41.998[0m
[1mSTEP:[0m validating provisioned PV [38;5;243m01/17/23 21:09:42.103[0m
[1mSTEP:[0m checking the PV [38;5;243m01/17/23 21:09:42.207[0m
[1mSTEP:[0m setting up the pod [38;5;243m01/17/23 21:09:42.208[0m
I0117 21:09:42.208102   41861 resource_setup.go:62] adding PV (pvc-f99f2be0-b4d3-4055-acc1-6337ddb600da) to pod ()
[1mSTEP:[0m deploying the pod [38;5;243m01/17/23 21:09:42.208[0m
[1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/17/23 21:09:42.322[0m
Jan 17 21:09:42.323: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-8g2x5" in namespace "azuredisk-3565" to be "Succeeded or Failed"
Jan 17 21:09:42.431: INFO: Pod "azuredisk-volume-tester-8g2x5": Phase="Pending", Reason="", readiness=false. Elapsed: 108.290018ms
Jan 17 21:09:44.540: INFO: Pod "azuredisk-volume-tester-8g2x5": Phase="Pending", Reason="", readiness=false. Elapsed: 2.217070124s
Jan 17 21:09:46.541: INFO: Pod "azuredisk-volume-tester-8g2x5": Phase="Pending", Reason="", readiness=false. Elapsed: 4.218049501s
Jan 17 21:09:48.541: INFO: Pod "azuredisk-volume-tester-8g2x5": Phase="Pending", Reason="", readiness=false. Elapsed: 6.217772738s
Jan 17 21:09:50.542: INFO: Pod "azuredisk-volume-tester-8g2x5": Phase="Pending", Reason="", readiness=false. Elapsed: 8.218756154s
Jan 17 21:09:52.542: INFO: Pod "azuredisk-volume-tester-8g2x5": Phase="Pending", Reason="", readiness=false. Elapsed: 10.219306575s
... skipping 33 lines ...
Jan 17 21:11:00.540: INFO: Pod "azuredisk-volume-tester-8g2x5": Phase="Running", Reason="", readiness=true. Elapsed: 1m18.216740126s
Jan 17 21:11:02.541: INFO: Pod "azuredisk-volume-tester-8g2x5": Phase="Running", Reason="", readiness=false. Elapsed: 1m20.218183301s
Jan 17 21:11:04.542: INFO: Pod "azuredisk-volume-tester-8g2x5": Phase="Running", Reason="", readiness=false. Elapsed: 1m22.219622065s
Jan 17 21:11:06.540: INFO: Pod "azuredisk-volume-tester-8g2x5": Phase="Running", Reason="", readiness=false. Elapsed: 1m24.21730921s
Jan 17 21:11:08.540: INFO: Pod "azuredisk-volume-tester-8g2x5": Phase="Succeeded", Reason="", readiness=false. Elapsed: 1m26.217462368s
[1mSTEP:[0m Saw pod success [38;5;243m01/17/23 21:11:08.54[0m
Jan 17 21:11:08.540: INFO: Pod "azuredisk-volume-tester-8g2x5" satisfied condition "Succeeded or Failed"
Jan 17 21:11:08.540: INFO: deleting Pod "azuredisk-3565"/"azuredisk-volume-tester-8g2x5"
Jan 17 21:11:08.671: INFO: Pod azuredisk-volume-tester-8g2x5 has the following logs: hello world
hello world
hello world

[1mSTEP:[0m Deleting pod azuredisk-volume-tester-8g2x5 in namespace azuredisk-3565 [38;5;243m01/17/23 21:11:08.671[0m
... skipping 89 lines ...
    [1mSTEP:[0m checking the PVC [38;5;243m01/17/23 21:09:41.998[0m
    [1mSTEP:[0m validating provisioned PV [38;5;243m01/17/23 21:09:42.103[0m
    [1mSTEP:[0m checking the PV [38;5;243m01/17/23 21:09:42.207[0m
    [1mSTEP:[0m setting up the pod [38;5;243m01/17/23 21:09:42.208[0m
    I0117 21:09:42.208102   41861 resource_setup.go:62] adding PV (pvc-f99f2be0-b4d3-4055-acc1-6337ddb600da) to pod ()
    [1mSTEP:[0m deploying the pod [38;5;243m01/17/23 21:09:42.208[0m
    [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/17/23 21:09:42.322[0m
    Jan 17 21:09:42.323: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-8g2x5" in namespace "azuredisk-3565" to be "Succeeded or Failed"
    Jan 17 21:09:42.431: INFO: Pod "azuredisk-volume-tester-8g2x5": Phase="Pending", Reason="", readiness=false. Elapsed: 108.290018ms
    Jan 17 21:09:44.540: INFO: Pod "azuredisk-volume-tester-8g2x5": Phase="Pending", Reason="", readiness=false. Elapsed: 2.217070124s
    Jan 17 21:09:46.541: INFO: Pod "azuredisk-volume-tester-8g2x5": Phase="Pending", Reason="", readiness=false. Elapsed: 4.218049501s
    Jan 17 21:09:48.541: INFO: Pod "azuredisk-volume-tester-8g2x5": Phase="Pending", Reason="", readiness=false. Elapsed: 6.217772738s
    Jan 17 21:09:50.542: INFO: Pod "azuredisk-volume-tester-8g2x5": Phase="Pending", Reason="", readiness=false. Elapsed: 8.218756154s
    Jan 17 21:09:52.542: INFO: Pod "azuredisk-volume-tester-8g2x5": Phase="Pending", Reason="", readiness=false. Elapsed: 10.219306575s
... skipping 33 lines ...
    Jan 17 21:11:00.540: INFO: Pod "azuredisk-volume-tester-8g2x5": Phase="Running", Reason="", readiness=true. Elapsed: 1m18.216740126s
    Jan 17 21:11:02.541: INFO: Pod "azuredisk-volume-tester-8g2x5": Phase="Running", Reason="", readiness=false. Elapsed: 1m20.218183301s
    Jan 17 21:11:04.542: INFO: Pod "azuredisk-volume-tester-8g2x5": Phase="Running", Reason="", readiness=false. Elapsed: 1m22.219622065s
    Jan 17 21:11:06.540: INFO: Pod "azuredisk-volume-tester-8g2x5": Phase="Running", Reason="", readiness=false. Elapsed: 1m24.21730921s
    Jan 17 21:11:08.540: INFO: Pod "azuredisk-volume-tester-8g2x5": Phase="Succeeded", Reason="", readiness=false. Elapsed: 1m26.217462368s
    [1mSTEP:[0m Saw pod success [38;5;243m01/17/23 21:11:08.54[0m
    Jan 17 21:11:08.540: INFO: Pod "azuredisk-volume-tester-8g2x5" satisfied condition "Succeeded or Failed"
    Jan 17 21:11:08.540: INFO: deleting Pod "azuredisk-3565"/"azuredisk-volume-tester-8g2x5"
    Jan 17 21:11:08.671: INFO: Pod azuredisk-volume-tester-8g2x5 has the following logs: hello world
    hello world
    hello world

    [1mSTEP:[0m Deleting pod azuredisk-volume-tester-8g2x5 in namespace azuredisk-3565 [38;5;243m01/17/23 21:11:08.671[0m
... skipping 94 lines ...
Jan 17 21:12:11.803: INFO: PersistentVolumeClaim pvc-7k529 found but phase is Pending instead of Bound.
Jan 17 21:12:13.909: INFO: PersistentVolumeClaim pvc-7k529 found and phase=Bound (4.316376662s)
[1mSTEP:[0m checking the PVC [38;5;243m01/17/23 21:12:13.909[0m
[1mSTEP:[0m validating provisioned PV [38;5;243m01/17/23 21:12:14.013[0m
[1mSTEP:[0m checking the PV [38;5;243m01/17/23 21:12:14.117[0m
[1mSTEP:[0m deploying the pod [38;5;243m01/17/23 21:12:14.117[0m
[1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/17/23 21:12:14.228[0m
Jan 17 21:12:14.228: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-7lm55" in namespace "azuredisk-273" to be "Succeeded or Failed"
Jan 17 21:12:14.332: INFO: Pod "azuredisk-volume-tester-7lm55": Phase="Pending", Reason="", readiness=false. Elapsed: 104.108216ms
Jan 17 21:12:16.437: INFO: Pod "azuredisk-volume-tester-7lm55": Phase="Pending", Reason="", readiness=false. Elapsed: 2.208695672s
Jan 17 21:12:18.447: INFO: Pod "azuredisk-volume-tester-7lm55": Phase="Pending", Reason="", readiness=false. Elapsed: 4.218774545s
Jan 17 21:12:20.443: INFO: Pod "azuredisk-volume-tester-7lm55": Phase="Pending", Reason="", readiness=false. Elapsed: 6.21468064s
Jan 17 21:12:22.438: INFO: Pod "azuredisk-volume-tester-7lm55": Phase="Pending", Reason="", readiness=false. Elapsed: 8.209790812s
Jan 17 21:12:24.439: INFO: Pod "azuredisk-volume-tester-7lm55": Phase="Pending", Reason="", readiness=false. Elapsed: 10.210824896s
... skipping 29 lines ...
Jan 17 21:13:24.443: INFO: Pod "azuredisk-volume-tester-7lm55": Phase="Running", Reason="", readiness=true. Elapsed: 1m10.214543854s
Jan 17 21:13:26.444: INFO: Pod "azuredisk-volume-tester-7lm55": Phase="Running", Reason="", readiness=false. Elapsed: 1m12.21535794s
Jan 17 21:13:28.443: INFO: Pod "azuredisk-volume-tester-7lm55": Phase="Running", Reason="", readiness=false. Elapsed: 1m14.215062586s
Jan 17 21:13:30.442: INFO: Pod "azuredisk-volume-tester-7lm55": Phase="Running", Reason="", readiness=false. Elapsed: 1m16.21342565s
Jan 17 21:13:32.444: INFO: Pod "azuredisk-volume-tester-7lm55": Phase="Succeeded", Reason="", readiness=false. Elapsed: 1m18.215343492s
[1mSTEP:[0m Saw pod success [38;5;243m01/17/23 21:13:32.444[0m
Jan 17 21:13:32.444: INFO: Pod "azuredisk-volume-tester-7lm55" satisfied condition "Succeeded or Failed"
Jan 17 21:13:32.549: INFO: deleting Pod "azuredisk-273"/"azuredisk-volume-tester-7lm55"
Jan 17 21:13:32.667: INFO: Pod azuredisk-volume-tester-7lm55 has the following logs: 
[1mSTEP:[0m Deleting pod azuredisk-volume-tester-7lm55 in namespace azuredisk-273 [38;5;243m01/17/23 21:13:32.667[0m
[1mSTEP:[0m waiting for disk to detach from node [38;5;243m01/17/23 21:13:32.79[0m
[1mSTEP:[0m Checking Prow test resource group [38;5;243m01/17/23 21:13:58.999[0m
[1mSTEP:[0m Prow test resource group: capz-ri100d [38;5;243m01/17/23 21:13:59.001[0m
... skipping 16 lines ...
[1mSTEP:[0m checking the PVC [38;5;243m01/17/23 21:14:20.871[0m
[1mSTEP:[0m validating provisioned PV [38;5;243m01/17/23 21:14:20.976[0m
[1mSTEP:[0m checking the PV [38;5;243m01/17/23 21:14:21.08[0m
[1mSTEP:[0m setting up the pod [38;5;243m01/17/23 21:14:21.08[0m
I0117 21:14:21.080834   41861 resource_setup.go:62] adding PV (pvc-fbec0ca5-0e6b-42e9-a8ce-cd338aafde99) to pod ()
[1mSTEP:[0m deploying a pod with a volume restored from the snapshot [38;5;243m01/17/23 21:14:21.08[0m
[1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/17/23 21:14:21.193[0m
Jan 17 21:14:21.193: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-89pdm" in namespace "azuredisk-273" to be "Succeeded or Failed"
Jan 17 21:14:21.299: INFO: Pod "azuredisk-volume-tester-89pdm": Phase="Pending", Reason="", readiness=false. Elapsed: 105.472349ms
Jan 17 21:14:23.404: INFO: Pod "azuredisk-volume-tester-89pdm": Phase="Pending", Reason="", readiness=false. Elapsed: 2.211117263s
Jan 17 21:14:25.405: INFO: Pod "azuredisk-volume-tester-89pdm": Phase="Pending", Reason="", readiness=false. Elapsed: 4.211431327s
Jan 17 21:14:27.403: INFO: Pod "azuredisk-volume-tester-89pdm": Phase="Pending", Reason="", readiness=false. Elapsed: 6.209407711s
Jan 17 21:14:29.406: INFO: Pod "azuredisk-volume-tester-89pdm": Phase="Pending", Reason="", readiness=false. Elapsed: 8.212661053s
Jan 17 21:14:31.409: INFO: Pod "azuredisk-volume-tester-89pdm": Phase="Pending", Reason="", readiness=false. Elapsed: 10.216155889s
... skipping 11 lines ...
Jan 17 21:14:55.411: INFO: Pod "azuredisk-volume-tester-89pdm": Phase="Running", Reason="", readiness=true. Elapsed: 34.21732827s
Jan 17 21:14:57.407: INFO: Pod "azuredisk-volume-tester-89pdm": Phase="Running", Reason="", readiness=false. Elapsed: 36.213514579s
Jan 17 21:14:59.411: INFO: Pod "azuredisk-volume-tester-89pdm": Phase="Running", Reason="", readiness=false. Elapsed: 38.218036004s
Jan 17 21:15:01.409: INFO: Pod "azuredisk-volume-tester-89pdm": Phase="Running", Reason="", readiness=false. Elapsed: 40.216141497s
Jan 17 21:15:03.409: INFO: Pod "azuredisk-volume-tester-89pdm": Phase="Succeeded", Reason="", readiness=false. Elapsed: 42.215316153s
[1mSTEP:[0m Saw pod success [38;5;243m01/17/23 21:15:03.409[0m
Jan 17 21:15:03.409: INFO: Pod "azuredisk-volume-tester-89pdm" satisfied condition "Succeeded or Failed"
Jan 17 21:15:03.409: INFO: deleting Pod "azuredisk-273"/"azuredisk-volume-tester-89pdm"
Jan 17 21:15:03.533: INFO: Pod azuredisk-volume-tester-89pdm has the following logs: hello world

[1mSTEP:[0m Deleting pod azuredisk-volume-tester-89pdm in namespace azuredisk-273 [38;5;243m01/17/23 21:15:03.534[0m
Jan 17 21:15:03.662: INFO: deleting PVC "azuredisk-273"/"pvc-npcsf"
Jan 17 21:15:03.662: INFO: Deleting PersistentVolumeClaim "pvc-npcsf"
... skipping 12 lines ...
Jan 17 21:15:44.846: INFO: Claim "pvc-npcsf" in namespace "azuredisk-273" doesn't exist in the system
Jan 17 21:15:44.846: INFO: deleting StorageClass azuredisk-273-disk.csi.azure.com-dynamic-sc-svhfr
[1mSTEP:[0m deleting a VolumeSnapshot volume-snapshot-bw5qm [38;5;243m01/17/23 21:15:44.958[0m
Jan 17 21:15:45.070: INFO: skip deleting VolumeSnapshotClass azuredisk-273-disk.csi.azure.com-dynamic-sc-pzhft
Jan 17 21:15:45.070: INFO: Deleting resource group azuredisk-csi-driver-test-dbd3d270-96ab-11ed-b7bd-36fad8c2e404
Jan 17 21:17:04.240: INFO: deleting Pod "azuredisk-273"/"azuredisk-volume-tester-7lm55"
Jan 17 21:17:04.344: INFO: Error getting logs for pod azuredisk-volume-tester-7lm55: the server could not find the requested resource (get pods azuredisk-volume-tester-7lm55)
[1mSTEP:[0m Deleting pod azuredisk-volume-tester-7lm55 in namespace azuredisk-273 [38;5;243m01/17/23 21:17:04.344[0m
Jan 17 21:17:04.448: INFO: deleting PVC "azuredisk-273"/"pvc-7k529"
Jan 17 21:17:04.448: INFO: Deleting PersistentVolumeClaim "pvc-7k529"
[1mSTEP:[0m waiting for claim's PV "pvc-1ec5be71-3969-45d8-8f62-9ffa7a56990f" to be deleted [38;5;243m01/17/23 21:17:04.558[0m
Jan 17 21:17:04.558: INFO: Waiting up to 10m0s for PersistentVolume pvc-1ec5be71-3969-45d8-8f62-9ffa7a56990f to get deleted
Jan 17 21:17:04.665: INFO: PersistentVolume pvc-1ec5be71-3969-45d8-8f62-9ffa7a56990f found and phase=Released (106.801323ms)
... skipping 29 lines ...
    Jan 17 21:12:11.803: INFO: PersistentVolumeClaim pvc-7k529 found but phase is Pending instead of Bound.
    Jan 17 21:12:13.909: INFO: PersistentVolumeClaim pvc-7k529 found and phase=Bound (4.316376662s)
    [1mSTEP:[0m checking the PVC [38;5;243m01/17/23 21:12:13.909[0m
    [1mSTEP:[0m validating provisioned PV [38;5;243m01/17/23 21:12:14.013[0m
    [1mSTEP:[0m checking the PV [38;5;243m01/17/23 21:12:14.117[0m
    [1mSTEP:[0m deploying the pod [38;5;243m01/17/23 21:12:14.117[0m
    [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/17/23 21:12:14.228[0m
    Jan 17 21:12:14.228: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-7lm55" in namespace "azuredisk-273" to be "Succeeded or Failed"
    Jan 17 21:12:14.332: INFO: Pod "azuredisk-volume-tester-7lm55": Phase="Pending", Reason="", readiness=false. Elapsed: 104.108216ms
    Jan 17 21:12:16.437: INFO: Pod "azuredisk-volume-tester-7lm55": Phase="Pending", Reason="", readiness=false. Elapsed: 2.208695672s
    Jan 17 21:12:18.447: INFO: Pod "azuredisk-volume-tester-7lm55": Phase="Pending", Reason="", readiness=false. Elapsed: 4.218774545s
    Jan 17 21:12:20.443: INFO: Pod "azuredisk-volume-tester-7lm55": Phase="Pending", Reason="", readiness=false. Elapsed: 6.21468064s
    Jan 17 21:12:22.438: INFO: Pod "azuredisk-volume-tester-7lm55": Phase="Pending", Reason="", readiness=false. Elapsed: 8.209790812s
    Jan 17 21:12:24.439: INFO: Pod "azuredisk-volume-tester-7lm55": Phase="Pending", Reason="", readiness=false. Elapsed: 10.210824896s
... skipping 29 lines ...
    Jan 17 21:13:24.443: INFO: Pod "azuredisk-volume-tester-7lm55": Phase="Running", Reason="", readiness=true. Elapsed: 1m10.214543854s
    Jan 17 21:13:26.444: INFO: Pod "azuredisk-volume-tester-7lm55": Phase="Running", Reason="", readiness=false. Elapsed: 1m12.21535794s
    Jan 17 21:13:28.443: INFO: Pod "azuredisk-volume-tester-7lm55": Phase="Running", Reason="", readiness=false. Elapsed: 1m14.215062586s
    Jan 17 21:13:30.442: INFO: Pod "azuredisk-volume-tester-7lm55": Phase="Running", Reason="", readiness=false. Elapsed: 1m16.21342565s
    Jan 17 21:13:32.444: INFO: Pod "azuredisk-volume-tester-7lm55": Phase="Succeeded", Reason="", readiness=false. Elapsed: 1m18.215343492s
    [1mSTEP:[0m Saw pod success [38;5;243m01/17/23 21:13:32.444[0m
    Jan 17 21:13:32.444: INFO: Pod "azuredisk-volume-tester-7lm55" satisfied condition "Succeeded or Failed"
    Jan 17 21:13:32.549: INFO: deleting Pod "azuredisk-273"/"azuredisk-volume-tester-7lm55"
    Jan 17 21:13:32.667: INFO: Pod azuredisk-volume-tester-7lm55 has the following logs: 
    [1mSTEP:[0m Deleting pod azuredisk-volume-tester-7lm55 in namespace azuredisk-273 [38;5;243m01/17/23 21:13:32.667[0m
    [1mSTEP:[0m waiting for disk to detach from node [38;5;243m01/17/23 21:13:32.79[0m
    [1mSTEP:[0m Checking Prow test resource group [38;5;243m01/17/23 21:13:58.999[0m
    [1mSTEP:[0m Prow test resource group: capz-ri100d [38;5;243m01/17/23 21:13:59.001[0m
... skipping 16 lines ...
    [1mSTEP:[0m checking the PVC [38;5;243m01/17/23 21:14:20.871[0m
    [1mSTEP:[0m validating provisioned PV [38;5;243m01/17/23 21:14:20.976[0m
    [1mSTEP:[0m checking the PV [38;5;243m01/17/23 21:14:21.08[0m
    [1mSTEP:[0m setting up the pod [38;5;243m01/17/23 21:14:21.08[0m
    I0117 21:14:21.080834   41861 resource_setup.go:62] adding PV (pvc-fbec0ca5-0e6b-42e9-a8ce-cd338aafde99) to pod ()
    [1mSTEP:[0m deploying a pod with a volume restored from the snapshot [38;5;243m01/17/23 21:14:21.08[0m
    [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/17/23 21:14:21.193[0m
    Jan 17 21:14:21.193: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-89pdm" in namespace "azuredisk-273" to be "Succeeded or Failed"
    Jan 17 21:14:21.299: INFO: Pod "azuredisk-volume-tester-89pdm": Phase="Pending", Reason="", readiness=false. Elapsed: 105.472349ms
    Jan 17 21:14:23.404: INFO: Pod "azuredisk-volume-tester-89pdm": Phase="Pending", Reason="", readiness=false. Elapsed: 2.211117263s
    Jan 17 21:14:25.405: INFO: Pod "azuredisk-volume-tester-89pdm": Phase="Pending", Reason="", readiness=false. Elapsed: 4.211431327s
    Jan 17 21:14:27.403: INFO: Pod "azuredisk-volume-tester-89pdm": Phase="Pending", Reason="", readiness=false. Elapsed: 6.209407711s
    Jan 17 21:14:29.406: INFO: Pod "azuredisk-volume-tester-89pdm": Phase="Pending", Reason="", readiness=false. Elapsed: 8.212661053s
    Jan 17 21:14:31.409: INFO: Pod "azuredisk-volume-tester-89pdm": Phase="Pending", Reason="", readiness=false. Elapsed: 10.216155889s
... skipping 11 lines ...
    Jan 17 21:14:55.411: INFO: Pod "azuredisk-volume-tester-89pdm": Phase="Running", Reason="", readiness=true. Elapsed: 34.21732827s
    Jan 17 21:14:57.407: INFO: Pod "azuredisk-volume-tester-89pdm": Phase="Running", Reason="", readiness=false. Elapsed: 36.213514579s
    Jan 17 21:14:59.411: INFO: Pod "azuredisk-volume-tester-89pdm": Phase="Running", Reason="", readiness=false. Elapsed: 38.218036004s
    Jan 17 21:15:01.409: INFO: Pod "azuredisk-volume-tester-89pdm": Phase="Running", Reason="", readiness=false. Elapsed: 40.216141497s
    Jan 17 21:15:03.409: INFO: Pod "azuredisk-volume-tester-89pdm": Phase="Succeeded", Reason="", readiness=false. Elapsed: 42.215316153s
    [1mSTEP:[0m Saw pod success [38;5;243m01/17/23 21:15:03.409[0m
    Jan 17 21:15:03.409: INFO: Pod "azuredisk-volume-tester-89pdm" satisfied condition "Succeeded or Failed"
    Jan 17 21:15:03.409: INFO: deleting Pod "azuredisk-273"/"azuredisk-volume-tester-89pdm"
    Jan 17 21:15:03.533: INFO: Pod azuredisk-volume-tester-89pdm has the following logs: hello world

    [1mSTEP:[0m Deleting pod azuredisk-volume-tester-89pdm in namespace azuredisk-273 [38;5;243m01/17/23 21:15:03.534[0m
    Jan 17 21:15:03.662: INFO: deleting PVC "azuredisk-273"/"pvc-npcsf"
    Jan 17 21:15:03.662: INFO: Deleting PersistentVolumeClaim "pvc-npcsf"
... skipping 12 lines ...
    Jan 17 21:15:44.846: INFO: Claim "pvc-npcsf" in namespace "azuredisk-273" doesn't exist in the system
    Jan 17 21:15:44.846: INFO: deleting StorageClass azuredisk-273-disk.csi.azure.com-dynamic-sc-svhfr
    [1mSTEP:[0m deleting a VolumeSnapshot volume-snapshot-bw5qm [38;5;243m01/17/23 21:15:44.958[0m
    Jan 17 21:15:45.070: INFO: skip deleting VolumeSnapshotClass azuredisk-273-disk.csi.azure.com-dynamic-sc-pzhft
    Jan 17 21:15:45.070: INFO: Deleting resource group azuredisk-csi-driver-test-dbd3d270-96ab-11ed-b7bd-36fad8c2e404
    Jan 17 21:17:04.240: INFO: deleting Pod "azuredisk-273"/"azuredisk-volume-tester-7lm55"
    Jan 17 21:17:04.344: INFO: Error getting logs for pod azuredisk-volume-tester-7lm55: the server could not find the requested resource (get pods azuredisk-volume-tester-7lm55)
    [1mSTEP:[0m Deleting pod azuredisk-volume-tester-7lm55 in namespace azuredisk-273 [38;5;243m01/17/23 21:17:04.344[0m
    Jan 17 21:17:04.448: INFO: deleting PVC "azuredisk-273"/"pvc-7k529"
    Jan 17 21:17:04.448: INFO: Deleting PersistentVolumeClaim "pvc-7k529"
    [1mSTEP:[0m waiting for claim's PV "pvc-1ec5be71-3969-45d8-8f62-9ffa7a56990f" to be deleted [38;5;243m01/17/23 21:17:04.558[0m
    Jan 17 21:17:04.558: INFO: Waiting up to 10m0s for PersistentVolume pvc-1ec5be71-3969-45d8-8f62-9ffa7a56990f to get deleted
    Jan 17 21:17:04.665: INFO: PersistentVolume pvc-1ec5be71-3969-45d8-8f62-9ffa7a56990f found and phase=Released (106.801323ms)
... skipping 28 lines ...
Jan 17 21:17:19.253: INFO: PersistentVolumeClaim pvc-smbsh found but phase is Pending instead of Bound.
Jan 17 21:17:21.360: INFO: PersistentVolumeClaim pvc-smbsh found and phase=Bound (4.319080873s)
[1mSTEP:[0m checking the PVC [38;5;243m01/17/23 21:17:21.36[0m
[1mSTEP:[0m validating provisioned PV [38;5;243m01/17/23 21:17:21.468[0m
[1mSTEP:[0m checking the PV [38;5;243m01/17/23 21:17:21.573[0m
[1mSTEP:[0m deploying the pod [38;5;243m01/17/23 21:17:21.573[0m
[1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/17/23 21:17:21.69[0m
Jan 17 21:17:21.690: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-6xp72" in namespace "azuredisk-8041" to be "Succeeded or Failed"
Jan 17 21:17:21.800: INFO: Pod "azuredisk-volume-tester-6xp72": Phase="Pending", Reason="", readiness=false. Elapsed: 109.690226ms
Jan 17 21:17:23.905: INFO: Pod "azuredisk-volume-tester-6xp72": Phase="Pending", Reason="", readiness=false. Elapsed: 2.214880709s
Jan 17 21:17:25.912: INFO: Pod "azuredisk-volume-tester-6xp72": Phase="Pending", Reason="", readiness=false. Elapsed: 4.22209354s
Jan 17 21:17:27.907: INFO: Pod "azuredisk-volume-tester-6xp72": Phase="Pending", Reason="", readiness=false. Elapsed: 6.216502826s
Jan 17 21:17:29.908: INFO: Pod "azuredisk-volume-tester-6xp72": Phase="Pending", Reason="", readiness=false. Elapsed: 8.217536635s
Jan 17 21:17:31.907: INFO: Pod "azuredisk-volume-tester-6xp72": Phase="Pending", Reason="", readiness=false. Elapsed: 10.217367047s
... skipping 16 lines ...
Jan 17 21:18:05.910: INFO: Pod "azuredisk-volume-tester-6xp72": Phase="Running", Reason="", readiness=true. Elapsed: 44.219681081s
Jan 17 21:18:07.911: INFO: Pod "azuredisk-volume-tester-6xp72": Phase="Running", Reason="", readiness=true. Elapsed: 46.220577095s
Jan 17 21:18:09.909: INFO: Pod "azuredisk-volume-tester-6xp72": Phase="Running", Reason="", readiness=false. Elapsed: 48.219226389s
Jan 17 21:18:11.910: INFO: Pod "azuredisk-volume-tester-6xp72": Phase="Running", Reason="", readiness=false. Elapsed: 50.220327668s
Jan 17 21:18:13.910: INFO: Pod "azuredisk-volume-tester-6xp72": Phase="Succeeded", Reason="", readiness=false. Elapsed: 52.22021815s
[1mSTEP:[0m Saw pod success [38;5;243m01/17/23 21:18:13.91[0m
Jan 17 21:18:13.911: INFO: Pod "azuredisk-volume-tester-6xp72" satisfied condition "Succeeded or Failed"
Jan 17 21:18:14.015: INFO: deleting Pod "azuredisk-8041"/"azuredisk-volume-tester-6xp72"
Jan 17 21:18:14.140: INFO: Pod azuredisk-volume-tester-6xp72 has the following logs: 
[1mSTEP:[0m Deleting pod azuredisk-volume-tester-6xp72 in namespace azuredisk-8041 [38;5;243m01/17/23 21:18:14.14[0m
[1mSTEP:[0m waiting for disk to detach from node [38;5;243m01/17/23 21:18:14.274[0m
[1mSTEP:[0m Checking Prow test resource group [38;5;243m01/17/23 21:18:32.484[0m
[1mSTEP:[0m Prow test resource group: capz-ri100d [38;5;243m01/17/23 21:18:32.486[0m
... skipping 36 lines ...
[1mSTEP:[0m validating provisioned PV [38;5;243m01/17/23 21:19:21.931[0m
[1mSTEP:[0m checking the PV [38;5;243m01/17/23 21:19:22.035[0m
[1mSTEP:[0m setting up the pod [38;5;243m01/17/23 21:19:22.035[0m
I0117 21:19:22.035821   41861 resource_setup.go:62] adding PV (pvc-fd03070e-1579-4566-b3e1-e83b180aa1a5) to pod ()
[1mSTEP:[0m Set pod anti-affinity to make sure two pods are scheduled on different nodes [38;5;243m01/17/23 21:19:22.035[0m
[1mSTEP:[0m deploying a pod with a volume restored from the snapshot [38;5;243m01/17/23 21:19:22.036[0m
[1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/17/23 21:19:22.144[0m
Jan 17 21:19:22.144: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-2snz8" in namespace "azuredisk-8041" to be "Succeeded or Failed"
Jan 17 21:19:22.253: INFO: Pod "azuredisk-volume-tester-2snz8": Phase="Pending", Reason="", readiness=false. Elapsed: 108.396618ms
Jan 17 21:19:24.363: INFO: Pod "azuredisk-volume-tester-2snz8": Phase="Pending", Reason="", readiness=false. Elapsed: 2.218491126s
Jan 17 21:19:26.363: INFO: Pod "azuredisk-volume-tester-2snz8": Phase="Pending", Reason="", readiness=false. Elapsed: 4.218909239s
Jan 17 21:19:28.363: INFO: Pod "azuredisk-volume-tester-2snz8": Phase="Pending", Reason="", readiness=false. Elapsed: 6.219049196s
Jan 17 21:19:30.363: INFO: Pod "azuredisk-volume-tester-2snz8": Phase="Pending", Reason="", readiness=false. Elapsed: 8.218565799s
Jan 17 21:19:32.363: INFO: Pod "azuredisk-volume-tester-2snz8": Phase="Pending", Reason="", readiness=false. Elapsed: 10.218661178s
... skipping 87 lines ...
Jan 17 21:22:28.363: INFO: Pod "azuredisk-volume-tester-2snz8": Phase="Running", Reason="", readiness=true. Elapsed: 3m6.21820088s
Jan 17 21:22:30.362: INFO: Pod "azuredisk-volume-tester-2snz8": Phase="Running", Reason="", readiness=true. Elapsed: 3m8.217729178s
Jan 17 21:22:32.361: INFO: Pod "azuredisk-volume-tester-2snz8": Phase="Running", Reason="", readiness=false. Elapsed: 3m10.216830273s
Jan 17 21:22:34.363: INFO: Pod "azuredisk-volume-tester-2snz8": Phase="Running", Reason="", readiness=false. Elapsed: 3m12.218677016s
Jan 17 21:22:36.363: INFO: Pod "azuredisk-volume-tester-2snz8": Phase="Succeeded", Reason="", readiness=false. Elapsed: 3m14.219097705s
[1mSTEP:[0m Saw pod success [38;5;243m01/17/23 21:22:36.364[0m
Jan 17 21:22:36.364: INFO: Pod "azuredisk-volume-tester-2snz8" satisfied condition "Succeeded or Failed"
Jan 17 21:22:36.364: INFO: deleting Pod "azuredisk-8041"/"azuredisk-volume-tester-2snz8"
Jan 17 21:22:36.495: INFO: Pod azuredisk-volume-tester-2snz8 has the following logs: hello world

[1mSTEP:[0m Deleting pod azuredisk-volume-tester-2snz8 in namespace azuredisk-8041 [38;5;243m01/17/23 21:22:36.495[0m
Jan 17 21:22:36.640: INFO: deleting PVC "azuredisk-8041"/"pvc-kr8vh"
Jan 17 21:22:36.640: INFO: Deleting PersistentVolumeClaim "pvc-kr8vh"
... skipping 15 lines ...
Jan 17 21:23:18.066: INFO: Pod azuredisk-volume-tester-vgr7w has the following logs: 
[1mSTEP:[0m Deleting pod azuredisk-volume-tester-vgr7w in namespace azuredisk-8041 [38;5;243m01/17/23 21:23:18.066[0m
[1mSTEP:[0m deleting a VolumeSnapshot volume-snapshot-47mgw [38;5;243m01/17/23 21:23:18.188[0m
Jan 17 21:23:18.302: INFO: skip deleting VolumeSnapshotClass azuredisk-8041-disk.csi.azure.com-dynamic-sc-vzxcz
Jan 17 21:23:18.302: INFO: Deleting resource group azuredisk-csi-driver-test-7ed6379b-96ac-11ed-b7bd-36fad8c2e404
Jan 17 21:24:37.267: INFO: deleting Pod "azuredisk-8041"/"azuredisk-volume-tester-6xp72"
Jan 17 21:24:37.370: INFO: Error getting logs for pod azuredisk-volume-tester-6xp72: the server could not find the requested resource (get pods azuredisk-volume-tester-6xp72)
[1mSTEP:[0m Deleting pod azuredisk-volume-tester-6xp72 in namespace azuredisk-8041 [38;5;243m01/17/23 21:24:37.37[0m
Jan 17 21:24:37.474: INFO: deleting PVC "azuredisk-8041"/"pvc-smbsh"
Jan 17 21:24:37.475: INFO: Deleting PersistentVolumeClaim "pvc-smbsh"
[1mSTEP:[0m waiting for claim's PV "pvc-ffa3200d-4515-4a5a-85ab-c76e504d0dba" to be deleted [38;5;243m01/17/23 21:24:37.583[0m
Jan 17 21:24:37.583: INFO: Waiting up to 10m0s for PersistentVolume pvc-ffa3200d-4515-4a5a-85ab-c76e504d0dba to get deleted
Jan 17 21:24:37.686: INFO: PersistentVolume pvc-ffa3200d-4515-4a5a-85ab-c76e504d0dba found and phase=Released (103.339652ms)
... skipping 29 lines ...
    Jan 17 21:17:19.253: INFO: PersistentVolumeClaim pvc-smbsh found but phase is Pending instead of Bound.
    Jan 17 21:17:21.360: INFO: PersistentVolumeClaim pvc-smbsh found and phase=Bound (4.319080873s)
    [1mSTEP:[0m checking the PVC [38;5;243m01/17/23 21:17:21.36[0m
    [1mSTEP:[0m validating provisioned PV [38;5;243m01/17/23 21:17:21.468[0m
    [1mSTEP:[0m checking the PV [38;5;243m01/17/23 21:17:21.573[0m
    [1mSTEP:[0m deploying the pod [38;5;243m01/17/23 21:17:21.573[0m
    [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/17/23 21:17:21.69[0m
    Jan 17 21:17:21.690: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-6xp72" in namespace "azuredisk-8041" to be "Succeeded or Failed"
    Jan 17 21:17:21.800: INFO: Pod "azuredisk-volume-tester-6xp72": Phase="Pending", Reason="", readiness=false. Elapsed: 109.690226ms
    Jan 17 21:17:23.905: INFO: Pod "azuredisk-volume-tester-6xp72": Phase="Pending", Reason="", readiness=false. Elapsed: 2.214880709s
    Jan 17 21:17:25.912: INFO: Pod "azuredisk-volume-tester-6xp72": Phase="Pending", Reason="", readiness=false. Elapsed: 4.22209354s
    Jan 17 21:17:27.907: INFO: Pod "azuredisk-volume-tester-6xp72": Phase="Pending", Reason="", readiness=false. Elapsed: 6.216502826s
    Jan 17 21:17:29.908: INFO: Pod "azuredisk-volume-tester-6xp72": Phase="Pending", Reason="", readiness=false. Elapsed: 8.217536635s
    Jan 17 21:17:31.907: INFO: Pod "azuredisk-volume-tester-6xp72": Phase="Pending", Reason="", readiness=false. Elapsed: 10.217367047s
... skipping 16 lines ...
    Jan 17 21:18:05.910: INFO: Pod "azuredisk-volume-tester-6xp72": Phase="Running", Reason="", readiness=true. Elapsed: 44.219681081s
    Jan 17 21:18:07.911: INFO: Pod "azuredisk-volume-tester-6xp72": Phase="Running", Reason="", readiness=true. Elapsed: 46.220577095s
    Jan 17 21:18:09.909: INFO: Pod "azuredisk-volume-tester-6xp72": Phase="Running", Reason="", readiness=false. Elapsed: 48.219226389s
    Jan 17 21:18:11.910: INFO: Pod "azuredisk-volume-tester-6xp72": Phase="Running", Reason="", readiness=false. Elapsed: 50.220327668s
    Jan 17 21:18:13.910: INFO: Pod "azuredisk-volume-tester-6xp72": Phase="Succeeded", Reason="", readiness=false. Elapsed: 52.22021815s
    [1mSTEP:[0m Saw pod success [38;5;243m01/17/23 21:18:13.91[0m
    Jan 17 21:18:13.911: INFO: Pod "azuredisk-volume-tester-6xp72" satisfied condition "Succeeded or Failed"
    Jan 17 21:18:14.015: INFO: deleting Pod "azuredisk-8041"/"azuredisk-volume-tester-6xp72"
    Jan 17 21:18:14.140: INFO: Pod azuredisk-volume-tester-6xp72 has the following logs: 
    [1mSTEP:[0m Deleting pod azuredisk-volume-tester-6xp72 in namespace azuredisk-8041 [38;5;243m01/17/23 21:18:14.14[0m
    [1mSTEP:[0m waiting for disk to detach from node [38;5;243m01/17/23 21:18:14.274[0m
    [1mSTEP:[0m Checking Prow test resource group [38;5;243m01/17/23 21:18:32.484[0m
    [1mSTEP:[0m Prow test resource group: capz-ri100d [38;5;243m01/17/23 21:18:32.486[0m
... skipping 36 lines ...
    [1mSTEP:[0m validating provisioned PV [38;5;243m01/17/23 21:19:21.931[0m
    [1mSTEP:[0m checking the PV [38;5;243m01/17/23 21:19:22.035[0m
    [1mSTEP:[0m setting up the pod [38;5;243m01/17/23 21:19:22.035[0m
    I0117 21:19:22.035821   41861 resource_setup.go:62] adding PV (pvc-fd03070e-1579-4566-b3e1-e83b180aa1a5) to pod ()
    [1mSTEP:[0m Set pod anti-affinity to make sure two pods are scheduled on different nodes [38;5;243m01/17/23 21:19:22.035[0m
    [1mSTEP:[0m deploying a pod with a volume restored from the snapshot [38;5;243m01/17/23 21:19:22.036[0m
    [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/17/23 21:19:22.144[0m
    Jan 17 21:19:22.144: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-2snz8" in namespace "azuredisk-8041" to be "Succeeded or Failed"
    Jan 17 21:19:22.253: INFO: Pod "azuredisk-volume-tester-2snz8": Phase="Pending", Reason="", readiness=false. Elapsed: 108.396618ms
    Jan 17 21:19:24.363: INFO: Pod "azuredisk-volume-tester-2snz8": Phase="Pending", Reason="", readiness=false. Elapsed: 2.218491126s
    Jan 17 21:19:26.363: INFO: Pod "azuredisk-volume-tester-2snz8": Phase="Pending", Reason="", readiness=false. Elapsed: 4.218909239s
    Jan 17 21:19:28.363: INFO: Pod "azuredisk-volume-tester-2snz8": Phase="Pending", Reason="", readiness=false. Elapsed: 6.219049196s
    Jan 17 21:19:30.363: INFO: Pod "azuredisk-volume-tester-2snz8": Phase="Pending", Reason="", readiness=false. Elapsed: 8.218565799s
    Jan 17 21:19:32.363: INFO: Pod "azuredisk-volume-tester-2snz8": Phase="Pending", Reason="", readiness=false. Elapsed: 10.218661178s
... skipping 87 lines ...
    Jan 17 21:22:28.363: INFO: Pod "azuredisk-volume-tester-2snz8": Phase="Running", Reason="", readiness=true. Elapsed: 3m6.21820088s
    Jan 17 21:22:30.362: INFO: Pod "azuredisk-volume-tester-2snz8": Phase="Running", Reason="", readiness=true. Elapsed: 3m8.217729178s
    Jan 17 21:22:32.361: INFO: Pod "azuredisk-volume-tester-2snz8": Phase="Running", Reason="", readiness=false. Elapsed: 3m10.216830273s
    Jan 17 21:22:34.363: INFO: Pod "azuredisk-volume-tester-2snz8": Phase="Running", Reason="", readiness=false. Elapsed: 3m12.218677016s
    Jan 17 21:22:36.363: INFO: Pod "azuredisk-volume-tester-2snz8": Phase="Succeeded", Reason="", readiness=false. Elapsed: 3m14.219097705s
    [1mSTEP:[0m Saw pod success [38;5;243m01/17/23 21:22:36.364[0m
    Jan 17 21:22:36.364: INFO: Pod "azuredisk-volume-tester-2snz8" satisfied condition "Succeeded or Failed"
    Jan 17 21:22:36.364: INFO: deleting Pod "azuredisk-8041"/"azuredisk-volume-tester-2snz8"
    Jan 17 21:22:36.495: INFO: Pod azuredisk-volume-tester-2snz8 has the following logs: hello world

    [1mSTEP:[0m Deleting pod azuredisk-volume-tester-2snz8 in namespace azuredisk-8041 [38;5;243m01/17/23 21:22:36.495[0m
    Jan 17 21:22:36.640: INFO: deleting PVC "azuredisk-8041"/"pvc-kr8vh"
    Jan 17 21:22:36.640: INFO: Deleting PersistentVolumeClaim "pvc-kr8vh"
... skipping 15 lines ...
    Jan 17 21:23:18.066: INFO: Pod azuredisk-volume-tester-vgr7w has the following logs: 
    [1mSTEP:[0m Deleting pod azuredisk-volume-tester-vgr7w in namespace azuredisk-8041 [38;5;243m01/17/23 21:23:18.066[0m
    [1mSTEP:[0m deleting a VolumeSnapshot volume-snapshot-47mgw [38;5;243m01/17/23 21:23:18.188[0m
    Jan 17 21:23:18.302: INFO: skip deleting VolumeSnapshotClass azuredisk-8041-disk.csi.azure.com-dynamic-sc-vzxcz
    Jan 17 21:23:18.302: INFO: Deleting resource group azuredisk-csi-driver-test-7ed6379b-96ac-11ed-b7bd-36fad8c2e404
    Jan 17 21:24:37.267: INFO: deleting Pod "azuredisk-8041"/"azuredisk-volume-tester-6xp72"
    Jan 17 21:24:37.370: INFO: Error getting logs for pod azuredisk-volume-tester-6xp72: the server could not find the requested resource (get pods azuredisk-volume-tester-6xp72)
    [1mSTEP:[0m Deleting pod azuredisk-volume-tester-6xp72 in namespace azuredisk-8041 [38;5;243m01/17/23 21:24:37.37[0m
    Jan 17 21:24:37.474: INFO: deleting PVC "azuredisk-8041"/"pvc-smbsh"
    Jan 17 21:24:37.475: INFO: Deleting PersistentVolumeClaim "pvc-smbsh"
    [1mSTEP:[0m waiting for claim's PV "pvc-ffa3200d-4515-4a5a-85ab-c76e504d0dba" to be deleted [38;5;243m01/17/23 21:24:37.583[0m
    Jan 17 21:24:37.583: INFO: Waiting up to 10m0s for PersistentVolume pvc-ffa3200d-4515-4a5a-85ab-c76e504d0dba to get deleted
    Jan 17 21:24:37.686: INFO: PersistentVolume pvc-ffa3200d-4515-4a5a-85ab-c76e504d0dba found and phase=Released (103.339652ms)
... skipping 55 lines ...
Jan 17 21:25:03.667: INFO: PersistentVolumeClaim pvc-lvpc5 found and phase=Bound (4.311332195s)
[1mSTEP:[0m checking the PVC [38;5;243m01/17/23 21:25:03.667[0m
[1mSTEP:[0m validating provisioned PV [38;5;243m01/17/23 21:25:03.77[0m
[1mSTEP:[0m checking the PV [38;5;243m01/17/23 21:25:03.872[0m
I0117 21:25:03.872774   41861 resource_setup.go:89] adding PV (pvc-3e0d4726-4b6e-4dd2-a295-d4087824d228) to pod ()
[1mSTEP:[0m deploying the pod [38;5;243m01/17/23 21:25:03.872[0m
[1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/17/23 21:25:03.983[0m
Jan 17 21:25:03.983: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-gbcxj" in namespace "azuredisk-7624" to be "Succeeded or Failed"
Jan 17 21:25:04.096: INFO: Pod "azuredisk-volume-tester-gbcxj": Phase="Pending", Reason="", readiness=false. Elapsed: 112.630842ms
Jan 17 21:25:06.205: INFO: Pod "azuredisk-volume-tester-gbcxj": Phase="Pending", Reason="", readiness=false. Elapsed: 2.222007177s
Jan 17 21:25:08.205: INFO: Pod "azuredisk-volume-tester-gbcxj": Phase="Pending", Reason="", readiness=false. Elapsed: 4.221912939s
Jan 17 21:25:10.206: INFO: Pod "azuredisk-volume-tester-gbcxj": Phase="Pending", Reason="", readiness=false. Elapsed: 6.222450437s
Jan 17 21:25:12.206: INFO: Pod "azuredisk-volume-tester-gbcxj": Phase="Pending", Reason="", readiness=false. Elapsed: 8.222216088s
Jan 17 21:25:14.205: INFO: Pod "azuredisk-volume-tester-gbcxj": Phase="Pending", Reason="", readiness=false. Elapsed: 10.221438908s
... skipping 20 lines ...
Jan 17 21:25:56.206: INFO: Pod "azuredisk-volume-tester-gbcxj": Phase="Running", Reason="", readiness=true. Elapsed: 52.222698248s
Jan 17 21:25:58.206: INFO: Pod "azuredisk-volume-tester-gbcxj": Phase="Running", Reason="", readiness=true. Elapsed: 54.222821871s
Jan 17 21:26:00.204: INFO: Pod "azuredisk-volume-tester-gbcxj": Phase="Running", Reason="", readiness=false. Elapsed: 56.220692074s
Jan 17 21:26:02.204: INFO: Pod "azuredisk-volume-tester-gbcxj": Phase="Running", Reason="", readiness=false. Elapsed: 58.22087835s
Jan 17 21:26:04.205: INFO: Pod "azuredisk-volume-tester-gbcxj": Phase="Succeeded", Reason="", readiness=false. Elapsed: 1m0.221435825s
[1mSTEP:[0m Saw pod success [38;5;243m01/17/23 21:26:04.205[0m
Jan 17 21:26:04.205: INFO: Pod "azuredisk-volume-tester-gbcxj" satisfied condition "Succeeded or Failed"
Jan 17 21:26:04.205: INFO: deleting Pod "azuredisk-7624"/"azuredisk-volume-tester-gbcxj"
Jan 17 21:26:04.335: INFO: Pod azuredisk-volume-tester-gbcxj has the following logs: hello world

[1mSTEP:[0m Deleting pod azuredisk-volume-tester-gbcxj in namespace azuredisk-7624 [38;5;243m01/17/23 21:26:04.335[0m
Jan 17 21:26:04.458: INFO: deleting PVC "azuredisk-7624"/"pvc-lvpc5"
Jan 17 21:26:04.458: INFO: Deleting PersistentVolumeClaim "pvc-lvpc5"
... skipping 85 lines ...
    Jan 17 21:25:03.667: INFO: PersistentVolumeClaim pvc-lvpc5 found and phase=Bound (4.311332195s)
    [1mSTEP:[0m checking the PVC [38;5;243m01/17/23 21:25:03.667[0m
    [1mSTEP:[0m validating provisioned PV [38;5;243m01/17/23 21:25:03.77[0m
    [1mSTEP:[0m checking the PV [38;5;243m01/17/23 21:25:03.872[0m
    I0117 21:25:03.872774   41861 resource_setup.go:89] adding PV (pvc-3e0d4726-4b6e-4dd2-a295-d4087824d228) to pod ()
    [1mSTEP:[0m deploying the pod [38;5;243m01/17/23 21:25:03.872[0m
    [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/17/23 21:25:03.983[0m
    Jan 17 21:25:03.983: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-gbcxj" in namespace "azuredisk-7624" to be "Succeeded or Failed"
    Jan 17 21:25:04.096: INFO: Pod "azuredisk-volume-tester-gbcxj": Phase="Pending", Reason="", readiness=false. Elapsed: 112.630842ms
    Jan 17 21:25:06.205: INFO: Pod "azuredisk-volume-tester-gbcxj": Phase="Pending", Reason="", readiness=false. Elapsed: 2.222007177s
    Jan 17 21:25:08.205: INFO: Pod "azuredisk-volume-tester-gbcxj": Phase="Pending", Reason="", readiness=false. Elapsed: 4.221912939s
    Jan 17 21:25:10.206: INFO: Pod "azuredisk-volume-tester-gbcxj": Phase="Pending", Reason="", readiness=false. Elapsed: 6.222450437s
    Jan 17 21:25:12.206: INFO: Pod "azuredisk-volume-tester-gbcxj": Phase="Pending", Reason="", readiness=false. Elapsed: 8.222216088s
    Jan 17 21:25:14.205: INFO: Pod "azuredisk-volume-tester-gbcxj": Phase="Pending", Reason="", readiness=false. Elapsed: 10.221438908s
... skipping 20 lines ...
    Jan 17 21:25:56.206: INFO: Pod "azuredisk-volume-tester-gbcxj": Phase="Running", Reason="", readiness=true. Elapsed: 52.222698248s
    Jan 17 21:25:58.206: INFO: Pod "azuredisk-volume-tester-gbcxj": Phase="Running", Reason="", readiness=true. Elapsed: 54.222821871s
    Jan 17 21:26:00.204: INFO: Pod "azuredisk-volume-tester-gbcxj": Phase="Running", Reason="", readiness=false. Elapsed: 56.220692074s
    Jan 17 21:26:02.204: INFO: Pod "azuredisk-volume-tester-gbcxj": Phase="Running", Reason="", readiness=false. Elapsed: 58.22087835s
    Jan 17 21:26:04.205: INFO: Pod "azuredisk-volume-tester-gbcxj": Phase="Succeeded", Reason="", readiness=false. Elapsed: 1m0.221435825s
    [1mSTEP:[0m Saw pod success [38;5;243m01/17/23 21:26:04.205[0m
    Jan 17 21:26:04.205: INFO: Pod "azuredisk-volume-tester-gbcxj" satisfied condition "Succeeded or Failed"
    Jan 17 21:26:04.205: INFO: deleting Pod "azuredisk-7624"/"azuredisk-volume-tester-gbcxj"
    Jan 17 21:26:04.335: INFO: Pod azuredisk-volume-tester-gbcxj has the following logs: hello world

    [1mSTEP:[0m Deleting pod azuredisk-volume-tester-gbcxj in namespace azuredisk-7624 [38;5;243m01/17/23 21:26:04.335[0m
    Jan 17 21:26:04.458: INFO: deleting PVC "azuredisk-7624"/"pvc-lvpc5"
    Jan 17 21:26:04.458: INFO: Deleting PersistentVolumeClaim "pvc-lvpc5"
... skipping 839 lines ...
[1mSTEP:[0m begin to delete the pod [38;5;243m01/17/23 21:48:01.65[0m
Jan 17 21:48:01.650: INFO: deleting Pod "azuredisk-5210"/"azuredisk-volume-tester-7qctp"
Jan 17 21:48:01.782: INFO: Pod azuredisk-volume-tester-7qctp has the following logs: hello world

[1mSTEP:[0m Deleting pod azuredisk-volume-tester-7qctp in namespace azuredisk-5210 [38;5;243m01/17/23 21:48:01.782[0m
Jan 17 21:48:28.116: INFO: deleting Pod "azuredisk-5210"/"azuredisk-volume-tester-7qctp"
Jan 17 21:48:28.218: INFO: Error getting logs for pod azuredisk-volume-tester-7qctp: the server could not find the requested resource (get pods azuredisk-volume-tester-7qctp)
[1mSTEP:[0m Deleting pod azuredisk-volume-tester-7qctp in namespace azuredisk-5210 [38;5;243m01/17/23 21:48:28.218[0m
Jan 17 21:48:28.321: INFO: deleting PVC "azuredisk-5210"/"pvc-wc4dh"
Jan 17 21:48:28.321: INFO: Deleting PersistentVolumeClaim "pvc-wc4dh"
[1mSTEP:[0m waiting for claim's PV "pvc-aa5754cf-8d1d-419f-be98-e054e54aa91a" to be deleted [38;5;243m01/17/23 21:48:28.429[0m
Jan 17 21:48:28.429: INFO: Waiting up to 10m0s for PersistentVolume pvc-aa5754cf-8d1d-419f-be98-e054e54aa91a to get deleted
Jan 17 21:48:28.536: INFO: PersistentVolume pvc-aa5754cf-8d1d-419f-be98-e054e54aa91a found and phase=Released (107.566267ms)
... skipping 64 lines ...
    [1mSTEP:[0m begin to delete the pod [38;5;243m01/17/23 21:48:01.65[0m
    Jan 17 21:48:01.650: INFO: deleting Pod "azuredisk-5210"/"azuredisk-volume-tester-7qctp"
    Jan 17 21:48:01.782: INFO: Pod azuredisk-volume-tester-7qctp has the following logs: hello world

    [1mSTEP:[0m Deleting pod azuredisk-volume-tester-7qctp in namespace azuredisk-5210 [38;5;243m01/17/23 21:48:01.782[0m
    Jan 17 21:48:28.116: INFO: deleting Pod "azuredisk-5210"/"azuredisk-volume-tester-7qctp"
    Jan 17 21:48:28.218: INFO: Error getting logs for pod azuredisk-volume-tester-7qctp: the server could not find the requested resource (get pods azuredisk-volume-tester-7qctp)
    [1mSTEP:[0m Deleting pod azuredisk-volume-tester-7qctp in namespace azuredisk-5210 [38;5;243m01/17/23 21:48:28.218[0m
    Jan 17 21:48:28.321: INFO: deleting PVC "azuredisk-5210"/"pvc-wc4dh"
    Jan 17 21:48:28.321: INFO: Deleting PersistentVolumeClaim "pvc-wc4dh"
    [1mSTEP:[0m waiting for claim's PV "pvc-aa5754cf-8d1d-419f-be98-e054e54aa91a" to be deleted [38;5;243m01/17/23 21:48:28.429[0m
    Jan 17 21:48:28.429: INFO: Waiting up to 10m0s for PersistentVolume pvc-aa5754cf-8d1d-419f-be98-e054e54aa91a to get deleted
    Jan 17 21:48:28.536: INFO: PersistentVolume pvc-aa5754cf-8d1d-419f-be98-e054e54aa91a found and phase=Released (107.566267ms)
... skipping 66 lines ...
[1mSTEP:[0m begin to delete the pod [38;5;243m01/17/23 21:49:32.968[0m
Jan 17 21:49:32.968: INFO: deleting Pod "azuredisk-4406"/"azuredisk-volume-tester-hgdn6"
Jan 17 21:49:33.092: INFO: Pod azuredisk-volume-tester-hgdn6 has the following logs: hello world

[1mSTEP:[0m Deleting pod azuredisk-volume-tester-hgdn6 in namespace azuredisk-4406 [38;5;243m01/17/23 21:49:33.092[0m
Jan 17 21:49:47.447: INFO: deleting Pod "azuredisk-4406"/"azuredisk-volume-tester-hgdn6"
Jan 17 21:49:47.550: INFO: Error getting logs for pod azuredisk-volume-tester-hgdn6: the server could not find the requested resource (get pods azuredisk-volume-tester-hgdn6)
[1mSTEP:[0m Deleting pod azuredisk-volume-tester-hgdn6 in namespace azuredisk-4406 [38;5;243m01/17/23 21:49:47.55[0m
Jan 17 21:49:47.653: INFO: deleting PVC "azuredisk-4406"/"pvc-7rg2w"
Jan 17 21:49:47.654: INFO: Deleting PersistentVolumeClaim "pvc-7rg2w"
[1mSTEP:[0m waiting for claim's PV "pvc-b3757d81-34f0-4e4a-8287-71da13330385" to be deleted [38;5;243m01/17/23 21:49:47.76[0m
Jan 17 21:49:47.760: INFO: Waiting up to 10m0s for PersistentVolume pvc-b3757d81-34f0-4e4a-8287-71da13330385 to get deleted
Jan 17 21:49:47.863: INFO: PersistentVolume pvc-b3757d81-34f0-4e4a-8287-71da13330385 found and phase=Released (102.562327ms)
... skipping 70 lines ...
    [1mSTEP:[0m begin to delete the pod [38;5;243m01/17/23 21:49:32.968[0m
    Jan 17 21:49:32.968: INFO: deleting Pod "azuredisk-4406"/"azuredisk-volume-tester-hgdn6"
    Jan 17 21:49:33.092: INFO: Pod azuredisk-volume-tester-hgdn6 has the following logs: hello world

    [1mSTEP:[0m Deleting pod azuredisk-volume-tester-hgdn6 in namespace azuredisk-4406 [38;5;243m01/17/23 21:49:33.092[0m
    Jan 17 21:49:47.447: INFO: deleting Pod "azuredisk-4406"/"azuredisk-volume-tester-hgdn6"
    Jan 17 21:49:47.550: INFO: Error getting logs for pod azuredisk-volume-tester-hgdn6: the server could not find the requested resource (get pods azuredisk-volume-tester-hgdn6)
    [1mSTEP:[0m Deleting pod azuredisk-volume-tester-hgdn6 in namespace azuredisk-4406 [38;5;243m01/17/23 21:49:47.55[0m
    Jan 17 21:49:47.653: INFO: deleting PVC "azuredisk-4406"/"pvc-7rg2w"
    Jan 17 21:49:47.654: INFO: Deleting PersistentVolumeClaim "pvc-7rg2w"
    [1mSTEP:[0m waiting for claim's PV "pvc-b3757d81-34f0-4e4a-8287-71da13330385" to be deleted [38;5;243m01/17/23 21:49:47.76[0m
    Jan 17 21:49:47.760: INFO: Waiting up to 10m0s for PersistentVolume pvc-b3757d81-34f0-4e4a-8287-71da13330385 to get deleted
    Jan 17 21:49:47.863: INFO: PersistentVolume pvc-b3757d81-34f0-4e4a-8287-71da13330385 found and phase=Released (102.562327ms)
... skipping 49 lines ...
I0117 21:50:55.778004   41861 statefulset.go:175] 1/1 replicas in the StatefulSet are ready
[1mSTEP:[0m checking that the pod is running [38;5;243m01/17/23 21:50:55.987[0m
I0117 21:50:56.090613   41861 statefulset.go:175] 1/1 replicas in the StatefulSet are ready
[1mSTEP:[0m check pod exec [38;5;243m01/17/23 21:50:56.09[0m
Jan 17 21:50:56.090: INFO: Running '/usr/local/bin/kubectl --kubeconfig=/home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure/kubeconfig --namespace=azuredisk-5772 exec azuredisk-volume-tester-cdj7l-0 -- cmd /c type C:\mnt\test-1\data.txt'
Jan 17 21:50:57.451: INFO: rc: 1
Jan 17 21:50:57.452: INFO: Error waiting for output "hello world\r\n" in pod "azuredisk-volume-tester-cdj7l-0": error running /usr/local/bin/kubectl --kubeconfig=/home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure/kubeconfig --namespace=azuredisk-5772 exec azuredisk-volume-tester-cdj7l-0 -- cmd /c type C:\mnt\test-1\data.txt:
Command stdout:

stderr:
The system cannot find the file specified.
command terminated with exit code 1

error:
exit status 1.
Jan 17 21:50:59.453: INFO: Running '/usr/local/bin/kubectl --kubeconfig=/home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure/kubeconfig --namespace=azuredisk-5772 exec azuredisk-volume-tester-cdj7l-0 -- cmd /c type C:\mnt\test-1\data.txt'
Jan 17 21:51:00.580: INFO: stderr: ""
Jan 17 21:51:00.580: INFO: stdout: "hello world\r\n"
[1mSTEP:[0m deleting the pod for statefulset [38;5;243m01/17/23 21:51:00.58[0m
Jan 17 21:51:00.580: INFO: Deleting pod "azuredisk-volume-tester-cdj7l-0" in namespace "azuredisk-5772"
... skipping 61 lines ...
    I0117 21:50:55.778004   41861 statefulset.go:175] 1/1 replicas in the StatefulSet are ready
    [1mSTEP:[0m checking that the pod is running [38;5;243m01/17/23 21:50:55.987[0m
    I0117 21:50:56.090613   41861 statefulset.go:175] 1/1 replicas in the StatefulSet are ready
    [1mSTEP:[0m check pod exec [38;5;243m01/17/23 21:50:56.09[0m
    Jan 17 21:50:56.090: INFO: Running '/usr/local/bin/kubectl --kubeconfig=/home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure/kubeconfig --namespace=azuredisk-5772 exec azuredisk-volume-tester-cdj7l-0 -- cmd /c type C:\mnt\test-1\data.txt'
    Jan 17 21:50:57.451: INFO: rc: 1
    Jan 17 21:50:57.452: INFO: Error waiting for output "hello world\r\n" in pod "azuredisk-volume-tester-cdj7l-0": error running /usr/local/bin/kubectl --kubeconfig=/home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure/kubeconfig --namespace=azuredisk-5772 exec azuredisk-volume-tester-cdj7l-0 -- cmd /c type C:\mnt\test-1\data.txt:
    Command stdout:

    stderr:
    The system cannot find the file specified.
    command terminated with exit code 1

    error:
    exit status 1.
    Jan 17 21:50:59.453: INFO: Running '/usr/local/bin/kubectl --kubeconfig=/home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure/kubeconfig --namespace=azuredisk-5772 exec azuredisk-volume-tester-cdj7l-0 -- cmd /c type C:\mnt\test-1\data.txt'
    Jan 17 21:51:00.580: INFO: stderr: ""
    Jan 17 21:51:00.580: INFO: stdout: "hello world\r\n"
    [1mSTEP:[0m deleting the pod for statefulset [38;5;243m01/17/23 21:51:00.58[0m
    Jan 17 21:51:00.580: INFO: Deleting pod "azuredisk-volume-tester-cdj7l-0" in namespace "azuredisk-5772"
... skipping 761 lines ...
I0117 22:07:17.127855   41861 dynamically_provisioned_pod_affinity.go:145] found 1 AzVolumeAttachments for volume (pvc-07350749-804b-4c0f-981b-bcccc8633254)
I0117 22:07:19.125719   41861 dynamically_provisioned_pod_affinity.go:145] found 1 AzVolumeAttachments for volume (pvc-07350749-804b-4c0f-981b-bcccc8633254)
I0117 22:07:21.126817   41861 dynamically_provisioned_pod_affinity.go:145] found 1 AzVolumeAttachments for volume (pvc-07350749-804b-4c0f-981b-bcccc8633254)
I0117 22:07:23.125569   41861 dynamically_provisioned_pod_affinity.go:145] found 1 AzVolumeAttachments for volume (pvc-07350749-804b-4c0f-981b-bcccc8633254)
I0117 22:07:25.127419   41861 dynamically_provisioned_pod_affinity.go:145] found 1 AzVolumeAttachments for volume (pvc-07350749-804b-4c0f-981b-bcccc8633254)
I0117 22:07:25.236942   41861 dynamically_provisioned_pod_affinity.go:145] found 1 AzVolumeAttachments for volume (pvc-07350749-804b-4c0f-981b-bcccc8633254)
Jan 17 22:07:25.237: INFO: Unexpected error: 
    <*errors.errorString | 0xc000486fc0>: {
        s: "timed out waiting for the condition",
    }
Jan 17 22:07:25.237: FAIL: timed out waiting for the condition

Full Stack Trace
sigs.k8s.io/azuredisk-csi-driver/test/e2e/testsuites.(*PodAffinity).Run(0xc000ad0000, {0x26fb380, 0xc001040000}, 0xb?, {0x220ec17, 0x20})
	/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/testsuites/dynamically_provisioned_pod_affinity.go:168 +0x1e9
sigs.k8s.io/azuredisk-csi-driver/test/e2e.(*dynamicProvisioningTestSuite).defineTests.func31()
	/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning_test.go:1347 +0xcca
... skipping 37 lines ...
Jan 17 22:08:48.054: INFO: Waiting up to 5m0s for PersistentVolumeClaim pvc-s66lg to be removed
Jan 17 22:08:48.158: INFO: Claim "pvc-s66lg" in namespace "azuredisk-3283" doesn't exist in the system
Jan 17 22:08:48.158: INFO: deleting StorageClass azuredisk-3283-disk.csi.azure.com-dynamic-sc-hmvlq
[1mSTEP:[0m dump namespace information after failure [38;5;243m01/17/23 22:08:48.268[0m
[1mSTEP:[0m Destroying namespace "azuredisk-3283" for this suite. [38;5;243m01/17/23 22:08:48.269[0m
[38;5;243m------------------------------[0m
[38;5;9m• [FAILED] [773.740 seconds][0m
Dynamic Provisioning
[38;5;243m/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning_test.go:41[0m
  [single-az]
  [38;5;243m/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning_test.go:45[0m
    [38;5;9m[1m[It] Should create replicas on node with matching pod affinity[0m
    [38;5;243m/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning_test.go:1275[0m
... skipping 384 lines ...
    I0117 22:07:17.127855   41861 dynamically_provisioned_pod_affinity.go:145] found 1 AzVolumeAttachments for volume (pvc-07350749-804b-4c0f-981b-bcccc8633254)
    I0117 22:07:19.125719   41861 dynamically_provisioned_pod_affinity.go:145] found 1 AzVolumeAttachments for volume (pvc-07350749-804b-4c0f-981b-bcccc8633254)
    I0117 22:07:21.126817   41861 dynamically_provisioned_pod_affinity.go:145] found 1 AzVolumeAttachments for volume (pvc-07350749-804b-4c0f-981b-bcccc8633254)
    I0117 22:07:23.125569   41861 dynamically_provisioned_pod_affinity.go:145] found 1 AzVolumeAttachments for volume (pvc-07350749-804b-4c0f-981b-bcccc8633254)
    I0117 22:07:25.127419   41861 dynamically_provisioned_pod_affinity.go:145] found 1 AzVolumeAttachments for volume (pvc-07350749-804b-4c0f-981b-bcccc8633254)
    I0117 22:07:25.236942   41861 dynamically_provisioned_pod_affinity.go:145] found 1 AzVolumeAttachments for volume (pvc-07350749-804b-4c0f-981b-bcccc8633254)
    Jan 17 22:07:25.237: INFO: Unexpected error: 
        <*errors.errorString | 0xc000486fc0>: {
            s: "timed out waiting for the condition",
        }
    Jan 17 22:07:25.237: FAIL: timed out waiting for the condition

    Full Stack Trace
    sigs.k8s.io/azuredisk-csi-driver/test/e2e/testsuites.(*PodAffinity).Run(0xc000ad0000, {0x26fb380, 0xc001040000}, 0xb?, {0x220ec17, 0x20})
    	/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/testsuites/dynamically_provisioned_pod_affinity.go:168 +0x1e9
    sigs.k8s.io/azuredisk-csi-driver/test/e2e.(*dynamicProvisioningTestSuite).defineTests.func31()
    	/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning_test.go:1347 +0xcca
... skipping 46 lines ...

  [1mThere were additional failures detected after the initial failure:[0m
    [38;5;13m[PANICKED][0m
    [38;5;13mTest Panicked[0m
    [38;5;13mIn [1m[DeferCleanup (Each)][0m[38;5;13m at: [1m/usr/local/go/src/runtime/panic.go:260[0m

    [38;5;13mruntime error: invalid memory address or nil pointer dereference[0m

    [38;5;13mFull Stack Trace[0m
      k8s.io/kubernetes/test/e2e/framework.(*Framework).dumpNamespaceInfo.func1()
      	/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:274 +0x5c
      k8s.io/kubernetes/test/e2e/framework.(*Framework).dumpNamespaceInfo(0xc0005c40f0)
      	/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:271 +0x179
... skipping 367 lines ...
Jan 17 22:12:25.793: INFO: deployment status: v1.DeploymentStatus{ObservedGeneration:1, Replicas:1, UpdatedReplicas:1, ReadyReplicas:0, AvailableReplicas:0, UnavailableReplicas:1, Conditions:[]v1.DeploymentCondition{v1.DeploymentCondition{Type:"Available", Status:"False", LastUpdateTime:time.Date(2023, time.January, 17, 22, 11, 41, 0, time.Local), LastTransitionTime:time.Date(2023, time.January, 17, 22, 11, 41, 0, time.Local), Reason:"MinimumReplicasUnavailable", Message:"Deployment does not have minimum availability."}, v1.DeploymentCondition{Type:"Progressing", Status:"True", LastUpdateTime:time.Date(2023, time.January, 17, 22, 11, 41, 0, time.Local), LastTransitionTime:time.Date(2023, time.January, 17, 22, 11, 41, 0, time.Local), Reason:"ReplicaSetUpdated", Message:"ReplicaSet \"azuredisk-volume-tester-b9b8g-8484548c96\" is progressing."}}, CollisionCount:(*int32)(nil)}
Jan 17 22:12:27.793: INFO: deployment status: v1.DeploymentStatus{ObservedGeneration:1, Replicas:1, UpdatedReplicas:1, ReadyReplicas:0, AvailableReplicas:0, UnavailableReplicas:1, Conditions:[]v1.DeploymentCondition{v1.DeploymentCondition{Type:"Available", Status:"False", LastUpdateTime:time.Date(2023, time.January, 17, 22, 11, 41, 0, time.Local), LastTransitionTime:time.Date(2023, time.January, 17, 22, 11, 41, 0, time.Local), Reason:"MinimumReplicasUnavailable", Message:"Deployment does not have minimum availability."}, v1.DeploymentCondition{Type:"Progressing", Status:"True", LastUpdateTime:time.Date(2023, time.January, 17, 22, 11, 41, 0, time.Local), LastTransitionTime:time.Date(2023, time.January, 17, 22, 11, 41, 0, time.Local), Reason:"ReplicaSetUpdated", Message:"ReplicaSet \"azuredisk-volume-tester-b9b8g-8484548c96\" is progressing."}}, CollisionCount:(*int32)(nil)}
[1mSTEP:[0m checking that the pod is running [38;5;243m01/17/23 22:12:30.008[0m
[1mSTEP:[0m check pod exec [38;5;243m01/17/23 22:12:30.324[0m
Jan 17 22:12:30.324: INFO: Running '/usr/local/bin/kubectl --kubeconfig=/home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure/kubeconfig --namespace=azuredisk-3208 exec azuredisk-volume-tester-b9b8g-8484548c96-nc9ff -- cmd /c type C:\mnt\test-1\data.txt'
Jan 17 22:12:31.510: INFO: rc: 1
Jan 17 22:12:31.510: INFO: Error waiting for output "hello world\r\n" in pod "azuredisk-volume-tester-b9b8g-8484548c96-nc9ff": error running /usr/local/bin/kubectl --kubeconfig=/home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure/kubeconfig --namespace=azuredisk-3208 exec azuredisk-volume-tester-b9b8g-8484548c96-nc9ff -- cmd /c type C:\mnt\test-1\data.txt:
Command stdout:

stderr:
The system cannot find the file specified.
command terminated with exit code 1

error:
exit status 1.
Jan 17 22:12:33.512: INFO: Running '/usr/local/bin/kubectl --kubeconfig=/home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure/kubeconfig --namespace=azuredisk-3208 exec azuredisk-volume-tester-b9b8g-8484548c96-nc9ff -- cmd /c type C:\mnt\test-1\data.txt'
Jan 17 22:12:34.661: INFO: stderr: ""
Jan 17 22:12:34.662: INFO: stdout: "hello world\r\n"
[1mSTEP:[0m cordoning node 0 [38;5;243m01/17/23 22:12:34.662[0m
[1mSTEP:[0m deleting the pod for deployment [38;5;243m01/17/23 22:12:34.898[0m
... skipping 85 lines ...
    Jan 17 22:12:25.793: INFO: deployment status: v1.DeploymentStatus{ObservedGeneration:1, Replicas:1, UpdatedReplicas:1, ReadyReplicas:0, AvailableReplicas:0, UnavailableReplicas:1, Conditions:[]v1.DeploymentCondition{v1.DeploymentCondition{Type:"Available", Status:"False", LastUpdateTime:time.Date(2023, time.January, 17, 22, 11, 41, 0, time.Local), LastTransitionTime:time.Date(2023, time.January, 17, 22, 11, 41, 0, time.Local), Reason:"MinimumReplicasUnavailable", Message:"Deployment does not have minimum availability."}, v1.DeploymentCondition{Type:"Progressing", Status:"True", LastUpdateTime:time.Date(2023, time.January, 17, 22, 11, 41, 0, time.Local), LastTransitionTime:time.Date(2023, time.January, 17, 22, 11, 41, 0, time.Local), Reason:"ReplicaSetUpdated", Message:"ReplicaSet \"azuredisk-volume-tester-b9b8g-8484548c96\" is progressing."}}, CollisionCount:(*int32)(nil)}
    Jan 17 22:12:27.793: INFO: deployment status: v1.DeploymentStatus{ObservedGeneration:1, Replicas:1, UpdatedReplicas:1, ReadyReplicas:0, AvailableReplicas:0, UnavailableReplicas:1, Conditions:[]v1.DeploymentCondition{v1.DeploymentCondition{Type:"Available", Status:"False", LastUpdateTime:time.Date(2023, time.January, 17, 22, 11, 41, 0, time.Local), LastTransitionTime:time.Date(2023, time.January, 17, 22, 11, 41, 0, time.Local), Reason:"MinimumReplicasUnavailable", Message:"Deployment does not have minimum availability."}, v1.DeploymentCondition{Type:"Progressing", Status:"True", LastUpdateTime:time.Date(2023, time.January, 17, 22, 11, 41, 0, time.Local), LastTransitionTime:time.Date(2023, time.January, 17, 22, 11, 41, 0, time.Local), Reason:"ReplicaSetUpdated", Message:"ReplicaSet \"azuredisk-volume-tester-b9b8g-8484548c96\" is progressing."}}, CollisionCount:(*int32)(nil)}
    [1mSTEP:[0m checking that the pod is running [38;5;243m01/17/23 22:12:30.008[0m
    [1mSTEP:[0m check pod exec [38;5;243m01/17/23 22:12:30.324[0m
    Jan 17 22:12:30.324: INFO: Running '/usr/local/bin/kubectl --kubeconfig=/home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure/kubeconfig --namespace=azuredisk-3208 exec azuredisk-volume-tester-b9b8g-8484548c96-nc9ff -- cmd /c type C:\mnt\test-1\data.txt'
    Jan 17 22:12:31.510: INFO: rc: 1
    Jan 17 22:12:31.510: INFO: Error waiting for output "hello world\r\n" in pod "azuredisk-volume-tester-b9b8g-8484548c96-nc9ff": error running /usr/local/bin/kubectl --kubeconfig=/home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure/kubeconfig --namespace=azuredisk-3208 exec azuredisk-volume-tester-b9b8g-8484548c96-nc9ff -- cmd /c type C:\mnt\test-1\data.txt:
    Command stdout:

    stderr:
    The system cannot find the file specified.
    command terminated with exit code 1

    error:
    exit status 1.
    Jan 17 22:12:33.512: INFO: Running '/usr/local/bin/kubectl --kubeconfig=/home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure/kubeconfig --namespace=azuredisk-3208 exec azuredisk-volume-tester-b9b8g-8484548c96-nc9ff -- cmd /c type C:\mnt\test-1\data.txt'
    Jan 17 22:12:34.661: INFO: stderr: ""
    Jan 17 22:12:34.662: INFO: stdout: "hello world\r\n"
    [1mSTEP:[0m cordoning node 0 [38;5;243m01/17/23 22:12:34.662[0m
    [1mSTEP:[0m deleting the pod for deployment [38;5;243m01/17/23 22:12:34.898[0m
... skipping 79 lines ...
Jan 17 22:14:15.489: INFO: deployment status: v1.DeploymentStatus{ObservedGeneration:1, Replicas:1, UpdatedReplicas:1, ReadyReplicas:0, AvailableReplicas:0, UnavailableReplicas:1, Conditions:[]v1.DeploymentCondition{v1.DeploymentCondition{Type:"Available", Status:"False", LastUpdateTime:time.Date(2023, time.January, 17, 22, 13, 41, 0, time.Local), LastTransitionTime:time.Date(2023, time.January, 17, 22, 13, 41, 0, time.Local), Reason:"MinimumReplicasUnavailable", Message:"Deployment does not have minimum availability."}, v1.DeploymentCondition{Type:"Progressing", Status:"True", LastUpdateTime:time.Date(2023, time.January, 17, 22, 13, 41, 0, time.Local), LastTransitionTime:time.Date(2023, time.January, 17, 22, 13, 41, 0, time.Local), Reason:"ReplicaSetUpdated", Message:"ReplicaSet \"azuredisk-volume-tester-24q6b-5df9ffbdd9\" is progressing."}}, CollisionCount:(*int32)(nil)}
Jan 17 22:14:17.489: INFO: deployment status: v1.DeploymentStatus{ObservedGeneration:1, Replicas:1, UpdatedReplicas:1, ReadyReplicas:0, AvailableReplicas:0, UnavailableReplicas:1, Conditions:[]v1.DeploymentCondition{v1.DeploymentCondition{Type:"Available", Status:"False", LastUpdateTime:time.Date(2023, time.January, 17, 22, 13, 41, 0, time.Local), LastTransitionTime:time.Date(2023, time.January, 17, 22, 13, 41, 0, time.Local), Reason:"MinimumReplicasUnavailable", Message:"Deployment does not have minimum availability."}, v1.DeploymentCondition{Type:"Progressing", Status:"True", LastUpdateTime:time.Date(2023, time.January, 17, 22, 13, 41, 0, time.Local), LastTransitionTime:time.Date(2023, time.January, 17, 22, 13, 41, 0, time.Local), Reason:"ReplicaSetUpdated", Message:"ReplicaSet \"azuredisk-volume-tester-24q6b-5df9ffbdd9\" is progressing."}}, CollisionCount:(*int32)(nil)}
[1mSTEP:[0m checking that the pod is running [38;5;243m01/17/23 22:14:19.709[0m
[1mSTEP:[0m check pod exec [38;5;243m01/17/23 22:14:20.026[0m
Jan 17 22:14:20.026: INFO: Running '/usr/local/bin/kubectl --kubeconfig=/home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure/kubeconfig --namespace=azuredisk-7724 exec azuredisk-volume-tester-24q6b-5df9ffbdd9-gcb45 -- cmd /c type C:\mnt\test-1\data.txt'
Jan 17 22:14:21.161: INFO: rc: 1
Jan 17 22:14:21.161: INFO: Error waiting for output "hello world\r\n" in pod "azuredisk-volume-tester-24q6b-5df9ffbdd9-gcb45": error running /usr/local/bin/kubectl --kubeconfig=/home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure/kubeconfig --namespace=azuredisk-7724 exec azuredisk-volume-tester-24q6b-5df9ffbdd9-gcb45 -- cmd /c type C:\mnt\test-1\data.txt:
Command stdout:

stderr:
The system cannot find the file specified.
command terminated with exit code 1

error:
exit status 1.
Jan 17 22:14:23.163: INFO: Running '/usr/local/bin/kubectl --kubeconfig=/home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure/kubeconfig --namespace=azuredisk-7724 exec azuredisk-volume-tester-24q6b-5df9ffbdd9-gcb45 -- cmd /c type C:\mnt\test-1\data.txt'
Jan 17 22:14:24.318: INFO: stderr: ""
Jan 17 22:14:24.318: INFO: stdout: "hello world\r\n"
[1mSTEP:[0m cordoning node 0 [38;5;243m01/17/23 22:14:24.318[0m
[1mSTEP:[0m deleting the pod for deployment [38;5;243m01/17/23 22:14:24.555[0m
... skipping 80 lines ...
    Jan 17 22:14:15.489: INFO: deployment status: v1.DeploymentStatus{ObservedGeneration:1, Replicas:1, UpdatedReplicas:1, ReadyReplicas:0, AvailableReplicas:0, UnavailableReplicas:1, Conditions:[]v1.DeploymentCondition{v1.DeploymentCondition{Type:"Available", Status:"False", LastUpdateTime:time.Date(2023, time.January, 17, 22, 13, 41, 0, time.Local), LastTransitionTime:time.Date(2023, time.January, 17, 22, 13, 41, 0, time.Local), Reason:"MinimumReplicasUnavailable", Message:"Deployment does not have minimum availability."}, v1.DeploymentCondition{Type:"Progressing", Status:"True", LastUpdateTime:time.Date(2023, time.January, 17, 22, 13, 41, 0, time.Local), LastTransitionTime:time.Date(2023, time.January, 17, 22, 13, 41, 0, time.Local), Reason:"ReplicaSetUpdated", Message:"ReplicaSet \"azuredisk-volume-tester-24q6b-5df9ffbdd9\" is progressing."}}, CollisionCount:(*int32)(nil)}
    Jan 17 22:14:17.489: INFO: deployment status: v1.DeploymentStatus{ObservedGeneration:1, Replicas:1, UpdatedReplicas:1, ReadyReplicas:0, AvailableReplicas:0, UnavailableReplicas:1, Conditions:[]v1.DeploymentCondition{v1.DeploymentCondition{Type:"Available", Status:"False", LastUpdateTime:time.Date(2023, time.January, 17, 22, 13, 41, 0, time.Local), LastTransitionTime:time.Date(2023, time.January, 17, 22, 13, 41, 0, time.Local), Reason:"MinimumReplicasUnavailable", Message:"Deployment does not have minimum availability."}, v1.DeploymentCondition{Type:"Progressing", Status:"True", LastUpdateTime:time.Date(2023, time.January, 17, 22, 13, 41, 0, time.Local), LastTransitionTime:time.Date(2023, time.January, 17, 22, 13, 41, 0, time.Local), Reason:"ReplicaSetUpdated", Message:"ReplicaSet \"azuredisk-volume-tester-24q6b-5df9ffbdd9\" is progressing."}}, CollisionCount:(*int32)(nil)}
    [1mSTEP:[0m checking that the pod is running [38;5;243m01/17/23 22:14:19.709[0m
    [1mSTEP:[0m check pod exec [38;5;243m01/17/23 22:14:20.026[0m
    Jan 17 22:14:20.026: INFO: Running '/usr/local/bin/kubectl --kubeconfig=/home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure/kubeconfig --namespace=azuredisk-7724 exec azuredisk-volume-tester-24q6b-5df9ffbdd9-gcb45 -- cmd /c type C:\mnt\test-1\data.txt'
    Jan 17 22:14:21.161: INFO: rc: 1
    Jan 17 22:14:21.161: INFO: Error waiting for output "hello world\r\n" in pod "azuredisk-volume-tester-24q6b-5df9ffbdd9-gcb45": error running /usr/local/bin/kubectl --kubeconfig=/home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure/kubeconfig --namespace=azuredisk-7724 exec azuredisk-volume-tester-24q6b-5df9ffbdd9-gcb45 -- cmd /c type C:\mnt\test-1\data.txt:
    Command stdout:

    stderr:
    The system cannot find the file specified.
    command terminated with exit code 1

    error:
    exit status 1.
    Jan 17 22:14:23.163: INFO: Running '/usr/local/bin/kubectl --kubeconfig=/home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure/kubeconfig --namespace=azuredisk-7724 exec azuredisk-volume-tester-24q6b-5df9ffbdd9-gcb45 -- cmd /c type C:\mnt\test-1\data.txt'
    Jan 17 22:14:24.318: INFO: stderr: ""
    Jan 17 22:14:24.318: INFO: stdout: "hello world\r\n"
    [1mSTEP:[0m cordoning node 0 [38;5;243m01/17/23 22:14:24.318[0m
    [1mSTEP:[0m deleting the pod for deployment [38;5;243m01/17/23 22:14:24.555[0m
... skipping 138 lines ...
  [38;5;243m<< End Captured GinkgoWriter Output[0m

  [38;5;14mtest case not supported by Windows clusters[0m
  [38;5;14mIn [1m[It][0m[38;5;14m at: [1m/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/utils/testutil/env_utils.go:26[0m
[38;5;243m------------------------------[0m
[0mDynamic Provisioning [38;5;243m[single-az][0m
  [1mshould check failed replica attachments are recreated after space is made from a volume detaching.[0m
  [38;5;243m/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning_test.go:1778[0m
[1mSTEP:[0m Creating a kubernetes client [38;5;243m01/17/23 22:15:29.184[0m
Jan 17 22:15:29.185: INFO: >>> kubeConfig: /home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure/kubeconfig
[1mSTEP:[0m Building a namespace api object, basename azuredisk [38;5;243m01/17/23 22:15:29.186[0m
[1mSTEP:[0m Waiting for a default service account to be provisioned in namespace [38;5;243m01/17/23 22:15:29.504[0m
[1mSTEP:[0m Waiting for kube-root-ca.crt to be provisioned in namespace [38;5;243m01/17/23 22:15:29.71[0m
... skipping 6 lines ...
[38;5;243m------------------------------[0m
[38;5;14mS [SKIPPED] [1.586 seconds][0m
Dynamic Provisioning
[38;5;243m/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning_test.go:41[0m
  [single-az]
  [38;5;243m/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning_test.go:45[0m
    [38;5;14m[1m[It] should check failed replica attachments are recreated after space is made from a volume detaching.[0m
    [38;5;243m/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning_test.go:1778[0m

  [38;5;243mBegin Captured GinkgoWriter Output >>[0m
    [1mSTEP:[0m Creating a kubernetes client [38;5;243m01/17/23 22:15:29.184[0m
    Jan 17 22:15:29.185: INFO: >>> kubeConfig: /home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure/kubeconfig
    [1mSTEP:[0m Building a namespace api object, basename azuredisk [38;5;243m01/17/23 22:15:29.186[0m
... skipping 27 lines ...
Jan 17 22:15:36.976: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [pvc-kxfz5] to have phase Bound
Jan 17 22:15:37.080: INFO: PersistentVolumeClaim pvc-kxfz5 found and phase=Bound (104.337756ms)
[1mSTEP:[0m checking the PVC [38;5;243m01/17/23 22:15:37.081[0m
[1mSTEP:[0m validating provisioned PV [38;5;243m01/17/23 22:15:37.185[0m
[1mSTEP:[0m checking the PV [38;5;243m01/17/23 22:15:37.288[0m
[1mSTEP:[0m deploying the pod [38;5;243m01/17/23 22:15:37.289[0m
[1mSTEP:[0m checking that the pod's command exits with an error [38;5;243m01/17/23 22:15:37.4[0m
Jan 17 22:15:37.400: INFO: Waiting up to 10m0s for pod "azuredisk-volume-tester-d2jd8" in namespace "azuredisk-7381" to be "Error status code"
Jan 17 22:15:37.508: INFO: Pod "azuredisk-volume-tester-d2jd8": Phase="Pending", Reason="", readiness=false. Elapsed: 107.318092ms
Jan 17 22:15:39.612: INFO: Pod "azuredisk-volume-tester-d2jd8": Phase="Pending", Reason="", readiness=false. Elapsed: 2.211816594s
Jan 17 22:15:41.612: INFO: Pod "azuredisk-volume-tester-d2jd8": Phase="Pending", Reason="", readiness=false. Elapsed: 4.21170513s
Jan 17 22:15:43.611: INFO: Pod "azuredisk-volume-tester-d2jd8": Phase="Pending", Reason="", readiness=false. Elapsed: 6.210679385s
Jan 17 22:15:45.613: INFO: Pod "azuredisk-volume-tester-d2jd8": Phase="Pending", Reason="", readiness=false. Elapsed: 8.212905134s
Jan 17 22:15:47.612: INFO: Pod "azuredisk-volume-tester-d2jd8": Phase="Pending", Reason="", readiness=false. Elapsed: 10.211956825s
... skipping 14 lines ...
Jan 17 22:16:17.618: INFO: Pod "azuredisk-volume-tester-d2jd8": Phase="Running", Reason="", readiness=true. Elapsed: 40.217575867s
Jan 17 22:16:19.617: INFO: Pod "azuredisk-volume-tester-d2jd8": Phase="Running", Reason="", readiness=true. Elapsed: 42.216775239s
Jan 17 22:16:21.618: INFO: Pod "azuredisk-volume-tester-d2jd8": Phase="Running", Reason="", readiness=true. Elapsed: 44.217939902s
Jan 17 22:16:23.618: INFO: Pod "azuredisk-volume-tester-d2jd8": Phase="Running", Reason="", readiness=false. Elapsed: 46.217206722s
Jan 17 22:16:25.618: INFO: Pod "azuredisk-volume-tester-d2jd8": Phase="Running", Reason="", readiness=false. Elapsed: 48.217263803s
Jan 17 22:16:27.617: INFO: Pod "azuredisk-volume-tester-d2jd8": Phase="Running", Reason="", readiness=false. Elapsed: 50.216646127s
Jan 17 22:16:29.617: INFO: Pod "azuredisk-volume-tester-d2jd8": Phase="Failed", Reason="", readiness=false. Elapsed: 52.216343056s
[1mSTEP:[0m Saw pod failure [38;5;243m01/17/23 22:16:29.617[0m
Jan 17 22:16:29.617: INFO: Pod "azuredisk-volume-tester-d2jd8" satisfied condition "Error status code"
[1mSTEP:[0m checking that pod logs contain expected message [38;5;243m01/17/23 22:16:29.617[0m
Jan 17 22:16:29.750: INFO: deleting Pod "azuredisk-7381"/"azuredisk-volume-tester-d2jd8"
Jan 17 22:16:29.859: INFO: Pod azuredisk-volume-tester-d2jd8 has the following logs: Out-File : Access to the path 'C:\mnt\test-1\data.txt' is denied.
At line:1 char:22
+ ... cho 'hello world' | Out-File -FilePath C:\mnt\test-1\data.txt; Get-Co ...
+                         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
... skipping 55 lines ...
    Jan 17 22:15:36.976: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [pvc-kxfz5] to have phase Bound
    Jan 17 22:15:37.080: INFO: PersistentVolumeClaim pvc-kxfz5 found and phase=Bound (104.337756ms)
    [1mSTEP:[0m checking the PVC [38;5;243m01/17/23 22:15:37.081[0m
    [1mSTEP:[0m validating provisioned PV [38;5;243m01/17/23 22:15:37.185[0m
    [1mSTEP:[0m checking the PV [38;5;243m01/17/23 22:15:37.288[0m
    [1mSTEP:[0m deploying the pod [38;5;243m01/17/23 22:15:37.289[0m
    [1mSTEP:[0m checking that the pod's command exits with an error [38;5;243m01/17/23 22:15:37.4[0m
    Jan 17 22:15:37.400: INFO: Waiting up to 10m0s for pod "azuredisk-volume-tester-d2jd8" in namespace "azuredisk-7381" to be "Error status code"
    Jan 17 22:15:37.508: INFO: Pod "azuredisk-volume-tester-d2jd8": Phase="Pending", Reason="", readiness=false. Elapsed: 107.318092ms
    Jan 17 22:15:39.612: INFO: Pod "azuredisk-volume-tester-d2jd8": Phase="Pending", Reason="", readiness=false. Elapsed: 2.211816594s
    Jan 17 22:15:41.612: INFO: Pod "azuredisk-volume-tester-d2jd8": Phase="Pending", Reason="", readiness=false. Elapsed: 4.21170513s
    Jan 17 22:15:43.611: INFO: Pod "azuredisk-volume-tester-d2jd8": Phase="Pending", Reason="", readiness=false. Elapsed: 6.210679385s
    Jan 17 22:15:45.613: INFO: Pod "azuredisk-volume-tester-d2jd8": Phase="Pending", Reason="", readiness=false. Elapsed: 8.212905134s
    Jan 17 22:15:47.612: INFO: Pod "azuredisk-volume-tester-d2jd8": Phase="Pending", Reason="", readiness=false. Elapsed: 10.211956825s
... skipping 14 lines ...
    Jan 17 22:16:17.618: INFO: Pod "azuredisk-volume-tester-d2jd8": Phase="Running", Reason="", readiness=true. Elapsed: 40.217575867s
    Jan 17 22:16:19.617: INFO: Pod "azuredisk-volume-tester-d2jd8": Phase="Running", Reason="", readiness=true. Elapsed: 42.216775239s
    Jan 17 22:16:21.618: INFO: Pod "azuredisk-volume-tester-d2jd8": Phase="Running", Reason="", readiness=true. Elapsed: 44.217939902s
    Jan 17 22:16:23.618: INFO: Pod "azuredisk-volume-tester-d2jd8": Phase="Running", Reason="", readiness=false. Elapsed: 46.217206722s
    Jan 17 22:16:25.618: INFO: Pod "azuredisk-volume-tester-d2jd8": Phase="Running", Reason="", readiness=false. Elapsed: 48.217263803s
    Jan 17 22:16:27.617: INFO: Pod "azuredisk-volume-tester-d2jd8": Phase="Running", Reason="", readiness=false. Elapsed: 50.216646127s
    Jan 17 22:16:29.617: INFO: Pod "azuredisk-volume-tester-d2jd8": Phase="Failed", Reason="", readiness=false. Elapsed: 52.216343056s
    [1mSTEP:[0m Saw pod failure [38;5;243m01/17/23 22:16:29.617[0m
    Jan 17 22:16:29.617: INFO: Pod "azuredisk-volume-tester-d2jd8" satisfied condition "Error status code"
    [1mSTEP:[0m checking that pod logs contain expected message [38;5;243m01/17/23 22:16:29.617[0m
    Jan 17 22:16:29.750: INFO: deleting Pod "azuredisk-7381"/"azuredisk-volume-tester-d2jd8"
    Jan 17 22:16:29.859: INFO: Pod azuredisk-volume-tester-d2jd8 has the following logs: Out-File : Access to the path 'C:\mnt\test-1\data.txt' is denied.
    At line:1 char:22
    + ... cho 'hello world' | Out-File -FilePath C:\mnt\test-1\data.txt; Get-Co ...
    +                         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
... skipping 84 lines ...
Jan 17 22:17:12.118: INFO: PersistentVolumeClaim pvc-6wldd found and phase=Bound (104.047778ms)
[1mSTEP:[0m checking the PVC [38;5;243m01/17/23 22:17:12.118[0m
[1mSTEP:[0m validating provisioned PV [38;5;243m01/17/23 22:17:12.222[0m
[1mSTEP:[0m checking the PV [38;5;243m01/17/23 22:17:12.327[0m
[1mSTEP:[0m attaching disk to node#0 [38;5;243m01/17/23 22:17:12.46[0m
[1mSTEP:[0m deploying the pod [38;5;243m01/17/23 22:17:21.44[0m
[1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/17/23 22:17:21.55[0m
Jan 17 22:17:21.550: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-xsgxb" in namespace "azuredisk-3399" to be "Succeeded or Failed"
Jan 17 22:17:21.655: INFO: Pod "azuredisk-volume-tester-xsgxb": Phase="Pending", Reason="", readiness=false. Elapsed: 104.069246ms
Jan 17 22:17:23.760: INFO: Pod "azuredisk-volume-tester-xsgxb": Phase="Pending", Reason="", readiness=false. Elapsed: 2.209036321s
Jan 17 22:17:25.760: INFO: Pod "azuredisk-volume-tester-xsgxb": Phase="Pending", Reason="", readiness=false. Elapsed: 4.209035366s
Jan 17 22:17:27.761: INFO: Pod "azuredisk-volume-tester-xsgxb": Phase="Pending", Reason="", readiness=false. Elapsed: 6.210159849s
Jan 17 22:17:29.766: INFO: Pod "azuredisk-volume-tester-xsgxb": Phase="Pending", Reason="", readiness=false. Elapsed: 8.215580005s
Jan 17 22:17:31.761: INFO: Pod "azuredisk-volume-tester-xsgxb": Phase="Pending", Reason="", readiness=false. Elapsed: 10.210213973s
... skipping 22 lines ...
Jan 17 22:18:17.765: INFO: Pod "azuredisk-volume-tester-xsgxb": Phase="Running", Reason="", readiness=true. Elapsed: 56.214569207s
Jan 17 22:18:19.764: INFO: Pod "azuredisk-volume-tester-xsgxb": Phase="Running", Reason="", readiness=true. Elapsed: 58.213198506s
Jan 17 22:18:21.765: INFO: Pod "azuredisk-volume-tester-xsgxb": Phase="Running", Reason="", readiness=false. Elapsed: 1m0.214702048s
Jan 17 22:18:23.765: INFO: Pod "azuredisk-volume-tester-xsgxb": Phase="Running", Reason="", readiness=false. Elapsed: 1m2.214988999s
Jan 17 22:18:25.766: INFO: Pod "azuredisk-volume-tester-xsgxb": Phase="Succeeded", Reason="", readiness=false. Elapsed: 1m4.215066116s
[1mSTEP:[0m Saw pod success [38;5;243m01/17/23 22:18:25.766[0m
Jan 17 22:18:25.766: INFO: Pod "azuredisk-volume-tester-xsgxb" satisfied condition "Succeeded or Failed"
Jan 17 22:18:25.766: INFO: deleting Pod "azuredisk-3399"/"azuredisk-volume-tester-xsgxb"
Jan 17 22:18:25.898: INFO: Pod azuredisk-volume-tester-xsgxb has the following logs: hello world

[1mSTEP:[0m Deleting pod azuredisk-volume-tester-xsgxb in namespace azuredisk-3399 [38;5;243m01/17/23 22:18:25.898[0m
Jan 17 22:18:26.287: INFO: deleting PVC "azuredisk-3399"/"pvc-6wldd"
Jan 17 22:18:26.287: INFO: Deleting PersistentVolumeClaim "pvc-6wldd"
... skipping 37 lines ...
    Jan 17 22:17:12.118: INFO: PersistentVolumeClaim pvc-6wldd found and phase=Bound (104.047778ms)
    [1mSTEP:[0m checking the PVC [38;5;243m01/17/23 22:17:12.118[0m
    [1mSTEP:[0m validating provisioned PV [38;5;243m01/17/23 22:17:12.222[0m
    [1mSTEP:[0m checking the PV [38;5;243m01/17/23 22:17:12.327[0m
    [1mSTEP:[0m attaching disk to node#0 [38;5;243m01/17/23 22:17:12.46[0m
    [1mSTEP:[0m deploying the pod [38;5;243m01/17/23 22:17:21.44[0m
    [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/17/23 22:17:21.55[0m
    Jan 17 22:17:21.550: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-xsgxb" in namespace "azuredisk-3399" to be "Succeeded or Failed"
    Jan 17 22:17:21.655: INFO: Pod "azuredisk-volume-tester-xsgxb": Phase="Pending", Reason="", readiness=false. Elapsed: 104.069246ms
    Jan 17 22:17:23.760: INFO: Pod "azuredisk-volume-tester-xsgxb": Phase="Pending", Reason="", readiness=false. Elapsed: 2.209036321s
    Jan 17 22:17:25.760: INFO: Pod "azuredisk-volume-tester-xsgxb": Phase="Pending", Reason="", readiness=false. Elapsed: 4.209035366s
    Jan 17 22:17:27.761: INFO: Pod "azuredisk-volume-tester-xsgxb": Phase="Pending", Reason="", readiness=false. Elapsed: 6.210159849s
    Jan 17 22:17:29.766: INFO: Pod "azuredisk-volume-tester-xsgxb": Phase="Pending", Reason="", readiness=false. Elapsed: 8.215580005s
    Jan 17 22:17:31.761: INFO: Pod "azuredisk-volume-tester-xsgxb": Phase="Pending", Reason="", readiness=false. Elapsed: 10.210213973s
... skipping 22 lines ...
    Jan 17 22:18:17.765: INFO: Pod "azuredisk-volume-tester-xsgxb": Phase="Running", Reason="", readiness=true. Elapsed: 56.214569207s
    Jan 17 22:18:19.764: INFO: Pod "azuredisk-volume-tester-xsgxb": Phase="Running", Reason="", readiness=true. Elapsed: 58.213198506s
    Jan 17 22:18:21.765: INFO: Pod "azuredisk-volume-tester-xsgxb": Phase="Running", Reason="", readiness=false. Elapsed: 1m0.214702048s
    Jan 17 22:18:23.765: INFO: Pod "azuredisk-volume-tester-xsgxb": Phase="Running", Reason="", readiness=false. Elapsed: 1m2.214988999s
    Jan 17 22:18:25.766: INFO: Pod "azuredisk-volume-tester-xsgxb": Phase="Succeeded", Reason="", readiness=false. Elapsed: 1m4.215066116s
    [1mSTEP:[0m Saw pod success [38;5;243m01/17/23 22:18:25.766[0m
    Jan 17 22:18:25.766: INFO: Pod "azuredisk-volume-tester-xsgxb" satisfied condition "Succeeded or Failed"
    Jan 17 22:18:25.766: INFO: deleting Pod "azuredisk-3399"/"azuredisk-volume-tester-xsgxb"
    Jan 17 22:18:25.898: INFO: Pod azuredisk-volume-tester-xsgxb has the following logs: hello world

    [1mSTEP:[0m Deleting pod azuredisk-volume-tester-xsgxb in namespace azuredisk-3399 [38;5;243m01/17/23 22:18:25.898[0m
    Jan 17 22:18:26.287: INFO: deleting PVC "azuredisk-3399"/"pvc-6wldd"
    Jan 17 22:18:26.287: INFO: Deleting PersistentVolumeClaim "pvc-6wldd"
... skipping 150 lines ...
  [38;5;243m<< End Captured GinkgoWriter Output[0m

  [38;5;14mtest case not supported by Windows clusters[0m
  [38;5;14mIn [1m[It][0m[38;5;14m at: [1m/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/utils/testutil/env_utils.go:26[0m

  [1mThere were additional failures detected after the initial failure:[0m
    [38;5;9m[FAILED][0m
    [38;5;9mpre-provisioning disk got deleted with the deletion of pv: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {
      "error": {

        "code": "NotFound",
        "message": "Disk single-shared-disk is not found."
      }
    }[0m
    [38;5;9mIn [1m[AfterEach][0m[38;5;9m at: [1m/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/pre_provisioning_test.go:490[0m
[38;5;243m------------------------------[0m
... skipping 27 lines ...
  [38;5;243m<< End Captured GinkgoWriter Output[0m

  [38;5;14mtest case not supported by Windows clusters[0m
  [38;5;14mIn [1m[It][0m[38;5;14m at: [1m/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/utils/testutil/env_utils.go:26[0m

  [1mThere were additional failures detected after the initial failure:[0m
    [38;5;9m[FAILED][0m
    [38;5;9mpre-provisioning disk got deleted with the deletion of pv: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {
      "error": {

        "code": "NotFound",
        "message": "Disk single-shared-disk is not found."
      }
    }[0m
    [38;5;9mIn [1m[AfterEach][0m[38;5;9m at: [1m/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/pre_provisioning_test.go:490[0m
[38;5;243m------------------------------[0m
... skipping 27 lines ...
  [38;5;243m<< End Captured GinkgoWriter Output[0m

  [38;5;14mtest case is only available for csi driver[0m
  [38;5;14mIn [1m[It][0m[38;5;14m at: [1m/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/pre_provisioning_test.go:395[0m

  [1mThere were additional failures detected after the initial failure:[0m
    [38;5;9m[FAILED][0m
    [38;5;9mpre-provisioning disk got deleted with the deletion of pv: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {
      "error": {

        "code": "NotFound",
        "message": "Disk single-shared-disk is not found."
      }
    }[0m
    [38;5;9mIn [1m[AfterEach][0m[38;5;9m at: [1m/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/pre_provisioning_test.go:490[0m
[38;5;243m------------------------------[0m
... skipping 146 lines ...
I0117 20:25:42.171145       1 round_trippers.go:553] GET https://10.96.0.1:443/apis/apiextensions.k8s.io/v1/customresourcedefinitions?limit=500&resourceVersion=0 200 OK in 35 milliseconds
I0117 20:25:42.214727       1 round_trippers.go:553] GET https://10.96.0.1:443/apis/apiextensions.k8s.io/v1/customresourcedefinitions?allowWatchBookmarks=true&resourceVersion=2085&timeout=6m17s&timeoutSeconds=377&watch=true 200 OK in 1 milliseconds
I0117 20:25:42.235988       1 shared_informer.go:303] caches populated
I0117 20:25:42.236053       1 azuredisk_v2.go:225] driver userAgent: disk.csi.azure.com/latest-v2-9ef068a8cb36a997d4ea04b90c05c6f92a488a19 e2e-test
I0117 20:25:42.236151       1 azure_disk_utils.go:564] reading cloud config from secret kube-system/azure-cloud-provider
I0117 20:25:42.238467       1 round_trippers.go:553] GET https://10.96.0.1:443/api/v1/namespaces/kube-system/secrets/azure-cloud-provider 404 Not Found in 2 milliseconds
I0117 20:25:42.238788       1 azure_disk_utils.go:571] InitializeCloudFromSecret: failed to get cloud config from secret kube-system/azure-cloud-provider: failed to get secret kube-system/azure-cloud-provider: secrets "azure-cloud-provider" not found
I0117 20:25:42.238809       1 azure_disk_utils.go:576] could not read cloud config from secret kube-system/azure-cloud-provider
I0117 20:25:42.238815       1 azure_disk_utils.go:586] use default AZURE_CREDENTIAL_FILE env var: /etc/kubernetes/azure.json
I0117 20:25:42.238844       1 azure_disk_utils.go:594] read cloud config from file: /etc/kubernetes/azure.json successfully
I0117 20:25:42.239587       1 azure_auth.go:253] Using AzurePublicCloud environment
I0117 20:25:42.239627       1 azure_auth.go:138] azure: using client_id+client_secret to retrieve access token
I0117 20:25:42.239853       1 azure_diskclient.go:68] Azure DisksClient using API version: 2022-03-02
... skipping 9 lines ...
I0117 20:25:42.925693       1 azure_backoff.go:101] VirtualMachinesClient.List(capz-ri100d) success
I0117 20:25:42.925709       1 azure_vmss_cache.go:400] Node  has joined the cluster since the last VM cache refresh in NonVmssUniformNodesEntry, refreshing the cache
I0117 20:25:42.925717       1 azure_vmss_cache.go:327] refresh the cache of NonVmssUniformNodesCache in rg map[capz-ri100d:{}]
I0117 20:25:42.968361       1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 13869
I0117 20:25:42.969804       1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=0.044026419 request="vm_list" resource_group="capz-ri100d" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0117 20:25:42.969832       1 azure_backoff.go:101] VirtualMachinesClient.List(capz-ri100d) success
E0117 20:25:42.969871       1 azuredisk_v2.go:272] Failed to get node info. Error: NewNodeInfo: Failed to get instance type from Azure cloud provider, nodeName: , error: not a vmss instance
I0117 20:25:42.970013       1 mount_linux.go:208] Detected OS without systemd
I0117 20:25:42.970031       1 driver.go:81] Enabling controller service capability: CREATE_DELETE_VOLUME
I0117 20:25:42.970036       1 driver.go:81] Enabling controller service capability: PUBLISH_UNPUBLISH_VOLUME
I0117 20:25:42.970041       1 driver.go:81] Enabling controller service capability: CREATE_DELETE_SNAPSHOT
I0117 20:25:42.970046       1 driver.go:81] Enabling controller service capability: CLONE_VOLUME
I0117 20:25:42.970050       1 driver.go:81] Enabling controller service capability: EXPAND_VOLUME
... skipping 249 lines ...
I0117 20:25:43.230498       1 shared_state.go:490]  "msg"="Storing pod csi-azuredisk-node-win-tppwd and claim [] to podToClaimsMap map." "disk.csi.azure.com/request-id"="1db2ea60-96a5-11ed-866b-002248a3db84"
I0117 20:25:43.230549       1 pod.go:91]  "msg"="Creating replicas for pod kube-system/csi-azuredisk-node-win-tppwd." "disk.csi.azure.com/request-id"="1db2ea60-96a5-11ed-866b-002248a3db84" "disk.csi/azure.com/pod-name"="kube-system/csi-azuredisk-node-win-tppwd"
I0117 20:25:43.230749       1 shared_state.go:314]  "msg"="Getting requested volumes for pod (kube-system/csi-azuredisk-node-win-tppwd)." "disk.csi.azure.com/request-id"="1db2ea60-96a5-11ed-866b-002248a3db84" "disk.csi/azure.com/pod-name"="kube-system/csi-azuredisk-node-win-tppwd"
I0117 20:25:43.230786       1 pod.go:99]  "msg"="Pod kube-system/csi-azuredisk-node-win-tppwd has 0 volumes. Volumes: []" "disk.csi.azure.com/request-id"="1db2ea60-96a5-11ed-866b-002248a3db84" "disk.csi/azure.com/pod-name"="kube-system/csi-azuredisk-node-win-tppwd"
I0117 20:25:43.230897       1 pod.go:89]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="1db2ea60-96a5-11ed-866b-002248a3db84" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcilePod).createReplicas" "disk.csi/azure.com/pod-name"="kube-system/csi-azuredisk-node-win-tppwd" "latency"=285409
I0117 20:25:43.231007       1 shared_state.go:420]  "msg"="Adding pod csi-azuredisk-controller-7f6f8d54b5-l6m44 to shared map with keyName kube-system/csi-azuredisk-controller-7f6f8d54b5-l6m44." "disk.csi.azure.com/request-id"="1db2ea60-96a5-11ed-866b-002248a3db84"
I0117 20:25:43.231479       1 shared_state.go:426]  "msg"="Pod spec of pod csi-azuredisk-controller-7f6f8d54b5-l6m44 is: {Volumes:[{Name:socket-dir VolumeSource:{HostPath:nil EmptyDir:&EmptyDirVolumeSource{Medium:,SizeLimit:<nil>,} GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:azure-cred VolumeSource:{HostPath:&HostPathVolumeSource{Path:/etc/kubernetes/,Type:*DirectoryOrCreate,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:csi-azuredisk-controller-config VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:&ConfigMapVolumeSource{LocalObjectReference:LocalObjectReference{Name:csi-azuredisk-controller-config,},Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,} VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:kube-api-access-lg6t4 VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:&ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,} PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:csi-provisioner Image:mcr.microsoft.com/oss/kubernetes-csi/csi-provisioner:v3.2.0 Command:[] Args:[--feature-gates=Topology=true --csi-address=$(ADDRESS) --v=2 --timeout=30s --leader-election --leader-election-namespace=kube-system --worker-threads=100 --extra-create-metadata=true --strict-topology=true --kube-api-qps=50 --kube-api-burst=100] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:ADDRESS Value:/csi/csi.sock ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:524288000 scale:0} d:{Dec:<nil>} s:500Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}] Claims:[]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-lg6t4 ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:csi-attacher Image:mcr.microsoft.com/oss/kubernetes-csi/csi-attacher:v3.5.0 Command:[] Args:[-v=2 -csi-address=$(ADDRESS) -timeout=600s -leader-election --leader-election-namespace=kube-system -worker-threads=500 -kube-api-qps=50 -kube-api-burst=100] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:ADDRESS Value:/csi/csi.sock ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:524288000 scale:0} d:{Dec:<nil>} s:500Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}] Claims:[]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-lg6t4 ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:csi-snapshotter Image:mcr.microsoft.com/oss/kubernetes-csi/csi-snapshotter:v5.0.1 Command:[] Args:[-csi-address=$(ADDRESS) -leader-election --leader-election-namespace=kube-system -v=2] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:ADDRESS Value:/csi/csi.sock ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:104857600 scale:0} d:{Dec:<nil>} s:100Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}] Claims:[]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-lg6t4 ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:csi-resizer Image:mcr.microsoft.com/oss/kubernetes-csi/csi-resizer:v1.5.0 Command:[] Args:[-csi-address=$(ADDRESS) -v=2 -leader-election --leader-election-namespace=kube-system -handle-volume-inuse-error=false -feature-gates=RecoverVolumeExpansionFailure=true -timeout=240s] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:ADDRESS Value:/csi/csi.sock ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:524288000 scale:0} d:{Dec:<nil>} s:500Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}] Claims:[]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-lg6t4 ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:liveness-probe Image:mcr.microsoft.com/oss/kubernetes-csi/livenessprobe:v2.7.0 Command:[] Args:[--csi-address=/csi/csi.sock --probe-timeout=3s --health-port=29602 --v=2] WorkingDir: Ports:[] EnvFrom:[] Env:[] Resources:{Limits:map[memory:{i:{value:104857600 scale:0} d:{Dec:<nil>} s:100Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}] Claims:[]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-lg6t4 ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:azuredisk Image:capzci.azurecr.io/azuredisk-csi:latest-v2-9ef068a8cb36a997d4ea04b90c05c6f92a488a19 Command:[] Args:[--v=6 --config=/etc/csi-azuredisk-controller/config.yaml] WorkingDir: Ports:[{Name:healthz HostPort:29602 ContainerPort:29602 Protocol:TCP HostIP:} {Name:metrics HostPort:29604 ContainerPort:29604 Protocol:TCP HostIP:}] EnvFrom:[] Env:[{Name:AZURE_CREDENTIAL_FILE Value: ValueFrom:&EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:&ConfigMapKeySelector{LocalObjectReference:LocalObjectReference{Name:azure-cred-file,},Key:path,Optional:*true,},SecretKeyRef:nil,}} {Name:CSI_ENDPOINT Value:unix:///csi/csi.sock ValueFrom:nil} {Name:AZURE_GO_SDK_LOG_LEVEL Value: ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:524288000 scale:0} d:{Dec:<nil>} s:500Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}] Claims:[]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:azure-cred ReadOnly:false MountPath:/etc/kubernetes/ SubPath: MountPropagation:<nil> SubPathExpr:} {Name:csi-azuredisk-controller-config ReadOnly:false MountPath:/etc/csi-azuredisk-controller SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-lg6t4 ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/healthz,Port:{1 0 healthz},Host:,Scheme:HTTP,HTTPHeaders:[]HTTPHeader{},},TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:30,TimeoutSeconds:10,PeriodSeconds:30,SuccessThreshold:1,FailureThreshold:5,TerminationGracePeriodSeconds:nil,} ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:Always SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy:Always TerminationGracePeriodSeconds:0xc0008c5790 ActiveDeadlineSeconds:<nil> DNSPolicy:ClusterFirst NodeSelector:map[kubernetes.io/os:linux] ServiceAccountName:csi-azuredisk-controller-sa DeprecatedServiceAccount:csi-azuredisk-controller-sa AutomountServiceAccountToken:<nil> NodeName:capz-ri100d-control-plane-8n9zj HostNetwork:true HostPID:false HostIPC:false ShareProcessNamespace:<nil> SecurityContext:&PodSecurityContext{SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,SupplementalGroups:[],FSGroup:nil,RunAsGroup:nil,Sysctls:[]Sysctl{},WindowsOptions:nil,FSGroupChangePolicy:nil,SeccompProfile:nil,} ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName:default-scheduler Tolerations:[{Key:node-role.kubernetes.io/master Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>} {Key:node-role.kubernetes.io/controlplane Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>} {Key:node-role.kubernetes.io/control-plane Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>} {Key:node.kubernetes.io/not-ready Operator:Exists Value: Effect:NoExecute TolerationSeconds:0xc0008c5798} {Key:node.kubernetes.io/unreachable Operator:Exists Value: Effect:NoExecute TolerationSeconds:0xc0008c57a0}] HostAliases:[] PriorityClassName:system-cluster-critical Priority:0xc0008c57a8 DNSConfig:nil ReadinessGates:[] RuntimeClassName:<nil> EnableServiceLinks:0xc0008c57ac PreemptionPolicy:0xc000371440 Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN:<nil> OS:nil HostUsers:<nil> SchedulingGates:[] ResourceClaims:[]}. With volumes: [{Name:socket-dir VolumeSource:{HostPath:nil EmptyDir:&EmptyDirVolumeSource{Medium:,SizeLimit:<nil>,} GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:azure-cred VolumeSource:{HostPath:&HostPathVolumeSource{Path:/etc/kubernetes/,Type:*DirectoryOrCreate,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:csi-azuredisk-controller-config VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:&ConfigMapVolumeSource{LocalObjectReference:LocalObjectReference{Name:csi-azuredisk-controller-config,},Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,} VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:kube-api-access-lg6t4 VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:&ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,} PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}]" "disk.csi.azure.com/request-id"="1db2ea60-96a5-11ed-866b-002248a3db84"
I0117 20:25:43.231548       1 shared_state.go:464]  "msg"="Pod csi-azuredisk-controller-7f6f8d54b5-l6m44: Skipping Volume {socket-dir {nil &EmptyDirVolumeSource{Medium:,SizeLimit:<nil>,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}}. No persistent volume exists." "disk.csi.azure.com/request-id"="1db2ea60-96a5-11ed-866b-002248a3db84"
I0117 20:25:43.231618       1 shared_state.go:464]  "msg"="Pod csi-azuredisk-controller-7f6f8d54b5-l6m44: Skipping Volume {azure-cred {&HostPathVolumeSource{Path:/etc/kubernetes/,Type:*DirectoryOrCreate,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}}. No persistent volume exists." "disk.csi.azure.com/request-id"="1db2ea60-96a5-11ed-866b-002248a3db84"
I0117 20:25:43.231716       1 shared_state.go:464]  "msg"="Pod csi-azuredisk-controller-7f6f8d54b5-l6m44: Skipping Volume {csi-azuredisk-controller-config {nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil &ConfigMapVolumeSource{LocalObjectReference:LocalObjectReference{Name:csi-azuredisk-controller-config,},Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,} nil nil nil nil nil nil nil nil nil nil}}. No persistent volume exists." "disk.csi.azure.com/request-id"="1db2ea60-96a5-11ed-866b-002248a3db84"
I0117 20:25:43.231770       1 shared_state.go:464]  "msg"="Pod csi-azuredisk-controller-7f6f8d54b5-l6m44: Skipping Volume {kube-api-access-lg6t4 {nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil &ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,} nil nil nil nil nil}}. No persistent volume exists." "disk.csi.azure.com/request-id"="1db2ea60-96a5-11ed-866b-002248a3db84"
I0117 20:25:43.231786       1 shared_state.go:490]  "msg"="Storing pod csi-azuredisk-controller-7f6f8d54b5-l6m44 and claim [] to podToClaimsMap map." "disk.csi.azure.com/request-id"="1db2ea60-96a5-11ed-866b-002248a3db84"
I0117 20:25:43.231808       1 pod.go:91]  "msg"="Creating replicas for pod kube-system/csi-azuredisk-controller-7f6f8d54b5-l6m44." "disk.csi.azure.com/request-id"="1db2ea60-96a5-11ed-866b-002248a3db84" "disk.csi/azure.com/pod-name"="kube-system/csi-azuredisk-controller-7f6f8d54b5-l6m44"
... skipping 273 lines ...
I0117 20:25:43.378453       1 azuredisk_v2.go:435]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="1db2ea60-96a5-11ed-866b-002248a3db84" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).StartControllersAndDieOnExit.func1" "latency"=339554345
I0117 20:25:43.448460       1 azure_disk_utils.go:1048]  "msg"="Getting AzVolumeAttachment list for node (capz-ri10-zdjmx)" 
I0117 20:25:43.448733       1 azure_disk_utils.go:1048]  "msg"="Getting AzVolumeAttachment list for node (capz-ri10-8qjlr)" 
I0117 20:25:43.448986       1 azure_disk_utils.go:1048]  "msg"="Getting AzVolumeAttachment list for node (capz-ri10-ppsff)" 
I0117 20:25:43.449041       1 azure_disk_utils.go:1068]  "msg"="Label selector is: disk.csi.azure.com/node-name=capz-ri10-ppsff." 
I0117 20:25:43.449066       1 shared_state.go:1745] Added node(capz-ri10-ppsff) to availableAttachmentsMap with capacity: 8
I0117 20:25:43.449123       1 node.go:90] AzDiskControllerManager "msg"="Node is now available. Will requeue failed replica creation requests." "controller"="node" "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "namespace"="azure-disk-csi" "partition"="csi-azuredisk-controller"
I0117 20:25:43.449161       1 shared_state.go:1205]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="1df18607-96a5-11ed-866b-002248a3db84" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*SharedState).tryCreateFailedReplicas" "latency"=6700
I0117 20:25:43.448766       1 azure_disk_utils.go:1068]  "msg"="Label selector is: disk.csi.azure.com/node-name=capz-ri10-zdjmx." 
I0117 20:25:43.449344       1 shared_state.go:1745] Added node(capz-ri10-zdjmx) to availableAttachmentsMap with capacity: 8
I0117 20:25:43.449367       1 node.go:90] AzDiskControllerManager "msg"="Node is now available. Will requeue failed replica creation requests." "controller"="node" "disk.csi.azure.com/node-name"="capz-ri10-zdjmx" "namespace"="azure-disk-csi" "partition"="csi-azuredisk-controller"
I0117 20:25:43.449431       1 shared_state.go:1205]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="1df1908b-96a5-11ed-866b-002248a3db84" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*SharedState).tryCreateFailedReplicas" "latency"=37801
I0117 20:25:43.448881       1 azure_disk_utils.go:1048]  "msg"="Getting AzVolumeAttachment list for node (capz-ri100d-control-plane-8n9zj)" 
I0117 20:25:43.449570       1 azure_disk_utils.go:1068]  "msg"="Label selector is: disk.csi.azure.com/node-name=capz-ri10-8qjlr." 
I0117 20:25:43.450050       1 shared_state.go:1745] Added node(capz-ri10-8qjlr) to availableAttachmentsMap with capacity: 8
I0117 20:25:43.450386       1 node.go:90] AzDiskControllerManager "msg"="Node is now available. Will requeue failed replica creation requests." "controller"="node" "disk.csi.azure.com/node-name"="capz-ri10-8qjlr" "namespace"="azure-disk-csi" "partition"="csi-azuredisk-controller"
I0117 20:25:43.450749       1 shared_state.go:1205]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="1df1c3c1-96a5-11ed-866b-002248a3db84" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*SharedState).tryCreateFailedReplicas" "latency"=10300
I0117 20:25:43.449466       1 azure_disk_utils.go:1068]  "msg"="Label selector is: disk.csi.azure.com/node-name=capz-ri100d-control-plane-8n9zj." 
I0117 20:25:43.451268       1 shared_state.go:1745] Added node(capz-ri100d-control-plane-8n9zj) to availableAttachmentsMap with capacity: 4
I0117 20:25:43.451473       1 node.go:90] AzDiskControllerManager "msg"="Node is now available. Will requeue failed replica creation requests." "controller"="node" "disk.csi.azure.com/node-name"="capz-ri100d-control-plane-8n9zj" "namespace"="azure-disk-csi" "partition"="csi-azuredisk-controller"
I0117 20:25:43.451627       1 shared_state.go:1205]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="1df1e647-96a5-11ed-866b-002248a3db84" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*SharedState).tryCreateFailedReplicas" "latency"=7901
I0117 20:25:43.470572       1 shared_state.go:420]  "msg"="Adding pod calico-node-75sr7 to shared map with keyName calico-system/calico-node-75sr7." 
I0117 20:25:43.471548       1 shared_state.go:426]  "msg"="Pod spec of pod calico-node-75sr7 is: {Volumes:[{Name:lib-modules VolumeSource:{HostPath:&HostPathVolumeSource{Path:/lib/modules,Type:*,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:xtables-lock VolumeSource:{HostPath:&HostPathVolumeSource{Path:/run/xtables.lock,Type:*FileOrCreate,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:policysync VolumeSource:{HostPath:&HostPathVolumeSource{Path:/var/run/nodeagent,Type:*DirectoryOrCreate,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:tigera-ca-bundle VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:&ConfigMapVolumeSource{LocalObjectReference:LocalObjectReference{Name:tigera-ca-bundle,},Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,} VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:node-certs VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:&SecretVolumeSource{SecretName:node-certs,Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,} NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:var-run-calico VolumeSource:{HostPath:&HostPathVolumeSource{Path:/var/run/calico,Type:*,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:var-lib-calico VolumeSource:{HostPath:&HostPathVolumeSource{Path:/var/lib/calico,Type:*,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:cni-bin-dir VolumeSource:{HostPath:&HostPathVolumeSource{Path:/opt/cni/bin,Type:*,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:cni-net-dir VolumeSource:{HostPath:&HostPathVolumeSource{Path:/etc/cni/net.d,Type:*,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:cni-log-dir VolumeSource:{HostPath:&HostPathVolumeSource{Path:/var/log/calico/cni,Type:*,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:flexvol-driver-host VolumeSource:{HostPath:&HostPathVolumeSource{Path:/usr/libexec/kubernetes/kubelet-plugins/volume/exec/nodeagent~uds,Type:*DirectoryOrCreate,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:kube-api-access-zskkl VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:&ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,} PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[{Name:flexvol-driver Image:docker.io/calico/pod2daemon-flexvol:v3.25.0 Command:[] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[] Resources:{Limits:map[] Requests:map[] Claims:[]} VolumeMounts:[{Name:flexvol-driver-host ReadOnly:false MountPath:/host/driver SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-zskkl ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:IfNotPresent SecurityContext:&SecurityContext{Capabilities:nil,Privileged:*true,SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,ReadOnlyRootFilesystem:nil,AllowPrivilegeEscalation:nil,RunAsGroup:nil,ProcMount:nil,WindowsOptions:nil,SeccompProfile:nil,} Stdin:false StdinOnce:false TTY:false} {Name:install-cni Image:docker.io/calico/cni:v3.25.0 Command:[/opt/cni/bin/install] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:CNI_CONF_NAME Value:10-calico.conflist ValueFrom:nil} {Name:SLEEP Value:false ValueFrom:nil} {Name:CNI_NET_DIR Value:/etc/cni/net.d ValueFrom:nil} {Name:CNI_NETWORK_CONFIG Value: ValueFrom:&EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:&ConfigMapKeySelector{LocalObjectReference:LocalObjectReference{Name:cni-config,},Key:config,Optional:nil,},SecretKeyRef:nil,}} {Name:KUBERNETES_SERVICE_HOST Value:10.96.0.1 ValueFrom:nil} {Name:KUBERNETES_SERVICE_PORT Value:443 ValueFrom:nil}] Resources:{Limits:map[] Requests:map[] Claims:[]} VolumeMounts:[{Name:cni-bin-dir ReadOnly:false MountPath:/host/opt/cni/bin SubPath: MountPropagation:<nil> SubPathExpr:} {Name:cni-net-dir ReadOnly:false MountPath:/host/etc/cni/net.d SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-zskkl ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:IfNotPresent SecurityContext:&SecurityContext{Capabilities:nil,Privileged:*true,SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,ReadOnlyRootFilesystem:nil,AllowPrivilegeEscalation:nil,RunAsGroup:nil,ProcMount:nil,WindowsOptions:nil,SeccompProfile:nil,} Stdin:false StdinOnce:false TTY:false}] Containers:[{Name:calico-node Image:docker.io/calico/node:v3.25.0 Command:[] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:DATASTORE_TYPE Value:kubernetes ValueFrom:nil} {Name:WAIT_FOR_DATASTORE Value:true ValueFrom:nil} {Name:CLUSTER_TYPE Value:k8s,operator ValueFrom:nil} {Name:CALICO_DISABLE_FILE_LOGGING Value:false ValueFrom:nil} {Name:FELIX_DEFAULTENDPOINTTOHOSTACTION Value:ACCEPT ValueFrom:nil} {Name:FELIX_HEALTHENABLED Value:true ValueFrom:nil} {Name:FELIX_HEALTHPORT Value:9099 ValueFrom:nil} {Name:NODENAME Value: ValueFrom:&EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:spec.nodeName,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}} {Name:NAMESPACE Value: ValueFrom:&EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}} {Name:FELIX_TYPHAK8SNAMESPACE Value:calico-system ValueFrom:nil} {Name:FELIX_TYPHAK8SSERVICENAME Value:calico-typha ValueFrom:nil} {Name:FELIX_TYPHACAFILE Value:/etc/pki/tls/certs/tigera-ca-bundle.crt ValueFrom:nil} {Name:FELIX_TYPHACERTFILE Value:/node-certs/tls.crt ValueFrom:nil} {Name:FELIX_TYPHAKEYFILE Value:/node-certs/tls.key ValueFrom:nil} {Name:FIPS_MODE_ENABLED Value:false ValueFrom:nil} {Name:FELIX_TYPHACN Value:typha-server ValueFrom:nil} {Name:CALICO_MANAGE_CNI Value:true ValueFrom:nil} {Name:CALICO_IPV4POOL_CIDR Value:192.168.0.0/16 ValueFrom:nil} {Name:CALICO_IPV4POOL_VXLAN Value:Always ValueFrom:nil} {Name:CALICO_IPV4POOL_BLOCK_SIZE Value:26 ValueFrom:nil} {Name:CALICO_IPV4POOL_NODE_SELECTOR Value:all() ValueFrom:nil} {Name:CALICO_IPV4POOL_DISABLE_BGP_EXPORT Value:false ValueFrom:nil} {Name:FELIX_VXLANMTU Value:1350 ValueFrom:nil} {Name:FELIX_WIREGUARDMTU Value:1350 ValueFrom:nil} {Name:CALICO_NETWORKING_BACKEND Value:vxlan ValueFrom:nil} {Name:IP Value:autodetect ValueFrom:nil} {Name:IP_AUTODETECTION_METHOD Value:first-found ValueFrom:nil} {Name:IP6 Value:none ValueFrom:nil} {Name:FELIX_IPV6SUPPORT Value:false ValueFrom:nil} {Name:KUBERNETES_SERVICE_HOST Value:10.96.0.1 ValueFrom:nil} {Name:KUBERNETES_SERVICE_PORT Value:443 ValueFrom:nil}] Resources:{Limits:map[] Requests:map[] Claims:[]} VolumeMounts:[{Name:lib-modules ReadOnly:true MountPath:/lib/modules SubPath: MountPropagation:<nil> SubPathExpr:} {Name:xtables-lock ReadOnly:false MountPath:/run/xtables.lock SubPath: MountPropagation:<nil> SubPathExpr:} {Name:policysync ReadOnly:false MountPath:/var/run/nodeagent SubPath: MountPropagation:<nil> SubPathExpr:} {Name:tigera-ca-bundle ReadOnly:true MountPath:/etc/pki/tls/certs/ SubPath: MountPropagation:<nil> SubPathExpr:} {Name:node-certs ReadOnly:true MountPath:/node-certs SubPath: MountPropagation:<nil> SubPathExpr:} {Name:var-run-calico ReadOnly:false MountPath:/var/run/calico SubPath: MountPropagation:<nil> SubPathExpr:} {Name:var-lib-calico ReadOnly:false MountPath:/var/lib/calico SubPath: MountPropagation:<nil> SubPathExpr:} {Name:cni-log-dir ReadOnly:false MountPath:/var/log/calico/cni SubPath: MountPropagation:<nil> SubPathExpr:} {Name:cni-net-dir ReadOnly:false MountPath:/host/etc/cni/net.d SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-zskkl ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/liveness,Port:{0 9099 },Host:localhost,Scheme:HTTP,HTTPHeaders:[]HTTPHeader{},},TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:0,TimeoutSeconds:10,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:3,TerminationGracePeriodSeconds:nil,} ReadinessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[/bin/calico-node -felix-ready],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:0,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:3,TerminationGracePeriodSeconds:nil,} StartupProbe:nil Lifecycle:&Lifecycle{PostStart:nil,PreStop:&LifecycleHandler{Exec:&ExecAction{Command:[/bin/calico-node -shutdown],},HTTPGet:nil,TCPSocket:nil,},} TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:IfNotPresent SecurityContext:&SecurityContext{Capabilities:nil,Privileged:*true,SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,ReadOnlyRootFilesystem:nil,AllowPrivilegeEscalation:nil,RunAsGroup:nil,ProcMount:nil,WindowsOptions:nil,SeccompProfile:nil,} Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy:Always TerminationGracePeriodSeconds:0xc000844210 ActiveDeadlineSeconds:<nil> DNSPolicy:ClusterFirst NodeSelector:map[kubernetes.io/os:linux] ServiceAccountName:calico-node DeprecatedServiceAccount:calico-node AutomountServiceAccountToken:<nil> NodeName:capz-ri100d-control-plane-8n9zj HostNetwork:true HostPID:false HostIPC:false ShareProcessNamespace:<nil> SecurityContext:&PodSecurityContext{SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,SupplementalGroups:[],FSGroup:nil,RunAsGroup:nil,Sysctls:[]Sysctl{},WindowsOptions:nil,FSGroupChangePolicy:nil,SeccompProfile:nil,} ImagePullSecrets:[] Hostname: Subdomain: Affinity:&Affinity{NodeAffinity:&NodeAffinity{RequiredDuringSchedulingIgnoredDuringExecution:&NodeSelector{NodeSelectorTerms:[]NodeSelectorTerm{NodeSelectorTerm{MatchExpressions:[]NodeSelectorRequirement{},MatchFields:[]NodeSelectorRequirement{NodeSelectorRequirement{Key:metadata.name,Operator:In,Values:[capz-ri100d-control-plane-8n9zj],},},},},},PreferredDuringSchedulingIgnoredDuringExecution:[]PreferredSchedulingTerm{},},PodAffinity:nil,PodAntiAffinity:nil,} SchedulerName:default-scheduler Tolerations:[{Key:CriticalAddonsOnly Operator:Exists Value: Effect: TolerationSeconds:<nil>} {Key: Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>} {Key: Operator:Exists Value: Effect:NoExecute TolerationSeconds:<nil>} {Key:node.kubernetes.io/not-ready Operator:Exists Value: Effect:NoExecute TolerationSeconds:<nil>} {Key:node.kubernetes.io/unreachable Operator:Exists Value: Effect:NoExecute TolerationSeconds:<nil>} {Key:node.kubernetes.io/disk-pressure Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>} {Key:node.kubernetes.io/memory-pressure Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>} {Key:node.kubernetes.io/pid-pressure Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>} {Key:node.kubernetes.io/unschedulable Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>} {Key:node.kubernetes.io/network-unavailable Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>}] HostAliases:[] PriorityClassName:system-node-critical Priority:0xc000844218 DNSConfig:nil ReadinessGates:[] RuntimeClassName:<nil> EnableServiceLinks:0xc00084421c PreemptionPolicy:0xc000c52130 Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN:<nil> OS:nil HostUsers:<nil> SchedulingGates:[] ResourceClaims:[]}. With volumes: [{Name:lib-modules VolumeSource:{HostPath:&HostPathVolumeSource{Path:/lib/modules,Type:*,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:xtables-lock VolumeSource:{HostPath:&HostPathVolumeSource{Path:/run/xtables.lock,Type:*FileOrCreate,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:policysync VolumeSource:{HostPath:&HostPathVolumeSource{Path:/var/run/nodeagent,Type:*DirectoryOrCreate,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:tigera-ca-bundle VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:&ConfigMapVolumeSource{LocalObjectReference:LocalObjectReference{Name:tigera-ca-bundle,},Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,} VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:node-certs VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:&SecretVolumeSource{SecretName:node-certs,Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,} NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:var-run-calico VolumeSource:{HostPath:&HostPathVolumeSource{Path:/var/run/calico,Type:*,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:var-lib-calico VolumeSource:{HostPath:&HostPathVolumeSource{Path:/var/lib/calico,Type:*,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:cni-bin-dir VolumeSource:{HostPath:&HostPathVolumeSource{Path:/opt/cni/bin,Type:*,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:cni-net-dir VolumeSource:{HostPath:&HostPathVolumeSource{Path:/etc/cni/net.d,Type:*,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:cni-log-dir VolumeSource:{HostPath:&HostPathVolumeSource{Path:/var/log/calico/cni,Type:*,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:flexvol-driver-host VolumeSource:{HostPath:&HostPathVolumeSource{Path:/usr/libexec/kubernetes/kubelet-plugins/volume/exec/nodeagent~uds,Type:*DirectoryOrCreate,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:kube-api-access-zskkl VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:&ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,} PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}]" 
I0117 20:25:43.471716       1 shared_state.go:464]  "msg"="Pod calico-node-75sr7: Skipping Volume {lib-modules {&HostPathVolumeSource{Path:/lib/modules,Type:*,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}}. No persistent volume exists." 
I0117 20:25:43.471957       1 shared_state.go:464]  "msg"="Pod calico-node-75sr7: Skipping Volume {xtables-lock {&HostPathVolumeSource{Path:/run/xtables.lock,Type:*FileOrCreate,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}}. No persistent volume exists." 
I0117 20:25:43.472118       1 shared_state.go:464]  "msg"="Pod calico-node-75sr7: Skipping Volume {policysync {&HostPathVolumeSource{Path:/var/run/nodeagent,Type:*DirectoryOrCreate,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}}. No persistent volume exists." 
... skipping 328 lines ...
I0117 20:25:46.370756       1 shared_state.go:426]  "msg"="Pod spec of pod csi-azuredisk-node-win-tppwd is: {Volumes:[{Name:init-script VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:&ConfigMapVolumeSource{LocalObjectReference:LocalObjectReference{Name:csi-azuredisk-node-win-init,},Items:[]KeyToPath{KeyToPath{Key:Initialize-Pod.ps1,Path:Initialize-Pod.ps1,Mode:nil,},},DefaultMode:*420,Optional:nil,} VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:csi-azuredisk-node-win-config VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:&ConfigMapVolumeSource{LocalObjectReference:LocalObjectReference{Name:csi-azuredisk-node-win-config,},Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,} VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:kube-api-access-dbft9 VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:&ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,} PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[{Name:init Image:capzci.azurecr.io/azuredisk-csi:latest-v2-9ef068a8cb36a997d4ea04b90c05c6f92a488a19 Command:[powershell.exe -c .\\etc\\config\\Initialize-Pod.ps1] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[] Resources:{Limits:map[] Requests:map[] Claims:[]} VolumeMounts:[{Name:init-script ReadOnly:true MountPath:/etc/config SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-dbft9 ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:Always SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] Containers:[{Name:liveness-probe Image:mcr.microsoft.com/oss/kubernetes-csi/livenessprobe:v2.7.0 Command:[livenessprobe.exe] Args:[--csi-address=$(CSI_ENDPOINT) --probe-timeout=3s --health-port=29603 --v=2] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:CSI_ENDPOINT Value:unix://C:\\\\var\\\\lib\\\\kubelet\\\\plugins\\\\disk.csi.azure.com\\\\csi.sock ValueFrom:nil}] Resources:{Limits:map[] Requests:map[] Claims:[]} VolumeMounts:[{Name:kube-api-access-dbft9 ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:node-driver-registrar Image:mcr.microsoft.com/oss/kubernetes-csi/csi-node-driver-registrar:v2.5.1 Command:[csi-node-driver-registrar.exe] Args:[--v=2 --csi-address=$(CSI_ENDPOINT) --kubelet-registration-path=$(DRIVER_REG_SOCK_PATH) --plugin-registration-path=$(PLUGIN_REG_DIR)] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:CSI_ENDPOINT Value:unix://C:\\\\var\\\\lib\\\\kubelet\\\\plugins\\\\disk.csi.azure.com\\\\csi.sock ValueFrom:nil} {Name:DRIVER_REG_SOCK_PATH Value:C:\\\\var\\\\lib\\\\kubelet\\\\plugins\\\\disk.csi.azure.com\\\\csi.sock ValueFrom:nil} {Name:PLUGIN_REG_DIR Value:C:\\\\var\\\\lib\\\\kubelet\\\\plugins_registry\\\\ ValueFrom:nil} {Name:KUBE_NODE_NAME Value: ValueFrom:&EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:spec.nodeName,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}}] Resources:{Limits:map[] Requests:map[] Claims:[]} VolumeMounts:[{Name:kube-api-access-dbft9 ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[csi-node-driver-registrar.exe --kubelet-registration-path=$(DRIVER_REG_SOCK_PATH) --mode=kubelet-registration-probe],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:60,TimeoutSeconds:30,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:3,TerminationGracePeriodSeconds:nil,} ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:azuredisk Image:capzci.azurecr.io/azuredisk-csi:latest-v2-9ef068a8cb36a997d4ea04b90c05c6f92a488a19 Command:[azurediskplugin.exe] Args:[--v=6 --nodeid=$(KUBE_NODE_NAME) --config=.\\etc\\csi-azuredisk-node-win\\config.yaml] WorkingDir: Ports:[{Name:healthz HostPort:29603 ContainerPort:29603 Protocol:TCP HostIP:}] EnvFrom:[] Env:[{Name:AZURE_CREDENTIAL_FILE Value: ValueFrom:&EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:&ConfigMapKeySelector{LocalObjectReference:LocalObjectReference{Name:azure-cred-file,},Key:path-windows,Optional:*true,},SecretKeyRef:nil,}} {Name:CSI_ENDPOINT Value:unix://C:\\\\var\\\\lib\\\\kubelet\\\\plugins\\\\disk.csi.azure.com\\\\csi.sock ValueFrom:nil} {Name:KUBE_NODE_NAME Value: ValueFrom:&EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:spec.nodeName,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}} {Name:AZURE_GO_SDK_LOG_LEVEL Value: ValueFrom:nil}] Resources:{Limits:map[] Requests:map[] Claims:[]} VolumeMounts:[{Name:csi-azuredisk-node-win-config ReadOnly:false MountPath:/etc/csi-azuredisk-node-win SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-dbft9 ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/healthz,Port:{1 0 healthz},Host:,Scheme:HTTP,HTTPHeaders:[]HTTPHeader{},},TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:30,TimeoutSeconds:10,PeriodSeconds:30,SuccessThreshold:1,FailureThreshold:5,TerminationGracePeriodSeconds:nil,} ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:Always SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy:Always TerminationGracePeriodSeconds:0xc0012a6f10 ActiveDeadlineSeconds:<nil> DNSPolicy:ClusterFirst NodeSelector:map[kubernetes.io/os:windows] ServiceAccountName:csi-azuredisk-node-sa DeprecatedServiceAccount:csi-azuredisk-node-sa AutomountServiceAccountToken:<nil> NodeName:capz-ri10-zdjmx HostNetwork:true HostPID:false HostIPC:false ShareProcessNamespace:<nil> SecurityContext:&PodSecurityContext{SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,SupplementalGroups:[],FSGroup:nil,RunAsGroup:nil,Sysctls:[]Sysctl{},WindowsOptions:&WindowsSecurityContextOptions{GMSACredentialSpecName:nil,GMSACredentialSpec:nil,RunAsUserName:*NT AUTHORITY\\SYSTEM,HostProcess:*true,},FSGroupChangePolicy:nil,SeccompProfile:nil,} ImagePullSecrets:[] Hostname: Subdomain: Affinity:&Affinity{NodeAffinity:&NodeAffinity{RequiredDuringSchedulingIgnoredDuringExecution:&NodeSelector{NodeSelectorTerms:[]NodeSelectorTerm{NodeSelectorTerm{MatchExpressions:[]NodeSelectorRequirement{},MatchFields:[]NodeSelectorRequirement{NodeSelectorRequirement{Key:metadata.name,Operator:In,Values:[capz-ri10-zdjmx],},},},},},PreferredDuringSchedulingIgnoredDuringExecution:[]PreferredSchedulingTerm{},},PodAffinity:nil,PodAntiAffinity:nil,} SchedulerName:default-scheduler Tolerations:[{Key:node.kubernetes.io/os Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>} {Key:node.kubernetes.io/not-ready Operator:Exists Value: Effect:NoExecute TolerationSeconds:<nil>} {Key:node.kubernetes.io/unreachable Operator:Exists Value: Effect:NoExecute TolerationSeconds:<nil>} {Key:node.kubernetes.io/disk-pressure Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>} {Key:node.kubernetes.io/memory-pressure Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>} {Key:node.kubernetes.io/pid-pressure Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>} {Key:node.kubernetes.io/unschedulable Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>} {Key:node.kubernetes.io/network-unavailable Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>}] HostAliases:[] PriorityClassName:system-node-critical Priority:0xc0012a6f1c DNSConfig:nil ReadinessGates:[] RuntimeClassName:<nil> EnableServiceLinks:0xc0012a6f20 PreemptionPolicy:0xc0010b4de0 Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN:<nil> OS:nil HostUsers:<nil> SchedulingGates:[] ResourceClaims:[]}. With volumes: [{Name:init-script VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:&ConfigMapVolumeSource{LocalObjectReference:LocalObjectReference{Name:csi-azuredisk-node-win-init,},Items:[]KeyToPath{KeyToPath{Key:Initialize-Pod.ps1,Path:Initialize-Pod.ps1,Mode:nil,},},DefaultMode:*420,Optional:nil,} VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:csi-azuredisk-node-win-config VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:&ConfigMapVolumeSource{LocalObjectReference:LocalObjectReference{Name:csi-azuredisk-node-win-config,},Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,} VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:kube-api-access-dbft9 VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:&ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,} PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}]" 
I0117 20:25:46.370831       1 shared_state.go:464]  "msg"="Pod csi-azuredisk-node-win-tppwd: Skipping Volume {init-script {nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil &ConfigMapVolumeSource{LocalObjectReference:LocalObjectReference{Name:csi-azuredisk-node-win-init,},Items:[]KeyToPath{KeyToPath{Key:Initialize-Pod.ps1,Path:Initialize-Pod.ps1,Mode:nil,},},DefaultMode:*420,Optional:nil,} nil nil nil nil nil nil nil nil nil nil}}. No persistent volume exists." 
I0117 20:25:46.370852       1 shared_state.go:464]  "msg"="Pod csi-azuredisk-node-win-tppwd: Skipping Volume {csi-azuredisk-node-win-config {nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil &ConfigMapVolumeSource{LocalObjectReference:LocalObjectReference{Name:csi-azuredisk-node-win-config,},Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,} nil nil nil nil nil nil nil nil nil nil}}. No persistent volume exists." 
I0117 20:25:46.370919       1 shared_state.go:464]  "msg"="Pod csi-azuredisk-node-win-tppwd: Skipping Volume {kube-api-access-dbft9 {nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil &ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,} nil nil nil nil nil}}. No persistent volume exists." 
I0117 20:25:46.370970       1 shared_state.go:490]  "msg"="Storing pod csi-azuredisk-node-win-tppwd and claim [] to podToClaimsMap map." 
I0117 20:25:46.470988       1 shared_state.go:420]  "msg"="Adding pod csi-azuredisk-controller-7f6f8d54b5-l6m44 to shared map with keyName kube-system/csi-azuredisk-controller-7f6f8d54b5-l6m44." 
I0117 20:25:46.471728       1 shared_state.go:426]  "msg"="Pod spec of pod csi-azuredisk-controller-7f6f8d54b5-l6m44 is: {Volumes:[{Name:socket-dir VolumeSource:{HostPath:nil EmptyDir:&EmptyDirVolumeSource{Medium:,SizeLimit:<nil>,} GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:azure-cred VolumeSource:{HostPath:&HostPathVolumeSource{Path:/etc/kubernetes/,Type:*DirectoryOrCreate,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:csi-azuredisk-controller-config VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:&ConfigMapVolumeSource{LocalObjectReference:LocalObjectReference{Name:csi-azuredisk-controller-config,},Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,} VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:kube-api-access-lg6t4 VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:&ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,} PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:csi-provisioner Image:mcr.microsoft.com/oss/kubernetes-csi/csi-provisioner:v3.2.0 Command:[] Args:[--feature-gates=Topology=true --csi-address=$(ADDRESS) --v=2 --timeout=30s --leader-election --leader-election-namespace=kube-system --worker-threads=100 --extra-create-metadata=true --strict-topology=true --kube-api-qps=50 --kube-api-burst=100] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:ADDRESS Value:/csi/csi.sock ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:524288000 scale:0} d:{Dec:<nil>} s:500Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}] Claims:[]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-lg6t4 ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:csi-attacher Image:mcr.microsoft.com/oss/kubernetes-csi/csi-attacher:v3.5.0 Command:[] Args:[-v=2 -csi-address=$(ADDRESS) -timeout=600s -leader-election --leader-election-namespace=kube-system -worker-threads=500 -kube-api-qps=50 -kube-api-burst=100] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:ADDRESS Value:/csi/csi.sock ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:524288000 scale:0} d:{Dec:<nil>} s:500Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}] Claims:[]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-lg6t4 ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:csi-snapshotter Image:mcr.microsoft.com/oss/kubernetes-csi/csi-snapshotter:v5.0.1 Command:[] Args:[-csi-address=$(ADDRESS) -leader-election --leader-election-namespace=kube-system -v=2] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:ADDRESS Value:/csi/csi.sock ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:104857600 scale:0} d:{Dec:<nil>} s:100Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}] Claims:[]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-lg6t4 ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:csi-resizer Image:mcr.microsoft.com/oss/kubernetes-csi/csi-resizer:v1.5.0 Command:[] Args:[-csi-address=$(ADDRESS) -v=2 -leader-election --leader-election-namespace=kube-system -handle-volume-inuse-error=false -feature-gates=RecoverVolumeExpansionFailure=true -timeout=240s] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:ADDRESS Value:/csi/csi.sock ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:524288000 scale:0} d:{Dec:<nil>} s:500Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}] Claims:[]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-lg6t4 ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:liveness-probe Image:mcr.microsoft.com/oss/kubernetes-csi/livenessprobe:v2.7.0 Command:[] Args:[--csi-address=/csi/csi.sock --probe-timeout=3s --health-port=29602 --v=2] WorkingDir: Ports:[] EnvFrom:[] Env:[] Resources:{Limits:map[memory:{i:{value:104857600 scale:0} d:{Dec:<nil>} s:100Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}] Claims:[]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-lg6t4 ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:azuredisk Image:capzci.azurecr.io/azuredisk-csi:latest-v2-9ef068a8cb36a997d4ea04b90c05c6f92a488a19 Command:[] Args:[--v=6 --config=/etc/csi-azuredisk-controller/config.yaml] WorkingDir: Ports:[{Name:healthz HostPort:29602 ContainerPort:29602 Protocol:TCP HostIP:} {Name:metrics HostPort:29604 ContainerPort:29604 Protocol:TCP HostIP:}] EnvFrom:[] Env:[{Name:AZURE_CREDENTIAL_FILE Value: ValueFrom:&EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:&ConfigMapKeySelector{LocalObjectReference:LocalObjectReference{Name:azure-cred-file,},Key:path,Optional:*true,},SecretKeyRef:nil,}} {Name:CSI_ENDPOINT Value:unix:///csi/csi.sock ValueFrom:nil} {Name:AZURE_GO_SDK_LOG_LEVEL Value: ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:524288000 scale:0} d:{Dec:<nil>} s:500Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}] Claims:[]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:azure-cred ReadOnly:false MountPath:/etc/kubernetes/ SubPath: MountPropagation:<nil> SubPathExpr:} {Name:csi-azuredisk-controller-config ReadOnly:false MountPath:/etc/csi-azuredisk-controller SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-lg6t4 ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/healthz,Port:{1 0 healthz},Host:,Scheme:HTTP,HTTPHeaders:[]HTTPHeader{},},TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:30,TimeoutSeconds:10,PeriodSeconds:30,SuccessThreshold:1,FailureThreshold:5,TerminationGracePeriodSeconds:nil,} ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:Always SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy:Always TerminationGracePeriodSeconds:0xc001281b38 ActiveDeadlineSeconds:<nil> DNSPolicy:ClusterFirst NodeSelector:map[kubernetes.io/os:linux] ServiceAccountName:csi-azuredisk-controller-sa DeprecatedServiceAccount:csi-azuredisk-controller-sa AutomountServiceAccountToken:<nil> NodeName:capz-ri100d-control-plane-8n9zj HostNetwork:true HostPID:false HostIPC:false ShareProcessNamespace:<nil> SecurityContext:&PodSecurityContext{SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,SupplementalGroups:[],FSGroup:nil,RunAsGroup:nil,Sysctls:[]Sysctl{},WindowsOptions:nil,FSGroupChangePolicy:nil,SeccompProfile:nil,} ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName:default-scheduler Tolerations:[{Key:node-role.kubernetes.io/master Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>} {Key:node-role.kubernetes.io/controlplane Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>} {Key:node-role.kubernetes.io/control-plane Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>} {Key:node.kubernetes.io/not-ready Operator:Exists Value: Effect:NoExecute TolerationSeconds:0xc001281b40} {Key:node.kubernetes.io/unreachable Operator:Exists Value: Effect:NoExecute TolerationSeconds:0xc001281b48}] HostAliases:[] PriorityClassName:system-cluster-critical Priority:0xc001281b50 DNSConfig:nil ReadinessGates:[] RuntimeClassName:<nil> EnableServiceLinks:0xc001281b54 PreemptionPolicy:0xc00110ca20 Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN:<nil> OS:nil HostUsers:<nil> SchedulingGates:[] ResourceClaims:[]}. With volumes: [{Name:socket-dir VolumeSource:{HostPath:nil EmptyDir:&EmptyDirVolumeSource{Medium:,SizeLimit:<nil>,} GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:azure-cred VolumeSource:{HostPath:&HostPathVolumeSource{Path:/etc/kubernetes/,Type:*DirectoryOrCreate,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:csi-azuredisk-controller-config VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:&ConfigMapVolumeSource{LocalObjectReference:LocalObjectReference{Name:csi-azuredisk-controller-config,},Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,} VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:kube-api-access-lg6t4 VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:&ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,} PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}]" 
I0117 20:25:46.471831       1 shared_state.go:464]  "msg"="Pod csi-azuredisk-controller-7f6f8d54b5-l6m44: Skipping Volume {socket-dir {nil &EmptyDirVolumeSource{Medium:,SizeLimit:<nil>,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}}. No persistent volume exists." 
I0117 20:25:46.471875       1 shared_state.go:464]  "msg"="Pod csi-azuredisk-controller-7f6f8d54b5-l6m44: Skipping Volume {azure-cred {&HostPathVolumeSource{Path:/etc/kubernetes/,Type:*DirectoryOrCreate,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}}. No persistent volume exists." 
I0117 20:25:46.471924       1 shared_state.go:464]  "msg"="Pod csi-azuredisk-controller-7f6f8d54b5-l6m44: Skipping Volume {csi-azuredisk-controller-config {nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil &ConfigMapVolumeSource{LocalObjectReference:LocalObjectReference{Name:csi-azuredisk-controller-config,},Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,} nil nil nil nil nil nil nil nil nil nil}}. No persistent volume exists." 
I0117 20:25:46.471961       1 shared_state.go:464]  "msg"="Pod csi-azuredisk-controller-7f6f8d54b5-l6m44: Skipping Volume {kube-api-access-lg6t4 {nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil &ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,} nil nil nil nil nil}}. No persistent volume exists." 
I0117 20:25:46.472000       1 shared_state.go:490]  "msg"="Storing pod csi-azuredisk-controller-7f6f8d54b5-l6m44 and claim [] to podToClaimsMap map." 
I0117 20:25:46.472027       1 pod.go:91]  "msg"="Creating replicas for pod kube-system/csi-azuredisk-controller-7f6f8d54b5-l6m44." "disk.csi.azure.com/request-id"="1fbec6fe-96a5-11ed-866b-002248a3db84" "disk.csi/azure.com/pod-name"="kube-system/csi-azuredisk-controller-7f6f8d54b5-l6m44"
... skipping 416 lines ...
I0117 20:29:08.088555       1 utils.go:79] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.disk.csi.azure.com/zone":""}}],"requisite":[{"segments":{"topology.disk.csi.azure.com/zone":""}}]},"capacity_range":{"required_bytes":10737418240},"name":"pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0","parameters":{"csi.storage.k8s.io/pv/name":"pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0","csi.storage.k8s.io/pvc/name":"pvc-6tkv4","csi.storage.k8s.io/pvc/namespace":"azuredisk-1538","skuname":"StandardSSD_LRS"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]}
I0117 20:29:08.089584       1 crdprovisioner.go:308]  "msg"="Creating AzVolume CRI" "csi.storage.k8s.io/pv/name"="pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0" "disk.csi.azure.com/request-id"="97eb2a4e-96a5-11ed-866b-002248a3db84" "disk.csi.azure.com/volume-name"="pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0"
I0117 20:29:08.149878       1 round_trippers.go:553] POST https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes 201 Created in 59 milliseconds
I0117 20:29:08.150315       1 crdprovisioner.go:316]  "msg"="Successfully created AzVolume CRI" "csi.storage.k8s.io/pv/name"="pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0" "disk.csi.azure.com/request-id"="97eb2a4e-96a5-11ed-866b-002248a3db84" "disk.csi.azure.com/volume-name"="pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0"
I0117 20:29:08.150348       1 conditionwatcher.go:99] Adding a condition function for azvolume (pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0)
I0117 20:29:08.154184       1 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0/status 200 OK in 8 milliseconds
I0117 20:29:08.154975       1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0117 20:29:08.155402       1 azure_disk_utils.go:1103]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate" "csi.storage.k8s.io/pv/name"="pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0" "disk.csi.azure.com/request-id"="97eb2a4e-96a5-11ed-866b-002248a3db84" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0" "latency"=10263856 "number_of_net_retry"=0 "number_of_retry"=0
I0117 20:29:08.155508       1 azvolume.go:159]  "msg"="Creating Volume..." "csi.storage.k8s.io/pv/name"="pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0" "disk.csi.azure.com/request-id"="97eb2a4e-96a5-11ed-866b-002248a3db84" "disk.csi.azure.com/volume-name"="pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0"
I0117 20:29:08.195420       1 util.go:124] Send.sendRequest got response with ContentLength 247, StatusCode 404 and responseBody length 247
I0117 20:29:08.195541       1 azure_diskclient.go:139] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-ri100d/providers/Microsoft.Compute/disks/pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0' under resource group 'capz-ri100d' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I0117 20:29:08.195785       1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=0.03986759 request="disks_get" resource_group="capz-ri100d" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code="ResourceNotFound"
I0117 20:29:08.196300       1 cloudprovisioner.go:246] begin to create disk(pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0) account type(StandardSSD_LRS) rg(capz-ri100d) location() size(10) selectedAvailabilityZone() maxShares(0)
I0117 20:29:08.244961       1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 10927
I0117 20:29:08.245689       1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=0.049298959 request="disks_get" resource_group="capz-ri100d" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0117 20:29:08.246051       1 azure_managedDiskController.go:94] azureDisk - creating new managed Name:pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0 StorageAccountType:StandardSSD_LRS Size:10
I0117 20:29:08.943038       1 round_trippers.go:553] GET https://10.96.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 5 milliseconds
... skipping 6 lines ...
I0117 20:29:10.621388       1 cloudprovisioner.go:320]  "msg"="create disk(pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0) account type(StandardSSD_LRS) rg(capz-ri100d) location() size(10) tags(map[kubernetes.io-created-for-pv-name:pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0 kubernetes.io-created-for-pvc-name:pvc-6tkv4 kubernetes.io-created-for-pvc-namespace:azuredisk-1538]) successfully" "csi.storage.k8s.io/pv/name"="pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0" "disk.csi.azure.com/request-id"="97eb2a4e-96a5-11ed-866b-002248a3db84" "disk.csi.azure.com/volume-name"="pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0"
I0117 20:29:10.621436       1 cloudprovisioner.go:142]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0" "disk.csi.azure.com/request-id"="97eb2a4e-96a5-11ed-866b-002248a3db84" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0" "latency"=2465734731
I0117 20:29:10.644142       1 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0/status 200 OK in 22 milliseconds
I0117 20:29:10.645921       1 azure_disk_utils.go:1103]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "csi.storage.k8s.io/pv/name"="pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0" "disk.csi.azure.com/request-id"="97eb2a4e-96a5-11ed-866b-002248a3db84" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0" "latency"=24428480 "number_of_net_retry"=0 "number_of_retry"=0
I0117 20:29:10.646939       1 azvolume.go:166]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0" "disk.csi.azure.com/request-id"="97eb2a4e-96a5-11ed-866b-002248a3db84" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "disk.csi.azure.com/volume-name"="pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0" "latency"=2491058413
I0117 20:29:10.646974       1 workflow.go:155]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0" "disk.csi.azure.com/request-id"="97eb2a4e-96a5-11ed-866b-002248a3db84" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate" "disk.csi.azure.com/volume-name"="pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0" "latency"=2501856972
I0117 20:29:10.647622       1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 20:29:10.647679       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).CreateVolume" "csi.storage.k8s.io/pv/name"="pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0" "disk.csi.azure.com/request-id"="97eb2a4e-96a5-11ed-866b-002248a3db84" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0" "latency"=2497181645
I0117 20:29:10.647721       1 crdprovisioner.go:228]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0" "disk.csi.azure.com/request-id"="97eb2a4e-96a5-11ed-866b-002248a3db84" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0" "latency"=2558187981
I0117 20:29:10.647757       1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=2.558247782 request="azuredisk_csi_driver_controller_create_volume" resource_group="capz-ri100d" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-ri100d/providers/Microsoft.Compute/disks/pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0" result_code="succeeded"
I0117 20:29:10.647769       1 utils.go:85] GRPC response: {"volume":{"accessible_topology":[{"segments":{"topology.disk.csi.azure.com/zone":""}}],"capacity_bytes":10737418240,"content_source":{"Type":{"Volume":{}}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0","csi.storage.k8s.io/pvc/name":"pvc-6tkv4","csi.storage.k8s.io/pvc/namespace":"azuredisk-1538","requestedsizegib":"10","skuname":"StandardSSD_LRS"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-ri100d/providers/Microsoft.Compute/disks/pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0"}}
I0117 20:29:10.958930       1 round_trippers.go:553] GET https://10.96.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 1 milliseconds
I0117 20:29:10.964507       1 round_trippers.go:553] PUT https://10.96.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 4 milliseconds
... skipping 14 lines ...
I0117 20:29:12.966040       1 round_trippers.go:553] POST https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments 201 Created in 19 milliseconds
I0117 20:29:12.967418       1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0-capz-ri10-ppsff-attachment)
I0117 20:29:12.971985       1 round_trippers.go:553] GET https://10.96.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 6 milliseconds
I0117 20:29:12.973293       1 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0-capz-ri10-ppsff-attachment/status 200 OK in 14 milliseconds
I0117 20:29:12.973759       1 azure_disk_utils.go:1103]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach" "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="9ad0371e-96a5-11ed-866b-002248a3db84" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0" "latency"=15325050 "number_of_net_retry"=0 "number_of_retry"=0
I0117 20:29:12.973931       1 attach_detach.go:192]  "msg"="Attaching volume" "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="9ad0371e-96a5-11ed-866b-002248a3db84" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0"
I0117 20:29:12.976155       1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0117 20:29:12.983942       1 round_trippers.go:553] PUT https://10.96.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 9 milliseconds
I0117 20:29:12.984446       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0117 20:29:12.997083       1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1191
I0117 20:29:12.997484       1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=0.022902874 request="disks_get" resource_group="capz-ri100d" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0117 20:29:13.059687       1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 5895
I0117 20:29:13.060744       1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=0.062987706 request="vm_get" resource_group="capz-ri100d" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0117 20:29:13.061015       1 cloudprovisioner.go:405]  "msg"="GetDiskLun returned: -1. Initiating attaching volume \"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-ri100d/providers/Microsoft.Compute/disks/pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0\" to node \"capz-ri10-ppsff\"." "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="9ad0371e-96a5-11ed-866b-002248a3db84" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0"
I0117 20:29:13.061245       1 cloudprovisioner.go:424]  "msg"="Trying to attach volume \"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-ri100d/providers/Microsoft.Compute/disks/pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0\" to node \"capz-ri10-ppsff\"." "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="9ad0371e-96a5-11ed-866b-002248a3db84" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0"
I0117 20:29:14.061589       1 batch.go:224] "cloud-provider-azure: Delayed processing of batch due to start delay" type="batch" operation="attach_disk" key="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e|capz-ri100d|capz-ri10-ppsff" delay="1s"
I0117 20:29:14.061884       1 azure_controller_standard.go:103] azureDisk - update(capz-ri100d): vm(capz-ri10-ppsff) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-ri100d/providers/Microsoft.Compute/disks/pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0:AttachDiskOptions{diskName: "pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0", lun: 0}])
I0117 20:29:14.089602       1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 20:29:14.089727       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="9ad0371e-96a5-11ed-866b-002248a3db84" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0" "latency"=1122219378
I0117 20:29:14.090282       1 crdprovisioner.go:727]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="9ad0371e-96a5-11ed-866b-002248a3db84" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLun" "disk.csi.azure.com/volume-name"="pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0" "latency"=1122852880
I0117 20:29:14.090376       1 crdprovisioner.go:486]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="9ad0371e-96a5-11ed-866b-002248a3db84" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).PublishVolume" "disk.csi.azure.com/volume-name"="pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0" "latency"=1144264649
I0117 20:29:14.096677       1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=1.14440615 request="azuredisk_csi_driver_controller_publish_volume" resource_group="capz-ri100d" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-ri100d/providers/Microsoft.Compute/disks/pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0" node="capz-ri10-ppsff" result_code="succeeded"
I0117 20:29:14.096730       1 utils.go:85] GRPC response: {"publish_context":{"LUN":"0"}}
I0117 20:29:14.097122       1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0-capz-ri10-ppsff-attachment)
... skipping 25 lines ...
I0117 20:29:24.413082       1 azure_controller_standard.go:272] updateCache(capz-ri10-ppsff) successfully
I0117 20:29:24.413178       1 cloudprovisioner.go:448]  "msg"="attach operation successful: volume \"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-ri100d/providers/Microsoft.Compute/disks/pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0\" attached to node \"capz-ri10-ppsff\"." "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="9ad0371e-96a5-11ed-866b-002248a3db84" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0"
I0117 20:29:24.413236       1 cloudprovisioner.go:440]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="9ad0371e-96a5-11ed-866b-002248a3db84" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).PublishVolume.func3" "disk.csi.azure.com/volume-name"="pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0" "latency"=11351779263
I0117 20:29:24.413262       1 workflow.go:155]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="9ad0371e-96a5-11ed-866b-002248a3db84" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).PublishVolume" "disk.csi.azure.com/volume-name"="pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0" "latency"=11438731149
I0117 20:29:24.413360       1 azure_disk_utils.go:1165]  "msg"="Skip update. No update needed." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).updateVolumeAttachmentWithResult" "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="9ad0371e-96a5-11ed-866b-002248a3db84" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0"
I0117 20:29:24.413423       1 azure_disk_utils.go:1103]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).updateVolumeAttachmentWithResult" "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="9ad0371e-96a5-11ed-866b-002248a3db84" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0" "latency"=114001 "number_of_net_retry"=0 "number_of_retry"=0
I0117 20:29:24.427826       1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 20:29:24.428666       1 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0-capz-ri10-ppsff-attachment/status 200 OK in 15 milliseconds
I0117 20:29:24.429028       1 azure_disk_utils.go:1103]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3.4" "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="9ad0371e-96a5-11ed-866b-002248a3db84" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0" "latency"=15557895 "number_of_net_retry"=0 "number_of_retry"=0
I0117 20:29:24.429070       1 attach_detach.go:198]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="9ad0371e-96a5-11ed-866b-002248a3db84" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3" "disk.csi.azure.com/volume-name"="pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0" "latency"=11454553245
I0117 20:29:24.429093       1 workflow.go:155]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="9ad0371e-96a5-11ed-866b-002248a3db84" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach" "disk.csi.azure.com/volume-name"="pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0" "latency"=11470719597
I0117 20:29:24.429129       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="9b7fd8a0-96a5-11ed-866b-002248a3db84" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0" "latency"=10331843565
I0117 20:29:24.429159       1 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).ControllerPublishVolume.func2" "disk.csi.azure.com/request-id"="9b7fd8a0-96a5-11ed-866b-002248a3db84" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=10332066667
... skipping 108 lines ...
I0117 20:30:15.017158       1 utils.go:79] GRPC request: {"node_id":"capz-ri10-ppsff","volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-ri100d/providers/Microsoft.Compute/disks/pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0"}
I0117 20:30:15.017652       1 crdprovisioner.go:927]  "msg"="Requesting AzVolumeAttachment (pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0-capz-ri10-ppsff-attachment) detachment" "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="9ad0371e-96a5-11ed-866b-002248a3db84" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0"
I0117 20:30:15.031072       1 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0-capz-ri10-ppsff-attachment/status 200 OK in 12 milliseconds
I0117 20:30:15.031614       1 azure_disk_utils.go:1103]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).detachVolume" "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="9ad0371e-96a5-11ed-866b-002248a3db84" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0" "latency"=13899979 "number_of_net_retry"=0 "number_of_retry"=0
I0117 20:30:15.031991       1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0-capz-ri10-ppsff-attachment)
I0117 20:30:15.031934       1 replica.go:157]  "msg"="Garbage collection of AzVolumeAttachments for AzVolume (pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0) scheduled in 5m0s." "disk.csi.azure.com/request-id"="bfd1c387-96a5-11ed-866b-002248a3db84" "disk.csi.azure.com/volume-name"="pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0"
I0117 20:30:15.044504       1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0117 20:30:15.045852       1 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0-capz-ri10-ppsff-attachment/status 200 OK in 13 milliseconds
I0117 20:30:15.046112       1 azure_disk_utils.go:1103]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach" "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="9ad0371e-96a5-11ed-866b-002248a3db84" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0" "latency"=14652784 "number_of_net_retry"=0 "number_of_retry"=0
I0117 20:30:15.046144       1 attach_detach.go:340]  "msg"="Detaching volume" "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="9ad0371e-96a5-11ed-866b-002248a3db84" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0"
I0117 20:30:15.046211       1 cloudprovisioner.go:481]  "msg"="Trying to detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-ri100d/providers/Microsoft.Compute/disks/pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0 from node capz-ri10-ppsff" "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="9ad0371e-96a5-11ed-866b-002248a3db84" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0"
I0117 20:30:15.454554       1 round_trippers.go:553] GET https://10.96.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 1 milliseconds
I0117 20:30:15.467819       1 round_trippers.go:553] PUT https://10.96.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 12 milliseconds
... skipping 31 lines ...
I0117 20:30:31.429059       1 azure_controller_standard.go:126] DeleteCacheForNode(capz-ri10-ppsff) successfully
I0117 20:30:31.429065       1 azure_controller_standard.go:272] updateCache(capz-ri10-ppsff) successfully
I0117 20:30:31.429071       1 azure_controller_common.go:473] azuredisk - successfully detached disks from node capz-ri10-ppsff: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-ri100d/providers/Microsoft.Compute/disks/pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0:pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0]
I0117 20:30:31.429109       1 azure_controller_common.go:434] azureDisk - detach disk(pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0, /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-ri100d/providers/Microsoft.Compute/disks/pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0) succeeded
I0117 20:30:31.429158       1 cloudprovisioner.go:491]  "msg"="detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-ri100d/providers/Microsoft.Compute/disks/pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0 from node capz-ri10-ppsff successfully" "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="9ad0371e-96a5-11ed-866b-002248a3db84" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0"
I0117 20:30:31.429198       1 cloudprovisioner.go:471]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="9ad0371e-96a5-11ed-866b-002248a3db84" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).UnpublishVolume" "disk.csi.azure.com/volume-name"="pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0" "latency"=16382993855
I0117 20:30:31.447100       1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0117 20:30:31.447974       1 round_trippers.go:553] DELETE https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0-capz-ri10-ppsff-attachment 200 OK in 18 milliseconds
I0117 20:30:31.448093       1 attach_detach.go:346]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="9ad0371e-96a5-11ed-866b-002248a3db84" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach.func3" "disk.csi.azure.com/volume-name"="pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0" "latency"=16401909578
I0117 20:30:31.448120       1 workflow.go:155]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="9ad0371e-96a5-11ed-866b-002248a3db84" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach" "disk.csi.azure.com/volume-name"="pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0" "latency"=16416685762
I0117 20:30:31.459467       1 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0-capz-ri10-ppsff-attachment 200 OK in 11 milliseconds
I0117 20:30:31.460619       1 azure_disk_utils.go:1103]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).removeFinalizer" "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="9ad0371e-96a5-11ed-866b-002248a3db84" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0" "latency"=12948916 "number_of_net_retry"=0 "number_of_retry"=0
I0117 20:30:31.461602       1 attach_detach.go:379]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="9ad0371e-96a5-11ed-866b-002248a3db84" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).removeFinalizer" "disk.csi.azure.com/volume-name"="pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0" "latency"=13914617
I0117 20:30:31.461276       1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 20:30:31.464008       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForDetach" "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="9ad0371e-96a5-11ed-866b-002248a3db84" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0" "latency"=16431890678
I0117 20:30:31.465656       1 crdprovisioner.go:938]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).detachVolume" "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="9ad0371e-96a5-11ed-866b-002248a3db84" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForDetach" "disk.csi.azure.com/volume-name"="pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0" "latency"=16432557679
I0117 20:30:31.465700       1 crdprovisioner.go:836]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="bfcf9527-96a5-11ed-866b-002248a3db84" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).UnpublishVolume" "disk.csi.azure.com/volume-name"="pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0" "latency"=16448076563
I0117 20:30:31.465740       1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=16.448263363 request="azuredisk_csi_driver_controller_unpublish_volume" resource_group="capz-ri100d" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-ri100d/providers/Microsoft.Compute/disks/pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0" node="capz-ri10-ppsff" result_code="succeeded"
I0117 20:30:31.465754       1 utils.go:85] GRPC response: {}
I0117 20:30:31.486272       1 utils.go:78] GRPC call: /csi.v1.Controller/ControllerUnpublishVolume
... skipping 28 lines ...
I0117 20:30:42.552246       1 utils.go:79] GRPC request: {"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-ri100d/providers/Microsoft.Compute/disks/pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0"}
I0117 20:30:42.552355       1 controllerserver_v2.go:201] deleting disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-ri100d/providers/Microsoft.Compute/disks/pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0)
I0117 20:30:42.583889       1 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0/status 200 OK in 31 milliseconds
I0117 20:30:42.584383       1 azure_disk_utils.go:1103]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).DeleteVolume" "csi.storage.k8s.io/pv/name"="pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0" "disk.csi.azure.com/request-id"="d0391511-96a5-11ed-866b-002248a3db84" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0" "latency"=31776372 "number_of_net_retry"=0 "number_of_retry"=0
I0117 20:30:42.584407       1 conditionwatcher.go:99] Adding a condition function for azvolume (pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0)
I0117 20:30:42.596474       1 round_trippers.go:553] DELETE https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0 200 OK in 11 milliseconds
I0117 20:30:42.601959       1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0117 20:30:42.610348       1 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0/status 200 OK in 12 milliseconds
I0117 20:30:42.611272       1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0117 20:30:42.612155       1 azure_disk_utils.go:1103]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerDelete" "csi.storage.k8s.io/pv/name"="pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0" "disk.csi.azure.com/request-id"="97eb2a4e-96a5-11ed-866b-002248a3db84" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0" "latency"=15021781 "number_of_net_retry"=0 "number_of_retry"=0
I0117 20:30:42.612323       1 azvolume.go:247]  "msg"="Deleting Volume..." "csi.storage.k8s.io/pv/name"="pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0" "disk.csi.azure.com/request-id"="97eb2a4e-96a5-11ed-866b-002248a3db84" "disk.csi.azure.com/volume-name"="pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0"
I0117 20:30:42.612484       1 shared_state.go:1038]  "msg"="AzVolumeAttachment clean up requested by azvolume-controller for AzVolume (pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0)" "csi.storage.k8s.io/pv/name"="pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0" "disk.csi.azure.com/request-id"="97eb2a4e-96a5-11ed-866b-002248a3db84" "disk.csi.azure.com/volume-name"="pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0"
I0117 20:30:42.612512       1 azure_disk_utils.go:1039]  "msg"="Getting AzVolumeAttachment list for volume (pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0)" "csi.storage.k8s.io/pv/name"="pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0" "disk.csi.azure.com/request-id"="97eb2a4e-96a5-11ed-866b-002248a3db84" "disk.csi.azure.com/volume-name"="pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0"
I0117 20:30:42.612569       1 azure_disk_utils.go:1068]  "msg"="Label selector is: disk.csi.azure.com/volume-name=pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0." "csi.storage.k8s.io/pv/name"="pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0" "disk.csi.azure.com/request-id"="97eb2a4e-96a5-11ed-866b-002248a3db84" "disk.csi.azure.com/volume-name"="pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0"
I0117 20:30:42.612815       1 shared_state.go:1036]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0" "disk.csi.azure.com/request-id"="97eb2a4e-96a5-11ed-866b-002248a3db84" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*SharedState).cleanUpAzVolumeAttachmentByVolume" "disk.csi.azure.com/volume-name"="pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0" "latency"=344602
... skipping 8 lines ...
I0117 20:30:47.737917       1 round_trippers.go:553] GET https://10.96.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 5 milliseconds
I0117 20:30:47.743093       1 round_trippers.go:553] PUT https://10.96.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 4 milliseconds
I0117 20:30:47.743347       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0117 20:30:47.804232       1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=5.167374686 request="disks_delete" resource_group="capz-ri100d" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0117 20:30:47.804269       1 azure_managedDiskController.go:317] azureDisk - deleted a managed disk: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-ri100d/providers/Microsoft.Compute/disks/pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0
I0117 20:30:47.804349       1 cloudprovisioner.go:337]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0" "disk.csi.azure.com/request-id"="97eb2a4e-96a5-11ed-866b-002248a3db84" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).DeleteVolume" "disk.csi.azure.com/volume-name"="pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0" "latency"=5191400615
I0117 20:30:47.822533       1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 20:30:47.822606       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).DeleteVolume" "csi.storage.k8s.io/pv/name"="pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0" "disk.csi.azure.com/request-id"="d0391511-96a5-11ed-866b-002248a3db84" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0" "latency"=5225784518
I0117 20:30:47.822806       1 crdprovisioner.go:375]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0" "disk.csi.azure.com/request-id"="d0391511-96a5-11ed-866b-002248a3db84" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).DeleteVolume" "disk.csi.azure.com/volume-name"="pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0" "latency"=5270210159
I0117 20:30:47.822972       1 controllerserver_v2.go:203] delete disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-ri100d/providers/Microsoft.Compute/disks/pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0) returned with <nil>
I0117 20:30:47.823202       1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=5.270817762 request="azuredisk_csi_driver_controller_delete_volume" resource_group="capz-ri100d" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-ri100d/providers/Microsoft.Compute/disks/pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0" result_code="succeeded"
I0117 20:30:47.823351       1 utils.go:85] GRPC response: {}
I0117 20:30:47.823150       1 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0 200 OK in 18 milliseconds
... skipping 20 lines ...
I0117 20:30:54.119152       1 utils.go:78] GRPC call: /csi.v1.Controller/CreateVolume
I0117 20:30:54.119201       1 utils.go:79] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.disk.csi.azure.com/zone":""}}],"requisite":[{"segments":{"topology.disk.csi.azure.com/zone":""}}]},"capacity_range":{"required_bytes":10737418240},"name":"pvc-4ca42c14-b312-449b-b9b1-ebfcde77416c","parameters":{"csi.storage.k8s.io/pv/name":"pvc-4ca42c14-b312-449b-b9b1-ebfcde77416c","csi.storage.k8s.io/pvc/name":"test-volume-1-azuredisk-volume-tester-qnsd7-0","csi.storage.k8s.io/pvc/namespace":"azuredisk-3759","skuname":"StandardSSD_LRS"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]}
I0117 20:30:54.119486       1 crdprovisioner.go:308]  "msg"="Creating AzVolume CRI" "csi.storage.k8s.io/pv/name"="pvc-4ca42c14-b312-449b-b9b1-ebfcde77416c" "disk.csi.azure.com/request-id"="d71e0c80-96a5-11ed-866b-002248a3db84" "disk.csi.azure.com/volume-name"="pvc-4ca42c14-b312-449b-b9b1-ebfcde77416c"
I0117 20:30:54.160037       1 round_trippers.go:553] POST https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes 201 Created in 40 milliseconds
I0117 20:30:54.166523       1 crdprovisioner.go:316]  "msg"="Successfully created AzVolume CRI" "csi.storage.k8s.io/pv/name"="pvc-4ca42c14-b312-449b-b9b1-ebfcde77416c" "disk.csi.azure.com/request-id"="d71e0c80-96a5-11ed-866b-002248a3db84" "disk.csi.azure.com/volume-name"="pvc-4ca42c14-b312-449b-b9b1-ebfcde77416c"
I0117 20:30:54.166975       1 conditionwatcher.go:99] Adding a condition function for azvolume (pvc-4ca42c14-b312-449b-b9b1-ebfcde77416c)
I0117 20:30:54.175627       1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0117 20:30:54.175686       1 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-4ca42c14-b312-449b-b9b1-ebfcde77416c/status 200 OK in 13 milliseconds
I0117 20:30:54.176488       1 azure_disk_utils.go:1103]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate" "csi.storage.k8s.io/pv/name"="pvc-4ca42c14-b312-449b-b9b1-ebfcde77416c" "disk.csi.azure.com/request-id"="d71e0c80-96a5-11ed-866b-002248a3db84" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-4ca42c14-b312-449b-b9b1-ebfcde77416c" "latency"=14343895 "number_of_net_retry"=0 "number_of_retry"=0
I0117 20:30:54.176520       1 azvolume.go:159]  "msg"="Creating Volume..." "csi.storage.k8s.io/pv/name"="pvc-4ca42c14-b312-449b-b9b1-ebfcde77416c" "disk.csi.azure.com/request-id"="d71e0c80-96a5-11ed-866b-002248a3db84" "disk.csi.azure.com/volume-name"="pvc-4ca42c14-b312-449b-b9b1-ebfcde77416c"
I0117 20:30:54.197424       1 util.go:124] Send.sendRequest got response with ContentLength 247, StatusCode 404 and responseBody length 247
I0117 20:30:54.197754       1 azure_diskclient.go:139] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-ri100d/providers/Microsoft.Compute/disks/pvc-4ca42c14-b312-449b-b9b1-ebfcde77416c, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-4ca42c14-b312-449b-b9b1-ebfcde77416c' under resource group 'capz-ri100d' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I0117 20:30:54.198149       1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=0.02113334 request="disks_get" resource_group="capz-ri100d" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code="ResourceNotFound"
I0117 20:30:54.198347       1 cloudprovisioner.go:246] begin to create disk(pvc-4ca42c14-b312-449b-b9b1-ebfcde77416c) account type(StandardSSD_LRS) rg(capz-ri100d) location() size(10) selectedAvailabilityZone() maxShares(0)
I0117 20:30:54.249794       1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 10927
I0117 20:30:54.250314       1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=0.051079437 request="disks_get" resource_group="capz-ri100d" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0117 20:30:54.250378       1 azure_managedDiskController.go:94] azureDisk - creating new managed Name:pvc-4ca42c14-b312-449b-b9b1-ebfcde77416c StorageAccountType:StandardSSD_LRS Size:10
I0117 20:30:55.827187       1 round_trippers.go:553] GET https://10.96.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 8 milliseconds
... skipping 2 lines ...
I0117 20:30:56.477890       1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=2.2274614169999998 request="disks_create_or_update" resource_group="capz-ri100d" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0117 20:30:56.500965       1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1227
I0117 20:30:56.501257       1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=0.023265954 request="disks_get" resource_group="capz-ri100d" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0117 20:30:56.501302       1 azure_managedDiskController.go:280] azureDisk - created new MD Name:pvc-4ca42c14-b312-449b-b9b1-ebfcde77416c StorageAccountType:StandardSSD_LRS Size:10
I0117 20:30:56.501443       1 cloudprovisioner.go:320]  "msg"="create disk(pvc-4ca42c14-b312-449b-b9b1-ebfcde77416c) account type(StandardSSD_LRS) rg(capz-ri100d) location() size(10) tags(map[kubernetes.io-created-for-pv-name:pvc-4ca42c14-b312-449b-b9b1-ebfcde77416c kubernetes.io-created-for-pvc-name:test-volume-1-azuredisk-volume-tester-qnsd7-0 kubernetes.io-created-for-pvc-namespace:azuredisk-3759]) successfully" "csi.storage.k8s.io/pv/name"="pvc-4ca42c14-b312-449b-b9b1-ebfcde77416c" "disk.csi.azure.com/request-id"="d71e0c80-96a5-11ed-866b-002248a3db84" "disk.csi.azure.com/volume-name"="pvc-4ca42c14-b312-449b-b9b1-ebfcde77416c"
I0117 20:30:56.501500       1 cloudprovisioner.go:142]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-4ca42c14-b312-449b-b9b1-ebfcde77416c" "disk.csi.azure.com/request-id"="d71e0c80-96a5-11ed-866b-002248a3db84" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-4ca42c14-b312-449b-b9b1-ebfcde77416c" "latency"=2324674559
I0117 20:30:56.523494       1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 20:30:56.523787       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).CreateVolume" "csi.storage.k8s.io/pv/name"="pvc-4ca42c14-b312-449b-b9b1-ebfcde77416c" "disk.csi.azure.com/request-id"="d71e0c80-96a5-11ed-866b-002248a3db84" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-4ca42c14-b312-449b-b9b1-ebfcde77416c" "latency"=2356549470
I0117 20:30:56.524074       1 crdprovisioner.go:228]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-4ca42c14-b312-449b-b9b1-ebfcde77416c" "disk.csi.azure.com/request-id"="d71e0c80-96a5-11ed-866b-002248a3db84" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-4ca42c14-b312-449b-b9b1-ebfcde77416c" "latency"=2404573687
I0117 20:30:56.524342       1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=2.404867688 request="azuredisk_csi_driver_controller_create_volume" resource_group="capz-ri100d" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-ri100d/providers/Microsoft.Compute/disks/pvc-4ca42c14-b312-449b-b9b1-ebfcde77416c" result_code="succeeded"
I0117 20:30:56.524526       1 utils.go:85] GRPC response: {"volume":{"accessible_topology":[{"segments":{"topology.disk.csi.azure.com/zone":""}}],"capacity_bytes":10737418240,"content_source":{"Type":{"Volume":{}}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-4ca42c14-b312-449b-b9b1-ebfcde77416c","csi.storage.k8s.io/pvc/name":"test-volume-1-azuredisk-volume-tester-qnsd7-0","csi.storage.k8s.io/pvc/namespace":"azuredisk-3759","requestedsizegib":"10","skuname":"StandardSSD_LRS"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-ri100d/providers/Microsoft.Compute/disks/pvc-4ca42c14-b312-449b-b9b1-ebfcde77416c"}}
I0117 20:30:56.525402       1 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-4ca42c14-b312-449b-b9b1-ebfcde77416c/status 200 OK in 23 milliseconds
I0117 20:30:56.526672       1 azure_disk_utils.go:1103]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "csi.storage.k8s.io/pv/name"="pvc-4ca42c14-b312-449b-b9b1-ebfcde77416c" "disk.csi.azure.com/request-id"="d71e0c80-96a5-11ed-866b-002248a3db84" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-4ca42c14-b312-449b-b9b1-ebfcde77416c" "latency"=25064266 "number_of_net_retry"=0 "number_of_retry"=0
... skipping 4 lines ...
I0117 20:30:57.640210       1 azure_disk_utils.go:1068]  "msg"="Label selector is: disk.csi.azure.com/node-name=capz-ri10-ppsff,disk.csi.azure.com/volume-name!=pvc-4ca42c14-b312-449b-b9b1-ebfcde77416c." "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="d93741b6-96a5-11ed-866b-002248a3db84" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-4ca42c14-b312-449b-b9b1-ebfcde77416c"
I0117 20:30:57.651625       1 round_trippers.go:553] POST https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments 201 Created in 11 milliseconds
I0117 20:30:57.654160       1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-4ca42c14-b312-449b-b9b1-ebfcde77416c-capz-ri10-ppsff-attachment)
I0117 20:30:57.678711       1 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-4ca42c14-b312-449b-b9b1-ebfcde77416c-capz-ri10-ppsff-attachment/status 200 OK in 25 milliseconds
I0117 20:30:57.679240       1 azure_disk_utils.go:1103]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach" "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="d93741b6-96a5-11ed-866b-002248a3db84" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-4ca42c14-b312-449b-b9b1-ebfcde77416c" "latency"=25712071 "number_of_net_retry"=0 "number_of_retry"=0
I0117 20:30:57.679287       1 attach_detach.go:192]  "msg"="Attaching volume" "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="d93741b6-96a5-11ed-866b-002248a3db84" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-4ca42c14-b312-449b-b9b1-ebfcde77416c"
I0117 20:30:57.681653       1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0117 20:30:57.704935       1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1227
I0117 20:30:57.705145       1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=0.02577097 request="disks_get" resource_group="capz-ri100d" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0117 20:30:57.705230       1 cloudprovisioner.go:405]  "msg"="GetDiskLun returned: -1. Initiating attaching volume \"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-ri100d/providers/Microsoft.Compute/disks/pvc-4ca42c14-b312-449b-b9b1-ebfcde77416c\" to node \"capz-ri10-ppsff\"." "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="d93741b6-96a5-11ed-866b-002248a3db84" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-4ca42c14-b312-449b-b9b1-ebfcde77416c"
I0117 20:30:57.705269       1 cloudprovisioner.go:424]  "msg"="Trying to attach volume \"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-ri100d/providers/Microsoft.Compute/disks/pvc-4ca42c14-b312-449b-b9b1-ebfcde77416c\" to node \"capz-ri10-ppsff\"." "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="d93741b6-96a5-11ed-866b-002248a3db84" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-4ca42c14-b312-449b-b9b1-ebfcde77416c"
I0117 20:30:57.840085       1 round_trippers.go:553] GET https://10.96.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 3 milliseconds
I0117 20:30:57.851572       1 round_trippers.go:553] PUT https://10.96.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 11 milliseconds
I0117 20:30:57.851886       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0117 20:30:58.163075       1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: Watch close - *v1beta2.AzVolumeAttachment total 10 items received
I0117 20:30:58.164760       1 round_trippers.go:553] GET https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumeattachments?allowWatchBookmarks=true&resourceVersion=3933&timeout=5m4s&timeoutSeconds=304&watch=true 200 OK in 1 milliseconds
I0117 20:30:58.705451       1 batch.go:224] "cloud-provider-azure: Delayed processing of batch due to start delay" type="batch" operation="attach_disk" key="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e|capz-ri100d|capz-ri10-ppsff" delay="1s"
I0117 20:30:58.705754       1 azure_controller_standard.go:103] azureDisk - update(capz-ri100d): vm(capz-ri10-ppsff) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-ri100d/providers/Microsoft.Compute/disks/pvc-4ca42c14-b312-449b-b9b1-ebfcde77416c:AttachDiskOptions{diskName: "pvc-4ca42c14-b312-449b-b9b1-ebfcde77416c", lun: 0}])
I0117 20:30:58.716339       1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 20:30:58.716539       1 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-4ca42c14-b312-449b-b9b1-ebfcde77416c-capz-ri10-ppsff-attachment/status 200 OK in 10 milliseconds
I0117 20:30:58.716762       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="d93741b6-96a5-11ed-866b-002248a3db84" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-4ca42c14-b312-449b-b9b1-ebfcde77416c" "latency"=1062521846
I0117 20:30:58.716983       1 crdprovisioner.go:727]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="d93741b6-96a5-11ed-866b-002248a3db84" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLun" "disk.csi.azure.com/volume-name"="pvc-4ca42c14-b312-449b-b9b1-ebfcde77416c" "latency"=1062848949
I0117 20:30:58.717148       1 azure_disk_utils.go:1103]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3.4" "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="d93741b6-96a5-11ed-866b-002248a3db84" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-4ca42c14-b312-449b-b9b1-ebfcde77416c" "latency"=11405676 "number_of_net_retry"=0 "number_of_retry"=0
I0117 20:30:58.717153       1 crdprovisioner.go:486]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="d93741b6-96a5-11ed-866b-002248a3db84" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).PublishVolume" "disk.csi.azure.com/volume-name"="pvc-4ca42c14-b312-449b-b9b1-ebfcde77416c" "latency"=1077023243
I0117 20:30:58.717423       1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=1.077336245 request="azuredisk_csi_driver_controller_publish_volume" resource_group="capz-ri100d" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-ri100d/providers/Microsoft.Compute/disks/pvc-4ca42c14-b312-449b-b9b1-ebfcde77416c" node="capz-ri10-ppsff" result_code="succeeded"
... skipping 24 lines ...
I0117 20:31:09.942432       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0117 20:31:11.950266       1 round_trippers.go:553] GET https://10.96.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 7 milliseconds
I0117 20:31:11.961397       1 round_trippers.go:553] PUT https://10.96.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 8 milliseconds
I0117 20:31:11.961742       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0117 20:31:12.172734       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 20:31:12.172726       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 20:31:12.172928       1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0117 20:31:12.172747       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 20:31:12.226794       1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync
I0117 20:31:13.964567       1 round_trippers.go:553] GET https://10.96.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 2 milliseconds
I0117 20:31:13.974186       1 round_trippers.go:553] PUT https://10.96.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 9 milliseconds
I0117 20:31:13.974329       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0117 20:31:15.979221       1 round_trippers.go:553] GET https://10.96.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 4 milliseconds
... skipping 42 lines ...
I0117 20:31:40.170702       1 round_trippers.go:553] GET https://10.96.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 3 milliseconds
I0117 20:31:40.180959       1 round_trippers.go:553] PUT https://10.96.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 9 milliseconds
I0117 20:31:40.181116       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0117 20:31:42.173812       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 20:31:42.173826       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 20:31:42.173838       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 20:31:42.173946       1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0117 20:31:42.184669       1 round_trippers.go:553] GET https://10.96.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 2 milliseconds
I0117 20:31:42.194501       1 round_trippers.go:553] PUT https://10.96.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 9 milliseconds
I0117 20:31:42.194978       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0117 20:31:42.227454       1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync
I0117 20:31:44.201836       1 round_trippers.go:553] GET https://10.96.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 5 milliseconds
I0117 20:31:44.215248       1 round_trippers.go:553] PUT https://10.96.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 12 milliseconds
... skipping 3 lines ...
I0117 20:31:44.298913       1 azure_controller_standard.go:272] updateCache(capz-ri10-ppsff) successfully
I0117 20:31:44.299109       1 cloudprovisioner.go:448]  "msg"="attach operation successful: volume \"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-ri100d/providers/Microsoft.Compute/disks/pvc-4ca42c14-b312-449b-b9b1-ebfcde77416c\" attached to node \"capz-ri10-ppsff\"." "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="d93741b6-96a5-11ed-866b-002248a3db84" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-4ca42c14-b312-449b-b9b1-ebfcde77416c"
I0117 20:31:44.299245       1 cloudprovisioner.go:440]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="d93741b6-96a5-11ed-866b-002248a3db84" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).PublishVolume.func3" "disk.csi.azure.com/volume-name"="pvc-4ca42c14-b312-449b-b9b1-ebfcde77416c" "latency"=46593824635
I0117 20:31:44.299376       1 workflow.go:155]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="d93741b6-96a5-11ed-866b-002248a3db84" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).PublishVolume" "disk.csi.azure.com/volume-name"="pvc-4ca42c14-b312-449b-b9b1-ebfcde77416c" "latency"=46619935709
I0117 20:31:44.299616       1 azure_disk_utils.go:1165]  "msg"="Skip update. No update needed." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).updateVolumeAttachmentWithResult" "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="d93741b6-96a5-11ed-866b-002248a3db84" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-4ca42c14-b312-449b-b9b1-ebfcde77416c"
I0117 20:31:44.299667       1 azure_disk_utils.go:1103]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).updateVolumeAttachmentWithResult" "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="d93741b6-96a5-11ed-866b-002248a3db84" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-4ca42c14-b312-449b-b9b1-ebfcde77416c" "latency"=239002 "number_of_net_retry"=0 "number_of_retry"=0
I0117 20:31:44.313182       1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 20:31:44.313774       1 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-4ca42c14-b312-449b-b9b1-ebfcde77416c-capz-ri10-ppsff-attachment/status 200 OK in 13 milliseconds
I0117 20:31:44.314024       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="d9dbab09-96a5-11ed-866b-002248a3db84" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-4ca42c14-b312-449b-b9b1-ebfcde77416c" "latency"=45596312822
I0117 20:31:44.314220       1 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).ControllerPublishVolume.func2" "disk.csi.azure.com/request-id"="d9dbab09-96a5-11ed-866b-002248a3db84" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=45596583724
I0117 20:31:44.314291       1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=46.674182671 request="azuredisk_csi_driver_attach_volume_latency" resource_group="capz-ri100d" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-ri100d/providers/Microsoft.Compute/disks/pvc-4ca42c14-b312-449b-b9b1-ebfcde77416c" node="capz-ri10-ppsff" result_code="succeeded"
I0117 20:31:44.314384       1 controllerserver_v2.go:264]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="d9dbab09-96a5-11ed-866b-002248a3db84" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).ControllerPublishVolume.func2" "latency"=45596743325
I0117 20:31:44.314655       1 azure_disk_utils.go:1103]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3.4" "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="d93741b6-96a5-11ed-866b-002248a3db84" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-4ca42c14-b312-449b-b9b1-ebfcde77416c" "latency"=14811400 "number_of_net_retry"=0 "number_of_retry"=0
... skipping 101 lines ...
I0117 20:32:28.330917       1 azure_disk_utils.go:1103]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).detachVolume" "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="d93741b6-96a5-11ed-866b-002248a3db84" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-4ca42c14-b312-449b-b9b1-ebfcde77416c" "latency"=18343824 "number_of_net_retry"=0 "number_of_retry"=0
I0117 20:32:28.331236       1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-4ca42c14-b312-449b-b9b1-ebfcde77416c-capz-ri10-ppsff-attachment)
I0117 20:32:28.340983       1 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-4ca42c14-b312-449b-b9b1-ebfcde77416c-capz-ri10-ppsff-attachment/status 200 OK in 12 milliseconds
I0117 20:32:28.341794       1 azure_disk_utils.go:1103]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach" "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="d93741b6-96a5-11ed-866b-002248a3db84" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-4ca42c14-b312-449b-b9b1-ebfcde77416c" "latency"=13705593 "number_of_net_retry"=0 "number_of_retry"=0
I0117 20:32:28.342054       1 attach_detach.go:340]  "msg"="Detaching volume" "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="d93741b6-96a5-11ed-866b-002248a3db84" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-4ca42c14-b312-449b-b9b1-ebfcde77416c"
I0117 20:32:28.342781       1 cloudprovisioner.go:481]  "msg"="Trying to detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-ri100d/providers/Microsoft.Compute/disks/pvc-4ca42c14-b312-449b-b9b1-ebfcde77416c from node capz-ri10-ppsff" "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="d93741b6-96a5-11ed-866b-002248a3db84" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-4ca42c14-b312-449b-b9b1-ebfcde77416c"
I0117 20:32:28.350577       1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0117 20:32:28.537055       1 round_trippers.go:553] GET https://10.96.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 5 milliseconds
I0117 20:32:28.545341       1 round_trippers.go:553] PUT https://10.96.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 7 milliseconds
I0117 20:32:28.545797       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0117 20:32:29.343477       1 batch.go:224] "cloud-provider-azure: Delayed processing of batch due to start delay" type="batch" operation="detach_disk" key="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e|capz-ri100d|capz-ri10-ppsff" delay="1s"
I0117 20:32:29.343727       1 azure_controller_common.go:461] azuredisk - trying to detach disks from node capz-ri10-ppsff: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-ri100d/providers/Microsoft.Compute/disks/pvc-4ca42c14-b312-449b-b9b1-ebfcde77416c:pvc-4ca42c14-b312-449b-b9b1-ebfcde77416c]
I0117 20:32:29.343908       1 azure_controller_standard.go:180] azureDisk - detach disk: name pvc-4ca42c14-b312-449b-b9b1-ebfcde77416c uri /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-ri100d/providers/Microsoft.Compute/disks/pvc-4ca42c14-b312-449b-b9b1-ebfcde77416c
... skipping 15 lines ...
I0117 20:32:38.612809       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0117 20:32:40.616219       1 round_trippers.go:553] GET https://10.96.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 2 milliseconds
I0117 20:32:40.629272       1 round_trippers.go:553] PUT https://10.96.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 12 milliseconds
I0117 20:32:40.629431       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0117 20:32:42.175332       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 20:32:42.175343       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 20:32:42.175581       1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0117 20:32:42.175356       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 20:32:42.228655       1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync
I0117 20:32:42.631533       1 round_trippers.go:553] GET https://10.96.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 1 milliseconds
I0117 20:32:42.639288       1 round_trippers.go:553] PUT https://10.96.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 7 milliseconds
I0117 20:32:42.639655       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0117 20:32:44.641604       1 round_trippers.go:553] GET https://10.96.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 1 milliseconds
... skipping 7 lines ...
I0117 20:32:44.830343       1 azure_controller_common.go:434] azureDisk - detach disk(pvc-4ca42c14-b312-449b-b9b1-ebfcde77416c, /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-ri100d/providers/Microsoft.Compute/disks/pvc-4ca42c14-b312-449b-b9b1-ebfcde77416c) succeeded
I0117 20:32:44.830488       1 cloudprovisioner.go:491]  "msg"="detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-ri100d/providers/Microsoft.Compute/disks/pvc-4ca42c14-b312-449b-b9b1-ebfcde77416c from node capz-ri10-ppsff successfully" "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="d93741b6-96a5-11ed-866b-002248a3db84" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-4ca42c14-b312-449b-b9b1-ebfcde77416c"
I0117 20:32:44.830556       1 cloudprovisioner.go:471]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="d93741b6-96a5-11ed-866b-002248a3db84" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).UnpublishVolume" "disk.csi.azure.com/volume-name"="pvc-4ca42c14-b312-449b-b9b1-ebfcde77416c" "latency"=16487800462
I0117 20:32:44.843345       1 round_trippers.go:553] DELETE https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-4ca42c14-b312-449b-b9b1-ebfcde77416c-capz-ri10-ppsff-attachment 200 OK in 12 milliseconds
I0117 20:32:44.844785       1 attach_detach.go:346]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="d93741b6-96a5-11ed-866b-002248a3db84" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach.func3" "disk.csi.azure.com/volume-name"="pvc-4ca42c14-b312-449b-b9b1-ebfcde77416c" "latency"=16502100162
I0117 20:32:44.844994       1 workflow.go:155]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="d93741b6-96a5-11ed-866b-002248a3db84" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach" "disk.csi.azure.com/volume-name"="pvc-4ca42c14-b312-449b-b9b1-ebfcde77416c" "latency"=16516916563
I0117 20:32:44.847006       1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0117 20:32:44.857071       1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 20:32:44.859512       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForDetach" "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="d93741b6-96a5-11ed-866b-002248a3db84" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-4ca42c14-b312-449b-b9b1-ebfcde77416c" "latency"=16528121244
I0117 20:32:44.859564       1 crdprovisioner.go:938]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).detachVolume" "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="d93741b6-96a5-11ed-866b-002248a3db84" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForDetach" "disk.csi.azure.com/volume-name"="pvc-4ca42c14-b312-449b-b9b1-ebfcde77416c" "latency"=16528341645
I0117 20:32:44.859654       1 crdprovisioner.go:836]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="0f42c371-96a6-11ed-866b-002248a3db84" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).UnpublishVolume" "disk.csi.azure.com/volume-name"="pvc-4ca42c14-b312-449b-b9b1-ebfcde77416c" "latency"=16547153072
I0117 20:32:44.859728       1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=16.547339374 request="azuredisk_csi_driver_controller_unpublish_volume" resource_group="capz-ri100d" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-ri100d/providers/Microsoft.Compute/disks/pvc-4ca42c14-b312-449b-b9b1-ebfcde77416c" node="capz-ri10-ppsff" result_code="succeeded"
I0117 20:32:44.859749       1 utils.go:85] GRPC response: {}
I0117 20:32:44.860000       1 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-4ca42c14-b312-449b-b9b1-ebfcde77416c-capz-ri10-ppsff-attachment 200 OK in 12 milliseconds
... skipping 23256 lines ...
I0117 20:25:52.204466    2284 utils.go:85] GRPC response: {"name":"disk.csi.azure.com","vendor_version":"latest-v2-9ef068a8cb36a997d4ea04b90c05c6f92a488a19"}
I0117 20:25:52.628625    2284 utils.go:78] GRPC call: /csi.v1.Identity/GetPluginInfo
I0117 20:25:52.628625    2284 utils.go:79] GRPC request: {}
I0117 20:25:52.628625    2284 utils.go:85] GRPC response: {"name":"disk.csi.azure.com","vendor_version":"latest-v2-9ef068a8cb36a997d4ea04b90c05c6f92a488a19"}
I0117 20:25:52.834467    2284 utils.go:78] GRPC call: /csi.v1.Node/NodeGetInfo
I0117 20:25:52.834974    2284 utils.go:79] GRPC request: {}
I0117 20:25:52.855963    2284 azure_instance_metadata.go:140] Warning: failed to get loadbalancer metadata: failure of getting loadbalancer metadata with response "404 Not Found"
I0117 20:25:52.856470    2284 nodeserver_v2.go:438] NodeGetInfo: nodeName(capz-ri10-8qjlr), VM Size(Standard_D4s_v3)
I0117 20:25:52.856470    2284 nodeserver.go:409] got a matching size in getMaxDataDiskCount, VM Size: STANDARD_D4S_V3, MaxDataDiskCount: 8
I0117 20:25:52.856470    2284 utils.go:85] GRPC response: {"accessible_topology":{"segments":{"topology.disk.csi.azure.com/zone":""}},"max_volumes_per_node":8,"node_id":"capz-ri10-8qjlr"}
I0117 20:26:21.602195    2284 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 20:26:21.602195    2284 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 20:26:21.602195    2284 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
... skipping 1261 lines ...
I0117 21:19:27.011673    2284 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0117 21:19:27.011673    2284 utils.go:79] GRPC request: {}
I0117 21:19:27.011673    2284 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I0117 21:19:27.013167    2284 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0117 21:19:27.013167    2284 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-fd03070e-1579-4566-b3e1-e83b180aa1a5\\globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-fd03070e-1579-4566-b3e1-e83b180aa1a5","csi.storage.k8s.io/pvc/name":"pvc-kr8vh","csi.storage.k8s.io/pvc/namespace":"azuredisk-8041","requestedsizegib":"10","skuname":"StandardSSD_ZRS","storage.kubernetes.io/csiProvisionerIdentity":"1673987143207-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-ri100d/providers/Microsoft.Compute/disks/pvc-fd03070e-1579-4566-b3e1-e83b180aa1a5"}
I0117 21:19:27.013683    2284 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-fd03070e-1579-4566-b3e1-e83b180aa1a5-capz-ri10-8qjlr-attachment)
I0117 21:19:28.750316    2284 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 21:19:28.750316    2284 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-ri10-8qjlr" "disk.csi.azure.com/request-id"="9f566ffe-96ac-11ed-ad40-000d3ab39166" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-fd03070e-1579-4566-b3e1-e83b180aa1a5" "latency"=1736131700
I0117 21:19:28.750316    2284 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="9f566ffe-96ac-11ed-ad40-000d3ab39166" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=1736616000
I0117 21:19:35.974401    2284 nodeserver_v2.go:211] NodeStageVolume: formatting 10 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-fd03070e-1579-4566-b3e1-e83b180aa1a5\globalmount with mount options([])
I0117 21:19:47.176626    2284 nodeserver_v2.go:215] NodeStageVolume: format 10 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-fd03070e-1579-4566-b3e1-e83b180aa1a5\globalmount successfully.
I0117 21:19:47.176626    2284 utils.go:85] GRPC response: {}
I0117 21:19:47.185123    2284 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
... skipping 228 lines ...
I0117 21:27:16.871105    2284 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0117 21:27:16.871606    2284 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-f1b5c7d6-1ac5-4033-8324-c1a8964c94fe\\globalmount","volume_capability":{"AccessType":{"Mount":{"fs_type":"btrfs"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-f1b5c7d6-1ac5-4033-8324-c1a8964c94fe","csi.storage.k8s.io/pvc/name":"test-volume-1-azuredisk-volume-tester-dfxwg-0","csi.storage.k8s.io/pvc/namespace":"azuredisk-1032","fsType":"btrfs","requestedsizegib":"10","skuname":"StandardSSD_ZRS","storage.kubernetes.io/csiProvisionerIdentity":"1673987143207-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-ri100d/providers/Microsoft.Compute/disks/pvc-f1b5c7d6-1ac5-4033-8324-c1a8964c94fe"}
I0117 21:27:16.871606    2284 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-f1b5c7d6-1ac5-4033-8324-c1a8964c94fe-capz-ri10-8qjlr-attachment)
I0117 21:27:18.611461    2284 azuredisk_v2.go:578]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-17T21:27:18Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-ri10-8qjlr" "disk.csi.azure.com/request-id"="b86e8c3a-96ad-11ed-ad40-000d3ab39166"
I0117 21:27:18.620949    2284 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-ri10-8qjlr/status 200 OK in 9 milliseconds
I0117 21:27:18.620986    2284 azuredisk_v2.go:565]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-ri10-8qjlr" "disk.csi.azure.com/request-id"="b86e8c3a-96ad-11ed-ad40-000d3ab39166" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=9524900
I0117 21:27:19.649653    2284 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 21:27:19.649653    2284 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-ri10-8qjlr" "disk.csi.azure.com/request-id"="b7651120-96ad-11ed-ad40-000d3ab39166" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-f1b5c7d6-1ac5-4033-8324-c1a8964c94fe" "latency"=2778015000
I0117 21:27:19.649653    2284 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="b7651120-96ad-11ed-ad40-000d3ab39166" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=2778015000
I0117 21:27:22.500756    2284 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 21:27:22.500756    2284 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 21:27:22.500756    2284 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 21:27:22.686760    2284 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync
... skipping 38 lines ...
I0117 21:27:54.908074    2284 utils.go:78] GRPC call: /csi.v1.Node/NodeUnstageVolume
I0117 21:27:54.908074    2284 utils.go:79] GRPC request: {"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-f1b5c7d6-1ac5-4033-8324-c1a8964c94fe\\globalmount","volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-ri100d/providers/Microsoft.Compute/disks/pvc-f1b5c7d6-1ac5-4033-8324-c1a8964c94fe"}
I0117 21:27:54.908074    2284 nodeserver_v2.go:257] NodeUnstageVolume: unmounting \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-f1b5c7d6-1ac5-4033-8324-c1a8964c94fe\globalmount
I0117 21:27:57.222144    2284 nodeserver_v2.go:262] NodeUnstageVolume: unmount \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-f1b5c7d6-1ac5-4033-8324-c1a8964c94fe\globalmount successfully
I0117 21:27:57.222144    2284 utils.go:85] GRPC response: {}
I0117 21:28:07.608020    2284 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-ri10-8qjlr/status  in 18981 milliseconds
W0117 21:28:07.608617    2284 reflector.go:347] k8s.io/client-go/informers/factory.go:150: watch of *v1.Node ended with: an error on the server ("unable to decode an event from the watch stream: read tcp 10.1.0.6:34556->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.") has prevented the request from succeeding
W0117 21:28:07.608617    2284 reflector.go:347] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: watch of *v1beta2.AzDriverNode ended with: an error on the server ("unable to decode an event from the watch stream: read tcp 10.1.0.6:34556->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.") has prevented the request from succeeding
W0117 21:28:07.609128    2284 reflector.go:347] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: watch of *v1beta2.AzDriverNode ended with: an error on the server ("unable to decode an event from the watch stream: read tcp 10.1.0.6:34556->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.") has prevented the request from succeeding
W0117 21:28:07.609128    2284 reflector.go:347] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: watch of *v1beta2.AzVolumeAttachment ended with: an error on the server ("unable to decode an event from the watch stream: read tcp 10.1.0.6:34556->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.") has prevented the request from succeeding
E0117 21:28:07.609128    2284 azuredisk_v2.go:613]  "msg"="Failed to update AzDriverNode status after creation" "error"="Put \"https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-ri10-8qjlr/status\": read tcp 10.1.0.6:34556->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-ri10-8qjlr" "disk.csi.azure.com/request-id"="ca524b97-96ad-11ed-ad40-000d3ab39166"
E0117 21:28:07.609128    2284 azuredisk_v2.go:565]  "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [Put \"https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-ri10-8qjlr/status\": read tcp 10.1.0.6:34556->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.]" "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-ri10-8qjlr" "disk.csi.azure.com/request-id"="ca524b97-96ad-11ed-ad40-000d3ab39166" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=18983475000
W0117 21:28:07.609128    2284 reflector.go:347] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: watch of *v1beta2.AzVolume ended with: an error on the server ("unable to decode an event from the watch stream: read tcp 10.1.0.6:34556->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.") has prevented the request from succeeding
W0117 21:28:07.609128    2284 reflector.go:347] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: watch of *v1.CustomResourceDefinition ended with: an error on the server ("unable to decode an event from the watch stream: read tcp 10.1.0.6:34556->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.") has prevented the request from succeeding
I0117 21:28:08.712053    2284 reflector.go:257] Listing and watching *v1.CustomResourceDefinition from k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132
I0117 21:28:08.712053    2284 reflector.go:257] Listing and watching *v1beta2.AzVolume from sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132
I0117 21:28:08.726034    2284 round_trippers.go:553] GET https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumes?resourceVersion=21452 200 OK in 13 milliseconds
I0117 21:28:08.730530    2284 round_trippers.go:553] GET https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumes?allowWatchBookmarks=true&resourceVersion=21452&timeout=9m42s&timeoutSeconds=582&watch=true 200 OK in 1 milliseconds
I0117 21:28:08.743244    2284 round_trippers.go:553] GET https://10.96.0.1:443/apis/apiextensions.k8s.io/v1/customresourcedefinitions?resourceVersion=20379 200 OK in 30 milliseconds
I0117 21:28:08.775529    2284 round_trippers.go:553] GET https://10.96.0.1:443/apis/apiextensions.k8s.io/v1/customresourcedefinitions?allowWatchBookmarks=true&resourceVersion=20379&timeout=9m25s&timeoutSeconds=565&watch=true 200 OK in 2 milliseconds
... skipping 68 lines ...
I0117 21:30:23.205666    2284 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0117 21:30:23.205666    2284 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-f1b5c7d6-1ac5-4033-8324-c1a8964c94fe\\globalmount","volume_capability":{"AccessType":{"Mount":{"fs_type":"btrfs"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-f1b5c7d6-1ac5-4033-8324-c1a8964c94fe","csi.storage.k8s.io/pvc/name":"test-volume-1-azuredisk-volume-tester-dfxwg-0","csi.storage.k8s.io/pvc/namespace":"azuredisk-1032","fsType":"btrfs","requestedsizegib":"10","skuname":"StandardSSD_ZRS","storage.kubernetes.io/csiProvisionerIdentity":"1673987143207-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-ri100d/providers/Microsoft.Compute/disks/pvc-f1b5c7d6-1ac5-4033-8324-c1a8964c94fe"}
I0117 21:30:23.206163    2284 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-f1b5c7d6-1ac5-4033-8324-c1a8964c94fe-capz-ri10-8qjlr-attachment)
I0117 21:30:23.762867    2284 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I0117 21:30:23.762867    2284 utils.go:79] GRPC request: {}
I0117 21:30:23.762867    2284 utils.go:85] GRPC response: {"ready":{"value":true}}
I0117 21:30:32.939409    2284 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 21:30:32.939873    2284 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-ri10-8qjlr" "disk.csi.azure.com/request-id"="267576c7-96ae-11ed-ad40-000d3ab39166" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-f1b5c7d6-1ac5-4033-8324-c1a8964c94fe" "latency"=9733606500
I0117 21:30:32.939873    2284 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="267576c7-96ae-11ed-ad40-000d3ab39166" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9733606500
I0117 21:30:37.125952    2284 nodeserver_v2.go:211] NodeStageVolume: formatting 10 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-f1b5c7d6-1ac5-4033-8324-c1a8964c94fe\globalmount with mount options([])
I0117 21:30:38.767033    2284 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 21:30:38.829041    2284 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync
I0117 21:30:38.829041    2284 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
... skipping 79 lines ...
I0117 21:32:03.588965    2284 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0117 21:32:03.588965    2284 utils.go:79] GRPC request: {}
I0117 21:32:03.588965    2284 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I0117 21:32:03.590450    2284 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0117 21:32:03.590450    2284 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-3e460b08-badd-420b-b852-a83462fa6d6e\\globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-3e460b08-badd-420b-b852-a83462fa6d6e","csi.storage.k8s.io/pvc/name":"test-volume-1-azuredisk-volume-tester-gp4pv-0","csi.storage.k8s.io/pvc/namespace":"azuredisk-5764","requestedsizegib":"10","skuname":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1673987143207-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-ri100d/providers/Microsoft.Compute/disks/pvc-3e460b08-badd-420b-b852-a83462fa6d6e"}
I0117 21:32:03.590450    2284 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-3e460b08-badd-420b-b852-a83462fa6d6e-capz-ri10-8qjlr-attachment)
I0117 21:32:06.687160    2284 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 21:32:06.687160    2284 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-ri10-8qjlr" "disk.csi.azure.com/request-id"="624ae3fd-96ae-11ed-ad40-000d3ab39166" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-3e460b08-badd-420b-b852-a83462fa6d6e" "latency"=3096674100
I0117 21:32:06.687160    2284 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="624ae3fd-96ae-11ed-ad40-000d3ab39166" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=3096674100
I0117 21:32:08.786795    2284 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 21:32:08.856361    2284 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync
I0117 21:32:08.856688    2284 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 21:32:09.127795    2284 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
... skipping 201 lines ...
I0117 21:38:15.956543    2284 nodeserver_v2.go:262] NodeUnstageVolume: unmount \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-3e460b08-badd-420b-b852-a83462fa6d6e\globalmount successfully
I0117 21:38:15.956543    2284 utils.go:85] GRPC response: {}
I0117 21:38:18.617683    2284 azuredisk_v2.go:578]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-17T21:38:18Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-ri10-8qjlr" "disk.csi.azure.com/request-id"="41d379ab-96af-11ed-ad40-000d3ab39166"
I0117 21:38:23.770047    2284 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I0117 21:38:23.770047    2284 utils.go:79] GRPC request: {}
I0117 21:38:23.770047    2284 utils.go:85] GRPC response: {"ready":{"value":true}}
W0117 21:38:37.607639    2284 reflector.go:347] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: watch of *v1beta2.AzDriverNode ended with: an error on the server ("unable to decode an event from the watch stream: read tcp 10.1.0.6:35354->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.") has prevented the request from succeeding
W0117 21:38:37.607639    2284 reflector.go:347] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: watch of *v1beta2.AzVolume ended with: an error on the server ("unable to decode an event from the watch stream: read tcp 10.1.0.6:35354->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.") has prevented the request from succeeding
W0117 21:38:37.607639    2284 reflector.go:347] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: watch of *v1.CustomResourceDefinition ended with: an error on the server ("unable to decode an event from the watch stream: read tcp 10.1.0.6:35354->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.") has prevented the request from succeeding
I0117 21:38:37.607639    2284 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-ri10-8qjlr/status  in 18989 milliseconds
W0117 21:38:37.607639    2284 reflector.go:347] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: watch of *v1beta2.AzDriverNode ended with: an error on the server ("unable to decode an event from the watch stream: read tcp 10.1.0.6:35354->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.") has prevented the request from succeeding
W0117 21:38:37.607639    2284 reflector.go:347] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: watch of *v1beta2.AzVolumeAttachment ended with: an error on the server ("unable to decode an event from the watch stream: read tcp 10.1.0.6:35354->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.") has prevented the request from succeeding
W0117 21:38:37.607639    2284 reflector.go:347] k8s.io/client-go/informers/factory.go:150: watch of *v1.Node ended with: an error on the server ("unable to decode an event from the watch stream: read tcp 10.1.0.6:35354->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.") has prevented the request from succeeding
E0117 21:38:37.607639    2284 azuredisk_v2.go:613]  "msg"="Failed to update AzDriverNode status after creation" "error"="Put \"https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-ri10-8qjlr/status\": read tcp 10.1.0.6:35354->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-ri10-8qjlr" "disk.csi.azure.com/request-id"="41d379ab-96af-11ed-ad40-000d3ab39166"
E0117 21:38:37.607639    2284 azuredisk_v2.go:565]  "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [Put \"https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-ri10-8qjlr/status\": read tcp 10.1.0.6:35354->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.]" "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-ri10-8qjlr" "disk.csi.azure.com/request-id"="41d379ab-96af-11ed-ad40-000d3ab39166" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=18989948400
I0117 21:38:38.529002    2284 reflector.go:257] Listing and watching *v1beta2.AzVolumeAttachment from sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132
I0117 21:38:38.540901    2284 round_trippers.go:553] GET https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumeattachments?resourceVersion=22928 200 OK in 11 milliseconds
E0117 21:38:38.541921    2284 conditionwatcher.go:138] object (%!v(MISSING)) has not implemented meta object interface.
E0117 21:38:38.541921    2284 conditionwatcher.go:151] unsupported object type cache.DeletedFinalStateUnknown
I0117 21:38:38.543911    2284 round_trippers.go:553] GET https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumeattachments?allowWatchBookmarks=true&resourceVersion=24725&timeout=5m2s&timeoutSeconds=302&watch=true 200 OK in 1 milliseconds
I0117 21:38:38.559903    2284 reflector.go:257] Listing and watching *v1beta2.AzDriverNode from sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132
... skipping 31 lines ...
I0117 21:39:11.843115    2284 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0117 21:39:11.843115    2284 utils.go:79] GRPC request: {}
I0117 21:39:11.843115    2284 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I0117 21:39:11.844612    2284 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0117 21:39:11.844612    2284 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-65e6c67a-668c-4672-ab77-db4571a6a498\\globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-65e6c67a-668c-4672-ab77-db4571a6a498","csi.storage.k8s.io/pvc/name":"1-azuredisk-volume-tester-dr8c7-0","csi.storage.k8s.io/pvc/namespace":"azuredisk-9678","requestedsizegib":"10","skuname":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1673987143207-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-ri100d/providers/Microsoft.Compute/disks/pvc-65e6c67a-668c-4672-ab77-db4571a6a498"}
I0117 21:39:11.844612    2284 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-65e6c67a-668c-4672-ab77-db4571a6a498-capz-ri10-8qjlr-attachment)
I0117 21:39:17.083387    2284 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 21:39:17.083856    2284 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-ri10-8qjlr" "disk.csi.azure.com/request-id"="618d4a2f-96af-11ed-ad40-000d3ab39166" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-65e6c67a-668c-4672-ab77-db4571a6a498" "latency"=5238722100
I0117 21:39:17.083856    2284 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="618d4a2f-96af-11ed-ad40-000d3ab39166" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=5239190700
I0117 21:39:18.627583    2284 azuredisk_v2.go:578]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-17T21:39:18Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-ri10-8qjlr" "disk.csi.azure.com/request-id"="659849f8-96af-11ed-ad40-000d3ab39166"
I0117 21:39:18.645372    2284 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-ri10-8qjlr/status 200 OK in 17 milliseconds
I0117 21:39:18.645372    2284 azuredisk_v2.go:565]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-ri10-8qjlr" "disk.csi.azure.com/request-id"="659849f8-96af-11ed-ad40-000d3ab39166" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=17789000
I0117 21:39:21.279425    2284 nodeserver_v2.go:211] NodeStageVolume: formatting 10 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-65e6c67a-668c-4672-ab77-db4571a6a498\globalmount with mount options([])
... skipping 166 lines ...
I0117 21:44:19.755073    2284 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0117 21:44:19.755073    2284 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-b6bbc058-f009-4d9d-bfc4-30be0d7d127f\\globalmount","volume_capability":{"AccessType":{"Mount":{"mount_flags":["barrier=1","acl"]}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-b6bbc058-f009-4d9d-bfc4-30be0d7d127f","csi.storage.k8s.io/pvc/name":"pvc-dcdqw","csi.storage.k8s.io/pvc/namespace":"azuredisk-1671","requestedsizegib":"10","skuname":"StandardSSD_ZRS","storage.kubernetes.io/csiProvisionerIdentity":"1673987143207-8081-disk.csi.azure.com","tags":"disk=test"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-ri100d/providers/Microsoft.Compute/disks/pvc-b6bbc058-f009-4d9d-bfc4-30be0d7d127f"}
I0117 21:44:19.755073    2284 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-b6bbc058-f009-4d9d-bfc4-30be0d7d127f-capz-ri10-8qjlr-attachment)
I0117 21:44:23.776531    2284 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I0117 21:44:23.776531    2284 utils.go:79] GRPC request: {}
I0117 21:44:23.776531    2284 utils.go:85] GRPC response: {"ready":{"value":true}}
I0117 21:44:29.223601    2284 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 21:44:29.223601    2284 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-ri10-8qjlr" "disk.csi.azure.com/request-id"="1914b293-96b0-11ed-ad40-000d3ab39166" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-b6bbc058-f009-4d9d-bfc4-30be0d7d127f" "latency"=9468314500
I0117 21:44:29.223601    2284 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="1914b293-96b0-11ed-ad40-000d3ab39166" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9468314500
I0117 21:44:33.273218    2284 nodeserver_v2.go:211] NodeStageVolume: formatting 10 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-b6bbc058-f009-4d9d-bfc4-30be0d7d127f\globalmount with mount options([barrier=1 acl])
I0117 21:44:38.635008    2284 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 21:44:38.650414    2284 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 21:44:38.650414    2284 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
... skipping 213 lines ...
I0117 21:50:26.398125    2284 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0117 21:50:26.398125    2284 utils.go:79] GRPC request: {}
I0117 21:50:26.398125    2284 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I0117 21:50:26.399137    2284 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0117 21:50:26.399137    2284 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-6d6eb8a9-7295-4cde-bac9-e4ce558387df\\globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-6d6eb8a9-7295-4cde-bac9-e4ce558387df","csi.storage.k8s.io/pvc/name":"pvc1-azuredisk-volume-tester-cdj7l-0","csi.storage.k8s.io/pvc/namespace":"azuredisk-5772","requestedsizegib":"10","skuname":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1673987143207-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-ri100d/providers/Microsoft.Compute/disks/pvc-6d6eb8a9-7295-4cde-bac9-e4ce558387df"}
I0117 21:50:26.399627    2284 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-6d6eb8a9-7295-4cde-bac9-e4ce558387df-capz-ri10-8qjlr-attachment)
I0117 21:50:26.684549    2284 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 21:50:26.685050    2284 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-ri10-8qjlr" "disk.csi.azure.com/request-id"="f39e3770-96b0-11ed-ad40-000d3ab39166" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-6d6eb8a9-7295-4cde-bac9-e4ce558387df" "latency"=284919900
I0117 21:50:26.685050    2284 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="f39e3770-96b0-11ed-ad40-000d3ab39166" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=285420400
I0117 21:50:30.885587    2284 nodeserver_v2.go:211] NodeStageVolume: formatting 10 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-6d6eb8a9-7295-4cde-bac9-e4ce558387df\globalmount with mount options([])
I0117 21:50:38.757712    2284 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 21:50:38.757712    2284 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 21:50:38.757712    2284 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
... skipping 68 lines ...
I0117 21:51:22.925799    2284 nodeserver_v2.go:348] NodePublishVolume: mounting \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-6d6eb8a9-7295-4cde-bac9-e4ce558387df\globalmount at c:\var\lib\kubelet\pods\4e32a37f-2f1b-4718-944b-cead38f4b2a7\volumes\kubernetes.io~csi\pvc-6d6eb8a9-7295-4cde-bac9-e4ce558387df\mount
I0117 21:51:22.926795    2284 nodeserver_v2.go:353] NodePublishVolume: mount \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-6d6eb8a9-7295-4cde-bac9-e4ce558387df\globalmount at c:\var\lib\kubelet\pods\4e32a37f-2f1b-4718-944b-cead38f4b2a7\volumes\kubernetes.io~csi\pvc-6d6eb8a9-7295-4cde-bac9-e4ce558387df\mount successfully
I0117 21:51:22.926795    2284 utils.go:85] GRPC response: {}
I0117 21:51:23.789539    2284 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I0117 21:51:23.789539    2284 utils.go:79] GRPC request: {}
I0117 21:51:23.789539    2284 utils.go:85] GRPC response: {"ready":{"value":true}}
W0117 21:51:33.661892    2284 reflector.go:347] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: watch of *v1beta2.AzVolume ended with: an error on the server ("unable to decode an event from the watch stream: http2: client connection lost") has prevented the request from succeeding
W0117 21:51:33.661892    2284 reflector.go:347] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: watch of *v1beta2.AzDriverNode ended with: an error on the server ("unable to decode an event from the watch stream: http2: client connection lost") has prevented the request from succeeding
W0117 21:51:33.661892    2284 reflector.go:347] k8s.io/client-go/informers/factory.go:150: watch of *v1.Node ended with: an error on the server ("unable to decode an event from the watch stream: http2: client connection lost") has prevented the request from succeeding
W0117 21:51:33.661892    2284 reflector.go:347] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: watch of *v1.CustomResourceDefinition ended with: an error on the server ("unable to decode an event from the watch stream: http2: client connection lost") has prevented the request from succeeding
I0117 21:51:33.661892    2284 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-ri10-8qjlr/status  in 15019 milliseconds
W0117 21:51:33.661892    2284 reflector.go:347] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: watch of *v1beta2.AzVolumeAttachment ended with: an error on the server ("unable to decode an event from the watch stream: http2: client connection lost") has prevented the request from succeeding
W0117 21:51:33.661892    2284 reflector.go:347] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: watch of *v1beta2.AzDriverNode ended with: an error on the server ("unable to decode an event from the watch stream: http2: client connection lost") has prevented the request from succeeding
E0117 21:51:33.661892    2284 azuredisk_v2.go:613]  "msg"="Failed to update AzDriverNode status after creation" "error"="Put \"https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-ri10-8qjlr/status\": http2: client connection lost" "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-ri10-8qjlr" "disk.csi.azure.com/request-id"="12c1a987-96b1-11ed-ad40-000d3ab39166"
E0117 21:51:33.662288    2284 azuredisk_v2.go:565]  "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [Put \"https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-ri10-8qjlr/status\": http2: client connection lost]" "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-ri10-8qjlr" "disk.csi.azure.com/request-id"="12c1a987-96b1-11ed-ad40-000d3ab39166" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=15020436400
I0117 21:51:34.606030    2284 reflector.go:257] Listing and watching *v1.Node from k8s.io/client-go/informers/factory.go:150
I0117 21:51:34.621980    2284 round_trippers.go:553] GET https://10.96.0.1:443/api/v1/nodes?resourceVersion=28309 200 OK in 15 milliseconds
I0117 21:51:34.625981    2284 round_trippers.go:553] GET https://10.96.0.1:443/api/v1/nodes?allowWatchBookmarks=true&resourceVersion=28309&timeout=8m35s&timeoutSeconds=515&watch=true 200 OK in 1 milliseconds
I0117 21:51:34.807288    2284 reflector.go:257] Listing and watching *v1beta2.AzDriverNode from sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132
I0117 21:51:34.810700    2284 round_trippers.go:553] GET https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes?fieldSelector=metadata.name%3Dcapz-ri10-8qjlr&resourceVersion=28406 200 OK in 2 milliseconds
I0117 21:51:34.812700    2284 round_trippers.go:553] GET https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dcapz-ri10-8qjlr&resourceVersion=28607&timeout=5m37s&timeoutSeconds=337&watch=true 200 OK in 1 milliseconds
... skipping 138 lines ...
I0117 21:54:31.679535    2284 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-29a919d8-9a1b-442f-a03c-0afae97395f2-capz-ri10-8qjlr-attachment)
I0117 21:54:34.858515    2284 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 21:54:35.029313    2284 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync
I0117 21:54:35.106141    2284 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 21:54:35.183573    2284 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 21:54:35.199117    2284 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 21:54:35.199117    2284 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0117 21:54:36.542380    2284 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 21:54:36.542380    2284 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-ri10-8qjlr" "disk.csi.azure.com/request-id"="85d0f5d6-96b1-11ed-ad40-000d3ab39166" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-29a919d8-9a1b-442f-a03c-0afae97395f2" "latency"=4862786400
I0117 21:54:36.542380    2284 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="85d0f5d6-96b1-11ed-ad40-000d3ab39166" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=4862786400
I0117 21:54:40.857168    2284 nodeserver_v2.go:211] NodeStageVolume: formatting 10 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-29a919d8-9a1b-442f-a03c-0afae97395f2\globalmount with mount options([barrier=1 acl])
I0117 21:54:48.635616    2284 azuredisk_v2.go:578]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-17T21:54:48Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-ri10-8qjlr" "disk.csi.azure.com/request-id"="8fec3ba4-96b1-11ed-ad40-000d3ab39166"
I0117 21:54:48.660304    2284 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-ri10-8qjlr/status 200 OK in 23 milliseconds
I0117 21:54:48.660814    2284 azuredisk_v2.go:565]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-ri10-8qjlr" "disk.csi.azure.com/request-id"="8fec3ba4-96b1-11ed-ad40-000d3ab39166" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=25404200
... skipping 72 lines ...
I0117 21:56:03.725763    2284 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-6af0faa6-72ed-4638-ba8e-bcf98c6bfc90-capz-ri10-8qjlr-attachment)
I0117 21:56:04.882014    2284 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 21:56:05.052499    2284 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync
I0117 21:56:05.130089    2284 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 21:56:05.207416    2284 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 21:56:05.222916    2284 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 21:56:05.222916    2284 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0117 21:56:07.556801    2284 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 21:56:07.556801    2284 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-ri10-8qjlr" "disk.csi.azure.com/request-id"="bcae199f-96b1-11ed-ad40-000d3ab39166" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-6af0faa6-72ed-4638-ba8e-bcf98c6bfc90" "latency"=3831008600
I0117 21:56:07.556801    2284 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="bcae199f-96b1-11ed-ad40-000d3ab39166" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=3831506100
I0117 21:56:11.720362    2284 nodeserver_v2.go:211] NodeStageVolume: formatting 10 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-6af0faa6-72ed-4638-ba8e-bcf98c6bfc90\globalmount with mount options([barrier=1 acl])
I0117 21:56:18.645436    2284 azuredisk_v2.go:578]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-17T21:56:18Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-ri10-8qjlr" "disk.csi.azure.com/request-id"="c592a95c-96b1-11ed-ad40-000d3ab39166"
I0117 21:56:18.662225    2284 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-ri10-8qjlr/status 200 OK in 16 milliseconds
I0117 21:56:18.662432    2284 azuredisk_v2.go:565]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-ri10-8qjlr" "disk.csi.azure.com/request-id"="c592a95c-96b1-11ed-ad40-000d3ab39166" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=16995500
... skipping 36 lines ...
I0117 21:56:48.645279    2284 azuredisk_v2.go:578]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-17T21:56:48Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-ri10-8qjlr" "disk.csi.azure.com/request-id"="d7743e20-96b1-11ed-ad40-000d3ab39166"
I0117 21:56:48.657995    2284 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-ri10-8qjlr/status 200 OK in 11 milliseconds
I0117 21:56:48.658089    2284 azuredisk_v2.go:565]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-ri10-8qjlr" "disk.csi.azure.com/request-id"="d7743e20-96b1-11ed-ad40-000d3ab39166" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=13017300
I0117 21:56:53.797060    2284 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I0117 21:56:53.797060    2284 utils.go:79] GRPC request: {}
I0117 21:56:53.797060    2284 utils.go:85] GRPC response: {"ready":{"value":true}}
I0117 21:56:54.103343    2284 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 21:56:54.103343    2284 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-ri10-8qjlr" "disk.csi.azure.com/request-id"="d4f50387-96b1-11ed-ad40-000d3ab39166" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-07350749-804b-4c0f-981b-bcccc8633254" "latency"=9647378400
I0117 21:56:54.103343    2284 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="d4f50387-96b1-11ed-ad40-000d3ab39166" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9647378400
I0117 21:56:58.352608    2284 nodeserver_v2.go:211] NodeStageVolume: formatting 13 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-07350749-804b-4c0f-981b-bcccc8633254\globalmount with mount options([barrier=1 acl])
I0117 21:57:00.168459    2284 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: Watch close - *v1beta2.AzVolumeAttachment total 60 items received
I0117 21:57:00.174625    2284 round_trippers.go:553] GET https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumeattachments?allowWatchBookmarks=true&resourceVersion=30346&timeout=6m39s&timeoutSeconds=399&watch=true 200 OK in 6 milliseconds
I0117 21:57:04.907138    2284 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
... skipping 402 lines ...
I0117 22:08:23.800085    2284 utils.go:85] GRPC response: {"ready":{"value":true}}
I0117 22:08:35.107843    2284 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 22:08:35.246906    2284 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync
I0117 22:08:35.278092    2284 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 22:08:35.386403    2284 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 22:08:35.386403    2284 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
W0117 22:08:37.616529    2284 reflector.go:347] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: watch of *v1beta2.AzVolume ended with: an error on the server ("unable to decode an event from the watch stream: read tcp 10.1.0.6:36621->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.") has prevented the request from succeeding
W0117 22:08:37.616529    2284 reflector.go:347] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: watch of *v1beta2.AzDriverNode ended with: an error on the server ("unable to decode an event from the watch stream: read tcp 10.1.0.6:36621->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.") has prevented the request from succeeding
W0117 22:08:37.616683    2284 reflector.go:347] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: watch of *v1beta2.AzDriverNode ended with: an error on the server ("unable to decode an event from the watch stream: read tcp 10.1.0.6:36621->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.") has prevented the request from succeeding
W0117 22:08:37.616529    2284 reflector.go:347] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: watch of *v1beta2.AzVolumeAttachment ended with: an error on the server ("unable to decode an event from the watch stream: read tcp 10.1.0.6:36621->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.") has prevented the request from succeeding
W0117 22:08:37.616683    2284 reflector.go:347] k8s.io/client-go/informers/factory.go:150: watch of *v1.Node ended with: an error on the server ("unable to decode an event from the watch stream: read tcp 10.1.0.6:36621->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.") has prevented the request from succeeding
W0117 22:08:37.616683    2284 reflector.go:347] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: watch of *v1.CustomResourceDefinition ended with: an error on the server ("unable to decode an event from the watch stream: read tcp 10.1.0.6:36621->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.") has prevented the request from succeeding
I0117 22:08:37.616529    2284 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-ri10-8qjlr/status  in 18973 milliseconds
E0117 22:08:37.616683    2284 azuredisk_v2.go:613]  "msg"="Failed to update AzDriverNode status after creation" "error"="Put \"https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-ri10-8qjlr/status\": read tcp 10.1.0.6:36621->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-ri10-8qjlr" "disk.csi.azure.com/request-id"="72b97c59-96b3-11ed-ad40-000d3ab39166"
E0117 22:08:37.616683    2284 azuredisk_v2.go:565]  "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [Put \"https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-ri10-8qjlr/status\": read tcp 10.1.0.6:36621->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.]" "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-ri10-8qjlr" "disk.csi.azure.com/request-id"="72b97c59-96b3-11ed-ad40-000d3ab39166" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=18974025100
I0117 22:08:38.466321    2284 reflector.go:257] Listing and watching *v1.Node from k8s.io/client-go/informers/factory.go:150
I0117 22:08:38.480952    2284 round_trippers.go:553] GET https://10.96.0.1:443/api/v1/nodes?resourceVersion=33186 200 OK in 13 milliseconds
I0117 22:08:38.485333    2284 round_trippers.go:553] GET https://10.96.0.1:443/api/v1/nodes?allowWatchBookmarks=true&resourceVersion=33375&timeout=5m44s&timeoutSeconds=344&watch=true 200 OK in 1 milliseconds
I0117 22:08:38.683857    2284 reflector.go:257] Listing and watching *v1beta2.AzVolumeAttachment from sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132
I0117 22:08:38.687258    2284 round_trippers.go:553] GET https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumeattachments?resourceVersion=33197 200 OK in 2 milliseconds
E0117 22:08:38.687763    2284 conditionwatcher.go:138] object (%!v(MISSING)) has not implemented meta object interface.
... skipping 28 lines ...
I0117 22:08:58.393960    2284 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0117 22:08:58.393960    2284 utils.go:79] GRPC request: {}
I0117 22:08:58.393960    2284 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I0117 22:08:58.395454    2284 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0117 22:08:58.395454    2284 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-2f370b01-ab02-45b6-a92f-98dc2f982e06\\globalmount","volume_capability":{"AccessType":{"Mount":{"mount_flags":["barrier=1","acl"]}},"access_mode":{"mode":7}},"volume_context":{"cachingmode":"None","csi.storage.k8s.io/pv/name":"pvc-2f370b01-ab02-45b6-a92f-98dc2f982e06","csi.storage.k8s.io/pvc/name":"pvc-wrvlb","csi.storage.k8s.io/pvc/namespace":"azuredisk-8582","maxshares":"2","requestedsizegib":"10","skuname":"Premium_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1673987143207-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-ri100d/providers/Microsoft.Compute/disks/pvc-2f370b01-ab02-45b6-a92f-98dc2f982e06"}
I0117 22:08:58.395454    2284 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-2f370b01-ab02-45b6-a92f-98dc2f982e06-capz-ri10-8qjlr-attachment)
I0117 22:09:07.930384    2284 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 22:09:07.930384    2284 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-ri10-8qjlr" "disk.csi.azure.com/request-id"="8a6b5072-96b3-11ed-ad40-000d3ab39166" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-2f370b01-ab02-45b6-a92f-98dc2f982e06" "latency"=9534809100
I0117 22:09:07.930384    2284 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="8a6b5072-96b3-11ed-ad40-000d3ab39166" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9534809100
I0117 22:09:08.692998    2284 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 22:09:08.894539    2284 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 22:09:08.956600    2284 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 22:09:08.982894    2284 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
... skipping 237 lines ...
I0117 22:16:27.739947    2284 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I0117 22:16:27.741451    2284 utils.go:78] GRPC call: /csi.v1.Node/NodeUnstageVolume
I0117 22:16:27.741451    2284 utils.go:79] GRPC request: {"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\azuredisk-7381-disk.csi.azure.com-preprovisioned-pv-6m697\\globalmount","volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-ri100d/providers/Microsoft.Compute/disks/pre-provisioned-read-only"}
I0117 22:16:27.741451    2284 nodeserver_v2.go:257] NodeUnstageVolume: unmounting \var\lib\kubelet\plugins\kubernetes.io\csi\pv\azuredisk-7381-disk.csi.azure.com-preprovisioned-pv-6m697\globalmount
I0117 22:16:30.262002    2284 nodeserver_v2.go:262] NodeUnstageVolume: unmount \var\lib\kubelet\plugins\kubernetes.io\csi\pv\azuredisk-7381-disk.csi.azure.com-preprovisioned-pv-6m697\globalmount successfully
I0117 22:16:30.262002    2284 utils.go:85] GRPC response: {}
W0117 22:16:37.634556    2284 reflector.go:347] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: watch of *v1beta2.AzVolumeAttachment ended with: an error on the server ("unable to decode an event from the watch stream: read tcp 10.1.0.6:37414->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.") has prevented the request from succeeding
W0117 22:16:37.634556    2284 reflector.go:347] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: watch of *v1beta2.AzDriverNode ended with: an error on the server ("unable to decode an event from the watch stream: read tcp 10.1.0.6:37414->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.") has prevented the request from succeeding
W0117 22:16:37.634556    2284 reflector.go:347] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: watch of *v1beta2.AzDriverNode ended with: an error on the server ("unable to decode an event from the watch stream: read tcp 10.1.0.6:37414->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.") has prevented the request from succeeding
W0117 22:16:37.634556    2284 reflector.go:347] k8s.io/client-go/informers/factory.go:150: watch of *v1.Node ended with: an error on the server ("unable to decode an event from the watch stream: read tcp 10.1.0.6:37414->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.") has prevented the request from succeeding
I0117 22:16:37.634556    2284 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-ri10-8qjlr/status  in 18980 milliseconds
W0117 22:16:37.634556    2284 reflector.go:347] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: watch of *v1.CustomResourceDefinition ended with: an error on the server ("unable to decode an event from the watch stream: read tcp 10.1.0.6:37414->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.") has prevented the request from succeeding
W0117 22:16:37.634556    2284 reflector.go:347] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: watch of *v1beta2.AzVolume ended with: an error on the server ("unable to decode an event from the watch stream: read tcp 10.1.0.6:37414->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.") has prevented the request from succeeding
E0117 22:16:37.634556    2284 azuredisk_v2.go:613]  "msg"="Failed to update AzDriverNode status after creation" "error"="Put \"https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-ri10-8qjlr/status\": read tcp 10.1.0.6:37414->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-ri10-8qjlr" "disk.csi.azure.com/request-id"="90d5545d-96b4-11ed-ad40-000d3ab39166"
E0117 22:16:37.634556    2284 azuredisk_v2.go:565]  "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [Put \"https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-ri10-8qjlr/status\": read tcp 10.1.0.6:37414->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.]" "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-ri10-8qjlr" "disk.csi.azure.com/request-id"="90d5545d-96b4-11ed-ad40-000d3ab39166" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=18980934700
I0117 22:16:38.609577    2284 reflector.go:257] Listing and watching *v1.Node from k8s.io/client-go/informers/factory.go:150
I0117 22:16:38.620483    2284 round_trippers.go:553] GET https://10.96.0.1:443/api/v1/nodes?resourceVersion=35843 200 OK in 9 milliseconds
I0117 22:16:38.625090    2284 round_trippers.go:553] GET https://10.96.0.1:443/api/v1/nodes?allowWatchBookmarks=true&resourceVersion=36085&timeout=8m51s&timeoutSeconds=531&watch=true 200 OK in 1 milliseconds
I0117 22:16:38.702486    2284 reflector.go:257] Listing and watching *v1beta2.AzVolumeAttachment from sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132
I0117 22:16:38.705490    2284 round_trippers.go:553] GET https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumeattachments?resourceVersion=35832 200 OK in 1 milliseconds
I0117 22:16:38.709987    2284 round_trippers.go:553] GET https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumeattachments?allowWatchBookmarks=true&resourceVersion=36089&timeout=7m5s&timeoutSeconds=425&watch=true 200 OK in 3 milliseconds
... skipping 95 lines ...
I0117 22:20:12.926967    2284 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0117 22:20:12.927480    2284 utils.go:79] GRPC request: {}
I0117 22:20:12.927480    2284 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I0117 22:20:12.928457    2284 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0117 22:20:12.928457    2284 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-0f723d82-f59c-4ec2-9b3b-d069faa319eb\\globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-0f723d82-f59c-4ec2-9b3b-d069faa319eb","csi.storage.k8s.io/pvc/name":"persistent-storage-statefulset-azuredisk-win-0","csi.storage.k8s.io/pvc/namespace":"default","requestedsizegib":"100","skuName":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1673987143207-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-ri100d/providers/Microsoft.Compute/disks/pvc-0f723d82-f59c-4ec2-9b3b-d069faa319eb"}
I0117 22:20:12.928969    2284 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-0f723d82-f59c-4ec2-9b3b-d069faa319eb-capz-ri10-8qjlr-attachment)
I0117 22:20:14.624491    2284 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 22:20:14.624491    2284 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-ri10-8qjlr" "disk.csi.azure.com/request-id"="1c78f5db-96b5-11ed-ad40-000d3ab39166" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-0f723d82-f59c-4ec2-9b3b-d069faa319eb" "latency"=1695495700
I0117 22:20:14.624491    2284 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="1c78f5db-96b5-11ed-ad40-000d3ab39166" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=1695495700
I0117 22:20:18.657705    2284 azuredisk_v2.go:578]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-17T22:20:18Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-ri10-8qjlr" "disk.csi.azure.com/request-id"="1fe3189c-96b5-11ed-ad40-000d3ab39166"
I0117 22:20:18.669725    2284 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-ri10-8qjlr/status 200 OK in 11 milliseconds
I0117 22:20:18.669725    2284 azuredisk_v2.go:565]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-ri10-8qjlr" "disk.csi.azure.com/request-id"="1fe3189c-96b5-11ed-ad40-000d3ab39166" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=12020400
I0117 22:20:19.217723    2284 nodeserver_v2.go:211] NodeStageVolume: formatting 10 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-0f723d82-f59c-4ec2-9b3b-d069faa319eb\globalmount with mount options([])
... skipping 80 lines ...
I0117 20:25:59.243856    6004 round_trippers.go:553] GET https://10.96.0.1:443/apis/apiextensions.k8s.io/v1/customresourcedefinitions?limit=500&resourceVersion=0 200 OK in 78 milliseconds
I0117 20:25:59.286544    6004 round_trippers.go:553] GET https://10.96.0.1:443/apis/apiextensions.k8s.io/v1/customresourcedefinitions?allowWatchBookmarks=true&resourceVersion=2085&timeout=9m1s&timeoutSeconds=541&watch=true 200 OK in 1 milliseconds
I0117 20:25:59.370981    6004 shared_informer.go:303] caches populated
I0117 20:25:59.371009    6004 azuredisk_v2.go:225] driver userAgent: disk.csi.azure.com/latest-v2-9ef068a8cb36a997d4ea04b90c05c6f92a488a19 e2e-test
I0117 20:25:59.371070    6004 azure_disk_utils.go:564] reading cloud config from secret kube-system/azure-cloud-provider
I0117 20:25:59.375488    6004 round_trippers.go:553] GET https://10.96.0.1:443/api/v1/namespaces/kube-system/secrets/azure-cloud-provider 404 Not Found in 3 milliseconds
I0117 20:25:59.376080    6004 azure_disk_utils.go:571] InitializeCloudFromSecret: failed to get cloud config from secret kube-system/azure-cloud-provider: failed to get secret kube-system/azure-cloud-provider: secrets "azure-cloud-provider" not found
I0117 20:25:59.376080    6004 azure_disk_utils.go:576] could not read cloud config from secret kube-system/azure-cloud-provider
I0117 20:25:59.376080    6004 azure_disk_utils.go:586] use default AZURE_CREDENTIAL_FILE env var: C:\k\azure.json
I0117 20:25:59.376610    6004 azure_disk_utils.go:594] read cloud config from file: C:\k\azure.json successfully
I0117 20:25:59.378971    6004 azure_auth.go:253] Using AzurePublicCloud environment
I0117 20:25:59.378971    6004 azure_auth.go:138] azure: using client_id+client_secret to retrieve access token
I0117 20:25:59.380890    6004 azure_diskclient.go:68] Azure DisksClient using API version: 2022-03-02
... skipping 33 lines ...
I0117 20:25:59.909078    6004 utils.go:85] GRPC response: {"name":"disk.csi.azure.com","vendor_version":"latest-v2-9ef068a8cb36a997d4ea04b90c05c6f92a488a19"}
I0117 20:26:00.357912    6004 utils.go:78] GRPC call: /csi.v1.Identity/GetPluginInfo
I0117 20:26:00.357912    6004 utils.go:79] GRPC request: {}
I0117 20:26:00.357912    6004 utils.go:85] GRPC response: {"name":"disk.csi.azure.com","vendor_version":"latest-v2-9ef068a8cb36a997d4ea04b90c05c6f92a488a19"}
I0117 20:26:01.039207    6004 utils.go:78] GRPC call: /csi.v1.Node/NodeGetInfo
I0117 20:26:01.039242    6004 utils.go:79] GRPC request: {}
I0117 20:26:01.060918    6004 azure_instance_metadata.go:140] Warning: failed to get loadbalancer metadata: failure of getting loadbalancer metadata with response "404 Not Found"
I0117 20:26:01.060918    6004 nodeserver_v2.go:438] NodeGetInfo: nodeName(capz-ri10-ppsff), VM Size(Standard_D4s_v3)
I0117 20:26:01.060918    6004 nodeserver.go:409] got a matching size in getMaxDataDiskCount, VM Size: STANDARD_D4S_V3, MaxDataDiskCount: 8
I0117 20:26:01.060918    6004 utils.go:85] GRPC response: {"accessible_topology":{"segments":{"topology.disk.csi.azure.com/zone":""}},"max_volumes_per_node":8,"node_id":"capz-ri10-ppsff"}
I0117 20:26:29.217421    6004 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 20:26:29.232702    6004 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 20:26:29.232765    6004 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
... skipping 66 lines ...
I0117 20:29:14.859039    6004 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0117 20:29:14.859039    6004 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0\\globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0","csi.storage.k8s.io/pvc/name":"pvc-6tkv4","csi.storage.k8s.io/pvc/namespace":"azuredisk-1538","requestedsizegib":"10","skuname":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1673987143207-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-ri100d/providers/Microsoft.Compute/disks/pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0"}
I0117 20:29:14.859809    6004 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0-capz-ri10-ppsff-attachment)
I0117 20:29:23.759501    6004 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I0117 20:29:23.759501    6004 utils.go:79] GRPC request: {}
I0117 20:29:23.759501    6004 utils.go:85] GRPC response: {"ready":{"value":true}}
I0117 20:29:24.424412    6004 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 20:29:24.424412    6004 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="9bf42502-96a5-11ed-a766-000d3a65271f" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-ef3850c1-7774-47a7-bd88-a2abf284d9a0" "latency"=9564548400
I0117 20:29:24.424412    6004 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="9bf42502-96a5-11ed-a766-000d3a65271f" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9565119700
I0117 20:29:28.622339    6004 azuredisk_v2.go:578]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-17T20:29:28Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="a4283a21-96a5-11ed-a766-000d3a65271f"
I0117 20:29:28.635706    6004 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-ri10-ppsff/status 200 OK in 12 milliseconds
I0117 20:29:28.641115    6004 azuredisk_v2.go:565]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="a4283a21-96a5-11ed-a766-000d3a65271f" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=18776000
I0117 20:29:29.262165    6004 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
... skipping 84 lines ...
I0117 20:31:28.595689    6004 azuredisk_v2.go:578]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-17T20:31:28Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="ebaab51a-96a5-11ed-a766-000d3a65271f"
I0117 20:31:28.611199    6004 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-ri10-ppsff/status 200 OK in 15 milliseconds
I0117 20:31:28.611199    6004 azuredisk_v2.go:565]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="ebaab51a-96a5-11ed-a766-000d3a65271f" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=15510700
I0117 20:31:29.307070    6004 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 20:31:29.307070    6004 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 20:31:29.327471    6004 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 20:31:29.327722    6004 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0117 20:31:29.409999    6004 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync
I0117 20:31:29.478746    6004 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 20:31:44.309137    6004 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 20:31:44.309137    6004 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="deb0bdff-96a5-11ed-a766-000d3a65271f" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-4ca42c14-b312-449b-b9b1-ebfcde77416c" "latency"=37484058700
I0117 20:31:44.309692    6004 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="deb0bdff-96a5-11ed-a766-000d3a65271f" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=37484058700
I0117 20:31:48.509531    6004 nodeserver_v2.go:211] NodeStageVolume: formatting 10 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-4ca42c14-b312-449b-b9b1-ebfcde77416c\globalmount with mount options([])
I0117 20:31:53.767719    6004 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I0117 20:31:53.767719    6004 utils.go:79] GRPC request: {}
I0117 20:31:53.767719    6004 utils.go:85] GRPC response: {"ready":{"value":true}}
... skipping 144 lines ...
I0117 20:36:23.061354    6004 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0117 20:36:23.061354    6004 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-4ca42c14-b312-449b-b9b1-ebfcde77416c\\globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-4ca42c14-b312-449b-b9b1-ebfcde77416c","csi.storage.k8s.io/pvc/name":"test-volume-1-azuredisk-volume-tester-qnsd7-0","csi.storage.k8s.io/pvc/namespace":"azuredisk-3759","requestedsizegib":"10","skuname":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1673987143207-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-ri100d/providers/Microsoft.Compute/disks/pvc-4ca42c14-b312-449b-b9b1-ebfcde77416c"}
I0117 20:36:23.061714    6004 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-4ca42c14-b312-449b-b9b1-ebfcde77416c-capz-ri10-ppsff-attachment)
I0117 20:36:23.764330    6004 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I0117 20:36:23.764357    6004 utils.go:79] GRPC request: {}
I0117 20:36:23.764401    6004 utils.go:85] GRPC response: {"ready":{"value":true}}
I0117 20:36:26.672531    6004 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 20:36:26.672531    6004 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="9b2ea3df-96a6-11ed-a766-000d3a65271f" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-4ca42c14-b312-449b-b9b1-ebfcde77416c" "latency"=3610730300
I0117 20:36:26.672531    6004 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="9b2ea3df-96a6-11ed-a766-000d3a65271f" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=3610875500
I0117 20:36:28.606985    6004 azuredisk_v2.go:578]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-17T20:36:28Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="9e7ccc61-96a6-11ed-a766-000d3a65271f"
I0117 20:36:28.634998    6004 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-ri10-ppsff/status 200 OK in 21 milliseconds
I0117 20:36:28.635763    6004 azuredisk_v2.go:565]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="9e7ccc61-96a6-11ed-a766-000d3a65271f" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=29245900
I0117 20:36:29.402687    6004 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
... skipping 40 lines ...
I0117 20:36:59.413516    6004 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 20:36:59.413516    6004 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 20:36:59.454502    6004 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 20:36:59.527411    6004 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync
I0117 20:36:59.563695    6004 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 20:37:17.609761    6004 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-ri10-ppsff/status  in 19008 milliseconds
E0117 20:37:17.610663    6004 azuredisk_v2.go:613]  "msg"="Failed to update AzDriverNode status after creation" "error"="Put \"https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-ri10-ppsff/status\": read tcp 10.1.0.5:34553->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="b05d45a0-96a6-11ed-a766-000d3a65271f"
W0117 20:37:17.610663    6004 reflector.go:347] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: watch of *v1beta2.AzDriverNode ended with: an error on the server ("unable to decode an event from the watch stream: read tcp 10.1.0.5:34553->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.") has prevented the request from succeeding
W0117 20:37:17.610663    6004 reflector.go:347] k8s.io/client-go/informers/factory.go:150: watch of *v1.Node ended with: an error on the server ("unable to decode an event from the watch stream: read tcp 10.1.0.5:34553->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.") has prevented the request from succeeding
W0117 20:37:17.610663    6004 reflector.go:347] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: watch of *v1beta2.AzVolume ended with: an error on the server ("unable to decode an event from the watch stream: read tcp 10.1.0.5:34553->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.") has prevented the request from succeeding
E0117 20:37:17.610663    6004 azuredisk_v2.go:565]  "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [Put \"https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-ri10-ppsff/status\": read tcp 10.1.0.5:34553->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.]" "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="b05d45a0-96a6-11ed-a766-000d3a65271f" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=19011136000
W0117 20:37:17.610663    6004 reflector.go:347] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: watch of *v1beta2.AzVolumeAttachment ended with: an error on the server ("unable to decode an event from the watch stream: read tcp 10.1.0.5:34553->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.") has prevented the request from succeeding
W0117 20:37:17.610663    6004 reflector.go:347] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: watch of *v1beta2.AzDriverNode ended with: an error on the server ("unable to decode an event from the watch stream: read tcp 10.1.0.5:34553->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.") has prevented the request from succeeding
W0117 20:37:17.610663    6004 reflector.go:347] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: watch of *v1.CustomResourceDefinition ended with: an error on the server ("unable to decode an event from the watch stream: read tcp 10.1.0.5:34553->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.") has prevented the request from succeeding
I0117 20:37:18.422239    6004 reflector.go:257] Listing and watching *v1.CustomResourceDefinition from k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132
I0117 20:37:18.425830    6004 reflector.go:257] Listing and watching *v1beta2.AzVolumeAttachment from sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132
I0117 20:37:18.434045    6004 round_trippers.go:553] GET https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumeattachments?resourceVersion=5403 200 OK in 7 milliseconds
I0117 20:37:18.436632    6004 round_trippers.go:553] GET https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumeattachments?allowWatchBookmarks=true&resourceVersion=5591&timeout=5m8s&timeoutSeconds=308&watch=true 200 OK in 2 milliseconds
I0117 20:37:18.450126    6004 round_trippers.go:553] GET https://10.96.0.1:443/apis/apiextensions.k8s.io/v1/customresourcedefinitions?resourceVersion=2085 200 OK in 20 milliseconds
I0117 20:37:18.479390    6004 round_trippers.go:553] GET https://10.96.0.1:443/apis/apiextensions.k8s.io/v1/customresourcedefinitions?allowWatchBookmarks=true&resourceVersion=2085&timeout=5m49s&timeoutSeconds=349&watch=true 200 OK in 0 milliseconds
... skipping 59 lines ...
I0117 20:37:55.999739    6004 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0117 20:37:55.999739    6004 utils.go:79] GRPC request: {"publish_context":{"LUN":"1"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-2b22494f-523f-45eb-b871-958030de6e5d\\globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-2b22494f-523f-45eb-b871-958030de6e5d","csi.storage.k8s.io/pvc/name":"pvc-hbgsq","csi.storage.k8s.io/pvc/namespace":"azuredisk-9092","requestedsizegib":"10","skuname":"Premium_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1673987143207-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-ri100d/providers/Microsoft.Compute/disks/pvc-2b22494f-523f-45eb-b871-958030de6e5d"}
I0117 20:37:55.999739    6004 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-2b22494f-523f-45eb-b871-958030de6e5d-capz-ri10-ppsff-attachment)
I0117 20:37:58.601417    6004 azuredisk_v2.go:578]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-17T20:37:58Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="d420d07a-96a6-11ed-a767-000d3a65271f"
I0117 20:37:58.623746    6004 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-ri10-ppsff/status 200 OK in 8 milliseconds
I0117 20:37:58.623746    6004 azuredisk_v2.go:565]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="d420d07a-96a6-11ed-a767-000d3a65271f" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=22620400
I0117 20:38:10.529895    6004 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 20:38:10.532328    6004 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="d2826223-96a6-11ed-a767-000d3a65271f" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-5aea05f2-b729-4024-867d-387eaab33883" "latency"=14644835200
I0117 20:38:10.532328    6004 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="d2826223-96a6-11ed-a767-000d3a65271f" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=14647268200
I0117 20:38:10.552525    6004 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 20:38:10.552525    6004 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 20:38:10.552525    6004 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="d2826223-96a6-11ed-a766-000d3a65271f" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-2204d595-ccd6-4f82-8d53-c44dac6a1b13" "latency"=14667466100
I0117 20:38:10.553616    6004 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="d2826223-96a6-11ed-a766-000d3a65271f" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=14668502000
I0117 20:38:10.553616    6004 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="d293dfd6-96a6-11ed-a767-000d3a65271f" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-2b22494f-523f-45eb-b871-958030de6e5d" "latency"=14553927600
I0117 20:38:10.553616    6004 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="d293dfd6-96a6-11ed-a767-000d3a65271f" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=14553927600
I0117 20:38:15.933112    6004 nodeserver_v2.go:211] NodeStageVolume: formatting 12 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-2b22494f-523f-45eb-b871-958030de6e5d\globalmount with mount options([])
I0117 20:38:15.938217    6004 nodeserver_v2.go:211] NodeStageVolume: formatting 11 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-2204d595-ccd6-4f82-8d53-c44dac6a1b13\globalmount with mount options([])
... skipping 163 lines ...
I0117 20:40:09.843425    6004 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-e5a8b2ac-bfe0-4682-bc24-96c1b4873d6a-capz-ri10-ppsff-attachment)
I0117 20:40:09.843592    6004 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-bddbdb0b-5a3c-4977-b311-7830407e9c30-capz-ri10-ppsff-attachment)
I0117 20:40:09.843592    6004 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0117 20:40:09.844214    6004 utils.go:79] GRPC request: {"publish_context":{"LUN":"1"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-3dd9b88b-a1e1-4d97-ac4c-0f697ce12802\\globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-3dd9b88b-a1e1-4d97-ac4c-0f697ce12802","csi.storage.k8s.io/pvc/name":"test-volume-2-azuredisk-volume-tester-pr4rh-0","csi.storage.k8s.io/pvc/namespace":"azuredisk-3198","requestedsizegib":"10","skuname":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1673987143207-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-ri100d/providers/Microsoft.Compute/disks/pvc-3dd9b88b-a1e1-4d97-ac4c-0f697ce12802"}
I0117 20:40:09.844359    6004 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-3dd9b88b-a1e1-4d97-ac4c-0f697ce12802-capz-ri10-ppsff-attachment)
I0117 20:40:18.516138    6004 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 20:40:18.516393    6004 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0117 20:40:18.516393    6004 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0117 20:40:18.516393    6004 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0117 20:40:18.543757    6004 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync
I0117 20:40:18.627363    6004 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 20:40:19.190698    6004 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 20:40:19.247897    6004 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 20:40:23.768089    6004 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I0117 20:40:23.768625    6004 utils.go:79] GRPC request: {}
I0117 20:40:23.768667    6004 utils.go:85] GRPC response: {"ready":{"value":true}}
I0117 20:40:24.642094    6004 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 20:40:24.642296    6004 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="225ac653-96a7-11ed-a767-000d3a65271f" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-e5a8b2ac-bfe0-4682-bc24-96c1b4873d6a" "latency"=14798477100
I0117 20:40:24.642394    6004 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="225ac653-96a7-11ed-a767-000d3a65271f" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=14798986500
I0117 20:40:24.702003    6004 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 20:40:24.702003    6004 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="225ace5e-96a7-11ed-a767-000d3a65271f" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-bddbdb0b-5a3c-4977-b311-7830407e9c30" "latency"=14858300300
I0117 20:40:24.702003    6004 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="225ace5e-96a7-11ed-a767-000d3a65271f" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=14858427400
I0117 20:40:24.725005    6004 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 20:40:24.725005    6004 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="225af14b-96a7-11ed-a767-000d3a65271f" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-3dd9b88b-a1e1-4d97-ac4c-0f697ce12802" "latency"=14880534400
I0117 20:40:24.725005    6004 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="225af14b-96a7-11ed-a767-000d3a65271f" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=14880534400
I0117 20:40:25.738023    6004 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0117 20:40:25.738023    6004 utils.go:79] GRPC request: {}
I0117 20:40:25.738023    6004 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I0117 20:40:25.744950    6004 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
... skipping 32 lines ...
I0117 20:40:28.619474    6004 azuredisk_v2.go:578]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-17T20:40:28Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="2d8bcc3d-96a7-11ed-a767-000d3a65271f"
I0117 20:40:28.631394    6004 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-ri10-ppsff/status 200 OK in 11 milliseconds
I0117 20:40:28.631394    6004 azuredisk_v2.go:565]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="2d8bcc3d-96a7-11ed-a767-000d3a65271f" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=11919500
I0117 20:40:30.129353    6004 nodeserver_v2.go:211] NodeStageVolume: formatting 12 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-bddbdb0b-5a3c-4977-b311-7830407e9c30\globalmount with mount options([])
I0117 20:40:30.173008    6004 nodeserver_v2.go:211] NodeStageVolume: formatting 10 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-e5a8b2ac-bfe0-4682-bc24-96c1b4873d6a\globalmount with mount options([])
I0117 20:40:30.226465    6004 nodeserver_v2.go:211] NodeStageVolume: formatting 11 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-3dd9b88b-a1e1-4d97-ac4c-0f697ce12802\globalmount with mount options([])
I0117 20:40:34.980394    6004 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 20:40:34.980394    6004 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="2bd7a1b9-96a7-11ed-a767-000d3a65271f" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-10f78768-20ee-4af9-90b0-e2802639b1ad" "latency"=9219164900
I0117 20:40:34.980394    6004 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="2bd7a1b9-96a7-11ed-a767-000d3a65271f" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9219164900
I0117 20:40:34.993750    6004 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 20:40:34.993750    6004 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 20:40:34.993750    6004 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="2be52176-96a7-11ed-a767-000d3a65271f" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-ab4ce96f-bc25-418c-9d03-647e6d67e815" "latency"=9144057800
I0117 20:40:34.993750    6004 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="2be52176-96a7-11ed-a767-000d3a65271f" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9144057800
I0117 20:40:34.993750    6004 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="2bf6eaf6-96a7-11ed-a767-000d3a65271f" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-48d5b7cc-6597-4e3f-bde0-aa3eb545a993" "latency"=9026956500
I0117 20:40:34.993750    6004 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="2bf6eaf6-96a7-11ed-a767-000d3a65271f" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9027492100
I0117 20:40:44.473960    6004 nodeserver_v2.go:211] NodeStageVolume: formatting 15 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-10f78768-20ee-4af9-90b0-e2802639b1ad\globalmount with mount options([])
I0117 20:40:44.639709    6004 nodeserver_v2.go:211] NodeStageVolume: formatting 13 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-48d5b7cc-6597-4e3f-bde0-aa3eb545a993\globalmount with mount options([])
... skipping 227 lines ...
I0117 20:41:53.666636    6004 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I0117 20:41:53.666636    6004 utils.go:78] GRPC call: /csi.v1.Node/NodeGetVolumeStats
I0117 20:41:53.669075    6004 utils.go:79] GRPC request: {"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-ri100d/providers/Microsoft.Compute/disks/pvc-10f78768-20ee-4af9-90b0-e2802639b1ad","volume_path":"c:\\var\\lib\\kubelet\\pods\\cfa5bfaa-0dd8-491e-b9ba-2b8a452490a2\\volumes\\kubernetes.io~csi\\pvc-10f78768-20ee-4af9-90b0-e2802639b1ad\\mount"}
I0117 20:41:53.792344    6004 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I0117 20:41:53.792344    6004 utils.go:79] GRPC request: {}
I0117 20:41:53.792344    6004 utils.go:85] GRPC response: {"ready":{"value":true}}
E0117 20:41:55.296981    6004 utils.go:83] GRPC error: GetVolumeIDFromMount(c:\var\lib\kubelet\pods\cfa5bfaa-0dd8-491e-b9ba-2b8a452490a2\volumes\kubernetes.io~csi\pvc-10f78768-20ee-4af9-90b0-e2802639b1ad\mount) failed with error: rpc error: code = Unknown desc = error getting the volume for the mount c:\var\lib\kubelet\pods\cfa5bfaa-0dd8-491e-b9ba-2b8a452490a2\volumes\kubernetes.io~csi\pvc-10f78768-20ee-4af9-90b0-e2802639b1ad\mount, internal error error getting volume from mount. cmd: (Get-Item -Path Get-Item : Cannot find path 'C:\var\lib\kubelet\pods\cfa5bfaa-0dd8-491e-b9ba-2b8a452490a2\volumes\kubernetes.io~csi\pvc
-10f78768-20ee-4af9-90b0-e2802639b1ad\mount' because it does not exist.
At line:1 char:2
+ (Get-Item -Path c:\var\lib\kubelet\pods\cfa5bfaa-0dd8-491e-b9ba-2b8a4 ...
+  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : ObjectNotFound: (C:\var\lib\kube...02639b1ad\mount:String) [Get-Item], ItemNotFoundExcep 
   tion
... skipping 37 lines ...
+ ... d : PathNotFound,Microsoft.PowerShell.Commands.GetItemCommand).Target
+                                                                  ~
Unexpected token ')' in expression or statement.
    + CategoryInfo          : ParserError: (:) [], ParentContainsErrorRecordException
    + FullyQualifiedErrorId : MissingEndParenthesisInExpression
 
, error: exit status 1, response: <nil>
I0117 20:41:56.157019    6004 nodeserver_v2.go:262] NodeUnstageVolume: unmount \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-e5a8b2ac-bfe0-4682-bc24-96c1b4873d6a\globalmount successfully
I0117 20:41:56.157019    6004 utils.go:85] GRPC response: {}
I0117 20:41:56.271307    6004 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0117 20:41:56.271307    6004 utils.go:79] GRPC request: {}
I0117 20:41:56.271307    6004 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I0117 20:41:56.277482    6004 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
... skipping 930 lines ...
I0117 20:46:57.379037    6004 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0117 20:46:57.379037    6004 utils.go:79] GRPC request: {"publish_context":{"LUN":"1"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-460c23aa-05cf-4c9a-ac26-31487aef4611\\globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"cachingmode":"None","csi.storage.k8s.io/pv/name":"pvc-460c23aa-05cf-4c9a-ac26-31487aef4611","csi.storage.k8s.io/pvc/name":"test-volume-2-azuredisk-volume-tester-gbzf6-0","csi.storage.k8s.io/pvc/namespace":"azuredisk-2582","maxShares":"2","requestedsizegib":"256","skuname":"Premium_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1673987143207-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-ri100d/providers/Microsoft.Compute/disks/pvc-460c23aa-05cf-4c9a-ac26-31487aef4611"}
I0117 20:46:57.379037    6004 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-460c23aa-05cf-4c9a-ac26-31487aef4611-capz-ri10-ppsff-attachment)
I0117 20:46:58.613385    6004 azuredisk_v2.go:578]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-17T20:46:58Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="1600255d-96a8-11ed-a767-000d3a65271f"
I0117 20:46:58.640850    6004 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-ri10-ppsff/status 200 OK in 26 milliseconds
I0117 20:46:58.640951    6004 azuredisk_v2.go:565]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="1600255d-96a8-11ed-a767-000d3a65271f" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=27565700
I0117 20:46:59.874710    6004 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 20:46:59.874710    6004 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="1543ccab-96a8-11ed-a767-000d3a65271f" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-460c23aa-05cf-4c9a-ac26-31487aef4611" "latency"=2495650100
I0117 20:46:59.875234    6004 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="1543ccab-96a8-11ed-a767-000d3a65271f" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=2495650100
I0117 20:46:59.884701    6004 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 20:46:59.884701    6004 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="15262fd3-96a8-11ed-a767-000d3a65271f" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-663927c5-b139-4bd9-9f8c-146c8c85dd2e" "latency"=2699709000
I0117 20:46:59.884701    6004 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="15262fd3-96a8-11ed-a767-000d3a65271f" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=2699709000
I0117 20:46:59.884701    6004 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 20:46:59.884701    6004 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="151ae41d-96a8-11ed-a767-000d3a65271f" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-2a802f2a-2732-4477-b9af-eba414a72087" "latency"=2773735900
I0117 20:46:59.884701    6004 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="151ae41d-96a8-11ed-a767-000d3a65271f" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=2773735900
I0117 20:46:59.891821    6004 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 20:46:59.891821    6004 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="151b14ac-96a8-11ed-a767-000d3a65271f" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-8970da8b-55ce-42b3-949b-0d9ce3daeb78" "latency"=2779541400
I0117 20:46:59.891821    6004 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="151b14ac-96a8-11ed-a767-000d3a65271f" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=2779653900
I0117 20:47:06.157759    6004 nodeserver_v2.go:211] NodeStageVolume: formatting 13 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-8970da8b-55ce-42b3-949b-0d9ce3daeb78\globalmount with mount options([])
I0117 20:47:06.170956    6004 nodeserver_v2.go:211] NodeStageVolume: formatting 10 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-663927c5-b139-4bd9-9f8c-146c8c85dd2e\globalmount with mount options([])
I0117 20:47:06.170956    6004 nodeserver_v2.go:211] NodeStageVolume: formatting 11 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-2a802f2a-2732-4477-b9af-eba414a72087\globalmount with mount options([])
I0117 20:47:06.380321    6004 nodeserver_v2.go:211] NodeStageVolume: formatting 12 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-460c23aa-05cf-4c9a-ac26-31487aef4611\globalmount with mount options([])
... skipping 713 lines ...
I0117 20:52:00.049113    6004 utils.go:78] GRPC call: /csi.v1.Node/NodeUnstageVolume
I0117 20:52:00.049113    6004 utils.go:79] GRPC request: {"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-8970da8b-55ce-42b3-949b-0d9ce3daeb78\\globalmount","volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-ri100d/providers/Microsoft.Compute/disks/pvc-8970da8b-55ce-42b3-949b-0d9ce3daeb78"}
I0117 20:52:00.049113    6004 nodeserver_v2.go:257] NodeUnstageVolume: unmounting \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-8970da8b-55ce-42b3-949b-0d9ce3daeb78\globalmount
I0117 20:52:00.497172    6004 utils.go:78] GRPC call: /csi.v1.Node/NodeUnpublishVolume
I0117 20:52:00.497172    6004 utils.go:79] GRPC request: {"target_path":"c:\\var\\lib\\kubelet\\pods\\2e14f23e-9881-40f8-bd55-ba67f4ea4fe8\\volumes\\kubernetes.io~csi\\pvc-2a802f2a-2732-4477-b9af-eba414a72087\\mount","volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-ri100d/providers/Microsoft.Compute/disks/pvc-2a802f2a-2732-4477-b9af-eba414a72087"}
I0117 20:52:00.497172    6004 nodeserver_v2.go:369] NodeUnpublishVolume: unmounting volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-ri100d/providers/Microsoft.Compute/disks/pvc-2a802f2a-2732-4477-b9af-eba414a72087 on c:\var\lib\kubelet\pods\2e14f23e-9881-40f8-bd55-ba67f4ea4fe8\volumes\kubernetes.io~csi\pvc-2a802f2a-2732-4477-b9af-eba414a72087\mount
W0117 20:52:01.167546    6004 safe_mounter_windows.go:113] GetVolumeIDFromTargetPath(c:\var\lib\kubelet\pods\2e14f23e-9881-40f8-bd55-ba67f4ea4fe8\volumes\kubernetes.io~csi\pvc-2a802f2a-2732-4477-b9af-eba414a72087\mount) failed with error: rpc error: code = Unknown desc = error getting the volume for the mount c:\var\lib\kubelet\pods\2e14f23e-9881-40f8-bd55-ba67f4ea4fe8\volumes\kubernetes.io~csi\pvc-2a802f2a-2732-4477-b9af-eba414a72087\mount, internal error error getting volume from mount. cmd: (Get-Item -Path Get-Item : Cannot find path 'C:\var\lib\kubelet\pods\2e14f23e-9881-40f8-bd55-ba67f4ea4fe8\volumes\kubernetes.io~csi\pvc
-2a802f2a-2732-4477-b9af-eba414a72087\mount' because it does not exist.
At line:1 char:2
+ (Get-Item -Path c:\var\lib\kubelet\pods\2e14f23e-9881-40f8-bd55-ba67f ...
+  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : ObjectNotFound: (C:\var\lib\kube...414a72087\mount:String) [Get-Item], ItemNotFoundExcep 
   tion
... skipping 37 lines ...
+ ... d : PathNotFound,Microsoft.PowerShell.Commands.GetItemCommand).Target
+                                                                  ~
Unexpected token ')' in expression or statement.
    + CategoryInfo          : ParserError: (:) [], ParentContainsErrorRecordException
    + FullyQualifiedErrorId : MissingEndParenthesisInExpression
 
, error: exit status 1, response: <nil>
I0117 20:52:01.168451    6004 nodeserver_v2.go:375] NodeUnpublishVolume: unmount volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-ri100d/providers/Microsoft.Compute/disks/pvc-2a802f2a-2732-4477-b9af-eba414a72087 on c:\var\lib\kubelet\pods\2e14f23e-9881-40f8-bd55-ba67f4ea4fe8\volumes\kubernetes.io~csi\pvc-2a802f2a-2732-4477-b9af-eba414a72087\mount successfully
I0117 20:52:01.168451    6004 utils.go:85] GRPC response: {}
I0117 20:52:01.271223    6004 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0117 20:52:01.271256    6004 utils.go:79] GRPC request: {}
I0117 20:52:01.271293    6004 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I0117 20:52:01.272705    6004 utils.go:78] GRPC call: /csi.v1.Node/NodeUnstageVolume
... skipping 34 lines ...
I0117 20:53:04.352403    6004 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0117 20:53:04.352403    6004 utils.go:79] GRPC request: {}
I0117 20:53:04.352484    6004 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I0117 20:53:04.353697    6004 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0117 20:53:04.353697    6004 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-d125a20e-7c64-4017-bd41-6edcd24569ac\\globalmount","volume_capability":{"AccessType":{"Mount":{"mount_flags":["barrier=1","acl"]}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-d125a20e-7c64-4017-bd41-6edcd24569ac","csi.storage.k8s.io/pvc/name":"pvc-jp2x5","csi.storage.k8s.io/pvc/namespace":"azuredisk-6615","requestedsizegib":"10","skuname":"Standard_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1673987143207-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-ri100d/providers/Microsoft.Compute/disks/pvc-d125a20e-7c64-4017-bd41-6edcd24569ac"}
I0117 20:53:04.354292    6004 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-d125a20e-7c64-4017-bd41-6edcd24569ac-capz-ri10-ppsff-attachment)
I0117 20:53:14.216952    6004 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 20:53:14.216952    6004 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="efffb04d-96a8-11ed-a767-000d3a65271f" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-d125a20e-7c64-4017-bd41-6edcd24569ac" "latency"=9862479100
I0117 20:53:14.217617    6004 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="efffb04d-96a8-11ed-a767-000d3a65271f" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9863759400
I0117 20:53:18.166641    6004 nodeserver_v2.go:211] NodeStageVolume: formatting 10 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-d125a20e-7c64-4017-bd41-6edcd24569ac\globalmount with mount options([barrier=1 acl])
I0117 20:53:18.737220    6004 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 20:53:18.737220    6004 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync
I0117 20:53:18.855718    6004 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
... skipping 71 lines ...
I0117 20:54:46.445554    6004 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I0117 20:54:46.447416    6004 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0117 20:54:46.447416    6004 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-ad240643-fdd7-4c1b-bd72-b833cf89dc0b\\globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-ad240643-fdd7-4c1b-bd72-b833cf89dc0b","csi.storage.k8s.io/pvc/name":"pvc-t4t66","csi.storage.k8s.io/pvc/namespace":"azuredisk-4970","enableAsyncAttach":"false","enableasyncattach":"false","networkAccessPolicy":"DenyAll","requestedsizegib":"10","skuName":"Standard_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1673987143207-8081-disk.csi.azure.com","userAgent":"azuredisk-e2e-test"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-ri100d/providers/Microsoft.Compute/disks/pvc-ad240643-fdd7-4c1b-bd72-b833cf89dc0b"}
I0117 20:54:46.447416    6004 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-ad240643-fdd7-4c1b-bd72-b833cf89dc0b-capz-ri10-ppsff-attachment)
I0117 20:54:48.744923    6004 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 20:54:48.745144    6004 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync
I0117 20:54:48.745144    6004 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0117 20:54:48.875188    6004 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 20:54:49.431953    6004 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 20:54:49.432632    6004 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="2cd9f435-96a9-11ed-a767-000d3a65271f" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-ad240643-fdd7-4c1b-bd72-b833cf89dc0b" "latency"=2985175000
I0117 20:54:49.432632    6004 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="2cd9f435-96a9-11ed-a767-000d3a65271f" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=2985175000
I0117 20:54:49.469302    6004 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 20:54:49.633305    6004 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 20:54:53.540162    6004 nodeserver_v2.go:211] NodeStageVolume: formatting 10 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-ad240643-fdd7-4c1b-bd72-b833cf89dc0b\globalmount with mount options([])
I0117 20:54:53.784042    6004 utils.go:78] GRPC call: /csi.v1.Identity/Probe
... skipping 86 lines ...
I0117 20:56:38.966340    6004 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0117 20:56:38.966379    6004 utils.go:79] GRPC request: {}
I0117 20:56:38.966437    6004 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I0117 20:56:38.967526    6004 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0117 20:56:38.967526    6004 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-dc1acede-46e2-48e8-9a2c-a5abfa872f05\\globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-dc1acede-46e2-48e8-9a2c-a5abfa872f05","csi.storage.k8s.io/pvc/name":"pvc-ls478","csi.storage.k8s.io/pvc/namespace":"azuredisk-6530","enableAsyncAttach":"false","enableBursting":"true","enableasyncattach":"false","perfProfile":"Basic","requestedsizegib":"1024","skuname":"Premium_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1673987143207-8081-disk.csi.azure.com","userAgent":"azuredisk-e2e-test"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-ri100d/providers/Microsoft.Compute/disks/pvc-dc1acede-46e2-48e8-9a2c-a5abfa872f05"}
I0117 20:56:38.967526    6004 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-dc1acede-46e2-48e8-9a2c-a5abfa872f05-capz-ri10-ppsff-attachment)
I0117 20:56:48.034913    6004 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 20:56:48.034913    6004 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="6feb2900-96a9-11ed-a767-000d3a65271f" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-dc1acede-46e2-48e8-9a2c-a5abfa872f05" "latency"=9067219100
I0117 20:56:48.034913    6004 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="6feb2900-96a9-11ed-a767-000d3a65271f" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9067219100
I0117 20:56:48.778117    6004 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync
I0117 20:56:48.778048    6004 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 20:56:48.916736    6004 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 20:56:49.491268    6004 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
... skipping 83 lines ...
I0117 20:58:23.784433    6004 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I0117 20:58:23.784494    6004 utils.go:79] GRPC request: {}
I0117 20:58:23.784494    6004 utils.go:85] GRPC response: {"ready":{"value":true}}
I0117 20:58:28.616870    6004 azuredisk_v2.go:578]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-17T20:58:28Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="b1464b36-96a9-11ed-a767-000d3a65271f"
I0117 20:58:28.628725    6004 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-ri10-ppsff/status 200 OK in 11 milliseconds
I0117 20:58:28.631851    6004 azuredisk_v2.go:565]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="b1464b36-96a9-11ed-a767-000d3a65271f" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=15168000
I0117 20:58:30.267305    6004 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 20:58:30.267305    6004 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="acc8747e-96a9-11ed-a767-000d3a65271f" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-12d12ae2-b6a2-4170-a917-98abe1860bf7" "latency"=9186170800
I0117 20:58:30.267305    6004 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="acc8747e-96a9-11ed-a767-000d3a65271f" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9186170800
I0117 20:58:34.354507    6004 nodeserver_v2.go:211] NodeStageVolume: formatting 10 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-12d12ae2-b6a2-4170-a917-98abe1860bf7\globalmount with mount options([])
I0117 20:58:48.812774    6004 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync
I0117 20:58:48.812774    6004 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 20:58:48.957610    6004 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
... skipping 69 lines ...
I0117 21:00:04.106806    6004 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I0117 21:00:04.107943    6004 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0117 21:00:04.107943    6004 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-3e929c22-2b53-4acd-b131-0b489234b929\\globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-3e929c22-2b53-4acd-b131-0b489234b929","csi.storage.k8s.io/pvc/name":"pvc-j5q78","csi.storage.k8s.io/pvc/namespace":"azuredisk-5474","requestedsizegib":"10","skuname":"Premium_ZRS","storage.kubernetes.io/csiProvisionerIdentity":"1673987143207-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-ri100d/providers/Microsoft.Compute/disks/pvc-3e929c22-2b53-4acd-b131-0b489234b929"}
I0117 21:00:04.107943    6004 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-3e929c22-2b53-4acd-b131-0b489234b929-capz-ri10-ppsff-attachment)
I0117 21:00:18.846960    6004 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 21:00:18.847095    6004 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync
I0117 21:00:18.847151    6004 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0117 21:00:18.970914    6004 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 21:00:19.589009    6004 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 21:00:19.722769    6004 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 21:00:23.782308    6004 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I0117 21:00:23.782308    6004 utils.go:79] GRPC request: {}
I0117 21:00:23.782308    6004 utils.go:85] GRPC response: {"ready":{"value":true}}
I0117 21:00:28.615329    6004 azuredisk_v2.go:578]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-17T21:00:28Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="f8cca24b-96a9-11ed-a767-000d3a65271f"
I0117 21:00:28.633356    6004 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-ri10-ppsff/status 200 OK in 10 milliseconds
I0117 21:00:28.633356    6004 azuredisk_v2.go:565]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="f8cca24b-96a9-11ed-a767-000d3a65271f" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=18246100
I0117 21:00:38.995151    6004 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 21:00:38.995151    6004 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="ea311a8e-96a9-11ed-a767-000d3a65271f" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-3e929c22-2b53-4acd-b131-0b489234b929" "latency"=34886708800
I0117 21:00:38.995151    6004 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="ea311a8e-96a9-11ed-a767-000d3a65271f" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=34886708800
I0117 21:00:43.115051    6004 nodeserver_v2.go:211] NodeStageVolume: formatting 10 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-3e929c22-2b53-4acd-b131-0b489234b929\globalmount with mount options([])
I0117 21:00:48.864236    6004 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 21:00:48.864236    6004 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync
I0117 21:00:48.973023    6004 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
... skipping 80 lines ...
I0117 21:02:16.461507    6004 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I0117 21:02:16.462824    6004 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0117 21:02:16.462824    6004 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-af5535c1-24c6-4a7d-97cd-5ad16ff79ada\\globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-af5535c1-24c6-4a7d-97cd-5ad16ff79ada","csi.storage.k8s.io/pvc/name":"pvc-zxl74","csi.storage.k8s.io/pvc/namespace":"azuredisk-2839","requestedsizegib":"10","skuname":"StandardSSD_ZRS","storage.kubernetes.io/csiProvisionerIdentity":"1673987143207-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-ri100d/providers/Microsoft.Compute/disks/pvc-af5535c1-24c6-4a7d-97cd-5ad16ff79ada"}
I0117 21:02:16.462824    6004 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-af5535c1-24c6-4a7d-97cd-5ad16ff79ada-capz-ri10-ppsff-attachment)
I0117 21:02:18.883213    6004 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync
I0117 21:02:18.883213    6004 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 21:02:18.883500    6004 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0117 21:02:18.981387    6004 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 21:02:19.640508    6004 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 21:02:19.764762    6004 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 21:02:21.167277    6004 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: Watch close - *v1beta2.AzDriverNode total 20 items received
I0117 21:02:21.170628    6004 round_trippers.go:553] GET https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dcapz-ri10-ppsff&resourceVersion=13865&timeout=9m26s&timeoutSeconds=566&watch=true 200 OK in 3 milliseconds
I0117 21:02:23.782272    6004 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I0117 21:02:23.782272    6004 utils.go:79] GRPC request: {}
I0117 21:02:23.782272    6004 utils.go:85] GRPC response: {"ready":{"value":true}}
I0117 21:02:28.629207    6004 azuredisk_v2.go:578]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-17T21:02:28Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="40554c6a-96aa-11ed-a767-000d3a65271f"
I0117 21:02:28.644782    6004 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-ri10-ppsff/status 200 OK in 12 milliseconds
I0117 21:02:28.644852    6004 azuredisk_v2.go:565]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="40554c6a-96aa-11ed-a767-000d3a65271f" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=15644500
I0117 21:02:45.216915    6004 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 21:02:45.216915    6004 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="3914db11-96aa-11ed-a767-000d3a65271f" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-af5535c1-24c6-4a7d-97cd-5ad16ff79ada" "latency"=28753990800
I0117 21:02:45.216915    6004 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="3914db11-96aa-11ed-a767-000d3a65271f" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=28753990800
I0117 21:02:48.893072    6004 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync
I0117 21:02:48.893072    6004 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 21:02:48.985417    6004 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 21:02:49.238669    6004 nodeserver_v2.go:211] NodeStageVolume: formatting 10 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-af5535c1-24c6-4a7d-97cd-5ad16ff79ada\globalmount with mount options([])
... skipping 43 lines ...
I0117 21:03:28.316719    6004 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0117 21:03:28.316719    6004 utils.go:79] GRPC request: {"publish_context":{"LUN":"1"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-5a37ee85-19f8-4a74-90c1-35cb69f739ce\\globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-5a37ee85-19f8-4a74-90c1-35cb69f739ce","csi.storage.k8s.io/pvc/name":"pvc-pxnmp","csi.storage.k8s.io/pvc/namespace":"azuredisk-2839","requestedsizegib":"10","skuname":"StandardSSD_ZRS","storage.kubernetes.io/csiProvisionerIdentity":"1673987143207-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-ri100d/providers/Microsoft.Compute/disks/pvc-5a37ee85-19f8-4a74-90c1-35cb69f739ce"}
I0117 21:03:28.317169    6004 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-5a37ee85-19f8-4a74-90c1-35cb69f739ce-capz-ri10-ppsff-attachment)
I0117 21:03:28.615896    6004 azuredisk_v2.go:578]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-17T21:03:28Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="641687cf-96aa-11ed-a767-000d3a65271f"
I0117 21:03:28.628954    6004 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-ri10-ppsff/status 200 OK in 12 milliseconds
I0117 21:03:28.628995    6004 azuredisk_v2.go:565]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="641687cf-96aa-11ed-a767-000d3a65271f" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=13194100
I0117 21:03:33.233251    6004 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 21:03:33.233423    6004 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="63e8f31e-96aa-11ed-a767-000d3a65271f" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-5a37ee85-19f8-4a74-90c1-35cb69f739ce" "latency"=4916073000
I0117 21:03:33.233467    6004 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="63e8f31e-96aa-11ed-a767-000d3a65271f" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=4916282300
I0117 21:03:37.309782    6004 nodeserver_v2.go:211] NodeStageVolume: formatting 13 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-5a37ee85-19f8-4a74-90c1-35cb69f739ce\globalmount with mount options([])
I0117 21:03:39.447446    6004 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0117 21:03:39.447446    6004 utils.go:79] GRPC request: {}
I0117 21:03:39.447446    6004 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
... skipping 46 lines ...
I0117 21:04:14.049636    6004 utils.go:78] GRPC call: /csi.v1.Node/NodeGetVolumeStats
I0117 21:04:14.050177    6004 utils.go:79] GRPC request: {"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-ri100d/providers/Microsoft.Compute/disks/pvc-5a37ee85-19f8-4a74-90c1-35cb69f739ce","volume_path":"c:\\var\\lib\\kubelet\\pods\\852ab65d-ce9c-4ab4-9165-f4d97fae65fd\\volumes\\kubernetes.io~csi\\pvc-5a37ee85-19f8-4a74-90c1-35cb69f739ce\\mount"}
I0117 21:04:14.519425    6004 safe_mounter_windows.go:356] GetVolumeStats(c:\var\lib\kubelet\pods\852ab65d-ce9c-4ab4-9165-f4d97fae65fd\volumes\kubernetes.io~csi\pvc-5a37ee85-19f8-4a74-90c1-35cb69f739ce\mount) returned volumeID(\\?\Volume{72c22f9e-c631-4967-93fd-66a2fd14f204}\)
I0117 21:04:16.643851    6004 utils.go:85] GRPC response: {"usage":[{"available":10677248000,"total":10719588352,"unit":1,"used":42340352}]}
I0117 21:04:18.940010    6004 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync
I0117 21:04:18.940010    6004 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 21:04:18.940010    6004 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0117 21:04:19.016773    6004 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 21:04:19.679269    6004 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 21:04:19.779786    6004 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 21:04:20.399039    6004 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 21:04:20.399039    6004 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="7d203bd6-96aa-11ed-a767-000d3a65271f" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-62de521a-131c-4664-b087-3dc3d324dfdf" "latency"=9776484600
I0117 21:04:20.399581    6004 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="7d203bd6-96aa-11ed-a767-000d3a65271f" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9776484600
I0117 21:04:23.778940    6004 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I0117 21:04:23.778940    6004 utils.go:79] GRPC request: {}
I0117 21:04:23.778940    6004 utils.go:85] GRPC response: {"ready":{"value":true}}
I0117 21:04:24.499913    6004 nodeserver_v2.go:211] NodeStageVolume: formatting 16 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-62de521a-131c-4664-b087-3dc3d324dfdf\globalmount with mount options([])
... skipping 140 lines ...
I0117 21:07:23.102805    6004 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0117 21:07:23.102805    6004 utils.go:79] GRPC request: {}
I0117 21:07:23.102805    6004 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I0117 21:07:23.104316    6004 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0117 21:07:23.104335    6004 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-956fabab-1cb2-4e02-86c4-837bcc2aa63a\\globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-956fabab-1cb2-4e02-86c4-837bcc2aa63a","csi.storage.k8s.io/pvc/name":"pvc-88k5h","csi.storage.k8s.io/pvc/namespace":"azuredisk-1183","requestedsizegib":"10","skuname":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1673987143207-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-ri100d/providers/Microsoft.Compute/disks/pvc-956fabab-1cb2-4e02-86c4-837bcc2aa63a"}
I0117 21:07:23.104772    6004 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-956fabab-1cb2-4e02-86c4-837bcc2aa63a-capz-ri10-ppsff-attachment)
I0117 21:07:23.682625    6004 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 21:07:23.682782    6004 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="efdaaec8-96aa-11ed-a767-000d3a65271f" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-956fabab-1cb2-4e02-86c4-837bcc2aa63a" "latency"=577886000
I0117 21:07:23.682829    6004 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="efdaaec8-96aa-11ed-a767-000d3a65271f" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=578187600
I0117 21:07:23.794001    6004 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I0117 21:07:23.794001    6004 utils.go:79] GRPC request: {}
I0117 21:07:23.794001    6004 utils.go:85] GRPC response: {"ready":{"value":true}}
I0117 21:07:27.782190    6004 nodeserver_v2.go:211] NodeStageVolume: formatting 10 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-956fabab-1cb2-4e02-86c4-837bcc2aa63a\globalmount with mount options([])
... skipping 140 lines ...
I0117 21:09:45.961288    6004 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-cbf2858c-5e36-49fa-a182-f0d2dcfbe29c-capz-ri10-ppsff-attachment)
I0117 21:09:45.961288    6004 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0117 21:09:45.961288    6004 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-f99f2be0-b4d3-4055-acc1-6337ddb600da\\globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-f99f2be0-b4d3-4055-acc1-6337ddb600da","csi.storage.k8s.io/pvc/name":"pvc-v2pmf","csi.storage.k8s.io/pvc/namespace":"azuredisk-3565","requestedsizegib":"10","skuname":"StandardSSD_ZRS","storage.kubernetes.io/csiProvisionerIdentity":"1673987143207-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-ri100d/providers/Microsoft.Compute/disks/pvc-f99f2be0-b4d3-4055-acc1-6337ddb600da"}
I0117 21:09:45.961288    6004 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-f99f2be0-b4d3-4055-acc1-6337ddb600da-capz-ri10-ppsff-attachment)
I0117 21:09:49.011638    6004 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync
I0117 21:09:49.011674    6004 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 21:09:49.011906    6004 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0117 21:09:49.011906    6004 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0117 21:09:49.011906    6004 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0117 21:09:49.135761    6004 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 21:09:49.785095    6004 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 21:09:49.867309    6004 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 21:09:53.787108    6004 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I0117 21:09:53.787108    6004 utils.go:79] GRPC request: {}
I0117 21:09:53.787218    6004 utils.go:85] GRPC response: {"ready":{"value":true}}
I0117 21:09:58.620073    6004 azuredisk_v2.go:578]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-17T21:09:58Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="4c8c74a3-96ab-11ed-a768-000d3a65271f"
I0117 21:09:58.637588    6004 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-ri10-ppsff/status 200 OK in 11 milliseconds
I0117 21:09:58.638212    6004 azuredisk_v2.go:565]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="4c8c74a3-96ab-11ed-a768-000d3a65271f" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=17606200
I0117 21:10:01.505839    6004 reflector.go:559] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: Watch close - *v1.CustomResourceDefinition total 0 items received
I0117 21:10:01.513614    6004 round_trippers.go:553] GET https://10.96.0.1:443/apis/apiextensions.k8s.io/v1/customresourcedefinitions?allowWatchBookmarks=true&resourceVersion=13716&timeout=7m50s&timeoutSeconds=470&watch=true 200 OK in 7 milliseconds
I0117 21:10:19.025704    6004 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync
I0117 21:10:19.025767    6004 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 21:10:19.025767    6004 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0117 21:10:19.025767    6004 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0117 21:10:19.025767    6004 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0117 21:10:19.143483    6004 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 21:10:19.210625    6004 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 21:10:19.210625    6004 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="4500e0d4-96ab-11ed-a768-000d3a65271f" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-f99f2be0-b4d3-4055-acc1-6337ddb600da" "latency"=33249135400
I0117 21:10:19.211598    6004 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="4500e0d4-96ab-11ed-a768-000d3a65271f" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=33249816700
I0117 21:10:19.214956    6004 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 21:10:19.215114    6004 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="4500d477-96ab-11ed-a767-000d3a65271f" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-ae7b4552-8ccd-4881-97d2-69d7fc252850" "latency"=33253678500
I0117 21:10:19.215114    6004 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="4500d477-96ab-11ed-a767-000d3a65271f" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=33253940800
I0117 21:10:19.217951    6004 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 21:10:19.218062    6004 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="4500e0d4-96ab-11ed-a767-000d3a65271f" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-cbf2858c-5e36-49fa-a182-f0d2dcfbe29c" "latency"=33256488500
I0117 21:10:19.218100    6004 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="4500e0d4-96ab-11ed-a767-000d3a65271f" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=33256572000
I0117 21:10:19.793003    6004 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 21:10:19.868887    6004 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 21:10:23.782807    6004 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I0117 21:10:23.782807    6004 utils.go:79] GRPC request: {}
... skipping 139 lines ...
I0117 21:12:18.893003    6004 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I0117 21:12:18.894421    6004 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0117 21:12:18.894421    6004 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-1ec5be71-3969-45d8-8f62-9ffa7a56990f\\globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-1ec5be71-3969-45d8-8f62-9ffa7a56990f","csi.storage.k8s.io/pvc/name":"pvc-7k529","csi.storage.k8s.io/pvc/namespace":"azuredisk-273","requestedsizegib":"10","skuname":"StandardSSD_ZRS","storage.kubernetes.io/csiProvisionerIdentity":"1673987143207-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-ri100d/providers/Microsoft.Compute/disks/pvc-1ec5be71-3969-45d8-8f62-9ffa7a56990f"}
I0117 21:12:18.894421    6004 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-1ec5be71-3969-45d8-8f62-9ffa7a56990f-capz-ri10-ppsff-attachment)
I0117 21:12:19.054827    6004 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 21:12:19.054827    6004 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync
I0117 21:12:19.054827    6004 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0117 21:12:19.179689    6004 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 21:12:19.816527    6004 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 21:12:19.891993    6004 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 21:12:23.797693    6004 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I0117 21:12:23.797693    6004 utils.go:79] GRPC request: {}
I0117 21:12:23.797693    6004 utils.go:85] GRPC response: {"ready":{"value":true}}
I0117 21:12:28.628420    6004 azuredisk_v2.go:578]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-17T21:12:28Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="a5f5e9af-96ab-11ed-a768-000d3a65271f"
I0117 21:12:28.646059    6004 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-ri10-ppsff/status 200 OK in 17 milliseconds
I0117 21:12:28.646059    6004 azuredisk_v2.go:565]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="a5f5e9af-96ab-11ed-a768-000d3a65271f" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=17638600
I0117 21:12:49.072870    6004 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 21:12:49.072870    6004 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0117 21:12:49.072870    6004 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync
I0117 21:12:49.186973    6004 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 21:12:49.832423    6004 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 21:12:49.899049    6004 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 21:12:50.764382    6004 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 21:12:50.764973    6004 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="a0289f54-96ab-11ed-a768-000d3a65271f" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-1ec5be71-3969-45d8-8f62-9ffa7a56990f" "latency"=31869547300
I0117 21:12:50.764973    6004 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="a0289f54-96ab-11ed-a768-000d3a65271f" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=31870138500
I0117 21:12:53.787344    6004 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I0117 21:12:53.787344    6004 utils.go:79] GRPC request: {}
I0117 21:12:53.787509    6004 utils.go:85] GRPC response: {"ready":{"value":true}}
I0117 21:12:54.887546    6004 nodeserver_v2.go:211] NodeStageVolume: formatting 10 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-1ec5be71-3969-45d8-8f62-9ffa7a56990f\globalmount with mount options([])
... skipping 178 lines ...
I0117 21:17:24.674593    6004 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0117 21:17:24.674593    6004 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-ffa3200d-4515-4a5a-85ab-c76e504d0dba\\globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-ffa3200d-4515-4a5a-85ab-c76e504d0dba","csi.storage.k8s.io/pvc/name":"pvc-smbsh","csi.storage.k8s.io/pvc/namespace":"azuredisk-8041","requestedsizegib":"10","skuname":"StandardSSD_ZRS","storage.kubernetes.io/csiProvisionerIdentity":"1673987143207-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-ri100d/providers/Microsoft.Compute/disks/pvc-ffa3200d-4515-4a5a-85ab-c76e504d0dba"}
I0117 21:17:24.675510    6004 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-ffa3200d-4515-4a5a-85ab-c76e504d0dba-capz-ri10-ppsff-attachment)
I0117 21:17:28.632510    6004 azuredisk_v2.go:578]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-17T21:17:28Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="58c6e078-96ac-11ed-a768-000d3a65271f"
I0117 21:17:28.651237    6004 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-ri10-ppsff/status 200 OK in 11 milliseconds
I0117 21:17:28.651853    6004 azuredisk_v2.go:565]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="58c6e078-96ac-11ed-a768-000d3a65271f" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=19521400
I0117 21:17:33.511740    6004 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 21:17:33.512520    6004 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="566b1af7-96ac-11ed-a768-000d3a65271f" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-ffa3200d-4515-4a5a-85ab-c76e504d0dba" "latency"=8836149900
I0117 21:17:33.512602    6004 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="566b1af7-96ac-11ed-a768-000d3a65271f" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=8837119500
I0117 21:17:37.686571    6004 nodeserver_v2.go:211] NodeStageVolume: formatting 10 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-ffa3200d-4515-4a5a-85ab-c76e504d0dba\globalmount with mount options([])
I0117 21:17:48.170861    6004 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: Watch close - *v1beta2.AzDriverNode total 19 items received
I0117 21:17:48.191256    6004 round_trippers.go:553] GET https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dcapz-ri10-ppsff&resourceVersion=18627&timeout=8m12s&timeoutSeconds=492&watch=true 200 OK in 11 milliseconds
I0117 21:17:48.462670    6004 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: Watch close - *v1beta2.AzVolumeAttachment total 54 items received
... skipping 301 lines ...
I0117 21:25:14.044707    6004 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-3e0d4726-4b6e-4dd2-a295-d4087824d228-capz-ri10-ppsff-attachment)
I0117 21:25:14.043337    6004 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I0117 21:25:14.044707    6004 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-2932dda6-9bda-46cd-a75e-b5fa806ed28e-capz-ri10-ppsff-attachment)
I0117 21:25:14.045868    6004 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0117 21:25:14.045905    6004 utils.go:79] GRPC request: {"publish_context":{"LUN":"1"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-7da24f0f-e324-4271-bd74-607ec3fd77d5\\globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-7da24f0f-e324-4271-bd74-607ec3fd77d5","csi.storage.k8s.io/pvc/name":"pvc-4bg5l","csi.storage.k8s.io/pvc/namespace":"azuredisk-7624","requestedsizegib":"10","skuname":"Standard_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1673987143207-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-ri100d/providers/Microsoft.Compute/disks/pvc-7da24f0f-e324-4271-bd74-607ec3fd77d5"}
I0117 21:25:14.046196    6004 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-7da24f0f-e324-4271-bd74-607ec3fd77d5-capz-ri10-ppsff-attachment)
I0117 21:25:15.764696    6004 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 21:25:15.764856    6004 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="6e2f29fc-96ad-11ed-a768-000d3a65271f" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-3e0d4726-4b6e-4dd2-a295-d4087824d228" "latency"=1720044000
I0117 21:25:15.764893    6004 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="6e2f29fc-96ad-11ed-a768-000d3a65271f" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=1720158600
I0117 21:25:15.774400    6004 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 21:25:15.774400    6004 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="6e2f62ff-96ad-11ed-a768-000d3a65271f" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-7da24f0f-e324-4271-bd74-607ec3fd77d5" "latency"=1728131800
I0117 21:25:15.774400    6004 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="6e2f62ff-96ad-11ed-a768-000d3a65271f" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=1728242800
I0117 21:25:15.776655    6004 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 21:25:15.777227    6004 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="6e2f2b60-96ad-11ed-a768-000d3a65271f" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-2932dda6-9bda-46cd-a75e-b5fa806ed28e" "latency"=1731922400
I0117 21:25:15.777227    6004 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="6e2f2b60-96ad-11ed-a768-000d3a65271f" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=1732493800
I0117 21:25:19.295447    6004 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync
I0117 21:25:19.295447    6004 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 21:25:19.450700    6004 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 21:25:20.047129    6004 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
... skipping 622 lines ...
I0117 21:47:24.073937    6004 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0117 21:47:24.074026    6004 utils.go:79] GRPC request: {}
I0117 21:47:24.074026    6004 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I0117 21:47:24.074612    6004 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0117 21:47:24.075149    6004 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-aa5754cf-8d1d-419f-be98-e054e54aa91a\\globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"cachingmode":"None","csi.storage.k8s.io/pv/name":"pvc-aa5754cf-8d1d-419f-be98-e054e54aa91a","csi.storage.k8s.io/pvc/name":"pvc-wc4dh","csi.storage.k8s.io/pvc/namespace":"azuredisk-5210","maxshares":"1","requestedsizegib":"10","skuname":"Premium_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1673987143207-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-ri100d/providers/Microsoft.Compute/disks/pvc-aa5754cf-8d1d-419f-be98-e054e54aa91a"}
I0117 21:47:24.075930    6004 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-aa5754cf-8d1d-419f-be98-e054e54aa91a-capz-ri10-ppsff-attachment)
I0117 21:47:27.078880    6004 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 21:47:27.078880    6004 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="86f1d408-96b0-11ed-a768-000d3a65271f" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-aa5754cf-8d1d-419f-be98-e054e54aa91a" "latency"=3002946500
I0117 21:47:27.078880    6004 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="86f1d408-96b0-11ed-a768-000d3a65271f" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=3002946500
I0117 21:47:28.641353    6004 azuredisk_v2.go:578]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-17T21:47:28Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="89aa6b19-96b0-11ed-a768-000d3a65271f"
I0117 21:47:28.654850    6004 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-ri10-ppsff/status 200 OK in 13 milliseconds
I0117 21:47:28.655433    6004 azuredisk_v2.go:565]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="89aa6b19-96b0-11ed-a768-000d3a65271f" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=14421900
I0117 21:47:31.868844    6004 nodeserver_v2.go:211] NodeStageVolume: formatting 10 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-aa5754cf-8d1d-419f-be98-e054e54aa91a\globalmount with mount options([])
... skipping 29 lines ...
I0117 21:48:04.605749    6004 nodeserver_v2.go:369] NodeUnpublishVolume: unmounting volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-ri100d/providers/Microsoft.Compute/disks/pvc-aa5754cf-8d1d-419f-be98-e054e54aa91a on c:\var\lib\kubelet\pods\748b2a91-897f-4add-a355-4934db95e95c\volumes\kubernetes.io~csi\pvc-aa5754cf-8d1d-419f-be98-e054e54aa91a\mount
I0117 21:48:07.230052    6004 nodeserver_v2.go:375] NodeUnpublishVolume: unmount volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-ri100d/providers/Microsoft.Compute/disks/pvc-aa5754cf-8d1d-419f-be98-e054e54aa91a on c:\var\lib\kubelet\pods\748b2a91-897f-4add-a355-4934db95e95c\volumes\kubernetes.io~csi\pvc-aa5754cf-8d1d-419f-be98-e054e54aa91a\mount successfully
I0117 21:48:07.230052    6004 utils.go:85] GRPC response: {}
I0117 21:48:07.760425    6004 utils.go:78] GRPC call: /csi.v1.Node/NodeUnpublishVolume
I0117 21:48:07.760425    6004 utils.go:79] GRPC request: {"target_path":"c:\\var\\lib\\kubelet\\pods\\748b2a91-897f-4add-a355-4934db95e95c\\volumes\\kubernetes.io~csi\\pvc-aa5754cf-8d1d-419f-be98-e054e54aa91a\\mount","volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-ri100d/providers/Microsoft.Compute/disks/pvc-aa5754cf-8d1d-419f-be98-e054e54aa91a"}
I0117 21:48:07.760425    6004 nodeserver_v2.go:369] NodeUnpublishVolume: unmounting volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-ri100d/providers/Microsoft.Compute/disks/pvc-aa5754cf-8d1d-419f-be98-e054e54aa91a on c:\var\lib\kubelet\pods\748b2a91-897f-4add-a355-4934db95e95c\volumes\kubernetes.io~csi\pvc-aa5754cf-8d1d-419f-be98-e054e54aa91a\mount
W0117 21:48:08.401058    6004 safe_mounter_windows.go:113] GetVolumeIDFromTargetPath(c:\var\lib\kubelet\pods\748b2a91-897f-4add-a355-4934db95e95c\volumes\kubernetes.io~csi\pvc-aa5754cf-8d1d-419f-be98-e054e54aa91a\mount) failed with error: rpc error: code = Unknown desc = error getting the volume for the mount c:\var\lib\kubelet\pods\748b2a91-897f-4add-a355-4934db95e95c\volumes\kubernetes.io~csi\pvc-aa5754cf-8d1d-419f-be98-e054e54aa91a\mount, internal error error getting volume from mount. cmd: (Get-Item -Path Get-Item : Cannot find path 'C:\var\lib\kubelet\pods\748b2a91-897f-4add-a355-4934db95e95c\volumes\kubernetes.io~csi\pvc
-aa5754cf-8d1d-419f-be98-e054e54aa91a\mount' because it does not exist.
At line:1 char:2
+ (Get-Item -Path c:\var\lib\kubelet\pods\748b2a91-897f-4add-a355-4934d ...
+  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : ObjectNotFound: (C:\var\lib\kube...4e54aa91a\mount:String) [Get-Item], ItemNotFoundExcep 
   tion
... skipping 37 lines ...
+ ... d : PathNotFound,Microsoft.PowerShell.Commands.GetItemCommand).Target
+                                                                  ~
Unexpected token ')' in expression or statement.
    + CategoryInfo          : ParserError: (:) [], ParentContainsErrorRecordException
    + FullyQualifiedErrorId : MissingEndParenthesisInExpression
 
, error: exit status 1, response: <nil>
I0117 21:48:08.407180    6004 nodeserver_v2.go:375] NodeUnpublishVolume: unmount volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-ri100d/providers/Microsoft.Compute/disks/pvc-aa5754cf-8d1d-419f-be98-e054e54aa91a on c:\var\lib\kubelet\pods\748b2a91-897f-4add-a355-4934db95e95c\volumes\kubernetes.io~csi\pvc-aa5754cf-8d1d-419f-be98-e054e54aa91a\mount successfully
I0117 21:48:08.407180    6004 utils.go:85] GRPC response: {}
I0117 21:48:08.502078    6004 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0117 21:48:08.502078    6004 utils.go:79] GRPC request: {}
I0117 21:48:08.502078    6004 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I0117 21:48:08.503672    6004 utils.go:78] GRPC call: /csi.v1.Node/NodeUnstageVolume
I0117 21:48:08.503699    6004 utils.go:79] GRPC request: {"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-aa5754cf-8d1d-419f-be98-e054e54aa91a\\globalmount","volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-ri100d/providers/Microsoft.Compute/disks/pvc-aa5754cf-8d1d-419f-be98-e054e54aa91a"}
I0117 21:48:08.503738    6004 nodeserver_v2.go:257] NodeUnstageVolume: unmounting \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-aa5754cf-8d1d-419f-be98-e054e54aa91a\globalmount
I0117 21:48:10.781070    6004 nodeserver_v2.go:262] NodeUnstageVolume: unmount \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-aa5754cf-8d1d-419f-be98-e054e54aa91a\globalmount successfully
I0117 21:48:10.781070    6004 utils.go:85] GRPC response: {}
W0117 21:48:17.616420    6004 reflector.go:347] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: watch of *v1beta2.AzVolume ended with: an error on the server ("unable to decode an event from the watch stream: read tcp 10.1.0.5:35295->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.") has prevented the request from succeeding
W0117 21:48:17.616420    6004 reflector.go:347] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: watch of *v1beta2.AzDriverNode ended with: an error on the server ("unable to decode an event from the watch stream: read tcp 10.1.0.5:35295->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.") has prevented the request from succeeding
I0117 21:48:17.616420    6004 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-ri10-ppsff/status  in 18974 milliseconds
W0117 21:48:17.616589    6004 reflector.go:347] k8s.io/client-go/informers/factory.go:150: watch of *v1.Node ended with: an error on the server ("unable to decode an event from the watch stream: read tcp 10.1.0.5:35295->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.") has prevented the request from succeeding
W0117 21:48:17.616589    6004 reflector.go:347] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: watch of *v1beta2.AzDriverNode ended with: an error on the server ("unable to decode an event from the watch stream: read tcp 10.1.0.5:35295->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.") has prevented the request from succeeding
W0117 21:48:17.616420    6004 reflector.go:347] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: watch of *v1.CustomResourceDefinition ended with: an error on the server ("unable to decode an event from the watch stream: read tcp 10.1.0.5:35295->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.") has prevented the request from succeeding
W0117 21:48:17.616420    6004 reflector.go:347] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: watch of *v1beta2.AzVolumeAttachment ended with: an error on the server ("unable to decode an event from the watch stream: read tcp 10.1.0.5:35295->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.") has prevented the request from succeeding
E0117 21:48:17.616589    6004 azuredisk_v2.go:613]  "msg"="Failed to update AzDriverNode status after creation" "error"="Put \"https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-ri10-ppsff/status\": read tcp 10.1.0.5:35295->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="9b8c1cbb-96b0-11ed-a768-000d3a65271f"
E0117 21:48:17.616589    6004 azuredisk_v2.go:565]  "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [Put \"https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-ri10-ppsff/status\": read tcp 10.1.0.5:35295->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.]" "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="9b8c1cbb-96b0-11ed-a768-000d3a65271f" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=18974874500
I0117 21:48:18.522152    6004 reflector.go:257] Listing and watching *v1beta2.AzVolumeAttachment from sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132
I0117 21:48:18.536893    6004 round_trippers.go:553] GET https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumeattachments?resourceVersion=27356 200 OK in 14 milliseconds
I0117 21:48:18.538084    6004 round_trippers.go:553] GET https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumeattachments?allowWatchBookmarks=true&resourceVersion=27586&timeout=9m10s&timeoutSeconds=550&watch=true 200 OK in 1 milliseconds
I0117 21:48:18.884296    6004 reflector.go:257] Listing and watching *v1beta2.AzDriverNode from sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132
I0117 21:48:18.887774    6004 round_trippers.go:553] GET https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes?fieldSelector=metadata.name%3Dcapz-ri10-ppsff&resourceVersion=27364 200 OK in 3 milliseconds
I0117 21:48:18.890421    6004 round_trippers.go:553] GET https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dcapz-ri10-ppsff&resourceVersion=27600&timeout=8m57s&timeoutSeconds=537&watch=true 200 OK in 2 milliseconds
... skipping 25 lines ...
I0117 21:48:47.503327    6004 utils.go:79] GRPC request: {}
I0117 21:48:47.503372    6004 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I0117 21:48:47.506055    6004 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0117 21:48:47.506055    6004 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-b3757d81-34f0-4e4a-8287-71da13330385\\globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"cachingmode":"None","csi.storage.k8s.io/pv/name":"pvc-b3757d81-34f0-4e4a-8287-71da13330385","csi.storage.k8s.io/pvc/name":"pvc-7rg2w","csi.storage.k8s.io/pvc/namespace":"azuredisk-4406","maxshares":"2","requestedsizegib":"10","skuname":"Premium_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1673987143207-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-ri100d/providers/Microsoft.Compute/disks/pvc-b3757d81-34f0-4e4a-8287-71da13330385"}
I0117 21:48:47.506633    6004 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-b3757d81-34f0-4e4a-8287-71da13330385-capz-ri10-ppsff-attachment)
I0117 21:48:48.544527    6004 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 21:48:48.544616    6004 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0117 21:48:48.894408    6004 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 21:48:49.036347    6004 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync
I0117 21:48:49.067597    6004 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 21:48:49.067597    6004 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 21:48:53.809181    6004 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I0117 21:48:53.809230    6004 utils.go:79] GRPC request: {}
I0117 21:48:53.809313    6004 utils.go:85] GRPC response: {"ready":{"value":true}}
I0117 21:48:57.249662    6004 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 21:48:57.252937    6004 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="b8ac57e1-96b0-11ed-a768-000d3a65271f" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-b3757d81-34f0-4e4a-8287-71da13330385" "latency"=9742853600
I0117 21:48:57.252937    6004 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="b8ac57e1-96b0-11ed-a768-000d3a65271f" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9746128800
I0117 21:48:58.656722    6004 azuredisk_v2.go:578]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-17T21:48:58Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="bf51ae00-96b0-11ed-a768-000d3a65271f"
I0117 21:48:58.670784    6004 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-ri10-ppsff/status 200 OK in 9 milliseconds
I0117 21:48:58.670784    6004 azuredisk_v2.go:565]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="bf51ae00-96b0-11ed-a768-000d3a65271f" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=14272500
I0117 21:49:01.360042    6004 nodeserver_v2.go:211] NodeStageVolume: formatting 10 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-b3757d81-34f0-4e4a-8287-71da13330385\globalmount with mount options([])
... skipping 527 lines ...
I0117 22:09:53.819182    6004 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I0117 22:09:53.819182    6004 utils.go:79] GRPC request: {}
I0117 22:09:53.819182    6004 utils.go:85] GRPC response: {"ready":{"value":true}}
I0117 22:09:58.659168    6004 azuredisk_v2.go:578]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-17T22:09:58Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="ae56d3c3-96b3-11ed-a768-000d3a65271f"
I0117 22:09:58.676610    6004 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-ri10-ppsff/status 200 OK in 13 milliseconds
I0117 22:09:58.676610    6004 azuredisk_v2.go:565]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="ae56d3c3-96b3-11ed-a768-000d3a65271f" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=17441600
I0117 22:10:00.587709    6004 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 22:10:00.587709    6004 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="ab5f743b-96b3-11ed-a768-000d3a65271f" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-5b2ee8af-f4f3-4398-8768-b99debc04e65" "latency"=6905029200
I0117 22:10:00.587709    6004 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="ab5f743b-96b3-11ed-a768-000d3a65271f" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=6905116400
I0117 22:10:05.414448    6004 nodeserver_v2.go:211] NodeStageVolume: formatting 11 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-5b2ee8af-f4f3-4398-8768-b99debc04e65\globalmount with mount options([barrier=1 acl])
I0117 22:10:18.872568    6004 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 22:10:19.228218    6004 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 22:10:19.346550    6004 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync
... skipping 29 lines ...
I0117 22:10:38.632809    6004 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I0117 22:10:38.633655    6004 utils.go:78] GRPC call: /csi.v1.Node/NodeUnstageVolume
I0117 22:10:38.633655    6004 utils.go:79] GRPC request: {"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-5b2ee8af-f4f3-4398-8768-b99debc04e65\\globalmount","volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-ri100d/providers/Microsoft.Compute/disks/pvc-5b2ee8af-f4f3-4398-8768-b99debc04e65"}
I0117 22:10:38.633655    6004 nodeserver_v2.go:257] NodeUnstageVolume: unmounting \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-5b2ee8af-f4f3-4398-8768-b99debc04e65\globalmount
I0117 22:10:41.012323    6004 nodeserver_v2.go:262] NodeUnstageVolume: unmount \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-5b2ee8af-f4f3-4398-8768-b99debc04e65\globalmount successfully
I0117 22:10:41.012323    6004 utils.go:85] GRPC response: {}
W0117 22:10:47.644794    6004 reflector.go:347] k8s.io/client-go/informers/factory.go:150: watch of *v1.Node ended with: an error on the server ("unable to decode an event from the watch stream: read tcp 10.1.0.5:41026->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.") has prevented the request from succeeding
I0117 22:10:47.644794    6004 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-ri10-ppsff/status  in 18979 milliseconds
W0117 22:10:47.644794    6004 reflector.go:347] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: watch of *v1beta2.AzVolumeAttachment ended with: an error on the server ("unable to decode an event from the watch stream: read tcp 10.1.0.5:41026->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.") has prevented the request from succeeding
W0117 22:10:47.644794    6004 reflector.go:347] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: watch of *v1beta2.AzDriverNode ended with: an error on the server ("unable to decode an event from the watch stream: read tcp 10.1.0.5:41026->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.") has prevented the request from succeeding
W0117 22:10:47.644794    6004 reflector.go:347] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: watch of *v1.CustomResourceDefinition ended with: an error on the server ("unable to decode an event from the watch stream: read tcp 10.1.0.5:41026->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.") has prevented the request from succeeding
W0117 22:10:47.644794    6004 reflector.go:347] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: watch of *v1beta2.AzDriverNode ended with: an error on the server ("unable to decode an event from the watch stream: read tcp 10.1.0.5:41026->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.") has prevented the request from succeeding
W0117 22:10:47.644794    6004 reflector.go:347] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: watch of *v1beta2.AzVolume ended with: an error on the server ("unable to decode an event from the watch stream: read tcp 10.1.0.5:41026->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.") has prevented the request from succeeding
E0117 22:10:47.644794    6004 azuredisk_v2.go:613]  "msg"="Failed to update AzDriverNode status after creation" "error"="Put \"https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-ri10-ppsff/status\": read tcp 10.1.0.5:41026->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="c03937c6-96b3-11ed-a768-000d3a65271f"
E0117 22:10:47.644794    6004 azuredisk_v2.go:565]  "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [Put \"https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-ri10-ppsff/status\": read tcp 10.1.0.5:41026->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.]" "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="c03937c6-96b3-11ed-a768-000d3a65271f" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=18980460100
I0117 22:10:48.533444    6004 reflector.go:257] Listing and watching *v1beta2.AzVolumeAttachment from sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132
I0117 22:10:48.540374    6004 reflector.go:257] Listing and watching *v1beta2.AzDriverNode from sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132
I0117 22:10:48.544387    6004 round_trippers.go:553] GET https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes?fieldSelector=metadata.name%3Dcapz-ri10-ppsff&resourceVersion=33882 200 OK in 4 milliseconds
I0117 22:10:48.544521    6004 round_trippers.go:553] GET https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumeattachments?resourceVersion=33890 200 OK in 11 milliseconds
I0117 22:10:48.546629    6004 round_trippers.go:553] GET https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dcapz-ri10-ppsff&resourceVersion=34061&timeout=8m34s&timeoutSeconds=514&watch=true 200 OK in 1 milliseconds
I0117 22:10:48.548066    6004 round_trippers.go:553] GET https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumeattachments?allowWatchBookmarks=true&resourceVersion=34127&timeout=9m14s&timeoutSeconds=554&watch=true 200 OK in 1 milliseconds
... skipping 37 lines ...
I0117 22:11:45.918997    6004 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I0117 22:11:45.920500    6004 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0117 22:11:45.920548    6004 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-687b1bee-92d6-4707-8497-e59679fcaa76\\globalmount","volume_capability":{"AccessType":{"Mount":{"mount_flags":["barrier=1","acl"]}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-687b1bee-92d6-4707-8497-e59679fcaa76","csi.storage.k8s.io/pvc/name":"pvc-6b6rk","csi.storage.k8s.io/pvc/namespace":"azuredisk-3208","requestedsizegib":"10","skuname":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1673987143207-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-ri100d/providers/Microsoft.Compute/disks/pvc-687b1bee-92d6-4707-8497-e59679fcaa76"}
I0117 22:11:45.920857    6004 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-687b1bee-92d6-4707-8497-e59679fcaa76-capz-ri10-ppsff-attachment)
I0117 22:11:48.551037    6004 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 22:11:48.551037    6004 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 22:11:48.551425    6004 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0117 22:11:48.672115    6004 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 22:11:48.697178    6004 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 22:11:48.713056    6004 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync
I0117 22:11:53.825582    6004 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I0117 22:11:53.825582    6004 utils.go:79] GRPC request: {}
I0117 22:11:53.825582    6004 utils.go:85] GRPC response: {"ready":{"value":true}}
I0117 22:11:58.178116    6004 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 22:11:58.178242    6004 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="ee45a748-96b3-11ed-a768-000d3a65271f" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-687b1bee-92d6-4707-8497-e59679fcaa76" "latency"=12257188300
I0117 22:11:58.178354    6004 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="ee45a748-96b3-11ed-a768-000d3a65271f" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=12257435400
I0117 22:11:58.657701    6004 azuredisk_v2.go:578]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-17T22:11:58Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="f5dd1fe9-96b3-11ed-a768-000d3a65271f"
I0117 22:11:58.667956    6004 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-ri10-ppsff/status 200 OK in 9 milliseconds
I0117 22:11:58.667956    6004 azuredisk_v2.go:565]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="f5dd1fe9-96b3-11ed-a768-000d3a65271f" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=10423100
I0117 22:12:02.285179    6004 nodeserver_v2.go:211] NodeStageVolume: formatting 10 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-687b1bee-92d6-4707-8497-e59679fcaa76\globalmount with mount options([barrier=1 acl])
... skipping 248 lines ...
I0117 22:17:23.825819    6004 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I0117 22:17:23.826385    6004 utils.go:79] GRPC request: {}
I0117 22:17:23.828019    6004 utils.go:85] GRPC response: {"ready":{"value":true}}
I0117 22:17:28.663050    6004 azuredisk_v2.go:578]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-17T22:17:28Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="ba8ff869-96b4-11ed-a768-000d3a65271f"
I0117 22:17:28.674127    6004 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-ri10-ppsff/status 200 OK in 10 milliseconds
I0117 22:17:28.678311    6004 azuredisk_v2.go:565]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="ba8ff869-96b4-11ed-a768-000d3a65271f" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=15260800
I0117 22:17:39.214355    6004 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0117 22:17:44.802415    6004 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 22:17:44.802415    6004 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="b75c8f3a-96b4-11ed-a768-000d3a65271f" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="reattach-disk-multiple-nodes" "latency"=21509225700
I0117 22:17:44.802415    6004 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="b75c8f3a-96b4-11ed-a768-000d3a65271f" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=21509225700
I0117 22:17:48.637217    6004 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 22:17:48.637217    6004 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 22:17:48.790870    6004 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 22:17:48.821757    6004 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync
... skipping 87 lines ...
I0117 22:19:26.294847    6004 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0117 22:19:26.294847    6004 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-705d56c4-c202-4c7d-8090-4ccae8e129ee\\globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-705d56c4-c202-4c7d-8090-4ccae8e129ee","csi.storage.k8s.io/pvc/name":"pvc-azuredisk","csi.storage.k8s.io/pvc/namespace":"default","requestedsizegib":"10","skuName":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1673987143207-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-ri100d/providers/Microsoft.Compute/disks/pvc-705d56c4-c202-4c7d-8090-4ccae8e129ee"}
I0117 22:19:26.294847    6004 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-705d56c4-c202-4c7d-8090-4ccae8e129ee-capz-ri10-ppsff-attachment)
I0117 22:19:28.664225    6004 azuredisk_v2.go:578]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-17T22:19:28Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="0216b24e-96b5-11ed-a768-000d3a65271f"
I0117 22:19:28.678342    6004 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-ri10-ppsff/status 200 OK in 14 milliseconds
I0117 22:19:28.678342    6004 azuredisk_v2.go:565]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="0216b24e-96b5-11ed-a768-000d3a65271f" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=14116800
I0117 22:19:29.161797    6004 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0117 22:19:29.163011    6004 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-ri10-ppsff" "disk.csi.azure.com/request-id"="00ad287f-96b5-11ed-a768-000d3a65271f" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-705d56c4-c202-4c7d-8090-4ccae8e129ee" "latency"=2866292800
I0117 22:19:29.163295    6004 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="00ad287f-96b5-11ed-a768-000d3a65271f" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=2868423900
I0117 22:19:33.072883    6004 nodeserver_v2.go:211] NodeStageVolume: formatting 10 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-705d56c4-c202-4c7d-8090-4ccae8e129ee\globalmount with mount options([])
I0117 22:19:48.677723    6004 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 22:19:48.678656    6004 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 22:19:48.817384    6004 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
... skipping 105 lines ...
I0117 20:25:50.866728    5972 round_trippers.go:553] GET https://10.96.0.1:443/apis/apiextensions.k8s.io/v1/customresourcedefinitions?limit=500&resourceVersion=0 200 OK in 37 milliseconds
I0117 20:25:50.903236    5972 round_trippers.go:553] GET https://10.96.0.1:443/apis/apiextensions.k8s.io/v1/customresourcedefinitions?allowWatchBookmarks=true&resourceVersion=2085&timeout=9m3s&timeoutSeconds=543&watch=true 200 OK in 1 milliseconds
I0117 20:25:50.935738    5972 shared_informer.go:303] caches populated
I0117 20:25:50.935738    5972 azuredisk_v2.go:225] driver userAgent: disk.csi.azure.com/latest-v2-9ef068a8cb36a997d4ea04b90c05c6f92a488a19 e2e-test
I0117 20:25:50.935738    5972 azure_disk_utils.go:564] reading cloud config from secret kube-system/azure-cloud-provider
I0117 20:25:50.941233    5972 round_trippers.go:553] GET https://10.96.0.1:443/api/v1/namespaces/kube-system/secrets/azure-cloud-provider 404 Not Found in 2 milliseconds
I0117 20:25:50.941749    5972 azure_disk_utils.go:571] InitializeCloudFromSecret: failed to get cloud config from secret kube-system/azure-cloud-provider: failed to get secret kube-system/azure-cloud-provider: secrets "azure-cloud-provider" not found
I0117 20:25:50.941749    5972 azure_disk_utils.go:576] could not read cloud config from secret kube-system/azure-cloud-provider
I0117 20:25:50.941749    5972 azure_disk_utils.go:586] use default AZURE_CREDENTIAL_FILE env var: C:\k\azure.json
I0117 20:25:50.942238    5972 azure_disk_utils.go:594] read cloud config from file: C:\k\azure.json successfully
I0117 20:25:50.943731    5972 azure_auth.go:253] Using AzurePublicCloud environment
I0117 20:25:50.944266    5972 azure_auth.go:138] azure: using client_id+client_secret to retrieve access token
I0117 20:25:50.947241    5972 azure_diskclient.go:68] Azure DisksClient using API version: 2022-03-02
... skipping 33 lines ...
I0117 20:25:51.378825    5972 utils.go:85] GRPC response: {"name":"disk.csi.azure.com","vendor_version":"latest-v2-9ef068a8cb36a997d4ea04b90c05c6f92a488a19"}
I0117 20:25:51.544330    5972 utils.go:78] GRPC call: /csi.v1.Identity/GetPluginInfo
I0117 20:25:51.544826    5972 utils.go:79] GRPC request: {}
I0117 20:25:51.544826    5972 utils.go:85] GRPC response: {"name":"disk.csi.azure.com","vendor_version":"latest-v2-9ef068a8cb36a997d4ea04b90c05c6f92a488a19"}
I0117 20:25:52.043404    5972 utils.go:78] GRPC call: /csi.v1.Node/NodeGetInfo
I0117 20:25:52.043404    5972 utils.go:79] GRPC request: {}
I0117 20:25:52.070889    5972 azure_instance_metadata.go:140] Warning: failed to get loadbalancer metadata: failure of getting loadbalancer metadata with response "404 Not Found"
I0117 20:25:52.070889    5972 nodeserver_v2.go:438] NodeGetInfo: nodeName(capz-ri10-zdjmx), VM Size(Standard_D4s_v3)
I0117 20:25:52.070889    5972 nodeserver.go:409] got a matching size in getMaxDataDiskCount, VM Size: STANDARD_D4S_V3, MaxDataDiskCount: 8
I0117 20:25:52.070889    5972 utils.go:85] GRPC response: {"accessible_topology":{"segments":{"topology.disk.csi.azure.com/zone":""}},"max_volumes_per_node":8,"node_id":"capz-ri10-zdjmx"}
I0117 20:26:20.861487    5972 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 20:26:20.861487    5972 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0117 20:26:20.861487    5972 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
... skipping 3152 lines ...
cloudprovider_azure_op_duration_seconds_bucket{request="azuredisk_csi_driver_controller_unpublish_volume",resource_group="capz-ri100d",source="disk.csi.azure.com",subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e",le="300"} 73
cloudprovider_azure_op_duration_seconds_bucket{request="azuredisk_csi_driver_controller_unpublish_volume",resource_group="capz-ri100d",source="disk.csi.azure.com",subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e",le="600"} 73
cloudprovider_azure_op_duration_seconds_bucket{request="azuredisk_csi_driver_controller_unpublish_volume",resource_group="capz-ri100d",source="disk.csi.azure.com",subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e",le="1200"} 73
cloudprovider_azure_op_duration_seconds_bucket{request="azuredisk_csi_driver_controller_unpublish_volume",resource_group="capz-ri100d",source="disk.csi.azure.com",subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e",le="+Inf"} 73
cloudprovider_azure_op_duration_seconds_sum{request="azuredisk_csi_driver_controller_unpublish_volume",resource_group="capz-ri100d",source="disk.csi.azure.com",subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e"} 718.2565781829998
cloudprovider_azure_op_duration_seconds_count{request="azuredisk_csi_driver_controller_unpublish_volume",resource_group="capz-ri100d",source="disk.csi.azure.com",subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e"} 73
# HELP cloudprovider_azure_op_failure_count [ALPHA] Number of failed Azure service operations
# TYPE cloudprovider_azure_op_failure_count counter
cloudprovider_azure_op_failure_count{request="azuredisk_csi_driver_controller_expand_volume",resource_group="capz-ri100d",source="disk.csi.azure.com",subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e"} 2
# HELP disabled_metric_total [ALPHA] The count of disabled metrics.
# TYPE disabled_metric_total counter
disabled_metric_total 0
# HELP go_cgo_go_to_c_calls_calls_total Count of calls made from Go to C by the current process.
... skipping 67 lines ...
# HELP go_gc_heap_objects_objects Number of objects, live or unswept, occupying heap memory.
# TYPE go_gc_heap_objects_objects gauge
go_gc_heap_objects_objects 135676
# HELP go_gc_heap_tiny_allocs_objects_total Count of small allocations that are packed together into blocks. These allocations are counted separately from other allocations because each individual allocation is not tracked by the runtime, only their block. Each block is already accounted for in allocs-by-size and frees-by-size.
# TYPE go_gc_heap_tiny_allocs_objects_total counter
go_gc_heap_tiny_allocs_objects_total 336785
# HELP go_gc_limiter_last_enabled_gc_cycle GC cycle the last time the GC CPU limiter was enabled. This metric is useful for diagnosing the root cause of an out-of-memory error, because the limiter trades memory for CPU time when the GC's CPU time gets too high. This is most likely to occur with use of SetMemoryLimit. The first GC cycle is cycle 1, so a value of 0 indicates that it was never enabled.
# TYPE go_gc_limiter_last_enabled_gc_cycle gauge
go_gc_limiter_last_enabled_gc_cycle 0
# HELP go_gc_pauses_seconds Distribution individual GC-related stop-the-world pause latencies.
# TYPE go_gc_pauses_seconds histogram
go_gc_pauses_seconds_bucket{le="9.999999999999999e-10"} 0
go_gc_pauses_seconds_bucket{le="9.999999999999999e-09"} 0
... skipping 211 lines ...
[ReportAfterSuite] Autogenerated ReportAfterSuite for --junit-report
[38;5;243mautogenerated by Ginkgo[0m
[38;5;243m------------------------------[0m


[38;5;9m[1mSummarizing 1 Failure:[0m
  [38;5;9m[FAIL][0m [0mDynamic Provisioning [38;5;243m[single-az] [0m[38;5;9m[1m[It] Should create replicas on node with matching pod affinity[0m[0m
  [38;5;243m/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/testsuites/dynamically_provisioned_pod_affinity.go:168[0m

[38;5;9m[1mRan 39 of 108 Specs in 7031.643 seconds[0m
[38;5;9m[1mFAIL![0m -- [38;5;10m[1m38 Passed[0m | [38;5;9m[1m1 Failed[0m | [38;5;11m[1m0 Pending[0m | [38;5;14m[1m69 Skipped[0m
--- FAIL: TestE2E (7031.67s)
FAIL
FAIL	sigs.k8s.io/azuredisk-csi-driver/test/e2e	7031.771s
FAIL
make: *** [Makefile:341: e2e-test] Error 1
NAME                              STATUS   ROLES                  AGE    VERSION                           INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                         KERNEL-VERSION     CONTAINER-RUNTIME
capz-ri10-8qjlr                   Ready    <none>                 120m   v1.23.16-rc.0.61+c8e964281208da   10.1.0.6      <none>        Windows Server 2019 Datacenter   10.0.17763.3770    containerd://1.6.2
capz-ri10-ppsff                   Ready    <none>                 120m   v1.23.16-rc.0.61+c8e964281208da   10.1.0.5      <none>        Windows Server 2019 Datacenter   10.0.17763.3770    containerd://1.6.2
capz-ri10-zdjmx                   Ready    <none>                 120m   v1.23.16-rc.0.61+c8e964281208da   10.1.0.4      <none>        Windows Server 2019 Datacenter   10.0.17763.3770    containerd://1.6.2
capz-ri100d-control-plane-8n9zj   Ready    control-plane,master   123m   v1.23.16-rc.0.61+c8e964281208da   10.0.0.4      <none>        Ubuntu 18.04.6 LTS               5.4.0-1098-azure   containerd://1.6.2
NAMESPACE          NAME                                                      READY   STATUS    RESTARTS       AGE    IP                NODE                              NOMINATED NODE   READINESS GATES
... skipping 36 lines ...

  Jan 17 22:24:20.709: INFO: Collecting logs for Windows node capz-ri10-zdjmx in cluster capz-ri100d in namespace default

  Jan 17 22:26:50.719: INFO: Attempting to copy file /c:/crashdumps.tar on node capz-ri10-zdjmx to /logs/artifacts/clusters/capz-ri100d/machines/capz-ri100d-md-win-86bd746bfb-98bf6/crashdumps.tar
  Jan 17 22:26:54.083: INFO: Collecting boot logs for AzureMachine capz-ri100d-md-win-zdjmx

Failed to get logs for Machine capz-ri100d-md-win-86bd746bfb-98bf6, Cluster default/capz-ri100d: [running command "Get-Content "C:\\cni.log"": Process exited with status 1, running command "$p = 'c:\localdumps' ; if (Test-Path $p) { tar.exe -cvzf c:\crashdumps.tar $p *>&1 | %{ Write-Output "$_"} } else { Write-Host "No crash dumps found at $p" }": Process exited with status 1]
  Jan 17 22:26:55.635: INFO: Collecting logs for Windows node capz-ri10-ppsff in cluster capz-ri100d in namespace default

  Jan 17 22:28:37.389: INFO: Attempting to copy file /c:/crashdumps.tar on node capz-ri10-ppsff to /logs/artifacts/clusters/capz-ri100d/machines/capz-ri100d-md-win-86bd746bfb-bfbkl/crashdumps.tar
  Jan 17 22:28:40.819: INFO: Collecting boot logs for AzureMachine capz-ri100d-md-win-ppsff

Failed to get logs for Machine capz-ri100d-md-win-86bd746bfb-bfbkl, Cluster default/capz-ri100d: running command "$p = 'c:\localdumps' ; if (Test-Path $p) { tar.exe -cvzf c:\crashdumps.tar $p *>&1 | %{ Write-Output "$_"} } else { Write-Host "No crash dumps found at $p" }": Process exited with status 1
  Jan 17 22:28:42.102: INFO: Collecting logs for Windows node capz-ri10-8qjlr in cluster capz-ri100d in namespace default

  Jan 17 22:30:17.030: INFO: Attempting to copy file /c:/crashdumps.tar on node capz-ri10-8qjlr to /logs/artifacts/clusters/capz-ri100d/machines/capz-ri100d-md-win-86bd746bfb-g7dzx/crashdumps.tar
  Jan 17 22:30:20.483: INFO: Collecting boot logs for AzureMachine capz-ri100d-md-win-8qjlr

Failed to get logs for Machine capz-ri100d-md-win-86bd746bfb-g7dzx, Cluster default/capz-ri100d: running command "$p = 'c:\localdumps' ; if (Test-Path $p) { tar.exe -cvzf c:\crashdumps.tar $p *>&1 | %{ Write-Output "$_"} } else { Write-Host "No crash dumps found at $p" }": Process exited with status 1
  Jan 17 22:30:21.822: INFO: Dumping workload cluster default/capz-ri100d kube-system pod logs
  Jan 17 22:30:22.802: INFO: Collecting events for Pod calico-apiserver/calico-apiserver-79dcb9855c-2zd6v
  Jan 17 22:30:22.802: INFO: Creating log watcher for controller calico-apiserver/calico-apiserver-79dcb9855c-2zd6v, container calico-apiserver
  Jan 17 22:30:22.803: INFO: Creating log watcher for controller calico-apiserver/calico-apiserver-79dcb9855c-krv4b, container calico-apiserver
  Jan 17 22:30:22.803: INFO: Collecting events for Pod calico-apiserver/calico-apiserver-79dcb9855c-krv4b
  Jan 17 22:30:22.909: INFO: Collecting events for Pod calico-system/calico-typha-5886d9f58d-kklxl
... skipping 49 lines ...
  Jan 17 22:30:23.181: INFO: Creating log watcher for controller kube-system/coredns-bd6b6df9f-xbhfb, container coredns
  Jan 17 22:30:23.182: INFO: Collecting events for Pod kube-system/metrics-server-7bdcf69694-45pdg
  Jan 17 22:30:23.530: INFO: Fetching kube-system pod logs took 1.707229554s
  Jan 17 22:30:23.530: INFO: Dumping workload cluster default/capz-ri100d Azure activity log
  Jan 17 22:30:23.530: INFO: Creating log watcher for controller tigera-operator/tigera-operator-6bbf97c9cf-bbp4n, container tigera-operator
  Jan 17 22:30:23.530: INFO: Collecting events for Pod tigera-operator/tigera-operator-6bbf97c9cf-bbp4n
  Jan 17 22:30:29.054: INFO: Got error while iterating over activity logs for resource group capz-ri100d: insights.ActivityLogsClient#listNextResults: Failure responding to next results request: StatusCode=404 -- Original Error: autorest/azure: error response cannot be parsed: {"<!DOCTYPE html PUBLIC \"-//W3C//DTD XHTML 1.0 Strict//EN\" \"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd\">\r\n<html xmlns=\"http://www.w3.org/1999/xhtml\">\r\n<head>\r\n<meta http-equiv=\"Content-Type\" content=\"text/html; charset=iso-8859-1\"/>\r\n<title>404 - File or directory not found.</title>\r\n<style type=\"text/css\">\r\n<!--\r\nbody{margin:0;font-size:.7em;font-family:Verdana, Arial, Helvetica, sans-serif;background:#EEEEEE;}\r\nfieldset{padding:0 15px 10px 15px;} \r\nh1{font-size:2.4em;margin:0;color:#FFF;}\r\nh2{font-si" '\x00' '\x00'} error: invalid character '<' looking for beginning of value
  Jan 17 22:30:29.054: INFO: Fetching activity logs took 5.523714031s
================ REDACTING LOGS ================
All sensitive variables are redacted
cluster.cluster.x-k8s.io "capz-ri100d" deleted
/home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure/hack/tools/bin/kind-v0.17.0 delete cluster --name=capz || true
Deleting cluster "capz" ...
... skipping 12 lines ...
PR	hccheng72: [V2] fix: only count replica attachments w/t deletion timestamp from cache client
Result	FAILURE
Tests	1 failed / 40 succeeded
Started	2023-01-17 20:04
Elapsed	2h34m
Revision	23b9a8b4a23ed46f3c0622bc10b1ef0956cdbc9e
Refs	1691
kubernetes-sigs/azuredisk-csi-driver PR #1691 pull-azuredisk-csi-driver-e2e-capz-windows-2019-mainv2 #1615439868979056640

Test Failures

AzureDisk CSI Driver End-to-End Tests [It] Dynamic Provisioning [single-az] Should create replicas on node with matching pod affinity 12m53s

Passed

Skipped

Error lines from build-log.txt
