pull-azuredisk-csi-driver-e2e-capz-windows-2019-mainv2 #1614049527289876480 Results

Error lines from build-log.txt

Raw build-log.txt
... skipping 788 lines ...
certificate.cert-manager.io "selfsigned-cert" deleted
# Create secret for AzureClusterIdentity
./hack/create-identity-secret.sh
make[2]: Entering directory '/home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure'
make[2]: Nothing to be done for 'kubectl'.
make[2]: Leaving directory '/home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure'
Error from server (NotFound): secrets "cluster-identity-secret" not found
secret/cluster-identity-secret created
secret/cluster-identity-secret labeled
# Create customized cloud provider configs
./hack/create-custom-cloud-provider-config.sh
make[2]: Entering directory '/home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure'
make[2]: Nothing to be done for 'kubectl'.
... skipping 226 lines ...

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 11345  100 11345    0     0   184k      0 --:--:-- --:--:-- --:--:--  184k
Downloading https://get.helm.sh/helm-v3.10.3-linux-amd64.tar.gz
Verifying checksum... Done.
Preparing to install helm into /usr/local/bin
helm installed into /usr/local/bin/helm
docker pull capzci.azurecr.io/azuredisk-csi:latest-v2-9ef068a8cb36a997d4ea04b90c05c6f92a488a19 || make container-all push-manifest
Error response from daemon: manifest for capzci.azurecr.io/azuredisk-csi:latest-v2-9ef068a8cb36a997d4ea04b90c05c6f92a488a19 not found: manifest unknown: manifest tagged by "latest-v2-9ef068a8cb36a997d4ea04b90c05c6f92a488a19" is not found
make[2]: Entering directory '/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver'
CGO_ENABLED=0 GOOS=windows go build -a -ldflags "-X sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.driverVersion=latest-v2-9ef068a8cb36a997d4ea04b90c05c6f92a488a19 -X sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.gitCommit=9ef068a8cb36a997d4ea04b90c05c6f92a488a19 -X sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.buildDate=2023-01-14T00:26:32Z -extldflags "-static"" -tags azurediskv2 -mod vendor -o _output/amd64/azurediskpluginv2.exe ./pkg/azurediskplugin
docker buildx rm container-builder || true
ERROR: no builder "container-builder" found
docker buildx create --use --name=container-builder
container-builder
# enable qemu for arm64 build
# https://github.com/docker/buildx/issues/464#issuecomment-741507760
docker run --privileged --rm tonistiigi/binfmt --uninstall qemu-aarch64
Unable to find image 'tonistiigi/binfmt:latest' locally
... skipping 643 lines ...
         }
      }
   ]
}
make[2]: Leaving directory '/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver'
docker pull capzci.azurecr.io/azdiskschedulerextender-csi:latest-v2-9ef068a8cb36a997d4ea04b90c05c6f92a488a19 || make azdiskschedulerextender-all push-manifest-azdiskschedulerextender
Error response from daemon: manifest for capzci.azurecr.io/azdiskschedulerextender-csi:latest-v2-9ef068a8cb36a997d4ea04b90c05c6f92a488a19 not found: manifest unknown: manifest tagged by "latest-v2-9ef068a8cb36a997d4ea04b90c05c6f92a488a19" is not found
make[2]: Entering directory '/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver'
docker buildx rm container-builder || true
container-builder removed
docker buildx create --use --name=container-builder
container-builder
# enable qemu for arm64 build
... skipping 1056 lines ...
                    type: string
                type: object
                oneOf:
                - required: ["persistentVolumeClaimName"]
                - required: ["volumeSnapshotContentName"]
              volumeSnapshotClassName:
                description: 'VolumeSnapshotClassName is the name of the VolumeSnapshotClass requested by the VolumeSnapshot. VolumeSnapshotClassName may be left nil to indicate that the default SnapshotClass should be used. A given cluster may have multiple default Volume SnapshotClasses: one default per CSI Driver. If a VolumeSnapshot does not specify a SnapshotClass, VolumeSnapshotSource will be checked to figure out what the associated CSI Driver is, and the default VolumeSnapshotClass associated with that CSI Driver will be used. If more than one VolumeSnapshotClass exist for a given CSI Driver and more than one have been marked as default, CreateSnapshot will fail and generate an event. Empty string is not allowed for this field.'
                type: string
            required:
            - source
            type: object
          status:
            description: status represents the current information of a snapshot. Consumers must verify binding between VolumeSnapshot and VolumeSnapshotContent objects is successful (by validating that both VolumeSnapshot and VolumeSnapshotContent point at each other) before using this object.
... skipping 2 lines ...
                description: 'boundVolumeSnapshotContentName is the name of the VolumeSnapshotContent object to which this VolumeSnapshot object intends to bind to. If not specified, it indicates that the VolumeSnapshot object has not been successfully bound to a VolumeSnapshotContent object yet. NOTE: To avoid possible security issues, consumers must verify binding between VolumeSnapshot and VolumeSnapshotContent objects is successful (by validating that both VolumeSnapshot and VolumeSnapshotContent point at each other) before using this object.'
                type: string
              creationTime:
                description: creationTime is the timestamp when the point-in-time snapshot is taken by the underlying storage system. In dynamic snapshot creation case, this field will be filled in by the snapshot controller with the "creation_time" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "creation_time" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. If not specified, it may indicate that the creation time of the snapshot is unknown.
                format: date-time
                type: string
              error:
                description: error is the last observed error during snapshot creation, if any. This field could be helpful to upper level controllers(i.e., application controller) to decide whether they should continue on waiting for the snapshot to be created based on the type of error reported. The snapshot controller will keep retrying when an error occurs during the snapshot creation. Upon success, this error field will be cleared.
                properties:
                  message:
                    description: 'message is a string detailing the encountered error during snapshot creation if specified. NOTE: message may be logged, and it should not contain sensitive information.'
                    type: string
                  time:
                    description: time is the timestamp when the error was encountered.
                    format: date-time
                    type: string
                type: object
              readyToUse:
                description: readyToUse indicates if the snapshot is ready to be used to restore a volume. In dynamic snapshot creation case, this field will be filled in by the snapshot controller with the "ready_to_use" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "ready_to_use" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it, otherwise, this field will be set to "True". If not specified, it means the readiness of a snapshot is unknown.
                type: boolean
              restoreSize:
                type: string
                description: restoreSize represents the minimum size of volume required to create a volume from this snapshot. In dynamic snapshot creation case, this field will be filled in by the snapshot controller with the "size_bytes" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "size_bytes" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. When restoring a volume from this snapshot, the size of the volume MUST NOT be smaller than the restoreSize if it is specified, otherwise the restoration will fail. If not specified, it indicates that the size is unknown.
                pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
                x-kubernetes-int-or-string: true
            type: object
        required:
        - spec
        type: object
... skipping 60 lines ...
                    type: string
                  volumeSnapshotContentName:
                    description: volumeSnapshotContentName specifies the name of a pre-existing VolumeSnapshotContent object representing an existing volume snapshot. This field should be set if the snapshot already exists and only needs a representation in Kubernetes. This field is immutable.
                    type: string
                type: object
              volumeSnapshotClassName:
                description: 'VolumeSnapshotClassName is the name of the VolumeSnapshotClass requested by the VolumeSnapshot. VolumeSnapshotClassName may be left nil to indicate that the default SnapshotClass should be used. A given cluster may have multiple default Volume SnapshotClasses: one default per CSI Driver. If a VolumeSnapshot does not specify a SnapshotClass, VolumeSnapshotSource will be checked to figure out what the associated CSI Driver is, and the default VolumeSnapshotClass associated with that CSI Driver will be used. If more than one VolumeSnapshotClass exist for a given CSI Driver and more than one have been marked as default, CreateSnapshot will fail and generate an event. Empty string is not allowed for this field.'
                type: string
            required:
            - source
            type: object
          status:
            description: status represents the current information of a snapshot. Consumers must verify binding between VolumeSnapshot and VolumeSnapshotContent objects is successful (by validating that both VolumeSnapshot and VolumeSnapshotContent point at each other) before using this object.
... skipping 2 lines ...
                description: 'boundVolumeSnapshotContentName is the name of the VolumeSnapshotContent object to which this VolumeSnapshot object intends to bind to. If not specified, it indicates that the VolumeSnapshot object has not been successfully bound to a VolumeSnapshotContent object yet. NOTE: To avoid possible security issues, consumers must verify binding between VolumeSnapshot and VolumeSnapshotContent objects is successful (by validating that both VolumeSnapshot and VolumeSnapshotContent point at each other) before using this object.'
                type: string
              creationTime:
                description: creationTime is the timestamp when the point-in-time snapshot is taken by the underlying storage system. In dynamic snapshot creation case, this field will be filled in by the snapshot controller with the "creation_time" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "creation_time" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. If not specified, it may indicate that the creation time of the snapshot is unknown.
                format: date-time
                type: string
              error:
                description: error is the last observed error during snapshot creation, if any. This field could be helpful to upper level controllers(i.e., application controller) to decide whether they should continue on waiting for the snapshot to be created based on the type of error reported. The snapshot controller will keep retrying when an error occurs during the snapshot creation. Upon success, this error field will be cleared.
                properties:
                  message:
                    description: 'message is a string detailing the encountered error during snapshot creation if specified. NOTE: message may be logged, and it should not contain sensitive information.'
                    type: string
                  time:
                    description: time is the timestamp when the error was encountered.
                    format: date-time
                    type: string
                type: object
              readyToUse:
                description: readyToUse indicates if the snapshot is ready to be used to restore a volume. In dynamic snapshot creation case, this field will be filled in by the snapshot controller with the "ready_to_use" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "ready_to_use" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it, otherwise, this field will be set to "True". If not specified, it means the readiness of a snapshot is unknown.
                type: boolean
              restoreSize:
                type: string
                description: restoreSize represents the minimum size of volume required to create a volume from this snapshot. In dynamic snapshot creation case, this field will be filled in by the snapshot controller with the "size_bytes" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "size_bytes" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. When restoring a volume from this snapshot, the size of the volume MUST NOT be smaller than the restoreSize if it is specified, otherwise the restoration will fail. If not specified, it indicates that the size is unknown.
                pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
                x-kubernetes-int-or-string: true
            type: object
        required:
        - spec
        type: object
... skipping 254 lines ...
            description: status represents the current information of a snapshot.
            properties:
              creationTime:
                description: creationTime is the timestamp when the point-in-time snapshot is taken by the underlying storage system. In dynamic snapshot creation case, this field will be filled in by the CSI snapshotter sidecar with the "creation_time" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "creation_time" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. If not specified, it indicates the creation time is unknown. The format of this field is a Unix nanoseconds time encoded as an int64. On Unix, the command `date +%s%N` returns the current time in nanoseconds since 1970-01-01 00:00:00 UTC.
                format: int64
                type: integer
              error:
                description: error is the last observed error during snapshot creation, if any. Upon success after retry, this error field will be cleared.
                properties:
                  message:
                    description: 'message is a string detailing the encountered error during snapshot creation if specified. NOTE: message may be logged, and it should not contain sensitive information.'
                    type: string
                  time:
                    description: time is the timestamp when the error was encountered.
                    format: date-time
                    type: string
                type: object
              readyToUse:
                description: readyToUse indicates if a snapshot is ready to be used to restore a volume. In dynamic snapshot creation case, this field will be filled in by the CSI snapshotter sidecar with the "ready_to_use" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "ready_to_use" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it, otherwise, this field will be set to "True". If not specified, it means the readiness of a snapshot is unknown.
                type: boolean
              restoreSize:
                description: restoreSize represents the complete size of the snapshot in bytes. In dynamic snapshot creation case, this field will be filled in by the CSI snapshotter sidecar with the "size_bytes" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "size_bytes" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. When restoring a volume from this snapshot, the size of the volume MUST NOT be smaller than the restoreSize if it is specified, otherwise the restoration will fail. If not specified, it indicates that the size is unknown.
                format: int64
                minimum: 0
                type: integer
              snapshotHandle:
                description: snapshotHandle is the CSI "snapshot_id" of a snapshot on the underlying storage system. If not specified, it indicates that dynamic snapshot creation has either failed or it is still in progress.
                type: string
            type: object
        required:
        - spec
        type: object
    served: true
... skipping 108 lines ...
            description: status represents the current information of a snapshot.
            properties:
              creationTime:
                description: creationTime is the timestamp when the point-in-time snapshot is taken by the underlying storage system. In dynamic snapshot creation case, this field will be filled in by the CSI snapshotter sidecar with the "creation_time" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "creation_time" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. If not specified, it indicates the creation time is unknown. The format of this field is a Unix nanoseconds time encoded as an int64. On Unix, the command `date +%s%N` returns the current time in nanoseconds since 1970-01-01 00:00:00 UTC.
                format: int64
                type: integer
              error:
                description: error is the last observed error during snapshot creation, if any. Upon success after retry, this error field will be cleared.
                properties:
                  message:
                    description: 'message is a string detailing the encountered error during snapshot creation if specified. NOTE: message may be logged, and it should not contain sensitive information.'
                    type: string
                  time:
                    description: time is the timestamp when the error was encountered.
                    format: date-time
                    type: string
                type: object
              readyToUse:
                description: readyToUse indicates if a snapshot is ready to be used to restore a volume. In dynamic snapshot creation case, this field will be filled in by the CSI snapshotter sidecar with the "ready_to_use" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "ready_to_use" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it, otherwise, this field will be set to "True". If not specified, it means the readiness of a snapshot is unknown.
                type: boolean
              restoreSize:
                description: restoreSize represents the complete size of the snapshot in bytes. In dynamic snapshot creation case, this field will be filled in by the CSI snapshotter sidecar with the "size_bytes" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "size_bytes" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. When restoring a volume from this snapshot, the size of the volume MUST NOT be smaller than the restoreSize if it is specified, otherwise the restoration will fail. If not specified, it indicates that the size is unknown.
                format: int64
                minimum: 0
                type: integer
              snapshotHandle:
                description: snapshotHandle is the CSI "snapshot_id" of a snapshot on the underlying storage system. If not specified, it indicates that dynamic snapshot creation has either failed or it is still in progress.
                type: string
            type: object
        required:
        - spec
        type: object
    served: true
... skipping 359 lines ...
            - volumeName
            - volume_context
            - volume_id
            type: object
          status:
            description: status represents the current state of AzVolumeAttachment.
              includes error, state, and attachment status Required
            properties:
              detail:
                description: Status summarizes the current attachment state of the
                  volume attachment Nil Status indicates that the volume has not yet
                  been attached to the node
                properties:
... skipping 7 lines ...
                  role:
                    description: The current attachment role.
                    type: string
                required:
                - role
                type: object
              error:
                description: Error occurred during attach/detach of volume
                properties:
                  code:
                    type: string
                  message:
                    type: string
                  parameters:
... skipping 90 lines ...
            - volumeName
            - volume_context
            - volume_id
            type: object
          status:
            description: status represents the current state of AzVolumeAttachment.
              includes error, state, and attachment status
            properties:
              annotation:
                additionalProperties:
                  type: string
                description: Annotations contains additional resource information
                  to guide driver actions
... skipping 13 lines ...
                  role:
                    description: The current attachment role.
                    type: string
                required:
                - role
                type: object
              error:
                description: Error occurred during attach/detach of volume
                properties:
                  code:
                    type: string
                  message:
                    type: string
                  parameters:
... skipping 169 lines ...
            - maxMountReplicaCount
            - volumeCapability
            - volumeName
            type: object
          status:
            description: status represents the current state of AzVolume. includes
              error, state, and volume status
            properties:
              detail:
                description: Current status detail of the AzVolume Nil detail indicates
                  that the volume has not been created
                properties:
                  accessible_topology:
... skipping 28 lines ...
                    type: string
                required:
                - capacity_bytes
                - node_expansion_required
                - volume_id
                type: object
              error:
                description: Error occurred during creation/deletion of volume
                properties:
                  code:
                    type: string
                  message:
                    type: string
                  parameters:
... skipping 154 lines ...
            - maxMountReplicaCount
            - volumeCapability
            - volumeName
            type: object
          status:
            description: status represents the current state of AzVolume. includes
              error, state, and volume status
            properties:
              annotation:
                additionalProperties:
                  type: string
                description: Annotations contains additional resource information
                  to guide driver actions
... skipping 34 lines ...
                    type: string
                required:
                - capacity_bytes
                - node_expansion_required
                - volume_id
                type: object
              error:
                description: Error occurred during creation/deletion of volume
                properties:
                  code:
                    type: string
                  message:
                    type: string
                  parameters:
... skipping 831 lines ...
          image: "mcr.microsoft.com/oss/kubernetes-csi/csi-resizer:v1.5.0"
          args:
            - "-csi-address=$(ADDRESS)"
            - "-v=2"
            - "-leader-election"
            - "--leader-election-namespace=kube-system"
            - '-handle-volume-inuse-error=false'
            - '-feature-gates=RecoverVolumeExpansionFailure=true'
            - "-timeout=240s"
          env:
            - name: ADDRESS
              value: /csi/csi.sock
          volumeMounts:
... skipping 296 lines ...
Jan 14 00:40:31.958: INFO: PersistentVolumeClaim pvc-5lsdn found but phase is Pending instead of Bound.
Jan 14 00:40:33.991: INFO: PersistentVolumeClaim pvc-5lsdn found and phase=Bound (2.064582665s)
[1mSTEP:[0m checking the PVC [38;5;243m01/14/23 00:40:33.991[0m
[1mSTEP:[0m validating provisioned PV [38;5;243m01/14/23 00:40:34.024[0m
[1mSTEP:[0m checking the PV [38;5;243m01/14/23 00:40:34.057[0m
[1mSTEP:[0m deploying the pod [38;5;243m01/14/23 00:40:34.057[0m
[1mSTEP:[0m checking that the pod's command exits with an error [38;5;243m01/14/23 00:40:34.097[0m
Jan 14 00:40:34.097: INFO: Waiting up to 10m0s for pod "azuredisk-volume-tester-rcl8n" in namespace "azuredisk-3481" to be "Error status code"
Jan 14 00:40:34.132: INFO: Pod "azuredisk-volume-tester-rcl8n": Phase="Pending", Reason="", readiness=false. Elapsed: 34.879762ms
Jan 14 00:40:36.166: INFO: Pod "azuredisk-volume-tester-rcl8n": Phase="Pending", Reason="", readiness=false. Elapsed: 2.069072542s
Jan 14 00:40:38.165: INFO: Pod "azuredisk-volume-tester-rcl8n": Phase="Pending", Reason="", readiness=false. Elapsed: 4.067664356s
Jan 14 00:40:40.165: INFO: Pod "azuredisk-volume-tester-rcl8n": Phase="Pending", Reason="", readiness=false. Elapsed: 6.068337946s
Jan 14 00:40:42.165: INFO: Pod "azuredisk-volume-tester-rcl8n": Phase="Pending", Reason="", readiness=false. Elapsed: 8.068069479s
Jan 14 00:40:44.166: INFO: Pod "azuredisk-volume-tester-rcl8n": Phase="Pending", Reason="", readiness=false. Elapsed: 10.069244741s
... skipping 120 lines ...
Jan 14 00:44:46.168: INFO: Pod "azuredisk-volume-tester-rcl8n": Phase="Running", Reason="", readiness=true. Elapsed: 4m12.070769052s
Jan 14 00:44:48.168: INFO: Pod "azuredisk-volume-tester-rcl8n": Phase="Running", Reason="", readiness=true. Elapsed: 4m14.070599882s
Jan 14 00:44:50.169: INFO: Pod "azuredisk-volume-tester-rcl8n": Phase="Running", Reason="", readiness=true. Elapsed: 4m16.071896898s
Jan 14 00:44:52.167: INFO: Pod "azuredisk-volume-tester-rcl8n": Phase="Running", Reason="", readiness=false. Elapsed: 4m18.070155609s
Jan 14 00:44:54.168: INFO: Pod "azuredisk-volume-tester-rcl8n": Phase="Running", Reason="", readiness=false. Elapsed: 4m20.071253321s
Jan 14 00:44:56.168: INFO: Pod "azuredisk-volume-tester-rcl8n": Phase="Running", Reason="", readiness=false. Elapsed: 4m22.071092977s
Jan 14 00:44:58.167: INFO: Pod "azuredisk-volume-tester-rcl8n": Phase="Failed", Reason="", readiness=false. Elapsed: 4m24.070219085s
[1mSTEP:[0m Saw pod failure [38;5;243m01/14/23 00:44:58.167[0m
Jan 14 00:44:58.167: INFO: Pod "azuredisk-volume-tester-rcl8n" satisfied condition "Error status code"
[1mSTEP:[0m checking that pod logs contain expected message [38;5;243m01/14/23 00:44:58.167[0m
Jan 14 00:44:58.235: INFO: deleting Pod "azuredisk-3481"/"azuredisk-volume-tester-rcl8n"
Jan 14 00:44:58.273: INFO: Pod azuredisk-volume-tester-rcl8n has the following logs: Out-File : Access to the path 'C:\mnt\test-1\data.txt' is denied.
At line:1 char:22
+ ... cho 'hello world' | Out-File -FilePath C:\mnt\test-1\data.txt; Get-Co ...
+                         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
... skipping 57 lines ...
    Jan 14 00:40:31.958: INFO: PersistentVolumeClaim pvc-5lsdn found but phase is Pending instead of Bound.
    Jan 14 00:40:33.991: INFO: PersistentVolumeClaim pvc-5lsdn found and phase=Bound (2.064582665s)
    [1mSTEP:[0m checking the PVC [38;5;243m01/14/23 00:40:33.991[0m
    [1mSTEP:[0m validating provisioned PV [38;5;243m01/14/23 00:40:34.024[0m
    [1mSTEP:[0m checking the PV [38;5;243m01/14/23 00:40:34.057[0m
    [1mSTEP:[0m deploying the pod [38;5;243m01/14/23 00:40:34.057[0m
    [1mSTEP:[0m checking that the pod's command exits with an error [38;5;243m01/14/23 00:40:34.097[0m
    Jan 14 00:40:34.097: INFO: Waiting up to 10m0s for pod "azuredisk-volume-tester-rcl8n" in namespace "azuredisk-3481" to be "Error status code"
    Jan 14 00:40:34.132: INFO: Pod "azuredisk-volume-tester-rcl8n": Phase="Pending", Reason="", readiness=false. Elapsed: 34.879762ms
    Jan 14 00:40:36.166: INFO: Pod "azuredisk-volume-tester-rcl8n": Phase="Pending", Reason="", readiness=false. Elapsed: 2.069072542s
    Jan 14 00:40:38.165: INFO: Pod "azuredisk-volume-tester-rcl8n": Phase="Pending", Reason="", readiness=false. Elapsed: 4.067664356s
    Jan 14 00:40:40.165: INFO: Pod "azuredisk-volume-tester-rcl8n": Phase="Pending", Reason="", readiness=false. Elapsed: 6.068337946s
    Jan 14 00:40:42.165: INFO: Pod "azuredisk-volume-tester-rcl8n": Phase="Pending", Reason="", readiness=false. Elapsed: 8.068069479s
    Jan 14 00:40:44.166: INFO: Pod "azuredisk-volume-tester-rcl8n": Phase="Pending", Reason="", readiness=false. Elapsed: 10.069244741s
... skipping 120 lines ...
    Jan 14 00:44:46.168: INFO: Pod "azuredisk-volume-tester-rcl8n": Phase="Running", Reason="", readiness=true. Elapsed: 4m12.070769052s
    Jan 14 00:44:48.168: INFO: Pod "azuredisk-volume-tester-rcl8n": Phase="Running", Reason="", readiness=true. Elapsed: 4m14.070599882s
    Jan 14 00:44:50.169: INFO: Pod "azuredisk-volume-tester-rcl8n": Phase="Running", Reason="", readiness=true. Elapsed: 4m16.071896898s
    Jan 14 00:44:52.167: INFO: Pod "azuredisk-volume-tester-rcl8n": Phase="Running", Reason="", readiness=false. Elapsed: 4m18.070155609s
    Jan 14 00:44:54.168: INFO: Pod "azuredisk-volume-tester-rcl8n": Phase="Running", Reason="", readiness=false. Elapsed: 4m20.071253321s
    Jan 14 00:44:56.168: INFO: Pod "azuredisk-volume-tester-rcl8n": Phase="Running", Reason="", readiness=false. Elapsed: 4m22.071092977s
    Jan 14 00:44:58.167: INFO: Pod "azuredisk-volume-tester-rcl8n": Phase="Failed", Reason="", readiness=false. Elapsed: 4m24.070219085s
    [1mSTEP:[0m Saw pod failure [38;5;243m01/14/23 00:44:58.167[0m
    Jan 14 00:44:58.167: INFO: Pod "azuredisk-volume-tester-rcl8n" satisfied condition "Error status code"
    [1mSTEP:[0m checking that pod logs contain expected message [38;5;243m01/14/23 00:44:58.167[0m
    Jan 14 00:44:58.235: INFO: deleting Pod "azuredisk-3481"/"azuredisk-volume-tester-rcl8n"
    Jan 14 00:44:58.273: INFO: Pod azuredisk-volume-tester-rcl8n has the following logs: Out-File : Access to the path 'C:\mnt\test-1\data.txt' is denied.
    At line:1 char:22
    + ... cho 'hello world' | Out-File -FilePath C:\mnt\test-1\data.txt; Get-Co ...
    +                         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
... skipping 85 lines ...
Jan 14 00:45:46.368: INFO: PersistentVolumeClaim pvc-7qphz found and phase=Bound (31.496615ms)
[1mSTEP:[0m checking the PVC [38;5;243m01/14/23 00:45:46.368[0m
[1mSTEP:[0m validating provisioned PV [38;5;243m01/14/23 00:45:46.4[0m
[1mSTEP:[0m checking the PV [38;5;243m01/14/23 00:45:46.433[0m
[1mSTEP:[0m attaching disk to node#0 [38;5;243m01/14/23 00:45:46.479[0m
[1mSTEP:[0m deploying the pod [38;5;243m01/14/23 00:45:58.72[0m
[1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/14/23 00:45:58.757[0m
Jan 14 00:45:58.757: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-lf5ll" in namespace "azuredisk-4454" to be "Succeeded or Failed"
Jan 14 00:45:58.790: INFO: Pod "azuredisk-volume-tester-lf5ll": Phase="Pending", Reason="", readiness=false. Elapsed: 33.480858ms
Jan 14 00:46:00.825: INFO: Pod "azuredisk-volume-tester-lf5ll": Phase="Pending", Reason="", readiness=false. Elapsed: 2.067925466s
Jan 14 00:46:02.829: INFO: Pod "azuredisk-volume-tester-lf5ll": Phase="Pending", Reason="", readiness=false. Elapsed: 4.071758214s
Jan 14 00:46:04.824: INFO: Pod "azuredisk-volume-tester-lf5ll": Phase="Pending", Reason="", readiness=false. Elapsed: 6.066976533s
Jan 14 00:46:06.823: INFO: Pod "azuredisk-volume-tester-lf5ll": Phase="Pending", Reason="", readiness=false. Elapsed: 8.066413562s
Jan 14 00:46:08.826: INFO: Pod "azuredisk-volume-tester-lf5ll": Phase="Pending", Reason="", readiness=false. Elapsed: 10.068894062s
... skipping 25 lines ...
Jan 14 00:47:00.827: INFO: Pod "azuredisk-volume-tester-lf5ll": Phase="Running", Reason="", readiness=true. Elapsed: 1m2.070099905s
Jan 14 00:47:02.827: INFO: Pod "azuredisk-volume-tester-lf5ll": Phase="Running", Reason="", readiness=false. Elapsed: 1m4.069892382s
Jan 14 00:47:04.825: INFO: Pod "azuredisk-volume-tester-lf5ll": Phase="Running", Reason="", readiness=false. Elapsed: 1m6.068552682s
Jan 14 00:47:06.826: INFO: Pod "azuredisk-volume-tester-lf5ll": Phase="Running", Reason="", readiness=false. Elapsed: 1m8.069522994s
Jan 14 00:47:08.827: INFO: Pod "azuredisk-volume-tester-lf5ll": Phase="Succeeded", Reason="", readiness=false. Elapsed: 1m10.069702387s
[1mSTEP:[0m Saw pod success [38;5;243m01/14/23 00:47:08.827[0m
Jan 14 00:47:08.827: INFO: Pod "azuredisk-volume-tester-lf5ll" satisfied condition "Succeeded or Failed"
Jan 14 00:47:08.827: INFO: deleting Pod "azuredisk-4454"/"azuredisk-volume-tester-lf5ll"
Jan 14 00:47:08.885: INFO: Pod azuredisk-volume-tester-lf5ll has the following logs: hello world

[1mSTEP:[0m Deleting pod azuredisk-volume-tester-lf5ll in namespace azuredisk-4454 [38;5;243m01/14/23 00:47:08.885[0m
Jan 14 00:47:09.009: INFO: deleting PVC "azuredisk-4454"/"pvc-7qphz"
Jan 14 00:47:09.009: INFO: Deleting PersistentVolumeClaim "pvc-7qphz"
... skipping 36 lines ...
    Jan 14 00:45:46.368: INFO: PersistentVolumeClaim pvc-7qphz found and phase=Bound (31.496615ms)
    [1mSTEP:[0m checking the PVC [38;5;243m01/14/23 00:45:46.368[0m
    [1mSTEP:[0m validating provisioned PV [38;5;243m01/14/23 00:45:46.4[0m
    [1mSTEP:[0m checking the PV [38;5;243m01/14/23 00:45:46.433[0m
    [1mSTEP:[0m attaching disk to node#0 [38;5;243m01/14/23 00:45:46.479[0m
    [1mSTEP:[0m deploying the pod [38;5;243m01/14/23 00:45:58.72[0m
    [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/14/23 00:45:58.757[0m
    Jan 14 00:45:58.757: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-lf5ll" in namespace "azuredisk-4454" to be "Succeeded or Failed"
    Jan 14 00:45:58.790: INFO: Pod "azuredisk-volume-tester-lf5ll": Phase="Pending", Reason="", readiness=false. Elapsed: 33.480858ms
    Jan 14 00:46:00.825: INFO: Pod "azuredisk-volume-tester-lf5ll": Phase="Pending", Reason="", readiness=false. Elapsed: 2.067925466s
    Jan 14 00:46:02.829: INFO: Pod "azuredisk-volume-tester-lf5ll": Phase="Pending", Reason="", readiness=false. Elapsed: 4.071758214s
    Jan 14 00:46:04.824: INFO: Pod "azuredisk-volume-tester-lf5ll": Phase="Pending", Reason="", readiness=false. Elapsed: 6.066976533s
    Jan 14 00:46:06.823: INFO: Pod "azuredisk-volume-tester-lf5ll": Phase="Pending", Reason="", readiness=false. Elapsed: 8.066413562s
    Jan 14 00:46:08.826: INFO: Pod "azuredisk-volume-tester-lf5ll": Phase="Pending", Reason="", readiness=false. Elapsed: 10.068894062s
... skipping 25 lines ...
    Jan 14 00:47:00.827: INFO: Pod "azuredisk-volume-tester-lf5ll": Phase="Running", Reason="", readiness=true. Elapsed: 1m2.070099905s
    Jan 14 00:47:02.827: INFO: Pod "azuredisk-volume-tester-lf5ll": Phase="Running", Reason="", readiness=false. Elapsed: 1m4.069892382s
    Jan 14 00:47:04.825: INFO: Pod "azuredisk-volume-tester-lf5ll": Phase="Running", Reason="", readiness=false. Elapsed: 1m6.068552682s
    Jan 14 00:47:06.826: INFO: Pod "azuredisk-volume-tester-lf5ll": Phase="Running", Reason="", readiness=false. Elapsed: 1m8.069522994s
    Jan 14 00:47:08.827: INFO: Pod "azuredisk-volume-tester-lf5ll": Phase="Succeeded", Reason="", readiness=false. Elapsed: 1m10.069702387s
    [1mSTEP:[0m Saw pod success [38;5;243m01/14/23 00:47:08.827[0m
    Jan 14 00:47:08.827: INFO: Pod "azuredisk-volume-tester-lf5ll" satisfied condition "Succeeded or Failed"
    Jan 14 00:47:08.827: INFO: deleting Pod "azuredisk-4454"/"azuredisk-volume-tester-lf5ll"
    Jan 14 00:47:08.885: INFO: Pod azuredisk-volume-tester-lf5ll has the following logs: hello world

    [1mSTEP:[0m Deleting pod azuredisk-volume-tester-lf5ll in namespace azuredisk-4454 [38;5;243m01/14/23 00:47:08.885[0m
    Jan 14 00:47:09.009: INFO: deleting PVC "azuredisk-4454"/"pvc-7qphz"
    Jan 14 00:47:09.009: INFO: Deleting PersistentVolumeClaim "pvc-7qphz"
... skipping 149 lines ...
  [38;5;243m<< End Captured GinkgoWriter Output[0m

  [38;5;14mtest case not supported by Windows clusters[0m
  [38;5;14mIn [1m[It][0m[38;5;14m at: [1m/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/utils/testutil/env_utils.go:26[0m

  [1mThere were additional failures detected after the initial failure:[0m
    [38;5;9m[FAILED][0m
    [38;5;9mpre-provisioning disk got deleted with the deletion of pv: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {
      "error": {

        "code": "NotFound",
        "message": "Disk single-shared-disk is not found."
      }
    }[0m
    [38;5;9mIn [1m[AfterEach][0m[38;5;9m at: [1m/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/pre_provisioning_test.go:490[0m
[38;5;243m------------------------------[0m
... skipping 27 lines ...
  [38;5;243m<< End Captured GinkgoWriter Output[0m

  [38;5;14mtest case not supported by Windows clusters[0m
  [38;5;14mIn [1m[It][0m[38;5;14m at: [1m/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/utils/testutil/env_utils.go:26[0m

  [1mThere were additional failures detected after the initial failure:[0m
    [38;5;9m[FAILED][0m
    [38;5;9mpre-provisioning disk got deleted with the deletion of pv: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {
      "error": {

        "code": "NotFound",
        "message": "Disk single-shared-disk is not found."
      }
    }[0m
    [38;5;9mIn [1m[AfterEach][0m[38;5;9m at: [1m/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/pre_provisioning_test.go:490[0m
[38;5;243m------------------------------[0m
... skipping 27 lines ...
  [38;5;243m<< End Captured GinkgoWriter Output[0m

  [38;5;14mtest case is only available for csi driver[0m
  [38;5;14mIn [1m[It][0m[38;5;14m at: [1m/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/pre_provisioning_test.go:395[0m

  [1mThere were additional failures detected after the initial failure:[0m
    [38;5;9m[FAILED][0m
    [38;5;9mpre-provisioning disk got deleted with the deletion of pv: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {
      "error": {

        "code": "NotFound",
        "message": "Disk single-shared-disk is not found."
      }
    }[0m
    [38;5;9mIn [1m[AfterEach][0m[38;5;9m at: [1m/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/pre_provisioning_test.go:490[0m
[38;5;243m------------------------------[0m
... skipping 51 lines ...
[1mSTEP:[0m checking the PVC [38;5;243m01/14/23 00:47:59.324[0m
[1mSTEP:[0m validating provisioned PV [38;5;243m01/14/23 00:47:59.356[0m
[1mSTEP:[0m checking the PV [38;5;243m01/14/23 00:47:59.388[0m
[1mSTEP:[0m setting up the pod [38;5;243m01/14/23 00:47:59.388[0m
I0114 00:47:59.389059   42005 resource_setup.go:62] adding PV (pvc-0f97f108-fafd-4633-a41a-df2b21087ede) to pod ()
[1mSTEP:[0m deploying the pod [38;5;243m01/14/23 00:47:59.389[0m
[1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/14/23 00:47:59.424[0m
Jan 14 00:47:59.424: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-jsls5" in namespace "azuredisk-8406" to be "Succeeded or Failed"
Jan 14 00:47:59.459: INFO: Pod "azuredisk-volume-tester-jsls5": Phase="Pending", Reason="", readiness=false. Elapsed: 34.231836ms
Jan 14 00:48:01.493: INFO: Pod "azuredisk-volume-tester-jsls5": Phase="Pending", Reason="", readiness=false. Elapsed: 2.068805013s
Jan 14 00:48:03.494: INFO: Pod "azuredisk-volume-tester-jsls5": Phase="Pending", Reason="", readiness=false. Elapsed: 4.069936662s
Jan 14 00:48:05.493: INFO: Pod "azuredisk-volume-tester-jsls5": Phase="Pending", Reason="", readiness=false. Elapsed: 6.069041645s
Jan 14 00:48:07.492: INFO: Pod "azuredisk-volume-tester-jsls5": Phase="Pending", Reason="", readiness=false. Elapsed: 8.068057103s
Jan 14 00:48:09.493: INFO: Pod "azuredisk-volume-tester-jsls5": Phase="Pending", Reason="", readiness=false. Elapsed: 10.068956318s
... skipping 19 lines ...
Jan 14 00:48:49.496: INFO: Pod "azuredisk-volume-tester-jsls5": Phase="Running", Reason="", readiness=true. Elapsed: 50.071898148s
Jan 14 00:48:51.495: INFO: Pod "azuredisk-volume-tester-jsls5": Phase="Running", Reason="", readiness=false. Elapsed: 52.070469965s
Jan 14 00:48:53.495: INFO: Pod "azuredisk-volume-tester-jsls5": Phase="Running", Reason="", readiness=false. Elapsed: 54.070712598s
Jan 14 00:48:55.494: INFO: Pod "azuredisk-volume-tester-jsls5": Phase="Running", Reason="", readiness=false. Elapsed: 56.06947537s
Jan 14 00:48:57.494: INFO: Pod "azuredisk-volume-tester-jsls5": Phase="Succeeded", Reason="", readiness=false. Elapsed: 58.069954247s
[1mSTEP:[0m Saw pod success [38;5;243m01/14/23 00:48:57.494[0m
Jan 14 00:48:57.494: INFO: Pod "azuredisk-volume-tester-jsls5" satisfied condition "Succeeded or Failed"
Jan 14 00:48:57.495: INFO: deleting Pod "azuredisk-8406"/"azuredisk-volume-tester-jsls5"
Jan 14 00:48:57.549: INFO: Pod azuredisk-volume-tester-jsls5 has the following logs: hello world

[1mSTEP:[0m Deleting pod azuredisk-volume-tester-jsls5 in namespace azuredisk-8406 [38;5;243m01/14/23 00:48:57.549[0m
Jan 14 00:48:57.593: INFO: deleting PVC "azuredisk-8406"/"pvc-tb5dz"
Jan 14 00:48:57.593: INFO: Deleting PersistentVolumeClaim "pvc-tb5dz"
... skipping 40 lines ...
    [1mSTEP:[0m checking the PVC [38;5;243m01/14/23 00:47:59.324[0m
    [1mSTEP:[0m validating provisioned PV [38;5;243m01/14/23 00:47:59.356[0m
    [1mSTEP:[0m checking the PV [38;5;243m01/14/23 00:47:59.388[0m
    [1mSTEP:[0m setting up the pod [38;5;243m01/14/23 00:47:59.388[0m
    I0114 00:47:59.389059   42005 resource_setup.go:62] adding PV (pvc-0f97f108-fafd-4633-a41a-df2b21087ede) to pod ()
    [1mSTEP:[0m deploying the pod [38;5;243m01/14/23 00:47:59.389[0m
    [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/14/23 00:47:59.424[0m
    Jan 14 00:47:59.424: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-jsls5" in namespace "azuredisk-8406" to be "Succeeded or Failed"
    Jan 14 00:47:59.459: INFO: Pod "azuredisk-volume-tester-jsls5": Phase="Pending", Reason="", readiness=false. Elapsed: 34.231836ms
    Jan 14 00:48:01.493: INFO: Pod "azuredisk-volume-tester-jsls5": Phase="Pending", Reason="", readiness=false. Elapsed: 2.068805013s
    Jan 14 00:48:03.494: INFO: Pod "azuredisk-volume-tester-jsls5": Phase="Pending", Reason="", readiness=false. Elapsed: 4.069936662s
    Jan 14 00:48:05.493: INFO: Pod "azuredisk-volume-tester-jsls5": Phase="Pending", Reason="", readiness=false. Elapsed: 6.069041645s
    Jan 14 00:48:07.492: INFO: Pod "azuredisk-volume-tester-jsls5": Phase="Pending", Reason="", readiness=false. Elapsed: 8.068057103s
    Jan 14 00:48:09.493: INFO: Pod "azuredisk-volume-tester-jsls5": Phase="Pending", Reason="", readiness=false. Elapsed: 10.068956318s
... skipping 19 lines ...
    Jan 14 00:48:49.496: INFO: Pod "azuredisk-volume-tester-jsls5": Phase="Running", Reason="", readiness=true. Elapsed: 50.071898148s
    Jan 14 00:48:51.495: INFO: Pod "azuredisk-volume-tester-jsls5": Phase="Running", Reason="", readiness=false. Elapsed: 52.070469965s
    Jan 14 00:48:53.495: INFO: Pod "azuredisk-volume-tester-jsls5": Phase="Running", Reason="", readiness=false. Elapsed: 54.070712598s
    Jan 14 00:48:55.494: INFO: Pod "azuredisk-volume-tester-jsls5": Phase="Running", Reason="", readiness=false. Elapsed: 56.06947537s
    Jan 14 00:48:57.494: INFO: Pod "azuredisk-volume-tester-jsls5": Phase="Succeeded", Reason="", readiness=false. Elapsed: 58.069954247s
    [1mSTEP:[0m Saw pod success [38;5;243m01/14/23 00:48:57.494[0m
    Jan 14 00:48:57.494: INFO: Pod "azuredisk-volume-tester-jsls5" satisfied condition "Succeeded or Failed"
    Jan 14 00:48:57.495: INFO: deleting Pod "azuredisk-8406"/"azuredisk-volume-tester-jsls5"
    Jan 14 00:48:57.549: INFO: Pod azuredisk-volume-tester-jsls5 has the following logs: hello world

    [1mSTEP:[0m Deleting pod azuredisk-volume-tester-jsls5 in namespace azuredisk-8406 [38;5;243m01/14/23 00:48:57.549[0m
    Jan 14 00:48:57.593: INFO: deleting PVC "azuredisk-8406"/"pvc-tb5dz"
    Jan 14 00:48:57.593: INFO: Deleting PersistentVolumeClaim "pvc-tb5dz"
... skipping 37 lines ...
Jan 14 00:49:40.912: INFO: PersistentVolumeClaim pvc-gvflv found but phase is Pending instead of Bound.
Jan 14 00:49:42.945: INFO: PersistentVolumeClaim pvc-gvflv found and phase=Bound (4.10096476s)
[1mSTEP:[0m checking the PVC [38;5;243m01/14/23 00:49:42.945[0m
[1mSTEP:[0m validating provisioned PV [38;5;243m01/14/23 00:49:42.98[0m
[1mSTEP:[0m checking the PV [38;5;243m01/14/23 00:49:43.014[0m
[1mSTEP:[0m deploying the pod [38;5;243m01/14/23 00:49:43.014[0m
[1mSTEP:[0m checking that the pods command exits with no error [38;5;243m01/14/23 00:49:43.049[0m
Jan 14 00:49:43.050: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-vqdxt" in namespace "azuredisk-7655" to be "Succeeded or Failed"
Jan 14 00:49:43.085: INFO: Pod "azuredisk-volume-tester-vqdxt": Phase="Pending", Reason="", readiness=false. Elapsed: 35.214203ms
Jan 14 00:49:45.119: INFO: Pod "azuredisk-volume-tester-vqdxt": Phase="Pending", Reason="", readiness=false. Elapsed: 2.06950333s
Jan 14 00:49:47.119: INFO: Pod "azuredisk-volume-tester-vqdxt": Phase="Pending", Reason="", readiness=false. Elapsed: 4.06973583s
Jan 14 00:49:49.120: INFO: Pod "azuredisk-volume-tester-vqdxt": Phase="Pending", Reason="", readiness=false. Elapsed: 6.070064239s
Jan 14 00:49:51.121: INFO: Pod "azuredisk-volume-tester-vqdxt": Phase="Pending", Reason="", readiness=false. Elapsed: 8.070882039s
Jan 14 00:49:53.120: INFO: Pod "azuredisk-volume-tester-vqdxt": Phase="Pending", Reason="", readiness=false. Elapsed: 10.0702679s
... skipping 20 lines ...
Jan 14 00:50:35.120: INFO: Pod "azuredisk-volume-tester-vqdxt": Phase="Running", Reason="", readiness=true. Elapsed: 52.070711123s
Jan 14 00:50:37.121: INFO: Pod "azuredisk-volume-tester-vqdxt": Phase="Running", Reason="", readiness=true. Elapsed: 54.070948226s
Jan 14 00:50:39.121: INFO: Pod "azuredisk-volume-tester-vqdxt": Phase="Running", Reason="", readiness=false. Elapsed: 56.071241518s
Jan 14 00:50:41.120: INFO: Pod "azuredisk-volume-tester-vqdxt": Phase="Running", Reason="", readiness=false. Elapsed: 58.070502087s
Jan 14 00:50:43.122: INFO: Pod "azuredisk-volume-tester-vqdxt": Phase="Succeeded", Reason="", readiness=false. Elapsed: 1m0.072711923s
[1mSTEP:[0m Saw pod success [38;5;243m01/14/23 00:50:43.122[0m
Jan 14 00:50:43.123: INFO: Pod "azuredisk-volume-tester-vqdxt" satisfied condition "Succeeded or Failed"
Jan 14 00:50:43.123: INFO: deleting Pod "azuredisk-7655"/"azuredisk-volume-tester-vqdxt"
Jan 14 00:50:43.186: INFO: Pod azuredisk-volume-tester-vqdxt has the following logs: hello world

[1mSTEP:[0m Deleting pod azuredisk-volume-tester-vqdxt in namespace azuredisk-7655 [38;5;243m01/14/23 00:50:43.186[0m
Jan 14 00:50:43.228: INFO: deleting PVC "azuredisk-7655"/"pvc-gvflv"
Jan 14 00:50:43.229: INFO: Deleting PersistentVolumeClaim "pvc-gvflv"
... skipping 38 lines ...
    Jan 14 00:49:40.912: INFO: PersistentVolumeClaim pvc-gvflv found but phase is Pending instead of Bound.
    Jan 14 00:49:42.945: INFO: PersistentVolumeClaim pvc-gvflv found and phase=Bound (4.10096476s)
    [1mSTEP:[0m checking the PVC [38;5;243m01/14/23 00:49:42.945[0m
    [1mSTEP:[0m validating provisioned PV [38;5;243m01/14/23 00:49:42.98[0m
    [1mSTEP:[0m checking the PV [38;5;243m01/14/23 00:49:43.014[0m
    [1mSTEP:[0m deploying the pod [38;5;243m01/14/23 00:49:43.014[0m
    [1mSTEP:[0m checking that the pods command exits with no error [38;5;243m01/14/23 00:49:43.049[0m
    Jan 14 00:49:43.050: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-vqdxt" in namespace "azuredisk-7655" to be "Succeeded or Failed"
    Jan 14 00:49:43.085: INFO: Pod "azuredisk-volume-tester-vqdxt": Phase="Pending", Reason="", readiness=false. Elapsed: 35.214203ms
    Jan 14 00:49:45.119: INFO: Pod "azuredisk-volume-tester-vqdxt": Phase="Pending", Reason="", readiness=false. Elapsed: 2.06950333s
    Jan 14 00:49:47.119: INFO: Pod "azuredisk-volume-tester-vqdxt": Phase="Pending", Reason="", readiness=false. Elapsed: 4.06973583s
    Jan 14 00:49:49.120: INFO: Pod "azuredisk-volume-tester-vqdxt": Phase="Pending", Reason="", readiness=false. Elapsed: 6.070064239s
    Jan 14 00:49:51.121: INFO: Pod "azuredisk-volume-tester-vqdxt": Phase="Pending", Reason="", readiness=false. Elapsed: 8.070882039s
    Jan 14 00:49:53.120: INFO: Pod "azuredisk-volume-tester-vqdxt": Phase="Pending", Reason="", readiness=false. Elapsed: 10.0702679s
... skipping 20 lines ...
    Jan 14 00:50:35.120: INFO: Pod "azuredisk-volume-tester-vqdxt": Phase="Running", Reason="", readiness=true. Elapsed: 52.070711123s
    Jan 14 00:50:37.121: INFO: Pod "azuredisk-volume-tester-vqdxt": Phase="Running", Reason="", readiness=true. Elapsed: 54.070948226s
    Jan 14 00:50:39.121: INFO: Pod "azuredisk-volume-tester-vqdxt": Phase="Running", Reason="", readiness=false. Elapsed: 56.071241518s
    Jan 14 00:50:41.120: INFO: Pod "azuredisk-volume-tester-vqdxt": Phase="Running", Reason="", readiness=false. Elapsed: 58.070502087s
    Jan 14 00:50:43.122: INFO: Pod "azuredisk-volume-tester-vqdxt": Phase="Succeeded", Reason="", readiness=false. Elapsed: 1m0.072711923s
    [1mSTEP:[0m Saw pod success [38;5;243m01/14/23 00:50:43.122[0m
    Jan 14 00:50:43.123: INFO: Pod "azuredisk-volume-tester-vqdxt" satisfied condition "Succeeded or Failed"
    Jan 14 00:50:43.123: INFO: deleting Pod "azuredisk-7655"/"azuredisk-volume-tester-vqdxt"
    Jan 14 00:50:43.186: INFO: Pod azuredisk-volume-tester-vqdxt has the following logs: hello world

    [1mSTEP:[0m Deleting pod azuredisk-volume-tester-vqdxt in namespace azuredisk-7655 [38;5;243m01/14/23 00:50:43.186[0m
    Jan 14 00:50:43.228: INFO: deleting PVC "azuredisk-7655"/"pvc-gvflv"
    Jan 14 00:50:43.229: INFO: Deleting PersistentVolumeClaim "pvc-gvflv"
... skipping 39 lines ...
[1mSTEP:[0m checking the PVC [38;5;243m01/14/23 00:51:28.625[0m
[1mSTEP:[0m validating provisioned PV [38;5;243m01/14/23 00:51:28.657[0m
[1mSTEP:[0m checking the PV [38;5;243m01/14/23 00:51:28.69[0m
[1mSTEP:[0m setting up the pod [38;5;243m01/14/23 00:51:28.69[0m
I0114 00:51:28.690881   42005 resource_setup.go:62] adding PV (pvc-dd61a03e-1d58-4a05-8002-22175d54d846) to pod ()
[1mSTEP:[0m deploying the pod [38;5;243m01/14/23 00:51:28.691[0m
[1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/14/23 00:51:28.727[0m
Jan 14 00:51:28.727: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-r4fmk" in namespace "azuredisk-586" to be "Succeeded or Failed"
Jan 14 00:51:28.764: INFO: Pod "azuredisk-volume-tester-r4fmk": Phase="Pending", Reason="", readiness=false. Elapsed: 37.418789ms
Jan 14 00:51:30.799: INFO: Pod "azuredisk-volume-tester-r4fmk": Phase="Pending", Reason="", readiness=false. Elapsed: 2.071922008s
Jan 14 00:51:32.798: INFO: Pod "azuredisk-volume-tester-r4fmk": Phase="Pending", Reason="", readiness=false. Elapsed: 4.070534868s
Jan 14 00:51:34.799: INFO: Pod "azuredisk-volume-tester-r4fmk": Phase="Pending", Reason="", readiness=false. Elapsed: 6.072361504s
Jan 14 00:51:36.800: INFO: Pod "azuredisk-volume-tester-r4fmk": Phase="Pending", Reason="", readiness=false. Elapsed: 8.073504795s
Jan 14 00:51:38.800: INFO: Pod "azuredisk-volume-tester-r4fmk": Phase="Pending", Reason="", readiness=false. Elapsed: 10.072945648s
... skipping 21 lines ...
Jan 14 00:52:22.800: INFO: Pod "azuredisk-volume-tester-r4fmk": Phase="Running", Reason="", readiness=true. Elapsed: 54.072696172s
Jan 14 00:52:24.800: INFO: Pod "azuredisk-volume-tester-r4fmk": Phase="Running", Reason="", readiness=true. Elapsed: 56.0734609s
Jan 14 00:52:26.802: INFO: Pod "azuredisk-volume-tester-r4fmk": Phase="Running", Reason="", readiness=false. Elapsed: 58.075028221s
Jan 14 00:52:28.801: INFO: Pod "azuredisk-volume-tester-r4fmk": Phase="Running", Reason="", readiness=false. Elapsed: 1m0.073572914s
Jan 14 00:52:30.801: INFO: Pod "azuredisk-volume-tester-r4fmk": Phase="Succeeded", Reason="", readiness=false. Elapsed: 1m2.074158779s
[1mSTEP:[0m Saw pod success [38;5;243m01/14/23 00:52:30.801[0m
Jan 14 00:52:30.801: INFO: Pod "azuredisk-volume-tester-r4fmk" satisfied condition "Succeeded or Failed"
Jan 14 00:52:30.801: INFO: deleting Pod "azuredisk-586"/"azuredisk-volume-tester-r4fmk"
Jan 14 00:52:30.863: INFO: Pod azuredisk-volume-tester-r4fmk has the following logs: hello world

[1mSTEP:[0m Deleting pod azuredisk-volume-tester-r4fmk in namespace azuredisk-586 [38;5;243m01/14/23 00:52:30.863[0m
Jan 14 00:52:30.908: INFO: deleting PVC "azuredisk-586"/"pvc-9qcsv"
Jan 14 00:52:30.908: INFO: Deleting PersistentVolumeClaim "pvc-9qcsv"
... skipping 40 lines ...
    [1mSTEP:[0m checking the PVC [38;5;243m01/14/23 00:51:28.625[0m
    [1mSTEP:[0m validating provisioned PV [38;5;243m01/14/23 00:51:28.657[0m
    [1mSTEP:[0m checking the PV [38;5;243m01/14/23 00:51:28.69[0m
    [1mSTEP:[0m setting up the pod [38;5;243m01/14/23 00:51:28.69[0m
    I0114 00:51:28.690881   42005 resource_setup.go:62] adding PV (pvc-dd61a03e-1d58-4a05-8002-22175d54d846) to pod ()
    [1mSTEP:[0m deploying the pod [38;5;243m01/14/23 00:51:28.691[0m
    [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/14/23 00:51:28.727[0m
    Jan 14 00:51:28.727: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-r4fmk" in namespace "azuredisk-586" to be "Succeeded or Failed"
    Jan 14 00:51:28.764: INFO: Pod "azuredisk-volume-tester-r4fmk": Phase="Pending", Reason="", readiness=false. Elapsed: 37.418789ms
    Jan 14 00:51:30.799: INFO: Pod "azuredisk-volume-tester-r4fmk": Phase="Pending", Reason="", readiness=false. Elapsed: 2.071922008s
    Jan 14 00:51:32.798: INFO: Pod "azuredisk-volume-tester-r4fmk": Phase="Pending", Reason="", readiness=false. Elapsed: 4.070534868s
    Jan 14 00:51:34.799: INFO: Pod "azuredisk-volume-tester-r4fmk": Phase="Pending", Reason="", readiness=false. Elapsed: 6.072361504s
    Jan 14 00:51:36.800: INFO: Pod "azuredisk-volume-tester-r4fmk": Phase="Pending", Reason="", readiness=false. Elapsed: 8.073504795s
    Jan 14 00:51:38.800: INFO: Pod "azuredisk-volume-tester-r4fmk": Phase="Pending", Reason="", readiness=false. Elapsed: 10.072945648s
... skipping 21 lines ...
    Jan 14 00:52:22.800: INFO: Pod "azuredisk-volume-tester-r4fmk": Phase="Running", Reason="", readiness=true. Elapsed: 54.072696172s
    Jan 14 00:52:24.800: INFO: Pod "azuredisk-volume-tester-r4fmk": Phase="Running", Reason="", readiness=true. Elapsed: 56.0734609s
    Jan 14 00:52:26.802: INFO: Pod "azuredisk-volume-tester-r4fmk": Phase="Running", Reason="", readiness=false. Elapsed: 58.075028221s
    Jan 14 00:52:28.801: INFO: Pod "azuredisk-volume-tester-r4fmk": Phase="Running", Reason="", readiness=false. Elapsed: 1m0.073572914s
    Jan 14 00:52:30.801: INFO: Pod "azuredisk-volume-tester-r4fmk": Phase="Succeeded", Reason="", readiness=false. Elapsed: 1m2.074158779s
    [1mSTEP:[0m Saw pod success [38;5;243m01/14/23 00:52:30.801[0m
    Jan 14 00:52:30.801: INFO: Pod "azuredisk-volume-tester-r4fmk" satisfied condition "Succeeded or Failed"
    Jan 14 00:52:30.801: INFO: deleting Pod "azuredisk-586"/"azuredisk-volume-tester-r4fmk"
    Jan 14 00:52:30.863: INFO: Pod azuredisk-volume-tester-r4fmk has the following logs: hello world

    [1mSTEP:[0m Deleting pod azuredisk-volume-tester-r4fmk in namespace azuredisk-586 [38;5;243m01/14/23 00:52:30.863[0m
    Jan 14 00:52:30.908: INFO: deleting PVC "azuredisk-586"/"pvc-9qcsv"
    Jan 14 00:52:30.908: INFO: Deleting PersistentVolumeClaim "pvc-9qcsv"
... skipping 39 lines ...
[1mSTEP:[0m checking the PVC [38;5;243m01/14/23 00:53:16.354[0m
[1mSTEP:[0m validating provisioned PV [38;5;243m01/14/23 00:53:16.387[0m
[1mSTEP:[0m checking the PV [38;5;243m01/14/23 00:53:16.421[0m
[1mSTEP:[0m setting up the pod [38;5;243m01/14/23 00:53:16.421[0m
I0114 00:53:16.422263   42005 resource_setup.go:62] adding PV (pvc-f20ea531-8640-4d75-857e-839afa44b163) to pod ()
[1mSTEP:[0m deploying the pod [38;5;243m01/14/23 00:53:16.422[0m
[1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/14/23 00:53:16.46[0m
Jan 14 00:53:16.460: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-2ghv9" in namespace "azuredisk-2453" to be "Succeeded or Failed"
Jan 14 00:53:16.495: INFO: Pod "azuredisk-volume-tester-2ghv9": Phase="Pending", Reason="", readiness=false. Elapsed: 34.949909ms
Jan 14 00:53:18.531: INFO: Pod "azuredisk-volume-tester-2ghv9": Phase="Pending", Reason="", readiness=false. Elapsed: 2.07120351s
Jan 14 00:53:20.529: INFO: Pod "azuredisk-volume-tester-2ghv9": Phase="Pending", Reason="", readiness=false. Elapsed: 4.068603089s
Jan 14 00:53:22.532: INFO: Pod "azuredisk-volume-tester-2ghv9": Phase="Pending", Reason="", readiness=false. Elapsed: 6.071909091s
Jan 14 00:53:24.530: INFO: Pod "azuredisk-volume-tester-2ghv9": Phase="Pending", Reason="", readiness=false. Elapsed: 8.06994233s
Jan 14 00:53:26.529: INFO: Pod "azuredisk-volume-tester-2ghv9": Phase="Pending", Reason="", readiness=false. Elapsed: 10.068587188s
... skipping 18 lines ...
Jan 14 00:54:04.532: INFO: Pod "azuredisk-volume-tester-2ghv9": Phase="Running", Reason="", readiness=true. Elapsed: 48.071842113s
Jan 14 00:54:06.531: INFO: Pod "azuredisk-volume-tester-2ghv9": Phase="Running", Reason="", readiness=true. Elapsed: 50.070671898s
Jan 14 00:54:08.531: INFO: Pod "azuredisk-volume-tester-2ghv9": Phase="Running", Reason="", readiness=true. Elapsed: 52.070798261s
Jan 14 00:54:10.530: INFO: Pod "azuredisk-volume-tester-2ghv9": Phase="Running", Reason="", readiness=false. Elapsed: 54.070196873s
Jan 14 00:54:12.530: INFO: Pod "azuredisk-volume-tester-2ghv9": Phase="Succeeded", Reason="", readiness=false. Elapsed: 56.070305108s
[1mSTEP:[0m Saw pod success [38;5;243m01/14/23 00:54:12.53[0m
Jan 14 00:54:12.531: INFO: Pod "azuredisk-volume-tester-2ghv9" satisfied condition "Succeeded or Failed"
Jan 14 00:54:12.531: INFO: deleting Pod "azuredisk-2453"/"azuredisk-volume-tester-2ghv9"
Jan 14 00:54:12.595: INFO: Pod azuredisk-volume-tester-2ghv9 has the following logs: hello world

[1mSTEP:[0m Deleting pod azuredisk-volume-tester-2ghv9 in namespace azuredisk-2453 [38;5;243m01/14/23 00:54:12.595[0m
Jan 14 00:54:12.638: INFO: deleting PVC "azuredisk-2453"/"pvc-b2jcb"
Jan 14 00:54:12.638: INFO: Deleting PersistentVolumeClaim "pvc-b2jcb"
... skipping 44 lines ...
    [1mSTEP:[0m checking the PVC [38;5;243m01/14/23 00:53:16.354[0m
    [1mSTEP:[0m validating provisioned PV [38;5;243m01/14/23 00:53:16.387[0m
    [1mSTEP:[0m checking the PV [38;5;243m01/14/23 00:53:16.421[0m
    [1mSTEP:[0m setting up the pod [38;5;243m01/14/23 00:53:16.421[0m
    I0114 00:53:16.422263   42005 resource_setup.go:62] adding PV (pvc-f20ea531-8640-4d75-857e-839afa44b163) to pod ()
    [1mSTEP:[0m deploying the pod [38;5;243m01/14/23 00:53:16.422[0m
    [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/14/23 00:53:16.46[0m
    Jan 14 00:53:16.460: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-2ghv9" in namespace "azuredisk-2453" to be "Succeeded or Failed"
    Jan 14 00:53:16.495: INFO: Pod "azuredisk-volume-tester-2ghv9": Phase="Pending", Reason="", readiness=false. Elapsed: 34.949909ms
    Jan 14 00:53:18.531: INFO: Pod "azuredisk-volume-tester-2ghv9": Phase="Pending", Reason="", readiness=false. Elapsed: 2.07120351s
    Jan 14 00:53:20.529: INFO: Pod "azuredisk-volume-tester-2ghv9": Phase="Pending", Reason="", readiness=false. Elapsed: 4.068603089s
    Jan 14 00:53:22.532: INFO: Pod "azuredisk-volume-tester-2ghv9": Phase="Pending", Reason="", readiness=false. Elapsed: 6.071909091s
    Jan 14 00:53:24.530: INFO: Pod "azuredisk-volume-tester-2ghv9": Phase="Pending", Reason="", readiness=false. Elapsed: 8.06994233s
    Jan 14 00:53:26.529: INFO: Pod "azuredisk-volume-tester-2ghv9": Phase="Pending", Reason="", readiness=false. Elapsed: 10.068587188s
... skipping 18 lines ...
    Jan 14 00:54:04.532: INFO: Pod "azuredisk-volume-tester-2ghv9": Phase="Running", Reason="", readiness=true. Elapsed: 48.071842113s
    Jan 14 00:54:06.531: INFO: Pod "azuredisk-volume-tester-2ghv9": Phase="Running", Reason="", readiness=true. Elapsed: 50.070671898s
    Jan 14 00:54:08.531: INFO: Pod "azuredisk-volume-tester-2ghv9": Phase="Running", Reason="", readiness=true. Elapsed: 52.070798261s
    Jan 14 00:54:10.530: INFO: Pod "azuredisk-volume-tester-2ghv9": Phase="Running", Reason="", readiness=false. Elapsed: 54.070196873s
    Jan 14 00:54:12.530: INFO: Pod "azuredisk-volume-tester-2ghv9": Phase="Succeeded", Reason="", readiness=false. Elapsed: 56.070305108s
    [1mSTEP:[0m Saw pod success [38;5;243m01/14/23 00:54:12.53[0m
    Jan 14 00:54:12.531: INFO: Pod "azuredisk-volume-tester-2ghv9" satisfied condition "Succeeded or Failed"
    Jan 14 00:54:12.531: INFO: deleting Pod "azuredisk-2453"/"azuredisk-volume-tester-2ghv9"
    Jan 14 00:54:12.595: INFO: Pod azuredisk-volume-tester-2ghv9 has the following logs: hello world

    [1mSTEP:[0m Deleting pod azuredisk-volume-tester-2ghv9 in namespace azuredisk-2453 [38;5;243m01/14/23 00:54:12.595[0m
    Jan 14 00:54:12.638: INFO: deleting PVC "azuredisk-2453"/"pvc-b2jcb"
    Jan 14 00:54:12.638: INFO: Deleting PersistentVolumeClaim "pvc-b2jcb"
... skipping 115 lines ...
[1mSTEP:[0m checking the PVC [38;5;243m01/14/23 00:55:19.219[0m
[1mSTEP:[0m validating provisioned PV [38;5;243m01/14/23 00:55:19.254[0m
[1mSTEP:[0m checking the PV [38;5;243m01/14/23 00:55:19.287[0m
[1mSTEP:[0m setting up the pod [38;5;243m01/14/23 00:55:19.287[0m
I0114 00:55:19.287235   42005 resource_setup.go:62] adding PV (pvc-36583e7e-1b88-40ea-844e-7d41221bd24b) to pod ()
[1mSTEP:[0m deploying the pod [38;5;243m01/14/23 00:55:19.287[0m
[1mSTEP:[0m checking that the pod's command exits with an error [38;5;243m01/14/23 00:55:19.321[0m
Jan 14 00:55:19.321: INFO: Waiting up to 10m0s for pod "azuredisk-volume-tester-gkdf4" in namespace "azuredisk-7540" to be "Error status code"
Jan 14 00:55:19.364: INFO: Pod "azuredisk-volume-tester-gkdf4": Phase="Pending", Reason="", readiness=false. Elapsed: 42.419477ms
Jan 14 00:55:21.396: INFO: Pod "azuredisk-volume-tester-gkdf4": Phase="Pending", Reason="", readiness=false. Elapsed: 2.075041101s
Jan 14 00:55:23.398: INFO: Pod "azuredisk-volume-tester-gkdf4": Phase="Pending", Reason="", readiness=false. Elapsed: 4.076831801s
Jan 14 00:55:25.397: INFO: Pod "azuredisk-volume-tester-gkdf4": Phase="Pending", Reason="", readiness=false. Elapsed: 6.076177121s
Jan 14 00:55:27.397: INFO: Pod "azuredisk-volume-tester-gkdf4": Phase="Pending", Reason="", readiness=false. Elapsed: 8.075274976s
Jan 14 00:55:29.398: INFO: Pod "azuredisk-volume-tester-gkdf4": Phase="Pending", Reason="", readiness=false. Elapsed: 10.076422409s
... skipping 17 lines ...
Jan 14 00:56:05.399: INFO: Pod "azuredisk-volume-tester-gkdf4": Phase="Running", Reason="", readiness=true. Elapsed: 46.078014468s
Jan 14 00:56:07.400: INFO: Pod "azuredisk-volume-tester-gkdf4": Phase="Running", Reason="", readiness=true. Elapsed: 48.079075188s
Jan 14 00:56:09.399: INFO: Pod "azuredisk-volume-tester-gkdf4": Phase="Running", Reason="", readiness=true. Elapsed: 50.078103892s
Jan 14 00:56:11.399: INFO: Pod "azuredisk-volume-tester-gkdf4": Phase="Running", Reason="", readiness=false. Elapsed: 52.077576849s
Jan 14 00:56:13.399: INFO: Pod "azuredisk-volume-tester-gkdf4": Phase="Running", Reason="", readiness=false. Elapsed: 54.077943285s
Jan 14 00:56:15.404: INFO: Pod "azuredisk-volume-tester-gkdf4": Phase="Running", Reason="", readiness=false. Elapsed: 56.082877743s
Jan 14 00:56:17.400: INFO: Pod "azuredisk-volume-tester-gkdf4": Phase="Failed", Reason="", readiness=false. Elapsed: 58.078814579s
[1mSTEP:[0m Saw pod failure [38;5;243m01/14/23 00:56:17.4[0m
Jan 14 00:56:17.400: INFO: Pod "azuredisk-volume-tester-gkdf4" satisfied condition "Error status code"
[1mSTEP:[0m checking that pod logs contain expected message [38;5;243m01/14/23 00:56:17.4[0m
Jan 14 00:56:17.465: INFO: deleting Pod "azuredisk-7540"/"azuredisk-volume-tester-gkdf4"
Jan 14 00:56:17.502: INFO: Pod azuredisk-volume-tester-gkdf4 has the following logs: out-file : Access to the path 'C:\mnt\test-1\data' is denied.
At line:1 char:1
+ echo $null >> C:\mnt\test-1\data
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
... skipping 49 lines ...
    [1mSTEP:[0m checking the PVC [38;5;243m01/14/23 00:55:19.219[0m
    [1mSTEP:[0m validating provisioned PV [38;5;243m01/14/23 00:55:19.254[0m
    [1mSTEP:[0m checking the PV [38;5;243m01/14/23 00:55:19.287[0m
    [1mSTEP:[0m setting up the pod [38;5;243m01/14/23 00:55:19.287[0m
    I0114 00:55:19.287235   42005 resource_setup.go:62] adding PV (pvc-36583e7e-1b88-40ea-844e-7d41221bd24b) to pod ()
    [1mSTEP:[0m deploying the pod [38;5;243m01/14/23 00:55:19.287[0m
    [1mSTEP:[0m checking that the pod's command exits with an error [38;5;243m01/14/23 00:55:19.321[0m
    Jan 14 00:55:19.321: INFO: Waiting up to 10m0s for pod "azuredisk-volume-tester-gkdf4" in namespace "azuredisk-7540" to be "Error status code"
    Jan 14 00:55:19.364: INFO: Pod "azuredisk-volume-tester-gkdf4": Phase="Pending", Reason="", readiness=false. Elapsed: 42.419477ms
    Jan 14 00:55:21.396: INFO: Pod "azuredisk-volume-tester-gkdf4": Phase="Pending", Reason="", readiness=false. Elapsed: 2.075041101s
    Jan 14 00:55:23.398: INFO: Pod "azuredisk-volume-tester-gkdf4": Phase="Pending", Reason="", readiness=false. Elapsed: 4.076831801s
    Jan 14 00:55:25.397: INFO: Pod "azuredisk-volume-tester-gkdf4": Phase="Pending", Reason="", readiness=false. Elapsed: 6.076177121s
    Jan 14 00:55:27.397: INFO: Pod "azuredisk-volume-tester-gkdf4": Phase="Pending", Reason="", readiness=false. Elapsed: 8.075274976s
    Jan 14 00:55:29.398: INFO: Pod "azuredisk-volume-tester-gkdf4": Phase="Pending", Reason="", readiness=false. Elapsed: 10.076422409s
... skipping 17 lines ...
    Jan 14 00:56:05.399: INFO: Pod "azuredisk-volume-tester-gkdf4": Phase="Running", Reason="", readiness=true. Elapsed: 46.078014468s
    Jan 14 00:56:07.400: INFO: Pod "azuredisk-volume-tester-gkdf4": Phase="Running", Reason="", readiness=true. Elapsed: 48.079075188s
    Jan 14 00:56:09.399: INFO: Pod "azuredisk-volume-tester-gkdf4": Phase="Running", Reason="", readiness=true. Elapsed: 50.078103892s
    Jan 14 00:56:11.399: INFO: Pod "azuredisk-volume-tester-gkdf4": Phase="Running", Reason="", readiness=false. Elapsed: 52.077576849s
    Jan 14 00:56:13.399: INFO: Pod "azuredisk-volume-tester-gkdf4": Phase="Running", Reason="", readiness=false. Elapsed: 54.077943285s
    Jan 14 00:56:15.404: INFO: Pod "azuredisk-volume-tester-gkdf4": Phase="Running", Reason="", readiness=false. Elapsed: 56.082877743s
    Jan 14 00:56:17.400: INFO: Pod "azuredisk-volume-tester-gkdf4": Phase="Failed", Reason="", readiness=false. Elapsed: 58.078814579s
    [1mSTEP:[0m Saw pod failure [38;5;243m01/14/23 00:56:17.4[0m
    Jan 14 00:56:17.400: INFO: Pod "azuredisk-volume-tester-gkdf4" satisfied condition "Error status code"
    [1mSTEP:[0m checking that pod logs contain expected message [38;5;243m01/14/23 00:56:17.4[0m
    Jan 14 00:56:17.465: INFO: deleting Pod "azuredisk-7540"/"azuredisk-volume-tester-gkdf4"
    Jan 14 00:56:17.502: INFO: Pod azuredisk-volume-tester-gkdf4 has the following logs: out-file : Access to the path 'C:\mnt\test-1\data' is denied.
    At line:1 char:1
    + echo $null >> C:\mnt\test-1\data
    + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
... skipping 477 lines ...
Jan 14 01:02:33.194: INFO: deployment status: v1.DeploymentStatus{ObservedGeneration:1, Replicas:1, UpdatedReplicas:1, ReadyReplicas:0, AvailableReplicas:0, UnavailableReplicas:1, Conditions:[]v1.DeploymentCondition{v1.DeploymentCondition{Type:"Available", Status:"False", LastUpdateTime:time.Date(2023, time.January, 14, 1, 1, 47, 0, time.Local), LastTransitionTime:time.Date(2023, time.January, 14, 1, 1, 47, 0, time.Local), Reason:"MinimumReplicasUnavailable", Message:"Deployment does not have minimum availability."}, v1.DeploymentCondition{Type:"Progressing", Status:"True", LastUpdateTime:time.Date(2023, time.January, 14, 1, 1, 47, 0, time.Local), LastTransitionTime:time.Date(2023, time.January, 14, 1, 1, 47, 0, time.Local), Reason:"ReplicaSetUpdated", Message:"ReplicaSet \"azuredisk-volume-tester-d6cl6-6f9f44f987\" is progressing."}}, CollisionCount:(*int32)(nil)}
Jan 14 01:02:35.193: INFO: deployment status: v1.DeploymentStatus{ObservedGeneration:1, Replicas:1, UpdatedReplicas:1, ReadyReplicas:0, AvailableReplicas:0, UnavailableReplicas:1, Conditions:[]v1.DeploymentCondition{v1.DeploymentCondition{Type:"Available", Status:"False", LastUpdateTime:time.Date(2023, time.January, 14, 1, 1, 47, 0, time.Local), LastTransitionTime:time.Date(2023, time.January, 14, 1, 1, 47, 0, time.Local), Reason:"MinimumReplicasUnavailable", Message:"Deployment does not have minimum availability."}, v1.DeploymentCondition{Type:"Progressing", Status:"True", LastUpdateTime:time.Date(2023, time.January, 14, 1, 1, 47, 0, time.Local), LastTransitionTime:time.Date(2023, time.January, 14, 1, 1, 47, 0, time.Local), Reason:"ReplicaSetUpdated", Message:"ReplicaSet \"azuredisk-volume-tester-d6cl6-6f9f44f987\" is progressing."}}, CollisionCount:(*int32)(nil)}
[1mSTEP:[0m checking that the pod is running [38;5;243m01/14/23 01:02:37.26[0m
[1mSTEP:[0m check pod exec [38;5;243m01/14/23 01:02:37.36[0m
Jan 14 01:02:37.361: INFO: Running '/usr/local/bin/kubectl --kubeconfig=/home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure/kubeconfig --namespace=azuredisk-3267 exec azuredisk-volume-tester-d6cl6-6f9f44f987-nb8qb -- cmd /c type C:\mnt\test-1\data.txt'
Jan 14 01:02:37.981: INFO: rc: 1
Jan 14 01:02:37.981: INFO: Error waiting for output "hello world\r\n" in pod "azuredisk-volume-tester-d6cl6-6f9f44f987-nb8qb": error running /usr/local/bin/kubectl --kubeconfig=/home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure/kubeconfig --namespace=azuredisk-3267 exec azuredisk-volume-tester-d6cl6-6f9f44f987-nb8qb -- cmd /c type C:\mnt\test-1\data.txt:
Command stdout:

stderr:
The system cannot find the file specified.
command terminated with exit code 1

error:
exit status 1.
Jan 14 01:02:39.982: INFO: Running '/usr/local/bin/kubectl --kubeconfig=/home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure/kubeconfig --namespace=azuredisk-3267 exec azuredisk-volume-tester-d6cl6-6f9f44f987-nb8qb -- cmd /c type C:\mnt\test-1\data.txt'
Jan 14 01:02:40.569: INFO: rc: 1
Jan 14 01:02:40.569: INFO: Error waiting for output "hello world\r\n" in pod "azuredisk-volume-tester-d6cl6-6f9f44f987-nb8qb": error running /usr/local/bin/kubectl --kubeconfig=/home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure/kubeconfig --namespace=azuredisk-3267 exec azuredisk-volume-tester-d6cl6-6f9f44f987-nb8qb -- cmd /c type C:\mnt\test-1\data.txt:
Command stdout:

stderr:
The system cannot find the file specified.
command terminated with exit code 1

error:
exit status 1.
Jan 14 01:02:41.981: INFO: Running '/usr/local/bin/kubectl --kubeconfig=/home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure/kubeconfig --namespace=azuredisk-3267 exec azuredisk-volume-tester-d6cl6-6f9f44f987-nb8qb -- cmd /c type C:\mnt\test-1\data.txt'
Jan 14 01:02:42.554: INFO: stderr: ""
Jan 14 01:02:42.554: INFO: stdout: "hello world\r\n"
[1mSTEP:[0m deleting the pod for deployment [38;5;243m01/14/23 01:02:42.554[0m
Jan 14 01:02:42.554: INFO: Deleting pod "azuredisk-volume-tester-d6cl6-6f9f44f987-nb8qb" in namespace "azuredisk-3267"
... skipping 81 lines ...
    Jan 14 01:02:33.194: INFO: deployment status: v1.DeploymentStatus{ObservedGeneration:1, Replicas:1, UpdatedReplicas:1, ReadyReplicas:0, AvailableReplicas:0, UnavailableReplicas:1, Conditions:[]v1.DeploymentCondition{v1.DeploymentCondition{Type:"Available", Status:"False", LastUpdateTime:time.Date(2023, time.January, 14, 1, 1, 47, 0, time.Local), LastTransitionTime:time.Date(2023, time.January, 14, 1, 1, 47, 0, time.Local), Reason:"MinimumReplicasUnavailable", Message:"Deployment does not have minimum availability."}, v1.DeploymentCondition{Type:"Progressing", Status:"True", LastUpdateTime:time.Date(2023, time.January, 14, 1, 1, 47, 0, time.Local), LastTransitionTime:time.Date(2023, time.January, 14, 1, 1, 47, 0, time.Local), Reason:"ReplicaSetUpdated", Message:"ReplicaSet \"azuredisk-volume-tester-d6cl6-6f9f44f987\" is progressing."}}, CollisionCount:(*int32)(nil)}
    Jan 14 01:02:35.193: INFO: deployment status: v1.DeploymentStatus{ObservedGeneration:1, Replicas:1, UpdatedReplicas:1, ReadyReplicas:0, AvailableReplicas:0, UnavailableReplicas:1, Conditions:[]v1.DeploymentCondition{v1.DeploymentCondition{Type:"Available", Status:"False", LastUpdateTime:time.Date(2023, time.January, 14, 1, 1, 47, 0, time.Local), LastTransitionTime:time.Date(2023, time.January, 14, 1, 1, 47, 0, time.Local), Reason:"MinimumReplicasUnavailable", Message:"Deployment does not have minimum availability."}, v1.DeploymentCondition{Type:"Progressing", Status:"True", LastUpdateTime:time.Date(2023, time.January, 14, 1, 1, 47, 0, time.Local), LastTransitionTime:time.Date(2023, time.January, 14, 1, 1, 47, 0, time.Local), Reason:"ReplicaSetUpdated", Message:"ReplicaSet \"azuredisk-volume-tester-d6cl6-6f9f44f987\" is progressing."}}, CollisionCount:(*int32)(nil)}
    [1mSTEP:[0m checking that the pod is running [38;5;243m01/14/23 01:02:37.26[0m
    [1mSTEP:[0m check pod exec [38;5;243m01/14/23 01:02:37.36[0m
    Jan 14 01:02:37.361: INFO: Running '/usr/local/bin/kubectl --kubeconfig=/home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure/kubeconfig --namespace=azuredisk-3267 exec azuredisk-volume-tester-d6cl6-6f9f44f987-nb8qb -- cmd /c type C:\mnt\test-1\data.txt'
    Jan 14 01:02:37.981: INFO: rc: 1
    Jan 14 01:02:37.981: INFO: Error waiting for output "hello world\r\n" in pod "azuredisk-volume-tester-d6cl6-6f9f44f987-nb8qb": error running /usr/local/bin/kubectl --kubeconfig=/home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure/kubeconfig --namespace=azuredisk-3267 exec azuredisk-volume-tester-d6cl6-6f9f44f987-nb8qb -- cmd /c type C:\mnt\test-1\data.txt:
    Command stdout:

    stderr:
    The system cannot find the file specified.
    command terminated with exit code 1

    error:
    exit status 1.
    Jan 14 01:02:39.982: INFO: Running '/usr/local/bin/kubectl --kubeconfig=/home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure/kubeconfig --namespace=azuredisk-3267 exec azuredisk-volume-tester-d6cl6-6f9f44f987-nb8qb -- cmd /c type C:\mnt\test-1\data.txt'
    Jan 14 01:02:40.569: INFO: rc: 1
    Jan 14 01:02:40.569: INFO: Error waiting for output "hello world\r\n" in pod "azuredisk-volume-tester-d6cl6-6f9f44f987-nb8qb": error running /usr/local/bin/kubectl --kubeconfig=/home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure/kubeconfig --namespace=azuredisk-3267 exec azuredisk-volume-tester-d6cl6-6f9f44f987-nb8qb -- cmd /c type C:\mnt\test-1\data.txt:
    Command stdout:

    stderr:
    The system cannot find the file specified.
    command terminated with exit code 1

    error:
    exit status 1.
    Jan 14 01:02:41.981: INFO: Running '/usr/local/bin/kubectl --kubeconfig=/home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure/kubeconfig --namespace=azuredisk-3267 exec azuredisk-volume-tester-d6cl6-6f9f44f987-nb8qb -- cmd /c type C:\mnt\test-1\data.txt'
    Jan 14 01:02:42.554: INFO: stderr: ""
    Jan 14 01:02:42.554: INFO: stdout: "hello world\r\n"
    [1mSTEP:[0m deleting the pod for deployment [38;5;243m01/14/23 01:02:42.554[0m
    Jan 14 01:02:42.554: INFO: Deleting pod "azuredisk-volume-tester-d6cl6-6f9f44f987-nb8qb" in namespace "azuredisk-3267"
... skipping 308 lines ...
[1mSTEP:[0m checking the PVC [38;5;243m01/14/23 01:04:19.129[0m
[1mSTEP:[0m validating provisioned PV [38;5;243m01/14/23 01:04:19.162[0m
[1mSTEP:[0m checking the PV [38;5;243m01/14/23 01:04:19.194[0m
[1mSTEP:[0m setting up the pod [38;5;243m01/14/23 01:04:19.194[0m
I0114 01:04:19.194636   42005 resource_setup.go:62] adding PV (pvc-31106f4c-6012-43b0-904c-68f7d4c3512f) to pod ()
[1mSTEP:[0m deploying the pod [38;5;243m01/14/23 01:04:19.194[0m
[1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/14/23 01:04:19.233[0m
Jan 14 01:04:19.233: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-2q4dc" in namespace "azuredisk-7247" to be "Succeeded or Failed"
Jan 14 01:04:19.267: INFO: Pod "azuredisk-volume-tester-2q4dc": Phase="Pending", Reason="", readiness=false. Elapsed: 33.82989ms
Jan 14 01:04:21.302: INFO: Pod "azuredisk-volume-tester-2q4dc": Phase="Pending", Reason="", readiness=false. Elapsed: 2.069091536s
Jan 14 01:04:23.302: INFO: Pod "azuredisk-volume-tester-2q4dc": Phase="Pending", Reason="", readiness=false. Elapsed: 4.069210106s
Jan 14 01:04:25.302: INFO: Pod "azuredisk-volume-tester-2q4dc": Phase="Pending", Reason="", readiness=false. Elapsed: 6.06927912s
Jan 14 01:04:27.302: INFO: Pod "azuredisk-volume-tester-2q4dc": Phase="Pending", Reason="", readiness=false. Elapsed: 8.069250307s
Jan 14 01:04:29.302: INFO: Pod "azuredisk-volume-tester-2q4dc": Phase="Pending", Reason="", readiness=false. Elapsed: 10.069220773s
... skipping 25 lines ...
Jan 14 01:05:21.303: INFO: Pod "azuredisk-volume-tester-2q4dc": Phase="Running", Reason="", readiness=true. Elapsed: 1m2.069944044s
Jan 14 01:05:23.303: INFO: Pod "azuredisk-volume-tester-2q4dc": Phase="Running", Reason="", readiness=false. Elapsed: 1m4.070056026s
Jan 14 01:05:25.304: INFO: Pod "azuredisk-volume-tester-2q4dc": Phase="Running", Reason="", readiness=false. Elapsed: 1m6.071308544s
Jan 14 01:05:27.303: INFO: Pod "azuredisk-volume-tester-2q4dc": Phase="Running", Reason="", readiness=false. Elapsed: 1m8.069902804s
Jan 14 01:05:29.302: INFO: Pod "azuredisk-volume-tester-2q4dc": Phase="Succeeded", Reason="", readiness=false. Elapsed: 1m10.06911628s
[1mSTEP:[0m Saw pod success [38;5;243m01/14/23 01:05:29.302[0m
Jan 14 01:05:29.302: INFO: Pod "azuredisk-volume-tester-2q4dc" satisfied condition "Succeeded or Failed"
Jan 14 01:05:29.302: INFO: deleting Pod "azuredisk-7247"/"azuredisk-volume-tester-2q4dc"
Jan 14 01:05:29.361: INFO: Pod azuredisk-volume-tester-2q4dc has the following logs: hello world
hello world
hello world

[1mSTEP:[0m Deleting pod azuredisk-volume-tester-2q4dc in namespace azuredisk-7247 [38;5;243m01/14/23 01:05:29.361[0m
... skipping 89 lines ...
    [1mSTEP:[0m checking the PVC [38;5;243m01/14/23 01:04:19.129[0m
    [1mSTEP:[0m validating provisioned PV [38;5;243m01/14/23 01:04:19.162[0m
    [1mSTEP:[0m checking the PV [38;5;243m01/14/23 01:04:19.194[0m
    [1mSTEP:[0m setting up the pod [38;5;243m01/14/23 01:04:19.194[0m
    I0114 01:04:19.194636   42005 resource_setup.go:62] adding PV (pvc-31106f4c-6012-43b0-904c-68f7d4c3512f) to pod ()
    [1mSTEP:[0m deploying the pod [38;5;243m01/14/23 01:04:19.194[0m
    [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/14/23 01:04:19.233[0m
    Jan 14 01:04:19.233: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-2q4dc" in namespace "azuredisk-7247" to be "Succeeded or Failed"
    Jan 14 01:04:19.267: INFO: Pod "azuredisk-volume-tester-2q4dc": Phase="Pending", Reason="", readiness=false. Elapsed: 33.82989ms
    Jan 14 01:04:21.302: INFO: Pod "azuredisk-volume-tester-2q4dc": Phase="Pending", Reason="", readiness=false. Elapsed: 2.069091536s
    Jan 14 01:04:23.302: INFO: Pod "azuredisk-volume-tester-2q4dc": Phase="Pending", Reason="", readiness=false. Elapsed: 4.069210106s
    Jan 14 01:04:25.302: INFO: Pod "azuredisk-volume-tester-2q4dc": Phase="Pending", Reason="", readiness=false. Elapsed: 6.06927912s
    Jan 14 01:04:27.302: INFO: Pod "azuredisk-volume-tester-2q4dc": Phase="Pending", Reason="", readiness=false. Elapsed: 8.069250307s
    Jan 14 01:04:29.302: INFO: Pod "azuredisk-volume-tester-2q4dc": Phase="Pending", Reason="", readiness=false. Elapsed: 10.069220773s
... skipping 25 lines ...
    Jan 14 01:05:21.303: INFO: Pod "azuredisk-volume-tester-2q4dc": Phase="Running", Reason="", readiness=true. Elapsed: 1m2.069944044s
    Jan 14 01:05:23.303: INFO: Pod "azuredisk-volume-tester-2q4dc": Phase="Running", Reason="", readiness=false. Elapsed: 1m4.070056026s
    Jan 14 01:05:25.304: INFO: Pod "azuredisk-volume-tester-2q4dc": Phase="Running", Reason="", readiness=false. Elapsed: 1m6.071308544s
    Jan 14 01:05:27.303: INFO: Pod "azuredisk-volume-tester-2q4dc": Phase="Running", Reason="", readiness=false. Elapsed: 1m8.069902804s
    Jan 14 01:05:29.302: INFO: Pod "azuredisk-volume-tester-2q4dc": Phase="Succeeded", Reason="", readiness=false. Elapsed: 1m10.06911628s
    [1mSTEP:[0m Saw pod success [38;5;243m01/14/23 01:05:29.302[0m
    Jan 14 01:05:29.302: INFO: Pod "azuredisk-volume-tester-2q4dc" satisfied condition "Succeeded or Failed"
    Jan 14 01:05:29.302: INFO: deleting Pod "azuredisk-7247"/"azuredisk-volume-tester-2q4dc"
    Jan 14 01:05:29.361: INFO: Pod azuredisk-volume-tester-2q4dc has the following logs: hello world
    hello world
    hello world

    [1mSTEP:[0m Deleting pod azuredisk-volume-tester-2q4dc in namespace azuredisk-7247 [38;5;243m01/14/23 01:05:29.361[0m
... skipping 94 lines ...
Jan 14 01:06:28.601: INFO: PersistentVolumeClaim pvc-zvjtn found but phase is Pending instead of Bound.
Jan 14 01:06:30.633: INFO: PersistentVolumeClaim pvc-zvjtn found and phase=Bound (4.106748676s)
[1mSTEP:[0m checking the PVC [38;5;243m01/14/23 01:06:30.633[0m
[1mSTEP:[0m validating provisioned PV [38;5;243m01/14/23 01:06:30.666[0m
[1mSTEP:[0m checking the PV [38;5;243m01/14/23 01:06:30.699[0m
[1mSTEP:[0m deploying the pod [38;5;243m01/14/23 01:06:30.699[0m
[1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/14/23 01:06:30.735[0m
Jan 14 01:06:30.736: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-n78bx" in namespace "azuredisk-1594" to be "Succeeded or Failed"
Jan 14 01:06:30.767: INFO: Pod "azuredisk-volume-tester-n78bx": Phase="Pending", Reason="", readiness=false. Elapsed: 31.464758ms
Jan 14 01:06:32.800: INFO: Pod "azuredisk-volume-tester-n78bx": Phase="Pending", Reason="", readiness=false. Elapsed: 2.064184412s
Jan 14 01:06:34.801: INFO: Pod "azuredisk-volume-tester-n78bx": Phase="Pending", Reason="", readiness=false. Elapsed: 4.065794151s
Jan 14 01:06:36.800: INFO: Pod "azuredisk-volume-tester-n78bx": Phase="Pending", Reason="", readiness=false. Elapsed: 6.064328543s
Jan 14 01:06:38.801: INFO: Pod "azuredisk-volume-tester-n78bx": Phase="Pending", Reason="", readiness=false. Elapsed: 8.065266048s
Jan 14 01:06:40.801: INFO: Pod "azuredisk-volume-tester-n78bx": Phase="Pending", Reason="", readiness=false. Elapsed: 10.065412647s
... skipping 19 lines ...
Jan 14 01:07:20.803: INFO: Pod "azuredisk-volume-tester-n78bx": Phase="Running", Reason="", readiness=true. Elapsed: 50.067508455s
Jan 14 01:07:22.803: INFO: Pod "azuredisk-volume-tester-n78bx": Phase="Running", Reason="", readiness=false. Elapsed: 52.067543959s
Jan 14 01:07:24.802: INFO: Pod "azuredisk-volume-tester-n78bx": Phase="Running", Reason="", readiness=false. Elapsed: 54.066748262s
Jan 14 01:07:26.802: INFO: Pod "azuredisk-volume-tester-n78bx": Phase="Running", Reason="", readiness=false. Elapsed: 56.066924543s
Jan 14 01:07:28.803: INFO: Pod "azuredisk-volume-tester-n78bx": Phase="Succeeded", Reason="", readiness=false. Elapsed: 58.067024313s
[1mSTEP:[0m Saw pod success [38;5;243m01/14/23 01:07:28.803[0m
Jan 14 01:07:28.803: INFO: Pod "azuredisk-volume-tester-n78bx" satisfied condition "Succeeded or Failed"
Jan 14 01:07:28.836: INFO: deleting Pod "azuredisk-1594"/"azuredisk-volume-tester-n78bx"
Jan 14 01:07:28.884: INFO: Pod azuredisk-volume-tester-n78bx has the following logs: 
[1mSTEP:[0m Deleting pod azuredisk-volume-tester-n78bx in namespace azuredisk-1594 [38;5;243m01/14/23 01:07:28.884[0m
[1mSTEP:[0m waiting for disk to detach from node [38;5;243m01/14/23 01:07:28.927[0m
[1mSTEP:[0m Checking Prow test resource group [38;5;243m01/14/23 01:07:54.992[0m
[1mSTEP:[0m Prow test resource group: capz-thn7c4 [38;5;243m01/14/23 01:07:54.994[0m
... skipping 16 lines ...
[1mSTEP:[0m checking the PVC [38;5;243m01/14/23 01:08:15.528[0m
[1mSTEP:[0m validating provisioned PV [38;5;243m01/14/23 01:08:15.561[0m
[1mSTEP:[0m checking the PV [38;5;243m01/14/23 01:08:15.593[0m
[1mSTEP:[0m setting up the pod [38;5;243m01/14/23 01:08:15.593[0m
I0114 01:08:15.594035   42005 resource_setup.go:62] adding PV (pvc-6dc7122d-cd59-4713-8699-fcc00d6b3df7) to pod ()
[1mSTEP:[0m deploying a pod with a volume restored from the snapshot [38;5;243m01/14/23 01:08:15.594[0m
[1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/14/23 01:08:15.633[0m
Jan 14 01:08:15.634: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-ndrnp" in namespace "azuredisk-1594" to be "Succeeded or Failed"
Jan 14 01:08:15.666: INFO: Pod "azuredisk-volume-tester-ndrnp": Phase="Pending", Reason="", readiness=false. Elapsed: 32.02418ms
Jan 14 01:08:17.699: INFO: Pod "azuredisk-volume-tester-ndrnp": Phase="Pending", Reason="", readiness=false. Elapsed: 2.065730275s
Jan 14 01:08:19.700: INFO: Pod "azuredisk-volume-tester-ndrnp": Phase="Pending", Reason="", readiness=false. Elapsed: 4.06629342s
Jan 14 01:08:21.700: INFO: Pod "azuredisk-volume-tester-ndrnp": Phase="Pending", Reason="", readiness=false. Elapsed: 6.066786472s
Jan 14 01:08:23.700: INFO: Pod "azuredisk-volume-tester-ndrnp": Phase="Pending", Reason="", readiness=false. Elapsed: 8.066273699s
Jan 14 01:08:25.699: INFO: Pod "azuredisk-volume-tester-ndrnp": Phase="Pending", Reason="", readiness=false. Elapsed: 10.065630779s
... skipping 13 lines ...
Jan 14 01:08:53.701: INFO: Pod "azuredisk-volume-tester-ndrnp": Phase="Running", Reason="", readiness=true. Elapsed: 38.067711299s
Jan 14 01:08:55.702: INFO: Pod "azuredisk-volume-tester-ndrnp": Phase="Running", Reason="", readiness=true. Elapsed: 40.068787445s
Jan 14 01:08:57.700: INFO: Pod "azuredisk-volume-tester-ndrnp": Phase="Running", Reason="", readiness=false. Elapsed: 42.066529178s
Jan 14 01:08:59.702: INFO: Pod "azuredisk-volume-tester-ndrnp": Phase="Running", Reason="", readiness=false. Elapsed: 44.068174966s
Jan 14 01:09:01.700: INFO: Pod "azuredisk-volume-tester-ndrnp": Phase="Succeeded", Reason="", readiness=false. Elapsed: 46.066739726s
[1mSTEP:[0m Saw pod success [38;5;243m01/14/23 01:09:01.7[0m
Jan 14 01:09:01.701: INFO: Pod "azuredisk-volume-tester-ndrnp" satisfied condition "Succeeded or Failed"
Jan 14 01:09:01.701: INFO: deleting Pod "azuredisk-1594"/"azuredisk-volume-tester-ndrnp"
Jan 14 01:09:01.760: INFO: Pod azuredisk-volume-tester-ndrnp has the following logs: hello world

[1mSTEP:[0m Deleting pod azuredisk-volume-tester-ndrnp in namespace azuredisk-1594 [38;5;243m01/14/23 01:09:01.76[0m
Jan 14 01:09:01.829: INFO: deleting PVC "azuredisk-1594"/"pvc-r567t"
Jan 14 01:09:01.829: INFO: Deleting PersistentVolumeClaim "pvc-r567t"
... skipping 12 lines ...
Jan 14 01:09:42.209: INFO: Claim "pvc-r567t" in namespace "azuredisk-1594" doesn't exist in the system
Jan 14 01:09:42.209: INFO: deleting StorageClass azuredisk-1594-disk.csi.azure.com-dynamic-sc-wnm8s
[1mSTEP:[0m deleting a VolumeSnapshot volume-snapshot-dqbhx [38;5;243m01/14/23 01:09:42.244[0m
Jan 14 01:09:42.283: INFO: skip deleting VolumeSnapshotClass azuredisk-1594-disk.csi.azure.com-dynamic-sc-t4csv
Jan 14 01:09:42.283: INFO: Deleting resource group azuredisk-csi-driver-test-e04751e7-93a7-11ed-abfb-260d6d03632e
Jan 14 01:10:58.901: INFO: deleting Pod "azuredisk-1594"/"azuredisk-volume-tester-n78bx"
Jan 14 01:10:58.933: INFO: Error getting logs for pod azuredisk-volume-tester-n78bx: the server could not find the requested resource (get pods azuredisk-volume-tester-n78bx)
[1mSTEP:[0m Deleting pod azuredisk-volume-tester-n78bx in namespace azuredisk-1594 [38;5;243m01/14/23 01:10:58.933[0m
Jan 14 01:10:58.966: INFO: deleting PVC "azuredisk-1594"/"pvc-zvjtn"
Jan 14 01:10:58.966: INFO: Deleting PersistentVolumeClaim "pvc-zvjtn"
[1mSTEP:[0m waiting for claim's PV "pvc-604c4585-6ac4-459f-869c-a9124bb6d3c0" to be deleted [38;5;243m01/14/23 01:10:59.004[0m
Jan 14 01:10:59.004: INFO: Waiting up to 10m0s for PersistentVolume pvc-604c4585-6ac4-459f-869c-a9124bb6d3c0 to get deleted
Jan 14 01:10:59.037: INFO: PersistentVolume pvc-604c4585-6ac4-459f-869c-a9124bb6d3c0 found and phase=Released (32.73236ms)
... skipping 29 lines ...
    Jan 14 01:06:28.601: INFO: PersistentVolumeClaim pvc-zvjtn found but phase is Pending instead of Bound.
    Jan 14 01:06:30.633: INFO: PersistentVolumeClaim pvc-zvjtn found and phase=Bound (4.106748676s)
    [1mSTEP:[0m checking the PVC [38;5;243m01/14/23 01:06:30.633[0m
    [1mSTEP:[0m validating provisioned PV [38;5;243m01/14/23 01:06:30.666[0m
    [1mSTEP:[0m checking the PV [38;5;243m01/14/23 01:06:30.699[0m
    [1mSTEP:[0m deploying the pod [38;5;243m01/14/23 01:06:30.699[0m
    [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/14/23 01:06:30.735[0m
    Jan 14 01:06:30.736: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-n78bx" in namespace "azuredisk-1594" to be "Succeeded or Failed"
    Jan 14 01:06:30.767: INFO: Pod "azuredisk-volume-tester-n78bx": Phase="Pending", Reason="", readiness=false. Elapsed: 31.464758ms
    Jan 14 01:06:32.800: INFO: Pod "azuredisk-volume-tester-n78bx": Phase="Pending", Reason="", readiness=false. Elapsed: 2.064184412s
    Jan 14 01:06:34.801: INFO: Pod "azuredisk-volume-tester-n78bx": Phase="Pending", Reason="", readiness=false. Elapsed: 4.065794151s
    Jan 14 01:06:36.800: INFO: Pod "azuredisk-volume-tester-n78bx": Phase="Pending", Reason="", readiness=false. Elapsed: 6.064328543s
    Jan 14 01:06:38.801: INFO: Pod "azuredisk-volume-tester-n78bx": Phase="Pending", Reason="", readiness=false. Elapsed: 8.065266048s
    Jan 14 01:06:40.801: INFO: Pod "azuredisk-volume-tester-n78bx": Phase="Pending", Reason="", readiness=false. Elapsed: 10.065412647s
... skipping 19 lines ...
    Jan 14 01:07:20.803: INFO: Pod "azuredisk-volume-tester-n78bx": Phase="Running", Reason="", readiness=true. Elapsed: 50.067508455s
    Jan 14 01:07:22.803: INFO: Pod "azuredisk-volume-tester-n78bx": Phase="Running", Reason="", readiness=false. Elapsed: 52.067543959s
    Jan 14 01:07:24.802: INFO: Pod "azuredisk-volume-tester-n78bx": Phase="Running", Reason="", readiness=false. Elapsed: 54.066748262s
    Jan 14 01:07:26.802: INFO: Pod "azuredisk-volume-tester-n78bx": Phase="Running", Reason="", readiness=false. Elapsed: 56.066924543s
    Jan 14 01:07:28.803: INFO: Pod "azuredisk-volume-tester-n78bx": Phase="Succeeded", Reason="", readiness=false. Elapsed: 58.067024313s
    [1mSTEP:[0m Saw pod success [38;5;243m01/14/23 01:07:28.803[0m
    Jan 14 01:07:28.803: INFO: Pod "azuredisk-volume-tester-n78bx" satisfied condition "Succeeded or Failed"
    Jan 14 01:07:28.836: INFO: deleting Pod "azuredisk-1594"/"azuredisk-volume-tester-n78bx"
    Jan 14 01:07:28.884: INFO: Pod azuredisk-volume-tester-n78bx has the following logs: 
    [1mSTEP:[0m Deleting pod azuredisk-volume-tester-n78bx in namespace azuredisk-1594 [38;5;243m01/14/23 01:07:28.884[0m
    [1mSTEP:[0m waiting for disk to detach from node [38;5;243m01/14/23 01:07:28.927[0m
    [1mSTEP:[0m Checking Prow test resource group [38;5;243m01/14/23 01:07:54.992[0m
    [1mSTEP:[0m Prow test resource group: capz-thn7c4 [38;5;243m01/14/23 01:07:54.994[0m
... skipping 16 lines ...
    [1mSTEP:[0m checking the PVC [38;5;243m01/14/23 01:08:15.528[0m
    [1mSTEP:[0m validating provisioned PV [38;5;243m01/14/23 01:08:15.561[0m
    [1mSTEP:[0m checking the PV [38;5;243m01/14/23 01:08:15.593[0m
    [1mSTEP:[0m setting up the pod [38;5;243m01/14/23 01:08:15.593[0m
    I0114 01:08:15.594035   42005 resource_setup.go:62] adding PV (pvc-6dc7122d-cd59-4713-8699-fcc00d6b3df7) to pod ()
    [1mSTEP:[0m deploying a pod with a volume restored from the snapshot [38;5;243m01/14/23 01:08:15.594[0m
    [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/14/23 01:08:15.633[0m
    Jan 14 01:08:15.634: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-ndrnp" in namespace "azuredisk-1594" to be "Succeeded or Failed"
    Jan 14 01:08:15.666: INFO: Pod "azuredisk-volume-tester-ndrnp": Phase="Pending", Reason="", readiness=false. Elapsed: 32.02418ms
    Jan 14 01:08:17.699: INFO: Pod "azuredisk-volume-tester-ndrnp": Phase="Pending", Reason="", readiness=false. Elapsed: 2.065730275s
    Jan 14 01:08:19.700: INFO: Pod "azuredisk-volume-tester-ndrnp": Phase="Pending", Reason="", readiness=false. Elapsed: 4.06629342s
    Jan 14 01:08:21.700: INFO: Pod "azuredisk-volume-tester-ndrnp": Phase="Pending", Reason="", readiness=false. Elapsed: 6.066786472s
    Jan 14 01:08:23.700: INFO: Pod "azuredisk-volume-tester-ndrnp": Phase="Pending", Reason="", readiness=false. Elapsed: 8.066273699s
    Jan 14 01:08:25.699: INFO: Pod "azuredisk-volume-tester-ndrnp": Phase="Pending", Reason="", readiness=false. Elapsed: 10.065630779s
... skipping 13 lines ...
    Jan 14 01:08:53.701: INFO: Pod "azuredisk-volume-tester-ndrnp": Phase="Running", Reason="", readiness=true. Elapsed: 38.067711299s
    Jan 14 01:08:55.702: INFO: Pod "azuredisk-volume-tester-ndrnp": Phase="Running", Reason="", readiness=true. Elapsed: 40.068787445s
    Jan 14 01:08:57.700: INFO: Pod "azuredisk-volume-tester-ndrnp": Phase="Running", Reason="", readiness=false. Elapsed: 42.066529178s
    Jan 14 01:08:59.702: INFO: Pod "azuredisk-volume-tester-ndrnp": Phase="Running", Reason="", readiness=false. Elapsed: 44.068174966s
    Jan 14 01:09:01.700: INFO: Pod "azuredisk-volume-tester-ndrnp": Phase="Succeeded", Reason="", readiness=false. Elapsed: 46.066739726s
    [1mSTEP:[0m Saw pod success [38;5;243m01/14/23 01:09:01.7[0m
    Jan 14 01:09:01.701: INFO: Pod "azuredisk-volume-tester-ndrnp" satisfied condition "Succeeded or Failed"
    Jan 14 01:09:01.701: INFO: deleting Pod "azuredisk-1594"/"azuredisk-volume-tester-ndrnp"
    Jan 14 01:09:01.760: INFO: Pod azuredisk-volume-tester-ndrnp has the following logs: hello world

    [1mSTEP:[0m Deleting pod azuredisk-volume-tester-ndrnp in namespace azuredisk-1594 [38;5;243m01/14/23 01:09:01.76[0m
    Jan 14 01:09:01.829: INFO: deleting PVC "azuredisk-1594"/"pvc-r567t"
    Jan 14 01:09:01.829: INFO: Deleting PersistentVolumeClaim "pvc-r567t"
... skipping 12 lines ...
    Jan 14 01:09:42.209: INFO: Claim "pvc-r567t" in namespace "azuredisk-1594" doesn't exist in the system
    Jan 14 01:09:42.209: INFO: deleting StorageClass azuredisk-1594-disk.csi.azure.com-dynamic-sc-wnm8s
    [1mSTEP:[0m deleting a VolumeSnapshot volume-snapshot-dqbhx [38;5;243m01/14/23 01:09:42.244[0m
    Jan 14 01:09:42.283: INFO: skip deleting VolumeSnapshotClass azuredisk-1594-disk.csi.azure.com-dynamic-sc-t4csv
    Jan 14 01:09:42.283: INFO: Deleting resource group azuredisk-csi-driver-test-e04751e7-93a7-11ed-abfb-260d6d03632e
    Jan 14 01:10:58.901: INFO: deleting Pod "azuredisk-1594"/"azuredisk-volume-tester-n78bx"
    Jan 14 01:10:58.933: INFO: Error getting logs for pod azuredisk-volume-tester-n78bx: the server could not find the requested resource (get pods azuredisk-volume-tester-n78bx)
    [1mSTEP:[0m Deleting pod azuredisk-volume-tester-n78bx in namespace azuredisk-1594 [38;5;243m01/14/23 01:10:58.933[0m
    Jan 14 01:10:58.966: INFO: deleting PVC "azuredisk-1594"/"pvc-zvjtn"
    Jan 14 01:10:58.966: INFO: Deleting PersistentVolumeClaim "pvc-zvjtn"
    [1mSTEP:[0m waiting for claim's PV "pvc-604c4585-6ac4-459f-869c-a9124bb6d3c0" to be deleted [38;5;243m01/14/23 01:10:59.004[0m
    Jan 14 01:10:59.004: INFO: Waiting up to 10m0s for PersistentVolume pvc-604c4585-6ac4-459f-869c-a9124bb6d3c0 to get deleted
    Jan 14 01:10:59.037: INFO: PersistentVolume pvc-604c4585-6ac4-459f-869c-a9124bb6d3c0 found and phase=Released (32.73236ms)
... skipping 28 lines ...
Jan 14 01:11:11.945: INFO: PersistentVolumeClaim pvc-nd58z found but phase is Pending instead of Bound.
Jan 14 01:11:13.979: INFO: PersistentVolumeClaim pvc-nd58z found and phase=Bound (4.101884399s)
[1mSTEP:[0m checking the PVC [38;5;243m01/14/23 01:11:13.979[0m
[1mSTEP:[0m validating provisioned PV [38;5;243m01/14/23 01:11:14.011[0m
[1mSTEP:[0m checking the PV [38;5;243m01/14/23 01:11:14.044[0m
[1mSTEP:[0m deploying the pod [38;5;243m01/14/23 01:11:14.044[0m
[1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/14/23 01:11:14.08[0m
Jan 14 01:11:14.080: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-mtf7r" in namespace "azuredisk-3957" to be "Succeeded or Failed"
Jan 14 01:11:14.113: INFO: Pod "azuredisk-volume-tester-mtf7r": Phase="Pending", Reason="", readiness=false. Elapsed: 32.262409ms
Jan 14 01:11:16.146: INFO: Pod "azuredisk-volume-tester-mtf7r": Phase="Pending", Reason="", readiness=false. Elapsed: 2.065470785s
Jan 14 01:11:18.145: INFO: Pod "azuredisk-volume-tester-mtf7r": Phase="Pending", Reason="", readiness=false. Elapsed: 4.064982123s
Jan 14 01:11:20.147: INFO: Pod "azuredisk-volume-tester-mtf7r": Phase="Pending", Reason="", readiness=false. Elapsed: 6.066206443s
Jan 14 01:11:22.147: INFO: Pod "azuredisk-volume-tester-mtf7r": Phase="Pending", Reason="", readiness=false. Elapsed: 8.066919066s
Jan 14 01:11:24.145: INFO: Pod "azuredisk-volume-tester-mtf7r": Phase="Pending", Reason="", readiness=false. Elapsed: 10.064907896s
... skipping 22 lines ...
Jan 14 01:12:10.148: INFO: Pod "azuredisk-volume-tester-mtf7r": Phase="Running", Reason="", readiness=true. Elapsed: 56.067074923s
Jan 14 01:12:12.149: INFO: Pod "azuredisk-volume-tester-mtf7r": Phase="Running", Reason="", readiness=false. Elapsed: 58.068570834s
Jan 14 01:12:14.149: INFO: Pod "azuredisk-volume-tester-mtf7r": Phase="Running", Reason="", readiness=false. Elapsed: 1m0.06875007s
Jan 14 01:12:16.149: INFO: Pod "azuredisk-volume-tester-mtf7r": Phase="Running", Reason="", readiness=false. Elapsed: 1m2.068499665s
Jan 14 01:12:18.148: INFO: Pod "azuredisk-volume-tester-mtf7r": Phase="Succeeded", Reason="", readiness=false. Elapsed: 1m4.067891134s
[1mSTEP:[0m Saw pod success [38;5;243m01/14/23 01:12:18.148[0m
Jan 14 01:12:18.149: INFO: Pod "azuredisk-volume-tester-mtf7r" satisfied condition "Succeeded or Failed"
Jan 14 01:12:18.181: INFO: deleting Pod "azuredisk-3957"/"azuredisk-volume-tester-mtf7r"
Jan 14 01:12:18.231: INFO: Pod azuredisk-volume-tester-mtf7r has the following logs: 
[1mSTEP:[0m Deleting pod azuredisk-volume-tester-mtf7r in namespace azuredisk-3957 [38;5;243m01/14/23 01:12:18.231[0m
[1mSTEP:[0m waiting for disk to detach from node [38;5;243m01/14/23 01:12:18.274[0m
[1mSTEP:[0m Checking Prow test resource group [38;5;243m01/14/23 01:12:40.34[0m
[1mSTEP:[0m Prow test resource group: capz-thn7c4 [38;5;243m01/14/23 01:12:40.341[0m
... skipping 39 lines ...
[1mSTEP:[0m validating provisioned PV [38;5;243m01/14/23 01:13:34.993[0m
[1mSTEP:[0m checking the PV [38;5;243m01/14/23 01:13:35.025[0m
[1mSTEP:[0m setting up the pod [38;5;243m01/14/23 01:13:35.025[0m
I0114 01:13:35.025996   42005 resource_setup.go:62] adding PV (pvc-ffdb31a0-a4c4-4951-9364-bedd15b4eebe) to pod ()
[1mSTEP:[0m Set pod anti-affinity to make sure two pods are scheduled on different nodes [38;5;243m01/14/23 01:13:35.026[0m
[1mSTEP:[0m deploying a pod with a volume restored from the snapshot [38;5;243m01/14/23 01:13:35.026[0m
[1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/14/23 01:13:35.06[0m
Jan 14 01:13:35.060: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-tn9cp" in namespace "azuredisk-3957" to be "Succeeded or Failed"
Jan 14 01:13:35.094: INFO: Pod "azuredisk-volume-tester-tn9cp": Phase="Pending", Reason="", readiness=false. Elapsed: 33.453541ms
Jan 14 01:13:37.130: INFO: Pod "azuredisk-volume-tester-tn9cp": Phase="Pending", Reason="", readiness=false. Elapsed: 2.06945976s
Jan 14 01:13:39.128: INFO: Pod "azuredisk-volume-tester-tn9cp": Phase="Pending", Reason="", readiness=false. Elapsed: 4.068225751s
Jan 14 01:13:41.128: INFO: Pod "azuredisk-volume-tester-tn9cp": Phase="Pending", Reason="", readiness=false. Elapsed: 6.067946714s
Jan 14 01:13:43.129: INFO: Pod "azuredisk-volume-tester-tn9cp": Phase="Pending", Reason="", readiness=false. Elapsed: 8.069077768s
Jan 14 01:13:45.129: INFO: Pod "azuredisk-volume-tester-tn9cp": Phase="Pending", Reason="", readiness=false. Elapsed: 10.068685776s
... skipping 112 lines ...
Jan 14 01:17:31.128: INFO: Pod "azuredisk-volume-tester-tn9cp": Phase="Running", Reason="", readiness=true. Elapsed: 3m56.068287332s
Jan 14 01:17:33.130: INFO: Pod "azuredisk-volume-tester-tn9cp": Phase="Running", Reason="", readiness=false. Elapsed: 3m58.069711293s
Jan 14 01:17:35.129: INFO: Pod "azuredisk-volume-tester-tn9cp": Phase="Running", Reason="", readiness=false. Elapsed: 4m0.068584865s
Jan 14 01:17:37.129: INFO: Pod "azuredisk-volume-tester-tn9cp": Phase="Running", Reason="", readiness=false. Elapsed: 4m2.068649699s
Jan 14 01:17:39.129: INFO: Pod "azuredisk-volume-tester-tn9cp": Phase="Succeeded", Reason="", readiness=false. Elapsed: 4m4.068823895s
[1mSTEP:[0m Saw pod success [38;5;243m01/14/23 01:17:39.129[0m
Jan 14 01:17:39.129: INFO: Pod "azuredisk-volume-tester-tn9cp" satisfied condition "Succeeded or Failed"
Jan 14 01:17:39.129: INFO: deleting Pod "azuredisk-3957"/"azuredisk-volume-tester-tn9cp"
Jan 14 01:17:39.190: INFO: Pod azuredisk-volume-tester-tn9cp has the following logs: hello world

[1mSTEP:[0m Deleting pod azuredisk-volume-tester-tn9cp in namespace azuredisk-3957 [38;5;243m01/14/23 01:17:39.19[0m
Jan 14 01:17:39.260: INFO: deleting PVC "azuredisk-3957"/"pvc-v9n6q"
Jan 14 01:17:39.260: INFO: Deleting PersistentVolumeClaim "pvc-v9n6q"
... skipping 15 lines ...
Jan 14 01:18:19.740: INFO: Pod azuredisk-volume-tester-864qh has the following logs: 
[1mSTEP:[0m Deleting pod azuredisk-volume-tester-864qh in namespace azuredisk-3957 [38;5;243m01/14/23 01:18:19.74[0m
[1mSTEP:[0m deleting a VolumeSnapshot volume-snapshot-5svlh [38;5;243m01/14/23 01:18:19.778[0m
Jan 14 01:18:19.814: INFO: skip deleting VolumeSnapshotClass azuredisk-3957-disk.csi.azure.com-dynamic-sc-xwh4g
Jan 14 01:18:19.814: INFO: Deleting resource group azuredisk-csi-driver-test-8a5bdacb-93a8-11ed-abfb-260d6d03632e
Jan 14 01:19:37.052: INFO: deleting Pod "azuredisk-3957"/"azuredisk-volume-tester-mtf7r"
Jan 14 01:19:37.085: INFO: Error getting logs for pod azuredisk-volume-tester-mtf7r: the server could not find the requested resource (get pods azuredisk-volume-tester-mtf7r)
[1mSTEP:[0m Deleting pod azuredisk-volume-tester-mtf7r in namespace azuredisk-3957 [38;5;243m01/14/23 01:19:37.085[0m
Jan 14 01:19:37.117: INFO: deleting PVC "azuredisk-3957"/"pvc-nd58z"
Jan 14 01:19:37.117: INFO: Deleting PersistentVolumeClaim "pvc-nd58z"
[1mSTEP:[0m waiting for claim's PV "pvc-c8664149-b16e-449c-960d-680e2012190e" to be deleted [38;5;243m01/14/23 01:19:37.154[0m
Jan 14 01:19:37.154: INFO: Waiting up to 10m0s for PersistentVolume pvc-c8664149-b16e-449c-960d-680e2012190e to get deleted
Jan 14 01:19:37.188: INFO: PersistentVolume pvc-c8664149-b16e-449c-960d-680e2012190e found and phase=Released (34.295376ms)
... skipping 29 lines ...
    Jan 14 01:11:11.945: INFO: PersistentVolumeClaim pvc-nd58z found but phase is Pending instead of Bound.
    Jan 14 01:11:13.979: INFO: PersistentVolumeClaim pvc-nd58z found and phase=Bound (4.101884399s)
    [1mSTEP:[0m checking the PVC [38;5;243m01/14/23 01:11:13.979[0m
    [1mSTEP:[0m validating provisioned PV [38;5;243m01/14/23 01:11:14.011[0m
    [1mSTEP:[0m checking the PV [38;5;243m01/14/23 01:11:14.044[0m
    [1mSTEP:[0m deploying the pod [38;5;243m01/14/23 01:11:14.044[0m
    [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/14/23 01:11:14.08[0m
    Jan 14 01:11:14.080: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-mtf7r" in namespace "azuredisk-3957" to be "Succeeded or Failed"
    Jan 14 01:11:14.113: INFO: Pod "azuredisk-volume-tester-mtf7r": Phase="Pending", Reason="", readiness=false. Elapsed: 32.262409ms
    Jan 14 01:11:16.146: INFO: Pod "azuredisk-volume-tester-mtf7r": Phase="Pending", Reason="", readiness=false. Elapsed: 2.065470785s
    Jan 14 01:11:18.145: INFO: Pod "azuredisk-volume-tester-mtf7r": Phase="Pending", Reason="", readiness=false. Elapsed: 4.064982123s
    Jan 14 01:11:20.147: INFO: Pod "azuredisk-volume-tester-mtf7r": Phase="Pending", Reason="", readiness=false. Elapsed: 6.066206443s
    Jan 14 01:11:22.147: INFO: Pod "azuredisk-volume-tester-mtf7r": Phase="Pending", Reason="", readiness=false. Elapsed: 8.066919066s
    Jan 14 01:11:24.145: INFO: Pod "azuredisk-volume-tester-mtf7r": Phase="Pending", Reason="", readiness=false. Elapsed: 10.064907896s
... skipping 22 lines ...
    Jan 14 01:12:10.148: INFO: Pod "azuredisk-volume-tester-mtf7r": Phase="Running", Reason="", readiness=true. Elapsed: 56.067074923s
    Jan 14 01:12:12.149: INFO: Pod "azuredisk-volume-tester-mtf7r": Phase="Running", Reason="", readiness=false. Elapsed: 58.068570834s
    Jan 14 01:12:14.149: INFO: Pod "azuredisk-volume-tester-mtf7r": Phase="Running", Reason="", readiness=false. Elapsed: 1m0.06875007s
    Jan 14 01:12:16.149: INFO: Pod "azuredisk-volume-tester-mtf7r": Phase="Running", Reason="", readiness=false. Elapsed: 1m2.068499665s
    Jan 14 01:12:18.148: INFO: Pod "azuredisk-volume-tester-mtf7r": Phase="Succeeded", Reason="", readiness=false. Elapsed: 1m4.067891134s
    [1mSTEP:[0m Saw pod success [38;5;243m01/14/23 01:12:18.148[0m
    Jan 14 01:12:18.149: INFO: Pod "azuredisk-volume-tester-mtf7r" satisfied condition "Succeeded or Failed"
    Jan 14 01:12:18.181: INFO: deleting Pod "azuredisk-3957"/"azuredisk-volume-tester-mtf7r"
    Jan 14 01:12:18.231: INFO: Pod azuredisk-volume-tester-mtf7r has the following logs: 
    [1mSTEP:[0m Deleting pod azuredisk-volume-tester-mtf7r in namespace azuredisk-3957 [38;5;243m01/14/23 01:12:18.231[0m
    [1mSTEP:[0m waiting for disk to detach from node [38;5;243m01/14/23 01:12:18.274[0m
    [1mSTEP:[0m Checking Prow test resource group [38;5;243m01/14/23 01:12:40.34[0m
    [1mSTEP:[0m Prow test resource group: capz-thn7c4 [38;5;243m01/14/23 01:12:40.341[0m
... skipping 39 lines ...
    [1mSTEP:[0m validating provisioned PV [38;5;243m01/14/23 01:13:34.993[0m
    [1mSTEP:[0m checking the PV [38;5;243m01/14/23 01:13:35.025[0m
    [1mSTEP:[0m setting up the pod [38;5;243m01/14/23 01:13:35.025[0m
    I0114 01:13:35.025996   42005 resource_setup.go:62] adding PV (pvc-ffdb31a0-a4c4-4951-9364-bedd15b4eebe) to pod ()
    [1mSTEP:[0m Set pod anti-affinity to make sure two pods are scheduled on different nodes [38;5;243m01/14/23 01:13:35.026[0m
    [1mSTEP:[0m deploying a pod with a volume restored from the snapshot [38;5;243m01/14/23 01:13:35.026[0m
    [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/14/23 01:13:35.06[0m
    Jan 14 01:13:35.060: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-tn9cp" in namespace "azuredisk-3957" to be "Succeeded or Failed"
    Jan 14 01:13:35.094: INFO: Pod "azuredisk-volume-tester-tn9cp": Phase="Pending", Reason="", readiness=false. Elapsed: 33.453541ms
    Jan 14 01:13:37.130: INFO: Pod "azuredisk-volume-tester-tn9cp": Phase="Pending", Reason="", readiness=false. Elapsed: 2.06945976s
    Jan 14 01:13:39.128: INFO: Pod "azuredisk-volume-tester-tn9cp": Phase="Pending", Reason="", readiness=false. Elapsed: 4.068225751s
    Jan 14 01:13:41.128: INFO: Pod "azuredisk-volume-tester-tn9cp": Phase="Pending", Reason="", readiness=false. Elapsed: 6.067946714s
    Jan 14 01:13:43.129: INFO: Pod "azuredisk-volume-tester-tn9cp": Phase="Pending", Reason="", readiness=false. Elapsed: 8.069077768s
    Jan 14 01:13:45.129: INFO: Pod "azuredisk-volume-tester-tn9cp": Phase="Pending", Reason="", readiness=false. Elapsed: 10.068685776s
... skipping 112 lines ...
    Jan 14 01:17:31.128: INFO: Pod "azuredisk-volume-tester-tn9cp": Phase="Running", Reason="", readiness=true. Elapsed: 3m56.068287332s
    Jan 14 01:17:33.130: INFO: Pod "azuredisk-volume-tester-tn9cp": Phase="Running", Reason="", readiness=false. Elapsed: 3m58.069711293s
    Jan 14 01:17:35.129: INFO: Pod "azuredisk-volume-tester-tn9cp": Phase="Running", Reason="", readiness=false. Elapsed: 4m0.068584865s
    Jan 14 01:17:37.129: INFO: Pod "azuredisk-volume-tester-tn9cp": Phase="Running", Reason="", readiness=false. Elapsed: 4m2.068649699s
    Jan 14 01:17:39.129: INFO: Pod "azuredisk-volume-tester-tn9cp": Phase="Succeeded", Reason="", readiness=false. Elapsed: 4m4.068823895s
    [1mSTEP:[0m Saw pod success [38;5;243m01/14/23 01:17:39.129[0m
    Jan 14 01:17:39.129: INFO: Pod "azuredisk-volume-tester-tn9cp" satisfied condition "Succeeded or Failed"
    Jan 14 01:17:39.129: INFO: deleting Pod "azuredisk-3957"/"azuredisk-volume-tester-tn9cp"
    Jan 14 01:17:39.190: INFO: Pod azuredisk-volume-tester-tn9cp has the following logs: hello world

    [1mSTEP:[0m Deleting pod azuredisk-volume-tester-tn9cp in namespace azuredisk-3957 [38;5;243m01/14/23 01:17:39.19[0m
    Jan 14 01:17:39.260: INFO: deleting PVC "azuredisk-3957"/"pvc-v9n6q"
    Jan 14 01:17:39.260: INFO: Deleting PersistentVolumeClaim "pvc-v9n6q"
... skipping 15 lines ...
    Jan 14 01:18:19.740: INFO: Pod azuredisk-volume-tester-864qh has the following logs: 
    [1mSTEP:[0m Deleting pod azuredisk-volume-tester-864qh in namespace azuredisk-3957 [38;5;243m01/14/23 01:18:19.74[0m
    [1mSTEP:[0m deleting a VolumeSnapshot volume-snapshot-5svlh [38;5;243m01/14/23 01:18:19.778[0m
    Jan 14 01:18:19.814: INFO: skip deleting VolumeSnapshotClass azuredisk-3957-disk.csi.azure.com-dynamic-sc-xwh4g
    Jan 14 01:18:19.814: INFO: Deleting resource group azuredisk-csi-driver-test-8a5bdacb-93a8-11ed-abfb-260d6d03632e
    Jan 14 01:19:37.052: INFO: deleting Pod "azuredisk-3957"/"azuredisk-volume-tester-mtf7r"
    Jan 14 01:19:37.085: INFO: Error getting logs for pod azuredisk-volume-tester-mtf7r: the server could not find the requested resource (get pods azuredisk-volume-tester-mtf7r)
    [1mSTEP:[0m Deleting pod azuredisk-volume-tester-mtf7r in namespace azuredisk-3957 [38;5;243m01/14/23 01:19:37.085[0m
    Jan 14 01:19:37.117: INFO: deleting PVC "azuredisk-3957"/"pvc-nd58z"
    Jan 14 01:19:37.117: INFO: Deleting PersistentVolumeClaim "pvc-nd58z"
    [1mSTEP:[0m waiting for claim's PV "pvc-c8664149-b16e-449c-960d-680e2012190e" to be deleted [38;5;243m01/14/23 01:19:37.154[0m
    Jan 14 01:19:37.154: INFO: Waiting up to 10m0s for PersistentVolume pvc-c8664149-b16e-449c-960d-680e2012190e to get deleted
    Jan 14 01:19:37.188: INFO: PersistentVolume pvc-c8664149-b16e-449c-960d-680e2012190e found and phase=Released (34.295376ms)
... skipping 55 lines ...
Jan 14 01:20:00.570: INFO: PersistentVolumeClaim pvc-fqf2z found and phase=Bound (4.099998083s)
[1mSTEP:[0m checking the PVC [38;5;243m01/14/23 01:20:00.57[0m
[1mSTEP:[0m validating provisioned PV [38;5;243m01/14/23 01:20:00.603[0m
[1mSTEP:[0m checking the PV [38;5;243m01/14/23 01:20:00.635[0m
I0114 01:20:00.635677   42005 resource_setup.go:89] adding PV (pvc-5165bb1e-ec89-4ead-bdb9-3dda7588cdf2) to pod ()
[1mSTEP:[0m deploying the pod [38;5;243m01/14/23 01:20:00.635[0m
[1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/14/23 01:20:00.673[0m
Jan 14 01:20:00.673: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-l9tt6" in namespace "azuredisk-4098" to be "Succeeded or Failed"
Jan 14 01:20:00.705: INFO: Pod "azuredisk-volume-tester-l9tt6": Phase="Pending", Reason="", readiness=false. Elapsed: 31.735035ms
Jan 14 01:20:02.741: INFO: Pod "azuredisk-volume-tester-l9tt6": Phase="Pending", Reason="", readiness=false. Elapsed: 2.067596755s
Jan 14 01:20:04.740: INFO: Pod "azuredisk-volume-tester-l9tt6": Phase="Pending", Reason="", readiness=false. Elapsed: 4.066952492s
Jan 14 01:20:06.742: INFO: Pod "azuredisk-volume-tester-l9tt6": Phase="Pending", Reason="", readiness=false. Elapsed: 6.068843458s
Jan 14 01:20:08.740: INFO: Pod "azuredisk-volume-tester-l9tt6": Phase="Pending", Reason="", readiness=false. Elapsed: 8.067427354s
Jan 14 01:20:10.742: INFO: Pod "azuredisk-volume-tester-l9tt6": Phase="Pending", Reason="", readiness=false. Elapsed: 10.068920714s
... skipping 24 lines ...
Jan 14 01:21:00.741: INFO: Pod "azuredisk-volume-tester-l9tt6": Phase="Running", Reason="", readiness=true. Elapsed: 1m0.067845973s
Jan 14 01:21:02.740: INFO: Pod "azuredisk-volume-tester-l9tt6": Phase="Running", Reason="", readiness=false. Elapsed: 1m2.067294873s
Jan 14 01:21:04.743: INFO: Pod "azuredisk-volume-tester-l9tt6": Phase="Running", Reason="", readiness=false. Elapsed: 1m4.069651057s
Jan 14 01:21:06.741: INFO: Pod "azuredisk-volume-tester-l9tt6": Phase="Running", Reason="", readiness=false. Elapsed: 1m6.067751248s
Jan 14 01:21:08.740: INFO: Pod "azuredisk-volume-tester-l9tt6": Phase="Succeeded", Reason="", readiness=false. Elapsed: 1m8.067413061s
[1mSTEP:[0m Saw pod success [38;5;243m01/14/23 01:21:08.74[0m
Jan 14 01:21:08.741: INFO: Pod "azuredisk-volume-tester-l9tt6" satisfied condition "Succeeded or Failed"
Jan 14 01:21:08.741: INFO: deleting Pod "azuredisk-4098"/"azuredisk-volume-tester-l9tt6"
Jan 14 01:21:08.804: INFO: Pod azuredisk-volume-tester-l9tt6 has the following logs: hello world

[1mSTEP:[0m Deleting pod azuredisk-volume-tester-l9tt6 in namespace azuredisk-4098 [38;5;243m01/14/23 01:21:08.804[0m
Jan 14 01:21:08.852: INFO: deleting PVC "azuredisk-4098"/"pvc-fqf2z"
Jan 14 01:21:08.852: INFO: Deleting PersistentVolumeClaim "pvc-fqf2z"
... skipping 85 lines ...
    Jan 14 01:20:00.570: INFO: PersistentVolumeClaim pvc-fqf2z found and phase=Bound (4.099998083s)
    [1mSTEP:[0m checking the PVC [38;5;243m01/14/23 01:20:00.57[0m
    [1mSTEP:[0m validating provisioned PV [38;5;243m01/14/23 01:20:00.603[0m
    [1mSTEP:[0m checking the PV [38;5;243m01/14/23 01:20:00.635[0m
    I0114 01:20:00.635677   42005 resource_setup.go:89] adding PV (pvc-5165bb1e-ec89-4ead-bdb9-3dda7588cdf2) to pod ()
    [1mSTEP:[0m deploying the pod [38;5;243m01/14/23 01:20:00.635[0m
    [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/14/23 01:20:00.673[0m
    Jan 14 01:20:00.673: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-l9tt6" in namespace "azuredisk-4098" to be "Succeeded or Failed"
    Jan 14 01:20:00.705: INFO: Pod "azuredisk-volume-tester-l9tt6": Phase="Pending", Reason="", readiness=false. Elapsed: 31.735035ms
    Jan 14 01:20:02.741: INFO: Pod "azuredisk-volume-tester-l9tt6": Phase="Pending", Reason="", readiness=false. Elapsed: 2.067596755s
    Jan 14 01:20:04.740: INFO: Pod "azuredisk-volume-tester-l9tt6": Phase="Pending", Reason="", readiness=false. Elapsed: 4.066952492s
    Jan 14 01:20:06.742: INFO: Pod "azuredisk-volume-tester-l9tt6": Phase="Pending", Reason="", readiness=false. Elapsed: 6.068843458s
    Jan 14 01:20:08.740: INFO: Pod "azuredisk-volume-tester-l9tt6": Phase="Pending", Reason="", readiness=false. Elapsed: 8.067427354s
    Jan 14 01:20:10.742: INFO: Pod "azuredisk-volume-tester-l9tt6": Phase="Pending", Reason="", readiness=false. Elapsed: 10.068920714s
... skipping 24 lines ...
    Jan 14 01:21:00.741: INFO: Pod "azuredisk-volume-tester-l9tt6": Phase="Running", Reason="", readiness=true. Elapsed: 1m0.067845973s
    Jan 14 01:21:02.740: INFO: Pod "azuredisk-volume-tester-l9tt6": Phase="Running", Reason="", readiness=false. Elapsed: 1m2.067294873s
    Jan 14 01:21:04.743: INFO: Pod "azuredisk-volume-tester-l9tt6": Phase="Running", Reason="", readiness=false. Elapsed: 1m4.069651057s
    Jan 14 01:21:06.741: INFO: Pod "azuredisk-volume-tester-l9tt6": Phase="Running", Reason="", readiness=false. Elapsed: 1m6.067751248s
    Jan 14 01:21:08.740: INFO: Pod "azuredisk-volume-tester-l9tt6": Phase="Succeeded", Reason="", readiness=false. Elapsed: 1m8.067413061s
    [1mSTEP:[0m Saw pod success [38;5;243m01/14/23 01:21:08.74[0m
    Jan 14 01:21:08.741: INFO: Pod "azuredisk-volume-tester-l9tt6" satisfied condition "Succeeded or Failed"
    Jan 14 01:21:08.741: INFO: deleting Pod "azuredisk-4098"/"azuredisk-volume-tester-l9tt6"
    Jan 14 01:21:08.804: INFO: Pod azuredisk-volume-tester-l9tt6 has the following logs: hello world

    [1mSTEP:[0m Deleting pod azuredisk-volume-tester-l9tt6 in namespace azuredisk-4098 [38;5;243m01/14/23 01:21:08.804[0m
    Jan 14 01:21:08.852: INFO: deleting PVC "azuredisk-4098"/"pvc-fqf2z"
    Jan 14 01:21:08.852: INFO: Deleting PersistentVolumeClaim "pvc-fqf2z"
... skipping 864 lines ...
I0114 01:38:56.488097   42005 dynamically_provisioned_pod_delete.go:101] volumes: [{Name:test-volume-1 VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:&PersistentVolumeClaimVolumeSource{ClaimName:pvc-j29qz,ReadOnly:false,} RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}]
[1mSTEP:[0m begin to delete the pod [38;5;243m01/14/23 01:38:56.488[0m
Jan 14 01:38:56.488: INFO: deleting Pod "azuredisk-1219"/"azuredisk-volume-tester-gvl7s"
Jan 14 01:38:56.540: INFO: Pod azuredisk-volume-tester-gvl7s has the following logs: 
[1mSTEP:[0m Deleting pod azuredisk-volume-tester-gvl7s in namespace azuredisk-1219 [38;5;243m01/14/23 01:38:56.54[0m
Jan 14 01:39:24.647: INFO: deleting Pod "azuredisk-1219"/"azuredisk-volume-tester-gvl7s"
Jan 14 01:39:24.679: INFO: Error getting logs for pod azuredisk-volume-tester-gvl7s: the server could not find the requested resource (get pods azuredisk-volume-tester-gvl7s)
[1mSTEP:[0m Deleting pod azuredisk-volume-tester-gvl7s in namespace azuredisk-1219 [38;5;243m01/14/23 01:39:24.679[0m
Jan 14 01:39:24.712: INFO: deleting PVC "azuredisk-1219"/"pvc-j29qz"
Jan 14 01:39:24.712: INFO: Deleting PersistentVolumeClaim "pvc-j29qz"
[1mSTEP:[0m waiting for claim's PV "pvc-b0a4977e-2912-42b3-8180-95044202fa70" to be deleted [38;5;243m01/14/23 01:39:24.75[0m
Jan 14 01:39:24.750: INFO: Waiting up to 10m0s for PersistentVolume pvc-b0a4977e-2912-42b3-8180-95044202fa70 to get deleted
Jan 14 01:39:24.795: INFO: PersistentVolume pvc-b0a4977e-2912-42b3-8180-95044202fa70 found and phase=Released (44.747415ms)
... skipping 67 lines ...
    I0114 01:38:56.488097   42005 dynamically_provisioned_pod_delete.go:101] volumes: [{Name:test-volume-1 VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:&PersistentVolumeClaimVolumeSource{ClaimName:pvc-j29qz,ReadOnly:false,} RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}]
    [1mSTEP:[0m begin to delete the pod [38;5;243m01/14/23 01:38:56.488[0m
    Jan 14 01:38:56.488: INFO: deleting Pod "azuredisk-1219"/"azuredisk-volume-tester-gvl7s"
    Jan 14 01:38:56.540: INFO: Pod azuredisk-volume-tester-gvl7s has the following logs: 
    [1mSTEP:[0m Deleting pod azuredisk-volume-tester-gvl7s in namespace azuredisk-1219 [38;5;243m01/14/23 01:38:56.54[0m
    Jan 14 01:39:24.647: INFO: deleting Pod "azuredisk-1219"/"azuredisk-volume-tester-gvl7s"
    Jan 14 01:39:24.679: INFO: Error getting logs for pod azuredisk-volume-tester-gvl7s: the server could not find the requested resource (get pods azuredisk-volume-tester-gvl7s)
    [1mSTEP:[0m Deleting pod azuredisk-volume-tester-gvl7s in namespace azuredisk-1219 [38;5;243m01/14/23 01:39:24.679[0m
    Jan 14 01:39:24.712: INFO: deleting PVC "azuredisk-1219"/"pvc-j29qz"
    Jan 14 01:39:24.712: INFO: Deleting PersistentVolumeClaim "pvc-j29qz"
    [1mSTEP:[0m waiting for claim's PV "pvc-b0a4977e-2912-42b3-8180-95044202fa70" to be deleted [38;5;243m01/14/23 01:39:24.75[0m
    Jan 14 01:39:24.750: INFO: Waiting up to 10m0s for PersistentVolume pvc-b0a4977e-2912-42b3-8180-95044202fa70 to get deleted
    Jan 14 01:39:24.795: INFO: PersistentVolume pvc-b0a4977e-2912-42b3-8180-95044202fa70 found and phase=Released (44.747415ms)
... skipping 66 lines ...
I0114 01:40:29.007574   42005 dynamically_provisioned_pod_delete.go:101] volumes: [{Name:test-volume-1 VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:&PersistentVolumeClaimVolumeSource{ClaimName:pvc-9t9dh,ReadOnly:false,} RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}]
[1mSTEP:[0m begin to delete the pod [38;5;243m01/14/23 01:40:29.007[0m
Jan 14 01:40:29.007: INFO: deleting Pod "azuredisk-6068"/"azuredisk-volume-tester-wjdvv"
Jan 14 01:40:29.063: INFO: Pod azuredisk-volume-tester-wjdvv has the following logs: 
[1mSTEP:[0m Deleting pod azuredisk-volume-tester-wjdvv in namespace azuredisk-6068 [38;5;243m01/14/23 01:40:29.063[0m
Jan 14 01:40:47.177: INFO: deleting Pod "azuredisk-6068"/"azuredisk-volume-tester-wjdvv"
Jan 14 01:40:47.210: INFO: Error getting logs for pod azuredisk-volume-tester-wjdvv: the server could not find the requested resource (get pods azuredisk-volume-tester-wjdvv)
[1mSTEP:[0m Deleting pod azuredisk-volume-tester-wjdvv in namespace azuredisk-6068 [38;5;243m01/14/23 01:40:47.21[0m
Jan 14 01:40:47.242: INFO: deleting PVC "azuredisk-6068"/"pvc-9t9dh"
Jan 14 01:40:47.242: INFO: Deleting PersistentVolumeClaim "pvc-9t9dh"
[1mSTEP:[0m waiting for claim's PV "pvc-ca224d18-e297-4ff6-87d1-108549471be4" to be deleted [38;5;243m01/14/23 01:40:47.279[0m
Jan 14 01:40:47.279: INFO: Waiting up to 10m0s for PersistentVolume pvc-ca224d18-e297-4ff6-87d1-108549471be4 to get deleted
Jan 14 01:40:47.312: INFO: PersistentVolume pvc-ca224d18-e297-4ff6-87d1-108549471be4 found and phase=Released (33.429286ms)
... skipping 70 lines ...
    I0114 01:40:29.007574   42005 dynamically_provisioned_pod_delete.go:101] volumes: [{Name:test-volume-1 VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:&PersistentVolumeClaimVolumeSource{ClaimName:pvc-9t9dh,ReadOnly:false,} RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}]
    [1mSTEP:[0m begin to delete the pod [38;5;243m01/14/23 01:40:29.007[0m
    Jan 14 01:40:29.007: INFO: deleting Pod "azuredisk-6068"/"azuredisk-volume-tester-wjdvv"
    Jan 14 01:40:29.063: INFO: Pod azuredisk-volume-tester-wjdvv has the following logs: 
    [1mSTEP:[0m Deleting pod azuredisk-volume-tester-wjdvv in namespace azuredisk-6068 [38;5;243m01/14/23 01:40:29.063[0m
    Jan 14 01:40:47.177: INFO: deleting Pod "azuredisk-6068"/"azuredisk-volume-tester-wjdvv"
    Jan 14 01:40:47.210: INFO: Error getting logs for pod azuredisk-volume-tester-wjdvv: the server could not find the requested resource (get pods azuredisk-volume-tester-wjdvv)
    [1mSTEP:[0m Deleting pod azuredisk-volume-tester-wjdvv in namespace azuredisk-6068 [38;5;243m01/14/23 01:40:47.21[0m
    Jan 14 01:40:47.242: INFO: deleting PVC "azuredisk-6068"/"pvc-9t9dh"
    Jan 14 01:40:47.242: INFO: Deleting PersistentVolumeClaim "pvc-9t9dh"
    [1mSTEP:[0m waiting for claim's PV "pvc-ca224d18-e297-4ff6-87d1-108549471be4" to be deleted [38;5;243m01/14/23 01:40:47.279[0m
    Jan 14 01:40:47.279: INFO: Waiting up to 10m0s for PersistentVolume pvc-ca224d18-e297-4ff6-87d1-108549471be4 to get deleted
    Jan 14 01:40:47.312: INFO: PersistentVolume pvc-ca224d18-e297-4ff6-87d1-108549471be4 found and phase=Released (33.429286ms)
... skipping 56 lines ...
I0114 01:42:07.396364   42005 statefulset.go:175] 1/1 replicas in the StatefulSet are ready
[1mSTEP:[0m checking that the pod is running [38;5;243m01/14/23 01:42:07.463[0m
I0114 01:42:07.497447   42005 statefulset.go:175] 1/1 replicas in the StatefulSet are ready
[1mSTEP:[0m check pod exec [38;5;243m01/14/23 01:42:07.497[0m
Jan 14 01:42:07.498: INFO: Running '/usr/local/bin/kubectl --kubeconfig=/home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure/kubeconfig --namespace=azuredisk-7207 exec azuredisk-volume-tester-czvvb-0 -- cmd /c type C:\mnt\test-1\data.txt'
Jan 14 01:42:08.249: INFO: rc: 1
Jan 14 01:42:08.249: INFO: Error waiting for output "hello world\r\n" in pod "azuredisk-volume-tester-czvvb-0": error running /usr/local/bin/kubectl --kubeconfig=/home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure/kubeconfig --namespace=azuredisk-7207 exec azuredisk-volume-tester-czvvb-0 -- cmd /c type C:\mnt\test-1\data.txt:
Command stdout:

stderr:
The system cannot find the file specified.
command terminated with exit code 1

error:
exit status 1.
Jan 14 01:42:10.249: INFO: Running '/usr/local/bin/kubectl --kubeconfig=/home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure/kubeconfig --namespace=azuredisk-7207 exec azuredisk-volume-tester-czvvb-0 -- cmd /c type C:\mnt\test-1\data.txt'
Jan 14 01:42:10.824: INFO: rc: 1
Jan 14 01:42:10.824: INFO: Error waiting for output "hello world\r\n" in pod "azuredisk-volume-tester-czvvb-0": error running /usr/local/bin/kubectl --kubeconfig=/home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure/kubeconfig --namespace=azuredisk-7207 exec azuredisk-volume-tester-czvvb-0 -- cmd /c type C:\mnt\test-1\data.txt:
Command stdout:

stderr:
The system cannot find the file specified.
command terminated with exit code 1

error:
exit status 1.
Jan 14 01:42:12.250: INFO: Running '/usr/local/bin/kubectl --kubeconfig=/home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure/kubeconfig --namespace=azuredisk-7207 exec azuredisk-volume-tester-czvvb-0 -- cmd /c type C:\mnt\test-1\data.txt'
Jan 14 01:42:12.842: INFO: stderr: ""
Jan 14 01:42:12.842: INFO: stdout: "hello world\r\n"
[1mSTEP:[0m deleting the pod for statefulset [38;5;243m01/14/23 01:42:12.842[0m
Jan 14 01:42:12.842: INFO: Deleting pod "azuredisk-volume-tester-czvvb-0" in namespace "azuredisk-7207"
... skipping 68 lines ...
    I0114 01:42:07.396364   42005 statefulset.go:175] 1/1 replicas in the StatefulSet are ready
    [1mSTEP:[0m checking that the pod is running [38;5;243m01/14/23 01:42:07.463[0m
    I0114 01:42:07.497447   42005 statefulset.go:175] 1/1 replicas in the StatefulSet are ready
    [1mSTEP:[0m check pod exec [38;5;243m01/14/23 01:42:07.497[0m
    Jan 14 01:42:07.498: INFO: Running '/usr/local/bin/kubectl --kubeconfig=/home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure/kubeconfig --namespace=azuredisk-7207 exec azuredisk-volume-tester-czvvb-0 -- cmd /c type C:\mnt\test-1\data.txt'
    Jan 14 01:42:08.249: INFO: rc: 1
    Jan 14 01:42:08.249: INFO: Error waiting for output "hello world\r\n" in pod "azuredisk-volume-tester-czvvb-0": error running /usr/local/bin/kubectl --kubeconfig=/home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure/kubeconfig --namespace=azuredisk-7207 exec azuredisk-volume-tester-czvvb-0 -- cmd /c type C:\mnt\test-1\data.txt:
    Command stdout:

    stderr:
    The system cannot find the file specified.
    command terminated with exit code 1

    error:
    exit status 1.
    Jan 14 01:42:10.249: INFO: Running '/usr/local/bin/kubectl --kubeconfig=/home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure/kubeconfig --namespace=azuredisk-7207 exec azuredisk-volume-tester-czvvb-0 -- cmd /c type C:\mnt\test-1\data.txt'
    Jan 14 01:42:10.824: INFO: rc: 1
    Jan 14 01:42:10.824: INFO: Error waiting for output "hello world\r\n" in pod "azuredisk-volume-tester-czvvb-0": error running /usr/local/bin/kubectl --kubeconfig=/home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure/kubeconfig --namespace=azuredisk-7207 exec azuredisk-volume-tester-czvvb-0 -- cmd /c type C:\mnt\test-1\data.txt:
    Command stdout:

    stderr:
    The system cannot find the file specified.
    command terminated with exit code 1

    error:
    exit status 1.
    Jan 14 01:42:12.250: INFO: Running '/usr/local/bin/kubectl --kubeconfig=/home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure/kubeconfig --namespace=azuredisk-7207 exec azuredisk-volume-tester-czvvb-0 -- cmd /c type C:\mnt\test-1\data.txt'
    Jan 14 01:42:12.842: INFO: stderr: ""
    Jan 14 01:42:12.842: INFO: stdout: "hello world\r\n"
    [1mSTEP:[0m deleting the pod for statefulset [38;5;243m01/14/23 01:42:12.842[0m
    Jan 14 01:42:12.842: INFO: Deleting pod "azuredisk-volume-tester-czvvb-0" in namespace "azuredisk-7207"
... skipping 800 lines ...
I0114 01:59:16.228113   42005 dynamically_provisioned_pod_affinity.go:145] found 1 AzVolumeAttachments for volume (pvc-76f33908-edbe-4dd1-9cae-af5a16ffd9e6)
I0114 01:59:18.228039   42005 dynamically_provisioned_pod_affinity.go:145] found 1 AzVolumeAttachments for volume (pvc-76f33908-edbe-4dd1-9cae-af5a16ffd9e6)
I0114 01:59:20.228330   42005 dynamically_provisioned_pod_affinity.go:145] found 1 AzVolumeAttachments for volume (pvc-76f33908-edbe-4dd1-9cae-af5a16ffd9e6)
I0114 01:59:22.228866   42005 dynamically_provisioned_pod_affinity.go:145] found 1 AzVolumeAttachments for volume (pvc-76f33908-edbe-4dd1-9cae-af5a16ffd9e6)
I0114 01:59:24.229606   42005 dynamically_provisioned_pod_affinity.go:145] found 1 AzVolumeAttachments for volume (pvc-76f33908-edbe-4dd1-9cae-af5a16ffd9e6)
I0114 01:59:24.266110   42005 dynamically_provisioned_pod_affinity.go:145] found 1 AzVolumeAttachments for volume (pvc-76f33908-edbe-4dd1-9cae-af5a16ffd9e6)
Jan 14 01:59:24.266: INFO: Unexpected error: 
    <*errors.errorString | 0xc0003e2bf0>: {
        s: "timed out waiting for the condition",
    }
Jan 14 01:59:24.266: FAIL: timed out waiting for the condition

Full Stack Trace
sigs.k8s.io/azuredisk-csi-driver/test/e2e/testsuites.(*PodAffinity).Run(0xc000acedc0, {0x26fb3c0, 0xc0006b0680}, 0xb?, {0x220ec57, 0x20})
	/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/testsuites/dynamically_provisioned_pod_affinity.go:168 +0x1e9
sigs.k8s.io/azuredisk-csi-driver/test/e2e.(*dynamicProvisioningTestSuite).defineTests.func31()
	/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning_test.go:1347 +0xcca
... skipping 39 lines ...
Jan 14 02:00:55.312: INFO: Waiting up to 5m0s for PersistentVolumeClaim pvc-lkktc to be removed
Jan 14 02:00:55.345: INFO: Claim "pvc-lkktc" in namespace "azuredisk-5211" doesn't exist in the system
Jan 14 02:00:55.345: INFO: deleting StorageClass azuredisk-5211-disk.csi.azure.com-dynamic-sc-wpfrl
[1mSTEP:[0m dump namespace information after failure [38;5;243m01/14/23 02:00:55.382[0m
[1mSTEP:[0m Destroying namespace "azuredisk-5211" for this suite. [38;5;243m01/14/23 02:00:55.383[0m
[38;5;243m------------------------------[0m
[38;5;9m• [FAILED] [820.758 seconds][0m
Dynamic Provisioning
[38;5;243m/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning_test.go:41[0m
  [single-az]
  [38;5;243m/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning_test.go:45[0m
    [38;5;9m[1m[It] Should create replicas on node with matching pod affinity[0m
    [38;5;243m/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning_test.go:1275[0m
... skipping 405 lines ...
    I0114 01:59:16.228113   42005 dynamically_provisioned_pod_affinity.go:145] found 1 AzVolumeAttachments for volume (pvc-76f33908-edbe-4dd1-9cae-af5a16ffd9e6)
    I0114 01:59:18.228039   42005 dynamically_provisioned_pod_affinity.go:145] found 1 AzVolumeAttachments for volume (pvc-76f33908-edbe-4dd1-9cae-af5a16ffd9e6)
    I0114 01:59:20.228330   42005 dynamically_provisioned_pod_affinity.go:145] found 1 AzVolumeAttachments for volume (pvc-76f33908-edbe-4dd1-9cae-af5a16ffd9e6)
    I0114 01:59:22.228866   42005 dynamically_provisioned_pod_affinity.go:145] found 1 AzVolumeAttachments for volume (pvc-76f33908-edbe-4dd1-9cae-af5a16ffd9e6)
    I0114 01:59:24.229606   42005 dynamically_provisioned_pod_affinity.go:145] found 1 AzVolumeAttachments for volume (pvc-76f33908-edbe-4dd1-9cae-af5a16ffd9e6)
    I0114 01:59:24.266110   42005 dynamically_provisioned_pod_affinity.go:145] found 1 AzVolumeAttachments for volume (pvc-76f33908-edbe-4dd1-9cae-af5a16ffd9e6)
    Jan 14 01:59:24.266: INFO: Unexpected error: 
        <*errors.errorString | 0xc0003e2bf0>: {
            s: "timed out waiting for the condition",
        }
    Jan 14 01:59:24.266: FAIL: timed out waiting for the condition

    Full Stack Trace
    sigs.k8s.io/azuredisk-csi-driver/test/e2e/testsuites.(*PodAffinity).Run(0xc000acedc0, {0x26fb3c0, 0xc0006b0680}, 0xb?, {0x220ec57, 0x20})
    	/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/testsuites/dynamically_provisioned_pod_affinity.go:168 +0x1e9
    sigs.k8s.io/azuredisk-csi-driver/test/e2e.(*dynamicProvisioningTestSuite).defineTests.func31()
    	/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning_test.go:1347 +0xcca
... skipping 48 lines ...

  [1mThere were additional failures detected after the initial failure:[0m
    [38;5;13m[PANICKED][0m
    [38;5;13mTest Panicked[0m
    [38;5;13mIn [1m[DeferCleanup (Each)][0m[38;5;13m at: [1m/usr/local/go/src/runtime/panic.go:260[0m

    [38;5;13mruntime error: invalid memory address or nil pointer dereference[0m

    [38;5;13mFull Stack Trace[0m
      k8s.io/kubernetes/test/e2e/framework.(*Framework).dumpNamespaceInfo.func1()
      	/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:274 +0x5c
      k8s.io/kubernetes/test/e2e/framework.(*Framework).dumpNamespaceInfo(0xc0002b03c0)
      	/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:271 +0x179
... skipping 383 lines ...
Jan 14 02:04:59.792: INFO: deployment status: v1.DeploymentStatus{ObservedGeneration:1, Replicas:1, UpdatedReplicas:1, ReadyReplicas:0, AvailableReplicas:0, UnavailableReplicas:1, Conditions:[]v1.DeploymentCondition{v1.DeploymentCondition{Type:"Available", Status:"False", LastUpdateTime:time.Date(2023, time.January, 14, 2, 4, 15, 0, time.Local), LastTransitionTime:time.Date(2023, time.January, 14, 2, 4, 15, 0, time.Local), Reason:"MinimumReplicasUnavailable", Message:"Deployment does not have minimum availability."}, v1.DeploymentCondition{Type:"Progressing", Status:"True", LastUpdateTime:time.Date(2023, time.January, 14, 2, 4, 15, 0, time.Local), LastTransitionTime:time.Date(2023, time.January, 14, 2, 4, 15, 0, time.Local), Reason:"ReplicaSetUpdated", Message:"ReplicaSet \"azuredisk-volume-tester-9r9tg-56f6699778\" is progressing."}}, CollisionCount:(*int32)(nil)}
Jan 14 02:05:01.791: INFO: deployment status: v1.DeploymentStatus{ObservedGeneration:1, Replicas:1, UpdatedReplicas:1, ReadyReplicas:0, AvailableReplicas:0, UnavailableReplicas:1, Conditions:[]v1.DeploymentCondition{v1.DeploymentCondition{Type:"Available", Status:"False", LastUpdateTime:time.Date(2023, time.January, 14, 2, 4, 15, 0, time.Local), LastTransitionTime:time.Date(2023, time.January, 14, 2, 4, 15, 0, time.Local), Reason:"MinimumReplicasUnavailable", Message:"Deployment does not have minimum availability."}, v1.DeploymentCondition{Type:"Progressing", Status:"True", LastUpdateTime:time.Date(2023, time.January, 14, 2, 4, 15, 0, time.Local), LastTransitionTime:time.Date(2023, time.January, 14, 2, 4, 15, 0, time.Local), Reason:"ReplicaSetUpdated", Message:"ReplicaSet \"azuredisk-volume-tester-9r9tg-56f6699778\" is progressing."}}, CollisionCount:(*int32)(nil)}
[1mSTEP:[0m checking that the pod is running [38;5;243m01/14/23 02:05:03.859[0m
[1mSTEP:[0m check pod exec [38;5;243m01/14/23 02:05:03.959[0m
Jan 14 02:05:03.959: INFO: Running '/usr/local/bin/kubectl --kubeconfig=/home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure/kubeconfig --namespace=azuredisk-4765 exec azuredisk-volume-tester-9r9tg-56f6699778-2fhnq -- cmd /c type C:\mnt\test-1\data.txt'
Jan 14 02:05:04.559: INFO: rc: 1
Jan 14 02:05:04.560: INFO: Error waiting for output "hello world\r\n" in pod "azuredisk-volume-tester-9r9tg-56f6699778-2fhnq": error running /usr/local/bin/kubectl --kubeconfig=/home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure/kubeconfig --namespace=azuredisk-4765 exec azuredisk-volume-tester-9r9tg-56f6699778-2fhnq -- cmd /c type C:\mnt\test-1\data.txt:
Command stdout:

stderr:
The system cannot find the file specified.
command terminated with exit code 1

error:
exit status 1.
Jan 14 02:05:06.560: INFO: Running '/usr/local/bin/kubectl --kubeconfig=/home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure/kubeconfig --namespace=azuredisk-4765 exec azuredisk-volume-tester-9r9tg-56f6699778-2fhnq -- cmd /c type C:\mnt\test-1\data.txt'
Jan 14 02:05:07.079: INFO: stderr: ""
Jan 14 02:05:07.079: INFO: stdout: "hello world\r\n"
[1mSTEP:[0m cordoning node 0 [38;5;243m01/14/23 02:05:07.079[0m
[1mSTEP:[0m deleting the pod for deployment [38;5;243m01/14/23 02:05:07.157[0m
... skipping 119 lines ...
    Jan 14 02:04:59.792: INFO: deployment status: v1.DeploymentStatus{ObservedGeneration:1, Replicas:1, UpdatedReplicas:1, ReadyReplicas:0, AvailableReplicas:0, UnavailableReplicas:1, Conditions:[]v1.DeploymentCondition{v1.DeploymentCondition{Type:"Available", Status:"False", LastUpdateTime:time.Date(2023, time.January, 14, 2, 4, 15, 0, time.Local), LastTransitionTime:time.Date(2023, time.January, 14, 2, 4, 15, 0, time.Local), Reason:"MinimumReplicasUnavailable", Message:"Deployment does not have minimum availability."}, v1.DeploymentCondition{Type:"Progressing", Status:"True", LastUpdateTime:time.Date(2023, time.January, 14, 2, 4, 15, 0, time.Local), LastTransitionTime:time.Date(2023, time.January, 14, 2, 4, 15, 0, time.Local), Reason:"ReplicaSetUpdated", Message:"ReplicaSet \"azuredisk-volume-tester-9r9tg-56f6699778\" is progressing."}}, CollisionCount:(*int32)(nil)}
    Jan 14 02:05:01.791: INFO: deployment status: v1.DeploymentStatus{ObservedGeneration:1, Replicas:1, UpdatedReplicas:1, ReadyReplicas:0, AvailableReplicas:0, UnavailableReplicas:1, Conditions:[]v1.DeploymentCondition{v1.DeploymentCondition{Type:"Available", Status:"False", LastUpdateTime:time.Date(2023, time.January, 14, 2, 4, 15, 0, time.Local), LastTransitionTime:time.Date(2023, time.January, 14, 2, 4, 15, 0, time.Local), Reason:"MinimumReplicasUnavailable", Message:"Deployment does not have minimum availability."}, v1.DeploymentCondition{Type:"Progressing", Status:"True", LastUpdateTime:time.Date(2023, time.January, 14, 2, 4, 15, 0, time.Local), LastTransitionTime:time.Date(2023, time.January, 14, 2, 4, 15, 0, time.Local), Reason:"ReplicaSetUpdated", Message:"ReplicaSet \"azuredisk-volume-tester-9r9tg-56f6699778\" is progressing."}}, CollisionCount:(*int32)(nil)}
    [1mSTEP:[0m checking that the pod is running [38;5;243m01/14/23 02:05:03.859[0m
    [1mSTEP:[0m check pod exec [38;5;243m01/14/23 02:05:03.959[0m
    Jan 14 02:05:03.959: INFO: Running '/usr/local/bin/kubectl --kubeconfig=/home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure/kubeconfig --namespace=azuredisk-4765 exec azuredisk-volume-tester-9r9tg-56f6699778-2fhnq -- cmd /c type C:\mnt\test-1\data.txt'
    Jan 14 02:05:04.559: INFO: rc: 1
    Jan 14 02:05:04.560: INFO: Error waiting for output "hello world\r\n" in pod "azuredisk-volume-tester-9r9tg-56f6699778-2fhnq": error running /usr/local/bin/kubectl --kubeconfig=/home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure/kubeconfig --namespace=azuredisk-4765 exec azuredisk-volume-tester-9r9tg-56f6699778-2fhnq -- cmd /c type C:\mnt\test-1\data.txt:
    Command stdout:

    stderr:
    The system cannot find the file specified.
    command terminated with exit code 1

    error:
    exit status 1.
    Jan 14 02:05:06.560: INFO: Running '/usr/local/bin/kubectl --kubeconfig=/home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure/kubeconfig --namespace=azuredisk-4765 exec azuredisk-volume-tester-9r9tg-56f6699778-2fhnq -- cmd /c type C:\mnt\test-1\data.txt'
    Jan 14 02:05:07.079: INFO: stderr: ""
    Jan 14 02:05:07.079: INFO: stdout: "hello world\r\n"
    [1mSTEP:[0m cordoning node 0 [38;5;243m01/14/23 02:05:07.079[0m
    [1mSTEP:[0m deleting the pod for deployment [38;5;243m01/14/23 02:05:07.157[0m
... skipping 208 lines ...
  [38;5;243m<< End Captured GinkgoWriter Output[0m

  [38;5;14mtest case not supported by Windows clusters[0m
  [38;5;14mIn [1m[It][0m[38;5;14m at: [1m/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/utils/testutil/env_utils.go:26[0m
[38;5;243m------------------------------[0m
[0mDynamic Provisioning [38;5;243m[single-az][0m
  [1mshould check failed replica attachments are recreated after space is made from a volume detaching.[0m
  [38;5;243m/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning_test.go:1778[0m
[1mSTEP:[0m Creating a kubernetes client [38;5;243m01/14/23 02:07:00.845[0m
Jan 14 02:07:00.846: INFO: >>> kubeConfig: /home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure/kubeconfig
[1mSTEP:[0m Building a namespace api object, basename azuredisk [38;5;243m01/14/23 02:07:00.847[0m
[1mSTEP:[0m Waiting for a default service account to be provisioned in namespace [38;5;243m01/14/23 02:07:00.948[0m
[1mSTEP:[0m Waiting for kube-root-ca.crt to be provisioned in namespace [38;5;243m01/14/23 02:07:01.012[0m
... skipping 6 lines ...
[38;5;243m------------------------------[0m
[38;5;14mS [SKIPPED] [0.570 seconds][0m
Dynamic Provisioning
[38;5;243m/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning_test.go:41[0m
  [single-az]
  [38;5;243m/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning_test.go:45[0m
    [38;5;14m[1m[It] should check failed replica attachments are recreated after space is made from a volume detaching.[0m
    [38;5;243m/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning_test.go:1778[0m

  [38;5;243mBegin Captured GinkgoWriter Output >>[0m
    [1mSTEP:[0m Creating a kubernetes client [38;5;243m01/14/23 02:07:00.845[0m
    Jan 14 02:07:00.846: INFO: >>> kubeConfig: /home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure/kubeconfig
    [1mSTEP:[0m Building a namespace api object, basename azuredisk [38;5;243m01/14/23 02:07:00.847[0m
... skipping 15 lines ...
Jan 14 02:07:01.416: INFO: >>> kubeConfig: /home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure/kubeconfig
[1mSTEP:[0m Building a namespace api object, basename azuredisk [38;5;243m01/14/23 02:07:01.418[0m
[1mSTEP:[0m Waiting for a default service account to be provisioned in namespace [38;5;243m01/14/23 02:07:01.522[0m
[1mSTEP:[0m Waiting for kube-root-ca.crt to be provisioned in namespace [38;5;243m01/14/23 02:07:01.586[0m
I0114 02:07:01.651895   42005 azuredisk_driver.go:52] Using azure disk driver: disk.csi.azure.com
[1mSTEP:[0m deploying the pod [38;5;243m01/14/23 02:07:01.695[0m
[1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/14/23 02:07:01.734[0m
Jan 14 02:07:01.734: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-sjkv5" in namespace "azuredisk-6238" to be "Succeeded or Failed"
Jan 14 02:07:01.766: INFO: Pod "azuredisk-volume-tester-sjkv5": Phase="Pending", Reason="", readiness=false. Elapsed: 32.520865ms
Jan 14 02:07:03.801: INFO: Pod "azuredisk-volume-tester-sjkv5": Phase="Pending", Reason="", readiness=false. Elapsed: 2.06704966s
Jan 14 02:07:05.800: INFO: Pod "azuredisk-volume-tester-sjkv5": Phase="Pending", Reason="", readiness=false. Elapsed: 4.066078077s
Jan 14 02:07:07.803: INFO: Pod "azuredisk-volume-tester-sjkv5": Phase="Running", Reason="", readiness=true. Elapsed: 6.06899289s
Jan 14 02:07:09.802: INFO: Pod "azuredisk-volume-tester-sjkv5": Phase="Running", Reason="", readiness=true. Elapsed: 8.068669526s
Jan 14 02:07:11.801: INFO: Pod "azuredisk-volume-tester-sjkv5": Phase="Running", Reason="", readiness=true. Elapsed: 10.067686828s
Jan 14 02:07:13.801: INFO: Pod "azuredisk-volume-tester-sjkv5": Phase="Running", Reason="", readiness=false. Elapsed: 12.067636915s
Jan 14 02:07:15.802: INFO: Pod "azuredisk-volume-tester-sjkv5": Phase="Succeeded", Reason="", readiness=false. Elapsed: 14.068261484s
[1mSTEP:[0m Saw pod success [38;5;243m01/14/23 02:07:15.802[0m
Jan 14 02:07:15.803: INFO: Pod "azuredisk-volume-tester-sjkv5" satisfied condition "Succeeded or Failed"
Jan 14 02:07:15.803: INFO: deleting Pod "azuredisk-6238"/"azuredisk-volume-tester-sjkv5"
Jan 14 02:07:15.856: INFO: Pod azuredisk-volume-tester-sjkv5 has the following logs: hello world

[1mSTEP:[0m Deleting pod azuredisk-volume-tester-sjkv5 in namespace azuredisk-6238 [38;5;243m01/14/23 02:07:15.856[0m
[1mSTEP:[0m Destroying namespace "azuredisk-6238" for this suite. [38;5;243m01/14/23 02:07:15.897[0m
[38;5;243m------------------------------[0m
... skipping 10 lines ...
    Jan 14 02:07:01.416: INFO: >>> kubeConfig: /home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure/kubeconfig
    [1mSTEP:[0m Building a namespace api object, basename azuredisk [38;5;243m01/14/23 02:07:01.418[0m
    [1mSTEP:[0m Waiting for a default service account to be provisioned in namespace [38;5;243m01/14/23 02:07:01.522[0m
    [1mSTEP:[0m Waiting for kube-root-ca.crt to be provisioned in namespace [38;5;243m01/14/23 02:07:01.586[0m
    I0114 02:07:01.651895   42005 azuredisk_driver.go:52] Using azure disk driver: disk.csi.azure.com
    [1mSTEP:[0m deploying the pod [38;5;243m01/14/23 02:07:01.695[0m
    [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/14/23 02:07:01.734[0m
    Jan 14 02:07:01.734: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-sjkv5" in namespace "azuredisk-6238" to be "Succeeded or Failed"
    Jan 14 02:07:01.766: INFO: Pod "azuredisk-volume-tester-sjkv5": Phase="Pending", Reason="", readiness=false. Elapsed: 32.520865ms
    Jan 14 02:07:03.801: INFO: Pod "azuredisk-volume-tester-sjkv5": Phase="Pending", Reason="", readiness=false. Elapsed: 2.06704966s
    Jan 14 02:07:05.800: INFO: Pod "azuredisk-volume-tester-sjkv5": Phase="Pending", Reason="", readiness=false. Elapsed: 4.066078077s
    Jan 14 02:07:07.803: INFO: Pod "azuredisk-volume-tester-sjkv5": Phase="Running", Reason="", readiness=true. Elapsed: 6.06899289s
    Jan 14 02:07:09.802: INFO: Pod "azuredisk-volume-tester-sjkv5": Phase="Running", Reason="", readiness=true. Elapsed: 8.068669526s
    Jan 14 02:07:11.801: INFO: Pod "azuredisk-volume-tester-sjkv5": Phase="Running", Reason="", readiness=true. Elapsed: 10.067686828s
    Jan 14 02:07:13.801: INFO: Pod "azuredisk-volume-tester-sjkv5": Phase="Running", Reason="", readiness=false. Elapsed: 12.067636915s
    Jan 14 02:07:15.802: INFO: Pod "azuredisk-volume-tester-sjkv5": Phase="Succeeded", Reason="", readiness=false. Elapsed: 14.068261484s
    [1mSTEP:[0m Saw pod success [38;5;243m01/14/23 02:07:15.802[0m
    Jan 14 02:07:15.803: INFO: Pod "azuredisk-volume-tester-sjkv5" satisfied condition "Succeeded or Failed"
    Jan 14 02:07:15.803: INFO: deleting Pod "azuredisk-6238"/"azuredisk-volume-tester-sjkv5"
    Jan 14 02:07:15.856: INFO: Pod azuredisk-volume-tester-sjkv5 has the following logs: hello world

    [1mSTEP:[0m Deleting pod azuredisk-volume-tester-sjkv5 in namespace azuredisk-6238 [38;5;243m01/14/23 02:07:15.856[0m
    [1mSTEP:[0m Destroying namespace "azuredisk-6238" for this suite. [38;5;243m01/14/23 02:07:15.897[0m
  [38;5;243m<< End Captured GinkgoWriter Output[0m
... skipping 17 lines ...
Jan 14 02:07:18.305: INFO: PersistentVolumeClaim pvc-tllxr found but phase is Pending instead of Bound.
Jan 14 02:07:20.338: INFO: PersistentVolumeClaim pvc-tllxr found and phase=Bound (4.101042341s)
[1mSTEP:[0m checking the PVC [38;5;243m01/14/23 02:07:20.338[0m
[1mSTEP:[0m validating provisioned PV [38;5;243m01/14/23 02:07:20.371[0m
[1mSTEP:[0m checking the PV [38;5;243m01/14/23 02:07:20.404[0m
[1mSTEP:[0m deploying the pod [38;5;243m01/14/23 02:07:20.446[0m
[1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/14/23 02:07:20.483[0m
Jan 14 02:07:20.483: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-szthn" in namespace "azuredisk-6031" to be "Succeeded or Failed"
Jan 14 02:07:20.516: INFO: Pod "azuredisk-volume-tester-szthn": Phase="Pending", Reason="", readiness=false. Elapsed: 32.655553ms
Jan 14 02:07:22.551: INFO: Pod "azuredisk-volume-tester-szthn": Phase="Pending", Reason="", readiness=false. Elapsed: 2.067982529s
Jan 14 02:07:24.553: INFO: Pod "azuredisk-volume-tester-szthn": Phase="Pending", Reason="", readiness=false. Elapsed: 4.070136811s
Jan 14 02:07:26.565: INFO: Pod "azuredisk-volume-tester-szthn": Phase="Pending", Reason="", readiness=false. Elapsed: 6.08188298s
Jan 14 02:07:28.551: INFO: Pod "azuredisk-volume-tester-szthn": Phase="Pending", Reason="", readiness=false. Elapsed: 8.067646048s
Jan 14 02:07:30.550: INFO: Pod "azuredisk-volume-tester-szthn": Phase="Pending", Reason="", readiness=false. Elapsed: 10.067032449s
... skipping 19 lines ...
Jan 14 02:08:10.552: INFO: Pod "azuredisk-volume-tester-szthn": Phase="Running", Reason="", readiness=true. Elapsed: 50.069129463s
Jan 14 02:08:12.551: INFO: Pod "azuredisk-volume-tester-szthn": Phase="Running", Reason="", readiness=false. Elapsed: 52.068495622s
Jan 14 02:08:14.552: INFO: Pod "azuredisk-volume-tester-szthn": Phase="Running", Reason="", readiness=false. Elapsed: 54.068933983s
Jan 14 02:08:16.552: INFO: Pod "azuredisk-volume-tester-szthn": Phase="Running", Reason="", readiness=false. Elapsed: 56.069499826s
Jan 14 02:08:18.552: INFO: Pod "azuredisk-volume-tester-szthn": Phase="Succeeded", Reason="", readiness=false. Elapsed: 58.068740033s
[1mSTEP:[0m Saw pod success [38;5;243m01/14/23 02:08:18.552[0m
Jan 14 02:08:18.552: INFO: Pod "azuredisk-volume-tester-szthn" satisfied condition "Succeeded or Failed"
Jan 14 02:08:18.552: INFO: deleting Pod "azuredisk-6031"/"azuredisk-volume-tester-szthn"
Jan 14 02:08:18.602: INFO: Pod azuredisk-volume-tester-szthn has the following logs: hello world

[1mSTEP:[0m Deleting pod azuredisk-volume-tester-szthn in namespace azuredisk-6031 [38;5;243m01/14/23 02:08:18.602[0m
Jan 14 02:08:18.675: INFO: deleting PVC "azuredisk-6031"/"pvc-tllxr"
Jan 14 02:08:18.675: INFO: Deleting PersistentVolumeClaim "pvc-tllxr"
... skipping 38 lines ...
    Jan 14 02:07:18.305: INFO: PersistentVolumeClaim pvc-tllxr found but phase is Pending instead of Bound.
    Jan 14 02:07:20.338: INFO: PersistentVolumeClaim pvc-tllxr found and phase=Bound (4.101042341s)
    [1mSTEP:[0m checking the PVC [38;5;243m01/14/23 02:07:20.338[0m
    [1mSTEP:[0m validating provisioned PV [38;5;243m01/14/23 02:07:20.371[0m
    [1mSTEP:[0m checking the PV [38;5;243m01/14/23 02:07:20.404[0m
    [1mSTEP:[0m deploying the pod [38;5;243m01/14/23 02:07:20.446[0m
    [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/14/23 02:07:20.483[0m
    Jan 14 02:07:20.483: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-szthn" in namespace "azuredisk-6031" to be "Succeeded or Failed"
    Jan 14 02:07:20.516: INFO: Pod "azuredisk-volume-tester-szthn": Phase="Pending", Reason="", readiness=false. Elapsed: 32.655553ms
    Jan 14 02:07:22.551: INFO: Pod "azuredisk-volume-tester-szthn": Phase="Pending", Reason="", readiness=false. Elapsed: 2.067982529s
    Jan 14 02:07:24.553: INFO: Pod "azuredisk-volume-tester-szthn": Phase="Pending", Reason="", readiness=false. Elapsed: 4.070136811s
    Jan 14 02:07:26.565: INFO: Pod "azuredisk-volume-tester-szthn": Phase="Pending", Reason="", readiness=false. Elapsed: 6.08188298s
    Jan 14 02:07:28.551: INFO: Pod "azuredisk-volume-tester-szthn": Phase="Pending", Reason="", readiness=false. Elapsed: 8.067646048s
    Jan 14 02:07:30.550: INFO: Pod "azuredisk-volume-tester-szthn": Phase="Pending", Reason="", readiness=false. Elapsed: 10.067032449s
... skipping 19 lines ...
    Jan 14 02:08:10.552: INFO: Pod "azuredisk-volume-tester-szthn": Phase="Running", Reason="", readiness=true. Elapsed: 50.069129463s
    Jan 14 02:08:12.551: INFO: Pod "azuredisk-volume-tester-szthn": Phase="Running", Reason="", readiness=false. Elapsed: 52.068495622s
    Jan 14 02:08:14.552: INFO: Pod "azuredisk-volume-tester-szthn": Phase="Running", Reason="", readiness=false. Elapsed: 54.068933983s
    Jan 14 02:08:16.552: INFO: Pod "azuredisk-volume-tester-szthn": Phase="Running", Reason="", readiness=false. Elapsed: 56.069499826s
    Jan 14 02:08:18.552: INFO: Pod "azuredisk-volume-tester-szthn": Phase="Succeeded", Reason="", readiness=false. Elapsed: 58.068740033s
    [1mSTEP:[0m Saw pod success [38;5;243m01/14/23 02:08:18.552[0m
    Jan 14 02:08:18.552: INFO: Pod "azuredisk-volume-tester-szthn" satisfied condition "Succeeded or Failed"
    Jan 14 02:08:18.552: INFO: deleting Pod "azuredisk-6031"/"azuredisk-volume-tester-szthn"
    Jan 14 02:08:18.602: INFO: Pod azuredisk-volume-tester-szthn has the following logs: hello world

    [1mSTEP:[0m Deleting pod azuredisk-volume-tester-szthn in namespace azuredisk-6031 [38;5;243m01/14/23 02:08:18.602[0m
    Jan 14 02:08:18.675: INFO: deleting PVC "azuredisk-6031"/"pvc-tllxr"
    Jan 14 02:08:18.675: INFO: Deleting PersistentVolumeClaim "pvc-tllxr"
... skipping 187 lines ...
Jan 14 02:15:21.344: INFO: PersistentVolumeClaim pvc-7jls4 found and phase=Bound (4.098972722s)
[1mSTEP:[0m checking the PVC [38;5;243m01/14/23 02:15:21.344[0m
[1mSTEP:[0m validating provisioned PV [38;5;243m01/14/23 02:15:21.377[0m
[1mSTEP:[0m checking the PV [38;5;243m01/14/23 02:15:21.409[0m
I0114 02:15:21.409358   42005 resource_setup.go:89] adding PV (pvc-5beae643-7d84-4005-b845-a438d65b72f6) to pod ()
[1mSTEP:[0m deploying the pod [38;5;243m01/14/23 02:15:21.454[0m
[1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/14/23 02:15:21.496[0m
Jan 14 02:15:21.497: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-fj5ns" in namespace "azuredisk-5425" to be "Succeeded or Failed"
Jan 14 02:15:21.530: INFO: Pod "azuredisk-volume-tester-fj5ns": Phase="Pending", Reason="", readiness=false. Elapsed: 33.748651ms
Jan 14 02:15:23.565: INFO: Pod "azuredisk-volume-tester-fj5ns": Phase="Pending", Reason="", readiness=false. Elapsed: 2.068716703s
Jan 14 02:15:25.565: INFO: Pod "azuredisk-volume-tester-fj5ns": Phase="Pending", Reason="", readiness=false. Elapsed: 4.068636223s
Jan 14 02:15:27.566: INFO: Pod "azuredisk-volume-tester-fj5ns": Phase="Pending", Reason="", readiness=false. Elapsed: 6.069110083s
Jan 14 02:15:29.565: INFO: Pod "azuredisk-volume-tester-fj5ns": Phase="Pending", Reason="", readiness=false. Elapsed: 8.0687782s
Jan 14 02:15:31.565: INFO: Pod "azuredisk-volume-tester-fj5ns": Phase="Pending", Reason="", readiness=false. Elapsed: 10.068582092s
... skipping 24 lines ...
Jan 14 02:16:21.566: INFO: Pod "azuredisk-volume-tester-fj5ns": Phase="Running", Reason="", readiness=true. Elapsed: 1m0.069467861s
Jan 14 02:16:23.566: INFO: Pod "azuredisk-volume-tester-fj5ns": Phase="Running", Reason="", readiness=false. Elapsed: 1m2.069886483s
Jan 14 02:16:25.567: INFO: Pod "azuredisk-volume-tester-fj5ns": Phase="Running", Reason="", readiness=false. Elapsed: 1m4.069956574s
Jan 14 02:16:27.567: INFO: Pod "azuredisk-volume-tester-fj5ns": Phase="Running", Reason="", readiness=false. Elapsed: 1m6.06996905s
Jan 14 02:16:29.567: INFO: Pod "azuredisk-volume-tester-fj5ns": Phase="Succeeded", Reason="", readiness=false. Elapsed: 1m8.07002958s
[1mSTEP:[0m Saw pod success [38;5;243m01/14/23 02:16:29.567[0m
Jan 14 02:16:29.567: INFO: Pod "azuredisk-volume-tester-fj5ns" satisfied condition "Succeeded or Failed"
Jan 14 02:16:29.567: INFO: deleting Pod "azuredisk-5425"/"azuredisk-volume-tester-fj5ns"
Jan 14 02:16:29.631: INFO: Pod azuredisk-volume-tester-fj5ns has the following logs: hello world

[1mSTEP:[0m Deleting pod azuredisk-volume-tester-fj5ns in namespace azuredisk-5425 [38;5;243m01/14/23 02:16:29.631[0m
Jan 14 02:16:29.676: INFO: deleting PVC "azuredisk-5425"/"pvc-7jls4"
Jan 14 02:16:29.676: INFO: Deleting PersistentVolumeClaim "pvc-7jls4"
... skipping 85 lines ...
    Jan 14 02:15:21.344: INFO: PersistentVolumeClaim pvc-7jls4 found and phase=Bound (4.098972722s)
    [1mSTEP:[0m checking the PVC [38;5;243m01/14/23 02:15:21.344[0m
    [1mSTEP:[0m validating provisioned PV [38;5;243m01/14/23 02:15:21.377[0m
    [1mSTEP:[0m checking the PV [38;5;243m01/14/23 02:15:21.409[0m
    I0114 02:15:21.409358   42005 resource_setup.go:89] adding PV (pvc-5beae643-7d84-4005-b845-a438d65b72f6) to pod ()
    [1mSTEP:[0m deploying the pod [38;5;243m01/14/23 02:15:21.454[0m
    [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/14/23 02:15:21.496[0m
    Jan 14 02:15:21.497: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-fj5ns" in namespace "azuredisk-5425" to be "Succeeded or Failed"
    Jan 14 02:15:21.530: INFO: Pod "azuredisk-volume-tester-fj5ns": Phase="Pending", Reason="", readiness=false. Elapsed: 33.748651ms
    Jan 14 02:15:23.565: INFO: Pod "azuredisk-volume-tester-fj5ns": Phase="Pending", Reason="", readiness=false. Elapsed: 2.068716703s
    Jan 14 02:15:25.565: INFO: Pod "azuredisk-volume-tester-fj5ns": Phase="Pending", Reason="", readiness=false. Elapsed: 4.068636223s
    Jan 14 02:15:27.566: INFO: Pod "azuredisk-volume-tester-fj5ns": Phase="Pending", Reason="", readiness=false. Elapsed: 6.069110083s
    Jan 14 02:15:29.565: INFO: Pod "azuredisk-volume-tester-fj5ns": Phase="Pending", Reason="", readiness=false. Elapsed: 8.0687782s
    Jan 14 02:15:31.565: INFO: Pod "azuredisk-volume-tester-fj5ns": Phase="Pending", Reason="", readiness=false. Elapsed: 10.068582092s
... skipping 24 lines ...
    Jan 14 02:16:21.566: INFO: Pod "azuredisk-volume-tester-fj5ns": Phase="Running", Reason="", readiness=true. Elapsed: 1m0.069467861s
    Jan 14 02:16:23.566: INFO: Pod "azuredisk-volume-tester-fj5ns": Phase="Running", Reason="", readiness=false. Elapsed: 1m2.069886483s
    Jan 14 02:16:25.567: INFO: Pod "azuredisk-volume-tester-fj5ns": Phase="Running", Reason="", readiness=false. Elapsed: 1m4.069956574s
    Jan 14 02:16:27.567: INFO: Pod "azuredisk-volume-tester-fj5ns": Phase="Running", Reason="", readiness=false. Elapsed: 1m6.06996905s
    Jan 14 02:16:29.567: INFO: Pod "azuredisk-volume-tester-fj5ns": Phase="Succeeded", Reason="", readiness=false. Elapsed: 1m8.07002958s
    [1mSTEP:[0m Saw pod success [38;5;243m01/14/23 02:16:29.567[0m
    Jan 14 02:16:29.567: INFO: Pod "azuredisk-volume-tester-fj5ns" satisfied condition "Succeeded or Failed"
    Jan 14 02:16:29.567: INFO: deleting Pod "azuredisk-5425"/"azuredisk-volume-tester-fj5ns"
    Jan 14 02:16:29.631: INFO: Pod azuredisk-volume-tester-fj5ns has the following logs: hello world

    [1mSTEP:[0m Deleting pod azuredisk-volume-tester-fj5ns in namespace azuredisk-5425 [38;5;243m01/14/23 02:16:29.631[0m
    Jan 14 02:16:29.676: INFO: deleting PVC "azuredisk-5425"/"pvc-7jls4"
    Jan 14 02:16:29.676: INFO: Deleting PersistentVolumeClaim "pvc-7jls4"
... skipping 795 lines ...
I0114 00:40:03.934175       1 round_trippers.go:553] GET https://10.96.0.1:443/apis/apiextensions.k8s.io/v1/customresourcedefinitions?limit=500&resourceVersion=0 200 OK in 202 milliseconds
I0114 00:40:03.968327       1 round_trippers.go:553] GET https://10.96.0.1:443/apis/apiextensions.k8s.io/v1/customresourcedefinitions?allowWatchBookmarks=true&resourceVersion=3845&timeout=9m48s&timeoutSeconds=588&watch=true 200 OK in 0 milliseconds
I0114 00:40:04.030507       1 shared_informer.go:303] caches populated
I0114 00:40:04.030531       1 azuredisk_v2.go:225] driver userAgent: disk.csi.azure.com/latest-v2-9ef068a8cb36a997d4ea04b90c05c6f92a488a19 e2e-test
I0114 00:40:04.030542       1 azure_disk_utils.go:564] reading cloud config from secret kube-system/azure-cloud-provider
I0114 00:40:04.033026       1 round_trippers.go:553] GET https://10.96.0.1:443/api/v1/namespaces/kube-system/secrets/azure-cloud-provider 404 Not Found in 2 milliseconds
I0114 00:40:04.033224       1 azure_disk_utils.go:571] InitializeCloudFromSecret: failed to get cloud config from secret kube-system/azure-cloud-provider: failed to get secret kube-system/azure-cloud-provider: secrets "azure-cloud-provider" not found
I0114 00:40:04.033231       1 azure_disk_utils.go:576] could not read cloud config from secret kube-system/azure-cloud-provider
I0114 00:40:04.033236       1 azure_disk_utils.go:586] use default AZURE_CREDENTIAL_FILE env var: /etc/kubernetes/azure.json
I0114 00:40:04.033258       1 azure_disk_utils.go:594] read cloud config from file: /etc/kubernetes/azure.json successfully
I0114 00:40:04.033740       1 azure_auth.go:253] Using AzurePublicCloud environment
I0114 00:40:04.033765       1 azure_auth.go:138] azure: using client_id+client_secret to retrieve access token
I0114 00:40:04.033815       1 azure_diskclient.go:68] Azure DisksClient using API version: 2022-03-02
... skipping 9 lines ...
I0114 00:40:04.503284       1 azure_backoff.go:101] VirtualMachinesClient.List(capz-thn7c4) success
I0114 00:40:04.503296       1 azure_vmss_cache.go:400] Node  has joined the cluster since the last VM cache refresh in NonVmssUniformNodesEntry, refreshing the cache
I0114 00:40:04.503303       1 azure_vmss_cache.go:327] refresh the cache of NonVmssUniformNodesCache in rg map[capz-thn7c4:{}]
I0114 00:40:04.567862       1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 13878
I0114 00:40:04.568693       1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=0.065337928 request="vm_list" resource_group="capz-thn7c4" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0114 00:40:04.568709       1 azure_backoff.go:101] VirtualMachinesClient.List(capz-thn7c4) success
E0114 00:40:04.568736       1 azuredisk_v2.go:272] Failed to get node info. Error: NewNodeInfo: Failed to get instance type from Azure cloud provider, nodeName: , error: not a vmss instance
I0114 00:40:04.568837       1 mount_linux.go:208] Detected OS without systemd
I0114 00:40:04.568849       1 driver.go:81] Enabling controller service capability: CREATE_DELETE_VOLUME
I0114 00:40:04.568853       1 driver.go:81] Enabling controller service capability: PUBLISH_UNPUBLISH_VOLUME
I0114 00:40:04.568857       1 driver.go:81] Enabling controller service capability: CREATE_DELETE_SNAPSHOT
I0114 00:40:04.568861       1 driver.go:81] Enabling controller service capability: CLONE_VOLUME
I0114 00:40:04.568865       1 driver.go:81] Enabling controller service capability: EXPAND_VOLUME
... skipping 292 lines ...
I0114 00:40:04.643602       1 shared_state.go:490]  "msg"="Storing pod tigera-operator-6bbf97c9cf-jt7lx and claim [] to podToClaimsMap map." "disk.csi.azure.com/request-id"="fca6d0f6-93a3-11ed-906c-002248afe19a"
I0114 00:40:04.643638       1 pod.go:91]  "msg"="Creating replicas for pod tigera-operator/tigera-operator-6bbf97c9cf-jt7lx." "disk.csi.azure.com/request-id"="fca6d0f6-93a3-11ed-906c-002248afe19a" "disk.csi/azure.com/pod-name"="tigera-operator/tigera-operator-6bbf97c9cf-jt7lx"
I0114 00:40:04.643651       1 shared_state.go:314]  "msg"="Getting requested volumes for pod (tigera-operator/tigera-operator-6bbf97c9cf-jt7lx)." "disk.csi.azure.com/request-id"="fca6d0f6-93a3-11ed-906c-002248afe19a" "disk.csi/azure.com/pod-name"="tigera-operator/tigera-operator-6bbf97c9cf-jt7lx"
I0114 00:40:04.643662       1 pod.go:99]  "msg"="Pod tigera-operator/tigera-operator-6bbf97c9cf-jt7lx has 0 volumes. Volumes: []" "disk.csi.azure.com/request-id"="fca6d0f6-93a3-11ed-906c-002248afe19a" "disk.csi/azure.com/pod-name"="tigera-operator/tigera-operator-6bbf97c9cf-jt7lx"
I0114 00:40:04.643676       1 pod.go:89]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="fca6d0f6-93a3-11ed-906c-002248afe19a" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcilePod).createReplicas" "disk.csi/azure.com/pod-name"="tigera-operator/tigera-operator-6bbf97c9cf-jt7lx" "latency"=38998
I0114 00:40:04.643707       1 shared_state.go:420]  "msg"="Adding pod csi-azuredisk-controller-7f6f8d54b5-f5wbz to shared map with keyName kube-system/csi-azuredisk-controller-7f6f8d54b5-f5wbz." "disk.csi.azure.com/request-id"="fca6d0f6-93a3-11ed-906c-002248afe19a"
I0114 00:40:04.644043       1 shared_state.go:426]  "msg"="Pod spec of pod csi-azuredisk-controller-7f6f8d54b5-f5wbz is: {Volumes:[{Name:socket-dir VolumeSource:{HostPath:nil EmptyDir:&EmptyDirVolumeSource{Medium:,SizeLimit:<nil>,} GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:azure-cred VolumeSource:{HostPath:&HostPathVolumeSource{Path:/etc/kubernetes/,Type:*DirectoryOrCreate,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:csi-azuredisk-controller-config VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:&ConfigMapVolumeSource{LocalObjectReference:LocalObjectReference{Name:csi-azuredisk-controller-config,},Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,} VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:kube-api-access-qlxwh VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:&ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,} PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:csi-provisioner Image:mcr.microsoft.com/oss/kubernetes-csi/csi-provisioner:v3.2.0 Command:[] Args:[--feature-gates=Topology=true --csi-address=$(ADDRESS) --v=2 --timeout=30s --leader-election --leader-election-namespace=kube-system --worker-threads=100 --extra-create-metadata=true --strict-topology=true --kube-api-qps=50 --kube-api-burst=100] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:ADDRESS Value:/csi/csi.sock ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:524288000 scale:0} d:{Dec:<nil>} s:500Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}] Claims:[]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-qlxwh ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:csi-attacher Image:mcr.microsoft.com/oss/kubernetes-csi/csi-attacher:v3.5.0 Command:[] Args:[-v=2 -csi-address=$(ADDRESS) -timeout=600s -leader-election --leader-election-namespace=kube-system -worker-threads=500 -kube-api-qps=50 -kube-api-burst=100] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:ADDRESS Value:/csi/csi.sock ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:524288000 scale:0} d:{Dec:<nil>} s:500Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}] Claims:[]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-qlxwh ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:csi-snapshotter Image:mcr.microsoft.com/oss/kubernetes-csi/csi-snapshotter:v5.0.1 Command:[] Args:[-csi-address=$(ADDRESS) -leader-election --leader-election-namespace=kube-system -v=2] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:ADDRESS Value:/csi/csi.sock ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:104857600 scale:0} d:{Dec:<nil>} s:100Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}] Claims:[]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-qlxwh ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:csi-resizer Image:mcr.microsoft.com/oss/kubernetes-csi/csi-resizer:v1.5.0 Command:[] Args:[-csi-address=$(ADDRESS) -v=2 -leader-election --leader-election-namespace=kube-system -handle-volume-inuse-error=false -feature-gates=RecoverVolumeExpansionFailure=true -timeout=240s] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:ADDRESS Value:/csi/csi.sock ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:524288000 scale:0} d:{Dec:<nil>} s:500Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}] Claims:[]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-qlxwh ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:liveness-probe Image:mcr.microsoft.com/oss/kubernetes-csi/livenessprobe:v2.7.0 Command:[] Args:[--csi-address=/csi/csi.sock --probe-timeout=3s --health-port=29602 --v=2] WorkingDir: Ports:[] EnvFrom:[] Env:[] Resources:{Limits:map[memory:{i:{value:104857600 scale:0} d:{Dec:<nil>} s:100Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}] Claims:[]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-qlxwh ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:azuredisk Image:capzci.azurecr.io/azuredisk-csi:latest-v2-9ef068a8cb36a997d4ea04b90c05c6f92a488a19 Command:[] Args:[--v=6 --config=/etc/csi-azuredisk-controller/config.yaml] WorkingDir: Ports:[{Name:healthz HostPort:29602 ContainerPort:29602 Protocol:TCP HostIP:} {Name:metrics HostPort:29604 ContainerPort:29604 Protocol:TCP HostIP:}] EnvFrom:[] Env:[{Name:AZURE_CREDENTIAL_FILE Value: ValueFrom:&EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:&ConfigMapKeySelector{LocalObjectReference:LocalObjectReference{Name:azure-cred-file,},Key:path,Optional:*true,},SecretKeyRef:nil,}} {Name:CSI_ENDPOINT Value:unix:///csi/csi.sock ValueFrom:nil} {Name:AZURE_GO_SDK_LOG_LEVEL Value: ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:524288000 scale:0} d:{Dec:<nil>} s:500Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}] Claims:[]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:azure-cred ReadOnly:false MountPath:/etc/kubernetes/ SubPath: MountPropagation:<nil> SubPathExpr:} {Name:csi-azuredisk-controller-config ReadOnly:false MountPath:/etc/csi-azuredisk-controller SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-qlxwh ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/healthz,Port:{1 0 healthz},Host:,Scheme:HTTP,HTTPHeaders:[]HTTPHeader{},},TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:30,TimeoutSeconds:10,PeriodSeconds:30,SuccessThreshold:1,FailureThreshold:5,TerminationGracePeriodSeconds:nil,} ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:Always SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy:Always TerminationGracePeriodSeconds:0xc000a6bcb8 ActiveDeadlineSeconds:<nil> DNSPolicy:ClusterFirst NodeSelector:map[kubernetes.io/os:linux] ServiceAccountName:csi-azuredisk-controller-sa DeprecatedServiceAccount:csi-azuredisk-controller-sa AutomountServiceAccountToken:<nil> NodeName:capz-thn7c4-control-plane-4lzvn HostNetwork:true HostPID:false HostIPC:false ShareProcessNamespace:<nil> SecurityContext:&PodSecurityContext{SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,SupplementalGroups:[],FSGroup:nil,RunAsGroup:nil,Sysctls:[]Sysctl{},WindowsOptions:nil,FSGroupChangePolicy:nil,SeccompProfile:nil,} ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName:default-scheduler Tolerations:[{Key:node-role.kubernetes.io/master Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>} {Key:node-role.kubernetes.io/controlplane Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>} {Key:node-role.kubernetes.io/control-plane Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>} {Key:node.kubernetes.io/not-ready Operator:Exists Value: Effect:NoExecute TolerationSeconds:0xc000a6bcc0} {Key:node.kubernetes.io/unreachable Operator:Exists Value: Effect:NoExecute TolerationSeconds:0xc000a6bcc8}] HostAliases:[] PriorityClassName:system-cluster-critical Priority:0xc000a6bcd0 DNSConfig:nil ReadinessGates:[] RuntimeClassName:<nil> EnableServiceLinks:0xc000a6bcd4 PreemptionPolicy:0xc0003377e0 Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN:<nil> OS:nil HostUsers:<nil> SchedulingGates:[] ResourceClaims:[]}. With volumes: [{Name:socket-dir VolumeSource:{HostPath:nil EmptyDir:&EmptyDirVolumeSource{Medium:,SizeLimit:<nil>,} GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:azure-cred VolumeSource:{HostPath:&HostPathVolumeSource{Path:/etc/kubernetes/,Type:*DirectoryOrCreate,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:csi-azuredisk-controller-config VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:&ConfigMapVolumeSource{LocalObjectReference:LocalObjectReference{Name:csi-azuredisk-controller-config,},Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,} VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:kube-api-access-qlxwh VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:&ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,} PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}]" "disk.csi.azure.com/request-id"="fca6d0f6-93a3-11ed-906c-002248afe19a"
I0114 00:40:04.644099       1 shared_state.go:464]  "msg"="Pod csi-azuredisk-controller-7f6f8d54b5-f5wbz: Skipping Volume {socket-dir {nil &EmptyDirVolumeSource{Medium:,SizeLimit:<nil>,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}}. No persistent volume exists." "disk.csi.azure.com/request-id"="fca6d0f6-93a3-11ed-906c-002248afe19a"
I0114 00:40:04.644119       1 shared_state.go:464]  "msg"="Pod csi-azuredisk-controller-7f6f8d54b5-f5wbz: Skipping Volume {azure-cred {&HostPathVolumeSource{Path:/etc/kubernetes/,Type:*DirectoryOrCreate,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}}. No persistent volume exists." "disk.csi.azure.com/request-id"="fca6d0f6-93a3-11ed-906c-002248afe19a"
I0114 00:40:04.644137       1 shared_state.go:464]  "msg"="Pod csi-azuredisk-controller-7f6f8d54b5-f5wbz: Skipping Volume {csi-azuredisk-controller-config {nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil &ConfigMapVolumeSource{LocalObjectReference:LocalObjectReference{Name:csi-azuredisk-controller-config,},Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,} nil nil nil nil nil nil nil nil nil nil}}. No persistent volume exists." "disk.csi.azure.com/request-id"="fca6d0f6-93a3-11ed-906c-002248afe19a"
I0114 00:40:04.644192       1 shared_state.go:464]  "msg"="Pod csi-azuredisk-controller-7f6f8d54b5-f5wbz: Skipping Volume {kube-api-access-qlxwh {nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil &ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,} nil nil nil nil nil}}. No persistent volume exists." "disk.csi.azure.com/request-id"="fca6d0f6-93a3-11ed-906c-002248afe19a"
I0114 00:40:04.644207       1 shared_state.go:490]  "msg"="Storing pod csi-azuredisk-controller-7f6f8d54b5-f5wbz and claim [] to podToClaimsMap map." "disk.csi.azure.com/request-id"="fca6d0f6-93a3-11ed-906c-002248afe19a"
I0114 00:40:04.644222       1 pod.go:91]  "msg"="Creating replicas for pod kube-system/csi-azuredisk-controller-7f6f8d54b5-f5wbz." "disk.csi.azure.com/request-id"="fca6d0f6-93a3-11ed-906c-002248afe19a" "disk.csi/azure.com/pod-name"="kube-system/csi-azuredisk-controller-7f6f8d54b5-f5wbz"
... skipping 261 lines ...
I0114 00:40:04.713205       1 shared_informer.go:303] caches populated
I0114 00:40:04.713295       1 controller.go:228] AzDiskControllerManager "msg"="Starting workers" "controller"="watch " "namespace"="azure-disk-csi" "partition"="csi-azuredisk-controller" "worker count"=1
I0114 00:40:04.713414       1 shared_state.go:420]  "msg"="Adding pod calico-node-7dcnd to shared map with keyName calico-system/calico-node-7dcnd." 
I0114 00:40:04.713559       1 azure_disk_utils.go:1048]  "msg"="Getting AzVolumeAttachment list for node (capz-thn7-slzjr)" 
I0114 00:40:04.713660       1 azure_disk_utils.go:1068]  "msg"="Label selector is: disk.csi.azure.com/node-name=capz-thn7-slzjr." 
I0114 00:40:04.713746       1 shared_state.go:1745] Added node(capz-thn7-slzjr) to availableAttachmentsMap with capacity: 8
I0114 00:40:04.713826       1 node.go:90] AzDiskControllerManager "msg"="Node is now available. Will requeue failed replica creation requests." "controller"="node" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "namespace"="azure-disk-csi" "partition"="csi-azuredisk-controller"
I0114 00:40:04.713901       1 shared_state.go:1205]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="fcb6b81f-93a3-11ed-906c-002248afe19a" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*SharedState).tryCreateFailedReplicas" "latency"=5099
I0114 00:40:04.714023       1 azure_disk_utils.go:1048]  "msg"="Getting AzVolumeAttachment list for node (capz-thn7c4-control-plane-4lzvn)" 
I0114 00:40:04.714110       1 azure_disk_utils.go:1068]  "msg"="Label selector is: disk.csi.azure.com/node-name=capz-thn7c4-control-plane-4lzvn." 
I0114 00:40:04.714624       1 shared_state.go:1745] Added node(capz-thn7c4-control-plane-4lzvn) to availableAttachmentsMap with capacity: 4
I0114 00:40:04.714734       1 node.go:90] AzDiskControllerManager "msg"="Node is now available. Will requeue failed replica creation requests." "controller"="node" "disk.csi.azure.com/node-name"="capz-thn7c4-control-plane-4lzvn" "namespace"="azure-disk-csi" "partition"="csi-azuredisk-controller"
I0114 00:40:04.714799       1 shared_state.go:420]  "msg"="Adding pod calico-apiserver-5bd69d647b-sn5fd to shared map with keyName calico-apiserver/calico-apiserver-5bd69d647b-sn5fd." 
I0114 00:40:04.714881       1 shared_state.go:1205]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="fcb6dbf8-93a3-11ed-906c-002248afe19a" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*SharedState).tryCreateFailedReplicas" "latency"=5900
I0114 00:40:04.714634       1 shared_state.go:420]  "msg"="Adding pod calico-kube-controllers-fb49b9cf7-4zqzq to shared map with keyName calico-system/calico-kube-controllers-fb49b9cf7-4zqzq." 
I0114 00:40:04.715206       1 shared_state.go:426]  "msg"="Pod spec of pod calico-apiserver-5bd69d647b-sn5fd is: {Volumes:[{Name:calico-apiserver-certs VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:&SecretVolumeSource{SecretName:calico-apiserver-certs,Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,} NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:kube-api-access-9rzxx VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:&ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,} PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:calico-apiserver Image:docker.io/calico/apiserver:v3.25.0 Command:[] Args:[--secure-port=5443 --tls-private-key-file=/calico-apiserver-certs/tls.key --tls-cert-file=/calico-apiserver-certs/tls.crt] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:DATASTORE_TYPE Value:kubernetes ValueFrom:nil} {Name:KUBERNETES_SERVICE_HOST Value:10.96.0.1 ValueFrom:nil} {Name:KUBERNETES_SERVICE_PORT Value:443 ValueFrom:nil} {Name:MULTI_INTERFACE_MODE Value:none ValueFrom:nil}] Resources:{Limits:map[] Requests:map[] Claims:[]} VolumeMounts:[{Name:calico-apiserver-certs ReadOnly:true MountPath:/calico-apiserver-certs SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-9rzxx ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/version,Port:{0 5443 },Host:,Scheme:HTTPS,HTTPHeaders:[]HTTPHeader{},},TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:90,TimeoutSeconds:1,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:3,TerminationGracePeriodSeconds:nil,} ReadinessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[/code/filecheck],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:5,TimeoutSeconds:1,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:5,TerminationGracePeriodSeconds:nil,} StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:IfNotPresent SecurityContext:&SecurityContext{Capabilities:nil,Privileged:*false,SELinuxOptions:nil,RunAsUser:*0,RunAsNonRoot:nil,ReadOnlyRootFilesystem:nil,AllowPrivilegeEscalation:nil,RunAsGroup:nil,ProcMount:nil,WindowsOptions:nil,SeccompProfile:nil,} Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy:Always TerminationGracePeriodSeconds:0xc00147ffd8 ActiveDeadlineSeconds:<nil> DNSPolicy:ClusterFirst NodeSelector:map[kubernetes.io/os:linux] ServiceAccountName:calico-apiserver DeprecatedServiceAccount:calico-apiserver AutomountServiceAccountToken:<nil> NodeName:capz-thn7c4-control-plane-4lzvn HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace:<nil> SecurityContext:&PodSecurityContext{SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,SupplementalGroups:[],FSGroup:nil,RunAsGroup:nil,Sysctls:[]Sysctl{},WindowsOptions:nil,FSGroupChangePolicy:nil,SeccompProfile:nil,} ImagePullSecrets:[] Hostname: Subdomain: Affinity:&Affinity{NodeAffinity:nil,PodAffinity:nil,PodAntiAffinity:&PodAntiAffinity{RequiredDuringSchedulingIgnoredDuringExecution:[]PodAffinityTerm{},PreferredDuringSchedulingIgnoredDuringExecution:[]WeightedPodAffinityTerm{WeightedPodAffinityTerm{Weight:100,PodAffinityTerm:PodAffinityTerm{LabelSelector:&v1.LabelSelector{MatchLabels:map[string]string{k8s-app: calico-apiserver,},MatchExpressions:[]LabelSelectorRequirement{},},Namespaces:[calico-apiserver],TopologyKey:kubernetes.io/hostname,NamespaceSelector:nil,},},WeightedPodAffinityTerm{Weight:100,PodAffinityTerm:PodAffinityTerm{LabelSelector:&v1.LabelSelector{MatchLabels:map[string]string{k8s-app: calico-apiserver,},MatchExpressions:[]LabelSelectorRequirement{},},Namespaces:[calico-apiserver],TopologyKey:topology.kubernetes.io/zone,NamespaceSelector:nil,},},},},} SchedulerName:default-scheduler Tolerations:[{Key:node-role.kubernetes.io/master Operator: Value: Effect:NoSchedule TolerationSeconds:<nil>} {Key:node-role.kubernetes.io/control-plane Operator: Value: Effect:NoSchedule TolerationSeconds:<nil>} {Key:node.kubernetes.io/not-ready Operator:Exists Value: Effect:NoExecute TolerationSeconds:0xc00147ffe0} {Key:node.kubernetes.io/unreachable Operator:Exists Value: Effect:NoExecute TolerationSeconds:0xc00147ffe8}] HostAliases:[] PriorityClassName: Priority:0xc00147fff0 DNSConfig:nil ReadinessGates:[] RuntimeClassName:<nil> EnableServiceLinks:0xc00147fff4 PreemptionPolicy:0xc0014a63a0 Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN:<nil> OS:nil HostUsers:<nil> SchedulingGates:[] ResourceClaims:[]}. With volumes: [{Name:calico-apiserver-certs VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:&SecretVolumeSource{SecretName:calico-apiserver-certs,Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,} NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:kube-api-access-9rzxx VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:&ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,} PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}]" 
I0114 00:40:04.715267       1 shared_state.go:464]  "msg"="Pod calico-apiserver-5bd69d647b-sn5fd: Skipping Volume {calico-apiserver-certs {nil nil nil nil nil &SecretVolumeSource{SecretName:calico-apiserver-certs,Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}}. No persistent volume exists." 
I0114 00:40:04.715445       1 shared_state.go:464]  "msg"="Pod calico-apiserver-5bd69d647b-sn5fd: Skipping Volume {kube-api-access-9rzxx {nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil &ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,} nil nil nil nil nil}}. No persistent volume exists." 
... skipping 19 lines ...
I0114 00:40:04.717299       1 shared_state.go:420]  "msg"="Adding pod csi-azuredisk-controller-7f6f8d54b5-f5wbz to shared map with keyName kube-system/csi-azuredisk-controller-7f6f8d54b5-f5wbz." 
I0114 00:40:04.717848       1 shared_state.go:426]  "msg"="Pod spec of pod csi-azuredisk-node-win-rw959 is: {Volumes:[{Name:init-script VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:&ConfigMapVolumeSource{LocalObjectReference:LocalObjectReference{Name:csi-azuredisk-node-win-init,},Items:[]KeyToPath{KeyToPath{Key:Initialize-Pod.ps1,Path:Initialize-Pod.ps1,Mode:nil,},},DefaultMode:*420,Optional:nil,} VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:csi-azuredisk-node-win-config VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:&ConfigMapVolumeSource{LocalObjectReference:LocalObjectReference{Name:csi-azuredisk-node-win-config,},Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,} VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:kube-api-access-s6wtk VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:&ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,} PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[{Name:init Image:capzci.azurecr.io/azuredisk-csi:latest-v2-9ef068a8cb36a997d4ea04b90c05c6f92a488a19 Command:[powershell.exe -c .\\etc\\config\\Initialize-Pod.ps1] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[] Resources:{Limits:map[] Requests:map[] Claims:[]} VolumeMounts:[{Name:init-script ReadOnly:true MountPath:/etc/config SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-s6wtk ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:Always SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] Containers:[{Name:liveness-probe Image:mcr.microsoft.com/oss/kubernetes-csi/livenessprobe:v2.7.0 Command:[livenessprobe.exe] Args:[--csi-address=$(CSI_ENDPOINT) --probe-timeout=3s --health-port=29603 --v=2] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:CSI_ENDPOINT Value:unix://C:\\\\var\\\\lib\\\\kubelet\\\\plugins\\\\disk.csi.azure.com\\\\csi.sock ValueFrom:nil}] Resources:{Limits:map[] Requests:map[] Claims:[]} VolumeMounts:[{Name:kube-api-access-s6wtk ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:node-driver-registrar Image:mcr.microsoft.com/oss/kubernetes-csi/csi-node-driver-registrar:v2.5.1 Command:[csi-node-driver-registrar.exe] Args:[--v=2 --csi-address=$(CSI_ENDPOINT) --kubelet-registration-path=$(DRIVER_REG_SOCK_PATH) --plugin-registration-path=$(PLUGIN_REG_DIR)] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:CSI_ENDPOINT Value:unix://C:\\\\var\\\\lib\\\\kubelet\\\\plugins\\\\disk.csi.azure.com\\\\csi.sock ValueFrom:nil} {Name:DRIVER_REG_SOCK_PATH Value:C:\\\\var\\\\lib\\\\kubelet\\\\plugins\\\\disk.csi.azure.com\\\\csi.sock ValueFrom:nil} {Name:PLUGIN_REG_DIR Value:C:\\\\var\\\\lib\\\\kubelet\\\\plugins_registry\\\\ ValueFrom:nil} {Name:KUBE_NODE_NAME Value: ValueFrom:&EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:spec.nodeName,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}}] Resources:{Limits:map[] Requests:map[] Claims:[]} VolumeMounts:[{Name:kube-api-access-s6wtk ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[csi-node-driver-registrar.exe --kubelet-registration-path=$(DRIVER_REG_SOCK_PATH) --mode=kubelet-registration-probe],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:60,TimeoutSeconds:30,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:3,TerminationGracePeriodSeconds:nil,} ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:azuredisk Image:capzci.azurecr.io/azuredisk-csi:latest-v2-9ef068a8cb36a997d4ea04b90c05c6f92a488a19 Command:[azurediskplugin.exe] Args:[--v=6 --nodeid=$(KUBE_NODE_NAME) --config=.\\etc\\csi-azuredisk-node-win\\config.yaml] WorkingDir: Ports:[{Name:healthz HostPort:29603 ContainerPort:29603 Protocol:TCP HostIP:}] EnvFrom:[] Env:[{Name:AZURE_CREDENTIAL_FILE Value: ValueFrom:&EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:&ConfigMapKeySelector{LocalObjectReference:LocalObjectReference{Name:azure-cred-file,},Key:path-windows,Optional:*true,},SecretKeyRef:nil,}} {Name:CSI_ENDPOINT Value:unix://C:\\\\var\\\\lib\\\\kubelet\\\\plugins\\\\disk.csi.azure.com\\\\csi.sock ValueFrom:nil} {Name:KUBE_NODE_NAME Value: ValueFrom:&EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:spec.nodeName,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}} {Name:AZURE_GO_SDK_LOG_LEVEL Value: ValueFrom:nil}] Resources:{Limits:map[] Requests:map[] Claims:[]} VolumeMounts:[{Name:csi-azuredisk-node-win-config ReadOnly:false MountPath:/etc/csi-azuredisk-node-win SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-s6wtk ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/healthz,Port:{1 0 healthz},Host:,Scheme:HTTP,HTTPHeaders:[]HTTPHeader{},},TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:30,TimeoutSeconds:10,PeriodSeconds:30,SuccessThreshold:1,FailureThreshold:5,TerminationGracePeriodSeconds:nil,} ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:Always SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy:Always TerminationGracePeriodSeconds:0xc001512830 ActiveDeadlineSeconds:<nil> DNSPolicy:ClusterFirst NodeSelector:map[kubernetes.io/os:windows] ServiceAccountName:csi-azuredisk-node-sa DeprecatedServiceAccount:csi-azuredisk-node-sa AutomountServiceAccountToken:<nil> NodeName:capz-thn7-zhv6p HostNetwork:true HostPID:false HostIPC:false ShareProcessNamespace:<nil> SecurityContext:&PodSecurityContext{SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,SupplementalGroups:[],FSGroup:nil,RunAsGroup:nil,Sysctls:[]Sysctl{},WindowsOptions:&WindowsSecurityContextOptions{GMSACredentialSpecName:nil,GMSACredentialSpec:nil,RunAsUserName:*NT AUTHORITY\\SYSTEM,HostProcess:*true,},FSGroupChangePolicy:nil,SeccompProfile:nil,} ImagePullSecrets:[] Hostname: Subdomain: Affinity:&Affinity{NodeAffinity:&NodeAffinity{RequiredDuringSchedulingIgnoredDuringExecution:&NodeSelector{NodeSelectorTerms:[]NodeSelectorTerm{NodeSelectorTerm{MatchExpressions:[]NodeSelectorRequirement{},MatchFields:[]NodeSelectorRequirement{NodeSelectorRequirement{Key:metadata.name,Operator:In,Values:[capz-thn7-zhv6p],},},},},},PreferredDuringSchedulingIgnoredDuringExecution:[]PreferredSchedulingTerm{},},PodAffinity:nil,PodAntiAffinity:nil,} SchedulerName:default-scheduler Tolerations:[{Key:node.kubernetes.io/os Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>} {Key:node.kubernetes.io/not-ready Operator:Exists Value: Effect:NoExecute TolerationSeconds:<nil>} {Key:node.kubernetes.io/unreachable Operator:Exists Value: Effect:NoExecute TolerationSeconds:<nil>} {Key:node.kubernetes.io/disk-pressure Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>} {Key:node.kubernetes.io/memory-pressure Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>} {Key:node.kubernetes.io/pid-pressure Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>} {Key:node.kubernetes.io/unschedulable Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>} {Key:node.kubernetes.io/network-unavailable Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>}] HostAliases:[] PriorityClassName:system-node-critical Priority:0xc00151283c DNSConfig:nil ReadinessGates:[] RuntimeClassName:<nil> EnableServiceLinks:0xc001512840 PreemptionPolicy:0xc00139d8e0 Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN:<nil> OS:nil HostUsers:<nil> SchedulingGates:[] ResourceClaims:[]}. With volumes: [{Name:init-script VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:&ConfigMapVolumeSource{LocalObjectReference:LocalObjectReference{Name:csi-azuredisk-node-win-init,},Items:[]KeyToPath{KeyToPath{Key:Initialize-Pod.ps1,Path:Initialize-Pod.ps1,Mode:nil,},},DefaultMode:*420,Optional:nil,} VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:csi-azuredisk-node-win-config VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:&ConfigMapVolumeSource{LocalObjectReference:LocalObjectReference{Name:csi-azuredisk-node-win-config,},Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,} VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:kube-api-access-s6wtk VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:&ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,} PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}]" 
I0114 00:40:04.718016       1 shared_state.go:464]  "msg"="Pod csi-azuredisk-node-win-rw959: Skipping Volume {init-script {nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil &ConfigMapVolumeSource{LocalObjectReference:LocalObjectReference{Name:csi-azuredisk-node-win-init,},Items:[]KeyToPath{KeyToPath{Key:Initialize-Pod.ps1,Path:Initialize-Pod.ps1,Mode:nil,},},DefaultMode:*420,Optional:nil,} nil nil nil nil nil nil nil nil nil nil}}. No persistent volume exists." 
I0114 00:40:04.718116       1 shared_state.go:464]  "msg"="Pod csi-azuredisk-node-win-rw959: Skipping Volume {csi-azuredisk-node-win-config {nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil &ConfigMapVolumeSource{LocalObjectReference:LocalObjectReference{Name:csi-azuredisk-node-win-config,},Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,} nil nil nil nil nil nil nil nil nil nil}}. No persistent volume exists." 
I0114 00:40:04.718266       1 shared_state.go:464]  "msg"="Pod csi-azuredisk-node-win-rw959: Skipping Volume {kube-api-access-s6wtk {nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil &ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,} nil nil nil nil nil}}. No persistent volume exists." 
I0114 00:40:04.718347       1 shared_state.go:490]  "msg"="Storing pod csi-azuredisk-node-win-rw959 and claim [] to podToClaimsMap map." 
I0114 00:40:04.718266       1 shared_state.go:426]  "msg"="Pod spec of pod csi-azuredisk-controller-7f6f8d54b5-f5wbz is: {Volumes:[{Name:socket-dir VolumeSource:{HostPath:nil EmptyDir:&EmptyDirVolumeSource{Medium:,SizeLimit:<nil>,} GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:azure-cred VolumeSource:{HostPath:&HostPathVolumeSource{Path:/etc/kubernetes/,Type:*DirectoryOrCreate,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:csi-azuredisk-controller-config VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:&ConfigMapVolumeSource{LocalObjectReference:LocalObjectReference{Name:csi-azuredisk-controller-config,},Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,} VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:kube-api-access-qlxwh VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:&ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,} PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:csi-provisioner Image:mcr.microsoft.com/oss/kubernetes-csi/csi-provisioner:v3.2.0 Command:[] Args:[--feature-gates=Topology=true --csi-address=$(ADDRESS) --v=2 --timeout=30s --leader-election --leader-election-namespace=kube-system --worker-threads=100 --extra-create-metadata=true --strict-topology=true --kube-api-qps=50 --kube-api-burst=100] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:ADDRESS Value:/csi/csi.sock ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:524288000 scale:0} d:{Dec:<nil>} s:500Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}] Claims:[]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-qlxwh ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:csi-attacher Image:mcr.microsoft.com/oss/kubernetes-csi/csi-attacher:v3.5.0 Command:[] Args:[-v=2 -csi-address=$(ADDRESS) -timeout=600s -leader-election --leader-election-namespace=kube-system -worker-threads=500 -kube-api-qps=50 -kube-api-burst=100] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:ADDRESS Value:/csi/csi.sock ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:524288000 scale:0} d:{Dec:<nil>} s:500Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}] Claims:[]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-qlxwh ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:csi-snapshotter Image:mcr.microsoft.com/oss/kubernetes-csi/csi-snapshotter:v5.0.1 Command:[] Args:[-csi-address=$(ADDRESS) -leader-election --leader-election-namespace=kube-system -v=2] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:ADDRESS Value:/csi/csi.sock ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:104857600 scale:0} d:{Dec:<nil>} s:100Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}] Claims:[]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-qlxwh ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:csi-resizer Image:mcr.microsoft.com/oss/kubernetes-csi/csi-resizer:v1.5.0 Command:[] Args:[-csi-address=$(ADDRESS) -v=2 -leader-election --leader-election-namespace=kube-system -handle-volume-inuse-error=false -feature-gates=RecoverVolumeExpansionFailure=true -timeout=240s] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:ADDRESS Value:/csi/csi.sock ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:524288000 scale:0} d:{Dec:<nil>} s:500Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}] Claims:[]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-qlxwh ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:liveness-probe Image:mcr.microsoft.com/oss/kubernetes-csi/livenessprobe:v2.7.0 Command:[] Args:[--csi-address=/csi/csi.sock --probe-timeout=3s --health-port=29602 --v=2] WorkingDir: Ports:[] EnvFrom:[] Env:[] Resources:{Limits:map[memory:{i:{value:104857600 scale:0} d:{Dec:<nil>} s:100Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}] Claims:[]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-qlxwh ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:azuredisk Image:capzci.azurecr.io/azuredisk-csi:latest-v2-9ef068a8cb36a997d4ea04b90c05c6f92a488a19 Command:[] Args:[--v=6 --config=/etc/csi-azuredisk-controller/config.yaml] WorkingDir: Ports:[{Name:healthz HostPort:29602 ContainerPort:29602 Protocol:TCP HostIP:} {Name:metrics HostPort:29604 ContainerPort:29604 Protocol:TCP HostIP:}] EnvFrom:[] Env:[{Name:AZURE_CREDENTIAL_FILE Value: ValueFrom:&EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:&ConfigMapKeySelector{LocalObjectReference:LocalObjectReference{Name:azure-cred-file,},Key:path,Optional:*true,},SecretKeyRef:nil,}} {Name:CSI_ENDPOINT Value:unix:///csi/csi.sock ValueFrom:nil} {Name:AZURE_GO_SDK_LOG_LEVEL Value: ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:524288000 scale:0} d:{Dec:<nil>} s:500Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}] Claims:[]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:azure-cred ReadOnly:false MountPath:/etc/kubernetes/ SubPath: MountPropagation:<nil> SubPathExpr:} {Name:csi-azuredisk-controller-config ReadOnly:false MountPath:/etc/csi-azuredisk-controller SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-qlxwh ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/healthz,Port:{1 0 healthz},Host:,Scheme:HTTP,HTTPHeaders:[]HTTPHeader{},},TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:30,TimeoutSeconds:10,PeriodSeconds:30,SuccessThreshold:1,FailureThreshold:5,TerminationGracePeriodSeconds:nil,} ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:Always SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy:Always TerminationGracePeriodSeconds:0xc0014f6360 ActiveDeadlineSeconds:<nil> DNSPolicy:ClusterFirst NodeSelector:map[kubernetes.io/os:linux] ServiceAccountName:csi-azuredisk-controller-sa DeprecatedServiceAccount:csi-azuredisk-controller-sa AutomountServiceAccountToken:<nil> NodeName:capz-thn7c4-control-plane-4lzvn HostNetwork:true HostPID:false HostIPC:false ShareProcessNamespace:<nil> SecurityContext:&PodSecurityContext{SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,SupplementalGroups:[],FSGroup:nil,RunAsGroup:nil,Sysctls:[]Sysctl{},WindowsOptions:nil,FSGroupChangePolicy:nil,SeccompProfile:nil,} ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName:default-scheduler Tolerations:[{Key:node-role.kubernetes.io/master Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>} {Key:node-role.kubernetes.io/controlplane Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>} {Key:node-role.kubernetes.io/control-plane Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>} {Key:node.kubernetes.io/not-ready Operator:Exists Value: Effect:NoExecute TolerationSeconds:0xc0014f6368} {Key:node.kubernetes.io/unreachable Operator:Exists Value: Effect:NoExecute TolerationSeconds:0xc0014f6370}] HostAliases:[] PriorityClassName:system-cluster-critical Priority:0xc0014f6378 DNSConfig:nil ReadinessGates:[] RuntimeClassName:<nil> EnableServiceLinks:0xc0014f637c PreemptionPolicy:0xc0014a7b50 Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN:<nil> OS:nil HostUsers:<nil> SchedulingGates:[] ResourceClaims:[]}. With volumes: [{Name:socket-dir VolumeSource:{HostPath:nil EmptyDir:&EmptyDirVolumeSource{Medium:,SizeLimit:<nil>,} GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:azure-cred VolumeSource:{HostPath:&HostPathVolumeSource{Path:/etc/kubernetes/,Type:*DirectoryOrCreate,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:csi-azuredisk-controller-config VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:&ConfigMapVolumeSource{LocalObjectReference:LocalObjectReference{Name:csi-azuredisk-controller-config,},Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,} VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:kube-api-access-qlxwh VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:&ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,} PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}]" 
I0114 00:40:04.720069       1 shared_state.go:420]  "msg"="Adding pod csi-snapshot-controller-6b69b99bbc-5nfkp to shared map with keyName kube-system/csi-snapshot-controller-6b69b99bbc-5nfkp." 
I0114 00:40:04.720217       1 shared_informer.go:303] caches populated
I0114 00:40:04.720314       1 shared_informer.go:303] caches populated
I0114 00:40:04.720372       1 shared_state.go:426]  "msg"="Pod spec of pod csi-snapshot-controller-6b69b99bbc-5nfkp is: {Volumes:[{Name:kube-api-access-rmjzx VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:&ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,} PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:csi-snapshot-controller Image:mcr.microsoft.com/oss/kubernetes-csi/snapshot-controller:v5.0.1 Command:[] Args:[--v=2 --leader-election=true --leader-election-namespace=kube-system] WorkingDir: Ports:[] EnvFrom:[] Env:[] Resources:{Limits:map[memory:{i:{value:104857600 scale:0} d:{Dec:<nil>} s:100Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}] Claims:[]} VolumeMounts:[{Name:kube-api-access-rmjzx ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy:Always TerminationGracePeriodSeconds:0xc0014f7f38 ActiveDeadlineSeconds:<nil> DNSPolicy:ClusterFirst NodeSelector:map[kubernetes.io/os:linux] ServiceAccountName:csi-snapshot-controller-sa DeprecatedServiceAccount:csi-snapshot-controller-sa AutomountServiceAccountToken:<nil> NodeName:capz-thn7c4-control-plane-4lzvn HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace:<nil> SecurityContext:&PodSecurityContext{SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,SupplementalGroups:[],FSGroup:nil,RunAsGroup:nil,Sysctls:[]Sysctl{},WindowsOptions:nil,FSGroupChangePolicy:nil,SeccompProfile:nil,} ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName:default-scheduler Tolerations:[{Key:node-role.kubernetes.io/master Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>} {Key:node-role.kubernetes.io/controlplane Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>} {Key:node-role.kubernetes.io/control-plane Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>} {Key:node.kubernetes.io/not-ready Operator:Exists Value: Effect:NoExecute TolerationSeconds:0xc0014f7f40} {Key:node.kubernetes.io/unreachable Operator:Exists Value: Effect:NoExecute TolerationSeconds:0xc0014f7f48}] HostAliases:[] PriorityClassName:system-cluster-critical Priority:0xc0014f7f50 DNSConfig:nil ReadinessGates:[] RuntimeClassName:<nil> EnableServiceLinks:0xc0014f7f54 PreemptionPolicy:0xc0015a2a60 Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN:<nil> OS:nil HostUsers:<nil> SchedulingGates:[] ResourceClaims:[]}. With volumes: [{Name:kube-api-access-rmjzx VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:&ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,} PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}]" 
I0114 00:40:04.720450       1 shared_state.go:464]  "msg"="Pod csi-snapshot-controller-6b69b99bbc-5nfkp: Skipping Volume {kube-api-access-rmjzx {nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil &ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,} nil nil nil nil nil}}. No persistent volume exists." 
I0114 00:40:04.720478       1 shared_state.go:490]  "msg"="Storing pod csi-snapshot-controller-6b69b99bbc-5nfkp and claim [] to podToClaimsMap map." 
... skipping 35 lines ...
I0114 00:40:04.721083       1 shared_state.go:426]  "msg"="Pod spec of pod csi-azuredisk-scheduler-extender-65d59fb785-p5k5x is: {Volumes:[{Name:csi-azuredisk-scheduler-extender-config VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:&ConfigMapVolumeSource{LocalObjectReference:LocalObjectReference{Name:csi-azuredisk-scheduler-extender-config,},Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,} VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:kube-api-access-glnsl VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:&ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,} PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:csi-azuredisk-scheduler Image:mcr.microsoft.com/oss/kubernetes/kube-scheduler:v1.21.2 Command:[] Args:[kube-scheduler --config=/azdiskschedulerextender/config.yaml --v=5 --feature-gates CSIInlineVolume=true,LocalStorageCapacityIsolation=true,ServiceNodeExclusion=true] WorkingDir: Ports:[] EnvFrom:[] Env:[] Resources:{Limits:map[] Requests:map[] Claims:[]} VolumeMounts:[{Name:csi-azuredisk-scheduler-extender-config ReadOnly:false MountPath:/azdiskschedulerextender SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-glnsl ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:csi-azuredisk-scheduler-extender Image:capzci.azurecr.io/azdiskschedulerextender-csi:latest-v2-9ef068a8cb36a997d4ea04b90c05c6f92a488a19 Command:[] Args:[--port=8889 --metrics-address=0.0.0.0:29606 --driver-object-namespace=azure-disk-csi --drivername=disk.csi.azure.com --v=5] WorkingDir: Ports:[{Name:extender-port HostPort:8889 ContainerPort:8889 Protocol:TCP HostIP:} {Name:metrics-port HostPort:0 ContainerPort:29606 Protocol:TCP HostIP:}] EnvFrom:[] Env:[] Resources:{Limits:map[cpu:{i:{value:300 scale:-3} d:{Dec:<nil>} s:300m Format:DecimalSI} memory:{i:{value:314572800 scale:0} d:{Dec:<nil>} s:300Mi Format:BinarySI}] Requests:map[cpu:{i:{value:20 scale:-3} d:{Dec:<nil>} s:20m Format:DecimalSI} memory:{i:{value:52428800 scale:0} d:{Dec:<nil>} s:50Mi Format:BinarySI}] Claims:[]} VolumeMounts:[{Name:kube-api-access-glnsl ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/ping,Port:{1 0 extender-port},Host:,Scheme:HTTP,HTTPHeaders:[]HTTPHeader{},},TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:0,TimeoutSeconds:1,PeriodSeconds:5,SuccessThreshold:1,FailureThreshold:2,TerminationGracePeriodSeconds:nil,} ReadinessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/ping,Port:{1 0 extender-port},Host:,Scheme:HTTP,HTTPHeaders:[]HTTPHeader{},},TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:0,TimeoutSeconds:1,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:3,TerminationGracePeriodSeconds:nil,} StartupProbe:&Probe{ProbeHandler:ProbeHandler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/ping,Port:{1 0 extender-port},Host:,Scheme:HTTP,HTTPHeaders:[]HTTPHeader{},},TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:0,TimeoutSeconds:1,PeriodSeconds:3,SuccessThreshold:1,FailureThreshold:20,TerminationGracePeriodSeconds:nil,} Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:Always SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy:Always TerminationGracePeriodSeconds:0xc001513e98 ActiveDeadlineSeconds:<nil> DNSPolicy:ClusterFirst NodeSelector:map[kubernetes.io/os:linux] ServiceAccountName:csi-azuredisk-scheduler-extender-sa DeprecatedServiceAccount:csi-azuredisk-scheduler-extender-sa AutomountServiceAccountToken:<nil> NodeName:capz-thn7c4-control-plane-4lzvn HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace:<nil> SecurityContext:&PodSecurityContext{SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,SupplementalGroups:[],FSGroup:nil,RunAsGroup:nil,Sysctls:[]Sysctl{},WindowsOptions:nil,FSGroupChangePolicy:nil,SeccompProfile:nil,} ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName:default-scheduler Tolerations:[{Key:node-role.kubernetes.io/master Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>} {Key:node-role.kubernetes.io/controlplane Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>} {Key:node.kubernetes.io/not-ready Operator:Exists Value: Effect:NoExecute TolerationSeconds:0xc001513ea0} {Key:node.kubernetes.io/unreachable Operator:Exists Value: Effect:NoExecute TolerationSeconds:0xc001513ea8}] HostAliases:[] PriorityClassName:system-cluster-critical Priority:0xc001513eb0 DNSConfig:nil ReadinessGates:[] RuntimeClassName:<nil> EnableServiceLinks:0xc001513eb4 PreemptionPolicy:0xc001532880 Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN:<nil> OS:nil HostUsers:<nil> SchedulingGates:[] ResourceClaims:[]}. With volumes: [{Name:csi-azuredisk-scheduler-extender-config VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:&ConfigMapVolumeSource{LocalObjectReference:LocalObjectReference{Name:csi-azuredisk-scheduler-extender-config,},Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,} VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:kube-api-access-glnsl VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:&ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,} PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}]" 
I0114 00:40:04.714387       1 azure_disk_utils.go:1048]  "msg"="Getting AzVolumeAttachment list for node (capz-thn7-fl7d5)" 
I0114 00:40:04.720734       1 shared_state.go:420]  "msg"="Adding pod metrics-server-7bdcf69694-s9tcd to shared map with keyName kube-system/metrics-server-7bdcf69694-s9tcd." 
I0114 00:40:04.713254       1 shared_state.go:464]  "msg"="Pod kube-controller-manager-capz-thn7c4-control-plane-4lzvn: Skipping Volume {ca-certs {&HostPathVolumeSource{Path:/etc/ssl/certs,Type:*DirectoryOrCreate,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}}. No persistent volume exists." 
I0114 00:40:04.720551       1 controller.go:228] AzDiskControllerManager "msg"="Starting workers" "controller"="replica" "namespace"="azure-disk-csi" "partition"="csi-azuredisk-controller" "worker count"=100
I0114 00:40:04.708307       1 shared_state.go:420]  "msg"="Adding pod csi-proxy-txfnj to shared map with keyName kube-system/csi-proxy-txfnj." 
I0114 00:40:04.708369       1 node.go:90] AzDiskControllerManager "msg"="Node is now available. Will requeue failed replica creation requests." "controller"="node" "disk.csi.azure.com/node-name"="capz-thn7-zhv6p" "namespace"="azure-disk-csi" "partition"="csi-azuredisk-controller"
I0114 00:40:04.720595       1 shared_informer.go:303] caches populated
I0114 00:40:04.720067       1 shared_state.go:464]  "msg"="Pod csi-azuredisk-controller-7f6f8d54b5-f5wbz: Skipping Volume {socket-dir {nil &EmptyDirVolumeSource{Medium:,SizeLimit:<nil>,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}}. No persistent volume exists." 
I0114 00:40:04.708223       1 shared_state.go:420]  "msg"="Adding pod calico-apiserver-5bd69d647b-vn5fm to shared map with keyName calico-apiserver/calico-apiserver-5bd69d647b-vn5fm." 
I0114 00:40:04.714348       1 shared_state.go:426]  "msg"="Pod spec of pod calico-node-7dcnd is: {Volumes:[{Name:lib-modules VolumeSource:{HostPath:&HostPathVolumeSource{Path:/lib/modules,Type:*,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:xtables-lock VolumeSource:{HostPath:&HostPathVolumeSource{Path:/run/xtables.lock,Type:*FileOrCreate,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:policysync VolumeSource:{HostPath:&HostPathVolumeSource{Path:/var/run/nodeagent,Type:*DirectoryOrCreate,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:tigera-ca-bundle VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:&ConfigMapVolumeSource{LocalObjectReference:LocalObjectReference{Name:tigera-ca-bundle,},Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,} VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:node-certs VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:&SecretVolumeSource{SecretName:node-certs,Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,} NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:var-run-calico VolumeSource:{HostPath:&HostPathVolumeSource{Path:/var/run/calico,Type:*,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:var-lib-calico VolumeSource:{HostPath:&HostPathVolumeSource{Path:/var/lib/calico,Type:*,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:cni-bin-dir VolumeSource:{HostPath:&HostPathVolumeSource{Path:/opt/cni/bin,Type:*,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:cni-net-dir VolumeSource:{HostPath:&HostPathVolumeSource{Path:/etc/cni/net.d,Type:*,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:cni-log-dir VolumeSource:{HostPath:&HostPathVolumeSource{Path:/var/log/calico/cni,Type:*,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:flexvol-driver-host VolumeSource:{HostPath:&HostPathVolumeSource{Path:/usr/libexec/kubernetes/kubelet-plugins/volume/exec/nodeagent~uds,Type:*DirectoryOrCreate,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:kube-api-access-zbq25 VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:&ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,} PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[{Name:flexvol-driver Image:docker.io/calico/pod2daemon-flexvol:v3.25.0 Command:[] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[] Resources:{Limits:map[] Requests:map[] Claims:[]} VolumeMounts:[{Name:flexvol-driver-host ReadOnly:false MountPath:/host/driver SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-zbq25 ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:IfNotPresent SecurityContext:&SecurityContext{Capabilities:nil,Privileged:*true,SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,ReadOnlyRootFilesystem:nil,AllowPrivilegeEscalation:nil,RunAsGroup:nil,ProcMount:nil,WindowsOptions:nil,SeccompProfile:nil,} Stdin:false StdinOnce:false TTY:false} {Name:install-cni Image:docker.io/calico/cni:v3.25.0 Command:[/opt/cni/bin/install] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:CNI_CONF_NAME Value:10-calico.conflist ValueFrom:nil} {Name:SLEEP Value:false ValueFrom:nil} {Name:CNI_NET_DIR Value:/etc/cni/net.d ValueFrom:nil} {Name:CNI_NETWORK_CONFIG Value: ValueFrom:&EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:&ConfigMapKeySelector{LocalObjectReference:LocalObjectReference{Name:cni-config,},Key:config,Optional:nil,},SecretKeyRef:nil,}} {Name:KUBERNETES_SERVICE_HOST Value:10.96.0.1 ValueFrom:nil} {Name:KUBERNETES_SERVICE_PORT Value:443 ValueFrom:nil}] Resources:{Limits:map[] Requests:map[] Claims:[]} VolumeMounts:[{Name:cni-bin-dir ReadOnly:false MountPath:/host/opt/cni/bin SubPath: MountPropagation:<nil> SubPathExpr:} {Name:cni-net-dir ReadOnly:false MountPath:/host/etc/cni/net.d SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-zbq25 ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:IfNotPresent SecurityContext:&SecurityContext{Capabilities:nil,Privileged:*true,SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,ReadOnlyRootFilesystem:nil,AllowPrivilegeEscalation:nil,RunAsGroup:nil,ProcMount:nil,WindowsOptions:nil,SeccompProfile:nil,} Stdin:false StdinOnce:false TTY:false}] Containers:[{Name:calico-node Image:docker.io/calico/node:v3.25.0 Command:[] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:DATASTORE_TYPE Value:kubernetes ValueFrom:nil} {Name:WAIT_FOR_DATASTORE Value:true ValueFrom:nil} {Name:CLUSTER_TYPE Value:k8s,operator ValueFrom:nil} {Name:CALICO_DISABLE_FILE_LOGGING Value:false ValueFrom:nil} {Name:FELIX_DEFAULTENDPOINTTOHOSTACTION Value:ACCEPT ValueFrom:nil} {Name:FELIX_HEALTHENABLED Value:true ValueFrom:nil} {Name:FELIX_HEALTHPORT Value:9099 ValueFrom:nil} {Name:NODENAME Value: ValueFrom:&EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:spec.nodeName,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}} {Name:NAMESPACE Value: ValueFrom:&EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}} {Name:FELIX_TYPHAK8SNAMESPACE Value:calico-system ValueFrom:nil} {Name:FELIX_TYPHAK8SSERVICENAME Value:calico-typha ValueFrom:nil} {Name:FELIX_TYPHACAFILE Value:/etc/pki/tls/certs/tigera-ca-bundle.crt ValueFrom:nil} {Name:FELIX_TYPHACERTFILE Value:/node-certs/tls.crt ValueFrom:nil} {Name:FELIX_TYPHAKEYFILE Value:/node-certs/tls.key ValueFrom:nil} {Name:FIPS_MODE_ENABLED Value:false ValueFrom:nil} {Name:FELIX_TYPHACN Value:typha-server ValueFrom:nil} {Name:CALICO_MANAGE_CNI Value:true ValueFrom:nil} {Name:CALICO_IPV4POOL_CIDR Value:192.168.0.0/16 ValueFrom:nil} {Name:CALICO_IPV4POOL_VXLAN Value:Always ValueFrom:nil} {Name:CALICO_IPV4POOL_BLOCK_SIZE Value:26 ValueFrom:nil} {Name:CALICO_IPV4POOL_NODE_SELECTOR Value:all() ValueFrom:nil} {Name:CALICO_IPV4POOL_DISABLE_BGP_EXPORT Value:false ValueFrom:nil} {Name:FELIX_VXLANMTU Value:1350 ValueFrom:nil} {Name:FELIX_WIREGUARDMTU Value:1350 ValueFrom:nil} {Name:CALICO_NETWORKING_BACKEND Value:vxlan ValueFrom:nil} {Name:IP Value:autodetect ValueFrom:nil} {Name:IP_AUTODETECTION_METHOD Value:first-found ValueFrom:nil} {Name:IP6 Value:none ValueFrom:nil} {Name:FELIX_IPV6SUPPORT Value:false ValueFrom:nil} {Name:KUBERNETES_SERVICE_HOST Value:10.96.0.1 ValueFrom:nil} {Name:KUBERNETES_SERVICE_PORT Value:443 ValueFrom:nil}] Resources:{Limits:map[] Requests:map[] Claims:[]} VolumeMounts:[{Name:lib-modules ReadOnly:true MountPath:/lib/modules SubPath: MountPropagation:<nil> SubPathExpr:} {Name:xtables-lock ReadOnly:false MountPath:/run/xtables.lock SubPath: MountPropagation:<nil> SubPathExpr:} {Name:policysync ReadOnly:false MountPath:/var/run/nodeagent SubPath: MountPropagation:<nil> SubPathExpr:} {Name:tigera-ca-bundle ReadOnly:true MountPath:/etc/pki/tls/certs/ SubPath: MountPropagation:<nil> SubPathExpr:} {Name:node-certs ReadOnly:true MountPath:/node-certs SubPath: MountPropagation:<nil> SubPathExpr:} {Name:var-run-calico ReadOnly:false MountPath:/var/run/calico SubPath: MountPropagation:<nil> SubPathExpr:} {Name:var-lib-calico ReadOnly:false MountPath:/var/lib/calico SubPath: MountPropagation:<nil> SubPathExpr:} {Name:cni-log-dir ReadOnly:false MountPath:/var/log/calico/cni SubPath: MountPropagation:<nil> SubPathExpr:} {Name:cni-net-dir ReadOnly:false MountPath:/host/etc/cni/net.d SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-zbq25 ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/liveness,Port:{0 9099 },Host:localhost,Scheme:HTTP,HTTPHeaders:[]HTTPHeader{},},TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:0,TimeoutSeconds:10,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:3,TerminationGracePeriodSeconds:nil,} ReadinessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[/bin/calico-node -felix-ready],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:0,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:3,TerminationGracePeriodSeconds:nil,} StartupProbe:nil Lifecycle:&Lifecycle{PostStart:nil,PreStop:&LifecycleHandler{Exec:&ExecAction{Command:[/bin/calico-node -shutdown],},HTTPGet:nil,TCPSocket:nil,},} TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:IfNotPresent SecurityContext:&SecurityContext{Capabilities:nil,Privileged:*true,SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,ReadOnlyRootFilesystem:nil,AllowPrivilegeEscalation:nil,RunAsGroup:nil,ProcMount:nil,WindowsOptions:nil,SeccompProfile:nil,} Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy:Always TerminationGracePeriodSeconds:0xc001475448 ActiveDeadlineSeconds:<nil> DNSPolicy:ClusterFirst NodeSelector:map[kubernetes.io/os:linux] ServiceAccountName:calico-node DeprecatedServiceAccount:calico-node AutomountServiceAccountToken:<nil> NodeName:capz-thn7c4-control-plane-4lzvn HostNetwork:true HostPID:false HostIPC:false ShareProcessNamespace:<nil> SecurityContext:&PodSecurityContext{SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,SupplementalGroups:[],FSGroup:nil,RunAsGroup:nil,Sysctls:[]Sysctl{},WindowsOptions:nil,FSGroupChangePolicy:nil,SeccompProfile:nil,} ImagePullSecrets:[] Hostname: Subdomain: Affinity:&Affinity{NodeAffinity:&NodeAffinity{RequiredDuringSchedulingIgnoredDuringExecution:&NodeSelector{NodeSelectorTerms:[]NodeSelectorTerm{NodeSelectorTerm{MatchExpressions:[]NodeSelectorRequirement{},MatchFields:[]NodeSelectorRequirement{NodeSelectorRequirement{Key:metadata.name,Operator:In,Values:[capz-thn7c4-control-plane-4lzvn],},},},},},PreferredDuringSchedulingIgnoredDuringExecution:[]PreferredSchedulingTerm{},},PodAffinity:nil,PodAntiAffinity:nil,} SchedulerName:default-scheduler Tolerations:[{Key:CriticalAddonsOnly Operator:Exists Value: Effect: TolerationSeconds:<nil>} {Key: Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>} {Key: Operator:Exists Value: Effect:NoExecute TolerationSeconds:<nil>} {Key:node.kubernetes.io/not-ready Operator:Exists Value: Effect:NoExecute TolerationSeconds:<nil>} {Key:node.kubernetes.io/unreachable Operator:Exists Value: Effect:NoExecute TolerationSeconds:<nil>} {Key:node.kubernetes.io/disk-pressure Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>} {Key:node.kubernetes.io/memory-pressure Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>} {Key:node.kubernetes.io/pid-pressure Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>} {Key:node.kubernetes.io/unschedulable Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>} {Key:node.kubernetes.io/network-unavailable Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>}] HostAliases:[] PriorityClassName:system-node-critical Priority:0xc001475450 DNSConfig:nil ReadinessGates:[] RuntimeClassName:<nil> EnableServiceLinks:0xc001475454 PreemptionPolicy:0xc001460f00 Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN:<nil> OS:nil HostUsers:<nil> SchedulingGates:[] ResourceClaims:[]}. With volumes: [{Name:lib-modules VolumeSource:{HostPath:&HostPathVolumeSource{Path:/lib/modules,Type:*,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:xtables-lock VolumeSource:{HostPath:&HostPathVolumeSource{Path:/run/xtables.lock,Type:*FileOrCreate,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:policysync VolumeSource:{HostPath:&HostPathVolumeSource{Path:/var/run/nodeagent,Type:*DirectoryOrCreate,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:tigera-ca-bundle VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:&ConfigMapVolumeSource{LocalObjectReference:LocalObjectReference{Name:tigera-ca-bundle,},Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,} VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:node-certs VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:&SecretVolumeSource{SecretName:node-certs,Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,} NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:var-run-calico VolumeSource:{HostPath:&HostPathVolumeSource{Path:/var/run/calico,Type:*,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:var-lib-calico VolumeSource:{HostPath:&HostPathVolumeSource{Path:/var/lib/calico,Type:*,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:cni-bin-dir VolumeSource:{HostPath:&HostPathVolumeSource{Path:/opt/cni/bin,Type:*,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:cni-net-dir VolumeSource:{HostPath:&HostPathVolumeSource{Path:/etc/cni/net.d,Type:*,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:cni-log-dir VolumeSource:{HostPath:&HostPathVolumeSource{Path:/var/log/calico/cni,Type:*,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:flexvol-driver-host VolumeSource:{HostPath:&HostPathVolumeSource{Path:/usr/libexec/kubernetes/kubelet-plugins/volume/exec/nodeagent~uds,Type:*DirectoryOrCreate,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:kube-api-access-zbq25 VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:&ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,} PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}]" 
I0114 00:40:04.714476       1 shared_informer.go:303] caches populated
I0114 00:40:04.720614       1 shared_informer.go:303] caches populated
... skipping 133 lines ...
I0114 00:40:04.729955       1 shared_state.go:314]  "msg"="Getting requested volumes for pod (calico-system/calico-node-windows-96v7h)." "disk.csi.azure.com/request-id"="fcb92af1-93a3-11ed-906c-002248afe19a" "disk.csi/azure.com/pod-name"="calico-system/calico-node-windows-96v7h"
I0114 00:40:04.729965       1 pod.go:99]  "msg"="Pod calico-system/calico-node-windows-96v7h has 0 volumes. Volumes: []" "disk.csi.azure.com/request-id"="fcb92af1-93a3-11ed-906c-002248afe19a" "disk.csi/azure.com/pod-name"="calico-system/calico-node-windows-96v7h"
I0114 00:40:04.729982       1 pod.go:89]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="fcb92af1-93a3-11ed-906c-002248afe19a" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcilePod).createReplicas" "disk.csi/azure.com/pod-name"="calico-system/calico-node-windows-96v7h" "latency"=40298
I0114 00:40:04.730010       1 shared_state.go:464]  "msg"="Pod csi-azuredisk-scheduler-extender-65d59fb785-p5k5x: Skipping Volume {csi-azuredisk-scheduler-extender-config {nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil &ConfigMapVolumeSource{LocalObjectReference:LocalObjectReference{Name:csi-azuredisk-scheduler-extender-config,},Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,} nil nil nil nil nil nil nil nil nil nil}}. No persistent volume exists." 
I0114 00:40:04.730157       1 azure_disk_utils.go:1068]  "msg"="Label selector is: disk.csi.azure.com/node-name=capz-thn7-fl7d5." 
I0114 00:40:04.730173       1 shared_state.go:1745] Added node(capz-thn7-fl7d5) to availableAttachmentsMap with capacity: 8
I0114 00:40:04.730196       1 node.go:90] AzDiskControllerManager "msg"="Node is now available. Will requeue failed replica creation requests." "controller"="node" "disk.csi.azure.com/node-name"="capz-thn7-fl7d5" "namespace"="azure-disk-csi" "partition"="csi-azuredisk-controller"
I0114 00:40:04.730216       1 shared_state.go:1205]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="fcb93582-93a3-11ed-906c-002248afe19a" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*SharedState).tryCreateFailedReplicas" "latency"=3900
I0114 00:40:04.730446       1 shared_state.go:426]  "msg"="Pod spec of pod metrics-server-7bdcf69694-s9tcd is: {Volumes:[{Name:tmp-dir VolumeSource:{HostPath:nil EmptyDir:&EmptyDirVolumeSource{Medium:,SizeLimit:<nil>,} GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:kube-api-access-s8f9z VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:&ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,} PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:metrics-server Image:k8s.gcr.io/metrics-server/metrics-server:v0.5.2 Command:[] Args:[--cert-dir=/tmp --secure-port=4443 --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --kubelet-use-node-status-port --metric-resolution=15s --kubelet-insecure-tls] WorkingDir: Ports:[{Name:https HostPort:0 ContainerPort:4443 Protocol:TCP HostIP:}] EnvFrom:[] Env:[] Resources:{Limits:map[] Requests:map[cpu:{i:{value:100 scale:-3} d:{Dec:<nil>} s:100m Format:DecimalSI} memory:{i:{value:209715200 scale:0} d:{Dec:<nil>} s: Format:BinarySI}] Claims:[]} VolumeMounts:[{Name:tmp-dir ReadOnly:false MountPath:/tmp SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-s8f9z ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/livez,Port:{1 0 https},Host:,Scheme:HTTPS,HTTPHeaders:[]HTTPHeader{},},TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:0,TimeoutSeconds:1,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:3,TerminationGracePeriodSeconds:nil,} ReadinessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/readyz,Port:{1 0 https},Host:,Scheme:HTTPS,HTTPHeaders:[]HTTPHeader{},},TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:20,TimeoutSeconds:1,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:3,TerminationGracePeriodSeconds:nil,} StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:IfNotPresent SecurityContext:&SecurityContext{Capabilities:nil,Privileged:nil,SELinuxOptions:nil,RunAsUser:*1000,RunAsNonRoot:*true,ReadOnlyRootFilesystem:*true,AllowPrivilegeEscalation:nil,RunAsGroup:nil,ProcMount:nil,WindowsOptions:nil,SeccompProfile:nil,} Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy:Always TerminationGracePeriodSeconds:0xc001513e30 ActiveDeadlineSeconds:<nil> DNSPolicy:ClusterFirst NodeSelector:map[kubernetes.io/os:linux] ServiceAccountName:metrics-server DeprecatedServiceAccount:metrics-server AutomountServiceAccountToken:<nil> NodeName:capz-thn7c4-control-plane-4lzvn HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace:<nil> SecurityContext:&PodSecurityContext{SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,SupplementalGroups:[],FSGroup:nil,RunAsGroup:nil,Sysctls:[]Sysctl{},WindowsOptions:nil,FSGroupChangePolicy:nil,SeccompProfile:nil,} ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName:default-scheduler Tolerations:[{Key:node-role.kubernetes.io/master Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>} {Key:node-role.kubernetes.io/control-plane Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>} {Key:node.kubernetes.io/not-ready Operator:Exists Value: Effect:NoExecute TolerationSeconds:0xc001513e38} {Key:node.kubernetes.io/unreachable Operator:Exists Value: Effect:NoExecute TolerationSeconds:0xc001513e40}] HostAliases:[] PriorityClassName:system-cluster-critical Priority:0xc001513e48 DNSConfig:nil ReadinessGates:[] RuntimeClassName:<nil> EnableServiceLinks:0xc001513e4c PreemptionPolicy:0xc001532770 Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN:<nil> OS:nil HostUsers:<nil> SchedulingGates:[] ResourceClaims:[]}. With volumes: [{Name:tmp-dir VolumeSource:{HostPath:nil EmptyDir:&EmptyDirVolumeSource{Medium:,SizeLimit:<nil>,} GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:kube-api-access-s8f9z VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:&ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,} PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}]" 
I0114 00:40:04.730477       1 shared_state.go:464]  "msg"="Pod metrics-server-7bdcf69694-s9tcd: Skipping Volume {tmp-dir {nil &EmptyDirVolumeSource{Medium:,SizeLimit:<nil>,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}}. No persistent volume exists." 
I0114 00:40:04.730509       1 shared_state.go:464]  "msg"="Pod metrics-server-7bdcf69694-s9tcd: Skipping Volume {kube-api-access-s8f9z {nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil &ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,} nil nil nil nil nil}}. No persistent volume exists." 
I0114 00:40:04.730518       1 shared_state.go:490]  "msg"="Storing pod metrics-server-7bdcf69694-s9tcd and claim [] to podToClaimsMap map." 
I0114 00:40:04.730537       1 pod.go:91]  "msg"="Creating replicas for pod kube-system/metrics-server-7bdcf69694-s9tcd." "disk.csi.azure.com/request-id"="fcb9421b-93a3-11ed-906c-002248afe19a" "disk.csi/azure.com/pod-name"="kube-system/metrics-server-7bdcf69694-s9tcd"
... skipping 216 lines ...
I0114 00:40:34.170876       1 utils.go:78] GRPC call: /csi.v1.Controller/ControllerPublishVolume
I0114 00:40:34.170892       1 utils.go:79] GRPC request: {"node_id":"capz-thn7-slzjr","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-thn7c4/providers/Microsoft.Compute/disks/pre-provisioned-read-only"}
I0114 00:40:34.171153       1 azure_disk_utils.go:1068]  "msg"="Label selector is: disk.csi.azure.com/node-name=capz-thn7-slzjr,disk.csi.azure.com/volume-name!=pre-provisioned-read-only." "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="0e458702-93a4-11ed-906c-002248afe19a" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pre-provisioned-read-only"
I0114 00:40:34.176787       1 round_trippers.go:553] POST https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments 201 Created in 5 milliseconds
I0114 00:40:34.177235       1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pre-provisioned-read-only-capz-thn7-slzjr-attachment)
I0114 00:40:34.183031       1 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pre-provisioned-read-only-capz-thn7-slzjr-attachment/status 200 OK in 5 milliseconds
I0114 00:40:34.184365       1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0114 00:40:34.184931       1 azure_disk_utils.go:1103]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="0e458702-93a4-11ed-906c-002248afe19a" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pre-provisioned-read-only" "latency"=7862346 "number_of_net_retry"=0 "number_of_retry"=0
I0114 00:40:34.185139       1 attach_detach.go:192]  "msg"="Attaching volume" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="0e458702-93a4-11ed-906c-002248afe19a" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pre-provisioned-read-only"
I0114 00:40:34.260764       1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 955
I0114 00:40:34.261037       1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=0.075264583 request="disks_get" resource_group="capz-thn7c4" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0114 00:40:34.335407       1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=2.382507877 request="disks_create_or_update" resource_group="capz-thn7c4" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0114 00:40:34.364016       1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 955
... skipping 12 lines ...
I0114 00:40:34.882627       1 round_trippers.go:553] GET https://10.96.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 2 milliseconds
I0114 00:40:34.886411       1 round_trippers.go:553] PUT https://10.96.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 3 milliseconds
I0114 00:40:34.886583       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0114 00:40:35.417478       1 batch.go:224] "cloud-provider-azure: Delayed processing of batch due to start delay" type="batch" operation="attach_disk" key="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e|capz-thn7c4|capz-thn7-slzjr" delay="1s"
I0114 00:40:35.417571       1 azure_controller_standard.go:103] azureDisk - update(capz-thn7c4): vm(capz-thn7-slzjr) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-thn7c4/providers/Microsoft.Compute/disks/pre-provisioned-read-only:AttachDiskOptions{diskName: "pre-provisioned-read-only", lun: 0}])
I0114 00:40:35.425908       1 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pre-provisioned-read-only-capz-thn7-slzjr-attachment/status 200 OK in 7 milliseconds
I0114 00:40:35.426540       1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0114 00:40:35.426593       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="0e458702-93a4-11ed-906c-002248afe19a" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pre-provisioned-read-only" "latency"=1248988017
I0114 00:40:35.426630       1 crdprovisioner.go:727]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="0e458702-93a4-11ed-906c-002248afe19a" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLun" "disk.csi.azure.com/volume-name"="pre-provisioned-read-only" "latency"=1249398214
I0114 00:40:35.426647       1 crdprovisioner.go:486]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="0e458702-93a4-11ed-906c-002248afe19a" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).PublishVolume" "disk.csi.azure.com/volume-name"="pre-provisioned-read-only" "latency"=1255577872
I0114 00:40:35.426676       1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=1.255616971 request="azuredisk_csi_driver_controller_publish_volume" resource_group="capz-thn7c4" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-thn7c4/providers/Microsoft.Compute/disks/pre-provisioned-read-only" node="capz-thn7-slzjr" result_code="succeeded"
I0114 00:40:35.426710       1 utils.go:85] GRPC response: {"publish_context":{"LUN":"0"}}
I0114 00:40:35.426900       1 azure_disk_utils.go:1103]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3.4" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="0e458702-93a4-11ed-906c-002248afe19a" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pre-provisioned-read-only" "latency"=9011042 "number_of_net_retry"=0 "number_of_retry"=0
... skipping 21 lines ...
I0114 00:40:46.173328       1 azure_controller_standard.go:272] updateCache(capz-thn7-slzjr) successfully
I0114 00:40:46.173439       1 cloudprovisioner.go:448]  "msg"="attach operation successful: volume \"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-thn7c4/providers/Microsoft.Compute/disks/pre-provisioned-read-only\" attached to node \"capz-thn7-slzjr\"." "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="0e458702-93a4-11ed-906c-002248afe19a" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pre-provisioned-read-only"
I0114 00:40:46.173553       1 cloudprovisioner.go:440]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="0e458702-93a4-11ed-906c-002248afe19a" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).PublishVolume.func3" "disk.csi.azure.com/volume-name"="pre-provisioned-read-only" "latency"=11757052794
I0114 00:40:46.173643       1 workflow.go:155]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="0e458702-93a4-11ed-906c-002248afe19a" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).PublishVolume" "disk.csi.azure.com/volume-name"="pre-provisioned-read-only" "latency"=11987883706
I0114 00:40:46.173707       1 azure_disk_utils.go:1165]  "msg"="Skip update. No update needed." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).updateVolumeAttachmentWithResult" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="0e458702-93a4-11ed-906c-002248afe19a" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pre-provisioned-read-only"
I0114 00:40:46.173790       1 azure_disk_utils.go:1103]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).updateVolumeAttachmentWithResult" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="0e458702-93a4-11ed-906c-002248afe19a" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pre-provisioned-read-only" "latency"=244498 "number_of_net_retry"=0 "number_of_retry"=0
I0114 00:40:46.184790       1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0114 00:40:46.185290       1 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pre-provisioned-read-only-capz-thn7-slzjr-attachment/status 200 OK in 11 milliseconds
I0114 00:40:46.185530       1 azure_disk_utils.go:1103]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3.4" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="0e458702-93a4-11ed-906c-002248afe19a" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pre-provisioned-read-only" "latency"=11681533 "number_of_net_retry"=0 "number_of_retry"=0
I0114 00:40:46.185584       1 attach_detach.go:198]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="0e458702-93a4-11ed-906c-002248afe19a" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3" "disk.csi.azure.com/volume-name"="pre-provisioned-read-only" "latency"=11999829938
I0114 00:40:46.185603       1 workflow.go:155]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="0e458702-93a4-11ed-906c-002248afe19a" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach" "disk.csi.azure.com/volume-name"="pre-provisioned-read-only" "latency"=12008579778
I0114 00:40:46.185684       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="0f0524f3-93a4-11ed-906c-002248afe19a" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pre-provisioned-read-only" "latency"=10758369970
I0114 00:40:46.185721       1 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).ControllerPublishVolume.func2" "disk.csi.azure.com/request-id"="0f0524f3-93a4-11ed-906c-002248afe19a" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=10758851666
... skipping 477 lines ...
I0114 00:45:08.810518       1 utils.go:78] GRPC call: /csi.v1.Controller/ControllerUnpublishVolume
I0114 00:45:08.810691       1 utils.go:79] GRPC request: {"node_id":"capz-thn7-slzjr","volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-thn7c4/providers/Microsoft.Compute/disks/pre-provisioned-read-only"}
I0114 00:45:08.810913       1 crdprovisioner.go:927]  "msg"="Requesting AzVolumeAttachment (pre-provisioned-read-only-capz-thn7-slzjr-attachment) detachment" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="0e458702-93a4-11ed-906c-002248afe19a" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pre-provisioned-read-only"
I0114 00:45:08.819037       1 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pre-provisioned-read-only-capz-thn7-slzjr-attachment/status 200 OK in 7 milliseconds
I0114 00:45:08.819486       1 azure_disk_utils.go:1103]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).detachVolume" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="0e458702-93a4-11ed-906c-002248afe19a" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pre-provisioned-read-only" "latency"=8439545 "number_of_net_retry"=0 "number_of_retry"=0
I0114 00:45:08.819629       1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pre-provisioned-read-only-capz-thn7-slzjr-attachment)
I0114 00:45:08.820016       1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0114 00:45:08.820303       1 replica.go:157]  "msg"="Garbage collection of AzVolumeAttachments for AzVolume (pre-provisioned-read-only) scheduled in 5m0s." "disk.csi.azure.com/request-id"="b1f9ac1f-93a4-11ed-906c-002248afe19a" "disk.csi.azure.com/volume-name"="pre-provisioned-read-only"
I0114 00:45:08.826527       1 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pre-provisioned-read-only-capz-thn7-slzjr-attachment/status 200 OK in 6 milliseconds
I0114 00:45:08.826839       1 azure_disk_utils.go:1103]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="0e458702-93a4-11ed-906c-002248afe19a" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pre-provisioned-read-only" "latency"=6886056 "number_of_net_retry"=0 "number_of_retry"=0
I0114 00:45:08.827185       1 attach_detach.go:340]  "msg"="Detaching volume" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="0e458702-93a4-11ed-906c-002248afe19a" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pre-provisioned-read-only"
I0114 00:45:08.827395       1 cloudprovisioner.go:481]  "msg"="Trying to detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-thn7c4/providers/Microsoft.Compute/disks/pre-provisioned-read-only from node capz-thn7-slzjr" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="0e458702-93a4-11ed-906c-002248afe19a" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pre-provisioned-read-only"
I0114 00:45:08.827441       1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0114 00:45:09.828035       1 batch.go:224] "cloud-provider-azure: Delayed processing of batch due to start delay" type="batch" operation="detach_disk" key="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e|capz-thn7c4|capz-thn7-slzjr" delay="1s"
I0114 00:45:09.828099       1 azure_controller_common.go:461] azuredisk - trying to detach disks from node capz-thn7-slzjr: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-thn7c4/providers/Microsoft.Compute/disks/pre-provisioned-read-only:pre-provisioned-read-only]
I0114 00:45:09.966326       1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 6811
I0114 00:45:09.966788       1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=0.138633704 request="vm_get" resource_group="capz-thn7c4" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0114 00:45:09.966809       1 azure_controller_standard.go:180] azureDisk - detach disk: name pre-provisioned-read-only uri /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-thn7c4/providers/Microsoft.Compute/disks/pre-provisioned-read-only
I0114 00:45:09.966816       1 azure_controller_standard.go:210] azureDisk - update(capz-thn7c4): vm(capz-thn7-slzjr) - detach disk list(capz-thn7-slzjr)%!(EXTRA map[string]string=map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-thn7c4/providers/Microsoft.Compute/disks/pre-provisioned-read-only:pre-provisioned-read-only])
... skipping 26 lines ...
I0114 00:45:25.540879       1 azure_controller_standard.go:126] DeleteCacheForNode(capz-thn7-slzjr) successfully
I0114 00:45:25.540884       1 azure_controller_standard.go:272] updateCache(capz-thn7-slzjr) successfully
I0114 00:45:25.540890       1 azure_controller_common.go:473] azuredisk - successfully detached disks from node capz-thn7-slzjr: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-thn7c4/providers/Microsoft.Compute/disks/pre-provisioned-read-only:pre-provisioned-read-only]
I0114 00:45:25.540918       1 azure_controller_common.go:434] azureDisk - detach disk(pre-provisioned-read-only, /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-thn7c4/providers/Microsoft.Compute/disks/pre-provisioned-read-only) succeeded
I0114 00:45:25.540974       1 cloudprovisioner.go:491]  "msg"="detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-thn7c4/providers/Microsoft.Compute/disks/pre-provisioned-read-only from node capz-thn7-slzjr successfully" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="0e458702-93a4-11ed-906c-002248afe19a" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pre-provisioned-read-only"
I0114 00:45:25.540998       1 cloudprovisioner.go:471]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="0e458702-93a4-11ed-906c-002248afe19a" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).UnpublishVolume" "disk.csi.azure.com/volume-name"="pre-provisioned-read-only" "latency"=16713618378
I0114 00:45:25.550984       1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0114 00:45:25.551008       1 round_trippers.go:553] DELETE https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pre-provisioned-read-only-capz-thn7-slzjr-attachment 200 OK in 9 milliseconds
I0114 00:45:25.551816       1 attach_detach.go:346]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="0e458702-93a4-11ed-906c-002248afe19a" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach.func3" "disk.csi.azure.com/volume-name"="pre-provisioned-read-only" "latency"=16724444118
I0114 00:45:25.551942       1 workflow.go:155]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="0e458702-93a4-11ed-906c-002248afe19a" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach" "disk.csi.azure.com/volume-name"="pre-provisioned-read-only" "latency"=16732014169
I0114 00:45:25.558232       1 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pre-provisioned-read-only-capz-thn7-slzjr-attachment 200 OK in 6 milliseconds
I0114 00:45:25.559089       1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0114 00:45:25.559180       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForDetach" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="0e458702-93a4-11ed-906c-002248afe19a" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pre-provisioned-read-only" "latency"=16739428327
I0114 00:45:25.559345       1 crdprovisioner.go:938]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).detachVolume" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="0e458702-93a4-11ed-906c-002248afe19a" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForDetach" "disk.csi.azure.com/volume-name"="pre-provisioned-read-only" "latency"=16739599226
I0114 00:45:25.559382       1 crdprovisioner.go:836]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="b1f83cd8-93a4-11ed-906c-002248afe19a" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).UnpublishVolume" "disk.csi.azure.com/volume-name"="pre-provisioned-read-only" "latency"=16748482669
I0114 00:45:25.559454       1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=16.748592368 request="azuredisk_csi_driver_controller_unpublish_volume" resource_group="capz-thn7c4" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-thn7c4/providers/Microsoft.Compute/disks/pre-provisioned-read-only" node="capz-thn7-slzjr" result_code="succeeded"
I0114 00:45:25.559500       1 utils.go:85] GRPC response: {}
I0114 00:45:25.560286       1 azure_disk_utils.go:1103]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).removeFinalizer" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="0e458702-93a4-11ed-906c-002248afe19a" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pre-provisioned-read-only" "latency"=8814850 "number_of_net_retry"=0 "number_of_retry"=0
... skipping 3 lines ...
I0114 00:45:25.571867       1 crdprovisioner.go:840]  "msg"="Volume detach successful" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="bbf5c22d-93a4-11ed-906c-002248afe19a" "disk.csi.azure.com/requested-role"="" "disk.csi.azure.com/volume-name"="pre-provisioned-read-only"
I0114 00:45:25.571933       1 crdprovisioner.go:836]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="" "disk.csi.azure.com/request-id"="bbf5c22d-93a4-11ed-906c-002248afe19a" "disk.csi.azure.com/requested-role"="" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).UnpublishVolume" "disk.csi.azure.com/volume-name"="" "latency"=73999
I0114 00:45:25.571993       1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=0.000175099 request="azuredisk_csi_driver_controller_unpublish_volume" resource_group="capz-thn7c4" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-thn7c4/providers/Microsoft.Compute/disks/pre-provisioned-read-only" node="capz-thn7-slzjr" result_code="succeeded"
I0114 00:45:25.572111       1 utils.go:85] GRPC response: {}
I0114 00:45:25.584467       1 round_trippers.go:553] DELETE https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pre-provisioned-read-only 200 OK in 5 milliseconds
I0114 00:45:25.585114       1 conditionwatcher.go:99] Adding a condition function for azvolume (pre-provisioned-read-only)
I0114 00:45:25.585600       1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0114 00:45:25.592291       1 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pre-provisioned-read-only/status 200 OK in 6 milliseconds
I0114 00:45:25.592495       1 azure_disk_utils.go:1103]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerDelete" "csi.storage.k8s.io/pv/name"="azuredisk-3481-disk.csi.azure.com-preprovisioned-pv-kqrsq" "disk.csi.azure.com/request-id"="bbf7c9b8-93a4-11ed-906c-002248afe19a" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pre-provisioned-read-only" "latency"=7167060 "number_of_net_retry"=0 "number_of_retry"=0
I0114 00:45:25.592569       1 shared_state.go:1038]  "msg"="AzVolumeAttachment clean up requested by azvolume-controller for AzVolume (pre-provisioned-read-only)" "csi.storage.k8s.io/pv/name"="azuredisk-3481-disk.csi.azure.com-preprovisioned-pv-kqrsq" "disk.csi.azure.com/request-id"="bbf7c9b8-93a4-11ed-906c-002248afe19a" "disk.csi.azure.com/volume-name"="pre-provisioned-read-only"
I0114 00:45:25.592688       1 azure_disk_utils.go:1039]  "msg"="Getting AzVolumeAttachment list for volume (pre-provisioned-read-only)" "csi.storage.k8s.io/pv/name"="azuredisk-3481-disk.csi.azure.com-preprovisioned-pv-kqrsq" "disk.csi.azure.com/request-id"="bbf7c9b8-93a4-11ed-906c-002248afe19a" "disk.csi.azure.com/volume-name"="pre-provisioned-read-only"
I0114 00:45:25.592722       1 azure_disk_utils.go:1068]  "msg"="Label selector is: disk.csi.azure.com/volume-name=pre-provisioned-read-only." "csi.storage.k8s.io/pv/name"="azuredisk-3481-disk.csi.azure.com-preprovisioned-pv-kqrsq" "disk.csi.azure.com/request-id"="bbf7c9b8-93a4-11ed-906c-002248afe19a" "disk.csi.azure.com/volume-name"="pre-provisioned-read-only"
I0114 00:45:25.592779       1 shared_state.go:1036]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="azuredisk-3481-disk.csi.azure.com-preprovisioned-pv-kqrsq" "disk.csi.azure.com/request-id"="bbf7c9b8-93a4-11ed-906c-002248afe19a" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*SharedState).cleanUpAzVolumeAttachmentByVolume" "disk.csi.azure.com/volume-name"="pre-provisioned-read-only" "latency"=210698
I0114 00:45:25.592625       1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0114 00:45:25.598731       1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0114 00:45:25.598999       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*SharedState).deletePV.func3" "csi.storage.k8s.io/pv/name"="azuredisk-3481-disk.csi.azure.com-preprovisioned-pv-kqrsq" "disk.csi.azure.com/request-id"="bbf6d6ae-93a4-11ed-906c-002248afe19a" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pre-provisioned-read-only" "latency"=13690524
I0114 00:45:25.599137       1 shared_state.go:627]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="bbf6d6ae-93a4-11ed-906c-002248afe19a" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*SharedState).deletePV.func3" "latency"=14015322
I0114 00:45:25.599253       1 workflow.go:155]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="bbf6d6ae-93a4-11ed-906c-002248afe19a" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*SharedState).deletePV" "latency"=20318486
I0114 00:45:25.599263       1 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pre-provisioned-read-only 200 OK in 6 milliseconds
I0114 00:45:25.599634       1 azure_disk_utils.go:1103]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerDelete.func4" "csi.storage.k8s.io/pv/name"="azuredisk-3481-disk.csi.azure.com-preprovisioned-pv-kqrsq" "disk.csi.azure.com/request-id"="bbf7c9b8-93a4-11ed-906c-002248afe19a" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pre-provisioned-read-only" "latency"=6827962 "number_of_net_retry"=0 "number_of_retry"=0
I0114 00:45:25.599659       1 azvolume.go:257]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="azuredisk-3481-disk.csi.azure.com-preprovisioned-pv-kqrsq" "disk.csi.azure.com/request-id"="bbf7c9b8-93a4-11ed-906c-002248afe19a" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerDelete.func4" "disk.csi.azure.com/volume-name"="pre-provisioned-read-only" "latency"=7131460
... skipping 98 lines ...
I0114 00:45:58.826109       1 utils.go:78] GRPC call: /csi.v1.Controller/ControllerPublishVolume
I0114 00:45:58.826294       1 utils.go:79] GRPC request: {"node_id":"capz-thn7-slzjr","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"cachingmode":"None","skuname":"Premium_LRS"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-thn7c4/providers/Microsoft.Compute/disks/reattach-disk-multiple-nodes"}
I0114 00:45:58.826673       1 azure_disk_utils.go:1068]  "msg"="Label selector is: disk.csi.azure.com/node-name=capz-thn7-slzjr,disk.csi.azure.com/volume-name!=reattach-disk-multiple-nodes." "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="cfc806b1-93a4-11ed-906c-002248afe19a" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="reattach-disk-multiple-nodes"
I0114 00:45:58.836655       1 round_trippers.go:553] POST https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments 201 Created in 9 milliseconds
I0114 00:45:58.837088       1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (reattach-disk-multiple-nodes-capz-thn7-slzjr-attachment)
I0114 00:45:58.843247       1 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/reattach-disk-multiple-nodes-capz-thn7-slzjr-attachment/status 200 OK in 6 milliseconds
I0114 00:45:58.843472       1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0114 00:45:58.843496       1 azure_disk_utils.go:1103]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="cfc806b1-93a4-11ed-906c-002248afe19a" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="reattach-disk-multiple-nodes" "latency"=7300353 "number_of_net_retry"=0 "number_of_retry"=0
I0114 00:45:58.843693       1 attach_detach.go:192]  "msg"="Attaching volume" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="cfc806b1-93a4-11ed-906c-002248afe19a" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="reattach-disk-multiple-nodes"
I0114 00:45:58.872233       1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1131
I0114 00:45:58.872388       1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=0.028583414 request="disks_get" resource_group="capz-thn7c4" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0114 00:45:58.872445       1 cloudprovisioner.go:405]  "msg"="GetDiskLun returned: -1. Initiating attaching volume \"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-thn7c4/providers/Microsoft.Compute/disks/reattach-disk-multiple-nodes\" to node \"capz-thn7-slzjr\"." "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="cfc806b1-93a4-11ed-906c-002248afe19a" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="reattach-disk-multiple-nodes"
I0114 00:45:58.872464       1 cloudprovisioner.go:424]  "msg"="Trying to attach volume \"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-thn7c4/providers/Microsoft.Compute/disks/reattach-disk-multiple-nodes\" to node \"capz-thn7-slzjr\"." "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="cfc806b1-93a4-11ed-906c-002248afe19a" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="reattach-disk-multiple-nodes"
I0114 00:45:58.872502       1 azure_controller_common.go:210] found dangling volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-thn7c4/providers/Microsoft.Compute/disks/reattach-disk-multiple-nodes attached to node capz-thn7-fl7d5, could not be attached to node(capz-thn7-slzjr)
E0114 00:45:58.872527       1 cloudprovisioner.go:446]  "msg"="attach volume \"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-thn7c4/providers/Microsoft.Compute/disks/reattach-disk-multiple-nodes\" to instance \"capz-thn7-slzjr\" failed" "error"="disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-thn7c4/providers/Microsoft.Compute/disks/reattach-disk-multiple-nodes) already attached to node(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-thn7c4/providers/Microsoft.Compute/virtualMachines/capz-thn7-fl7d5), could not be attached to node(capz-thn7-slzjr)" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="cfc806b1-93a4-11ed-906c-002248afe19a" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="reattach-disk-multiple-nodes"
E0114 00:45:58.872577       1 cloudprovisioner.go:440]  "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-thn7c4/providers/Microsoft.Compute/disks/reattach-disk-multiple-nodes) already attached to node(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-thn7c4/providers/Microsoft.Compute/virtualMachines/capz-thn7-fl7d5), could not be attached to node(capz-thn7-slzjr)]" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="cfc806b1-93a4-11ed-906c-002248afe19a" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).PublishVolume.func3" "disk.csi.azure.com/volume-name"="reattach-disk-multiple-nodes" "latency"=51199
E0114 00:45:58.872599       1 cloudprovisioner.go:379]  "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-thn7c4/providers/Microsoft.Compute/disks/reattach-disk-multiple-nodes) already attached to node(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-thn7c4/providers/Microsoft.Compute/virtualMachines/capz-thn7-fl7d5), could not be attached to node(capz-thn7-slzjr)]" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="cfc806b1-93a4-11ed-906c-002248afe19a" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).PublishVolume" "disk.csi.azure.com/volume-name"="reattach-disk-multiple-nodes" "latency"=28841413
I0114 00:45:58.879417       1 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/reattach-disk-multiple-nodes-capz-thn7-slzjr-attachment/status 200 OK in 6 milliseconds
I0114 00:45:58.879692       1 azure_disk_utils.go:1103]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3.4" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="cfc806b1-93a4-11ed-906c-002248afe19a" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="reattach-disk-multiple-nodes" "latency"=7050154 "number_of_net_retry"=0 "number_of_retry"=0
I0114 00:45:58.879731       1 attach_detach.go:244]  "msg"="Dangling attach detected for capz-thn7-fl7d5" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="cfc806b1-93a4-11ed-906c-002248afe19a" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="reattach-disk-multiple-nodes"
I0114 00:45:58.880203       1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0114 00:45:58.880426       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="cfc806b1-93a4-11ed-906c-002248afe19a" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="reattach-disk-multiple-nodes" "latency"=43247420
I0114 00:45:58.880547       1 crdprovisioner.go:727]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="cfc806b1-93a4-11ed-906c-002248afe19a" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLun" "disk.csi.azure.com/volume-name"="reattach-disk-multiple-nodes" "latency"=43470317
I0114 00:45:58.880694       1 crdprovisioner.go:486]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="cfc806b1-93a4-11ed-906c-002248afe19a" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).PublishVolume" "disk.csi.azure.com/volume-name"="reattach-disk-multiple-nodes" "latency"=54092549
I0114 00:45:58.880858       1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=0.054244548 request="azuredisk_csi_driver_controller_publish_volume" resource_group="capz-thn7c4" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-thn7c4/providers/Microsoft.Compute/disks/reattach-disk-multiple-nodes" node="capz-thn7-slzjr" result_code="succeeded"
I0114 00:45:58.881053       1 utils.go:85] GRPC response: {"publish_context":{"LUN":"-1"}}
I0114 00:45:58.881025       1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (reattach-disk-multiple-nodes-capz-thn7-slzjr-attachment)
... skipping 16 lines ...
I0114 00:46:02.638426       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I0114 00:46:02.638445       1 utils.go:79] GRPC request: {}
I0114 00:46:02.638597       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I0114 00:46:03.753510       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 00:46:03.753529       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 00:46:03.753570       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 00:46:03.753750       1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0114 00:46:03.976817       1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync
I0114 00:46:04.196093       1 round_trippers.go:553] GET https://10.96.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 2 milliseconds
I0114 00:46:04.200393       1 round_trippers.go:553] PUT https://10.96.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 3 milliseconds
I0114 00:46:04.200505       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0114 00:46:06.204025       1 round_trippers.go:553] GET https://10.96.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 2 milliseconds
I0114 00:46:06.207950       1 round_trippers.go:553] PUT https://10.96.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 3 milliseconds
... skipping 24 lines ...
I0114 00:46:16.100838       1 cloudprovisioner.go:424]  "msg"="Trying to attach volume \"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-thn7c4/providers/Microsoft.Compute/disks/reattach-disk-multiple-nodes\" to node \"capz-thn7-slzjr\"." "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="cfc806b1-93a4-11ed-906c-002248afe19a" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="reattach-disk-multiple-nodes"
I0114 00:46:16.241365       1 round_trippers.go:553] GET https://10.96.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 2 milliseconds
I0114 00:46:16.245620       1 round_trippers.go:553] PUT https://10.96.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 3 milliseconds
I0114 00:46:16.245844       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0114 00:46:17.101511       1 batch.go:224] "cloud-provider-azure: Delayed processing of batch due to start delay" type="batch" operation="attach_disk" key="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e|capz-thn7c4|capz-thn7-slzjr" delay="1s"
I0114 00:46:17.101660       1 azure_controller_standard.go:103] azureDisk - update(capz-thn7c4): vm(capz-thn7-slzjr) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-thn7c4/providers/Microsoft.Compute/disks/reattach-disk-multiple-nodes:AttachDiskOptions{diskName: "reattach-disk-multiple-nodes", lun: 0}])
I0114 00:46:17.112449       1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0114 00:46:17.112546       1 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/reattach-disk-multiple-nodes-capz-thn7-slzjr-attachment/status 200 OK in 10 milliseconds
I0114 00:46:17.112842       1 azure_disk_utils.go:1103]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3.4" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="cfc806b1-93a4-11ed-906c-002248afe19a" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="reattach-disk-multiple-nodes" "latency"=11062332 "number_of_net_retry"=0 "number_of_retry"=0
I0114 00:46:17.487283       1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=0.385542938 request="vm_updateasync" resource_group="capz-thn7c4" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="attach_disk" error_code=""
I0114 00:46:17.487306       1 azure_controller_standard.go:116] azureDisk - update(capz-thn7c4): vm(capz-thn7-slzjr) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-thn7c4/providers/Microsoft.Compute/disks/reattach-disk-multiple-nodes:AttachDiskOptions{diskName: "reattach-disk-multiple-nodes", lun: 0}]) returned with <nil>
I0114 00:46:17.487358       1 azure_controller_common.go:344] azuredisk - trying to attach disks to node capz-thn7-slzjr: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-thn7c4/providers/Microsoft.Compute/disks/reattach-disk-multiple-nodes:AttachDiskOptions{diskName: "reattach-disk-multiple-nodes", lun: 0}]
I0114 00:46:18.249639       1 round_trippers.go:553] GET https://10.96.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 2 milliseconds
... skipping 10 lines ...
I0114 00:46:22.551928       1 azure_controller_standard.go:272] updateCache(capz-thn7-slzjr) successfully
I0114 00:46:22.551985       1 cloudprovisioner.go:448]  "msg"="attach operation successful: volume \"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-thn7c4/providers/Microsoft.Compute/disks/reattach-disk-multiple-nodes\" attached to node \"capz-thn7-slzjr\"." "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="cfc806b1-93a4-11ed-906c-002248afe19a" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="reattach-disk-multiple-nodes"
I0114 00:46:22.552021       1 cloudprovisioner.go:440]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="cfc806b1-93a4-11ed-906c-002248afe19a" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).PublishVolume.func3" "disk.csi.azure.com/volume-name"="reattach-disk-multiple-nodes" "latency"=6451065019
I0114 00:46:22.552040       1 workflow.go:155]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="cfc806b1-93a4-11ed-906c-002248afe19a" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).PublishVolume" "disk.csi.azure.com/volume-name"="reattach-disk-multiple-nodes" "latency"=6507423473
I0114 00:46:22.560862       1 round_trippers.go:553] PUT https://10.96.0.1:443/apis/storage.k8s.io/v1/volumeattachments/csi-b32e1ede69f23fe070de2b447ef1f2413cc85545b2abe64fa6c5ccc30692e08e/status 200 OK in 8 milliseconds
I0114 00:46:22.561142       1 azure_disk_utils.go:1103]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).updateVolumeAttachmentWithResult" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="cfc806b1-93a4-11ed-906c-002248afe19a" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="reattach-disk-multiple-nodes" "latency"=9044748 "number_of_net_retry"=0 "number_of_retry"=0
I0114 00:46:22.567056       1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0114 00:46:22.567549       1 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/reattach-disk-multiple-nodes-capz-thn7-slzjr-attachment/status 200 OK in 6 milliseconds
I0114 00:46:22.567721       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="cfd05474-93a4-11ed-906c-002248afe19a" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="reattach-disk-multiple-nodes" "latency"=23686231170
I0114 00:46:22.567875       1 azure_disk_utils.go:1103]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3.4" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="cfc806b1-93a4-11ed-906c-002248afe19a" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="reattach-disk-multiple-nodes" "latency"=6694261 "number_of_net_retry"=0 "number_of_retry"=0
E0114 00:46:22.567923       1 attach_detach.go:198]  "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-thn7c4/providers/Microsoft.Compute/disks/reattach-disk-multiple-nodes) already attached to node(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-thn7c4/providers/Microsoft.Compute/virtualMachines/capz-thn7-fl7d5), could not be attached to node(capz-thn7-slzjr)]" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="cfc806b1-93a4-11ed-906c-002248afe19a" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3" "disk.csi.azure.com/volume-name"="reattach-disk-multiple-nodes" "latency"=23724159224
E0114 00:46:22.567956       1 workflow.go:155]  "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-thn7c4/providers/Microsoft.Compute/disks/reattach-disk-multiple-nodes) already attached to node(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-thn7c4/providers/Microsoft.Compute/virtualMachines/capz-thn7-fl7d5), could not be attached to node(capz-thn7-slzjr)]" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="cfc806b1-93a4-11ed-906c-002248afe19a" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach" "disk.csi.azure.com/volume-name"="reattach-disk-multiple-nodes" "latency"=23731785775
I0114 00:46:22.568021       1 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).ControllerPublishVolume.func2" "disk.csi.azure.com/request-id"="cfd05474-93a4-11ed-906c-002248afe19a" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=23686747566
I0114 00:46:22.568053       1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=23.741490411 request="azuredisk_csi_driver_attach_volume_latency" resource_group="capz-thn7c4" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-thn7c4/providers/Microsoft.Compute/disks/reattach-disk-multiple-nodes" node="capz-thn7-slzjr" result_code="succeeded"
I0114 00:46:22.568073       1 controllerserver_v2.go:264]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="cfd05474-93a4-11ed-906c-002248afe19a" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).ControllerPublishVolume.func2" "latency"=23687075564
I0114 00:46:24.274037       1 round_trippers.go:553] GET https://10.96.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 2 milliseconds
I0114 00:46:24.279093       1 round_trippers.go:553] PUT https://10.96.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 4 milliseconds
I0114 00:46:24.279325       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
... skipping 22334 lines ...
I0114 02:30:49.661259       1 azure_controller_standard.go:272] updateCache(capz-thn7-slzjr) successfully
I0114 02:30:49.661327       1 cloudprovisioner.go:448]  "msg"="attach operation successful: volume \"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-thn7c4/providers/Microsoft.Compute/disks/pvc-9ca63fc4-8fdd-47f0-a8be-d19a05379e5d\" attached to node \"capz-thn7-slzjr\"." "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="6e8c8150-93b3-11ed-906c-002248afe19a" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-9ca63fc4-8fdd-47f0-a8be-d19a05379e5d"
I0114 02:30:49.661421       1 cloudprovisioner.go:440]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="6e8c8150-93b3-11ed-906c-002248afe19a" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).PublishVolume.func3" "disk.csi.azure.com/volume-name"="pvc-9ca63fc4-8fdd-47f0-a8be-d19a05379e5d" "latency"=11429862567
I0114 02:30:49.661470       1 workflow.go:155]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="6e8c8150-93b3-11ed-906c-002248afe19a" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).PublishVolume" "disk.csi.azure.com/volume-name"="pvc-9ca63fc4-8fdd-47f0-a8be-d19a05379e5d" "latency"=11499329618
I0114 02:30:49.661606       1 azure_disk_utils.go:1165]  "msg"="Skip update. No update needed." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).updateVolumeAttachmentWithResult" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="6e8c8150-93b3-11ed-906c-002248afe19a" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-9ca63fc4-8fdd-47f0-a8be-d19a05379e5d"
I0114 02:30:49.661688       1 azure_disk_utils.go:1103]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).updateVolumeAttachmentWithResult" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="6e8c8150-93b3-11ed-906c-002248afe19a" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-9ca63fc4-8fdd-47f0-a8be-d19a05379e5d" "latency"=273798 "number_of_net_retry"=0 "number_of_retry"=0
I0114 02:30:49.669173       1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0114 02:30:49.669224       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="6f335e5f-93b3-11ed-906c-002248afe19a" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-9ca63fc4-8fdd-47f0-a8be-d19a05379e5d" "latency"=10427125436
I0114 02:30:49.669360       1 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-9ca63fc4-8fdd-47f0-a8be-d19a05379e5d-capz-thn7-slzjr-attachment/status 200 OK in 7 milliseconds
I0114 02:30:49.669382       1 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).ControllerPublishVolume.func2" "disk.csi.azure.com/request-id"="6f335e5f-93b3-11ed-906c-002248afe19a" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=10427366535
I0114 02:30:49.669524       1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=11.521072069 request="azuredisk_csi_driver_attach_volume_latency" resource_group="capz-thn7c4" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-thn7c4/providers/Microsoft.Compute/disks/pvc-9ca63fc4-8fdd-47f0-a8be-d19a05379e5d" node="capz-thn7-slzjr" result_code="succeeded"
I0114 02:30:49.669590       1 controllerserver_v2.go:264]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="6f335e5f-93b3-11ed-906c-002248afe19a" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).ControllerPublishVolume.func2" "latency"=10427542433
I0114 02:30:49.669818       1 azure_disk_utils.go:1103]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3.4" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="6e8c8150-93b3-11ed-906c-002248afe19a" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-9ca63fc4-8fdd-47f0-a8be-d19a05379e5d" "latency"=8058140 "number_of_net_retry"=0 "number_of_retry"=0
I0114 02:30:49.669885       1 attach_detach.go:198]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="6e8c8150-93b3-11ed-906c-002248afe19a" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3" "disk.csi.azure.com/volume-name"="pvc-9ca63fc4-8fdd-47f0-a8be-d19a05379e5d" "latency"=11507758055
I0114 02:30:49.669953       1 workflow.go:155]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="6e8c8150-93b3-11ed-906c-002248afe19a" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach" "disk.csi.azure.com/volume-name"="pvc-9ca63fc4-8fdd-47f0-a8be-d19a05379e5d" "latency"=11514416811
I0114 02:30:49.670037       1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0114 02:30:49.670233       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="6f344408-93b3-11ed-906c-002248afe19a" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-9ca63fc4-8fdd-47f0-a8be-d19a05379e5d" "latency"=10422109167
I0114 02:30:49.670361       1 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).ControllerPublishVolume.func2" "disk.csi.azure.com/request-id"="6f344408-93b3-11ed-906c-002248afe19a" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=10422451265
I0114 02:30:49.670431       1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=10.422938262 request="azuredisk_csi_driver_attach_volume_latency" resource_group="capz-thn7c4" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-thn7c4/providers/Microsoft.Compute/disks/pvc-9ca63fc4-8fdd-47f0-a8be-d19a05379e5d" node="capz-thn7-slzjr" result_code="succeeded"
I0114 02:30:49.670453       1 controllerserver_v2.go:264]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="6f344408-93b3-11ed-906c-002248afe19a" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).ControllerPublishVolume.func2" "latency"=10422569764
I0114 02:30:50.669024       1 round_trippers.go:553] GET https://10.96.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 2 milliseconds
I0114 02:30:50.674057       1 round_trippers.go:553] PUT https://10.96.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 4 milliseconds
... skipping 131 lines ...
I0114 00:40:24.192390    2828 round_trippers.go:553] GET https://10.96.0.1:443/apis/apiextensions.k8s.io/v1/customresourcedefinitions?limit=500&resourceVersion=0 200 OK in 40 milliseconds
I0114 00:40:24.240588    2828 round_trippers.go:553] GET https://10.96.0.1:443/apis/apiextensions.k8s.io/v1/customresourcedefinitions?allowWatchBookmarks=true&resourceVersion=3845&timeout=9m31s&timeoutSeconds=571&watch=true 200 OK in 1 milliseconds
I0114 00:40:24.246557    2828 shared_informer.go:303] caches populated
I0114 00:40:24.246557    2828 azuredisk_v2.go:225] driver userAgent: disk.csi.azure.com/latest-v2-9ef068a8cb36a997d4ea04b90c05c6f92a488a19 e2e-test
I0114 00:40:24.246557    2828 azure_disk_utils.go:564] reading cloud config from secret kube-system/azure-cloud-provider
I0114 00:40:24.250857    2828 round_trippers.go:553] GET https://10.96.0.1:443/api/v1/namespaces/kube-system/secrets/azure-cloud-provider 404 Not Found in 2 milliseconds
I0114 00:40:24.251620    2828 azure_disk_utils.go:571] InitializeCloudFromSecret: failed to get cloud config from secret kube-system/azure-cloud-provider: failed to get secret kube-system/azure-cloud-provider: secrets "azure-cloud-provider" not found
I0114 00:40:24.251620    2828 azure_disk_utils.go:576] could not read cloud config from secret kube-system/azure-cloud-provider
I0114 00:40:24.251620    2828 azure_disk_utils.go:586] use default AZURE_CREDENTIAL_FILE env var: C:\k\azure.json
I0114 00:40:24.251620    2828 azure_disk_utils.go:594] read cloud config from file: C:\k\azure.json successfully
I0114 00:40:24.253695    2828 azure_auth.go:253] Using AzurePublicCloud environment
I0114 00:40:24.254389    2828 azure_auth.go:138] azure: using client_id+client_secret to retrieve access token
I0114 00:40:24.257722    2828 azure_diskclient.go:68] Azure DisksClient using API version: 2022-03-02
... skipping 33 lines ...
I0114 00:40:24.999750    2828 utils.go:85] GRPC response: {"name":"disk.csi.azure.com","vendor_version":"latest-v2-9ef068a8cb36a997d4ea04b90c05c6f92a488a19"}
I0114 00:40:25.150319    2828 utils.go:78] GRPC call: /csi.v1.Identity/GetPluginInfo
I0114 00:40:25.150319    2828 utils.go:79] GRPC request: {}
I0114 00:40:25.150319    2828 utils.go:85] GRPC response: {"name":"disk.csi.azure.com","vendor_version":"latest-v2-9ef068a8cb36a997d4ea04b90c05c6f92a488a19"}
I0114 00:40:25.571167    2828 utils.go:78] GRPC call: /csi.v1.Node/NodeGetInfo
I0114 00:40:25.571214    2828 utils.go:79] GRPC request: {}
I0114 00:40:25.596632    2828 azure_instance_metadata.go:140] Warning: failed to get loadbalancer metadata: failure of getting loadbalancer metadata with response "404 Not Found"
I0114 00:40:25.596632    2828 nodeserver_v2.go:438] NodeGetInfo: nodeName(capz-thn7-slzjr), VM Size(Standard_D4s_v3)
I0114 00:40:25.596632    2828 nodeserver.go:409] got a matching size in getMaxDataDiskCount, VM Size: STANDARD_D4S_V3, MaxDataDiskCount: 8
I0114 00:40:25.596632    2828 utils.go:85] GRPC response: {"accessible_topology":{"segments":{"topology.disk.csi.azure.com/zone":""}},"max_volumes_per_node":8,"node_id":"capz-thn7-slzjr"}
I0114 00:40:43.476125    2828 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0114 00:40:43.476125    2828 utils.go:79] GRPC request: {}
I0114 00:40:43.476125    2828 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
... skipping 3 lines ...
I0114 00:40:43.488501    2828 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0114 00:40:43.488501    2828 utils.go:79] GRPC request: {}
I0114 00:40:43.488501    2828 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I0114 00:40:43.491739    2828 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0114 00:40:43.491790    2828 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\azuredisk-3481-disk.csi.azure.com-preprovisioned-pv-kqrsq\\globalmount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-thn7c4/providers/Microsoft.Compute/disks/pre-provisioned-read-only"}
I0114 00:40:43.492882    2828 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pre-provisioned-read-only-capz-thn7-slzjr-attachment)
I0114 00:40:46.160084    2828 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0114 00:40:46.160084    2828 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="13d3d857-93a4-11ed-9dd5-6045bd608314" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pre-provisioned-read-only" "latency"=2667086300
I0114 00:40:46.160698    2828 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="13d3d857-93a4-11ed-9dd5-6045bd608314" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=2668218400
I0114 00:40:54.209983    2828 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 00:40:54.210039    2828 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 00:40:54.210039    2828 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 00:40:54.251186    2828 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync
... skipping 180 lines ...
I0114 00:46:01.360269    2828 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I0114 00:46:01.360269    2828 utils.go:79] GRPC request: {}
I0114 00:46:01.360269    2828 utils.go:85] GRPC response: {"ready":{"value":true}}
I0114 00:46:15.342190    2828 azuredisk_v2.go:578]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-14T00:46:15Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="d9a01c4f-93a4-11ed-9dd5-6045bd608314"
I0114 00:46:15.353524    2828 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-thn7-slzjr/status 200 OK in 9 milliseconds
I0114 00:46:15.353617    2828 azuredisk_v2.go:565]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="d9a01c4f-93a4-11ed-9dd5-6045bd608314" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=11426200
I0114 00:46:17.091201    2828 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0114 00:46:22.547830    2828 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0114 00:46:22.548043    2828 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="d069fa48-93a4-11ed-9dd5-6045bd608314" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="reattach-disk-multiple-nodes" "latency"=22658929000
I0114 00:46:22.548183    2828 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="d069fa48-93a4-11ed-9dd5-6045bd608314" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=22659141300
I0114 00:46:24.328975    2828 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 00:46:24.369893    2828 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 00:46:24.369931    2828 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 00:46:24.376168    2828 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync
... skipping 75 lines ...
I0114 00:48:02.409627    2828 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0114 00:48:02.409627    2828 utils.go:79] GRPC request: {}
I0114 00:48:02.409726    2828 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I0114 00:48:02.410744    2828 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0114 00:48:02.410744    2828 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-0f97f108-fafd-4633-a41a-df2b21087ede\\globalmount","volume_capability":{"AccessType":{"Mount":{"mount_flags":["barrier=1","acl"]}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-0f97f108-fafd-4633-a41a-df2b21087ede","csi.storage.k8s.io/pvc/name":"pvc-tb5dz","csi.storage.k8s.io/pvc/namespace":"azuredisk-8406","requestedsizegib":"10","skuname":"Standard_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1673656805162-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-thn7c4/providers/Microsoft.Compute/disks/pvc-0f97f108-fafd-4633-a41a-df2b21087ede"}
I0114 00:48:02.411355    2828 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-0f97f108-fafd-4633-a41a-df2b21087ede-capz-thn7-slzjr-attachment)
I0114 00:48:11.287563    2828 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0114 00:48:11.287563    2828 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="19719192-93a5-11ed-9dd5-6045bd608314" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-0f97f108-fafd-4633-a41a-df2b21087ede" "latency"=8876100500
I0114 00:48:11.287563    2828 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="19719192-93a5-11ed-9dd5-6045bd608314" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=8876100500
I0114 00:48:15.345800    2828 azuredisk_v2.go:578]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-14T00:48:15Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="212730a0-93a5-11ed-9dd5-6045bd608314"
I0114 00:48:15.357187    2828 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-thn7-slzjr/status 200 OK in 10 milliseconds
I0114 00:48:15.357187    2828 azuredisk_v2.go:565]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="212730a0-93a5-11ed-9dd5-6045bd608314" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=11506800
I0114 00:48:16.073364    2828 nodeserver_v2.go:211] NodeStageVolume: formatting 10 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-0f97f108-fafd-4633-a41a-df2b21087ede\globalmount with mount options([barrier=1 acl])
... skipping 74 lines ...
I0114 00:49:45.350075    2828 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-thn7-slzjr/status 200 OK in 9 milliseconds
I0114 00:49:45.350075    2828 azuredisk_v2.go:565]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="56cb55f8-93a5-11ed-9dd5-6045bd608314" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=9403800
I0114 00:49:54.388411    2828 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 00:49:54.461298    2828 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 00:49:54.461298    2828 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 00:49:54.461298    2828 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync
I0114 00:49:54.461752    2828 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0114 00:49:54.549735    2828 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 00:49:54.848946    2828 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0114 00:49:54.849670    2828 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="569d361f-93a5-11ed-9dd5-6045bd608314" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-f82bd6a5-9f53-4b20-a79a-b284b52473b6" "latency"=9810735000
I0114 00:49:54.849670    2828 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="569d361f-93a5-11ed-9dd5-6045bd608314" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9810878800
I0114 00:49:55.242533    2828 reflector.go:559] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: Watch close - *v1.CustomResourceDefinition total 0 items received
I0114 00:49:55.243814    2828 round_trippers.go:553] GET https://10.96.0.1:443/apis/apiextensions.k8s.io/v1/customresourcedefinitions?allowWatchBookmarks=true&resourceVersion=3845&timeout=7m9s&timeoutSeconds=429&watch=true 200 OK in 0 milliseconds
I0114 00:49:59.893737    2828 nodeserver_v2.go:211] NodeStageVolume: formatting 10 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-f82bd6a5-9f53-4b20-a79a-b284b52473b6\globalmount with mount options([])
I0114 00:50:01.376948    2828 utils.go:78] GRPC call: /csi.v1.Identity/Probe
... skipping 72 lines ...
I0114 00:51:36.013368    2828 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0114 00:51:36.013368    2828 utils.go:79] GRPC request: {}
I0114 00:51:36.013368    2828 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I0114 00:51:36.015725    2828 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0114 00:51:36.015725    2828 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-dd61a03e-1d58-4a05-8002-22175d54d846\\globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-dd61a03e-1d58-4a05-8002-22175d54d846","csi.storage.k8s.io/pvc/name":"pvc-9qcsv","csi.storage.k8s.io/pvc/namespace":"azuredisk-586","enableAsyncAttach":"false","enableBursting":"true","enableasyncattach":"false","perfProfile":"Basic","requestedsizegib":"1024","skuname":"Premium_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1673656805162-8081-disk.csi.azure.com","userAgent":"azuredisk-e2e-test"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-thn7c4/providers/Microsoft.Compute/disks/pvc-dd61a03e-1d58-4a05-8002-22175d54d846"}
I0114 00:51:36.016426    2828 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-dd61a03e-1d58-4a05-8002-22175d54d846-capz-thn7-slzjr-attachment)
I0114 00:51:45.280662    2828 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0114 00:51:45.280662    2828 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="98c31da6-93a5-11ed-9dd5-6045bd608314" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-dd61a03e-1d58-4a05-8002-22175d54d846" "latency"=9263861300
I0114 00:51:45.291002    2828 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="98c31da6-93a5-11ed-9dd5-6045bd608314" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9274146700
I0114 00:51:45.347440    2828 azuredisk_v2.go:578]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-14T00:51:45Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="9e52ea66-93a5-11ed-9dd5-6045bd608314"
I0114 00:51:45.355980    2828 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-thn7-slzjr/status 200 OK in 8 milliseconds
I0114 00:51:45.355980    2828 azuredisk_v2.go:565]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="9e52ea66-93a5-11ed-9dd5-6045bd608314" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=8539200
I0114 00:51:50.178732    2828 nodeserver_v2.go:211] NodeStageVolume: formatting 10 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-dd61a03e-1d58-4a05-8002-22175d54d846\globalmount with mount options([])
... skipping 76 lines ...
I0114 00:53:18.019503    2828 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-f20ea531-8640-4d75-857e-839afa44b163\\globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-f20ea531-8640-4d75-857e-839afa44b163","csi.storage.k8s.io/pvc/name":"pvc-b2jcb","csi.storage.k8s.io/pvc/namespace":"azuredisk-2453","device-setting/queue/nomerges":"0","device-setting/queue/read_ahead_kb":"8","enableAsyncAttach":"false","enableBursting":"true","enableasyncattach":"false","perfProfile":"Advanced","requestedsizegib":"1024","skuname":"Premium_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1673656805162-8081-disk.csi.azure.com","userAgent":"azuredisk-e2e-test"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-thn7c4/providers/Microsoft.Compute/disks/pvc-f20ea531-8640-4d75-857e-839afa44b163"}
I0114 00:53:18.020230    2828 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-f20ea531-8640-4d75-857e-839afa44b163-capz-thn7-slzjr-attachment)
I0114 00:53:24.468276    2828 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 00:53:24.545374    2828 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync
I0114 00:53:24.545374    2828 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 00:53:24.545374    2828 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 00:53:24.545374    2828 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0114 00:53:24.596395    2828 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 00:53:28.660309    2828 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0114 00:53:28.660346    2828 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="d58fa945-93a5-11ed-9dd5-6045bd608314" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-f20ea531-8640-4d75-857e-839afa44b163" "latency"=10639971900
I0114 00:53:28.660346    2828 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="d58fa945-93a5-11ed-9dd5-6045bd608314" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=10639971900
I0114 00:53:31.375076    2828 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I0114 00:53:31.375076    2828 utils.go:79] GRPC request: {}
I0114 00:53:31.375076    2828 utils.go:85] GRPC response: {"ready":{"value":true}}
I0114 00:53:33.783783    2828 nodeserver_v2.go:211] NodeStageVolume: formatting 10 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-f20ea531-8640-4d75-857e-839afa44b163\globalmount with mount options([])
... skipping 39 lines ...
I0114 00:54:14.812874    2828 utils.go:78] GRPC call: /csi.v1.Node/NodeUnpublishVolume
I0114 00:54:14.812874    2828 utils.go:79] GRPC request: {"target_path":"c:\\var\\lib\\kubelet\\pods\\45f05775-9912-4b7f-b54b-b5138265009a\\volumes\\kubernetes.io~csi\\pvc-f20ea531-8640-4d75-857e-839afa44b163\\mount","volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-thn7c4/providers/Microsoft.Compute/disks/pvc-f20ea531-8640-4d75-857e-839afa44b163"}
I0114 00:54:14.815532    2828 nodeserver_v2.go:369] NodeUnpublishVolume: unmounting volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-thn7c4/providers/Microsoft.Compute/disks/pvc-f20ea531-8640-4d75-857e-839afa44b163 on c:\var\lib\kubelet\pods\45f05775-9912-4b7f-b54b-b5138265009a\volumes\kubernetes.io~csi\pvc-f20ea531-8640-4d75-857e-839afa44b163\mount
I0114 00:54:15.360060    2828 azuredisk_v2.go:578]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-14T00:54:15Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="f7bcfd25-93a5-11ed-9dd5-6045bd608314"
I0114 00:54:15.370613    2828 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-thn7-slzjr/status 200 OK in 9 milliseconds
I0114 00:54:15.370613    2828 azuredisk_v2.go:565]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="f7bcfd25-93a5-11ed-9dd5-6045bd608314" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=10788500
W0114 00:54:15.562217    2828 safe_mounter_windows.go:113] GetVolumeIDFromTargetPath(c:\var\lib\kubelet\pods\45f05775-9912-4b7f-b54b-b5138265009a\volumes\kubernetes.io~csi\pvc-f20ea531-8640-4d75-857e-839afa44b163\mount) failed with error: rpc error: code = Unknown desc = error getting the volume for the mount c:\var\lib\kubelet\pods\45f05775-9912-4b7f-b54b-b5138265009a\volumes\kubernetes.io~csi\pvc-f20ea531-8640-4d75-857e-839afa44b163\mount, internal error error getting volume from mount. cmd: (Get-Item -Path Get-Item : Cannot find path 'C:\var\lib\kubelet\pods\45f05775-9912-4b7f-b54b-b5138265009a\volumes\kubernetes.io~csi\pvc
-f20ea531-8640-4d75-857e-839afa44b163\mount' because it does not exist.
At line:1 char:2
+ (Get-Item -Path c:\var\lib\kubelet\pods\45f05775-9912-4b7f-b54b-b5138 ...
+  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : ObjectNotFound: (C:\var\lib\kube...afa44b163\mount:String) [Get-Item], ItemNotFoundExcep 
   tion
... skipping 37 lines ...
+ ... d : PathNotFound,Microsoft.PowerShell.Commands.GetItemCommand).Target
+                                                                  ~
Unexpected token ')' in expression or statement.
    + CategoryInfo          : ParserError: (:) [], ParentContainsErrorRecordException
    + FullyQualifiedErrorId : MissingEndParenthesisInExpression
 
, error: exit status 1, response: <nil>
I0114 00:54:15.563152    2828 nodeserver_v2.go:375] NodeUnpublishVolume: unmount volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-thn7c4/providers/Microsoft.Compute/disks/pvc-f20ea531-8640-4d75-857e-839afa44b163 on c:\var\lib\kubelet\pods\45f05775-9912-4b7f-b54b-b5138265009a\volumes\kubernetes.io~csi\pvc-f20ea531-8640-4d75-857e-839afa44b163\mount successfully
I0114 00:54:15.563152    2828 utils.go:85] GRPC response: {}
I0114 00:54:16.624488    2828 utils.go:78] GRPC call: /csi.v1.Node/NodeUnpublishVolume
I0114 00:54:16.624774    2828 utils.go:79] GRPC request: {"target_path":"c:\\var\\lib\\kubelet\\pods\\45f05775-9912-4b7f-b54b-b5138265009a\\volumes\\kubernetes.io~csi\\pvc-f20ea531-8640-4d75-857e-839afa44b163\\mount","volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-thn7c4/providers/Microsoft.Compute/disks/pvc-f20ea531-8640-4d75-857e-839afa44b163"}
I0114 00:54:16.624945    2828 nodeserver_v2.go:369] NodeUnpublishVolume: unmounting volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-thn7c4/providers/Microsoft.Compute/disks/pvc-f20ea531-8640-4d75-857e-839afa44b163 on c:\var\lib\kubelet\pods\45f05775-9912-4b7f-b54b-b5138265009a\volumes\kubernetes.io~csi\pvc-f20ea531-8640-4d75-857e-839afa44b163\mount
W0114 00:54:17.315679    2828 safe_mounter_windows.go:113] GetVolumeIDFromTargetPath(c:\var\lib\kubelet\pods\45f05775-9912-4b7f-b54b-b5138265009a\volumes\kubernetes.io~csi\pvc-f20ea531-8640-4d75-857e-839afa44b163\mount) failed with error: rpc error: code = Unknown desc = error getting the volume for the mount c:\var\lib\kubelet\pods\45f05775-9912-4b7f-b54b-b5138265009a\volumes\kubernetes.io~csi\pvc-f20ea531-8640-4d75-857e-839afa44b163\mount, internal error error getting volume from mount. cmd: (Get-Item -Path Get-Item : Cannot find path 'C:\var\lib\kubelet\pods\45f05775-9912-4b7f-b54b-b5138265009a\volumes\kubernetes.io~csi\pvc
-f20ea531-8640-4d75-857e-839afa44b163\mount' because it does not exist.
At line:1 char:2
+ (Get-Item -Path c:\var\lib\kubelet\pods\45f05775-9912-4b7f-b54b-b5138 ...
+  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : ObjectNotFound: (C:\var\lib\kube...afa44b163\mount:String) [Get-Item], ItemNotFoundExcep 
   tion
... skipping 37 lines ...
+ ... d : PathNotFound,Microsoft.PowerShell.Commands.GetItemCommand).Target
+                                                                  ~
Unexpected token ')' in expression or statement.
    + CategoryInfo          : ParserError: (:) [], ParentContainsErrorRecordException
    + FullyQualifiedErrorId : MissingEndParenthesisInExpression
 
, error: exit status 1, response: <nil>
I0114 00:54:17.316871    2828 nodeserver_v2.go:375] NodeUnpublishVolume: unmount volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-thn7c4/providers/Microsoft.Compute/disks/pvc-f20ea531-8640-4d75-857e-839afa44b163 on c:\var\lib\kubelet\pods\45f05775-9912-4b7f-b54b-b5138265009a\volumes\kubernetes.io~csi\pvc-f20ea531-8640-4d75-857e-839afa44b163\mount successfully
I0114 00:54:17.316929    2828 utils.go:85] GRPC response: {}
I0114 00:54:17.367901    2828 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0114 00:54:17.367901    2828 utils.go:79] GRPC request: {}
I0114 00:54:17.367901    2828 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I0114 00:54:17.369968    2828 utils.go:78] GRPC call: /csi.v1.Node/NodeUnstageVolume
... skipping 38 lines ...
I0114 00:55:21.427239    2828 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-36583e7e-1b88-40ea-844e-7d41221bd24b\\globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-36583e7e-1b88-40ea-844e-7d41221bd24b","csi.storage.k8s.io/pvc/name":"pvc-qsq8h","csi.storage.k8s.io/pvc/namespace":"azuredisk-7540","requestedsizegib":"10","skuname":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1673656805162-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-thn7c4/providers/Microsoft.Compute/disks/pvc-36583e7e-1b88-40ea-844e-7d41221bd24b"}
I0114 00:55:21.427644    2828 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-36583e7e-1b88-40ea-844e-7d41221bd24b-capz-thn7-slzjr-attachment)
I0114 00:55:24.513298    2828 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 00:55:24.560596    2828 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 00:55:24.560596    2828 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync
I0114 00:55:24.560596    2828 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 00:55:24.560596    2828 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0114 00:55:24.632678    2828 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 00:55:31.037000    2828 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0114 00:55:31.037236    2828 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="1f1e2127-93a6-11ed-9dd5-6045bd608314" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-36583e7e-1b88-40ea-844e-7d41221bd24b" "latency"=9608923700
I0114 00:55:31.037338    2828 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="1f1e2127-93a6-11ed-9dd5-6045bd608314" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9609179900
I0114 00:55:31.388944    2828 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I0114 00:55:31.388944    2828 utils.go:79] GRPC request: {}
I0114 00:55:31.388944    2828 utils.go:85] GRPC response: {"ready":{"value":true}}
I0114 00:55:35.851989    2828 nodeserver_v2.go:211] NodeStageVolume: formatting 10 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-36583e7e-1b88-40ea-844e-7d41221bd24b\globalmount with mount options([])
... skipping 81 lines ...
I0114 00:57:04.726804    2828 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0114 00:57:04.726804    2828 utils.go:79] GRPC request: {}
I0114 00:57:04.726804    2828 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I0114 00:57:04.732119    2828 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0114 00:57:04.732119    2828 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-4dd394df-a069-40e3-8b2f-221d8519cda1\\globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-4dd394df-a069-40e3-8b2f-221d8519cda1","csi.storage.k8s.io/pvc/name":"pvc-6fbtc","csi.storage.k8s.io/pvc/namespace":"azuredisk-7278","requestedsizegib":"10","skuname":"Premium_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1673656805162-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-thn7c4/providers/Microsoft.Compute/disks/pvc-4dd394df-a069-40e3-8b2f-221d8519cda1"}
I0114 00:57:04.732834    2828 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-4dd394df-a069-40e3-8b2f-221d8519cda1-capz-thn7-slzjr-attachment)
I0114 00:57:14.807829    2828 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0114 00:57:14.808359    2828 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="5cb13f15-93a6-11ed-9dd5-6045bd608314" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-4dd394df-a069-40e3-8b2f-221d8519cda1" "latency"=10073604800
I0114 00:57:14.808452    2828 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="5cb13f15-93a6-11ed-9dd5-6045bd608314" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=10074329000
I0114 00:57:15.370057    2828 azuredisk_v2.go:578]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-14T00:57:15Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="63085b4c-93a6-11ed-9dd5-6045bd608314"
I0114 00:57:15.379450    2828 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-thn7-slzjr/status 200 OK in 8 milliseconds
I0114 00:57:15.380464    2828 azuredisk_v2.go:565]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="63085b4c-93a6-11ed-9dd5-6045bd608314" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=10303200
I0114 00:57:19.611794    2828 nodeserver_v2.go:211] NodeStageVolume: formatting 10 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-4dd394df-a069-40e3-8b2f-221d8519cda1\globalmount with mount options([])
... skipping 37 lines ...
I0114 00:57:54.322325    2828 utils.go:79] GRPC request: {"publish_context":{"LUN":"1"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-4547d4ba-0080-4436-a09d-8b61c8646c6f\\globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-4547d4ba-0080-4436-a09d-8b61c8646c6f","csi.storage.k8s.io/pvc/name":"pvc-8bjwx","csi.storage.k8s.io/pvc/namespace":"azuredisk-7278","requestedsizegib":"10","skuname":"Premium_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1673656805162-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-thn7c4/providers/Microsoft.Compute/disks/pvc-4547d4ba-0080-4436-a09d-8b61c8646c6f"}
I0114 00:57:54.322803    2828 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-4547d4ba-0080-4436-a09d-8b61c8646c6f-capz-thn7-slzjr-attachment)
I0114 00:57:54.566818    2828 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 00:57:54.639379    2828 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync
I0114 00:57:54.639379    2828 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 00:57:54.639379    2828 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 00:57:54.639655    2828 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0114 00:57:54.697249    2828 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 00:58:01.392941    2828 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I0114 00:58:01.392941    2828 utils.go:79] GRPC request: {}
I0114 00:58:01.392941    2828 utils.go:85] GRPC response: {"ready":{"value":true}}
I0114 00:58:01.646534    2828 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0114 00:58:01.646534    2828 utils.go:79] GRPC request: {}
I0114 00:58:01.646534    2828 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I0114 00:58:01.648669    2828 utils.go:78] GRPC call: /csi.v1.Node/NodeGetVolumeStats
I0114 00:58:01.648669    2828 utils.go:79] GRPC request: {"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-thn7c4/providers/Microsoft.Compute/disks/pvc-4dd394df-a069-40e3-8b2f-221d8519cda1","volume_path":"c:\\var\\lib\\kubelet\\pods\\b0b29e61-6f15-45e0-8050-6b72c9f45ffe\\volumes\\kubernetes.io~csi\\pvc-4dd394df-a069-40e3-8b2f-221d8519cda1\\mount"}
I0114 00:58:02.183486    2828 safe_mounter_windows.go:356] GetVolumeStats(c:\var\lib\kubelet\pods\b0b29e61-6f15-45e0-8050-6b72c9f45ffe\volumes\kubernetes.io~csi\pvc-4dd394df-a069-40e3-8b2f-221d8519cda1\mount) returned volumeID(\\?\Volume{3dd6ef95-7ff5-4826-8751-234fc75a75f0}\)
I0114 00:58:03.074346    2828 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0114 00:58:03.074346    2828 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="7a4010b4-93a6-11ed-9dd5-6045bd608314" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-4547d4ba-0080-4436-a09d-8b61c8646c6f" "latency"=8750311200
I0114 00:58:03.074346    2828 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="7a4010b4-93a6-11ed-9dd5-6045bd608314" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=8750523500
I0114 00:58:04.869976    2828 utils.go:85] GRPC response: {"usage":[{"available":10677248000,"total":10719588352,"unit":1,"used":42340352}]}
I0114 00:58:07.932405    2828 nodeserver_v2.go:211] NodeStageVolume: formatting 13 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-4547d4ba-0080-4436-a09d-8b61c8646c6f\globalmount with mount options([])
I0114 00:58:15.210581    2828 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: Watch close - *v1beta2.AzDriverNode total 42 items received
I0114 00:58:15.214769    2828 round_trippers.go:553] GET https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/azdrivernodes?allowWatchBookmarks=true&resourceVersion=9564&timeout=6m48s&timeoutSeconds=408&watch=true 200 OK in 3 milliseconds
... skipping 36 lines ...
I0114 00:58:45.120716    2828 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0114 00:58:45.120716    2828 utils.go:79] GRPC request: {"publish_context":{"LUN":"2"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-3e136862-bf51-4962-9cf3-7a3473d2b459\\globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-3e136862-bf51-4962-9cf3-7a3473d2b459","csi.storage.k8s.io/pvc/name":"pvc-9t9wk","csi.storage.k8s.io/pvc/namespace":"azuredisk-7278","requestedsizegib":"10","skuname":"Premium_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1673656805162-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-thn7c4/providers/Microsoft.Compute/disks/pvc-3e136862-bf51-4962-9cf3-7a3473d2b459"}
I0114 00:58:45.121113    2828 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-3e136862-bf51-4962-9cf3-7a3473d2b459-capz-thn7-slzjr-attachment)
I0114 00:58:45.377724    2828 azuredisk_v2.go:578]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-14T00:58:45Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="98ae735b-93a6-11ed-9dd5-6045bd608314"
I0114 00:58:45.384736    2828 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-thn7-slzjr/status 200 OK in 6 milliseconds
I0114 00:58:45.385309    2828 azuredisk_v2.go:565]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="98ae735b-93a6-11ed-9dd5-6045bd608314" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=7584500
I0114 00:58:51.211521    2828 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0114 00:58:51.212448    2828 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="98874785-93a6-11ed-9dd5-6045bd608314" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-3e136862-bf51-4962-9cf3-7a3473d2b459" "latency"=6090181300
I0114 00:58:51.212448    2828 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="98874785-93a6-11ed-9dd5-6045bd608314" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=6091258100
I0114 00:58:54.585402    2828 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 00:58:54.667851    2828 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 00:58:54.667851    2828 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync
I0114 00:58:54.667851    2828 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
... skipping 151 lines ...
I0114 01:01:49.341861    2828 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-e7e93284-927f-4ae5-b370-cb0d2b708a35\\globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-e7e93284-927f-4ae5-b370-cb0d2b708a35","csi.storage.k8s.io/pvc/name":"pvc-vgw5j","csi.storage.k8s.io/pvc/namespace":"azuredisk-3267","requestedsizegib":"10","skuname":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1673656805162-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-thn7c4/providers/Microsoft.Compute/disks/pvc-e7e93284-927f-4ae5-b370-cb0d2b708a35"}
I0114 01:01:49.342388    2828 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-e7e93284-927f-4ae5-b370-cb0d2b708a35-capz-thn7-slzjr-attachment)
I0114 01:01:54.669091    2828 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 01:01:54.710603    2828 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync
I0114 01:01:54.710603    2828 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 01:01:54.710905    2828 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 01:01:54.711005    2828 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0114 01:01:54.774504    2828 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 01:01:58.289125    2828 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: Watch close - *v1beta2.AzDriverNode total 20 items received
I0114 01:01:58.293776    2828 round_trippers.go:553] GET https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dcapz-thn7-slzjr&resourceVersion=10658&timeout=8m5s&timeoutSeconds=485&watch=true 200 OK in 4 milliseconds
I0114 01:02:01.413214    2828 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I0114 01:02:01.413246    2828 utils.go:79] GRPC request: {}
I0114 01:02:01.413358    2828 utils.go:85] GRPC response: {"ready":{"value":true}}
I0114 01:02:03.718218    2828 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0114 01:02:03.718218    2828 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="0655267a-93a7-11ed-9dd5-6045bd608314" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-e7e93284-927f-4ae5-b370-cb0d2b708a35" "latency"=14375401200
I0114 01:02:03.718218    2828 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="0655267a-93a7-11ed-9dd5-6045bd608314" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=14375996300
I0114 01:02:08.454294    2828 nodeserver_v2.go:211] NodeStageVolume: formatting 10 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-e7e93284-927f-4ae5-b370-cb0d2b708a35\globalmount with mount options([])
I0114 01:02:15.400965    2828 azuredisk_v2.go:578]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-14T01:02:15Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="15dd7435-93a7-11ed-9dd5-6045bd608314"
I0114 01:02:15.408527    2828 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-thn7-slzjr/status 200 OK in 6 milliseconds
I0114 01:02:15.409165    2828 azuredisk_v2.go:565]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="15dd7435-93a7-11ed-9dd5-6045bd608314" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=8199600
... skipping 132 lines ...
I0114 01:04:22.189989    2828 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-0b54d7a1-4ffd-440e-b5d1-d4582feda973\\globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-0b54d7a1-4ffd-440e-b5d1-d4582feda973","csi.storage.k8s.io/pvc/name":"pvc-vc8n6","csi.storage.k8s.io/pvc/namespace":"azuredisk-7247","requestedsizegib":"10","skuname":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1673656805162-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-thn7c4/providers/Microsoft.Compute/disks/pvc-0b54d7a1-4ffd-440e-b5d1-d4582feda973"}
I0114 01:04:22.189989    2828 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-0b54d7a1-4ffd-440e-b5d1-d4582feda973-capz-thn7-slzjr-attachment)
I0114 01:04:24.717963    2828 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 01:04:24.770272    2828 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 01:04:24.770272    2828 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync
I0114 01:04:24.770272    2828 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 01:04:24.770272    2828 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0114 01:04:24.770272    2828 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0114 01:04:24.770272    2828 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0114 01:04:24.822286    2828 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 01:04:31.423019    2828 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I0114 01:04:31.423065    2828 utils.go:79] GRPC request: {}
I0114 01:04:31.423166    2828 utils.go:85] GRPC response: {"ready":{"value":true}}
I0114 01:04:33.942028    2828 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0114 01:04:33.942618    2828 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="616fd545-93a7-11ed-9dd5-6045bd608314" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-2fa2051a-5bd0-4d9b-a181-2c716b4870a5" "latency"=11752039800
I0114 01:04:33.942841    2828 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="616fd545-93a7-11ed-9dd5-6045bd608314" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=11752852200
I0114 01:04:33.945752    2828 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0114 01:04:33.945752    2828 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="616fec75-93a7-11ed-9dd6-6045bd608314" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-0b54d7a1-4ffd-440e-b5d1-d4582feda973" "latency"=11754614200
I0114 01:04:33.946290    2828 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="616fec75-93a7-11ed-9dd6-6045bd608314" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=11755169500
I0114 01:04:33.946290    2828 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0114 01:04:33.946290    2828 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="616fec75-93a7-11ed-9dd5-6045bd608314" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-31106f4c-6012-43b0-904c-68f7d4c3512f" "latency"=11755707500
I0114 01:04:33.946290    2828 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="616fec75-93a7-11ed-9dd5-6045bd608314" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=11755707500
I0114 01:04:40.379568    2828 nodeserver_v2.go:211] NodeStageVolume: formatting 11 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-31106f4c-6012-43b0-904c-68f7d4c3512f\globalmount with mount options([])
I0114 01:04:40.399161    2828 nodeserver_v2.go:211] NodeStageVolume: formatting 12 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-0b54d7a1-4ffd-440e-b5d1-d4582feda973\globalmount with mount options([])
I0114 01:04:40.613287    2828 nodeserver_v2.go:211] NodeStageVolume: formatting 10 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-2fa2051a-5bd0-4d9b-a181-2c716b4870a5\globalmount with mount options([])
I0114 01:04:45.406627    2828 azuredisk_v2.go:578]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-14T01:04:45Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="6f468060-93a7-11ed-9dd6-6045bd608314"
... skipping 138 lines ...
I0114 01:06:35.371718    2828 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0114 01:06:35.371758    2828 utils.go:79] GRPC request: {}
I0114 01:06:35.371758    2828 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I0114 01:06:35.373644    2828 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0114 01:06:35.373644    2828 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-604c4585-6ac4-459f-869c-a9124bb6d3c0\\globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-604c4585-6ac4-459f-869c-a9124bb6d3c0","csi.storage.k8s.io/pvc/name":"pvc-zvjtn","csi.storage.k8s.io/pvc/namespace":"azuredisk-1594","requestedsizegib":"10","skuname":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1673656805162-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-thn7c4/providers/Microsoft.Compute/disks/pvc-604c4585-6ac4-459f-869c-a9124bb6d3c0"}
I0114 01:06:35.373644    2828 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-604c4585-6ac4-459f-869c-a9124bb6d3c0-capz-thn7-slzjr-attachment)
I0114 01:06:42.497278    2828 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0114 01:06:42.497278    2828 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="b0d222fe-93a7-11ed-9dd6-6045bd608314" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-604c4585-6ac4-459f-869c-a9124bb6d3c0" "latency"=7123357100
I0114 01:06:42.497278    2828 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="b0d222fe-93a7-11ed-9dd6-6045bd608314" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=7123357100
I0114 01:06:45.416731    2828 azuredisk_v2.go:578]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-14T01:06:45Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="b6ce8c6d-93a7-11ed-9dd6-6045bd608314"
I0114 01:06:45.424744    2828 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-thn7-slzjr/status 200 OK in 7 milliseconds
I0114 01:06:45.425285    2828 azuredisk_v2.go:565]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="b6ce8c6d-93a7-11ed-9dd6-6045bd608314" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=8743300
I0114 01:06:47.334763    2828 nodeserver_v2.go:211] NodeStageVolume: formatting 10 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-604c4585-6ac4-459f-869c-a9124bb6d3c0\globalmount with mount options([])
... skipping 74 lines ...
I0114 01:08:17.676202    2828 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-6dc7122d-cd59-4713-8699-fcc00d6b3df7\\globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-6dc7122d-cd59-4713-8699-fcc00d6b3df7","csi.storage.k8s.io/pvc/name":"pvc-r567t","csi.storage.k8s.io/pvc/namespace":"azuredisk-1594","requestedsizegib":"10","skuname":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1673656805162-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-thn7c4/providers/Microsoft.Compute/disks/pvc-6dc7122d-cd59-4713-8699-fcc00d6b3df7"}
I0114 01:08:17.676730    2828 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-6dc7122d-cd59-4713-8699-fcc00d6b3df7-capz-thn7-slzjr-attachment)
I0114 01:08:24.788779    2828 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 01:08:24.820553    2828 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync
I0114 01:08:24.820553    2828 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 01:08:24.820553    2828 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 01:08:24.820553    2828 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0114 01:08:24.893454    2828 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 01:08:27.292060    2828 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0114 01:08:27.292276    2828 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="edcc44ad-93a7-11ed-9dd6-6045bd608314" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-6dc7122d-cd59-4713-8699-fcc00d6b3df7" "latency"=9615030500
I0114 01:08:27.292327    2828 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="edcc44ad-93a7-11ed-9dd6-6045bd608314" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9615931600
I0114 01:08:29.254979    2828 reflector.go:559] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: Watch close - *v1.CustomResourceDefinition total 0 items received
I0114 01:08:29.260196    2828 round_trippers.go:553] GET https://10.96.0.1:443/apis/apiextensions.k8s.io/v1/customresourcedefinitions?allowWatchBookmarks=true&resourceVersion=10579&timeout=7m11s&timeoutSeconds=431&watch=true 200 OK in 4 milliseconds
I0114 01:08:31.441655    2828 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I0114 01:08:31.441655    2828 utils.go:79] GRPC request: {}
... skipping 105 lines ...
I0114 01:11:24.210639    2828 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: Watch close - *v1beta2.AzDriverNode total 46 items received
I0114 01:11:24.221342    2828 round_trippers.go:553] GET https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/azdrivernodes?allowWatchBookmarks=true&resourceVersion=13744&timeout=9m47s&timeoutSeconds=587&watch=true 200 OK in 10 milliseconds
I0114 01:11:24.854448    2828 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 01:11:24.869575    2828 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync
I0114 01:11:24.869575    2828 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 01:11:24.869575    2828 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 01:11:24.869575    2828 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0114 01:11:24.933058    2828 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 01:11:30.837852    2828 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0114 01:11:30.837852    2828 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="5b3a7a13-93a8-11ed-9dd6-6045bd608314" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-c8664149-b16e-449c-960d-680e2012190e" "latency"=9567519400
I0114 01:11:30.837852    2828 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="5b3a7a13-93a8-11ed-9dd6-6045bd608314" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9567519400
I0114 01:11:31.452519    2828 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I0114 01:11:31.452564    2828 utils.go:79] GRPC request: {}
I0114 01:11:31.452663    2828 utils.go:85] GRPC response: {"ready":{"value":true}}
I0114 01:11:35.796290    2828 nodeserver_v2.go:211] NodeStageVolume: formatting 10 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-c8664149-b16e-449c-960d-680e2012190e\globalmount with mount options([])
... skipping 77 lines ...
I0114 01:13:03.411562    2828 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0114 01:13:03.411562    2828 utils.go:79] GRPC request: {}
I0114 01:13:03.411689    2828 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I0114 01:13:03.412744    2828 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0114 01:13:03.412744    2828 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-c8664149-b16e-449c-960d-680e2012190e\\globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-c8664149-b16e-449c-960d-680e2012190e","csi.storage.k8s.io/pvc/name":"pvc-nd58z","csi.storage.k8s.io/pvc/namespace":"azuredisk-3957","requestedsizegib":"10","skuname":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1673656805162-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-thn7c4/providers/Microsoft.Compute/disks/pvc-c8664149-b16e-449c-960d-680e2012190e"}
I0114 01:13:03.413670    2828 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-c8664149-b16e-449c-960d-680e2012190e-capz-thn7-slzjr-attachment)
I0114 01:13:08.700099    2828 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0114 01:13:08.700299    2828 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="981c56a1-93a8-11ed-9dd6-6045bd608314" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-c8664149-b16e-449c-960d-680e2012190e" "latency"=5285504700
I0114 01:13:08.700360    2828 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="981c56a1-93a8-11ed-9dd6-6045bd608314" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=5285710000
I0114 01:13:13.500616    2828 nodeserver_v2.go:211] NodeStageVolume: formatting 10 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-c8664149-b16e-449c-960d-680e2012190e\globalmount with mount options([])
I0114 01:13:15.428293    2828 azuredisk_v2.go:578]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-14T01:13:15Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="9f459704-93a8-11ed-9dd6-6045bd608314"
I0114 01:13:15.437134    2828 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-thn7-slzjr/status 200 OK in 7 milliseconds
I0114 01:13:15.437729    2828 azuredisk_v2.go:565]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="9f459704-93a8-11ed-9dd6-6045bd608314" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=9114400
... skipping 184 lines ...
I0114 01:18:54.994769    2828 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 01:18:54.995224    2828 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 01:18:55.090340    2828 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 01:19:01.481947    2828 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I0114 01:19:01.482004    2828 utils.go:79] GRPC request: {}
I0114 01:19:01.482104    2828 utils.go:85] GRPC response: {"ready":{"value":true}}
W0114 01:19:02.253943    2828 reflector.go:347] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: watch of *v1beta2.AzDriverNode ended with: an error on the server ("unable to decode an event from the watch stream: http2: client connection lost") has prevented the request from succeeding
W0114 01:19:02.253943    2828 reflector.go:347] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: watch of *v1beta2.AzVolumeAttachment ended with: an error on the server ("unable to decode an event from the watch stream: http2: client connection lost") has prevented the request from succeeding
W0114 01:19:02.254008    2828 reflector.go:347] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: watch of *v1beta2.AzDriverNode ended with: an error on the server ("unable to decode an event from the watch stream: http2: client connection lost") has prevented the request from succeeding
I0114 01:19:02.254083    2828 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-thn7-slzjr/status  in 16800 milliseconds
W0114 01:19:02.254120    2828 reflector.go:347] k8s.io/client-go/informers/factory.go:150: watch of *v1.Node ended with: an error on the server ("unable to decode an event from the watch stream: http2: client connection lost") has prevented the request from succeeding
W0114 01:19:02.254120    2828 reflector.go:347] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: watch of *v1beta2.AzVolume ended with: an error on the server ("unable to decode an event from the watch stream: http2: client connection lost") has prevented the request from succeeding
W0114 01:19:02.254120    2828 reflector.go:347] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: watch of *v1.CustomResourceDefinition ended with: an error on the server ("unable to decode an event from the watch stream: http2: client connection lost") has prevented the request from succeeding
E0114 01:19:02.254206    2828 azuredisk_v2.go:613]  "msg"="Failed to update AzDriverNode status after creation" "error"="Put \"https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-thn7-slzjr/status\": http2: client connection lost" "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="63fb3415-93a9-11ed-9dd6-6045bd608314"
E0114 01:19:02.254206    2828 azuredisk_v2.go:565]  "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [Put \"https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-thn7-slzjr/status\": http2: client connection lost]" "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="63fb3415-93a9-11ed-9dd6-6045bd608314" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=16801570400
I0114 01:19:03.075856    2828 reflector.go:257] Listing and watching *v1beta2.AzDriverNode from sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132
I0114 01:19:03.092591    2828 round_trippers.go:553] GET https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/azdrivernodes?resourceVersion=15658 200 OK in 10 milliseconds
I0114 01:19:03.093757    2828 round_trippers.go:553] GET https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/azdrivernodes?allowWatchBookmarks=true&resourceVersion=15774&timeout=5m39s&timeoutSeconds=339&watch=true 200 OK in 0 milliseconds
I0114 01:19:03.151079    2828 reflector.go:257] Listing and watching *v1beta2.AzVolumeAttachment from sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132
I0114 01:19:03.154410    2828 round_trippers.go:553] GET https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumeattachments?resourceVersion=15600 200 OK in 0 milliseconds
E0114 01:19:03.156777    2828 conditionwatcher.go:138] object (%!v(MISSING)) has not implemented meta object interface.
... skipping 63 lines ...
I0114 01:20:02.558963    2828 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-2349d479-4ea3-4b2a-ad01-ef6572d8d223-capz-thn7-slzjr-attachment)
I0114 01:20:02.560247    2828 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0114 01:20:02.560288    2828 utils.go:79] GRPC request: {"publish_context":{"LUN":"1"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-4cf3e50d-0de8-4ee8-b811-f5eab1628345\\globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-4cf3e50d-0de8-4ee8-b811-f5eab1628345","csi.storage.k8s.io/pvc/name":"pvc-bmd6n","csi.storage.k8s.io/pvc/namespace":"azuredisk-4098","requestedsizegib":"10","skuname":"Premium_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1673656805162-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-thn7c4/providers/Microsoft.Compute/disks/pvc-4cf3e50d-0de8-4ee8-b811-f5eab1628345"}
I0114 01:20:02.560598    2828 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-4cf3e50d-0de8-4ee8-b811-f5eab1628345-capz-thn7-slzjr-attachment)
I0114 01:20:03.109988    2828 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 01:20:03.166311    2828 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 01:20:03.166461    2828 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0114 01:20:03.166675    2828 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0114 01:20:03.166675    2828 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0114 01:20:03.468069    2828 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync
I0114 01:20:03.753007    2828 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 01:20:03.857268    2828 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 01:20:12.251174    2828 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0114 01:20:12.251369    2828 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="91f0d785-93a9-11ed-9dd6-6045bd608314" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-2349d479-4ea3-4b2a-ad01-ef6572d8d223" "latency"=9692086100
I0114 01:20:12.251369    2828 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="91f0d785-93a9-11ed-9dd6-6045bd608314" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9692179200
I0114 01:20:12.254580    2828 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0114 01:20:12.255144    2828 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="91f1160d-93a9-11ed-9dd6-6045bd608314" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-4cf3e50d-0de8-4ee8-b811-f5eab1628345" "latency"=9694207300
I0114 01:20:12.255228    2828 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="91f1160d-93a9-11ed-9dd6-6045bd608314" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9694353400
I0114 01:20:12.257972    2828 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0114 01:20:12.257972    2828 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="91f0c097-93a9-11ed-9dd6-6045bd608314" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-5165bb1e-ec89-4ead-bdb9-3dda7588cdf2" "latency"=9698781900
I0114 01:20:12.258101    2828 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="91f0c097-93a9-11ed-9dd6-6045bd608314" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9699458500
I0114 01:20:15.452225    2828 azuredisk_v2.go:578]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-14T01:20:15Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="99a03290-93a9-11ed-9dd6-6045bd608314"
I0114 01:20:15.464229    2828 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-thn7-slzjr/status 200 OK in 8 milliseconds
I0114 01:20:15.464229    2828 azuredisk_v2.go:565]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="99a03290-93a9-11ed-9dd6-6045bd608314" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=12003100
I0114 01:20:18.417213    2828 nodeserver_v2.go:211] NodeStageVolume: formatting 10 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-4cf3e50d-0de8-4ee8-b811-f5eab1628345\globalmount with mount options([])
... skipping 101 lines ...
I0114 01:21:07.993805    2828 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0114 01:21:07.993854    2828 utils.go:79] GRPC request: {}
I0114 01:21:07.993854    2828 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I0114 01:21:07.996932    2828 utils.go:78] GRPC call: /csi.v1.Node/NodeUnstageVolume
I0114 01:21:07.996979    2828 utils.go:79] GRPC request: {"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-2349d479-4ea3-4b2a-ad01-ef6572d8d223\\globalmount","volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-thn7c4/providers/Microsoft.Compute/disks/pvc-2349d479-4ea3-4b2a-ad01-ef6572d8d223"}
I0114 01:21:07.997112    2828 nodeserver_v2.go:257] NodeUnstageVolume: unmounting \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-2349d479-4ea3-4b2a-ad01-ef6572d8d223\globalmount
E0114 01:21:09.143327    2828 utils.go:83] GRPC error: GetVolumeIDFromMount(c:\var\lib\kubelet\pods\3e4be9ff-f23b-458d-b555-3a401a4196be\volumes\kubernetes.io~csi\pvc-2349d479-4ea3-4b2a-ad01-ef6572d8d223\mount) failed with error: rpc error: code = Unknown desc = error getting the volume for the mount c:\var\lib\kubelet\pods\3e4be9ff-f23b-458d-b555-3a401a4196be\volumes\kubernetes.io~csi\pvc-2349d479-4ea3-4b2a-ad01-ef6572d8d223\mount, internal error error getting volume from mount. cmd: (Get-Item -Path Get-Item : Cannot find path 'C:\var\lib\kubelet\pods\3e4be9ff-f23b-458d-b555-3a401a4196be\volumes\kubernetes.io~csi\pvc
-2349d479-4ea3-4b2a-ad01-ef6572d8d223\mount' because it does not exist.
At line:1 char:2
+ (Get-Item -Path c:\var\lib\kubelet\pods\3e4be9ff-f23b-458d-b555-3a401 ...
+  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : ObjectNotFound: (C:\var\lib\kube...572d8d223\mount:String) [Get-Item], ItemNotFoundExcep 
   tion
... skipping 37 lines ...
+ ... d : PathNotFound,Microsoft.PowerShell.Commands.GetItemCommand).Target
+                                                                  ~
Unexpected token ')' in expression or statement.
    + CategoryInfo          : ParserError: (:) [], ParentContainsErrorRecordException
    + FullyQualifiedErrorId : MissingEndParenthesisInExpression
 
, error: exit status 1, response: <nil>
I0114 01:21:11.846923    2828 nodeserver_v2.go:262] NodeUnstageVolume: unmount \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-5165bb1e-ec89-4ead-bdb9-3dda7588cdf2\globalmount successfully
I0114 01:21:11.846923    2828 utils.go:85] GRPC response: {}
I0114 01:21:11.936936    2828 nodeserver_v2.go:262] NodeUnstageVolume: unmount \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-4cf3e50d-0de8-4ee8-b811-f5eab1628345\globalmount successfully
I0114 01:21:11.936936    2828 utils.go:85] GRPC response: {}
I0114 01:21:11.943676    2828 nodeserver_v2.go:262] NodeUnstageVolume: unmount \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-2349d479-4ea3-4b2a-ad01-ef6572d8d223\globalmount successfully
I0114 01:21:11.943747    2828 utils.go:85] GRPC response: {}
... skipping 34 lines ...
I0114 01:22:16.219521    2828 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0114 01:22:16.219521    2828 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-1c383bc9-ed50-42d3-977e-866ccee3baa3\\globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-1c383bc9-ed50-42d3-977e-866ccee3baa3","csi.storage.k8s.io/pvc/name":"test-volume-1-azuredisk-volume-tester-vbz92-0","csi.storage.k8s.io/pvc/namespace":"azuredisk-800","requestedsizegib":"10","skuname":"Standard_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1673656805162-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-thn7c4/providers/Microsoft.Compute/disks/pvc-1c383bc9-ed50-42d3-977e-866ccee3baa3"}
I0114 01:22:16.219916    2828 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-1c383bc9-ed50-42d3-977e-866ccee3baa3-capz-thn7-slzjr-attachment)
I0114 01:22:31.487875    2828 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I0114 01:22:31.487875    2828 utils.go:79] GRPC request: {}
I0114 01:22:31.487966    2828 utils.go:85] GRPC response: {"ready":{"value":true}}
I0114 01:22:31.653614    2828 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0114 01:22:31.653873    2828 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="e19be048-93a9-11ed-9dd6-6045bd608314" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-1c383bc9-ed50-42d3-977e-866ccee3baa3" "latency"=15433124300
I0114 01:22:31.653978    2828 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="e19be048-93a9-11ed-9dd6-6045bd608314" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=15433489100
I0114 01:22:33.152622    2828 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 01:22:33.235098    2828 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 01:22:33.503388    2828 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync
I0114 01:22:33.828843    2828 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
... skipping 38 lines ...
I0114 01:23:10.183144    2828 nodeserver_v2.go:369] NodeUnpublishVolume: unmounting volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-thn7c4/providers/Microsoft.Compute/disks/pvc-1c383bc9-ed50-42d3-977e-866ccee3baa3 on c:\var\lib\kubelet\pods\6c566918-5508-47e3-bc1a-eb42cb422e50\volumes\kubernetes.io~csi\pvc-1c383bc9-ed50-42d3-977e-866ccee3baa3\mount
I0114 01:23:13.325418    2828 nodeserver_v2.go:375] NodeUnpublishVolume: unmount volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-thn7c4/providers/Microsoft.Compute/disks/pvc-1c383bc9-ed50-42d3-977e-866ccee3baa3 on c:\var\lib\kubelet\pods\6c566918-5508-47e3-bc1a-eb42cb422e50\volumes\kubernetes.io~csi\pvc-1c383bc9-ed50-42d3-977e-866ccee3baa3\mount successfully
I0114 01:23:13.325478    2828 utils.go:85] GRPC response: {}
I0114 01:23:13.869907    2828 utils.go:78] GRPC call: /csi.v1.Node/NodeUnpublishVolume
I0114 01:23:13.869907    2828 utils.go:79] GRPC request: {"target_path":"c:\\var\\lib\\kubelet\\pods\\6c566918-5508-47e3-bc1a-eb42cb422e50\\volumes\\kubernetes.io~csi\\pvc-1c383bc9-ed50-42d3-977e-866ccee3baa3\\mount","volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-thn7c4/providers/Microsoft.Compute/disks/pvc-1c383bc9-ed50-42d3-977e-866ccee3baa3"}
I0114 01:23:13.870439    2828 nodeserver_v2.go:369] NodeUnpublishVolume: unmounting volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-thn7c4/providers/Microsoft.Compute/disks/pvc-1c383bc9-ed50-42d3-977e-866ccee3baa3 on c:\var\lib\kubelet\pods\6c566918-5508-47e3-bc1a-eb42cb422e50\volumes\kubernetes.io~csi\pvc-1c383bc9-ed50-42d3-977e-866ccee3baa3\mount
W0114 01:23:14.667788    2828 safe_mounter_windows.go:113] GetVolumeIDFromTargetPath(c:\var\lib\kubelet\pods\6c566918-5508-47e3-bc1a-eb42cb422e50\volumes\kubernetes.io~csi\pvc-1c383bc9-ed50-42d3-977e-866ccee3baa3\mount) failed with error: rpc error: code = Unknown desc = error getting the volume for the mount c:\var\lib\kubelet\pods\6c566918-5508-47e3-bc1a-eb42cb422e50\volumes\kubernetes.io~csi\pvc-1c383bc9-ed50-42d3-977e-866ccee3baa3\mount, internal error error getting volume from mount. cmd: (Get-Item -Path Get-Item : Cannot find path 'C:\var\lib\kubelet\pods\6c566918-5508-47e3-bc1a-eb42cb422e50\volumes\kubernetes.io~csi\pvc
-1c383bc9-ed50-42d3-977e-866ccee3baa3\mount' because it does not exist.
At line:1 char:2
+ (Get-Item -Path c:\var\lib\kubelet\pods\6c566918-5508-47e3-bc1a-eb42c ...
+  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : ObjectNotFound: (C:\var\lib\kube...ccee3baa3\mount:String) [Get-Item], ItemNotFoundExcep 
   tion
... skipping 37 lines ...
+ ... d : PathNotFound,Microsoft.PowerShell.Commands.GetItemCommand).Target
+                                                                  ~
Unexpected token ')' in expression or statement.
    + CategoryInfo          : ParserError: (:) [], ParentContainsErrorRecordException
    + FullyQualifiedErrorId : MissingEndParenthesisInExpression
 
, error: exit status 1, response: <nil>
I0114 01:23:14.668721    2828 nodeserver_v2.go:375] NodeUnpublishVolume: unmount volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-thn7c4/providers/Microsoft.Compute/disks/pvc-1c383bc9-ed50-42d3-977e-866ccee3baa3 on c:\var\lib\kubelet\pods\6c566918-5508-47e3-bc1a-eb42cb422e50\volumes\kubernetes.io~csi\pvc-1c383bc9-ed50-42d3-977e-866ccee3baa3\mount successfully
I0114 01:23:14.668761    2828 utils.go:85] GRPC response: {}
I0114 01:23:14.741779    2828 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0114 01:23:14.741779    2828 utils.go:79] GRPC request: {}
I0114 01:23:14.741779    2828 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I0114 01:23:14.743675    2828 utils.go:78] GRPC call: /csi.v1.Node/NodeUnstageVolume
... skipping 71 lines ...
I0114 01:25:39.996144    2828 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0114 01:25:39.996199    2828 utils.go:79] GRPC request: {}
I0114 01:25:39.996282    2828 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I0114 01:25:39.997767    2828 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0114 01:25:39.997813    2828 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-1c383bc9-ed50-42d3-977e-866ccee3baa3\\globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-1c383bc9-ed50-42d3-977e-866ccee3baa3","csi.storage.k8s.io/pvc/name":"test-volume-1-azuredisk-volume-tester-vbz92-0","csi.storage.k8s.io/pvc/namespace":"azuredisk-800","requestedsizegib":"10","skuname":"Standard_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1673656805162-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-thn7c4/providers/Microsoft.Compute/disks/pvc-1c383bc9-ed50-42d3-977e-866ccee3baa3"}
I0114 01:25:39.998181    2828 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-1c383bc9-ed50-42d3-977e-866ccee3baa3-capz-thn7-slzjr-attachment)
I0114 01:25:45.145988    2828 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0114 01:25:45.146169    2828 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="5b11f904-93aa-11ed-9dd6-6045bd608314" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-1c383bc9-ed50-42d3-977e-866ccee3baa3" "latency"=5147652000
I0114 01:25:45.146169    2828 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="5b11f904-93aa-11ed-9dd6-6045bd608314" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=5147819000
I0114 01:25:45.471702    2828 azuredisk_v2.go:578]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-14T01:25:45Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="5e552c5e-93aa-11ed-9dd6-6045bd608314"
I0114 01:25:45.480797    2828 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-thn7-slzjr/status 200 OK in 9 milliseconds
I0114 01:25:45.481122    2828 azuredisk_v2.go:565]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="5e552c5e-93aa-11ed-9dd6-6045bd608314" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=9485800
I0114 01:25:49.831519    2828 nodeserver_v2.go:211] NodeStageVolume: formatting 10 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-1c383bc9-ed50-42d3-977e-866ccee3baa3\globalmount with mount options([])
... skipping 84 lines ...
I0114 01:27:11.594254    2828 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0114 01:27:11.594254    2828 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-a976ab0f-9989-44ff-b9c8-9928e9e37d61\\globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-a976ab0f-9989-44ff-b9c8-9928e9e37d61","csi.storage.k8s.io/pvc/name":"test-volume-1-azuredisk-volume-tester-9nsj4-0","csi.storage.k8s.io/pvc/namespace":"azuredisk-8324","requestedsizegib":"10","skuname":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1673656805162-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-thn7c4/providers/Microsoft.Compute/disks/pvc-a976ab0f-9989-44ff-b9c8-9928e9e37d61"}
I0114 01:27:11.594551    2828 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-a976ab0f-9989-44ff-b9c8-9928e9e37d61-capz-thn7-slzjr-attachment)
I0114 01:27:15.486653    2828 azuredisk_v2.go:578]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-14T01:27:15Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="93fc583b-93aa-11ed-9dd6-6045bd608314"
I0114 01:27:15.498279    2828 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-thn7-slzjr/status 200 OK in 10 milliseconds
I0114 01:27:15.498279    2828 azuredisk_v2.go:565]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="93fc583b-93aa-11ed-9dd6-6045bd608314" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=11758100
I0114 01:27:16.395168    2828 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0114 01:27:16.395168    2828 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="91aa7a26-93aa-11ed-9dd6-6045bd608314" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-a976ab0f-9989-44ff-b9c8-9928e9e37d61" "latency"=4800429900
I0114 01:27:16.395738    2828 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="91aa7a26-93aa-11ed-9dd6-6045bd608314" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=4800958400
I0114 01:27:21.196880    2828 nodeserver_v2.go:211] NodeStageVolume: formatting 10 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-a976ab0f-9989-44ff-b9c8-9928e9e37d61\globalmount with mount options([])
I0114 01:27:31.492584    2828 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I0114 01:27:31.492635    2828 utils.go:79] GRPC request: {}
I0114 01:27:31.492704    2828 utils.go:85] GRPC response: {"ready":{"value":true}}
... skipping 98 lines ...
I0114 01:29:54.352089    2828 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0114 01:29:54.352089    2828 utils.go:79] GRPC request: {}
I0114 01:29:54.352089    2828 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I0114 01:29:54.353901    2828 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0114 01:29:54.353901    2828 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-c8a37440-3028-465f-9fab-9657e2d641d6\\globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-c8a37440-3028-465f-9fab-9657e2d641d6","csi.storage.k8s.io/pvc/name":"1-azuredisk-volume-tester-5lhnw-0","csi.storage.k8s.io/pvc/namespace":"azuredisk-2145","requestedsizegib":"10","skuname":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1673656805162-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-thn7c4/providers/Microsoft.Compute/disks/pvc-c8a37440-3028-465f-9fab-9657e2d641d6"}
I0114 01:29:54.353901    2828 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-c8a37440-3028-465f-9fab-9657e2d641d6-capz-thn7-slzjr-attachment)
I0114 01:29:56.715578    2828 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0114 01:29:56.716142    2828 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="f2ad9547-93aa-11ed-9dd6-6045bd608314" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-c8a37440-3028-465f-9fab-9657e2d641d6" "latency"=2361279000
I0114 01:29:56.716331    2828 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="f2ad9547-93aa-11ed-9dd6-6045bd608314" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=2361990600
I0114 01:30:01.498558    2828 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I0114 01:30:01.498859    2828 utils.go:79] GRPC request: {}
I0114 01:30:01.499058    2828 utils.go:85] GRPC response: {"ready":{"value":true}}
I0114 01:30:01.540048    2828 nodeserver_v2.go:211] NodeStageVolume: formatting 10 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-c8a37440-3028-465f-9fab-9657e2d641d6\globalmount with mount options([])
... skipping 174 lines ...
I0114 01:35:00.956428    2828 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-a546d589-9aa5-4be8-9817-cdd3b1da69e7-capz-thn7-slzjr-attachment)
I0114 01:35:01.541652    2828 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I0114 01:35:01.541712    2828 utils.go:79] GRPC request: {}
I0114 01:35:01.541832    2828 utils.go:85] GRPC response: {"ready":{"value":true}}
I0114 01:35:03.397622    2828 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 01:35:03.465600    2828 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 01:35:03.465738    2828 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0114 01:35:03.763039    2828 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync
I0114 01:35:04.093584    2828 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 01:35:04.134835    2828 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 01:35:08.128110    2828 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0114 01:35:08.128798    2828 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="a96d68c5-93ab-11ed-9dd6-6045bd608314" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-a546d589-9aa5-4be8-9817-cdd3b1da69e7" "latency"=7171994100
I0114 01:35:08.128871    2828 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="a96d68c5-93ab-11ed-9dd6-6045bd608314" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=7172308600
I0114 01:35:13.059547    2828 nodeserver_v2.go:211] NodeStageVolume: formatting 10 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-a546d589-9aa5-4be8-9817-cdd3b1da69e7\globalmount with mount options([barrier=1 acl])
I0114 01:35:15.511160    2828 azuredisk_v2.go:578]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-14T01:35:15Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="b21a4665-93ab-11ed-9dd6-6045bd608314"
I0114 01:35:15.518948    2828 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-thn7-slzjr/status 200 OK in 6 milliseconds
I0114 01:35:15.520142    2828 azuredisk_v2.go:565]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="b21a4665-93ab-11ed-9dd6-6045bd608314" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=8831100
... skipping 74 lines ...
I0114 01:36:34.516521    2828 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0114 01:36:34.516521    2828 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-f8c871f0-1691-47e5-8e29-769da26b9b09\\globalmount","volume_capability":{"AccessType":{"Mount":{"mount_flags":["barrier=1","acl"]}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-f8c871f0-1691-47e5-8e29-769da26b9b09","csi.storage.k8s.io/pvc/name":"pvc-qkpd2","csi.storage.k8s.io/pvc/namespace":"azuredisk-571","requestedsizegib":"10","skuname":"Standard_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1673656805162-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-thn7c4/providers/Microsoft.Compute/disks/pvc-f8c871f0-1691-47e5-8e29-769da26b9b09"}
I0114 01:36:34.517064    2828 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-f8c871f0-1691-47e5-8e29-769da26b9b09-capz-thn7-slzjr-attachment)
I0114 01:36:45.518709    2828 azuredisk_v2.go:578]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-14T01:36:45Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="e7c056b5-93ab-11ed-9dd6-6045bd608314"
I0114 01:36:45.541173    2828 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-thn7-slzjr/status 200 OK in 22 milliseconds
I0114 01:36:45.541236    2828 azuredisk_v2.go:565]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="e7c056b5-93ab-11ed-9dd6-6045bd608314" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=22683900
I0114 01:36:46.492956    2828 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0114 01:36:46.493535    2828 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="e1318d9d-93ab-11ed-9dd6-6045bd608314" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-f8c871f0-1691-47e5-8e29-769da26b9b09" "latency"=11974612700
I0114 01:36:46.493924    2828 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="e1318d9d-93ab-11ed-9dd6-6045bd608314" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=11975819800
I0114 01:36:51.318698    2828 nodeserver_v2.go:211] NodeStageVolume: formatting 10 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-f8c871f0-1691-47e5-8e29-769da26b9b09\globalmount with mount options([barrier=1 acl])
I0114 01:37:01.533907    2828 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I0114 01:37:01.534061    2828 utils.go:79] GRPC request: {}
I0114 01:37:01.534178    2828 utils.go:85] GRPC response: {"ready":{"value":true}}
... skipping 67 lines ...
I0114 01:38:16.124876    2828 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0114 01:38:16.124876    2828 utils.go:79] GRPC request: {}
I0114 01:38:16.124876    2828 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I0114 01:38:16.126139    2828 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0114 01:38:16.126666    2828 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-b0a4977e-2912-42b3-8180-95044202fa70\\globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"cachingmode":"None","csi.storage.k8s.io/pv/name":"pvc-b0a4977e-2912-42b3-8180-95044202fa70","csi.storage.k8s.io/pvc/name":"pvc-j29qz","csi.storage.k8s.io/pvc/namespace":"azuredisk-1219","maxshares":"1","requestedsizegib":"10","skuname":"Premium_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1673656805162-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-thn7c4/providers/Microsoft.Compute/disks/pvc-b0a4977e-2912-42b3-8180-95044202fa70"}
I0114 01:38:16.126838    2828 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-b0a4977e-2912-42b3-8180-95044202fa70-capz-thn7-slzjr-attachment)
I0114 01:38:19.035226    2828 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0114 01:38:19.035457    2828 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="1dc20ea4-93ac-11ed-9dd6-6045bd608314" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-b0a4977e-2912-42b3-8180-95044202fa70" "latency"=2908422500
I0114 01:38:19.035508    2828 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="1dc20ea4-93ac-11ed-9dd6-6045bd608314" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=2908546200
I0114 01:38:24.163083    2828 nodeserver_v2.go:211] NodeStageVolume: formatting 10 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-b0a4977e-2912-42b3-8180-95044202fa70\globalmount with mount options([])
I0114 01:38:31.548429    2828 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I0114 01:38:31.548482    2828 utils.go:79] GRPC request: {}
I0114 01:38:31.548557    2828 utils.go:85] GRPC response: {"ready":{"value":true}}
... skipping 105 lines ...
I0114 01:41:19.527229    2828 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-3150f98a-cb80-4b06-994c-d71a6c69164f-capz-thn7-slzjr-attachment)
I0114 01:41:31.553128    2828 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I0114 01:41:31.553218    2828 utils.go:79] GRPC request: {}
I0114 01:41:31.553315    2828 utils.go:85] GRPC response: {"ready":{"value":true}}
I0114 01:41:33.496990    2828 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 01:41:33.604499    2828 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 01:41:33.604853    2828 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0114 01:41:33.820133    2828 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0114 01:41:33.820282    2828 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="8b12bc68-93ac-11ed-9dd6-6045bd608314" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-3150f98a-cb80-4b06-994c-d71a6c69164f" "latency"=14292576200
I0114 01:41:33.820282    2828 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="8b12bc68-93ac-11ed-9dd6-6045bd608314" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=14292725200
I0114 01:41:33.844163    2828 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync
I0114 01:41:34.215829    2828 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 01:41:34.277902    2828 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 01:41:38.817486    2828 nodeserver_v2.go:211] NodeStageVolume: formatting 10 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-3150f98a-cb80-4b06-994c-d71a6c69164f\globalmount with mount options([])
... skipping 217 lines ...
I0114 01:47:27.419927    2828 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0114 01:47:27.420000    2828 utils.go:79] GRPC request: {}
I0114 01:47:27.420000    2828 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I0114 01:47:27.421015    2828 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0114 01:47:27.421015    2828 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-019a1681-abdc-458a-bb75-047388404234\\globalmount","volume_capability":{"AccessType":{"Mount":{"mount_flags":["barrier=1","acl"]}},"access_mode":{"mode":7}},"volume_context":{"cachingmode":"None","csi.storage.k8s.io/pv/name":"pvc-019a1681-abdc-458a-bb75-047388404234","csi.storage.k8s.io/pvc/name":"pvc-lkktc","csi.storage.k8s.io/pvc/namespace":"azuredisk-5211","maxshares":"2","requestedsizegib":"10","skuname":"Premium_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1673656805162-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-thn7c4/providers/Microsoft.Compute/disks/pvc-019a1681-abdc-458a-bb75-047388404234"}
I0114 01:47:27.421802    2828 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-019a1681-abdc-458a-bb75-047388404234-capz-thn7-slzjr-attachment)
I0114 01:47:31.129374    2828 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0114 01:47:31.129374    2828 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="665afe24-93ad-11ed-9dd6-6045bd608314" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-019a1681-abdc-458a-bb75-047388404234" "latency"=3707026000
I0114 01:47:31.129374    2828 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="665afe24-93ad-11ed-9dd6-6045bd608314" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=3707026000
I0114 01:47:31.562607    2828 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I0114 01:47:31.562658    2828 utils.go:79] GRPC request: {}
I0114 01:47:31.562767    2828 utils.go:85] GRPC response: {"ready":{"value":true}}
I0114 01:47:33.646630    2828 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
... skipping 45 lines ...
I0114 01:48:18.413545    2828 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I0114 01:48:18.413545    2828 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0114 01:48:18.413545    2828 utils.go:79] GRPC request: {}
I0114 01:48:18.413545    2828 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I0114 01:48:18.416557    2828 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0114 01:48:18.416557    2828 utils.go:79] GRPC request: {"publish_context":{"LUN":"1"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-76f33908-edbe-4dd1-9cae-af5a16ffd9e6\\globalmount","volume_capability":{"AccessType":{"Mount":{"mount_flags":["barrier=1","acl"]}},"access_mode":{"mode":7}},"volume_context":{"cachingmode":"None","csi.storage.k8s.io/pv/name":"pvc-76f33908-edbe-4dd1-9cae-af5a16ffd9e6","csi.storage.k8s.io/pvc/name":"pvc-hs4nh","csi.storage.k8s.io/pvc/namespace":"azuredisk-5211","maxshares":"2","requestedsizegib":"10","skuname":"Premium_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1673656805162-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-thn7c4/providers/Microsoft.Compute/disks/pvc-76f33908-edbe-4dd1-9cae-af5a16ffd9e6"}
E0114 01:48:18.416557    2828 utils.go:83] GRPC error: rpc error: code = Internal desc = failed to get current attachment for volume (/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-thn7c4/providers/Microsoft.Compute/disks/pvc-76f33908-edbe-4dd1-9cae-af5a16ffd9e6) and node (capz-thn7-slzjr): azvolumeattachment.disk.csi.azure.com "pvc-76f33908-edbe-4dd1-9cae-af5a16ffd9e6-capz-thn7-slzjr-attachment" not found
I0114 01:48:19.024479    2828 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0114 01:48:19.024479    2828 utils.go:79] GRPC request: {}
I0114 01:48:19.024479    2828 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I0114 01:48:19.030932    2828 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0114 01:48:19.030932    2828 utils.go:79] GRPC request: {}
I0114 01:48:19.030932    2828 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I0114 01:48:19.032773    2828 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0114 01:48:19.032773    2828 utils.go:79] GRPC request: {}
I0114 01:48:19.032773    2828 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I0114 01:48:19.037719    2828 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0114 01:48:19.037719    2828 utils.go:79] GRPC request: {"publish_context":{"LUN":"1"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-76f33908-edbe-4dd1-9cae-af5a16ffd9e6\\globalmount","volume_capability":{"AccessType":{"Mount":{"mount_flags":["barrier=1","acl"]}},"access_mode":{"mode":7}},"volume_context":{"cachingmode":"None","csi.storage.k8s.io/pv/name":"pvc-76f33908-edbe-4dd1-9cae-af5a16ffd9e6","csi.storage.k8s.io/pvc/name":"pvc-hs4nh","csi.storage.k8s.io/pvc/namespace":"azuredisk-5211","maxshares":"2","requestedsizegib":"10","skuname":"Premium_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1673656805162-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-thn7c4/providers/Microsoft.Compute/disks/pvc-76f33908-edbe-4dd1-9cae-af5a16ffd9e6"}
E0114 01:48:19.038309    2828 utils.go:83] GRPC error: rpc error: code = Internal desc = failed to get current attachment for volume (/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-thn7c4/providers/Microsoft.Compute/disks/pvc-76f33908-edbe-4dd1-9cae-af5a16ffd9e6) and node (capz-thn7-slzjr): azvolumeattachment.disk.csi.azure.com "pvc-76f33908-edbe-4dd1-9cae-af5a16ffd9e6-capz-thn7-slzjr-attachment" not found
I0114 01:48:20.073613    2828 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0114 01:48:20.073613    2828 utils.go:79] GRPC request: {}
I0114 01:48:20.073613    2828 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I0114 01:48:20.079796    2828 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0114 01:48:20.079796    2828 utils.go:79] GRPC request: {}
I0114 01:48:20.079796    2828 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I0114 01:48:20.081467    2828 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0114 01:48:20.081467    2828 utils.go:79] GRPC request: {}
I0114 01:48:20.081467    2828 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I0114 01:48:20.083409    2828 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0114 01:48:20.083409    2828 utils.go:79] GRPC request: {"publish_context":{"LUN":"1"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-76f33908-edbe-4dd1-9cae-af5a16ffd9e6\\globalmount","volume_capability":{"AccessType":{"Mount":{"mount_flags":["barrier=1","acl"]}},"access_mode":{"mode":7}},"volume_context":{"cachingmode":"None","csi.storage.k8s.io/pv/name":"pvc-76f33908-edbe-4dd1-9cae-af5a16ffd9e6","csi.storage.k8s.io/pvc/name":"pvc-hs4nh","csi.storage.k8s.io/pvc/namespace":"azuredisk-5211","maxshares":"2","requestedsizegib":"10","skuname":"Premium_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1673656805162-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-thn7c4/providers/Microsoft.Compute/disks/pvc-76f33908-edbe-4dd1-9cae-af5a16ffd9e6"}
E0114 01:48:20.083409    2828 utils.go:83] GRPC error: rpc error: code = Internal desc = failed to get current attachment for volume (/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-thn7c4/providers/Microsoft.Compute/disks/pvc-76f33908-edbe-4dd1-9cae-af5a16ffd9e6) and node (capz-thn7-slzjr): azvolumeattachment.disk.csi.azure.com "pvc-76f33908-edbe-4dd1-9cae-af5a16ffd9e6-capz-thn7-slzjr-attachment" not found
I0114 01:48:22.188876    2828 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0114 01:48:22.188876    2828 utils.go:79] GRPC request: {}
I0114 01:48:22.188876    2828 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I0114 01:48:22.195439    2828 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0114 01:48:22.195489    2828 utils.go:79] GRPC request: {}
I0114 01:48:22.195561    2828 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I0114 01:48:22.196985    2828 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0114 01:48:22.196985    2828 utils.go:79] GRPC request: {}
I0114 01:48:22.196985    2828 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I0114 01:48:22.198486    2828 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0114 01:48:22.198486    2828 utils.go:79] GRPC request: {"publish_context":{"LUN":"1"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-76f33908-edbe-4dd1-9cae-af5a16ffd9e6\\globalmount","volume_capability":{"AccessType":{"Mount":{"mount_flags":["barrier=1","acl"]}},"access_mode":{"mode":7}},"volume_context":{"cachingmode":"None","csi.storage.k8s.io/pv/name":"pvc-76f33908-edbe-4dd1-9cae-af5a16ffd9e6","csi.storage.k8s.io/pvc/name":"pvc-hs4nh","csi.storage.k8s.io/pvc/namespace":"azuredisk-5211","maxshares":"2","requestedsizegib":"10","skuname":"Premium_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1673656805162-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-thn7c4/providers/Microsoft.Compute/disks/pvc-76f33908-edbe-4dd1-9cae-af5a16ffd9e6"}
E0114 01:48:22.199019    2828 utils.go:83] GRPC error: rpc error: code = Internal desc = failed to get current attachment for volume (/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-thn7c4/providers/Microsoft.Compute/disks/pvc-76f33908-edbe-4dd1-9cae-af5a16ffd9e6) and node (capz-thn7-slzjr): azvolumeattachment.disk.csi.azure.com "pvc-76f33908-edbe-4dd1-9cae-af5a16ffd9e6-capz-thn7-slzjr-attachment" not found
I0114 01:48:26.286533    2828 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0114 01:48:26.286533    2828 utils.go:79] GRPC request: {}
I0114 01:48:26.287339    2828 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I0114 01:48:26.298310    2828 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0114 01:48:26.298310    2828 utils.go:79] GRPC request: {}
I0114 01:48:26.298310    2828 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I0114 01:48:26.300374    2828 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0114 01:48:26.300374    2828 utils.go:79] GRPC request: {}
I0114 01:48:26.300374    2828 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I0114 01:48:26.302175    2828 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0114 01:48:26.302175    2828 utils.go:79] GRPC request: {"publish_context":{"LUN":"1"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-76f33908-edbe-4dd1-9cae-af5a16ffd9e6\\globalmount","volume_capability":{"AccessType":{"Mount":{"mount_flags":["barrier=1","acl"]}},"access_mode":{"mode":7}},"volume_context":{"cachingmode":"None","csi.storage.k8s.io/pv/name":"pvc-76f33908-edbe-4dd1-9cae-af5a16ffd9e6","csi.storage.k8s.io/pvc/name":"pvc-hs4nh","csi.storage.k8s.io/pvc/namespace":"azuredisk-5211","maxshares":"2","requestedsizegib":"10","skuname":"Premium_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1673656805162-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-thn7c4/providers/Microsoft.Compute/disks/pvc-76f33908-edbe-4dd1-9cae-af5a16ffd9e6"}
E0114 01:48:26.302175    2828 utils.go:83] GRPC error: rpc error: code = Internal desc = failed to get current attachment for volume (/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-thn7c4/providers/Microsoft.Compute/disks/pvc-76f33908-edbe-4dd1-9cae-af5a16ffd9e6) and node (capz-thn7-slzjr): azvolumeattachment.disk.csi.azure.com "pvc-76f33908-edbe-4dd1-9cae-af5a16ffd9e6-capz-thn7-slzjr-attachment" not found
I0114 01:48:31.565639    2828 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I0114 01:48:31.565639    2828 utils.go:79] GRPC request: {}
I0114 01:48:31.566259    2828 utils.go:85] GRPC response: {"ready":{"value":true}}
I0114 01:48:33.661967    2828 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 01:48:33.698222    2828 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 01:48:33.987819    2828 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync
... skipping 5 lines ...
I0114 01:48:34.334638    2828 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I0114 01:48:34.336489    2828 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0114 01:48:34.336489    2828 utils.go:79] GRPC request: {}
I0114 01:48:34.336489    2828 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I0114 01:48:34.338168    2828 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0114 01:48:34.338168    2828 utils.go:79] GRPC request: {"publish_context":{"LUN":"1"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-76f33908-edbe-4dd1-9cae-af5a16ffd9e6\\globalmount","volume_capability":{"AccessType":{"Mount":{"mount_flags":["barrier=1","acl"]}},"access_mode":{"mode":7}},"volume_context":{"cachingmode":"None","csi.storage.k8s.io/pv/name":"pvc-76f33908-edbe-4dd1-9cae-af5a16ffd9e6","csi.storage.k8s.io/pvc/name":"pvc-hs4nh","csi.storage.k8s.io/pvc/namespace":"azuredisk-5211","maxshares":"2","requestedsizegib":"10","skuname":"Premium_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1673656805162-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-thn7c4/providers/Microsoft.Compute/disks/pvc-76f33908-edbe-4dd1-9cae-af5a16ffd9e6"}
E0114 01:48:34.338532    2828 utils.go:83] GRPC error: rpc error: code = Internal desc = failed to get current attachment for volume (/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-thn7c4/providers/Microsoft.Compute/disks/pvc-76f33908-edbe-4dd1-9cae-af5a16ffd9e6) and node (capz-thn7-slzjr): azvolumeattachment.disk.csi.azure.com "pvc-76f33908-edbe-4dd1-9cae-af5a16ffd9e6-capz-thn7-slzjr-attachment" not found
I0114 01:48:34.345433    2828 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 01:48:34.486249    2828 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 01:48:34.508086    2828 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-thn7-slzjr/status  in 18954 milliseconds
E0114 01:48:34.508392    2828 azuredisk_v2.go:613]  "msg"="Failed to update AzDriverNode status after creation" "error"="Put \"https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-thn7-slzjr/status\": read tcp 10.1.0.5:37450->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="830ab909-93ad-11ed-9dd6-6045bd608314"
E0114 01:48:34.508542    2828 azuredisk_v2.go:565]  "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [Put \"https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-thn7-slzjr/status\": read tcp 10.1.0.5:37450->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.]" "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="830ab909-93ad-11ed-9dd6-6045bd608314" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=18956515100
W0114 01:48:34.508764    2828 reflector.go:347] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: watch of *v1beta2.AzVolumeAttachment ended with: an error on the server ("unable to decode an event from the watch stream: read tcp 10.1.0.5:37450->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.") has prevented the request from succeeding
W0114 01:48:34.508815    2828 reflector.go:347] k8s.io/client-go/informers/factory.go:150: watch of *v1.Node ended with: an error on the server ("unable to decode an event from the watch stream: read tcp 10.1.0.5:37450->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.") has prevented the request from succeeding
W0114 01:48:34.508815    2828 reflector.go:347] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: watch of *v1beta2.AzDriverNode ended with: an error on the server ("unable to decode an event from the watch stream: read tcp 10.1.0.5:37450->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.") has prevented the request from succeeding
W0114 01:48:34.509356    2828 reflector.go:347] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: watch of *v1beta2.AzDriverNode ended with: an error on the server ("unable to decode an event from the watch stream: read tcp 10.1.0.5:37450->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.") has prevented the request from succeeding
W0114 01:48:34.509399    2828 reflector.go:347] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: watch of *v1beta2.AzVolume ended with: an error on the server ("unable to decode an event from the watch stream: read tcp 10.1.0.5:37450->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.") has prevented the request from succeeding
W0114 01:48:34.509871    2828 reflector.go:347] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: watch of *v1.CustomResourceDefinition ended with: an error on the server ("unable to decode an event from the watch stream: read tcp 10.1.0.5:37450->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.") has prevented the request from succeeding
I0114 01:48:35.329898    2828 reflector.go:257] Listing and watching *v1.Node from k8s.io/client-go/informers/factory.go:150
I0114 01:48:35.341644    2828 round_trippers.go:553] GET https://10.96.0.1:443/api/v1/nodes?resourceVersion=24566 200 OK in 10 milliseconds
I0114 01:48:35.346259    2828 reflector.go:257] Listing and watching *v1beta2.AzDriverNode from sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132
I0114 01:48:35.348209    2828 round_trippers.go:553] GET https://10.96.0.1:443/api/v1/nodes?allowWatchBookmarks=true&resourceVersion=24825&timeout=5m29s&timeoutSeconds=329&watch=true 200 OK in 1 milliseconds
I0114 01:48:35.348778    2828 round_trippers.go:553] GET https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/azdrivernodes?resourceVersion=24647 200 OK in 2 milliseconds
I0114 01:48:35.352090    2828 round_trippers.go:553] GET https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/azdrivernodes?allowWatchBookmarks=true&resourceVersion=24902&timeout=8m59s&timeoutSeconds=539&watch=true 200 OK in 2 milliseconds
... skipping 450 lines ...
I0114 02:01:04.400506    2828 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0114 02:01:04.400506    2828 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-0f3b7735-5ade-4834-a5fd-5f159b593c6f\\globalmount","volume_capability":{"AccessType":{"Mount":{"mount_flags":["barrier=1","acl"]}},"access_mode":{"mode":7}},"volume_context":{"cachingmode":"None","csi.storage.k8s.io/pv/name":"pvc-0f3b7735-5ade-4834-a5fd-5f159b593c6f","csi.storage.k8s.io/pvc/name":"pvc-s4pcr","csi.storage.k8s.io/pvc/namespace":"azuredisk-8231","maxshares":"2","requestedsizegib":"10","skuname":"Premium_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1673656805162-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-thn7c4/providers/Microsoft.Compute/disks/pvc-0f3b7735-5ade-4834-a5fd-5f159b593c6f"}
I0114 02:01:04.401176    2828 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-0f3b7735-5ade-4834-a5fd-5f159b593c6f-capz-thn7-slzjr-attachment)
I0114 02:01:05.570604    2828 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 02:01:05.811549    2828 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 02:01:05.915750    2828 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 02:01:05.916137    2828 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0114 02:01:05.916647    2828 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync
I0114 02:01:06.066432    2828 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 02:01:12.613626    2828 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0114 02:01:12.613777    2828 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="4d502219-93af-11ed-9dd6-6045bd608314" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-0f3b7735-5ade-4834-a5fd-5f159b593c6f" "latency"=8212130100
I0114 02:01:12.613777    2828 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="4d502219-93af-11ed-9dd6-6045bd608314" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=8212431800
I0114 02:01:15.583456    2828 azuredisk_v2.go:578]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-14T02:01:15Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="53fa65f6-93af-11ed-9dd6-6045bd608314"
I0114 02:01:15.598347    2828 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-thn7-slzjr/status 200 OK in 14 milliseconds
I0114 02:01:15.598808    2828 azuredisk_v2.go:565]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="53fa65f6-93af-11ed-9dd6-6045bd608314" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=15623600
I0114 02:01:17.458868    2828 nodeserver_v2.go:211] NodeStageVolume: formatting 10 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-0f3b7735-5ade-4834-a5fd-5f159b593c6f\globalmount with mount options([barrier=1 acl])
... skipping 147 lines ...
I0114 02:05:33.029033    2828 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-72cb0fd3-460c-47f7-b549-477e22f09c6f\\globalmount","volume_capability":{"AccessType":{"Mount":{"mount_flags":["barrier=1","acl"]}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-72cb0fd3-460c-47f7-b549-477e22f09c6f","csi.storage.k8s.io/pvc/name":"pvc-r5tt2","csi.storage.k8s.io/pvc/namespace":"azuredisk-4765","requestedsizegib":"10","skuname":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1673656805162-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-thn7c4/providers/Microsoft.Compute/disks/pvc-72cb0fd3-460c-47f7-b549-477e22f09c6f"}
I0114 02:05:33.029852    2828 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-72cb0fd3-460c-47f7-b549-477e22f09c6f-capz-thn7-slzjr-attachment)
I0114 02:05:35.645223    2828 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 02:05:35.894126    2828 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 02:05:36.018170    2828 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 02:05:36.018393    2828 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync
I0114 02:05:36.018393    2828 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0114 02:05:36.164240    2828 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 02:05:42.838310    2828 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0114 02:05:42.838888    2828 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="ed6da29e-93af-11ed-9dd6-6045bd608314" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-72cb0fd3-460c-47f7-b549-477e22f09c6f" "latency"=9807732100
I0114 02:05:42.838888    2828 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="ed6da29e-93af-11ed-9dd6-6045bd608314" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9807732100
I0114 02:05:45.623215    2828 azuredisk_v2.go:578]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-14T02:05:45Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="f4ef3add-93af-11ed-9dd6-6045bd608314"
I0114 02:05:45.631751    2828 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-thn7-slzjr/status 200 OK in 8 milliseconds
I0114 02:05:45.631751    2828 azuredisk_v2.go:565]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="f4ef3add-93af-11ed-9dd6-6045bd608314" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=8570500
I0114 02:05:47.752035    2828 nodeserver_v2.go:211] NodeStageVolume: formatting 10 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-72cb0fd3-460c-47f7-b549-477e22f09c6f\globalmount with mount options([barrier=1 acl])
... skipping 290 lines ...
I0114 02:15:23.982718    2828 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0114 02:15:23.982718    2828 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-6b0a0ff1-1f55-4f03-a4b2-703bef3a74ed\\globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-6b0a0ff1-1f55-4f03-a4b2-703bef3a74ed","csi.storage.k8s.io/pvc/name":"pvc-g5fks","csi.storage.k8s.io/pvc/namespace":"azuredisk-5425","requestedsizegib":"10","skuname":"Standard_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1673656805162-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-thn7c4/providers/Microsoft.Compute/disks/pvc-6b0a0ff1-1f55-4f03-a4b2-703bef3a74ed"}
I0114 02:15:23.983374    2828 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-6b0a0ff1-1f55-4f03-a4b2-703bef3a74ed-capz-thn7-slzjr-attachment)
I0114 02:15:31.669415    2828 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I0114 02:15:31.669500    2828 utils.go:79] GRPC request: {}
I0114 02:15:31.669571    2828 utils.go:85] GRPC response: {"ready":{"value":true}}
I0114 02:15:33.656997    2828 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0114 02:15:33.657095    2828 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="4da9de7d-93b1-11ed-9dd6-6045bd608314" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-6ef9816c-2e3d-416a-942c-f25a0af13d9d" "latency"=9674077500
I0114 02:15:33.657095    2828 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="4da9de7d-93b1-11ed-9dd6-6045bd608314" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9674191300
I0114 02:15:33.661756    2828 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0114 02:15:33.661756    2828 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="4da9e2ef-93b1-11ed-9dd6-6045bd608314" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-5beae643-7d84-4005-b845-a438d65b72f6" "latency"=9678738700
I0114 02:15:33.661756    2828 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="4da9e2ef-93b1-11ed-9dd6-6045bd608314" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9678738700
I0114 02:15:33.661993    2828 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0114 02:15:33.662170    2828 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="4da9f79e-93b1-11ed-9dd6-6045bd608314" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-6b0a0ff1-1f55-4f03-a4b2-703bef3a74ed" "latency"=9678496700
I0114 02:15:33.662170    2828 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="4da9f79e-93b1-11ed-9dd6-6045bd608314" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9678622900
I0114 02:15:35.843044    2828 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 02:15:36.072816    2828 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 02:15:36.263624    2828 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 02:15:36.263624    2828 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync
... skipping 103 lines ...
I0114 02:16:33.220717    2828 utils.go:85] GRPC response: {}
I0114 02:16:33.235188    2828 nodeserver_v2.go:262] NodeUnstageVolume: unmount \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-6b0a0ff1-1f55-4f03-a4b2-703bef3a74ed\globalmount successfully
I0114 02:16:33.235188    2828 utils.go:85] GRPC response: {}
I0114 02:16:33.269718    2828 nodeserver_v2.go:262] NodeUnstageVolume: unmount \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-5beae643-7d84-4005-b845-a438d65b72f6\globalmount successfully
I0114 02:16:33.269718    2828 utils.go:85] GRPC response: {}
I0114 02:16:34.613369    2828 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-thn7-slzjr/status  in 18957 milliseconds
W0114 02:16:34.613412    2828 reflector.go:347] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: watch of *v1.CustomResourceDefinition ended with: an error on the server ("unable to decode an event from the watch stream: read tcp 10.1.0.5:39370->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.") has prevented the request from succeeding
W0114 02:16:34.613412    2828 reflector.go:347] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: watch of *v1beta2.AzVolumeAttachment ended with: an error on the server ("unable to decode an event from the watch stream: read tcp 10.1.0.5:39370->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.") has prevented the request from succeeding
W0114 02:16:34.613412    2828 reflector.go:347] k8s.io/client-go/informers/factory.go:150: watch of *v1.Node ended with: an error on the server ("unable to decode an event from the watch stream: read tcp 10.1.0.5:39370->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.") has prevented the request from succeeding
E0114 02:16:34.613412    2828 azuredisk_v2.go:613]  "msg"="Failed to update AzDriverNode status after creation" "error"="Put \"https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-thn7-slzjr/status\": read tcp 10.1.0.5:39370->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="6c7530c9-93b1-11ed-9dd6-6045bd608314"
W0114 02:16:34.613412    2828 reflector.go:347] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: watch of *v1beta2.AzVolume ended with: an error on the server ("unable to decode an event from the watch stream: read tcp 10.1.0.5:39370->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.") has prevented the request from succeeding
W0114 02:16:34.613412    2828 reflector.go:347] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: watch of *v1beta2.AzDriverNode ended with: an error on the server ("unable to decode an event from the watch stream: read tcp 10.1.0.5:39370->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.") has prevented the request from succeeding
W0114 02:16:34.613412    2828 reflector.go:347] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: watch of *v1beta2.AzDriverNode ended with: an error on the server ("unable to decode an event from the watch stream: read tcp 10.1.0.5:39370->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.") has prevented the request from succeeding
E0114 02:16:34.613657    2828 azuredisk_v2.go:565]  "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [Put \"https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-thn7-slzjr/status\": read tcp 10.1.0.5:39370->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.]" "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="6c7530c9-93b1-11ed-9dd6-6045bd608314" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=18966013700
I0114 02:16:35.671048    2828 reflector.go:257] Listing and watching *v1beta2.AzDriverNode from sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132
I0114 02:16:35.686264    2828 round_trippers.go:553] GET https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/azdrivernodes?resourceVersion=32840 200 OK in 14 milliseconds
I0114 02:16:35.688650    2828 round_trippers.go:553] GET https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/azdrivernodes?allowWatchBookmarks=true&resourceVersion=32976&timeout=6m19s&timeoutSeconds=379&watch=true 200 OK in 2 milliseconds
I0114 02:16:35.712902    2828 reflector.go:257] Listing and watching *v1.Node from k8s.io/client-go/informers/factory.go:150
I0114 02:16:35.716393    2828 round_trippers.go:553] GET https://10.96.0.1:443/api/v1/nodes?resourceVersion=32656 200 OK in 2 milliseconds
I0114 02:16:35.721867    2828 round_trippers.go:553] GET https://10.96.0.1:443/api/v1/nodes?allowWatchBookmarks=true&resourceVersion=32983&timeout=8m50s&timeoutSeconds=530&watch=true 200 OK in 2 milliseconds
... skipping 67 lines ...
I0114 02:17:37.556709    2828 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0114 02:17:37.556709    2828 utils.go:79] GRPC request: {"publish_context":{"LUN":"1"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-4974a98c-1a4a-4ee4-878e-1b433fa16dab\\globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-4974a98c-1a4a-4ee4-878e-1b433fa16dab","csi.storage.k8s.io/pvc/name":"test-volume-1-azuredisk-volume-tester-mws2q-0","csi.storage.k8s.io/pvc/namespace":"azuredisk-9681","requestedsizegib":"10","skuname":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1673656805162-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-thn7c4/providers/Microsoft.Compute/disks/pvc-4974a98c-1a4a-4ee4-878e-1b433fa16dab"}
I0114 02:17:37.557674    2828 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-4974a98c-1a4a-4ee4-878e-1b433fa16dab-capz-thn7-slzjr-attachment)
I0114 02:17:45.644523    2828 azuredisk_v2.go:578]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-14T02:17:45Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="a219af63-93b1-11ed-9dd6-6045bd608314"
I0114 02:17:45.653885    2828 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-thn7-slzjr/status 200 OK in 8 milliseconds
I0114 02:17:45.653885    2828 azuredisk_v2.go:565]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="a219af63-93b1-11ed-9dd6-6045bd608314" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=9868700
I0114 02:17:47.288171    2828 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0114 02:17:47.290758    2828 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="9d375b71-93b1-11ed-9dd6-6045bd608314" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-eab89b3d-16f7-49ac-b7c4-65b66f0d68e3" "latency"=9838023300
I0114 02:17:47.290758    2828 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="9d375b71-93b1-11ed-9dd6-6045bd608314" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9840610500
I0114 02:17:47.293887    2828 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0114 02:17:47.294083    2828 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="9d47cab4-93b1-11ed-9dd6-6045bd608314" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-4974a98c-1a4a-4ee4-878e-1b433fa16dab" "latency"=9735890000
I0114 02:17:47.294176    2828 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="9d47cab4-93b1-11ed-9dd6-6045bd608314" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9736232100
I0114 02:17:47.294322    2828 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0114 02:17:47.294450    2828 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="9d377af8-93b1-11ed-9dd6-6045bd608314" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-dd773983-b0e8-4e84-88e3-e395b8b19040" "latency"=9843258500
I0114 02:17:47.294517    2828 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="9d377af8-93b1-11ed-9dd6-6045bd608314" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9843495500
I0114 02:17:53.784453    2828 nodeserver_v2.go:211] NodeStageVolume: formatting 12 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-dd773983-b0e8-4e84-88e3-e395b8b19040\globalmount with mount options([])
I0114 02:17:53.795594    2828 nodeserver_v2.go:211] NodeStageVolume: formatting 11 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-eab89b3d-16f7-49ac-b7c4-65b66f0d68e3\globalmount with mount options([])
I0114 02:17:53.921542    2828 nodeserver_v2.go:211] NodeStageVolume: formatting 10 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-4974a98c-1a4a-4ee4-878e-1b433fa16dab\globalmount with mount options([])
I0114 02:18:01.699140    2828 utils.go:78] GRPC call: /csi.v1.Identity/Probe
... skipping 609 lines ...
I0114 02:23:44.127216    2828 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0114 02:23:44.127216    2828 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-8de4b674-c12b-4efe-9876-3fc13a92da14\\globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"cachingmode":"None","csi.storage.k8s.io/pv/name":"pvc-8de4b674-c12b-4efe-9876-3fc13a92da14","csi.storage.k8s.io/pvc/name":"test-volume-1-azuredisk-volume-tester-qh4sc-0","csi.storage.k8s.io/pvc/namespace":"azuredisk-2388","maxShares":"2","requestedsizegib":"256","skuname":"Premium_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1673656805162-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-thn7c4/providers/Microsoft.Compute/disks/pvc-8de4b674-c12b-4efe-9876-3fc13a92da14"}
I0114 02:23:44.127915    2828 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-8de4b674-c12b-4efe-9876-3fc13a92da14-capz-thn7-slzjr-attachment)
I0114 02:23:45.673341    2828 azuredisk_v2.go:578]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-14T02:23:45Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="78b1c81a-93b2-11ed-9dd6-6045bd608314"
I0114 02:23:45.693951    2828 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-thn7-slzjr/status 200 OK in 20 milliseconds
I0114 02:23:45.693951    2828 azuredisk_v2.go:565]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="78b1c81a-93b2-11ed-9dd6-6045bd608314" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=20775300
I0114 02:23:52.366444    2828 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0114 02:23:52.366528    2828 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="77c5f903-93b2-11ed-9dd6-6045bd608314" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-8de4b674-c12b-4efe-9876-3fc13a92da14" "latency"=8237785800
I0114 02:23:52.366528    2828 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="77c5f903-93b2-11ed-9dd6-6045bd608314" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=8237880600
I0114 02:23:52.367476    2828 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0114 02:23:52.367476    2828 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="77c5b9cc-93b2-11ed-9dd6-6045bd608314" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-2ee5b23e-ed42-4a06-8a29-3e21c35f47fe" "latency"=8240175400
I0114 02:23:52.367476    2828 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="77c5b9cc-93b2-11ed-9dd6-6045bd608314" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=8240446700
I0114 02:24:01.567013    2828 nodeserver_v2.go:211] NodeStageVolume: formatting 11 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-2ee5b23e-ed42-4a06-8a29-3e21c35f47fe\globalmount with mount options([])
I0114 02:24:01.567013    2828 nodeserver_v2.go:211] NodeStageVolume: formatting 10 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-8de4b674-c12b-4efe-9876-3fc13a92da14\globalmount with mount options([])
I0114 02:24:01.729326    2828 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I0114 02:24:01.729326    2828 utils.go:79] GRPC request: {}
... skipping 474 lines ...
I0114 02:30:42.727464    2828 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0114 02:30:42.728010    2828 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-9ca63fc4-8fdd-47f0-a8be-d19a05379e5d\\globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-9ca63fc4-8fdd-47f0-a8be-d19a05379e5d","csi.storage.k8s.io/pvc/name":"persistent-storage-statefulset-azuredisk-win-0","csi.storage.k8s.io/pvc/namespace":"default","requestedsizegib":"100","skuName":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1673656805162-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-thn7c4/providers/Microsoft.Compute/disks/pvc-9ca63fc4-8fdd-47f0-a8be-d19a05379e5d"}
I0114 02:30:42.728658    2828 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-9ca63fc4-8fdd-47f0-a8be-d19a05379e5d-capz-thn7-slzjr-attachment)
I0114 02:30:45.714586    2828 azuredisk_v2.go:578]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-14T02:30:45Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="730efb26-93b3-11ed-9dd6-6045bd608314"
I0114 02:30:45.724356    2828 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-thn7-slzjr/status 200 OK in 9 milliseconds
I0114 02:30:45.724356    2828 azuredisk_v2.go:565]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="730efb26-93b3-11ed-9dd6-6045bd608314" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=9945800
I0114 02:30:49.649040    2828 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0114 02:30:49.649245    2828 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-thn7-slzjr" "disk.csi.azure.com/request-id"="71476439-93b3-11ed-9dd6-6045bd608314" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-9ca63fc4-8fdd-47f0-a8be-d19a05379e5d" "latency"=6920075500
I0114 02:30:49.649325    2828 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="71476439-93b3-11ed-9dd6-6045bd608314" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=6920360100
I0114 02:30:54.797850    2828 nodeserver_v2.go:211] NodeStageVolume: formatting 10 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-9ca63fc4-8fdd-47f0-a8be-d19a05379e5d\globalmount with mount options([])
I0114 02:31:01.725189    2828 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I0114 02:31:01.725189    2828 utils.go:79] GRPC request: {}
I0114 02:31:01.725275    2828 utils.go:85] GRPC response: {"ready":{"value":true}}
... skipping 67 lines ...
I0114 00:40:21.678167    3488 round_trippers.go:553] GET https://10.96.0.1:443/apis/apiextensions.k8s.io/v1/customresourcedefinitions?limit=500&resourceVersion=0 200 OK in 32 milliseconds
I0114 00:40:21.718679    3488 round_trippers.go:553] GET https://10.96.0.1:443/apis/apiextensions.k8s.io/v1/customresourcedefinitions?allowWatchBookmarks=true&resourceVersion=3845&timeout=8m11s&timeoutSeconds=491&watch=true 200 OK in 2 milliseconds
I0114 00:40:21.738184    3488 shared_informer.go:303] caches populated
I0114 00:40:21.739908    3488 azuredisk_v2.go:225] driver userAgent: disk.csi.azure.com/latest-v2-9ef068a8cb36a997d4ea04b90c05c6f92a488a19 e2e-test
I0114 00:40:21.739942    3488 azure_disk_utils.go:564] reading cloud config from secret kube-system/azure-cloud-provider
I0114 00:40:21.744686    3488 round_trippers.go:553] GET https://10.96.0.1:443/api/v1/namespaces/kube-system/secrets/azure-cloud-provider 404 Not Found in 3 milliseconds
I0114 00:40:21.745189    3488 azure_disk_utils.go:571] InitializeCloudFromSecret: failed to get cloud config from secret kube-system/azure-cloud-provider: failed to get secret kube-system/azure-cloud-provider: secrets "azure-cloud-provider" not found
I0114 00:40:21.745189    3488 azure_disk_utils.go:576] could not read cloud config from secret kube-system/azure-cloud-provider
I0114 00:40:21.745189    3488 azure_disk_utils.go:586] use default AZURE_CREDENTIAL_FILE env var: C:\k\azure.json
I0114 00:40:21.745689    3488 azure_disk_utils.go:594] read cloud config from file: C:\k\azure.json successfully
I0114 00:40:21.749184    3488 azure_auth.go:253] Using AzurePublicCloud environment
I0114 00:40:21.749695    3488 azure_auth.go:138] azure: using client_id+client_secret to retrieve access token
I0114 00:40:21.753179    3488 azure_diskclient.go:68] Azure DisksClient using API version: 2022-03-02
... skipping 33 lines ...
I0114 00:40:22.575406    3488 utils.go:85] GRPC response: {"name":"disk.csi.azure.com","vendor_version":"latest-v2-9ef068a8cb36a997d4ea04b90c05c6f92a488a19"}
I0114 00:40:22.788531    3488 utils.go:78] GRPC call: /csi.v1.Identity/GetPluginInfo
I0114 00:40:22.788531    3488 utils.go:79] GRPC request: {}
I0114 00:40:22.788531    3488 utils.go:85] GRPC response: {"name":"disk.csi.azure.com","vendor_version":"latest-v2-9ef068a8cb36a997d4ea04b90c05c6f92a488a19"}
I0114 00:40:23.425257    3488 utils.go:78] GRPC call: /csi.v1.Node/NodeGetInfo
I0114 00:40:23.425257    3488 utils.go:79] GRPC request: {}
I0114 00:40:23.451493    3488 azure_instance_metadata.go:140] Warning: failed to get loadbalancer metadata: failure of getting loadbalancer metadata with response "404 Not Found"
I0114 00:40:23.451493    3488 nodeserver_v2.go:438] NodeGetInfo: nodeName(capz-thn7-fl7d5), VM Size(Standard_D4s_v3)
I0114 00:40:23.451757    3488 nodeserver.go:409] got a matching size in getMaxDataDiskCount, VM Size: STANDARD_D4S_V3, MaxDataDiskCount: 8
I0114 00:40:23.451757    3488 utils.go:85] GRPC response: {"accessible_topology":{"segments":{"topology.disk.csi.azure.com/zone":""}},"max_volumes_per_node":8,"node_id":"capz-thn7-fl7d5"}
I0114 00:40:51.684773    3488 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 00:40:51.684773    3488 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 00:40:51.684773    3488 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
... skipping 777 lines ...
I0114 01:13:42.092748    3488 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0114 01:13:42.092748    3488 utils.go:79] GRPC request: {}
I0114 01:13:42.092748    3488 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I0114 01:13:42.094759    3488 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0114 01:13:42.094759    3488 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-ffdb31a0-a4c4-4951-9364-bedd15b4eebe\\globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-ffdb31a0-a4c4-4951-9364-bedd15b4eebe","csi.storage.k8s.io/pvc/name":"pvc-v9n6q","csi.storage.k8s.io/pvc/namespace":"azuredisk-3957","requestedsizegib":"10","skuname":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1673656805162-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-thn7c4/providers/Microsoft.Compute/disks/pvc-ffdb31a0-a4c4-4951-9364-bedd15b4eebe"}
I0114 01:13:42.100665    3488 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-ffdb31a0-a4c4-4951-9364-bedd15b4eebe-capz-thn7-fl7d5-attachment)
I0114 01:13:46.801036    3488 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0114 01:13:46.801036    3488 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-thn7-fl7d5" "disk.csi.azure.com/request-id"="af2aaf07-93a8-11ed-81d6-002248b31186" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-ffdb31a0-a4c4-4951-9364-bedd15b4eebe" "latency"=4699580500
I0114 01:13:46.801036    3488 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="af2aaf07-93a8-11ed-81d6-002248b31186" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=4705288800
I0114 01:13:52.415677    3488 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 01:13:52.415677    3488 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 01:13:52.415677    3488 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync
I0114 01:13:52.415897    3488 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
... skipping 667 lines ...
I0114 01:39:45.025890    3488 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0114 01:39:45.025890    3488 utils.go:79] GRPC request: {}
I0114 01:39:45.025890    3488 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I0114 01:39:45.027394    3488 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0114 01:39:45.027394    3488 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-ca224d18-e297-4ff6-87d1-108549471be4\\globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"cachingmode":"None","csi.storage.k8s.io/pv/name":"pvc-ca224d18-e297-4ff6-87d1-108549471be4","csi.storage.k8s.io/pvc/name":"pvc-9t9dh","csi.storage.k8s.io/pvc/namespace":"azuredisk-6068","maxshares":"2","requestedsizegib":"10","skuname":"Premium_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1673656805162-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-thn7c4/providers/Microsoft.Compute/disks/pvc-ca224d18-e297-4ff6-87d1-108549471be4"}
I0114 01:39:45.027394    3488 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-ca224d18-e297-4ff6-87d1-108549471be4-capz-thn7-fl7d5-attachment)
I0114 01:39:51.752778    3488 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0114 01:39:51.752778    3488 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-thn7-fl7d5" "disk.csi.azure.com/request-id"="52bf3497-93ac-11ed-81d6-002248b31186" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-ca224d18-e297-4ff6-87d1-108549471be4" "latency"=6725141100
I0114 01:39:51.752778    3488 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="52bf3497-93ac-11ed-81d6-002248b31186" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=6725141100
I0114 01:39:52.967828    3488 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 01:39:52.967828    3488 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync
I0114 01:39:52.967828    3488 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 01:39:52.968045    3488 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
... skipping 46 lines ...
I0114 01:40:52.978854    3488 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 01:40:52.978854    3488 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 01:40:52.978854    3488 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 01:40:52.978854    3488 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 01:40:52.978854    3488 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync
I0114 01:40:54.215563    3488 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-thn7-fl7d5/status  in 18995 milliseconds
W0114 01:40:54.216736    3488 reflector.go:347] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: watch of *v1beta2.AzVolume ended with: an error on the server ("unable to decode an event from the watch stream: read tcp 10.1.0.6:34615->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.") has prevented the request from succeeding
W0114 01:40:54.217228    3488 reflector.go:347] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: watch of *v1beta2.AzDriverNode ended with: an error on the server ("unable to decode an event from the watch stream: read tcp 10.1.0.6:34615->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.") has prevented the request from succeeding
W0114 01:40:54.217228    3488 reflector.go:347] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: watch of *v1beta2.AzDriverNode ended with: an error on the server ("unable to decode an event from the watch stream: read tcp 10.1.0.6:34615->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.") has prevented the request from succeeding
W0114 01:40:54.217228    3488 reflector.go:347] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: watch of *v1.CustomResourceDefinition ended with: an error on the server ("unable to decode an event from the watch stream: read tcp 10.1.0.6:34615->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.") has prevented the request from succeeding
W0114 01:40:54.217228    3488 reflector.go:347] k8s.io/client-go/informers/factory.go:150: watch of *v1.Node ended with: an error on the server ("unable to decode an event from the watch stream: read tcp 10.1.0.6:34615->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.") has prevented the request from succeeding
W0114 01:40:54.217228    3488 reflector.go:347] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: watch of *v1beta2.AzVolumeAttachment ended with: an error on the server ("unable to decode an event from the watch stream: read tcp 10.1.0.6:34615->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.") has prevented the request from succeeding
E0114 01:40:54.217730    3488 azuredisk_v2.go:613]  "msg"="Failed to update AzDriverNode status after creation" "error"="Put \"https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-thn7-fl7d5/status\": read tcp 10.1.0.6:34615->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-thn7-fl7d5" "disk.csi.azure.com/request-id"="70a9d474-93ac-11ed-81d6-002248b31186"
E0114 01:40:54.217730    3488 azuredisk_v2.go:565]  "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [Put \"https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-thn7-fl7d5/status\": read tcp 10.1.0.6:34615->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.]" "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-thn7-fl7d5" "disk.csi.azure.com/request-id"="70a9d474-93ac-11ed-81d6-002248b31186" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=18997972800
I0114 01:40:55.122069    3488 reflector.go:257] Listing and watching *v1beta2.AzVolume from sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132
I0114 01:40:55.143578    3488 round_trippers.go:553] GET https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumes?resourceVersion=22082 200 OK in 21 milliseconds
I0114 01:40:55.146558    3488 round_trippers.go:553] GET https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumes?allowWatchBookmarks=true&resourceVersion=22446&timeout=5m38s&timeoutSeconds=338&watch=true 200 OK in 1 milliseconds
I0114 01:40:55.174159    3488 reflector.go:257] Listing and watching *v1beta2.AzDriverNode from sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132
I0114 01:40:55.176715    3488 round_trippers.go:553] GET https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/azdrivernodes?resourceVersion=22276 200 OK in 2 milliseconds
I0114 01:40:55.179069    3488 round_trippers.go:553] GET https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/azdrivernodes?allowWatchBookmarks=true&resourceVersion=22425&timeout=5m55s&timeoutSeconds=355&watch=true 200 OK in 1 milliseconds
... skipping 94 lines ...
I0114 01:44:11.810296    3488 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-602bc706-e416-41fb-8ee1-827ee430fd9d\\globalmount","volume_capability":{"AccessType":{"Mount":{"mount_flags":["barrier=1","acl"]}},"access_mode":{"mode":7}},"volume_context":{"cachingmode":"None","csi.storage.k8s.io/pv/name":"pvc-602bc706-e416-41fb-8ee1-827ee430fd9d","csi.storage.k8s.io/pvc/name":"pvc-4dxgz","csi.storage.k8s.io/pvc/namespace":"azuredisk-482","maxshares":"2","requestedsizegib":"10","skuname":"Premium_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1673656805162-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-thn7c4/providers/Microsoft.Compute/disks/pvc-602bc706-e416-41fb-8ee1-827ee430fd9d"}
I0114 01:44:11.810296    3488 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-602bc706-e416-41fb-8ee1-827ee430fd9d-capz-thn7-fl7d5-attachment)
I0114 01:44:25.192177    3488 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 01:44:25.253703    3488 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 01:44:25.269165    3488 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 01:44:25.423415    3488 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 01:44:25.423519    3488 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0114 01:44:25.939251    3488 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync
I0114 01:44:26.695916    3488 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0114 01:44:26.695916    3488 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-thn7-fl7d5" "disk.csi.azure.com/request-id"="f1c30bb0-93ac-11ed-81d6-002248b31186" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-602bc706-e416-41fb-8ee1-827ee430fd9d" "latency"=14884042100
I0114 01:44:26.695916    3488 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="f1c30bb0-93ac-11ed-81d6-002248b31186" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=14884042100
I0114 01:44:31.241485    3488 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I0114 01:44:31.241485    3488 utils.go:79] GRPC request: {}
I0114 01:44:31.241485    3488 utils.go:85] GRPC response: {"ready":{"value":true}}
I0114 01:44:31.613025    3488 nodeserver_v2.go:211] NodeStageVolume: formatting 10 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-602bc706-e416-41fb-8ee1-827ee430fd9d\globalmount with mount options([barrier=1 acl])
... skipping 60 lines ...
I0114 01:45:44.248276    3488 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0114 01:45:44.248276    3488 utils.go:79] GRPC request: {}
I0114 01:45:44.248276    3488 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I0114 01:45:44.249774    3488 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0114 01:45:44.249774    3488 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-6d3c8697-81a6-4653-957a-ca98659fe6d0\\globalmount","volume_capability":{"AccessType":{"Mount":{"mount_flags":["barrier=1","acl"]}},"access_mode":{"mode":7}},"volume_context":{"cachingmode":"None","csi.storage.k8s.io/pv/name":"pvc-6d3c8697-81a6-4653-957a-ca98659fe6d0","csi.storage.k8s.io/pvc/name":"pvc-j8tb5","csi.storage.k8s.io/pvc/namespace":"azuredisk-3950","maxshares":"2","requestedsizegib":"10","skuname":"Premium_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1673656805162-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-thn7c4/providers/Microsoft.Compute/disks/pvc-6d3c8697-81a6-4653-957a-ca98659fe6d0"}
I0114 01:45:44.250270    3488 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-6d3c8697-81a6-4653-957a-ca98659fe6d0-capz-thn7-fl7d5-attachment)
I0114 01:45:54.335716    3488 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0114 01:45:54.335716    3488 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-thn7-fl7d5" "disk.csi.azure.com/request-id"="28dc442f-93ad-11ed-81d6-002248b31186" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-6d3c8697-81a6-4653-957a-ca98659fe6d0" "latency"=10084240000
I0114 01:45:54.336227    3488 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="28dc442f-93ad-11ed-81d6-002248b31186" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=10084750700
I0114 01:45:55.232887    3488 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 01:45:55.279551    3488 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 01:45:55.281885    3488 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 01:45:55.455073    3488 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
... skipping 412 lines ...
I0114 02:01:54.255163    3488 utils.go:79] GRPC request: {"publish_context":{"LUN":"1"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-fbf75c70-754d-41bc-b42e-9b87d71b5f82\\globalmount","volume_capability":{"AccessType":{"Mount":{"mount_flags":["barrier=1","acl"]}},"access_mode":{"mode":7}},"volume_context":{"cachingmode":"None","csi.storage.k8s.io/pv/name":"pvc-fbf75c70-754d-41bc-b42e-9b87d71b5f82","csi.storage.k8s.io/pvc/name":"pvc-q6wx5","csi.storage.k8s.io/pvc/namespace":"azuredisk-8231","maxshares":"2","requestedsizegib":"10","skuname":"Premium_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1673656805162-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-thn7c4/providers/Microsoft.Compute/disks/pvc-fbf75c70-754d-41bc-b42e-9b87d71b5f82"}
I0114 02:01:54.255680    3488 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-fbf75c70-754d-41bc-b42e-9b87d71b5f82-capz-thn7-fl7d5-attachment)
I0114 02:01:55.568878    3488 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 02:01:55.568878    3488 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 02:01:55.568878    3488 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 02:01:55.740130    3488 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 02:01:55.740130    3488 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0114 02:01:56.257517    3488 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync
I0114 02:02:01.322999    3488 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I0114 02:02:01.322999    3488 utils.go:79] GRPC request: {}
I0114 02:02:01.322999    3488 utils.go:85] GRPC response: {"ready":{"value":true}}
I0114 02:02:04.849192    3488 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0114 02:02:04.849652    3488 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-thn7-fl7d5" "disk.csi.azure.com/request-id"="6b075882-93af-11ed-81d6-002248b31186" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-fbf75c70-754d-41bc-b42e-9b87d71b5f82" "latency"=10591908300
I0114 02:02:04.849652    3488 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="6b075882-93af-11ed-81d6-002248b31186" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=10592368900
I0114 02:02:05.316025    3488 azuredisk_v2.go:578]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-14T02:02:05Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-thn7-fl7d5" "disk.csi.azure.com/request-id"="719f053a-93af-11ed-81d6-002248b31186"
I0114 02:02:05.323903    3488 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-thn7-fl7d5/status 200 OK in 7 milliseconds
I0114 02:02:05.324184    3488 azuredisk_v2.go:565]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-thn7-fl7d5" "disk.csi.azure.com/request-id"="719f053a-93af-11ed-81d6-002248b31186" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=8158100
I0114 02:02:09.964535    3488 nodeserver_v2.go:211] NodeStageVolume: formatting 11 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-fbf75c70-754d-41bc-b42e-9b87d71b5f82\globalmount with mount options([barrier=1 acl])
... skipping 36 lines ...
I0114 02:02:45.630282    3488 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I0114 02:02:45.632293    3488 utils.go:78] GRPC call: /csi.v1.Node/NodeUnstageVolume
I0114 02:02:45.632293    3488 utils.go:79] GRPC request: {"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-fbf75c70-754d-41bc-b42e-9b87d71b5f82\\globalmount","volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-thn7c4/providers/Microsoft.Compute/disks/pvc-fbf75c70-754d-41bc-b42e-9b87d71b5f82"}
I0114 02:02:45.632293    3488 nodeserver_v2.go:257] NodeUnstageVolume: unmounting \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-fbf75c70-754d-41bc-b42e-9b87d71b5f82\globalmount
I0114 02:02:48.374487    3488 nodeserver_v2.go:262] NodeUnstageVolume: unmount \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-fbf75c70-754d-41bc-b42e-9b87d71b5f82\globalmount successfully
I0114 02:02:48.374487    3488 utils.go:85] GRPC response: {}
W0114 02:02:54.298560    3488 reflector.go:347] k8s.io/client-go/informers/factory.go:150: watch of *v1.Node ended with: an error on the server ("unable to decode an event from the watch stream: read tcp 10.1.0.6:35388->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.") has prevented the request from succeeding
W0114 02:02:54.298560    3488 reflector.go:347] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: watch of *v1.CustomResourceDefinition ended with: an error on the server ("unable to decode an event from the watch stream: read tcp 10.1.0.6:35388->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.") has prevented the request from succeeding
W0114 02:02:54.298560    3488 reflector.go:347] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: watch of *v1beta2.AzVolume ended with: an error on the server ("unable to decode an event from the watch stream: read tcp 10.1.0.6:35388->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.") has prevented the request from succeeding
W0114 02:02:54.298560    3488 reflector.go:347] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: watch of *v1beta2.AzDriverNode ended with: an error on the server ("unable to decode an event from the watch stream: read tcp 10.1.0.6:35388->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.") has prevented the request from succeeding
I0114 02:02:54.298755    3488 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-thn7-fl7d5/status  in 18973 milliseconds
W0114 02:02:54.298831    3488 reflector.go:347] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: watch of *v1beta2.AzDriverNode ended with: an error on the server ("unable to decode an event from the watch stream: read tcp 10.1.0.6:35388->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.") has prevented the request from succeeding
W0114 02:02:54.298831    3488 reflector.go:347] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: watch of *v1beta2.AzVolumeAttachment ended with: an error on the server ("unable to decode an event from the watch stream: read tcp 10.1.0.6:35388->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.") has prevented the request from succeeding
E0114 02:02:54.298921    3488 azuredisk_v2.go:613]  "msg"="Failed to update AzDriverNode status after creation" "error"="Put \"https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-thn7-fl7d5/status\": read tcp 10.1.0.6:35388->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-thn7-fl7d5" "disk.csi.azure.com/request-id"="8381a073-93af-11ed-81d6-002248b31186"
E0114 02:02:54.298959    3488 azuredisk_v2.go:565]  "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [Put \"https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-thn7-fl7d5/status\": read tcp 10.1.0.6:35388->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.]" "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-thn7-fl7d5" "disk.csi.azure.com/request-id"="8381a073-93af-11ed-81d6-002248b31186" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=18973921200
I0114 02:02:55.214825    3488 reflector.go:257] Listing and watching *v1beta2.AzDriverNode from sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132
I0114 02:02:55.228325    3488 round_trippers.go:553] GET https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes?fieldSelector=metadata.name%3Dcapz-thn7-fl7d5&resourceVersion=28566 200 OK in 13 milliseconds
I0114 02:02:55.230828    3488 round_trippers.go:553] GET https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dcapz-thn7-fl7d5&resourceVersion=28749&timeout=6m51s&timeoutSeconds=411&watch=true 200 OK in 2 milliseconds
I0114 02:02:55.292587    3488 reflector.go:257] Listing and watching *v1.CustomResourceDefinition from k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132
I0114 02:02:55.308107    3488 round_trippers.go:553] GET https://10.96.0.1:443/apis/apiextensions.k8s.io/v1/customresourcedefinitions?resourceVersion=26308 200 OK in 14 milliseconds
I0114 02:02:55.344428    3488 round_trippers.go:553] GET https://10.96.0.1:443/apis/apiextensions.k8s.io/v1/customresourcedefinitions?allowWatchBookmarks=true&resourceVersion=26308&timeout=8m21s&timeoutSeconds=501&watch=true 200 OK in 1 milliseconds
... skipping 50 lines ...
I0114 02:04:17.485568    3488 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-72cb0fd3-460c-47f7-b549-477e22f09c6f\\globalmount","volume_capability":{"AccessType":{"Mount":{"mount_flags":["barrier=1","acl"]}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-72cb0fd3-460c-47f7-b549-477e22f09c6f","csi.storage.k8s.io/pvc/name":"pvc-r5tt2","csi.storage.k8s.io/pvc/namespace":"azuredisk-4765","requestedsizegib":"10","skuname":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1673656805162-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-thn7c4/providers/Microsoft.Compute/disks/pvc-72cb0fd3-460c-47f7-b549-477e22f09c6f"}
I0114 02:04:17.486048    3488 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-72cb0fd3-460c-47f7-b549-477e22f09c6f-capz-thn7-fl7d5-attachment)
I0114 02:04:25.272480    3488 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 02:04:25.380913    3488 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync
I0114 02:04:25.504927    3488 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 02:04:25.675204    3488 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 02:04:25.675204    3488 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0114 02:04:25.927151    3488 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 02:04:27.393352    3488 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0114 02:04:27.393352    3488 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-thn7-fl7d5" "disk.csi.azure.com/request-id"="c06690c3-93af-11ed-81d6-002248b31186" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-72cb0fd3-460c-47f7-b549-477e22f09c6f" "latency"=9906655900
I0114 02:04:27.393352    3488 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="c06690c3-93af-11ed-81d6-002248b31186" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9906655900
I0114 02:04:31.331965    3488 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I0114 02:04:31.331965    3488 utils.go:79] GRPC request: {}
I0114 02:04:31.332455    3488 utils.go:85] GRPC response: {"ready":{"value":true}}
I0114 02:04:32.511007    3488 nodeserver_v2.go:211] NodeStageVolume: formatting 10 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-72cb0fd3-460c-47f7-b549-477e22f09c6f\globalmount with mount options([barrier=1 acl])
... skipping 104 lines ...
I0114 02:07:22.409814    3488 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-b123ace2-a423-499d-a9a9-80cb022bce33\\globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-b123ace2-a423-499d-a9a9-80cb022bce33","csi.storage.k8s.io/pvc/name":"pvc-tllxr","csi.storage.k8s.io/pvc/namespace":"azuredisk-6031","requestedsizegib":"10","skuname":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1673656805162-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-thn7c4/providers/Microsoft.Compute/disks/pvc-b123ace2-a423-499d-a9a9-80cb022bce33"}
I0114 02:07:22.410304    3488 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-b123ace2-a423-499d-a9a9-80cb022bce33-capz-thn7-fl7d5-attachment)
I0114 02:07:25.322984    3488 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 02:07:25.421244    3488 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync
I0114 02:07:25.555251    3488 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 02:07:25.740648    3488 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 02:07:25.740648    3488 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0114 02:07:25.971914    3488 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 02:07:31.342226    3488 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I0114 02:07:31.342226    3488 utils.go:79] GRPC request: {}
I0114 02:07:31.342226    3488 utils.go:85] GRPC response: {"ready":{"value":true}}
I0114 02:07:32.104738    3488 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0114 02:07:32.104738    3488 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-thn7-fl7d5" "disk.csi.azure.com/request-id"="2e9fc483-93b0-11ed-81d6-002248b31186" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-b123ace2-a423-499d-a9a9-80cb022bce33" "latency"=9693863400
I0114 02:07:32.104738    3488 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="2e9fc483-93b0-11ed-81d6-002248b31186" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9693863400
I0114 02:07:35.338500    3488 azuredisk_v2.go:578]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-14T02:07:35Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-thn7-fl7d5" "disk.csi.azure.com/request-id"="3654742b-93b0-11ed-81d6-002248b31186"
I0114 02:07:35.348401    3488 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-thn7-fl7d5/status 200 OK in 9 milliseconds
I0114 02:07:35.348401    3488 azuredisk_v2.go:565]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-thn7-fl7d5" "disk.csi.azure.com/request-id"="3654742b-93b0-11ed-81d6-002248b31186" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=9900600
I0114 02:07:36.936969    3488 nodeserver_v2.go:211] NodeStageVolume: formatting 10 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-b123ace2-a423-499d-a9a9-80cb022bce33\globalmount with mount options([])
... skipping 79 lines ...
I0114 02:09:04.584262    3488 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0114 02:09:04.584262    3488 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-9e37a05d-ad96-4f84-8461-6113c9b62413\\globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-9e37a05d-ad96-4f84-8461-6113c9b62413","csi.storage.k8s.io/pvc/name":"test-volume-1-azuredisk-volume-tester-gznjr-0","csi.storage.k8s.io/pvc/namespace":"azuredisk-6340","requestedsizegib":"10","skuname":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1673656805162-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-thn7c4/providers/Microsoft.Compute/disks/pvc-9e37a05d-ad96-4f84-8461-6113c9b62413"}
I0114 02:09:04.584262    3488 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-9e37a05d-ad96-4f84-8461-6113c9b62413-capz-thn7-fl7d5-attachment)
I0114 02:09:05.343262    3488 azuredisk_v2.go:578]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-14T02:09:05Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-thn7-fl7d5" "disk.csi.azure.com/request-id"="6bfa0cd1-93b0-11ed-81d6-002248b31186"
I0114 02:09:05.350339    3488 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-thn7-fl7d5/status 200 OK in 7 milliseconds
I0114 02:09:05.350339    3488 azuredisk_v2.go:565]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-thn7-fl7d5" "disk.csi.azure.com/request-id"="6bfa0cd1-93b0-11ed-81d6-002248b31186" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=7342600
I0114 02:09:09.809526    3488 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0114 02:09:09.809526    3488 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-thn7-fl7d5" "disk.csi.azure.com/request-id"="6b8646c2-93b0-11ed-81d6-002248b31186" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-9e37a05d-ad96-4f84-8461-6113c9b62413" "latency"=5224911300
I0114 02:09:09.810036    3488 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="6b8646c2-93b0-11ed-81d6-002248b31186" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=5225420900
I0114 02:09:14.724966    3488 nodeserver_v2.go:211] NodeStageVolume: formatting 10 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-9e37a05d-ad96-4f84-8461-6113c9b62413\globalmount with mount options([])
I0114 02:09:25.356711    3488 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 02:09:25.449606    3488 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync
I0114 02:09:25.589193    3488 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
... skipping 144 lines ...
I0114 02:13:50.559981    3488 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0114 02:13:50.559981    3488 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-9e37a05d-ad96-4f84-8461-6113c9b62413\\globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-9e37a05d-ad96-4f84-8461-6113c9b62413","csi.storage.k8s.io/pvc/name":"test-volume-1-azuredisk-volume-tester-gznjr-0","csi.storage.k8s.io/pvc/namespace":"azuredisk-6340","requestedsizegib":"10","skuname":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1673656805162-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-thn7c4/providers/Microsoft.Compute/disks/pvc-9e37a05d-ad96-4f84-8461-6113c9b62413"}
I0114 02:13:50.560473    3488 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-9e37a05d-ad96-4f84-8461-6113c9b62413-capz-thn7-fl7d5-attachment)
I0114 02:13:55.455375    3488 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 02:13:55.533840    3488 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync
I0114 02:13:55.672186    3488 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 02:13:55.812607    3488 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0114 02:13:55.812607    3488 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-thn7-fl7d5" "disk.csi.azure.com/request-id"="15fac881-93b1-11ed-81d6-002248b31186" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-9e37a05d-ad96-4f84-8461-6113c9b62413" "latency"=5251909700
I0114 02:13:55.812607    3488 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="15fac881-93b1-11ed-81d6-002248b31186" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=5251909700
I0114 02:13:55.827117    3488 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 02:13:56.090612    3488 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 02:14:00.479296    3488 nodeserver_v2.go:211] NodeStageVolume: formatting 10 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-9e37a05d-ad96-4f84-8461-6113c9b62413\globalmount with mount options([])
I0114 02:14:01.362335    3488 utils.go:78] GRPC call: /csi.v1.Identity/Probe
... skipping 37 lines ...
I0114 02:14:27.337469    3488 nodeserver_v2.go:262] NodeUnstageVolume: unmount \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-9e37a05d-ad96-4f84-8461-6113c9b62413\globalmount successfully
I0114 02:14:27.337469    3488 utils.go:85] GRPC response: {}
I0114 02:14:31.366432    3488 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I0114 02:14:31.366432    3488 utils.go:79] GRPC request: {}
I0114 02:14:31.366432    3488 utils.go:85] GRPC response: {"ready":{"value":true}}
I0114 02:14:35.360962    3488 azuredisk_v2.go:578]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-14T02:14:35Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-thn7-fl7d5" "disk.csi.azure.com/request-id"="30aecb99-93b1-11ed-81d6-002248b31186"
W0114 02:14:54.287814    3488 reflector.go:347] k8s.io/client-go/informers/factory.go:150: watch of *v1.Node ended with: an error on the server ("unable to decode an event from the watch stream: read tcp 10.1.0.6:36008->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.") has prevented the request from succeeding
W0114 02:14:54.287814    3488 reflector.go:347] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: watch of *v1beta2.AzDriverNode ended with: an error on the server ("unable to decode an event from the watch stream: read tcp 10.1.0.6:36008->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.") has prevented the request from succeeding
W0114 02:14:54.288314    3488 reflector.go:347] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: watch of *v1beta2.AzVolumeAttachment ended with: an error on the server ("unable to decode an event from the watch stream: read tcp 10.1.0.6:36008->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.") has prevented the request from succeeding
W0114 02:14:54.288314    3488 reflector.go:347] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: watch of *v1.CustomResourceDefinition ended with: an error on the server ("unable to decode an event from the watch stream: read tcp 10.1.0.6:36008->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.") has prevented the request from succeeding
W0114 02:14:54.288314    3488 reflector.go:347] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: watch of *v1beta2.AzDriverNode ended with: an error on the server ("unable to decode an event from the watch stream: read tcp 10.1.0.6:36008->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.") has prevented the request from succeeding
I0114 02:14:54.288314    3488 round_trippers.go:553] PUT https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-thn7-fl7d5/status  in 18925 milliseconds
W0114 02:14:54.288314    3488 reflector.go:347] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: watch of *v1beta2.AzVolume ended with: an error on the server ("unable to decode an event from the watch stream: read tcp 10.1.0.6:36008->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.") has prevented the request from succeeding
E0114 02:14:54.288536    3488 azuredisk_v2.go:613]  "msg"="Failed to update AzDriverNode status after creation" "error"="Put \"https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-thn7-fl7d5/status\": read tcp 10.1.0.6:36008->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-thn7-fl7d5" "disk.csi.azure.com/request-id"="30aecb99-93b1-11ed-81d6-002248b31186"
E0114 02:14:54.288536    3488 azuredisk_v2.go:565]  "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [Put \"https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/capz-thn7-fl7d5/status\": read tcp 10.1.0.6:36008->10.96.0.1:443: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.]" "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="capz-thn7-fl7d5" "disk.csi.azure.com/request-id"="30aecb99-93b1-11ed-81d6-002248b31186" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=18926723600
I0114 02:14:55.156248    3488 reflector.go:257] Listing and watching *v1beta2.AzDriverNode from sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132
I0114 02:14:55.168036    3488 round_trippers.go:553] GET https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/azdrivernodes?resourceVersion=32184 200 OK in 11 milliseconds
I0114 02:14:55.170632    3488 round_trippers.go:553] GET https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/azdrivernodes?allowWatchBookmarks=true&resourceVersion=32377&timeout=8m15s&timeoutSeconds=495&watch=true 200 OK in 1 milliseconds
I0114 02:14:55.373202    3488 reflector.go:257] Listing and watching *v1beta2.AzDriverNode from sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132
I0114 02:14:55.376653    3488 round_trippers.go:553] GET https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes?fieldSelector=metadata.name%3Dcapz-thn7-fl7d5&resourceVersion=32172 200 OK in 3 milliseconds
I0114 02:14:55.379647    3488 round_trippers.go:553] GET https://10.96.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dcapz-thn7-fl7d5&resourceVersion=32377&timeout=9m28s&timeoutSeconds=568&watch=true 200 OK in 2 milliseconds
... skipping 105 lines ...
I0114 02:17:45.575413    3488 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I0114 02:17:45.575413    3488 utils.go:79] GRPC request: {"publish_context":{"LUN":"2"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-65690be5-7762-498c-adbd-3edddf93411b\\globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-65690be5-7762-498c-adbd-3edddf93411b","csi.storage.k8s.io/pvc/name":"test-volume-3-azuredisk-volume-tester-kjk5l-0","csi.storage.k8s.io/pvc/namespace":"azuredisk-9681","requestedsizegib":"10","skuname":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1673656805162-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-thn7c4/providers/Microsoft.Compute/disks/pvc-65690be5-7762-498c-adbd-3edddf93411b"}
I0114 02:17:45.575413    3488 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-65690be5-7762-498c-adbd-3edddf93411b-capz-thn7-fl7d5-attachment)
I0114 02:17:45.576422    3488 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0114 02:17:45.576422    3488 utils.go:79] GRPC request: {"publish_context":{"LUN":"1"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-f4e2d594-0b28-4ada-af1c-1574be82ebdf\\globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-f4e2d594-0b28-4ada-af1c-1574be82ebdf","csi.storage.k8s.io/pvc/name":"test-volume-1-azuredisk-volume-tester-kjk5l-0","csi.storage.k8s.io/pvc/namespace":"azuredisk-9681","requestedsizegib":"10","skuname":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1673656805162-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-thn7c4/providers/Microsoft.Compute/disks/pvc-f4e2d594-0b28-4ada-af1c-1574be82ebdf"}
I0114 02:17:45.576944    3488 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-f4e2d594-0b28-4ada-af1c-1574be82ebdf-capz-thn7-fl7d5-attachment)
I0114 02:17:47.457280    3488 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0114 02:17:47.457280    3488 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-thn7-fl7d5" "disk.csi.azure.com/request-id"="a20f5efc-93b1-11ed-81d6-002248b31186" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-f4e2d594-0b28-4ada-af1c-1574be82ebdf" "latency"=1880210500
I0114 02:17:47.457797    3488 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="a20f5efc-93b1-11ed-81d6-002248b31186" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=1880733200
I0114 02:17:47.462282    3488 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0114 02:17:47.462282    3488 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-thn7-fl7d5" "disk.csi.azure.com/request-id"="a20f24af-93b1-11ed-81d6-002248b31186" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-2ce66fc3-7535-436c-9268-6658e3cbf20b" "latency"=1887227300
I0114 02:17:47.462282    3488 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="a20f24af-93b1-11ed-81d6-002248b31186" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=1887227300
I0114 02:17:47.462788    3488 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0114 02:17:47.462788    3488 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-thn7-fl7d5" "disk.csi.azure.com/request-id"="a20f3797-93b1-11ed-81d6-002248b31186" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-65690be5-7762-498c-adbd-3edddf93411b" "latency"=1886735500
I0114 02:17:47.462788    3488 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="a20f3797-93b1-11ed-81d6-002248b31186" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=1887249300
I0114 02:17:53.534640    3488 nodeserver_v2.go:211] NodeStageVolume: formatting 10 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-2ce66fc3-7535-436c-9268-6658e3cbf20b\globalmount with mount options([])
I0114 02:17:53.643655    3488 nodeserver_v2.go:211] NodeStageVolume: formatting 12 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-f4e2d594-0b28-4ada-af1c-1574be82ebdf\globalmount with mount options([])
I0114 02:17:53.761166    3488 nodeserver_v2.go:211] NodeStageVolume: formatting 11 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-65690be5-7762-498c-adbd-3edddf93411b\globalmount with mount options([])
I0114 02:17:55.222932    3488 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
... skipping 576 lines ...
I0114 02:22:55.580369    3488 utils.go:78] GRPC call: /csi.v1.Node/NodeUnstageVolume
I0114 02:22:55.580369    3488 utils.go:79] GRPC request: {"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-f4e2d594-0b28-4ada-af1c-1574be82ebdf\\globalmount","volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-thn7c4/providers/Microsoft.Compute/disks/pvc-f4e2d594-0b28-4ada-af1c-1574be82ebdf"}
I0114 02:22:55.580369    3488 nodeserver_v2.go:257] NodeUnstageVolume: unmounting \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-f4e2d594-0b28-4ada-af1c-1574be82ebdf\globalmount
I0114 02:22:55.734893    3488 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 02:22:55.738383    3488 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 02:22:56.055914    3488 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync
E0114 02:22:57.056483    3488 utils.go:83] GRPC error: GetVolumeIDFromMount(c:\var\lib\kubelet\pods\b0292632-dd5e-4c3d-8055-d10fab803128\volumes\kubernetes.io~csi\pvc-f4e2d594-0b28-4ada-af1c-1574be82ebdf\mount) failed with error: rpc error: code = Unknown desc = error getting the volume for the mount c:\var\lib\kubelet\pods\b0292632-dd5e-4c3d-8055-d10fab803128\volumes\kubernetes.io~csi\pvc-f4e2d594-0b28-4ada-af1c-1574be82ebdf\mount, internal error error getting volume from mount. cmd: (Get-Item -Path Get-Item : Cannot find path 'C:\var\lib\kubelet\pods\b0292632-dd5e-4c3d-8055-d10fab803128\volumes\kubernetes.io~csi\pvc
-f4e2d594-0b28-4ada-af1c-1574be82ebdf\mount' because it does not exist.
At line:1 char:2
+ (Get-Item -Path c:\var\lib\kubelet\pods\b0292632-dd5e-4c3d-8055-d10fa ...
+  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : ObjectNotFound: (C:\var\lib\kube...4be82ebdf\mount:String) [Get-Item], ItemNotFoundExcep 
   tion
... skipping 37 lines ...
+ ... d : PathNotFound,Microsoft.PowerShell.Commands.GetItemCommand).Target
+                                                                  ~
Unexpected token ')' in expression or statement.
    + CategoryInfo          : ParserError: (:) [], ParentContainsErrorRecordException
    + FullyQualifiedErrorId : MissingEndParenthesisInExpression
 
, error: exit status 1, response: <nil>
I0114 02:22:57.059486    3488 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0114 02:22:57.059486    3488 utils.go:79] GRPC request: {}
I0114 02:22:57.059486    3488 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I0114 02:22:57.061988    3488 utils.go:78] GRPC call: /csi.v1.Node/NodeGetVolumeStats
I0114 02:22:57.061988    3488 utils.go:79] GRPC request: {"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-thn7c4/providers/Microsoft.Compute/disks/pvc-65690be5-7762-498c-adbd-3edddf93411b","volume_path":"c:\\var\\lib\\kubelet\\pods\\b0292632-dd5e-4c3d-8055-d10fab803128\\volumes\\kubernetes.io~csi\\pvc-65690be5-7762-498c-adbd-3edddf93411b\\mount"}
E0114 02:22:58.478596    3488 utils.go:83] GRPC error: GetVolumeIDFromMount(c:\var\lib\kubelet\pods\b0292632-dd5e-4c3d-8055-d10fab803128\volumes\kubernetes.io~csi\pvc-65690be5-7762-498c-adbd-3edddf93411b\mount) failed with error: rpc error: code = Unknown desc = error getting the volume for the mount c:\var\lib\kubelet\pods\b0292632-dd5e-4c3d-8055-d10fab803128\volumes\kubernetes.io~csi\pvc-65690be5-7762-498c-adbd-3edddf93411b\mount, internal error error getting volume from mount. cmd: (Get-Item -Path Get-Item : Cannot find path 'C:\var\lib\kubelet\pods\b0292632-dd5e-4c3d-8055-d10fab803128\volumes\kubernetes.io~csi\pvc
-65690be5-7762-498c-adbd-3edddf93411b\mount' because it does not exist.
At line:1 char:2
+ (Get-Item -Path c:\var\lib\kubelet\pods\b0292632-dd5e-4c3d-8055-d10fa ...
+  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : ObjectNotFound: (C:\var\lib\kube...ddf93411b\mount:String) [Get-Item], ItemNotFoundExcep 
   tion
... skipping 37 lines ...
+ ... d : PathNotFound,Microsoft.PowerShell.Commands.GetItemCommand).Target
+                                                                  ~
Unexpected token ')' in expression or statement.
    + CategoryInfo          : ParserError: (:) [], ParentContainsErrorRecordException
    + FullyQualifiedErrorId : MissingEndParenthesisInExpression
 
, error: exit status 1, response: <nil>
I0114 02:23:00.540317    3488 nodeserver_v2.go:262] NodeUnstageVolume: unmount \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-2ce66fc3-7535-436c-9268-6658e3cbf20b\globalmount successfully
I0114 02:23:00.540317    3488 utils.go:85] GRPC response: {}
I0114 02:23:00.555823    3488 nodeserver_v2.go:262] NodeUnstageVolume: unmount \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-65690be5-7762-498c-adbd-3edddf93411b\globalmount successfully
I0114 02:23:00.555823    3488 utils.go:85] GRPC response: {}
I0114 02:23:00.568827    3488 nodeserver_v2.go:262] NodeUnstageVolume: unmount \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-f4e2d594-0b28-4ada-af1c-1574be82ebdf\globalmount successfully
I0114 02:23:00.568827    3488 utils.go:85] GRPC response: {}
... skipping 37 lines ...
I0114 02:23:42.942306    3488 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0114 02:23:42.942306    3488 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0114 02:23:42.942306    3488 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-fcb84751-9df9-4849-a9db-aa3b457f9b39\\globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"cachingmode":"None","csi.storage.k8s.io/pv/name":"pvc-fcb84751-9df9-4849-a9db-aa3b457f9b39","csi.storage.k8s.io/pvc/name":"test-volume-2-azuredisk-volume-tester-bgpjk-0","csi.storage.k8s.io/pvc/namespace":"azuredisk-2388","maxShares":"2","requestedsizegib":"256","skuname":"Premium_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1673656805162-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-thn7c4/providers/Microsoft.Compute/disks/pvc-fcb84751-9df9-4849-a9db-aa3b457f9b39"}
I0114 02:23:42.942306    3488 utils.go:79] GRPC request: {"publish_context":{"LUN":"1"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-444ffcc3-7d65-45f9-ac54-99e7f6c433d7\\globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"cachingmode":"None","csi.storage.k8s.io/pv/name":"pvc-444ffcc3-7d65-45f9-ac54-99e7f6c433d7","csi.storage.k8s.io/pvc/name":"test-volume-1-azuredisk-volume-tester-bgpjk-0","csi.storage.k8s.io/pvc/namespace":"azuredisk-2388","maxShares":"2","requestedsizegib":"256","skuname":"Premium_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1673656805162-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-thn7c4/providers/Microsoft.Compute/disks/pvc-444ffcc3-7d65-45f9-ac54-99e7f6c433d7"}
I0114 02:23:42.942804    3488 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-fcb84751-9df9-4849-a9db-aa3b457f9b39-capz-thn7-fl7d5-attachment)
I0114 02:23:42.942804    3488 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-444ffcc3-7d65-45f9-ac54-99e7f6c433d7-capz-thn7-fl7d5-attachment)
I0114 02:23:52.362343    3488 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0114 02:23:52.362343    3488 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-thn7-fl7d5" "disk.csi.azure.com/request-id"="7711274c-93b2-11ed-81d7-002248b31186" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-444ffcc3-7d65-45f9-ac54-99e7f6c433d7" "latency"=9418795400
I0114 02:23:52.362343    3488 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="7711274c-93b2-11ed-81d7-002248b31186" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9418795400
I0114 02:23:52.368338    3488 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0114 02:23:52.368841    3488 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-thn7-fl7d5" "disk.csi.azure.com/request-id"="7711274c-93b2-11ed-81d6-002248b31186" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-fcb84751-9df9-4849-a9db-aa3b457f9b39" "latency"=9424789600
I0114 02:23:52.368841    3488 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="7711274c-93b2-11ed-81d6-002248b31186" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9425292300
I0114 02:23:55.346585    3488 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 02:23:55.539663    3488 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 02:23:55.756810    3488 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 02:23:55.757120    3488 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
... skipping 466 lines ...
I0114 02:29:52.952577    3488 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-92a470fb-8e85-4494-b943-11384394ce20\\globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-92a470fb-8e85-4494-b943-11384394ce20","csi.storage.k8s.io/pvc/name":"pvc-azuredisk","csi.storage.k8s.io/pvc/namespace":"default","requestedsizegib":"10","skuName":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1673656805162-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/capz-thn7c4/providers/Microsoft.Compute/disks/pvc-92a470fb-8e85-4494-b943-11384394ce20"}
I0114 02:29:52.953092    3488 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-92a470fb-8e85-4494-b943-11384394ce20-capz-thn7-fl7d5-attachment)
I0114 02:29:55.480793    3488 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 02:29:55.651170    3488 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 02:29:55.883347    3488 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 02:29:55.883347    3488 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 02:29:55.883679    3488 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0114 02:29:56.195832    3488 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync
I0114 02:29:58.083779    3488 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0114 02:29:58.083779    3488 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="capz-thn7-fl7d5" "disk.csi.azure.com/request-id"="539c3e2c-93b3-11ed-81d7-002248b31186" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-92a470fb-8e85-4494-b943-11384394ce20" "latency"=5130455900
I0114 02:29:58.083779    3488 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="539c3e2c-93b3-11ed-81d7-002248b31186" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=5130455900
I0114 02:30:01.431418    3488 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I0114 02:30:01.431418    3488 utils.go:79] GRPC request: {}
I0114 02:30:01.431418    3488 utils.go:85] GRPC response: {"ready":{"value":true}}
I0114 02:30:03.615525    3488 nodeserver_v2.go:211] NodeStageVolume: formatting 10 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-92a470fb-8e85-4494-b943-11384394ce20\globalmount with mount options([])
... skipping 103 lines ...
I0114 00:40:25.135509    3484 round_trippers.go:553] GET https://10.96.0.1:443/api/v1/nodes?allowWatchBookmarks=true&resourceVersion=4217&timeout=7m49s&timeoutSeconds=469&watch=true 200 OK in 2 milliseconds
I0114 00:40:25.183801    3484 round_trippers.go:553] GET https://10.96.0.1:443/apis/apiextensions.k8s.io/v1/customresourcedefinitions?allowWatchBookmarks=true&resourceVersion=3845&timeout=6m53s&timeoutSeconds=413&watch=true 200 OK in 8 milliseconds
I0114 00:40:25.203354    3484 shared_informer.go:303] caches populated
I0114 00:40:25.203354    3484 azuredisk_v2.go:225] driver userAgent: disk.csi.azure.com/latest-v2-9ef068a8cb36a997d4ea04b90c05c6f92a488a19 e2e-test
I0114 00:40:25.203491    3484 azure_disk_utils.go:564] reading cloud config from secret kube-system/azure-cloud-provider
I0114 00:40:25.208742    3484 round_trippers.go:553] GET https://10.96.0.1:443/api/v1/namespaces/kube-system/secrets/azure-cloud-provider 404 Not Found in 2 milliseconds
I0114 00:40:25.209615    3484 azure_disk_utils.go:571] InitializeCloudFromSecret: failed to get cloud config from secret kube-system/azure-cloud-provider: failed to get secret kube-system/azure-cloud-provider: secrets "azure-cloud-provider" not found
I0114 00:40:25.209615    3484 azure_disk_utils.go:576] could not read cloud config from secret kube-system/azure-cloud-provider
I0114 00:40:25.209615    3484 azure_disk_utils.go:586] use default AZURE_CREDENTIAL_FILE env var: C:\k\azure.json
I0114 00:40:25.209615    3484 azure_disk_utils.go:594] read cloud config from file: C:\k\azure.json successfully
I0114 00:40:25.212072    3484 azure_auth.go:253] Using AzurePublicCloud environment
I0114 00:40:25.212737    3484 azure_auth.go:138] azure: using client_id+client_secret to retrieve access token
I0114 00:40:25.216615    3484 azure_diskclient.go:68] Azure DisksClient using API version: 2022-03-02
... skipping 33 lines ...
I0114 00:40:25.661128    3484 utils.go:85] GRPC response: {"name":"disk.csi.azure.com","vendor_version":"latest-v2-9ef068a8cb36a997d4ea04b90c05c6f92a488a19"}
I0114 00:40:25.929573    3484 utils.go:78] GRPC call: /csi.v1.Identity/GetPluginInfo
I0114 00:40:25.929789    3484 utils.go:79] GRPC request: {}
I0114 00:40:25.929932    3484 utils.go:85] GRPC response: {"name":"disk.csi.azure.com","vendor_version":"latest-v2-9ef068a8cb36a997d4ea04b90c05c6f92a488a19"}
I0114 00:40:26.833694    3484 utils.go:78] GRPC call: /csi.v1.Node/NodeGetInfo
I0114 00:40:26.833694    3484 utils.go:79] GRPC request: {}
I0114 00:40:26.861430    3484 azure_instance_metadata.go:140] Warning: failed to get loadbalancer metadata: failure of getting loadbalancer metadata with response "404 Not Found"
I0114 00:40:26.862860    3484 nodeserver_v2.go:438] NodeGetInfo: nodeName(capz-thn7-zhv6p), VM Size(Standard_D4s_v3)
I0114 00:40:26.862860    3484 nodeserver.go:409] got a matching size in getMaxDataDiskCount, VM Size: STANDARD_D4S_V3, MaxDataDiskCount: 8
I0114 00:40:26.862860    3484 utils.go:85] GRPC response: {"accessible_topology":{"segments":{"topology.disk.csi.azure.com/zone":""}},"max_volumes_per_node":8,"node_id":"capz-thn7-zhv6p"}
I0114 00:40:55.132337    3484 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 00:40:55.132337    3484 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0114 00:40:55.132494    3484 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
... skipping 3124 lines ...
# HELP go_gc_heap_objects_objects Number of objects, live or unswept, occupying heap memory.
# TYPE go_gc_heap_objects_objects gauge
go_gc_heap_objects_objects 143663
# HELP go_gc_heap_tiny_allocs_objects_total Count of small allocations that are packed together into blocks. These allocations are counted separately from other allocations because each individual allocation is not tracked by the runtime, only their block. Each block is already accounted for in allocs-by-size and frees-by-size.
# TYPE go_gc_heap_tiny_allocs_objects_total counter
go_gc_heap_tiny_allocs_objects_total 325887
# HELP go_gc_limiter_last_enabled_gc_cycle GC cycle the last time the GC CPU limiter was enabled. This metric is useful for diagnosing the root cause of an out-of-memory error, because the limiter trades memory for CPU time when the GC's CPU time gets too high. This is most likely to occur with use of SetMemoryLimit. The first GC cycle is cycle 1, so a value of 0 indicates that it was never enabled.
# TYPE go_gc_limiter_last_enabled_gc_cycle gauge
go_gc_limiter_last_enabled_gc_cycle 0
# HELP go_gc_pauses_seconds Distribution individual GC-related stop-the-world pause latencies.
# TYPE go_gc_pauses_seconds histogram
go_gc_pauses_seconds_bucket{le="9.999999999999999e-10"} 0
go_gc_pauses_seconds_bucket{le="9.999999999999999e-09"} 0
... skipping 211 lines ...
[ReportAfterSuite] Autogenerated ReportAfterSuite for --junit-report
[38;5;243mautogenerated by Ginkgo[0m
[38;5;243m------------------------------[0m


[38;5;9m[1mSummarizing 1 Failure:[0m
  [38;5;9m[FAIL][0m [0mDynamic Provisioning [38;5;243m[single-az] [0m[38;5;9m[1m[It] Should create replicas on node with matching pod affinity[0m[0m
  [38;5;243m/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/testsuites/dynamically_provisioned_pod_affinity.go:168[0m

[38;5;9m[1mRan 38 of 108 Specs in 7541.294 seconds[0m
[38;5;9m[1mFAIL![0m -- [38;5;10m[1m37 Passed[0m | [38;5;9m[1m1 Failed[0m | [38;5;11m[1m0 Pending[0m | [38;5;14m[1m70 Skipped[0m
--- FAIL: TestE2E (7541.33s)
FAIL
FAIL	sigs.k8s.io/azuredisk-csi-driver/test/e2e	7541.428s
FAIL
make: *** [Makefile:341: e2e-test] Error 1
NAME                              STATUS   ROLES                  AGE    VERSION                           INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                         KERNEL-VERSION     CONTAINER-RUNTIME
capz-thn7-fl7d5                   Ready    <none>                 130m   v1.23.16-rc.0.57+b5c72d8dc2059d   10.1.0.6      <none>        Windows Server 2019 Datacenter   10.0.17763.3770    containerd://1.6.2
capz-thn7-slzjr                   Ready    <none>                 129m   v1.23.16-rc.0.57+b5c72d8dc2059d   10.1.0.5      <none>        Windows Server 2019 Datacenter   10.0.17763.3770    containerd://1.6.2
capz-thn7-zhv6p                   Ready    <none>                 130m   v1.23.16-rc.0.57+b5c72d8dc2059d   10.1.0.4      <none>        Windows Server 2019 Datacenter   10.0.17763.3770    containerd://1.6.2
capz-thn7c4-control-plane-4lzvn   Ready    control-plane,master   132m   v1.23.16-rc.0.57+b5c72d8dc2059d   10.0.0.4      <none>        Ubuntu 18.04.6 LTS               5.4.0-1098-azure   containerd://1.6.2
NAMESPACE          NAME                                                      READY   STATUS    RESTARTS       AGE    IP                NODE                              NOMINATED NODE   READINESS GATES
... skipping 36 lines ...

  Jan 14 02:36:42.202: INFO: Collecting logs for Windows node capz-thn7-slzjr in cluster capz-thn7c4 in namespace default

  Jan 14 02:38:19.221: INFO: Attempting to copy file /c:/crashdumps.tar on node capz-thn7-slzjr to /logs/artifacts/clusters/capz-thn7c4/machines/capz-thn7c4-md-win-84877fd844-l7bp6/crashdumps.tar
  Jan 14 02:38:20.861: INFO: Collecting boot logs for AzureMachine capz-thn7c4-md-win-slzjr

Failed to get logs for Machine capz-thn7c4-md-win-84877fd844-l7bp6, Cluster default/capz-thn7c4: running command "$p = 'c:\localdumps' ; if (Test-Path $p) { tar.exe -cvzf c:\crashdumps.tar $p *>&1 | %{ Write-Output "$_"} } else { Write-Host "No crash dumps found at $p" }": Process exited with status 1
  Jan 14 02:38:21.875: INFO: Collecting logs for Windows node capz-thn7-zhv6p in cluster capz-thn7c4 in namespace default

  Jan 14 02:40:49.710: INFO: Attempting to copy file /c:/crashdumps.tar on node capz-thn7-zhv6p to /logs/artifacts/clusters/capz-thn7c4/machines/capz-thn7c4-md-win-84877fd844-lsxgt/crashdumps.tar
  Jan 14 02:40:51.309: INFO: Collecting boot logs for AzureMachine capz-thn7c4-md-win-zhv6p

Failed to get logs for Machine capz-thn7c4-md-win-84877fd844-lsxgt, Cluster default/capz-thn7c4: [running command "Get-Content "C:\\cni.log"": Process exited with status 1, running command "$p = 'c:\localdumps' ; if (Test-Path $p) { tar.exe -cvzf c:\crashdumps.tar $p *>&1 | %{ Write-Output "$_"} } else { Write-Host "No crash dumps found at $p" }": Process exited with status 1]
  Jan 14 02:40:52.326: INFO: Collecting logs for Windows node capz-thn7-fl7d5 in cluster capz-thn7c4 in namespace default

  Jan 14 02:42:23.734: INFO: Attempting to copy file /c:/crashdumps.tar on node capz-thn7-fl7d5 to /logs/artifacts/clusters/capz-thn7c4/machines/capz-thn7c4-md-win-84877fd844-pbqqb/crashdumps.tar
  Jan 14 02:42:25.380: INFO: Collecting boot logs for AzureMachine capz-thn7c4-md-win-fl7d5

Failed to get logs for Machine capz-thn7c4-md-win-84877fd844-pbqqb, Cluster default/capz-thn7c4: running command "$p = 'c:\localdumps' ; if (Test-Path $p) { tar.exe -cvzf c:\crashdumps.tar $p *>&1 | %{ Write-Output "$_"} } else { Write-Host "No crash dumps found at $p" }": Process exited with status 1
  Jan 14 02:42:26.336: INFO: Dumping workload cluster default/capz-thn7c4 kube-system pod logs
  Jan 14 02:42:26.759: INFO: Collecting events for Pod calico-apiserver/calico-apiserver-5bd69d647b-sn5fd
  Jan 14 02:42:26.759: INFO: Creating log watcher for controller calico-apiserver/calico-apiserver-5bd69d647b-vn5fm, container calico-apiserver
  Jan 14 02:42:26.759: INFO: Creating log watcher for controller calico-apiserver/calico-apiserver-5bd69d647b-sn5fd, container calico-apiserver
  Jan 14 02:42:26.759: INFO: Collecting events for Pod calico-apiserver/calico-apiserver-5bd69d647b-vn5fm
  Jan 14 02:42:26.793: INFO: Creating log watcher for controller calico-system/calico-kube-controllers-fb49b9cf7-4zqzq, container calico-kube-controllers
... skipping 49 lines ...
  Jan 14 02:42:26.890: INFO: Creating log watcher for controller kube-system/etcd-capz-thn7c4-control-plane-4lzvn, container etcd
  Jan 14 02:42:26.890: INFO: Collecting events for Pod kube-system/metrics-server-7bdcf69694-s9tcd
  Jan 14 02:42:26.965: INFO: Fetching kube-system pod logs took 628.745867ms
  Jan 14 02:42:26.965: INFO: Dumping workload cluster default/capz-thn7c4 Azure activity log
  Jan 14 02:42:26.965: INFO: Creating log watcher for controller tigera-operator/tigera-operator-6bbf97c9cf-jt7lx, container tigera-operator
  Jan 14 02:42:26.965: INFO: Collecting events for Pod tigera-operator/tigera-operator-6bbf97c9cf-jt7lx
  Jan 14 02:42:32.931: INFO: Got error while iterating over activity logs for resource group capz-thn7c4: insights.ActivityLogsClient#listNextResults: Failure responding to next results request: StatusCode=404 -- Original Error: autorest/azure: error response cannot be parsed: {"<!DOCTYPE html PUBLIC \"-//W3C//DTD XHTML 1.0 Strict//EN\" \"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd\">\r\n<html xmlns=\"http://www.w3.org/1999/xhtml\">\r\n<head>\r\n<meta http-equiv=\"Content-Type\" content=\"text/html; charset=iso-8859-1\"/>\r\n<title>404 - File or directory not found.</title>\r\n<style type=\"text/css\">\r\n<!--\r\nbody{margin:0;font-size:.7em;font-family:Verdana, Arial, Helvetica, sans-serif;background:#EEEEEE;}\r\nfieldset{padding:0 15px 10px 15px;} \r\nh1{font-size:2.4em;margin:0;color:#FFF;}\r\nh2{font-si" '\x00' '\x00'} error: invalid character '<' looking for beginning of value
  Jan 14 02:42:32.931: INFO: Fetching activity logs took 5.965551228s
================ REDACTING LOGS ================
All sensitive variables are redacted
cluster.cluster.x-k8s.io "capz-thn7c4" deleted
/home/prow/go/src/sigs.k8s.io/cluster-api-provider-azure/hack/tools/bin/kind-v0.17.0 delete cluster --name=capz || true
Deleting cluster "capz" ...
... skipping 13 lines ...
PR	hccheng72: [V2] fix: only count replica attachments w/t deletion timestamp from cache client
Result	FAILURE
Tests	1 failed / 39 succeeded
Started	2023-01-13 23:59
Elapsed	2h49m
Revision	23b9a8b4a23ed46f3c0622bc10b1ef0956cdbc9e
Refs	1691
kubernetes-sigs/azuredisk-csi-driver PR #1691 pull-azuredisk-csi-driver-e2e-capz-windows-2019-mainv2 #1614049527289876480

Test Failures

AzureDisk CSI Driver End-to-End Tests [It] Dynamic Provisioning [single-az] Should create replicas on node with matching pod affinity 13m40s

Passed

Skipped

Error lines from build-log.txt
