This job view page is being replaced by Spyglass soon. Check out the new job view.
PRandyzhangx: fix: install updated packages to fix CVE issues
ResultFAILURE
Tests 1 failed / 11 succeeded
Started2021-10-22 06:13
Elapsed1h13m
Revision881b6763b89095109dc6d41b238347bce9f9ad98
Refs 1055
job-versionv1.23.0-alpha.3.479+cc25656b00baa3
kubetest-version
revisionv1.23.0-alpha.3.479+cc25656b00baa3

Test Failures


kubetest Test 1h0m

error during make e2e-test: exit status 2
				from junit_runner.xml

Filter through log files | View test history on testgrid


Show 11 Passed Tests

Error lines from build-log.txt

... skipping 221 lines ...

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 11248  100 11248    0     0   168k      0 --:--:-- --:--:-- --:--:--  168k
Downloading https://get.helm.sh/helm-v3.7.1-linux-amd64.tar.gz
Verifying checksum... Done.
Preparing to install helm into /usr/local/bin
helm installed into /usr/local/bin/helm
docker pull k8sprow.azurecr.io/azuredisk-csi:v1.9.0-716f56a44514b799d2d6c6922145f22026e7d4e2 || make container-all push-manifest
Error response from daemon: manifest for k8sprow.azurecr.io/azuredisk-csi:v1.9.0-716f56a44514b799d2d6c6922145f22026e7d4e2 not found: manifest unknown: manifest tagged by "v1.9.0-716f56a44514b799d2d6c6922145f22026e7d4e2" is not found
make[2]: Entering directory '/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver'
CGO_ENABLED=0 GOOS=windows go build -a -ldflags "-X sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.driverVersion=v1.9.0-716f56a44514b799d2d6c6922145f22026e7d4e2 -X sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.gitCommit=716f56a44514b799d2d6c6922145f22026e7d4e2 -X sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.buildDate=2021-10-22T06:17:25Z -extldflags "-static""  -mod vendor -o _output/amd64/azurediskplugin.exe ./pkg/azurediskplugin
docker buildx rm container-builder || true
error: no builder "container-builder" found
docker buildx create --use --name=container-builder
container-builder
# enable qemu for arm64 build
# https://github.com/docker/buildx/issues/464#issuecomment-741507760
docker run --privileged --rm tonistiigi/binfmt --uninstall qemu-aarch64
Unable to find image 'tonistiigi/binfmt:latest' locally
... skipping 1844 lines ...
                    type: string
                type: object
                oneOf:
                - required: ["persistentVolumeClaimName"]
                - required: ["volumeSnapshotContentName"]
              volumeSnapshotClassName:
                description: 'VolumeSnapshotClassName is the name of the VolumeSnapshotClass requested by the VolumeSnapshot. VolumeSnapshotClassName may be left nil to indicate that the default SnapshotClass should be used. A given cluster may have multiple default Volume SnapshotClasses: one default per CSI Driver. If a VolumeSnapshot does not specify a SnapshotClass, VolumeSnapshotSource will be checked to figure out what the associated CSI Driver is, and the default VolumeSnapshotClass associated with that CSI Driver will be used. If more than one VolumeSnapshotClass exist for a given CSI Driver and more than one have been marked as default, CreateSnapshot will fail and generate an event. Empty string is not allowed for this field.'
                type: string
            required:
            - source
            type: object
          status:
            description: status represents the current information of a snapshot. Consumers must verify binding between VolumeSnapshot and VolumeSnapshotContent objects is successful (by validating that both VolumeSnapshot and VolumeSnapshotContent point at each other) before using this object.
... skipping 2 lines ...
                description: 'boundVolumeSnapshotContentName is the name of the VolumeSnapshotContent object to which this VolumeSnapshot object intends to bind to. If not specified, it indicates that the VolumeSnapshot object has not been successfully bound to a VolumeSnapshotContent object yet. NOTE: To avoid possible security issues, consumers must verify binding between VolumeSnapshot and VolumeSnapshotContent objects is successful (by validating that both VolumeSnapshot and VolumeSnapshotContent point at each other) before using this object.'
                type: string
              creationTime:
                description: creationTime is the timestamp when the point-in-time snapshot is taken by the underlying storage system. In dynamic snapshot creation case, this field will be filled in by the snapshot controller with the "creation_time" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "creation_time" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. If not specified, it may indicate that the creation time of the snapshot is unknown.
                format: date-time
                type: string
              error:
                description: error is the last observed error during snapshot creation, if any. This field could be helpful to upper level controllers(i.e., application controller) to decide whether they should continue on waiting for the snapshot to be created based on the type of error reported. The snapshot controller will keep retrying when an error occurrs during the snapshot creation. Upon success, this error field will be cleared.
                properties:
                  message:
                    description: 'message is a string detailing the encountered error during snapshot creation if specified. NOTE: message may be logged, and it should not contain sensitive information.'
                    type: string
                  time:
                    description: time is the timestamp when the error was encountered.
                    format: date-time
                    type: string
                type: object
              readyToUse:
                description: readyToUse indicates if the snapshot is ready to be used to restore a volume. In dynamic snapshot creation case, this field will be filled in by the snapshot controller with the "ready_to_use" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "ready_to_use" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it, otherwise, this field will be set to "True". If not specified, it means the readiness of a snapshot is unknown.
                type: boolean
              restoreSize:
                type: string
                description: restoreSize represents the minimum size of volume required to create a volume from this snapshot. In dynamic snapshot creation case, this field will be filled in by the snapshot controller with the "size_bytes" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "size_bytes" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. When restoring a volume from this snapshot, the size of the volume MUST NOT be smaller than the restoreSize if it is specified, otherwise the restoration will fail. If not specified, it indicates that the size is unknown.
                pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
                x-kubernetes-int-or-string: true
            type: object
        required:
        - spec
        type: object
... skipping 60 lines ...
                    type: string
                  volumeSnapshotContentName:
                    description: volumeSnapshotContentName specifies the name of a pre-existing VolumeSnapshotContent object representing an existing volume snapshot. This field should be set if the snapshot already exists and only needs a representation in Kubernetes. This field is immutable.
                    type: string
                type: object
              volumeSnapshotClassName:
                description: 'VolumeSnapshotClassName is the name of the VolumeSnapshotClass requested by the VolumeSnapshot. VolumeSnapshotClassName may be left nil to indicate that the default SnapshotClass should be used. A given cluster may have multiple default Volume SnapshotClasses: one default per CSI Driver. If a VolumeSnapshot does not specify a SnapshotClass, VolumeSnapshotSource will be checked to figure out what the associated CSI Driver is, and the default VolumeSnapshotClass associated with that CSI Driver will be used. If more than one VolumeSnapshotClass exist for a given CSI Driver and more than one have been marked as default, CreateSnapshot will fail and generate an event. Empty string is not allowed for this field.'
                type: string
            required:
            - source
            type: object
          status:
            description: status represents the current information of a snapshot. Consumers must verify binding between VolumeSnapshot and VolumeSnapshotContent objects is successful (by validating that both VolumeSnapshot and VolumeSnapshotContent point at each other) before using this object.
... skipping 2 lines ...
                description: 'boundVolumeSnapshotContentName is the name of the VolumeSnapshotContent object to which this VolumeSnapshot object intends to bind to. If not specified, it indicates that the VolumeSnapshot object has not been successfully bound to a VolumeSnapshotContent object yet. NOTE: To avoid possible security issues, consumers must verify binding between VolumeSnapshot and VolumeSnapshotContent objects is successful (by validating that both VolumeSnapshot and VolumeSnapshotContent point at each other) before using this object.'
                type: string
              creationTime:
                description: creationTime is the timestamp when the point-in-time snapshot is taken by the underlying storage system. In dynamic snapshot creation case, this field will be filled in by the snapshot controller with the "creation_time" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "creation_time" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. If not specified, it may indicate that the creation time of the snapshot is unknown.
                format: date-time
                type: string
              error:
                description: error is the last observed error during snapshot creation, if any. This field could be helpful to upper level controllers(i.e., application controller) to decide whether they should continue on waiting for the snapshot to be created based on the type of error reported. The snapshot controller will keep retrying when an error occurrs during the snapshot creation. Upon success, this error field will be cleared.
                properties:
                  message:
                    description: 'message is a string detailing the encountered error during snapshot creation if specified. NOTE: message may be logged, and it should not contain sensitive information.'
                    type: string
                  time:
                    description: time is the timestamp when the error was encountered.
                    format: date-time
                    type: string
                type: object
              readyToUse:
                description: readyToUse indicates if the snapshot is ready to be used to restore a volume. In dynamic snapshot creation case, this field will be filled in by the snapshot controller with the "ready_to_use" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "ready_to_use" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it, otherwise, this field will be set to "True". If not specified, it means the readiness of a snapshot is unknown.
                type: boolean
              restoreSize:
                type: string
                description: restoreSize represents the minimum size of volume required to create a volume from this snapshot. In dynamic snapshot creation case, this field will be filled in by the snapshot controller with the "size_bytes" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "size_bytes" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. When restoring a volume from this snapshot, the size of the volume MUST NOT be smaller than the restoreSize if it is specified, otherwise the restoration will fail. If not specified, it indicates that the size is unknown.
                pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
                x-kubernetes-int-or-string: true
            type: object
        required:
        - spec
        type: object
... skipping 248 lines ...
            description: status represents the current information of a snapshot.
            properties:
              creationTime:
                description: creationTime is the timestamp when the point-in-time snapshot is taken by the underlying storage system. In dynamic snapshot creation case, this field will be filled in by the CSI snapshotter sidecar with the "creation_time" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "creation_time" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. If not specified, it indicates the creation time is unknown. The format of this field is a Unix nanoseconds time encoded as an int64. On Unix, the command `date +%s%N` returns the current time in nanoseconds since 1970-01-01 00:00:00 UTC.
                format: int64
                type: integer
              error:
                description: error is the last observed error during snapshot creation, if any. Upon success after retry, this error field will be cleared.
                properties:
                  message:
                    description: 'message is a string detailing the encountered error during snapshot creation if specified. NOTE: message may be logged, and it should not contain sensitive information.'
                    type: string
                  time:
                    description: time is the timestamp when the error was encountered.
                    format: date-time
                    type: string
                type: object
              readyToUse:
                description: readyToUse indicates if a snapshot is ready to be used to restore a volume. In dynamic snapshot creation case, this field will be filled in by the CSI snapshotter sidecar with the "ready_to_use" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "ready_to_use" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it, otherwise, this field will be set to "True". If not specified, it means the readiness of a snapshot is unknown.
                type: boolean
              restoreSize:
                description: restoreSize represents the complete size of the snapshot in bytes. In dynamic snapshot creation case, this field will be filled in by the CSI snapshotter sidecar with the "size_bytes" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "size_bytes" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. When restoring a volume from this snapshot, the size of the volume MUST NOT be smaller than the restoreSize if it is specified, otherwise the restoration will fail. If not specified, it indicates that the size is unknown.
                format: int64
                minimum: 0
                type: integer
              snapshotHandle:
                description: snapshotHandle is the CSI "snapshot_id" of a snapshot on the underlying storage system. If not specified, it indicates that dynamic snapshot creation has either failed or it is still in progress.
                type: string
            type: object
        required:
        - spec
        type: object
    served: true
... skipping 108 lines ...
            description: status represents the current information of a snapshot.
            properties:
              creationTime:
                description: creationTime is the timestamp when the point-in-time snapshot is taken by the underlying storage system. In dynamic snapshot creation case, this field will be filled in by the CSI snapshotter sidecar with the "creation_time" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "creation_time" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. If not specified, it indicates the creation time is unknown. The format of this field is a Unix nanoseconds time encoded as an int64. On Unix, the command `date +%s%N` returns the current time in nanoseconds since 1970-01-01 00:00:00 UTC.
                format: int64
                type: integer
              error:
                description: error is the last observed error during snapshot creation, if any. Upon success after retry, this error field will be cleared.
                properties:
                  message:
                    description: 'message is a string detailing the encountered error during snapshot creation if specified. NOTE: message may be logged, and it should not contain sensitive information.'
                    type: string
                  time:
                    description: time is the timestamp when the error was encountered.
                    format: date-time
                    type: string
                type: object
              readyToUse:
                description: readyToUse indicates if a snapshot is ready to be used to restore a volume. In dynamic snapshot creation case, this field will be filled in by the CSI snapshotter sidecar with the "ready_to_use" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "ready_to_use" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it, otherwise, this field will be set to "True". If not specified, it means the readiness of a snapshot is unknown.
                type: boolean
              restoreSize:
                description: restoreSize represents the complete size of the snapshot in bytes. In dynamic snapshot creation case, this field will be filled in by the CSI snapshotter sidecar with the "size_bytes" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "size_bytes" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. When restoring a volume from this snapshot, the size of the volume MUST NOT be smaller than the restoreSize if it is specified, otherwise the restoration will fail. If not specified, it indicates that the size is unknown.
                format: int64
                minimum: 0
                type: integer
              snapshotHandle:
                description: snapshotHandle is the CSI "snapshot_id" of a snapshot on the underlying storage system. If not specified, it indicates that dynamic snapshot creation has either failed or it is still in progress.
                type: string
            type: object
        required:
        - spec
        type: object
    served: true
... skipping 858 lines ...
        - name: csi-resizer-disk
          image: "mcr.microsoft.com/oss/kubernetes-csi/csi-resizer:v1.3.0"
          args:
            - "-csi-address=$(ADDRESS)"
            - "-v=2"
            - "-leader-election"
            - '-handle-volume-inuse-error=true'
          env:
            - name: ADDRESS
              value: /csi/csi.sock
          volumeMounts:
            - name: socket-dir
              mountPath: /csi
... skipping 502 lines ...
Oct 22 06:27:29.816: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comp2wmc] to have phase Bound
Oct 22 06:27:29.937: INFO: PersistentVolumeClaim test.csi.azure.comp2wmc found but phase is Pending instead of Bound.
Oct 22 06:27:32.046: INFO: PersistentVolumeClaim test.csi.azure.comp2wmc found but phase is Pending instead of Bound.
Oct 22 06:27:34.156: INFO: PersistentVolumeClaim test.csi.azure.comp2wmc found and phase=Bound (4.340325093s)
STEP: Creating pod pod-subpath-test-dynamicpv-27r9
STEP: Creating a pod to test subpath
Oct 22 06:27:34.499: INFO: Waiting up to 5m0s for pod "pod-subpath-test-dynamicpv-27r9" in namespace "provisioning-5043" to be "Succeeded or Failed"
Oct 22 06:27:34.613: INFO: Pod "pod-subpath-test-dynamicpv-27r9": Phase="Pending", Reason="", readiness=false. Elapsed: 114.021668ms
Oct 22 06:27:36.723: INFO: Pod "pod-subpath-test-dynamicpv-27r9": Phase="Pending", Reason="", readiness=false. Elapsed: 2.224166695s
Oct 22 06:27:38.832: INFO: Pod "pod-subpath-test-dynamicpv-27r9": Phase="Pending", Reason="", readiness=false. Elapsed: 4.333215998s
Oct 22 06:27:40.942: INFO: Pod "pod-subpath-test-dynamicpv-27r9": Phase="Pending", Reason="", readiness=false. Elapsed: 6.442760839s
Oct 22 06:27:43.053: INFO: Pod "pod-subpath-test-dynamicpv-27r9": Phase="Pending", Reason="", readiness=false. Elapsed: 8.553719102s
Oct 22 06:27:45.162: INFO: Pod "pod-subpath-test-dynamicpv-27r9": Phase="Pending", Reason="", readiness=false. Elapsed: 10.662953042s
... skipping 11 lines ...
Oct 22 06:28:10.486: INFO: Pod "pod-subpath-test-dynamicpv-27r9": Phase="Pending", Reason="", readiness=false. Elapsed: 35.98655044s
Oct 22 06:28:12.598: INFO: Pod "pod-subpath-test-dynamicpv-27r9": Phase="Pending", Reason="", readiness=false. Elapsed: 38.098729802s
Oct 22 06:28:14.706: INFO: Pod "pod-subpath-test-dynamicpv-27r9": Phase="Pending", Reason="", readiness=false. Elapsed: 40.207127039s
Oct 22 06:28:16.815: INFO: Pod "pod-subpath-test-dynamicpv-27r9": Phase="Pending", Reason="", readiness=false. Elapsed: 42.31641195s
Oct 22 06:28:18.924: INFO: Pod "pod-subpath-test-dynamicpv-27r9": Phase="Succeeded", Reason="", readiness=false. Elapsed: 44.425259993s
STEP: Saw pod success
Oct 22 06:28:18.924: INFO: Pod "pod-subpath-test-dynamicpv-27r9" satisfied condition "Succeeded or Failed"
Oct 22 06:28:19.033: INFO: Trying to get logs from node k8s-agentpool1-22475134-vmss000001 pod pod-subpath-test-dynamicpv-27r9 container test-container-subpath-dynamicpv-27r9: <nil>
STEP: delete the pod
Oct 22 06:28:19.309: INFO: Waiting for pod pod-subpath-test-dynamicpv-27r9 to disappear
Oct 22 06:28:19.418: INFO: Pod pod-subpath-test-dynamicpv-27r9 no longer exists
STEP: Deleting pod pod-subpath-test-dynamicpv-27r9
Oct 22 06:28:19.418: INFO: Deleting pod "pod-subpath-test-dynamicpv-27r9" in namespace "provisioning-5043"
... skipping 29 lines ...
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (default fs)] subPath
  /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50
    should support readOnly file specified in the volumeMount [LinuxOnly]
    /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/subpath.go:380
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should support readOnly file specified in the volumeMount [LinuxOnly]","total":12,"completed":1,"skipped":105,"failed":0}

SS
------------------------------
[BeforeEach] [Testpattern: Pre-provisioned PV (default fs)] subPath
  /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:51
Oct 22 06:29:31.835: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping
... skipping 149 lines ...
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/external/external.go:174
  [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral
  /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50
    should support multiple inline ephemeral volumes
    /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/ephemeral.go:221
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral should support multiple inline ephemeral volumes","total":19,"completed":1,"skipped":75,"failed":0}

SSSS
------------------------------
[BeforeEach] [Testpattern: Inline-volume (ntfs)][Feature:Windows] volumes
  /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:51
Oct 22 06:29:35.831: INFO: Driver "test.csi.azure.com" does not support volume type "InlineVolume" - skipping
... skipping 179 lines ...
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy
  /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50
    (Always)[LinuxOnly], pod created with an initial fsgroup, new pod fsgroup applied to volume contents
    /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/fsgroupchangepolicy.go:208
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy (Always)[LinuxOnly], pod created with an initial fsgroup, new pod fsgroup applied to volume contents","total":18,"completed":1,"skipped":7,"failed":0}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Dynamic PV (block volmode)] provisioning
  /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:51
[BeforeEach] [Testpattern: Dynamic PV (block volmode)] provisioning
... skipping 50 lines ...
Oct 22 06:27:29.749: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.com6t7zx] to have phase Bound
Oct 22 06:27:29.878: INFO: PersistentVolumeClaim test.csi.azure.com6t7zx found but phase is Pending instead of Bound.
Oct 22 06:27:31.986: INFO: PersistentVolumeClaim test.csi.azure.com6t7zx found but phase is Pending instead of Bound.
Oct 22 06:27:34.094: INFO: PersistentVolumeClaim test.csi.azure.com6t7zx found and phase=Bound (4.344569757s)
STEP: Creating pod pod-subpath-test-dynamicpv-w5gl
STEP: Creating a pod to test atomic-volume-subpath
Oct 22 06:27:34.419: INFO: Waiting up to 5m0s for pod "pod-subpath-test-dynamicpv-w5gl" in namespace "provisioning-2124" to be "Succeeded or Failed"
Oct 22 06:27:34.533: INFO: Pod "pod-subpath-test-dynamicpv-w5gl": Phase="Pending", Reason="", readiness=false. Elapsed: 114.496854ms
Oct 22 06:27:36.642: INFO: Pod "pod-subpath-test-dynamicpv-w5gl": Phase="Pending", Reason="", readiness=false. Elapsed: 2.223226335s
Oct 22 06:27:38.750: INFO: Pod "pod-subpath-test-dynamicpv-w5gl": Phase="Pending", Reason="", readiness=false. Elapsed: 4.331504269s
Oct 22 06:27:40.858: INFO: Pod "pod-subpath-test-dynamicpv-w5gl": Phase="Pending", Reason="", readiness=false. Elapsed: 6.439617206s
Oct 22 06:27:42.966: INFO: Pod "pod-subpath-test-dynamicpv-w5gl": Phase="Pending", Reason="", readiness=false. Elapsed: 8.547278781s
Oct 22 06:27:45.074: INFO: Pod "pod-subpath-test-dynamicpv-w5gl": Phase="Pending", Reason="", readiness=false. Elapsed: 10.655306715s
... skipping 19 lines ...
Oct 22 06:28:27.252: INFO: Pod "pod-subpath-test-dynamicpv-w5gl": Phase="Running", Reason="", readiness=true. Elapsed: 52.832633935s
Oct 22 06:28:29.359: INFO: Pod "pod-subpath-test-dynamicpv-w5gl": Phase="Running", Reason="", readiness=true. Elapsed: 54.94012827s
Oct 22 06:28:31.468: INFO: Pod "pod-subpath-test-dynamicpv-w5gl": Phase="Running", Reason="", readiness=true. Elapsed: 57.049421342s
Oct 22 06:28:33.576: INFO: Pod "pod-subpath-test-dynamicpv-w5gl": Phase="Running", Reason="", readiness=true. Elapsed: 59.157388339s
Oct 22 06:28:35.685: INFO: Pod "pod-subpath-test-dynamicpv-w5gl": Phase="Succeeded", Reason="", readiness=false. Elapsed: 1m1.265878881s
STEP: Saw pod success
Oct 22 06:28:35.685: INFO: Pod "pod-subpath-test-dynamicpv-w5gl" satisfied condition "Succeeded or Failed"
Oct 22 06:28:35.792: INFO: Trying to get logs from node k8s-agentpool1-22475134-vmss000000 pod pod-subpath-test-dynamicpv-w5gl container test-container-subpath-dynamicpv-w5gl: <nil>
STEP: delete the pod
Oct 22 06:28:36.031: INFO: Waiting for pod pod-subpath-test-dynamicpv-w5gl to disappear
Oct 22 06:28:36.138: INFO: Pod pod-subpath-test-dynamicpv-w5gl no longer exists
STEP: Deleting pod pod-subpath-test-dynamicpv-w5gl
Oct 22 06:28:36.138: INFO: Deleting pod "pod-subpath-test-dynamicpv-w5gl" in namespace "provisioning-2124"
... skipping 29 lines ...
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (default fs)] subPath
  /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50
    should support file as subpath [LinuxOnly]
    /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/subpath.go:230
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should support file as subpath [LinuxOnly]","total":18,"completed":1,"skipped":72,"failed":0}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy 
  (OnRootMismatch)[LinuxOnly], pod created with an initial fsgroup, new pod fsgroup applied to volume contents
  /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/fsgroupchangepolicy.go:208
... skipping 96 lines ...
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy
  /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50
    (OnRootMismatch)[LinuxOnly], pod created with an initial fsgroup, new pod fsgroup applied to volume contents
    /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/fsgroupchangepolicy.go:208
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy (OnRootMismatch)[LinuxOnly], pod created with an initial fsgroup, new pod fsgroup applied to volume contents","total":9,"completed":1,"skipped":518,"failed":0}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Pre-provisioned PV (block volmode)] volumes
  /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:51
Oct 22 06:30:23.203: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping
... skipping 141 lines ...
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (default fs)(allowExpansion)] volume-expand
  /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50
    Verify if offline PVC expansion works
    /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/volume_expand.go:174
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)(allowExpansion)] volume-expand Verify if offline PVC expansion works","total":17,"completed":1,"skipped":279,"failed":0}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: CSI Ephemeral-volume (default fs)] ephemeral
  /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:51
Oct 22 06:32:05.828: INFO: Driver "test.csi.azure.com" does not support volume type "CSIInlineVolume" - skipping
... skipping 147 lines ...
Oct 22 06:35:50.879: INFO: PersistentVolume pvc-3470e515-3208-4ae7-8a58-48ebecf49398 found and phase=Released (4m35.956230608s)
Oct 22 06:35:55.988: INFO: PersistentVolume pvc-3470e515-3208-4ae7-8a58-48ebecf49398 found and phase=Released (4m41.06595111s)
Oct 22 06:36:01.096: INFO: PersistentVolume pvc-3470e515-3208-4ae7-8a58-48ebecf49398 found and phase=Released (4m46.173592829s)
Oct 22 06:36:06.203: INFO: PersistentVolume pvc-3470e515-3208-4ae7-8a58-48ebecf49398 found and phase=Released (4m51.280843071s)
Oct 22 06:36:11.310: INFO: PersistentVolume pvc-3470e515-3208-4ae7-8a58-48ebecf49398 found and phase=Released (4m56.387818834s)
STEP: Deleting sc
Oct 22 06:36:16.424: FAIL: while cleanup resource
Unexpected error:
    <errors.aggregate | len:1, cap:1>: [
        [
            {
                msg: "persistent Volume pvc-3470e515-3208-4ae7-8a58-48ebecf49398 not deleted by dynamic provisioner: PersistentVolume pvc-3470e515-3208-4ae7-8a58-48ebecf49398 still exists within 5m0s",
                err: {
                    s: "PersistentVolume pvc-3470e515-3208-4ae7-8a58-48ebecf49398 still exists within 5m0s",
... skipping 28 lines ...
Oct 22 06:36:16.532: INFO: At 2021-10-22 06:30:18 +0000 UTC - event for pod-3d17a7cb-b5c3-4730-a8d1-ea56e4c4077a: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-3470e515-3208-4ae7-8a58-48ebecf49398" 
Oct 22 06:36:16.532: INFO: At 2021-10-22 06:30:47 +0000 UTC - event for pod-3d17a7cb-b5c3-4730-a8d1-ea56e4c4077a: {kubelet k8s-agentpool1-22475134-vmss000000} Pulling: Pulling image "k8s.gcr.io/e2e-test-images/busybox:1.29-1"
Oct 22 06:36:16.532: INFO: At 2021-10-22 06:30:48 +0000 UTC - event for pod-3d17a7cb-b5c3-4730-a8d1-ea56e4c4077a: {kubelet k8s-agentpool1-22475134-vmss000000} Started: Started container write-pod
Oct 22 06:36:16.532: INFO: At 2021-10-22 06:30:48 +0000 UTC - event for pod-3d17a7cb-b5c3-4730-a8d1-ea56e4c4077a: {kubelet k8s-agentpool1-22475134-vmss000000} Pulled: Successfully pulled image "k8s.gcr.io/e2e-test-images/busybox:1.29-1" in 297.964482ms
Oct 22 06:36:16.532: INFO: At 2021-10-22 06:30:48 +0000 UTC - event for pod-3d17a7cb-b5c3-4730-a8d1-ea56e4c4077a: {kubelet k8s-agentpool1-22475134-vmss000000} Created: Created container write-pod
Oct 22 06:36:16.532: INFO: At 2021-10-22 06:30:58 +0000 UTC - event for pod-3d17a7cb-b5c3-4730-a8d1-ea56e4c4077a: {kubelet k8s-agentpool1-22475134-vmss000000} Killing: Stopping container write-pod
Oct 22 06:36:16.532: INFO: At 2021-10-22 06:31:00 +0000 UTC - event for pod-3d17a7cb-b5c3-4730-a8d1-ea56e4c4077a: {kubelet k8s-agentpool1-22475134-vmss000000} FailedKillPod: error killing pod: failed to "KillContainer" for "write-pod" with KillContainerError: "rpc error: code = Unknown desc = Error response from daemon: No such container: e9b406af1e38c619472c163f8b3da9b0780133191b2a9c97d33434e210231eb3"
Oct 22 06:36:16.532: INFO: At 2021-10-22 06:31:00 +0000 UTC - event for pod-617a3c7f-4afe-4e69-8780-452148c48d19: {default-scheduler } Scheduled: Successfully assigned fsgroupchangepolicy-1195/pod-617a3c7f-4afe-4e69-8780-452148c48d19 to k8s-agentpool1-22475134-vmss000000
Oct 22 06:36:16.532: INFO: At 2021-10-22 06:31:09 +0000 UTC - event for pod-617a3c7f-4afe-4e69-8780-452148c48d19: {kubelet k8s-agentpool1-22475134-vmss000000} Created: Created container write-pod
Oct 22 06:36:16.532: INFO: At 2021-10-22 06:31:09 +0000 UTC - event for pod-617a3c7f-4afe-4e69-8780-452148c48d19: {kubelet k8s-agentpool1-22475134-vmss000000} Pulling: Pulling image "k8s.gcr.io/e2e-test-images/busybox:1.29-1"
Oct 22 06:36:16.532: INFO: At 2021-10-22 06:31:09 +0000 UTC - event for pod-617a3c7f-4afe-4e69-8780-452148c48d19: {kubelet k8s-agentpool1-22475134-vmss000000} Pulled: Successfully pulled image "k8s.gcr.io/e2e-test-images/busybox:1.29-1" in 316.984843ms
Oct 22 06:36:16.532: INFO: At 2021-10-22 06:31:10 +0000 UTC - event for pod-617a3c7f-4afe-4e69-8780-452148c48d19: {kubelet k8s-agentpool1-22475134-vmss000000} Started: Started container write-pod
Oct 22 06:36:16.532: INFO: At 2021-10-22 06:31:12 +0000 UTC - event for pod-617a3c7f-4afe-4e69-8780-452148c48d19: {kubelet k8s-agentpool1-22475134-vmss000000} Killing: Stopping container write-pod
... skipping 105 lines ...
  [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy
  /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50
    (Always)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed in first pod, new pod with same fsgroup applied to the volume contents [Measurement]
    /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/fsgroupchangepolicy.go:208

    Oct 22 06:36:16.424: while cleanup resource
    Unexpected error:
        <errors.aggregate | len:1, cap:1>: [
            [
                {
                    msg: "persistent Volume pvc-3470e515-3208-4ae7-8a58-48ebecf49398 not deleted by dynamic provisioner: PersistentVolume pvc-3470e515-3208-4ae7-8a58-48ebecf49398 still exists within 5m0s",
                    err: {
                        s: "PersistentVolume pvc-3470e515-3208-4ae7-8a58-48ebecf49398 still exists within 5m0s",
... skipping 3 lines ...
        ]
        persistent Volume pvc-3470e515-3208-4ae7-8a58-48ebecf49398 not deleted by dynamic provisioner: PersistentVolume pvc-3470e515-3208-4ae7-8a58-48ebecf49398 still exists within 5m0s
    occurred

    /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/fsgroupchangepolicy.go:132
------------------------------
{"msg":"FAILED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy (Always)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed in first pod, new pod with same fsgroup applied to the volume contents","total":19,"completed":1,"skipped":206,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy (Always)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed in first pod, new pod with same fsgroup applied to the volume contents"]}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Dynamic PV (ntfs)][Feature:Windows] subPath
  /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:51
Oct 22 06:36:19.521: INFO: Driver test.csi.azure.com doesn't support ntfs -- skipping
... skipping 113 lines ...
Oct 22 06:36:57.287: INFO: PersistentVolume pvc-35f243f0-f6ed-410c-9d3f-999c356910d0 found and phase=Released (4m36.064838999s)
Oct 22 06:37:02.399: INFO: PersistentVolume pvc-35f243f0-f6ed-410c-9d3f-999c356910d0 found and phase=Released (4m41.177090668s)
Oct 22 06:37:07.508: INFO: PersistentVolume pvc-35f243f0-f6ed-410c-9d3f-999c356910d0 found and phase=Released (4m46.286413079s)
Oct 22 06:37:12.621: INFO: PersistentVolume pvc-35f243f0-f6ed-410c-9d3f-999c356910d0 found and phase=Released (4m51.398991898s)
Oct 22 06:37:17.730: INFO: PersistentVolume pvc-35f243f0-f6ed-410c-9d3f-999c356910d0 found and phase=Released (4m56.507998851s)
STEP: Deleting sc
Oct 22 06:37:22.846: FAIL: while cleaning up resource
Unexpected error:
    <errors.aggregate | len:1, cap:1>: [
        [
            {
                msg: "persistent Volume pvc-35f243f0-f6ed-410c-9d3f-999c356910d0 not deleted by dynamic provisioner: PersistentVolume pvc-35f243f0-f6ed-410c-9d3f-999c356910d0 still exists within 5m0s",
                err: {
                    s: "PersistentVolume pvc-35f243f0-f6ed-410c-9d3f-999c356910d0 still exists within 5m0s",
... skipping 33 lines ...
Oct 22 06:37:22.956: INFO: At 2021-10-22 06:32:14 +0000 UTC - event for hostexec-k8s-agentpool1-22475134-vmss000000-tswfn: {kubelet k8s-agentpool1-22475134-vmss000000} Pulling: Pulling image "k8s.gcr.io/e2e-test-images/agnhost:2.32"
Oct 22 06:37:22.956: INFO: At 2021-10-22 06:32:14 +0000 UTC - event for hostexec-k8s-agentpool1-22475134-vmss000000-tswfn: {kubelet k8s-agentpool1-22475134-vmss000000} Created: Created container agnhost-container
Oct 22 06:37:22.956: INFO: At 2021-10-22 06:32:14 +0000 UTC - event for hostexec-k8s-agentpool1-22475134-vmss000000-tswfn: {kubelet k8s-agentpool1-22475134-vmss000000} Pulled: Successfully pulled image "k8s.gcr.io/e2e-test-images/agnhost:2.32" in 304.978259ms
Oct 22 06:37:22.956: INFO: At 2021-10-22 06:32:14 +0000 UTC - event for hostexec-k8s-agentpool1-22475134-vmss000000-tswfn: {kubelet k8s-agentpool1-22475134-vmss000000} Started: Started container agnhost-container
Oct 22 06:37:22.956: INFO: At 2021-10-22 06:32:18 +0000 UTC - event for hostexec-k8s-agentpool1-22475134-vmss000000-tswfn: {kubelet k8s-agentpool1-22475134-vmss000000} Killing: Stopping container agnhost-container
Oct 22 06:37:22.956: INFO: At 2021-10-22 06:32:18 +0000 UTC - event for pod-2e35e4cb-ec41-4e76-badd-d0c73d6f7634: {kubelet k8s-agentpool1-22475134-vmss000000} Killing: Stopping container write-pod
Oct 22 06:37:22.956: INFO: At 2021-10-22 06:34:11 +0000 UTC - event for pod-2e35e4cb-ec41-4e76-badd-d0c73d6f7634: {attachdetach-controller } FailedAttachVolume: AttachVolume.Attach failed for volume "pvc-35f243f0-f6ed-410c-9d3f-999c356910d0" : Attach timeout for volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-35f243f0-f6ed-410c-9d3f-999c356910d0
Oct 22 06:37:23.066: INFO: POD  NODE  PHASE  GRACE  CONDITIONS
Oct 22 06:37:23.066: INFO: 
Oct 22 06:37:23.282: INFO: 
Logging node info for node k8s-agentpool1-22475134-vmss000000
Oct 22 06:37:23.400: INFO: Node Info: &Node{ObjectMeta:{k8s-agentpool1-22475134-vmss000000    72bf23fc-217b-4f1a-9ed6-d90fb28f8d04 4884 0 2021-10-22 06:15:58 +0000 UTC <nil> <nil> map[agentpool:agentpool1 beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_DS4_v2 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westeurope failure-domain.beta.kubernetes.io/zone:0 kubernetes.azure.com/cluster:kubetest-5p3qtork kubernetes.azure.com/role:agent kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-agentpool1-22475134-vmss000000 kubernetes.io/os:linux kubernetes.io/role:agent node-role.kubernetes.io/agent: node.kubernetes.io/instance-type:Standard_DS4_v2 storageprofile:managed storagetier:Premium_LRS topology.kubernetes.io/region:westeurope topology.kubernetes.io/zone:0 topology.test.csi.azure.com/zone:] map[csi.volume.kubernetes.io/nodeid:{"test.csi.azure.com":"k8s-agentpool1-22475134-vmss000000"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] []  [{kubelet Update v1 2021-10-22 06:15:58 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:agentpool":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.azure.com/cluster":{},"f:kubernetes.azure.com/role":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:storageprofile":{},"f:storagetier":{}}}} } {kubectl-label Update v1 2021-10-22 06:16:06 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/agent":{}}}} } {cloud-controller-manager Update v1 2021-10-22 06:16:12 +0000 UTC FieldsV1 {"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"10.244.0.0/24\"":{}}}} } {cloud-controller-manager Update v1 2021-10-22 06:16:12 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2021-10-22 06:16:15 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}}} } {cloud-node-manager Update v1 2021-10-22 06:17:01 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kubelet Update v1 2021-10-22 06:27:43 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.test.csi.azure.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status} {kube-controller-manager Update v1 2021-10-22 06:32:55 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status}]},Spec:NodeSpec{PodCIDR:10.244.0.0/24,DoNotUseExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool1-22475134-vmss/virtualMachines/0,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[10.244.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{8 0} {<nil>} 8 DecimalSI},ephemeral-storage: {{31036686336 0} {<nil>}  BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{29449355264 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{8 0} {<nil>} 8 DecimalSI},ephemeral-storage: {{27933017657 0} {<nil>} 27933017657 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{28662923264 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2021-10-22 06:17:09 +0000 UTC,LastTransitionTime:2021-10-22 06:17:09 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2021-10-22 06:36:27 +0000 UTC,LastTransitionTime:2021-10-22 06:15:55 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2021-10-22 06:36:27 +0000 UTC,LastTransitionTime:2021-10-22 06:15:55 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2021-10-22 06:36:27 +0000 UTC,LastTransitionTime:2021-10-22 06:15:55 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2021-10-22 06:36:27 +0000 UTC,LastTransitionTime:2021-10-22 06:16:12 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.240.0.4,},NodeAddress{Type:Hostname,Address:k8s-agentpool1-22475134-vmss000000,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:d2790c386f8b4816867c93ee5ec07afb,SystemUUID:99c573b5-645b-634d-bb56-59a354ec6405,BootID:87fd645e-5f99-4dfe-9c69-ecf7970fbbbf,KernelVersion:5.4.0-1059-azure,OSImage:Ubuntu 18.04.6 LTS,ContainerRuntimeVersion:docker://20.10.7+azure,KubeletVersion:v1.22.2,KubeProxyVersion:v1.22.2,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/azuredisk-csi@sha256:0743369a58a6620586dcf8671640b81f8dc66bc5986a5c2d3b23a4ce6a9886e8 k8sprow.azurecr.io/azuredisk-csi:v1.9.0-716f56a44514b799d2d6c6922145f22026e7d4e2],SizeBytes:239897501,},ContainerImage{Names:[mcr.microsoft.com/k8s/csi/azurefile-csi@sha256:91972069fc98aae86e3297b054cb0a347b256d0a81799337e1ec0c196c341065 mcr.microsoft.com/k8s/csi/azurefile-csi:v1.6.0],SizeBytes:223047280,},ContainerImage{Names:[mcr.microsoft.com/k8s/csi/azuredisk-csi@sha256:d1e335c6d52dc8b0f7db7650879e1673fda5c561761a52f9ba72a216d9ca0721 mcr.microsoft.com/k8s/csi/azuredisk-csi:v1.7.0],SizeBytes:212174459,},ContainerImage{Names:[mcr.microsoft.com/containernetworking/azure-npm@sha256:106f669f48e5e80c4ec0afb49858ead72cf4b901cd8664e7bf81f8d789e56e12 mcr.microsoft.com/containernetworking/azure-npm:v1.2.2_hotfix],SizeBytes:175230380,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-apiserver@sha256:6c37c0efdaf49d24dafcced601fb78750205670e4e41842ab3a71935ed23b753 mcr.microsoft.com/oss/kubernetes/kube-apiserver:v1.22.2],SizeBytes:128401987,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:758db666ac7028534dba72e7e9bb1e57bb81b8196f976f7a5cc351ef8b3529e1 k8s.gcr.io/e2e-test-images/agnhost:2.32],SizeBytes:125910179,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/secrets-store/driver@sha256:c0d040a1c4fbfceb65663e31c09ea40f4f78e356437610cbc3fbb4bb409bd6f1 mcr.microsoft.com/oss/kubernetes-csi/secrets-store/driver:v0.0.19],SizeBytes:123229697,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-controller-manager@sha256:76fa95d463bf8950324d68f9f53063bb73628adcc862708f4534d6f44546df63 mcr.microsoft.com/oss/kubernetes/kube-controller-manager:v1.22.2],SizeBytes:121930716,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-proxy@sha256:b1ec3f3090fc799799bee98619a33300d8ee9095219adc4bb688084718810d23 mcr.microsoft.com/oss/kubernetes/kube-proxy:v1.22.2],SizeBytes:103625747,},ContainerImage{Names:[mcr.microsoft.com/oss/azure/secrets-store/provider-azure@sha256:6f67f3d0c7cdde5702f8ce7f101b6519daa0237f0c34fecb7c058b6af8c22ad1 mcr.microsoft.com/oss/azure/secrets-store/provider-azure:0.0.12],SizeBytes:101061355,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/autoscaler/cluster-autoscaler@sha256:29ba45b98c8035adea245f124d08d54dffbd27068940edd879cfd86022db20f9 mcr.microsoft.com/oss/kubernetes/autoscaler/cluster-autoscaler:v1.22.0],SizeBytes:99406403,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-addon-manager@sha256:32e2836018c96e73533bd4642fe438e465b81dcbfa8b7b61935a6f4d0246c7ae mcr.microsoft.com/oss/kubernetes/kube-addon-manager:v9.1.3],SizeBytes:86832059,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-addon-manager@sha256:92c2c5aad9012ee32d2a43a74966cc0adc6ccb1705ad15abb10485ecf406d88b mcr.microsoft.com/oss/kubernetes/kube-addon-manager:v9.1.5],SizeBytes:84094027,},ContainerImage{Names:[mcr.microsoft.com/oss/nvidia/k8s-device-plugin@sha256:0f5b52bf28239234e831697d96db63ac03cde70fe68058f964504ab7564ee810 mcr.microsoft.com/oss/nvidia/k8s-device-plugin:1.0.0-beta6],SizeBytes:64160241,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/metrics-server@sha256:682d3a1a48abbe03df05e1084923ab668a6651fd80327f9694b0cf302cc49f90 mcr.microsoft.com/oss/kubernetes/metrics-server:v0.5.0],SizeBytes:63501445,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-provisioner@sha256:e9ddadc44ba87a4a27f67e54760a14f9986885b534b3dff170a14eae1e35d213 mcr.microsoft.com/oss/kubernetes-csi/csi-provisioner:v3.0.0],SizeBytes:56881280,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-resizer@sha256:c5bb71ceaac60b1a4b58739fa07b709f6248c452ff6272a384d2f7648895a750 mcr.microsoft.com/oss/kubernetes-csi/csi-resizer:v1.3.0],SizeBytes:54313772,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-snapshotter@sha256:e28e996e03519565907d7158e0e1a6613cec76220fa8625e0ebfc1aa1c3bacb2 mcr.microsoft.com/oss/kubernetes-csi/csi-snapshotter:v4.2.1],SizeBytes:54210936,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-attacher@sha256:6b41e7153ebdfdc1501aa65184624bc15fd33a52d93f88ec3a758d0f8c9b8c10 mcr.microsoft.com/oss/kubernetes-csi/csi-attacher:v3.3.0],SizeBytes:53842561,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-scheduler@sha256:0457773d5575596d4f898913b45293f77dea0d955bc2002a4dbf7ea827cfe4f0 mcr.microsoft.com/oss/kubernetes/kube-scheduler:v1.22.2],SizeBytes:52626428,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/snapshot-controller@sha256:bb61653b29713588c280999cbb25b472fef8794623164fd9177c270da1121db0 mcr.microsoft.com/oss/kubernetes-csi/snapshot-controller:v4.2.1],SizeBytes:51575245,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager@sha256:ea0065331ce28081b4a048bd18530bbe4fe675dacb3c9515a2022ef10abf95ff mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager:v1.1.1],SizeBytes:50901571,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager@sha256:c24707c5f947d9e384d0f111469ca3266a01c48de6557f306e718c23d699cadf mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager:v1.1.1],SizeBytes:50295363,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/ip-masq-agent@sha256:bddf19be25644c7176e8c501465dbad07d032288334f135877b984660b0d3ee6 mcr.microsoft.com/oss/kubernetes/ip-masq-agent:v2.5.0],SizeBytes:50146762,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/coredns@sha256:1c23b17c78ffbc90c068ee088b77456284b9649952fa127b0f86a0ccff575539 mcr.microsoft.com/oss/kubernetes/coredns:1.7.0],SizeBytes:45233936,},ContainerImage{Names:[mcr.microsoft.com/oss/etcd-io/etcd@sha256:cf587862e3f1b6fa4d9a2565520a34f164bdf72c50f37af8c3c668160593246e mcr.microsoft.com/oss/etcd-io/etcd:v3.3.25],SizeBytes:41832119,},ContainerImage{Names:[mcr.microsoft.com/k8s/aad-pod-identity/mic@sha256:bd9465be94966b9a917e1e3904fa5e63dd91772ccadf304e18ffd8e4ad8ccedd mcr.microsoft.com/k8s/aad-pod-identity/mic:1.6.1],SizeBytes:41374894,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/autoscaler/cluster-proportional-autoscaler@sha256:753e3ca79dc833feed24ee4107ff4f06a96abbd6cefba8d41d7dec6f4d3e75f0 mcr.microsoft.com/oss/kubernetes/autoscaler/cluster-proportional-autoscaler:1.7.1],SizeBytes:40051560,},ContainerImage{Names:[mcr.microsoft.com/k8s/aad-pod-identity/nmi@sha256:02128fefcdb7593ac53fc342e2c53a0fc6fabd813036bf60457bf43cc2940116 mcr.microsoft.com/k8s/aad-pod-identity/nmi:1.6.1],SizeBytes:38007982,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:921f301c44dda06a325164accf22e78ecc570b5c7d9d6ee4c66bd8cbb2b60b9a mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.16],SizeBytes:26970670,},ContainerImage{Names:[mcr.microsoft.com/k8s/kms/keyvault@sha256:1a27e175f8c125209e32d2957b5509fe20757bd8cb309ff9da598799b56326fb mcr.microsoft.com/k8s/kms/keyvault:v0.0.10],SizeBytes:23077387,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-node-driver-registrar@sha256:f1205ba7ae0b0783bb19442064f4d7d664f3dc0c9b0b5b33358e5eef5aedc554 mcr.microsoft.com/oss/kubernetes-csi/csi-node-driver-registrar:v2.3.0],SizeBytes:18746317,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/livenessprobe@sha256:3770688d4efa33f8f284cc2ef58d33efa4ff606147e0a8de20e67458e13fedc2 mcr.microsoft.com/oss/kubernetes-csi/livenessprobe:v2.4.0],SizeBytes:17183286,},ContainerImage{Names:[busybox@sha256:f7ca5a32c10d51aeda3b4d01c61c6061f497893d7f6628b92f822f7117182a57 busybox:latest],SizeBytes:1235724,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8],SizeBytes:1159025,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:1154361,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/pause@sha256:e3b8c20681593c21b344ad801fbb8abaf564427ee3a57a9fcfa3b455f917ce46 mcr.microsoft.com/oss/kubernetes/pause:3.4.1],SizeBytes:682696,},},VolumesInUse:[kubernetes.io/csi/test.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-4028b709-1233-4cd4-8837-fd82aeedf83a kubernetes.io/csi/test.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-6c25433f-b868-4830-abac-dc47802cc74d kubernetes.io/csi/test.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-bc294ea0-3146-4351-837c-8dba4447e4d2 kubernetes.io/csi/test.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-f3e47329-1be6-465a-8415-51a95da5fb5b],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/test.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-4028b709-1233-4cd4-8837-fd82aeedf83a,DevicePath:,},},Config:nil,},}
Oct 22 06:37:23.401: INFO: 
... skipping 100 lines ...
  [Testpattern: Dynamic PV (filesystem volmode)] volumeMode
  /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50
    should not mount / map unused volumes in a pod [LinuxOnly] [Measurement]
    /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/volumemode.go:351

    Oct 22 06:37:22.846: while cleaning up resource
    Unexpected error:
        <errors.aggregate | len:1, cap:1>: [
            [
                {
                    msg: "persistent Volume pvc-35f243f0-f6ed-410c-9d3f-999c356910d0 not deleted by dynamic provisioner: PersistentVolume pvc-35f243f0-f6ed-410c-9d3f-999c356910d0 still exists within 5m0s",
                    err: {
                        s: "PersistentVolume pvc-35f243f0-f6ed-410c-9d3f-999c356910d0 still exists within 5m0s",
... skipping 3 lines ...
        ]
        persistent Volume pvc-35f243f0-f6ed-410c-9d3f-999c356910d0 not deleted by dynamic provisioner: PersistentVolume pvc-35f243f0-f6ed-410c-9d3f-999c356910d0 still exists within 5m0s
    occurred

    /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/volumemode.go:188
------------------------------
{"msg":"FAILED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (filesystem volmode)] volumeMode should not mount / map unused volumes in a pod [LinuxOnly]","total":17,"completed":1,"skipped":364,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (filesystem volmode)] volumeMode should not mount / map unused volumes in a pod [LinuxOnly]"]}

SSSSSSSSS
------------------------------
External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath 
  should be able to unmount after the subpath directory is deleted [LinuxOnly]
  /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/subpath.go:445
... skipping 15 lines ...
Oct 22 06:30:23.912: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil
Oct 22 06:30:24.024: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comncjl2] to have phase Bound
Oct 22 06:30:24.133: INFO: PersistentVolumeClaim test.csi.azure.comncjl2 found but phase is Pending instead of Bound.
Oct 22 06:30:26.242: INFO: PersistentVolumeClaim test.csi.azure.comncjl2 found but phase is Pending instead of Bound.
Oct 22 06:30:28.352: INFO: PersistentVolumeClaim test.csi.azure.comncjl2 found and phase=Bound (4.327637343s)
STEP: Creating pod pod-subpath-test-dynamicpv-4gwr
Oct 22 06:35:29.006: FAIL: while waiting for pod to be running
Unexpected error:
    <*errors.errorString | 0xc0002c0240>: {
        s: "timed out waiting for the condition",
    }
    timed out waiting for the condition
occurred

... skipping 49 lines ...
STEP: Collecting events from namespace "provisioning-4223".
STEP: Found 6 events.
Oct 22 06:39:22.759: INFO: At 2021-10-22 06:30:23 +0000 UTC - event for test.csi.azure.comncjl2: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "test.csi.azure.com" or manually created by system administrator
Oct 22 06:39:22.759: INFO: At 2021-10-22 06:30:23 +0000 UTC - event for test.csi.azure.comncjl2: {test.csi.azure.com_k8s-agentpool1-22475134-vmss000001_643e5b82-d98f-4701-82bb-2ae23a5c6604 } Provisioning: External provisioner is provisioning volume for claim "provisioning-4223/test.csi.azure.comncjl2"
Oct 22 06:39:22.759: INFO: At 2021-10-22 06:30:26 +0000 UTC - event for test.csi.azure.comncjl2: {test.csi.azure.com_k8s-agentpool1-22475134-vmss000001_643e5b82-d98f-4701-82bb-2ae23a5c6604 } ProvisioningSucceeded: Successfully provisioned volume pvc-a8b788c1-65b1-446d-9813-2274459c86a0
Oct 22 06:39:22.759: INFO: At 2021-10-22 06:30:28 +0000 UTC - event for pod-subpath-test-dynamicpv-4gwr: {default-scheduler } Scheduled: Successfully assigned provisioning-4223/pod-subpath-test-dynamicpv-4gwr to k8s-agentpool1-22475134-vmss000001
Oct 22 06:39:22.759: INFO: At 2021-10-22 06:32:28 +0000 UTC - event for pod-subpath-test-dynamicpv-4gwr: {attachdetach-controller } FailedAttachVolume: AttachVolume.Attach failed for volume "pvc-a8b788c1-65b1-446d-9813-2274459c86a0" : Attach timeout for volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-a8b788c1-65b1-446d-9813-2274459c86a0
Oct 22 06:39:22.759: INFO: At 2021-10-22 06:32:31 +0000 UTC - event for pod-subpath-test-dynamicpv-4gwr: {kubelet k8s-agentpool1-22475134-vmss000001} FailedMount: Unable to attach or mount volumes: unmounted volumes=[test-volume], unattached volumes=[test-volume liveness-probe-volume kube-api-access-44fw8]: timed out waiting for the condition
Oct 22 06:39:22.868: INFO: POD  NODE  PHASE  GRACE  CONDITIONS
Oct 22 06:39:22.868: INFO: 
Oct 22 06:39:23.085: INFO: 
Logging node info for node k8s-agentpool1-22475134-vmss000000
Oct 22 06:39:23.194: INFO: Node Info: &Node{ObjectMeta:{k8s-agentpool1-22475134-vmss000000    72bf23fc-217b-4f1a-9ed6-d90fb28f8d04 5455 0 2021-10-22 06:15:58 +0000 UTC <nil> <nil> map[agentpool:agentpool1 beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_DS4_v2 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westeurope failure-domain.beta.kubernetes.io/zone:0 kubernetes.azure.com/cluster:kubetest-5p3qtork kubernetes.azure.com/role:agent kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-agentpool1-22475134-vmss000000 kubernetes.io/os:linux kubernetes.io/role:agent node-role.kubernetes.io/agent: node.kubernetes.io/instance-type:Standard_DS4_v2 storageprofile:managed storagetier:Premium_LRS topology.kubernetes.io/region:westeurope topology.kubernetes.io/zone:0 topology.test.csi.azure.com/zone:] map[csi.volume.kubernetes.io/nodeid:{"test.csi.azure.com":"k8s-agentpool1-22475134-vmss000000"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] []  [{kubelet Update v1 2021-10-22 06:15:58 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:agentpool":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.azure.com/cluster":{},"f:kubernetes.azure.com/role":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:storageprofile":{},"f:storagetier":{}}}} } {kubectl-label Update v1 2021-10-22 06:16:06 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/agent":{}}}} } {cloud-controller-manager Update v1 2021-10-22 06:16:12 +0000 UTC FieldsV1 {"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"10.244.0.0/24\"":{}}}} } {cloud-controller-manager Update v1 2021-10-22 06:16:12 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2021-10-22 06:16:15 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}}} } {cloud-node-manager Update v1 2021-10-22 06:17:01 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kubelet Update v1 2021-10-22 06:27:43 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.test.csi.azure.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status} {kube-controller-manager Update v1 2021-10-22 06:32:55 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status}]},Spec:NodeSpec{PodCIDR:10.244.0.0/24,DoNotUseExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool1-22475134-vmss/virtualMachines/0,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[10.244.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{8 0} {<nil>} 8 DecimalSI},ephemeral-storage: {{31036686336 0} {<nil>}  BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{29449355264 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{8 0} {<nil>} 8 DecimalSI},ephemeral-storage: {{27933017657 0} {<nil>} 27933017657 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{28662923264 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2021-10-22 06:17:09 +0000 UTC,LastTransitionTime:2021-10-22 06:17:09 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2021-10-22 06:36:27 +0000 UTC,LastTransitionTime:2021-10-22 06:15:55 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2021-10-22 06:36:27 +0000 UTC,LastTransitionTime:2021-10-22 06:15:55 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2021-10-22 06:36:27 +0000 UTC,LastTransitionTime:2021-10-22 06:15:55 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2021-10-22 06:36:27 +0000 UTC,LastTransitionTime:2021-10-22 06:16:12 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.240.0.4,},NodeAddress{Type:Hostname,Address:k8s-agentpool1-22475134-vmss000000,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:d2790c386f8b4816867c93ee5ec07afb,SystemUUID:99c573b5-645b-634d-bb56-59a354ec6405,BootID:87fd645e-5f99-4dfe-9c69-ecf7970fbbbf,KernelVersion:5.4.0-1059-azure,OSImage:Ubuntu 18.04.6 LTS,ContainerRuntimeVersion:docker://20.10.7+azure,KubeletVersion:v1.22.2,KubeProxyVersion:v1.22.2,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/azuredisk-csi@sha256:0743369a58a6620586dcf8671640b81f8dc66bc5986a5c2d3b23a4ce6a9886e8 k8sprow.azurecr.io/azuredisk-csi:v1.9.0-716f56a44514b799d2d6c6922145f22026e7d4e2],SizeBytes:239897501,},ContainerImage{Names:[mcr.microsoft.com/k8s/csi/azurefile-csi@sha256:91972069fc98aae86e3297b054cb0a347b256d0a81799337e1ec0c196c341065 mcr.microsoft.com/k8s/csi/azurefile-csi:v1.6.0],SizeBytes:223047280,},ContainerImage{Names:[mcr.microsoft.com/k8s/csi/azuredisk-csi@sha256:d1e335c6d52dc8b0f7db7650879e1673fda5c561761a52f9ba72a216d9ca0721 mcr.microsoft.com/k8s/csi/azuredisk-csi:v1.7.0],SizeBytes:212174459,},ContainerImage{Names:[mcr.microsoft.com/containernetworking/azure-npm@sha256:106f669f48e5e80c4ec0afb49858ead72cf4b901cd8664e7bf81f8d789e56e12 mcr.microsoft.com/containernetworking/azure-npm:v1.2.2_hotfix],SizeBytes:175230380,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-apiserver@sha256:6c37c0efdaf49d24dafcced601fb78750205670e4e41842ab3a71935ed23b753 mcr.microsoft.com/oss/kubernetes/kube-apiserver:v1.22.2],SizeBytes:128401987,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:758db666ac7028534dba72e7e9bb1e57bb81b8196f976f7a5cc351ef8b3529e1 k8s.gcr.io/e2e-test-images/agnhost:2.32],SizeBytes:125910179,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/secrets-store/driver@sha256:c0d040a1c4fbfceb65663e31c09ea40f4f78e356437610cbc3fbb4bb409bd6f1 mcr.microsoft.com/oss/kubernetes-csi/secrets-store/driver:v0.0.19],SizeBytes:123229697,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-controller-manager@sha256:76fa95d463bf8950324d68f9f53063bb73628adcc862708f4534d6f44546df63 mcr.microsoft.com/oss/kubernetes/kube-controller-manager:v1.22.2],SizeBytes:121930716,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-proxy@sha256:b1ec3f3090fc799799bee98619a33300d8ee9095219adc4bb688084718810d23 mcr.microsoft.com/oss/kubernetes/kube-proxy:v1.22.2],SizeBytes:103625747,},ContainerImage{Names:[mcr.microsoft.com/oss/azure/secrets-store/provider-azure@sha256:6f67f3d0c7cdde5702f8ce7f101b6519daa0237f0c34fecb7c058b6af8c22ad1 mcr.microsoft.com/oss/azure/secrets-store/provider-azure:0.0.12],SizeBytes:101061355,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/autoscaler/cluster-autoscaler@sha256:29ba45b98c8035adea245f124d08d54dffbd27068940edd879cfd86022db20f9 mcr.microsoft.com/oss/kubernetes/autoscaler/cluster-autoscaler:v1.22.0],SizeBytes:99406403,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-addon-manager@sha256:32e2836018c96e73533bd4642fe438e465b81dcbfa8b7b61935a6f4d0246c7ae mcr.microsoft.com/oss/kubernetes/kube-addon-manager:v9.1.3],SizeBytes:86832059,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-addon-manager@sha256:92c2c5aad9012ee32d2a43a74966cc0adc6ccb1705ad15abb10485ecf406d88b mcr.microsoft.com/oss/kubernetes/kube-addon-manager:v9.1.5],SizeBytes:84094027,},ContainerImage{Names:[mcr.microsoft.com/oss/nvidia/k8s-device-plugin@sha256:0f5b52bf28239234e831697d96db63ac03cde70fe68058f964504ab7564ee810 mcr.microsoft.com/oss/nvidia/k8s-device-plugin:1.0.0-beta6],SizeBytes:64160241,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/metrics-server@sha256:682d3a1a48abbe03df05e1084923ab668a6651fd80327f9694b0cf302cc49f90 mcr.microsoft.com/oss/kubernetes/metrics-server:v0.5.0],SizeBytes:63501445,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-provisioner@sha256:e9ddadc44ba87a4a27f67e54760a14f9986885b534b3dff170a14eae1e35d213 mcr.microsoft.com/oss/kubernetes-csi/csi-provisioner:v3.0.0],SizeBytes:56881280,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-resizer@sha256:c5bb71ceaac60b1a4b58739fa07b709f6248c452ff6272a384d2f7648895a750 mcr.microsoft.com/oss/kubernetes-csi/csi-resizer:v1.3.0],SizeBytes:54313772,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-snapshotter@sha256:e28e996e03519565907d7158e0e1a6613cec76220fa8625e0ebfc1aa1c3bacb2 mcr.microsoft.com/oss/kubernetes-csi/csi-snapshotter:v4.2.1],SizeBytes:54210936,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-attacher@sha256:6b41e7153ebdfdc1501aa65184624bc15fd33a52d93f88ec3a758d0f8c9b8c10 mcr.microsoft.com/oss/kubernetes-csi/csi-attacher:v3.3.0],SizeBytes:53842561,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-scheduler@sha256:0457773d5575596d4f898913b45293f77dea0d955bc2002a4dbf7ea827cfe4f0 mcr.microsoft.com/oss/kubernetes/kube-scheduler:v1.22.2],SizeBytes:52626428,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/snapshot-controller@sha256:bb61653b29713588c280999cbb25b472fef8794623164fd9177c270da1121db0 mcr.microsoft.com/oss/kubernetes-csi/snapshot-controller:v4.2.1],SizeBytes:51575245,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager@sha256:ea0065331ce28081b4a048bd18530bbe4fe675dacb3c9515a2022ef10abf95ff mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager:v1.1.1],SizeBytes:50901571,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager@sha256:c24707c5f947d9e384d0f111469ca3266a01c48de6557f306e718c23d699cadf mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager:v1.1.1],SizeBytes:50295363,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/ip-masq-agent@sha256:bddf19be25644c7176e8c501465dbad07d032288334f135877b984660b0d3ee6 mcr.microsoft.com/oss/kubernetes/ip-masq-agent:v2.5.0],SizeBytes:50146762,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/coredns@sha256:1c23b17c78ffbc90c068ee088b77456284b9649952fa127b0f86a0ccff575539 mcr.microsoft.com/oss/kubernetes/coredns:1.7.0],SizeBytes:45233936,},ContainerImage{Names:[mcr.microsoft.com/oss/etcd-io/etcd@sha256:cf587862e3f1b6fa4d9a2565520a34f164bdf72c50f37af8c3c668160593246e mcr.microsoft.com/oss/etcd-io/etcd:v3.3.25],SizeBytes:41832119,},ContainerImage{Names:[mcr.microsoft.com/k8s/aad-pod-identity/mic@sha256:bd9465be94966b9a917e1e3904fa5e63dd91772ccadf304e18ffd8e4ad8ccedd mcr.microsoft.com/k8s/aad-pod-identity/mic:1.6.1],SizeBytes:41374894,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/autoscaler/cluster-proportional-autoscaler@sha256:753e3ca79dc833feed24ee4107ff4f06a96abbd6cefba8d41d7dec6f4d3e75f0 mcr.microsoft.com/oss/kubernetes/autoscaler/cluster-proportional-autoscaler:1.7.1],SizeBytes:40051560,},ContainerImage{Names:[mcr.microsoft.com/k8s/aad-pod-identity/nmi@sha256:02128fefcdb7593ac53fc342e2c53a0fc6fabd813036bf60457bf43cc2940116 mcr.microsoft.com/k8s/aad-pod-identity/nmi:1.6.1],SizeBytes:38007982,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:921f301c44dda06a325164accf22e78ecc570b5c7d9d6ee4c66bd8cbb2b60b9a mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.16],SizeBytes:26970670,},ContainerImage{Names:[mcr.microsoft.com/k8s/kms/keyvault@sha256:1a27e175f8c125209e32d2957b5509fe20757bd8cb309ff9da598799b56326fb mcr.microsoft.com/k8s/kms/keyvault:v0.0.10],SizeBytes:23077387,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-node-driver-registrar@sha256:f1205ba7ae0b0783bb19442064f4d7d664f3dc0c9b0b5b33358e5eef5aedc554 mcr.microsoft.com/oss/kubernetes-csi/csi-node-driver-registrar:v2.3.0],SizeBytes:18746317,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/livenessprobe@sha256:3770688d4efa33f8f284cc2ef58d33efa4ff606147e0a8de20e67458e13fedc2 mcr.microsoft.com/oss/kubernetes-csi/livenessprobe:v2.4.0],SizeBytes:17183286,},ContainerImage{Names:[busybox@sha256:f7ca5a32c10d51aeda3b4d01c61c6061f497893d7f6628b92f822f7117182a57 busybox:latest],SizeBytes:1235724,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8],SizeBytes:1159025,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:1154361,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/pause@sha256:e3b8c20681593c21b344ad801fbb8abaf564427ee3a57a9fcfa3b455f917ce46 mcr.microsoft.com/oss/kubernetes/pause:3.4.1],SizeBytes:682696,},},VolumesInUse:[kubernetes.io/csi/test.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-4028b709-1233-4cd4-8837-fd82aeedf83a kubernetes.io/csi/test.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-6c25433f-b868-4830-abac-dc47802cc74d kubernetes.io/csi/test.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-bc294ea0-3146-4351-837c-8dba4447e4d2 kubernetes.io/csi/test.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-f3e47329-1be6-465a-8415-51a95da5fb5b],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/test.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-bc294ea0-3146-4351-837c-8dba4447e4d2,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/test.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-f3e47329-1be6-465a-8415-51a95da5fb5b,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/test.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-6c25433f-b868-4830-abac-dc47802cc74d,DevicePath:,},},Config:nil,},}
... skipping 105 lines ...
  [Testpattern: Dynamic PV (default fs)] subPath
  /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50
    should be able to unmount after the subpath directory is deleted [LinuxOnly] [Measurement]
    /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/subpath.go:445

    Oct 22 06:35:29.006: while waiting for pod to be running
    Unexpected error:
        <*errors.errorString | 0xc0002c0240>: {
            s: "timed out waiting for the condition",
        }
        timed out waiting for the condition
    occurred

    /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/subpath.go:465
------------------------------
{"msg":"FAILED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should be able to unmount after the subpath directory is deleted [LinuxOnly]","total":9,"completed":1,"skipped":654,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should be able to unmount after the subpath directory is deleted [LinuxOnly]"]}
[BeforeEach] [Testpattern: Dynamic PV (ext3)] volumes
  /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:51
Oct 22 06:39:25.640: INFO: Driver test.csi.azure.com doesn't support ext3 -- skipping
[AfterEach] [Testpattern: Dynamic PV (ext3)] volumes
  /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:186

... skipping 33 lines ...
Oct 22 06:29:45.601: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil
Oct 22 06:29:45.713: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comvx7qt] to have phase Bound
Oct 22 06:29:45.821: INFO: PersistentVolumeClaim test.csi.azure.comvx7qt found but phase is Pending instead of Bound.
Oct 22 06:29:47.930: INFO: PersistentVolumeClaim test.csi.azure.comvx7qt found but phase is Pending instead of Bound.
Oct 22 06:29:50.046: INFO: PersistentVolumeClaim test.csi.azure.comvx7qt found and phase=Bound (4.332871378s)
STEP: Creating Pod in namespace fsgroupchangepolicy-8367 with fsgroup 1000
Oct 22 06:34:50.705: FAIL: Unexpected error:
    <*errors.errorString | 0xc00372ace0>: {
        s: "pod \"pod-b7337dca-0c62-47b9-8679-cf9e0f5cc21d\" is not Running: timed out waiting for the condition",
    }
    pod "pod-b7337dca-0c62-47b9-8679-cf9e0f5cc21d" is not Running: timed out waiting for the condition
occurred

... skipping 70 lines ...
Oct 22 06:39:26.909: INFO: PersistentVolume pvc-f3e47329-1be6-465a-8415-51a95da5fb5b found and phase=Bound (4m36.090487061s)
Oct 22 06:39:32.018: INFO: PersistentVolume pvc-f3e47329-1be6-465a-8415-51a95da5fb5b found and phase=Bound (4m41.19928459s)
Oct 22 06:39:37.127: INFO: PersistentVolume pvc-f3e47329-1be6-465a-8415-51a95da5fb5b found and phase=Bound (4m46.307939926s)
Oct 22 06:39:42.236: INFO: PersistentVolume pvc-f3e47329-1be6-465a-8415-51a95da5fb5b found and phase=Bound (4m51.41667841s)
Oct 22 06:39:47.347: INFO: PersistentVolume pvc-f3e47329-1be6-465a-8415-51a95da5fb5b found and phase=Bound (4m56.527984802s)
STEP: Deleting sc
Oct 22 06:39:52.457: FAIL: while cleanup resource
Unexpected error:
    <errors.aggregate | len:1, cap:1>: [
        [
            {
                msg: "persistent Volume pvc-f3e47329-1be6-465a-8415-51a95da5fb5b not deleted by dynamic provisioner: PersistentVolume pvc-f3e47329-1be6-465a-8415-51a95da5fb5b still exists within 5m0s",
                err: {
                    s: "PersistentVolume pvc-f3e47329-1be6-465a-8415-51a95da5fb5b still exists within 5m0s",
... skipping 6 lines ...

Full Stack Trace
k8s.io/kubernetes/test/e2e/storage/testsuites.(*fsGroupChangePolicyTestSuite).DefineTests.func2()
	/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/fsgroupchangepolicy.go:132 +0xf0
panic(0x6bf5d00, 0xc003d40640)
	/usr/local/go/src/runtime/panic.go:965 +0x1b9
k8s.io/kubernetes/test/e2e/framework/ginkgowrapper.Fail.func1(0xc0001d0500, 0x13b, 0x8985882, 0x7f, 0xfa, 0xc0012f4800, 0x3a0)
	/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/ginkgowrapper/wrapper.go:63 +0xa5
panic(0x6323060, 0x77ced70)
	/usr/local/go/src/runtime/panic.go:965 +0x1b9
k8s.io/kubernetes/test/e2e/framework/ginkgowrapper.Fail(0xc0001d0500, 0x13b, 0xc002e7ad78, 0x1, 0x1)
	/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/ginkgowrapper/wrapper.go:67 +0x1b5
k8s.io/kubernetes/test/e2e/framework.Fail(0xc0001d0280, 0x126, 0xc000824850, 0x1, 0x1)
	/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/log.go:62 +0x1ea
k8s.io/kubernetes/vendor/github.com/onsi/gomega/internal/assertion.(*Assertion).match(0xc002e7af00, 0x79043f8, 0xa0cec48, 0x0, 0x0, 0x0, 0x0, 0xc00024c250)
	/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/onsi/gomega/internal/assertion/assertion.go:79 +0x216
k8s.io/kubernetes/vendor/github.com/onsi/gomega/internal/assertion.(*Assertion).NotTo(0xc002e7af00, 0x79043f8, 0xa0cec48, 0x0, 0x0, 0x0, 0x789aac0)
	/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/onsi/gomega/internal/assertion/assertion.go:48 +0xc7
k8s.io/kubernetes/test/e2e/framework.ExpectNoErrorWithOffset(0x1, 0x789aac0, 0xc00372ace0, 0x0, 0x0, 0x0)
... skipping 17 lines ...
STEP: Collecting events from namespace "fsgroupchangepolicy-8367".
STEP: Found 9 events.
Oct 22 06:39:52.567: INFO: At 2021-10-22 06:29:45 +0000 UTC - event for test.csi.azure.comvx7qt: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "test.csi.azure.com" or manually created by system administrator
Oct 22 06:39:52.567: INFO: At 2021-10-22 06:29:45 +0000 UTC - event for test.csi.azure.comvx7qt: {test.csi.azure.com_k8s-agentpool1-22475134-vmss000001_643e5b82-d98f-4701-82bb-2ae23a5c6604 } Provisioning: External provisioner is provisioning volume for claim "fsgroupchangepolicy-8367/test.csi.azure.comvx7qt"
Oct 22 06:39:52.567: INFO: At 2021-10-22 06:29:48 +0000 UTC - event for test.csi.azure.comvx7qt: {test.csi.azure.com_k8s-agentpool1-22475134-vmss000001_643e5b82-d98f-4701-82bb-2ae23a5c6604 } ProvisioningSucceeded: Successfully provisioned volume pvc-f3e47329-1be6-465a-8415-51a95da5fb5b
Oct 22 06:39:52.567: INFO: At 2021-10-22 06:29:50 +0000 UTC - event for pod-b7337dca-0c62-47b9-8679-cf9e0f5cc21d: {default-scheduler } Scheduled: Successfully assigned fsgroupchangepolicy-8367/pod-b7337dca-0c62-47b9-8679-cf9e0f5cc21d to k8s-agentpool1-22475134-vmss000000
Oct 22 06:39:52.567: INFO: At 2021-10-22 06:31:50 +0000 UTC - event for pod-b7337dca-0c62-47b9-8679-cf9e0f5cc21d: {attachdetach-controller } FailedAttachVolume: AttachVolume.Attach failed for volume "pvc-f3e47329-1be6-465a-8415-51a95da5fb5b" : Attach timeout for volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-f3e47329-1be6-465a-8415-51a95da5fb5b
Oct 22 06:39:52.567: INFO: At 2021-10-22 06:31:53 +0000 UTC - event for pod-b7337dca-0c62-47b9-8679-cf9e0f5cc21d: {kubelet k8s-agentpool1-22475134-vmss000000} FailedMount: Unable to attach or mount volumes: unmounted volumes=[volume1], unattached volumes=[volume1 kube-api-access-5sxhc]: timed out waiting for the condition
Oct 22 06:39:52.567: INFO: At 2021-10-22 06:36:42 +0000 UTC - event for pod-b7337dca-0c62-47b9-8679-cf9e0f5cc21d: {kubelet k8s-agentpool1-22475134-vmss000000} FailedMount: Unable to attach or mount volumes: unmounted volumes=[volume1], unattached volumes=[kube-api-access-5sxhc volume1]: error processing PVC fsgroupchangepolicy-8367/test.csi.azure.comvx7qt: PVC is being deleted
Oct 22 06:39:52.567: INFO: At 2021-10-22 06:36:55 +0000 UTC - event for pod-b7337dca-0c62-47b9-8679-cf9e0f5cc21d: {kubelet k8s-agentpool1-22475134-vmss000000} FailedMount: Unable to attach or mount volumes: unmounted volumes=[volume1], unattached volumes=[volume1 kube-api-access-5sxhc]: error processing PVC fsgroupchangepolicy-8367/test.csi.azure.comvx7qt: PVC is being deleted
Oct 22 06:39:52.567: INFO: At 2021-10-22 06:38:34 +0000 UTC - event for pod-b7337dca-0c62-47b9-8679-cf9e0f5cc21d: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-f3e47329-1be6-465a-8415-51a95da5fb5b" 
Oct 22 06:39:52.677: INFO: POD                                       NODE                                PHASE    GRACE  CONDITIONS
Oct 22 06:39:52.677: INFO: pod-b7337dca-0c62-47b9-8679-cf9e0f5cc21d  k8s-agentpool1-22475134-vmss000000  Pending         [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2021-10-22 06:29:50 +0000 UTC  } {Ready False 0001-01-01 00:00:00 +0000 UTC 2021-10-22 06:29:50 +0000 UTC ContainersNotReady containers with unready status: [write-pod]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2021-10-22 06:29:50 +0000 UTC ContainersNotReady containers with unready status: [write-pod]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2021-10-22 06:29:50 +0000 UTC  }]
Oct 22 06:39:52.677: INFO: 
Oct 22 06:39:52.891: INFO: 
Logging node info for node k8s-agentpool1-22475134-vmss000000
... skipping 103 lines ...
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy
  /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50
    (OnRootMismatch)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed in first pod, new pod with same fsgroup skips ownership changes to the volume contents [Measurement]
    /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/fsgroupchangepolicy.go:208

    Oct 22 06:34:50.705: Unexpected error:
        <*errors.errorString | 0xc00372ace0>: {
            s: "pod \"pod-b7337dca-0c62-47b9-8679-cf9e0f5cc21d\" is not Running: timed out waiting for the condition",
        }
        pod "pod-b7337dca-0c62-47b9-8679-cf9e0f5cc21d" is not Running: timed out waiting for the condition
    occurred

    /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/fsgroupchangepolicy.go:250
------------------------------
{"msg":"FAILED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy (OnRootMismatch)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed in first pod, new pod with same fsgroup skips ownership changes to the volume contents","total":18,"completed":1,"skipped":131,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy (OnRootMismatch)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed in first pod, new pod with same fsgroup skips ownership changes to the volume contents"]}

SSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Dynamic PV (ntfs)][Feature:Windows] subPath
  /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:51
Oct 22 06:39:55.359: INFO: Driver test.csi.azure.com doesn't support ntfs -- skipping
... skipping 178 lines ...
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/external/external.go:174
  [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral
  /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50
    should support two pods which share the same volume
    /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/ephemeral.go:183
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral should support two pods which share the same volume","total":13,"completed":1,"skipped":279,"failed":0}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Pre-provisioned PV (filesystem volmode)] volumeMode
  /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:51
Oct 22 06:39:55.887: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping
... skipping 51 lines ...
[BeforeEach] [Testpattern: Dynamic PV (immediate binding)] topology
  /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:185
STEP: Creating a kubernetes client
Oct 22 06:39:56.819: INFO: >>> kubeConfig: /root/tmp1541861128/kubeconfig/kubeconfig.westeurope.json
STEP: Building a namespace api object, basename topology
STEP: Waiting for a default service account to be provisioned in namespace
[It] should fail to schedule a pod which has topologies that conflict with AllowedTopologies
  /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/topology.go:192
Oct 22 06:39:57.364: INFO: Driver didn't provide topology keys -- skipping
[AfterEach] [Testpattern: Dynamic PV (immediate binding)] topology
  /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:186
Oct 22 06:39:57.364: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready
STEP: Destroying namespace "topology-5117" for this suite.


S [SKIPPING] [0.766 seconds]
External Storage [Driver: test.csi.azure.com]
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (immediate binding)] topology
  /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50
    should fail to schedule a pod which has topologies that conflict with AllowedTopologies [Measurement]
    /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/topology.go:192

    Driver didn't provide topology keys -- skipping

    /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/topology.go:124
------------------------------
... skipping 89 lines ...
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (block volmode)(allowExpansion)] volume-expand
  /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50
    Verify if offline PVC expansion works
    /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/volume_expand.go:174
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)(allowExpansion)] volume-expand Verify if offline PVC expansion works","total":18,"completed":2,"skipped":134,"failed":0}

SSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Inline-volume (default fs)] subPath
  /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:51
Oct 22 06:41:54.596: INFO: Driver "test.csi.azure.com" does not support volume type "InlineVolume" - skipping
... skipping 58 lines ...
Oct 22 06:37:26.543: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comv88b9] to have phase Bound
Oct 22 06:37:26.651: INFO: PersistentVolumeClaim test.csi.azure.comv88b9 found but phase is Pending instead of Bound.
Oct 22 06:37:28.760: INFO: PersistentVolumeClaim test.csi.azure.comv88b9 found but phase is Pending instead of Bound.
Oct 22 06:37:30.869: INFO: PersistentVolumeClaim test.csi.azure.comv88b9 found and phase=Bound (4.326537853s)
STEP: Creating pod pod-subpath-test-dynamicpv-2kgp
STEP: Creating a pod to test subpath
Oct 22 06:37:31.196: INFO: Waiting up to 5m0s for pod "pod-subpath-test-dynamicpv-2kgp" in namespace "provisioning-873" to be "Succeeded or Failed"
Oct 22 06:37:31.305: INFO: Pod "pod-subpath-test-dynamicpv-2kgp": Phase="Pending", Reason="", readiness=false. Elapsed: 108.528516ms
Oct 22 06:37:33.419: INFO: Pod "pod-subpath-test-dynamicpv-2kgp": Phase="Pending", Reason="", readiness=false. Elapsed: 2.222434478s
Oct 22 06:37:35.530: INFO: Pod "pod-subpath-test-dynamicpv-2kgp": Phase="Pending", Reason="", readiness=false. Elapsed: 4.333873805s
Oct 22 06:37:37.640: INFO: Pod "pod-subpath-test-dynamicpv-2kgp": Phase="Pending", Reason="", readiness=false. Elapsed: 6.444067518s
Oct 22 06:37:39.750: INFO: Pod "pod-subpath-test-dynamicpv-2kgp": Phase="Pending", Reason="", readiness=false. Elapsed: 8.553411002s
Oct 22 06:37:41.861: INFO: Pod "pod-subpath-test-dynamicpv-2kgp": Phase="Pending", Reason="", readiness=false. Elapsed: 10.665019643s
... skipping 56 lines ...
Oct 22 06:39:42.123: INFO: Pod "pod-subpath-test-dynamicpv-2kgp": Phase="Pending", Reason="", readiness=false. Elapsed: 2m10.926602194s
Oct 22 06:39:44.236: INFO: Pod "pod-subpath-test-dynamicpv-2kgp": Phase="Pending", Reason="", readiness=false. Elapsed: 2m13.039868244s
Oct 22 06:39:46.344: INFO: Pod "pod-subpath-test-dynamicpv-2kgp": Phase="Pending", Reason="", readiness=false. Elapsed: 2m15.148175678s
Oct 22 06:39:48.453: INFO: Pod "pod-subpath-test-dynamicpv-2kgp": Phase="Pending", Reason="", readiness=false. Elapsed: 2m17.25693482s
Oct 22 06:39:50.562: INFO: Pod "pod-subpath-test-dynamicpv-2kgp": Phase="Succeeded", Reason="", readiness=false. Elapsed: 2m19.365902308s
STEP: Saw pod success
Oct 22 06:39:50.562: INFO: Pod "pod-subpath-test-dynamicpv-2kgp" satisfied condition "Succeeded or Failed"
Oct 22 06:39:50.670: INFO: Trying to get logs from node k8s-agentpool1-22475134-vmss000001 pod pod-subpath-test-dynamicpv-2kgp container test-container-volume-dynamicpv-2kgp: <nil>
STEP: delete the pod
Oct 22 06:39:50.895: INFO: Waiting for pod pod-subpath-test-dynamicpv-2kgp to disappear
Oct 22 06:39:51.002: INFO: Pod pod-subpath-test-dynamicpv-2kgp no longer exists
STEP: Deleting pod pod-subpath-test-dynamicpv-2kgp
Oct 22 06:39:51.002: INFO: Deleting pod "pod-subpath-test-dynamicpv-2kgp" in namespace "provisioning-873"
... skipping 41 lines ...
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (default fs)] subPath
  /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50
    should support non-existent path
    /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/subpath.go:194
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should support non-existent path","total":17,"completed":2,"skipped":373,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (filesystem volmode)] volumeMode should not mount / map unused volumes in a pod [LinuxOnly]"]}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Dynamic PV (ntfs)][Feature:Windows] provisioning
  /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:51
Oct 22 06:42:04.905: INFO: Driver test.csi.azure.com doesn't support ntfs -- skipping
... skipping 71 lines ...
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/external/external.go:174
  [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral
  /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50
    should create read-only inline ephemeral volume
    /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/ephemeral.go:149
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral should create read-only inline ephemeral volume","total":19,"completed":2,"skipped":325,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy (Always)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed in first pod, new pod with same fsgroup applied to the volume contents"]}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Pre-provisioned PV (ext4)] volumes
  /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:51
Oct 22 06:42:25.069: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping
... skipping 35 lines ...
  /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:186
Oct 22 06:42:25.978: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready
STEP: Destroying namespace "volumelimits-9347" for this suite.

•
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (filesystem volmode)] volumeLimits should verify that all csinodes have volume limits","total":19,"completed":3,"skipped":472,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy (Always)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed in first pod, new pod with same fsgroup applied to the volume contents"]}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Generic Ephemeral-volume (default fs) (immediate-binding)] ephemeral
  /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:51
[BeforeEach] [Testpattern: Generic Ephemeral-volume (default fs) (immediate-binding)] ephemeral
... skipping 67 lines ...
Oct 22 06:42:05.774: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comhvs4f] to have phase Bound
Oct 22 06:42:05.881: INFO: PersistentVolumeClaim test.csi.azure.comhvs4f found but phase is Pending instead of Bound.
Oct 22 06:42:07.990: INFO: PersistentVolumeClaim test.csi.azure.comhvs4f found but phase is Pending instead of Bound.
Oct 22 06:42:10.101: INFO: PersistentVolumeClaim test.csi.azure.comhvs4f found and phase=Bound (4.327729778s)
STEP: Expanding non-expandable pvc
Oct 22 06:42:10.316: INFO: currentPvcSize {{5368709120 0} {<nil>} 5Gi BinarySI}, newSize {{6442450944 0} {<nil>}  BinarySI}
Oct 22 06:42:10.531: INFO: Error updating pvc test.csi.azure.comhvs4f: persistentvolumeclaims "test.csi.azure.comhvs4f" is forbidden: only dynamically provisioned pvc can be resized and the storageclass that provisions the pvc must support resize
Oct 22 06:42:12.749: INFO: Error updating pvc test.csi.azure.comhvs4f: persistentvolumeclaims "test.csi.azure.comhvs4f" is forbidden: only dynamically provisioned pvc can be resized and the storageclass that provisions the pvc must support resize
Oct 22 06:42:14.748: INFO: Error updating pvc test.csi.azure.comhvs4f: persistentvolumeclaims "test.csi.azure.comhvs4f" is forbidden: only dynamically provisioned pvc can be resized and the storageclass that provisions the pvc must support resize
Oct 22 06:42:16.749: INFO: Error updating pvc test.csi.azure.comhvs4f: persistentvolumeclaims "test.csi.azure.comhvs4f" is forbidden: only dynamically provisioned pvc can be resized and the storageclass that provisions the pvc must support resize
Oct 22 06:42:18.749: INFO: Error updating pvc test.csi.azure.comhvs4f: persistentvolumeclaims "test.csi.azure.comhvs4f" is forbidden: only dynamically provisioned pvc can be resized and the storageclass that provisions the pvc must support resize
Oct 22 06:42:20.748: INFO: Error updating pvc test.csi.azure.comhvs4f: persistentvolumeclaims "test.csi.azure.comhvs4f" is forbidden: only dynamically provisioned pvc can be resized and the storageclass that provisions the pvc must support resize
Oct 22 06:42:22.756: INFO: Error updating pvc test.csi.azure.comhvs4f: persistentvolumeclaims "test.csi.azure.comhvs4f" is forbidden: only dynamically provisioned pvc can be resized and the storageclass that provisions the pvc must support resize
Oct 22 06:42:24.757: INFO: Error updating pvc test.csi.azure.comhvs4f: persistentvolumeclaims "test.csi.azure.comhvs4f" is forbidden: only dynamically provisioned pvc can be resized and the storageclass that provisions the pvc must support resize
Oct 22 06:42:26.746: INFO: Error updating pvc test.csi.azure.comhvs4f: persistentvolumeclaims "test.csi.azure.comhvs4f" is forbidden: only dynamically provisioned pvc can be resized and the storageclass that provisions the pvc must support resize
Oct 22 06:42:28.748: INFO: Error updating pvc test.csi.azure.comhvs4f: persistentvolumeclaims "test.csi.azure.comhvs4f" is forbidden: only dynamically provisioned pvc can be resized and the storageclass that provisions the pvc must support resize
Oct 22 06:42:30.749: INFO: Error updating pvc test.csi.azure.comhvs4f: persistentvolumeclaims "test.csi.azure.comhvs4f" is forbidden: only dynamically provisioned pvc can be resized and the storageclass that provisions the pvc must support resize
Oct 22 06:42:32.748: INFO: Error updating pvc test.csi.azure.comhvs4f: persistentvolumeclaims "test.csi.azure.comhvs4f" is forbidden: only dynamically provisioned pvc can be resized and the storageclass that provisions the pvc must support resize
Oct 22 06:42:34.747: INFO: Error updating pvc test.csi.azure.comhvs4f: persistentvolumeclaims "test.csi.azure.comhvs4f" is forbidden: only dynamically provisioned pvc can be resized and the storageclass that provisions the pvc must support resize
Oct 22 06:42:36.760: INFO: Error updating pvc test.csi.azure.comhvs4f: persistentvolumeclaims "test.csi.azure.comhvs4f" is forbidden: only dynamically provisioned pvc can be resized and the storageclass that provisions the pvc must support resize
Oct 22 06:42:38.748: INFO: Error updating pvc test.csi.azure.comhvs4f: persistentvolumeclaims "test.csi.azure.comhvs4f" is forbidden: only dynamically provisioned pvc can be resized and the storageclass that provisions the pvc must support resize
Oct 22 06:42:40.748: INFO: Error updating pvc test.csi.azure.comhvs4f: persistentvolumeclaims "test.csi.azure.comhvs4f" is forbidden: only dynamically provisioned pvc can be resized and the storageclass that provisions the pvc must support resize
Oct 22 06:42:40.965: INFO: Error updating pvc test.csi.azure.comhvs4f: persistentvolumeclaims "test.csi.azure.comhvs4f" is forbidden: only dynamically provisioned pvc can be resized and the storageclass that provisions the pvc must support resize
STEP: Deleting pvc
Oct 22 06:42:40.965: INFO: Deleting PersistentVolumeClaim "test.csi.azure.comhvs4f"
Oct 22 06:42:41.072: INFO: Waiting up to 5m0s for PersistentVolume pvc-1b1ac790-3b7b-4f7f-a54b-bd639857f2f7 to get deleted
Oct 22 06:42:41.179: INFO: PersistentVolume pvc-1b1ac790-3b7b-4f7f-a54b-bd639857f2f7 found and phase=Released (106.5392ms)
Oct 22 06:42:46.287: INFO: PersistentVolume pvc-1b1ac790-3b7b-4f7f-a54b-bd639857f2f7 found and phase=Released (5.214359821s)
Oct 22 06:42:51.395: INFO: PersistentVolume pvc-1b1ac790-3b7b-4f7f-a54b-bd639857f2f7 was removed
... skipping 9 lines ...
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (block volmode)] volume-expand
  /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50
    should not allow expansion of pvcs without AllowVolumeExpansion property
    /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/volume_expand.go:157
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] volume-expand should not allow expansion of pvcs without AllowVolumeExpansion property","total":17,"completed":3,"skipped":689,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (filesystem volmode)] volumeMode should not mount / map unused volumes in a pod [LinuxOnly]"]}

SSSSSSSSSSSSSS
------------------------------
External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] provisioning 
  should provision storage with snapshot data source [Feature:VolumeSnapshotDataSource]
  /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/provisioning.go:201
... skipping 100 lines ...
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (block volmode)] provisioning
  /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50
    should provision storage with snapshot data source [Feature:VolumeSnapshotDataSource]
    /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/provisioning.go:201
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] provisioning should provision storage with snapshot data source [Feature:VolumeSnapshotDataSource]","total":9,"completed":2,"skipped":738,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should be able to unmount after the subpath directory is deleted [LinuxOnly]"]}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Inline-volume (default fs)] subPath
  /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:51
Oct 22 06:43:26.423: INFO: Driver "test.csi.azure.com" does not support volume type "InlineVolume" - skipping
... skipping 36 lines ...
Oct 22 06:29:33.395: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil
Oct 22 06:29:33.507: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comb6jzz] to have phase Bound
Oct 22 06:29:33.614: INFO: PersistentVolumeClaim test.csi.azure.comb6jzz found but phase is Pending instead of Bound.
Oct 22 06:29:35.724: INFO: PersistentVolumeClaim test.csi.azure.comb6jzz found but phase is Pending instead of Bound.
Oct 22 06:29:37.855: INFO: PersistentVolumeClaim test.csi.azure.comb6jzz found and phase=Bound (4.347894283s)
STEP: [init] starting a pod to use the claim
Oct 22 06:29:38.200: INFO: Waiting up to 15m0s for pod "pvc-snapshottable-tester-2lssq" in namespace "snapshotting-2296" to be "Succeeded or Failed"
Oct 22 06:29:38.308: INFO: Pod "pvc-snapshottable-tester-2lssq": Phase="Pending", Reason="", readiness=false. Elapsed: 108.116834ms
Oct 22 06:29:40.418: INFO: Pod "pvc-snapshottable-tester-2lssq": Phase="Pending", Reason="", readiness=false. Elapsed: 2.218120267s
Oct 22 06:29:42.529: INFO: Pod "pvc-snapshottable-tester-2lssq": Phase="Pending", Reason="", readiness=false. Elapsed: 4.329454458s
Oct 22 06:29:44.638: INFO: Pod "pvc-snapshottable-tester-2lssq": Phase="Pending", Reason="", readiness=false. Elapsed: 6.438666873s
Oct 22 06:29:46.748: INFO: Pod "pvc-snapshottable-tester-2lssq": Phase="Pending", Reason="", readiness=false. Elapsed: 8.547879348s
Oct 22 06:29:48.856: INFO: Pod "pvc-snapshottable-tester-2lssq": Phase="Pending", Reason="", readiness=false. Elapsed: 10.65672916s
... skipping 22 lines ...
Oct 22 06:30:37.474: INFO: Pod "pvc-snapshottable-tester-2lssq": Phase="Pending", Reason="", readiness=false. Elapsed: 59.274440521s
Oct 22 06:30:39.583: INFO: Pod "pvc-snapshottable-tester-2lssq": Phase="Pending", Reason="", readiness=false. Elapsed: 1m1.383701848s
Oct 22 06:30:41.694: INFO: Pod "pvc-snapshottable-tester-2lssq": Phase="Pending", Reason="", readiness=false. Elapsed: 1m3.49462767s
Oct 22 06:30:43.804: INFO: Pod "pvc-snapshottable-tester-2lssq": Phase="Pending", Reason="", readiness=false. Elapsed: 1m5.604209177s
Oct 22 06:30:45.913: INFO: Pod "pvc-snapshottable-tester-2lssq": Phase="Succeeded", Reason="", readiness=false. Elapsed: 1m7.713298463s
STEP: Saw pod success
Oct 22 06:30:45.913: INFO: Pod "pvc-snapshottable-tester-2lssq" satisfied condition "Succeeded or Failed"
Oct 22 06:30:46.136: INFO: Pod pvc-snapshottable-tester-2lssq has the following logs: 
Oct 22 06:30:46.136: INFO: Deleting pod "pvc-snapshottable-tester-2lssq" in namespace "snapshotting-2296"
Oct 22 06:30:46.251: INFO: Wait up to 5m0s for pod "pvc-snapshottable-tester-2lssq" to be fully deleted
Oct 22 06:30:46.360: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comb6jzz] to have phase Bound
Oct 22 06:30:46.469: INFO: PersistentVolumeClaim test.csi.azure.comb6jzz found and phase=Bound (109.352399ms)
STEP: [init] checking the claim
... skipping 37 lines ...
[It] should check snapshot fields, check restore correctly works after modifying source data, check deletion
  /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/snapshottable.go:248
STEP: checking the snapshot
STEP: checking the SnapshotContent
STEP: Modifying source data test
STEP: modifying the data in the source PVC
Oct 22 06:31:01.784: INFO: Waiting up to 15m0s for pod "pvc-snapshottable-data-tester-smvj2" in namespace "snapshotting-2296" to be "Succeeded or Failed"
Oct 22 06:31:01.893: INFO: Pod "pvc-snapshottable-data-tester-smvj2": Phase="Pending", Reason="", readiness=false. Elapsed: 108.698175ms
Oct 22 06:31:04.002: INFO: Pod "pvc-snapshottable-data-tester-smvj2": Phase="Pending", Reason="", readiness=false. Elapsed: 2.218157604s
Oct 22 06:31:06.113: INFO: Pod "pvc-snapshottable-data-tester-smvj2": Phase="Pending", Reason="", readiness=false. Elapsed: 4.328309874s
Oct 22 06:31:08.230: INFO: Pod "pvc-snapshottable-data-tester-smvj2": Phase="Pending", Reason="", readiness=false. Elapsed: 6.445602542s
Oct 22 06:31:10.340: INFO: Pod "pvc-snapshottable-data-tester-smvj2": Phase="Pending", Reason="", readiness=false. Elapsed: 8.555592667s
Oct 22 06:31:12.449: INFO: Pod "pvc-snapshottable-data-tester-smvj2": Phase="Pending", Reason="", readiness=false. Elapsed: 10.664469418s
... skipping 286 lines ...
Oct 22 06:41:18.153: INFO: Pod "pvc-snapshottable-data-tester-smvj2": Phase="Pending", Reason="", readiness=false. Elapsed: 10m16.368607991s
Oct 22 06:41:20.263: INFO: Pod "pvc-snapshottable-data-tester-smvj2": Phase="Pending", Reason="", readiness=false. Elapsed: 10m18.478951751s
Oct 22 06:41:22.374: INFO: Pod "pvc-snapshottable-data-tester-smvj2": Phase="Pending", Reason="", readiness=false. Elapsed: 10m20.589928095s
Oct 22 06:41:24.485: INFO: Pod "pvc-snapshottable-data-tester-smvj2": Phase="Pending", Reason="", readiness=false. Elapsed: 10m22.700618265s
Oct 22 06:41:26.594: INFO: Pod "pvc-snapshottable-data-tester-smvj2": Phase="Succeeded", Reason="", readiness=false. Elapsed: 10m24.810209334s
STEP: Saw pod success
Oct 22 06:41:26.594: INFO: Pod "pvc-snapshottable-data-tester-smvj2" satisfied condition "Succeeded or Failed"
Oct 22 06:41:26.822: INFO: Pod pvc-snapshottable-data-tester-smvj2 has the following logs: 
Oct 22 06:41:26.822: INFO: Deleting pod "pvc-snapshottable-data-tester-smvj2" in namespace "snapshotting-2296"
Oct 22 06:41:26.936: INFO: Wait up to 5m0s for pod "pvc-snapshottable-data-tester-smvj2" to be fully deleted
STEP: creating a pvc from the snapshot
STEP: starting a pod to use the claim
Oct 22 06:42:39.497: INFO: Running '/usr/local/bin/kubectl --server=https://kubetest-5p3qtork.westeurope.cloudapp.azure.com --kubeconfig=/root/tmp1541861128/kubeconfig/kubeconfig.westeurope.json --namespace=snapshotting-2296 exec restored-pvc-tester-ql6t9 --namespace=snapshotting-2296 -- cat /mnt/test/data'
... skipping 45 lines ...
    /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/snapshottable.go:108
      
      /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/snapshottable.go:230
        should check snapshot fields, check restore correctly works after modifying source data, check deletion
        /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/snapshottable.go:248
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Pre-provisioned Snapshot (delete policy)] snapshottable[Feature:VolumeSnapshotDataSource] volume snapshot controller  should check snapshot fields, check restore correctly works after modifying source data, check deletion","total":12,"completed":2,"skipped":250,"failed":0}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Dynamic PV (ntfs)][Feature:Windows] subPath
  /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:51
Oct 22 06:43:31.822: INFO: Driver test.csi.azure.com doesn't support ntfs -- skipping
... skipping 125 lines ...
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (block volmode)] volumes
  /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50
    should store data
    /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/volumes.go:159
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] volumes should store data","total":18,"completed":3,"skipped":203,"failed":0}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Pre-provisioned PV (default fs)] subPath
  /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:51
Oct 22 06:43:42.159: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping
... skipping 134 lines ...
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/external/external.go:174
  [Testpattern: Generic Ephemeral-volume (default fs) (immediate-binding)] ephemeral
  /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50
    should create read/write inline ephemeral volume
    /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/ephemeral.go:166
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Generic Ephemeral-volume (default fs) (immediate-binding)] ephemeral should create read/write inline ephemeral volume","total":13,"completed":2,"skipped":771,"failed":0}

SSSSSSSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Pre-provisioned PV (default fs)] volumes
  /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:51
Oct 22 06:44:07.351: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping
... skipping 20 lines ...
[BeforeEach] [Testpattern: Dynamic PV (delayed binding)] topology
  /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:185
STEP: Creating a kubernetes client
Oct 22 06:44:07.392: INFO: >>> kubeConfig: /root/tmp1541861128/kubeconfig/kubeconfig.westeurope.json
STEP: Building a namespace api object, basename topology
STEP: Waiting for a default service account to be provisioned in namespace
[It] should fail to schedule a pod which has topologies that conflict with AllowedTopologies
  /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/topology.go:192
Oct 22 06:44:07.936: INFO: Driver didn't provide topology keys -- skipping
[AfterEach] [Testpattern: Dynamic PV (delayed binding)] topology
  /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:186
Oct 22 06:44:07.936: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready
STEP: Destroying namespace "topology-7054" for this suite.


S [SKIPPING] [0.764 seconds]
External Storage [Driver: test.csi.azure.com]
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (delayed binding)] topology
  /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50
    should fail to schedule a pod which has topologies that conflict with AllowedTopologies [Measurement]
    /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/topology.go:192

    Driver didn't provide topology keys -- skipping

    /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/topology.go:124
------------------------------
... skipping 73 lines ...
Oct 22 06:43:43.117: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comvnkfv] to have phase Bound
Oct 22 06:43:43.224: INFO: PersistentVolumeClaim test.csi.azure.comvnkfv found but phase is Pending instead of Bound.
Oct 22 06:43:45.334: INFO: PersistentVolumeClaim test.csi.azure.comvnkfv found but phase is Pending instead of Bound.
Oct 22 06:43:47.442: INFO: PersistentVolumeClaim test.csi.azure.comvnkfv found and phase=Bound (4.325053215s)
STEP: Expanding non-expandable pvc
Oct 22 06:43:47.656: INFO: currentPvcSize {{5368709120 0} {<nil>} 5Gi BinarySI}, newSize {{6442450944 0} {<nil>}  BinarySI}
Oct 22 06:43:47.872: INFO: Error updating pvc test.csi.azure.comvnkfv: persistentvolumeclaims "test.csi.azure.comvnkfv" is forbidden: only dynamically provisioned pvc can be resized and the storageclass that provisions the pvc must support resize
Oct 22 06:43:50.087: INFO: Error updating pvc test.csi.azure.comvnkfv: persistentvolumeclaims "test.csi.azure.comvnkfv" is forbidden: only dynamically provisioned pvc can be resized and the storageclass that provisions the pvc must support resize
Oct 22 06:43:52.089: INFO: Error updating pvc test.csi.azure.comvnkfv: persistentvolumeclaims "test.csi.azure.comvnkfv" is forbidden: only dynamically provisioned pvc can be resized and the storageclass that provisions the pvc must support resize
Oct 22 06:43:54.090: INFO: Error updating pvc test.csi.azure.comvnkfv: persistentvolumeclaims "test.csi.azure.comvnkfv" is forbidden: only dynamically provisioned pvc can be resized and the storageclass that provisions the pvc must support resize
Oct 22 06:43:56.088: INFO: Error updating pvc test.csi.azure.comvnkfv: persistentvolumeclaims "test.csi.azure.comvnkfv" is forbidden: only dynamically provisioned pvc can be resized and the storageclass that provisions the pvc must support resize
Oct 22 06:43:58.088: INFO: Error updating pvc test.csi.azure.comvnkfv: persistentvolumeclaims "test.csi.azure.comvnkfv" is forbidden: only dynamically provisioned pvc can be resized and the storageclass that provisions the pvc must support resize
Oct 22 06:44:00.087: INFO: Error updating pvc test.csi.azure.comvnkfv: persistentvolumeclaims "test.csi.azure.comvnkfv" is forbidden: only dynamically provisioned pvc can be resized and the storageclass that provisions the pvc must support resize
Oct 22 06:44:02.089: INFO: Error updating pvc test.csi.azure.comvnkfv: persistentvolumeclaims "test.csi.azure.comvnkfv" is forbidden: only dynamically provisioned pvc can be resized and the storageclass that provisions the pvc must support resize
Oct 22 06:44:04.088: INFO: Error updating pvc test.csi.azure.comvnkfv: persistentvolumeclaims "test.csi.azure.comvnkfv" is forbidden: only dynamically provisioned pvc can be resized and the storageclass that provisions the pvc must support resize
Oct 22 06:44:06.087: INFO: Error updating pvc test.csi.azure.comvnkfv: persistentvolumeclaims "test.csi.azure.comvnkfv" is forbidden: only dynamically provisioned pvc can be resized and the storageclass that provisions the pvc must support resize
Oct 22 06:44:08.088: INFO: Error updating pvc test.csi.azure.comvnkfv: persistentvolumeclaims "test.csi.azure.comvnkfv" is forbidden: only dynamically provisioned pvc can be resized and the storageclass that provisions the pvc must support resize
Oct 22 06:44:10.089: INFO: Error updating pvc test.csi.azure.comvnkfv: persistentvolumeclaims "test.csi.azure.comvnkfv" is forbidden: only dynamically provisioned pvc can be resized and the storageclass that provisions the pvc must support resize
Oct 22 06:44:12.088: INFO: Error updating pvc test.csi.azure.comvnkfv: persistentvolumeclaims "test.csi.azure.comvnkfv" is forbidden: only dynamically provisioned pvc can be resized and the storageclass that provisions the pvc must support resize
Oct 22 06:44:14.089: INFO: Error updating pvc test.csi.azure.comvnkfv: persistentvolumeclaims "test.csi.azure.comvnkfv" is forbidden: only dynamically provisioned pvc can be resized and the storageclass that provisions the pvc must support resize
Oct 22 06:44:16.088: INFO: Error updating pvc test.csi.azure.comvnkfv: persistentvolumeclaims "test.csi.azure.comvnkfv" is forbidden: only dynamically provisioned pvc can be resized and the storageclass that provisions the pvc must support resize
Oct 22 06:44:18.089: INFO: Error updating pvc test.csi.azure.comvnkfv: persistentvolumeclaims "test.csi.azure.comvnkfv" is forbidden: only dynamically provisioned pvc can be resized and the storageclass that provisions the pvc must support resize
Oct 22 06:44:18.304: INFO: Error updating pvc test.csi.azure.comvnkfv: persistentvolumeclaims "test.csi.azure.comvnkfv" is forbidden: only dynamically provisioned pvc can be resized and the storageclass that provisions the pvc must support resize
STEP: Deleting pvc
Oct 22 06:44:18.304: INFO: Deleting PersistentVolumeClaim "test.csi.azure.comvnkfv"
Oct 22 06:44:18.412: INFO: Waiting up to 5m0s for PersistentVolume pvc-d6c052b6-f54f-44e8-a567-4abeecf87be1 to get deleted
Oct 22 06:44:18.519: INFO: PersistentVolume pvc-d6c052b6-f54f-44e8-a567-4abeecf87be1 found and phase=Released (107.21921ms)
Oct 22 06:44:23.627: INFO: PersistentVolume pvc-d6c052b6-f54f-44e8-a567-4abeecf87be1 found and phase=Released (5.215271603s)
Oct 22 06:44:28.736: INFO: PersistentVolume pvc-d6c052b6-f54f-44e8-a567-4abeecf87be1 was removed
... skipping 9 lines ...
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (default fs)] volume-expand
  /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50
    should not allow expansion of pvcs without AllowVolumeExpansion property
    /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/volume_expand.go:157
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] volume-expand should not allow expansion of pvcs without AllowVolumeExpansion property","total":18,"completed":4,"skipped":461,"failed":0}

SSSSS
------------------------------
[BeforeEach] [Testpattern: Dynamic PV (ext4)] volumes
  /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:51
Oct 22 06:44:29.176: INFO: Driver test.csi.azure.com doesn't support ext4 -- skipping
... skipping 132 lines ...
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy
  /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50
    (Always)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed in first pod, new pod with different fsgroup applied to the volume contents
    /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/fsgroupchangepolicy.go:208
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy (Always)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed in first pod, new pod with different fsgroup applied to the volume contents","total":17,"completed":4,"skipped":703,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (filesystem volmode)] volumeMode should not mount / map unused volumes in a pod [LinuxOnly]"]}

SSSSSSSSSSSSSS
------------------------------
External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] volumeMode 
  should not mount / map unused volumes in a pod [LinuxOnly]
  /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/volumemode.go:351
... skipping 92 lines ...
Oct 22 06:44:48.927: INFO: PersistentVolume pvc-1943503f-e1dc-4825-81a8-b38b96df76b0 found and phase=Released (4m36.147933225s)
Oct 22 06:44:54.037: INFO: PersistentVolume pvc-1943503f-e1dc-4825-81a8-b38b96df76b0 found and phase=Released (4m41.258430693s)
Oct 22 06:44:59.146: INFO: PersistentVolume pvc-1943503f-e1dc-4825-81a8-b38b96df76b0 found and phase=Released (4m46.366893488s)
Oct 22 06:45:04.255: INFO: PersistentVolume pvc-1943503f-e1dc-4825-81a8-b38b96df76b0 found and phase=Released (4m51.476160861s)
Oct 22 06:45:09.364: INFO: PersistentVolume pvc-1943503f-e1dc-4825-81a8-b38b96df76b0 found and phase=Released (4m56.58504106s)
STEP: Deleting sc
Oct 22 06:45:14.476: FAIL: while cleaning up resource
Unexpected error:
    <errors.aggregate | len:1, cap:1>: [
        [
            {
                msg: "persistent Volume pvc-1943503f-e1dc-4825-81a8-b38b96df76b0 not deleted by dynamic provisioner: PersistentVolume pvc-1943503f-e1dc-4825-81a8-b38b96df76b0 still exists within 5m0s",
                err: {
                    s: "PersistentVolume pvc-1943503f-e1dc-4825-81a8-b38b96df76b0 still exists within 5m0s",
... skipping 33 lines ...
Oct 22 06:45:14.587: INFO: At 2021-10-22 06:40:03 +0000 UTC - event for hostexec-k8s-agentpool1-22475134-vmss000001-6rlhs: {default-scheduler } Scheduled: Successfully assigned volumemode-2028/hostexec-k8s-agentpool1-22475134-vmss000001-6rlhs to k8s-agentpool1-22475134-vmss000001
Oct 22 06:45:14.587: INFO: At 2021-10-22 06:40:04 +0000 UTC - event for hostexec-k8s-agentpool1-22475134-vmss000001-6rlhs: {kubelet k8s-agentpool1-22475134-vmss000001} Created: Created container agnhost-container
Oct 22 06:45:14.587: INFO: At 2021-10-22 06:40:04 +0000 UTC - event for hostexec-k8s-agentpool1-22475134-vmss000001-6rlhs: {kubelet k8s-agentpool1-22475134-vmss000001} Pulled: Successfully pulled image "k8s.gcr.io/e2e-test-images/agnhost:2.32" in 880.569029ms
Oct 22 06:45:14.587: INFO: At 2021-10-22 06:40:05 +0000 UTC - event for hostexec-k8s-agentpool1-22475134-vmss000001-6rlhs: {kubelet k8s-agentpool1-22475134-vmss000001} Started: Started container agnhost-container
Oct 22 06:45:14.587: INFO: At 2021-10-22 06:40:10 +0000 UTC - event for hostexec-k8s-agentpool1-22475134-vmss000001-6rlhs: {kubelet k8s-agentpool1-22475134-vmss000001} Killing: Stopping container agnhost-container
Oct 22 06:45:14.587: INFO: At 2021-10-22 06:40:10 +0000 UTC - event for pod-90cb57c0-404c-4f88-8a4c-ba40d0d8a593: {kubelet k8s-agentpool1-22475134-vmss000001} Killing: Stopping container write-pod
Oct 22 06:45:14.587: INFO: At 2021-10-22 06:42:00 +0000 UTC - event for pod-90cb57c0-404c-4f88-8a4c-ba40d0d8a593: {attachdetach-controller } FailedAttachVolume: AttachVolume.Attach failed for volume "pvc-1943503f-e1dc-4825-81a8-b38b96df76b0" : Attach timeout for volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-1943503f-e1dc-4825-81a8-b38b96df76b0
Oct 22 06:45:14.697: INFO: POD  NODE  PHASE  GRACE  CONDITIONS
Oct 22 06:45:14.697: INFO: 
Oct 22 06:45:14.913: INFO: 
Logging node info for node k8s-agentpool1-22475134-vmss000000
Oct 22 06:45:15.022: INFO: Node Info: &Node{ObjectMeta:{k8s-agentpool1-22475134-vmss000000    72bf23fc-217b-4f1a-9ed6-d90fb28f8d04 7639 0 2021-10-22 06:15:58 +0000 UTC <nil> <nil> map[agentpool:agentpool1 beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_DS4_v2 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westeurope failure-domain.beta.kubernetes.io/zone:0 kubernetes.azure.com/cluster:kubetest-5p3qtork kubernetes.azure.com/role:agent kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-agentpool1-22475134-vmss000000 kubernetes.io/os:linux kubernetes.io/role:agent node-role.kubernetes.io/agent: node.kubernetes.io/instance-type:Standard_DS4_v2 storageprofile:managed storagetier:Premium_LRS topology.kubernetes.io/region:westeurope topology.kubernetes.io/zone:0 topology.test.csi.azure.com/zone:] map[csi.volume.kubernetes.io/nodeid:{"test.csi.azure.com":"k8s-agentpool1-22475134-vmss000000"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] []  [{kubelet Update v1 2021-10-22 06:15:58 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:agentpool":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.azure.com/cluster":{},"f:kubernetes.azure.com/role":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:storageprofile":{},"f:storagetier":{}}}} } {kubectl-label Update v1 2021-10-22 06:16:06 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/agent":{}}}} } {cloud-controller-manager Update v1 2021-10-22 06:16:12 +0000 UTC FieldsV1 {"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"10.244.0.0/24\"":{}}}} } {cloud-controller-manager Update v1 2021-10-22 06:16:12 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2021-10-22 06:16:15 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}}} } {cloud-node-manager Update v1 2021-10-22 06:17:01 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kubelet Update v1 2021-10-22 06:27:43 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.test.csi.azure.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status} {kube-controller-manager Update v1 2021-10-22 06:44:35 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status}]},Spec:NodeSpec{PodCIDR:10.244.0.0/24,DoNotUseExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool1-22475134-vmss/virtualMachines/0,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[10.244.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{8 0} {<nil>} 8 DecimalSI},ephemeral-storage: {{31036686336 0} {<nil>}  BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{29449355264 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{8 0} {<nil>} 8 DecimalSI},ephemeral-storage: {{27933017657 0} {<nil>} 27933017657 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{28662923264 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2021-10-22 06:17:09 +0000 UTC,LastTransitionTime:2021-10-22 06:17:09 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2021-10-22 06:44:51 +0000 UTC,LastTransitionTime:2021-10-22 06:15:55 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2021-10-22 06:44:51 +0000 UTC,LastTransitionTime:2021-10-22 06:15:55 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2021-10-22 06:44:51 +0000 UTC,LastTransitionTime:2021-10-22 06:15:55 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2021-10-22 06:44:51 +0000 UTC,LastTransitionTime:2021-10-22 06:16:12 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.240.0.4,},NodeAddress{Type:Hostname,Address:k8s-agentpool1-22475134-vmss000000,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:d2790c386f8b4816867c93ee5ec07afb,SystemUUID:99c573b5-645b-634d-bb56-59a354ec6405,BootID:87fd645e-5f99-4dfe-9c69-ecf7970fbbbf,KernelVersion:5.4.0-1059-azure,OSImage:Ubuntu 18.04.6 LTS,ContainerRuntimeVersion:docker://20.10.7+azure,KubeletVersion:v1.22.2,KubeProxyVersion:v1.22.2,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/azuredisk-csi@sha256:0743369a58a6620586dcf8671640b81f8dc66bc5986a5c2d3b23a4ce6a9886e8 k8sprow.azurecr.io/azuredisk-csi:v1.9.0-716f56a44514b799d2d6c6922145f22026e7d4e2],SizeBytes:239897501,},ContainerImage{Names:[mcr.microsoft.com/k8s/csi/azurefile-csi@sha256:91972069fc98aae86e3297b054cb0a347b256d0a81799337e1ec0c196c341065 mcr.microsoft.com/k8s/csi/azurefile-csi:v1.6.0],SizeBytes:223047280,},ContainerImage{Names:[mcr.microsoft.com/k8s/csi/azuredisk-csi@sha256:d1e335c6d52dc8b0f7db7650879e1673fda5c561761a52f9ba72a216d9ca0721 mcr.microsoft.com/k8s/csi/azuredisk-csi:v1.7.0],SizeBytes:212174459,},ContainerImage{Names:[mcr.microsoft.com/containernetworking/azure-npm@sha256:106f669f48e5e80c4ec0afb49858ead72cf4b901cd8664e7bf81f8d789e56e12 mcr.microsoft.com/containernetworking/azure-npm:v1.2.2_hotfix],SizeBytes:175230380,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-apiserver@sha256:6c37c0efdaf49d24dafcced601fb78750205670e4e41842ab3a71935ed23b753 mcr.microsoft.com/oss/kubernetes/kube-apiserver:v1.22.2],SizeBytes:128401987,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:758db666ac7028534dba72e7e9bb1e57bb81b8196f976f7a5cc351ef8b3529e1 k8s.gcr.io/e2e-test-images/agnhost:2.32],SizeBytes:125910179,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/secrets-store/driver@sha256:c0d040a1c4fbfceb65663e31c09ea40f4f78e356437610cbc3fbb4bb409bd6f1 mcr.microsoft.com/oss/kubernetes-csi/secrets-store/driver:v0.0.19],SizeBytes:123229697,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-controller-manager@sha256:76fa95d463bf8950324d68f9f53063bb73628adcc862708f4534d6f44546df63 mcr.microsoft.com/oss/kubernetes/kube-controller-manager:v1.22.2],SizeBytes:121930716,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-proxy@sha256:b1ec3f3090fc799799bee98619a33300d8ee9095219adc4bb688084718810d23 mcr.microsoft.com/oss/kubernetes/kube-proxy:v1.22.2],SizeBytes:103625747,},ContainerImage{Names:[mcr.microsoft.com/oss/azure/secrets-store/provider-azure@sha256:6f67f3d0c7cdde5702f8ce7f101b6519daa0237f0c34fecb7c058b6af8c22ad1 mcr.microsoft.com/oss/azure/secrets-store/provider-azure:0.0.12],SizeBytes:101061355,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/autoscaler/cluster-autoscaler@sha256:29ba45b98c8035adea245f124d08d54dffbd27068940edd879cfd86022db20f9 mcr.microsoft.com/oss/kubernetes/autoscaler/cluster-autoscaler:v1.22.0],SizeBytes:99406403,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-addon-manager@sha256:32e2836018c96e73533bd4642fe438e465b81dcbfa8b7b61935a6f4d0246c7ae mcr.microsoft.com/oss/kubernetes/kube-addon-manager:v9.1.3],SizeBytes:86832059,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-addon-manager@sha256:92c2c5aad9012ee32d2a43a74966cc0adc6ccb1705ad15abb10485ecf406d88b mcr.microsoft.com/oss/kubernetes/kube-addon-manager:v9.1.5],SizeBytes:84094027,},ContainerImage{Names:[mcr.microsoft.com/oss/nvidia/k8s-device-plugin@sha256:0f5b52bf28239234e831697d96db63ac03cde70fe68058f964504ab7564ee810 mcr.microsoft.com/oss/nvidia/k8s-device-plugin:1.0.0-beta6],SizeBytes:64160241,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/metrics-server@sha256:682d3a1a48abbe03df05e1084923ab668a6651fd80327f9694b0cf302cc49f90 mcr.microsoft.com/oss/kubernetes/metrics-server:v0.5.0],SizeBytes:63501445,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-provisioner@sha256:e9ddadc44ba87a4a27f67e54760a14f9986885b534b3dff170a14eae1e35d213 mcr.microsoft.com/oss/kubernetes-csi/csi-provisioner:v3.0.0],SizeBytes:56881280,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-resizer@sha256:c5bb71ceaac60b1a4b58739fa07b709f6248c452ff6272a384d2f7648895a750 mcr.microsoft.com/oss/kubernetes-csi/csi-resizer:v1.3.0],SizeBytes:54313772,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-snapshotter@sha256:e28e996e03519565907d7158e0e1a6613cec76220fa8625e0ebfc1aa1c3bacb2 mcr.microsoft.com/oss/kubernetes-csi/csi-snapshotter:v4.2.1],SizeBytes:54210936,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-attacher@sha256:6b41e7153ebdfdc1501aa65184624bc15fd33a52d93f88ec3a758d0f8c9b8c10 mcr.microsoft.com/oss/kubernetes-csi/csi-attacher:v3.3.0],SizeBytes:53842561,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-scheduler@sha256:0457773d5575596d4f898913b45293f77dea0d955bc2002a4dbf7ea827cfe4f0 mcr.microsoft.com/oss/kubernetes/kube-scheduler:v1.22.2],SizeBytes:52626428,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/snapshot-controller@sha256:bb61653b29713588c280999cbb25b472fef8794623164fd9177c270da1121db0 mcr.microsoft.com/oss/kubernetes-csi/snapshot-controller:v4.2.1],SizeBytes:51575245,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager@sha256:ea0065331ce28081b4a048bd18530bbe4fe675dacb3c9515a2022ef10abf95ff mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager:v1.1.1],SizeBytes:50901571,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager@sha256:c24707c5f947d9e384d0f111469ca3266a01c48de6557f306e718c23d699cadf mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager:v1.1.1],SizeBytes:50295363,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/ip-masq-agent@sha256:bddf19be25644c7176e8c501465dbad07d032288334f135877b984660b0d3ee6 mcr.microsoft.com/oss/kubernetes/ip-masq-agent:v2.5.0],SizeBytes:50146762,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/coredns@sha256:1c23b17c78ffbc90c068ee088b77456284b9649952fa127b0f86a0ccff575539 mcr.microsoft.com/oss/kubernetes/coredns:1.7.0],SizeBytes:45233936,},ContainerImage{Names:[mcr.microsoft.com/oss/etcd-io/etcd@sha256:cf587862e3f1b6fa4d9a2565520a34f164bdf72c50f37af8c3c668160593246e mcr.microsoft.com/oss/etcd-io/etcd:v3.3.25],SizeBytes:41832119,},ContainerImage{Names:[mcr.microsoft.com/k8s/aad-pod-identity/mic@sha256:bd9465be94966b9a917e1e3904fa5e63dd91772ccadf304e18ffd8e4ad8ccedd mcr.microsoft.com/k8s/aad-pod-identity/mic:1.6.1],SizeBytes:41374894,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/autoscaler/cluster-proportional-autoscaler@sha256:753e3ca79dc833feed24ee4107ff4f06a96abbd6cefba8d41d7dec6f4d3e75f0 mcr.microsoft.com/oss/kubernetes/autoscaler/cluster-proportional-autoscaler:1.7.1],SizeBytes:40051560,},ContainerImage{Names:[mcr.microsoft.com/k8s/aad-pod-identity/nmi@sha256:02128fefcdb7593ac53fc342e2c53a0fc6fabd813036bf60457bf43cc2940116 mcr.microsoft.com/k8s/aad-pod-identity/nmi:1.6.1],SizeBytes:38007982,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:921f301c44dda06a325164accf22e78ecc570b5c7d9d6ee4c66bd8cbb2b60b9a mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.16],SizeBytes:26970670,},ContainerImage{Names:[mcr.microsoft.com/k8s/kms/keyvault@sha256:1a27e175f8c125209e32d2957b5509fe20757bd8cb309ff9da598799b56326fb mcr.microsoft.com/k8s/kms/keyvault:v0.0.10],SizeBytes:23077387,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-node-driver-registrar@sha256:f1205ba7ae0b0783bb19442064f4d7d664f3dc0c9b0b5b33358e5eef5aedc554 mcr.microsoft.com/oss/kubernetes-csi/csi-node-driver-registrar:v2.3.0],SizeBytes:18746317,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/livenessprobe@sha256:3770688d4efa33f8f284cc2ef58d33efa4ff606147e0a8de20e67458e13fedc2 mcr.microsoft.com/oss/kubernetes-csi/livenessprobe:v2.4.0],SizeBytes:17183286,},ContainerImage{Names:[busybox@sha256:f7ca5a32c10d51aeda3b4d01c61c6061f497893d7f6628b92f822f7117182a57 busybox:latest],SizeBytes:1235724,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8],SizeBytes:1159025,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:1154361,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/pause@sha256:e3b8c20681593c21b344ad801fbb8abaf564427ee3a57a9fcfa3b455f917ce46 mcr.microsoft.com/oss/kubernetes/pause:3.4.1],SizeBytes:682696,},},VolumesInUse:[kubernetes.io/csi/test.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-16d55a1f-a769-40b5-8384-92940fcc54f1 kubernetes.io/csi/test.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-2ead2c64-3a60-46c4-940f-249658572375 kubernetes.io/csi/test.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-6894fdc9-9e02-446f-92bb-2b026c63a266],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/test.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-2ead2c64-3a60-46c4-940f-249658572375,DevicePath:,},},Config:nil,},}
Oct 22 06:45:15.023: INFO: 
... skipping 98 lines ...
  [Testpattern: Dynamic PV (block volmode)] volumeMode
  /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50
    should not mount / map unused volumes in a pod [LinuxOnly] [Measurement]
    /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/volumemode.go:351

    Oct 22 06:45:14.476: while cleaning up resource
    Unexpected error:
        <errors.aggregate | len:1, cap:1>: [
            [
                {
                    msg: "persistent Volume pvc-1943503f-e1dc-4825-81a8-b38b96df76b0 not deleted by dynamic provisioner: PersistentVolume pvc-1943503f-e1dc-4825-81a8-b38b96df76b0 still exists within 5m0s",
                    err: {
                        s: "PersistentVolume pvc-1943503f-e1dc-4825-81a8-b38b96df76b0 still exists within 5m0s",
... skipping 3 lines ...
        ]
        persistent Volume pvc-1943503f-e1dc-4825-81a8-b38b96df76b0 not deleted by dynamic provisioner: PersistentVolume pvc-1943503f-e1dc-4825-81a8-b38b96df76b0 still exists within 5m0s
    occurred

    /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/volumemode.go:188
------------------------------
{"msg":"FAILED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] volumeMode should not mount / map unused volumes in a pod [LinuxOnly]","total":18,"completed":1,"skipped":177,"failed":2,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy (OnRootMismatch)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed in first pod, new pod with same fsgroup skips ownership changes to the volume contents","External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] volumeMode should not mount / map unused volumes in a pod [LinuxOnly]"]}

SSSSSSSSSSSSSS
------------------------------
External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic Snapshot (delete policy)] snapshottable[Feature:VolumeSnapshotDataSource] volume snapshot controller  
  should check snapshot fields, check restore correctly works after modifying source data, check deletion
  /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/snapshottable.go:248
... skipping 15 lines ...
Oct 22 06:42:27.749: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil
Oct 22 06:42:27.859: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comhw7mn] to have phase Bound
Oct 22 06:42:27.966: INFO: PersistentVolumeClaim test.csi.azure.comhw7mn found but phase is Pending instead of Bound.
Oct 22 06:42:30.076: INFO: PersistentVolumeClaim test.csi.azure.comhw7mn found but phase is Pending instead of Bound.
Oct 22 06:42:32.184: INFO: PersistentVolumeClaim test.csi.azure.comhw7mn found and phase=Bound (4.325478174s)
STEP: [init] starting a pod to use the claim
Oct 22 06:42:32.509: INFO: Waiting up to 15m0s for pod "pvc-snapshottable-tester-zjtzn" in namespace "snapshotting-8809" to be "Succeeded or Failed"
Oct 22 06:42:32.617: INFO: Pod "pvc-snapshottable-tester-zjtzn": Phase="Pending", Reason="", readiness=false. Elapsed: 107.712282ms
Oct 22 06:42:34.726: INFO: Pod "pvc-snapshottable-tester-zjtzn": Phase="Pending", Reason="", readiness=false. Elapsed: 2.216209744s
Oct 22 06:42:36.835: INFO: Pod "pvc-snapshottable-tester-zjtzn": Phase="Pending", Reason="", readiness=false. Elapsed: 4.325129552s
Oct 22 06:42:38.943: INFO: Pod "pvc-snapshottable-tester-zjtzn": Phase="Pending", Reason="", readiness=false. Elapsed: 6.432990995s
Oct 22 06:42:41.051: INFO: Pod "pvc-snapshottable-tester-zjtzn": Phase="Pending", Reason="", readiness=false. Elapsed: 8.541007037s
Oct 22 06:42:43.158: INFO: Pod "pvc-snapshottable-tester-zjtzn": Phase="Pending", Reason="", readiness=false. Elapsed: 10.648332538s
Oct 22 06:42:45.267: INFO: Pod "pvc-snapshottable-tester-zjtzn": Phase="Pending", Reason="", readiness=false. Elapsed: 12.757409854s
Oct 22 06:42:47.376: INFO: Pod "pvc-snapshottable-tester-zjtzn": Phase="Pending", Reason="", readiness=false. Elapsed: 14.866809706s
Oct 22 06:42:49.484: INFO: Pod "pvc-snapshottable-tester-zjtzn": Phase="Pending", Reason="", readiness=false. Elapsed: 16.974717581s
Oct 22 06:42:51.592: INFO: Pod "pvc-snapshottable-tester-zjtzn": Phase="Succeeded", Reason="", readiness=false. Elapsed: 19.082083241s
STEP: Saw pod success
Oct 22 06:42:51.592: INFO: Pod "pvc-snapshottable-tester-zjtzn" satisfied condition "Succeeded or Failed"
Oct 22 06:42:51.813: INFO: Pod pvc-snapshottable-tester-zjtzn has the following logs: 
Oct 22 06:42:51.813: INFO: Deleting pod "pvc-snapshottable-tester-zjtzn" in namespace "snapshotting-8809"
Oct 22 06:42:51.930: INFO: Wait up to 5m0s for pod "pvc-snapshottable-tester-zjtzn" to be fully deleted
Oct 22 06:42:52.037: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comhw7mn] to have phase Bound
Oct 22 06:42:52.144: INFO: PersistentVolumeClaim test.csi.azure.comhw7mn found and phase=Bound (107.085586ms)
STEP: [init] checking the claim
... skipping 19 lines ...
[It] should check snapshot fields, check restore correctly works after modifying source data, check deletion
  /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/snapshottable.go:248
STEP: checking the snapshot
STEP: checking the SnapshotContent
STEP: Modifying source data test
STEP: modifying the data in the source PVC
Oct 22 06:43:06.523: INFO: Waiting up to 15m0s for pod "pvc-snapshottable-data-tester-fz8zv" in namespace "snapshotting-8809" to be "Succeeded or Failed"
Oct 22 06:43:06.630: INFO: Pod "pvc-snapshottable-data-tester-fz8zv": Phase="Pending", Reason="", readiness=false. Elapsed: 107.061037ms
Oct 22 06:43:08.739: INFO: Pod "pvc-snapshottable-data-tester-fz8zv": Phase="Pending", Reason="", readiness=false. Elapsed: 2.215912288s
Oct 22 06:43:10.848: INFO: Pod "pvc-snapshottable-data-tester-fz8zv": Phase="Pending", Reason="", readiness=false. Elapsed: 4.324636666s
Oct 22 06:43:12.957: INFO: Pod "pvc-snapshottable-data-tester-fz8zv": Phase="Pending", Reason="", readiness=false. Elapsed: 6.4332027s
Oct 22 06:43:15.066: INFO: Pod "pvc-snapshottable-data-tester-fz8zv": Phase="Pending", Reason="", readiness=false. Elapsed: 8.542291532s
Oct 22 06:43:17.174: INFO: Pod "pvc-snapshottable-data-tester-fz8zv": Phase="Pending", Reason="", readiness=false. Elapsed: 10.650703274s
... skipping 22 lines ...
Oct 22 06:44:05.697: INFO: Pod "pvc-snapshottable-data-tester-fz8zv": Phase="Pending", Reason="", readiness=false. Elapsed: 59.173796229s
Oct 22 06:44:07.807: INFO: Pod "pvc-snapshottable-data-tester-fz8zv": Phase="Pending", Reason="", readiness=false. Elapsed: 1m1.283614961s
Oct 22 06:44:09.917: INFO: Pod "pvc-snapshottable-data-tester-fz8zv": Phase="Pending", Reason="", readiness=false. Elapsed: 1m3.394156841s
Oct 22 06:44:12.027: INFO: Pod "pvc-snapshottable-data-tester-fz8zv": Phase="Pending", Reason="", readiness=false. Elapsed: 1m5.50320029s
Oct 22 06:44:14.135: INFO: Pod "pvc-snapshottable-data-tester-fz8zv": Phase="Succeeded", Reason="", readiness=false. Elapsed: 1m7.612134439s
STEP: Saw pod success
Oct 22 06:44:14.136: INFO: Pod "pvc-snapshottable-data-tester-fz8zv" satisfied condition "Succeeded or Failed"
Oct 22 06:44:14.359: INFO: Pod pvc-snapshottable-data-tester-fz8zv has the following logs: 
Oct 22 06:44:14.359: INFO: Deleting pod "pvc-snapshottable-data-tester-fz8zv" in namespace "snapshotting-8809"
Oct 22 06:44:14.475: INFO: Wait up to 5m0s for pod "pvc-snapshottable-data-tester-fz8zv" to be fully deleted
STEP: creating a pvc from the snapshot
STEP: starting a pod to use the claim
Oct 22 06:44:39.026: INFO: Running '/usr/local/bin/kubectl --server=https://kubetest-5p3qtork.westeurope.cloudapp.azure.com --kubeconfig=/root/tmp1541861128/kubeconfig/kubeconfig.westeurope.json --namespace=snapshotting-8809 exec restored-pvc-tester-cxwbq --namespace=snapshotting-8809 -- cat /mnt/test/data'
... skipping 45 lines ...
    /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/snapshottable.go:108
      
      /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/snapshottable.go:230
        should check snapshot fields, check restore correctly works after modifying source data, check deletion
        /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/snapshottable.go:248
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic Snapshot (delete policy)] snapshottable[Feature:VolumeSnapshotDataSource] volume snapshot controller  should check snapshot fields, check restore correctly works after modifying source data, check deletion","total":19,"completed":4,"skipped":624,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy (Always)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed in first pod, new pod with same fsgroup applied to the volume contents"]}

SSSSS
------------------------------
[BeforeEach] [Testpattern: CSI Ephemeral-volume (default fs)] ephemeral
  /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:51
Oct 22 06:45:31.080: INFO: Driver "test.csi.azure.com" does not support volume type "CSIInlineVolume" - skipping
... skipping 37 lines ...
Oct 22 06:44:30.057: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comztpm2] to have phase Bound
Oct 22 06:44:30.165: INFO: PersistentVolumeClaim test.csi.azure.comztpm2 found but phase is Pending instead of Bound.
Oct 22 06:44:32.274: INFO: PersistentVolumeClaim test.csi.azure.comztpm2 found but phase is Pending instead of Bound.
Oct 22 06:44:34.381: INFO: PersistentVolumeClaim test.csi.azure.comztpm2 found and phase=Bound (4.323992075s)
STEP: Creating pod pod-subpath-test-dynamicpv-s9ls
STEP: Creating a pod to test subpath
Oct 22 06:44:34.706: INFO: Waiting up to 5m0s for pod "pod-subpath-test-dynamicpv-s9ls" in namespace "provisioning-2162" to be "Succeeded or Failed"
Oct 22 06:44:34.814: INFO: Pod "pod-subpath-test-dynamicpv-s9ls": Phase="Pending", Reason="", readiness=false. Elapsed: 107.694348ms
Oct 22 06:44:36.922: INFO: Pod "pod-subpath-test-dynamicpv-s9ls": Phase="Pending", Reason="", readiness=false. Elapsed: 2.216022906s
Oct 22 06:44:39.044: INFO: Pod "pod-subpath-test-dynamicpv-s9ls": Phase="Pending", Reason="", readiness=false. Elapsed: 4.337796603s
Oct 22 06:44:41.151: INFO: Pod "pod-subpath-test-dynamicpv-s9ls": Phase="Pending", Reason="", readiness=false. Elapsed: 6.445014699s
Oct 22 06:44:43.260: INFO: Pod "pod-subpath-test-dynamicpv-s9ls": Phase="Pending", Reason="", readiness=false. Elapsed: 8.553601002s
Oct 22 06:44:45.367: INFO: Pod "pod-subpath-test-dynamicpv-s9ls": Phase="Pending", Reason="", readiness=false. Elapsed: 10.661342029s
Oct 22 06:44:47.475: INFO: Pod "pod-subpath-test-dynamicpv-s9ls": Phase="Pending", Reason="", readiness=false. Elapsed: 12.76868129s
Oct 22 06:44:49.583: INFO: Pod "pod-subpath-test-dynamicpv-s9ls": Phase="Pending", Reason="", readiness=false. Elapsed: 14.87687613s
Oct 22 06:44:51.690: INFO: Pod "pod-subpath-test-dynamicpv-s9ls": Phase="Pending", Reason="", readiness=false. Elapsed: 16.984362745s
Oct 22 06:44:53.799: INFO: Pod "pod-subpath-test-dynamicpv-s9ls": Phase="Pending", Reason="", readiness=false. Elapsed: 19.092799225s
Oct 22 06:44:55.908: INFO: Pod "pod-subpath-test-dynamicpv-s9ls": Phase="Pending", Reason="", readiness=false. Elapsed: 21.20172264s
Oct 22 06:44:58.016: INFO: Pod "pod-subpath-test-dynamicpv-s9ls": Phase="Succeeded", Reason="", readiness=false. Elapsed: 23.310008104s
STEP: Saw pod success
Oct 22 06:44:58.016: INFO: Pod "pod-subpath-test-dynamicpv-s9ls" satisfied condition "Succeeded or Failed"
Oct 22 06:44:58.123: INFO: Trying to get logs from node k8s-agentpool1-22475134-vmss000001 pod pod-subpath-test-dynamicpv-s9ls container test-container-subpath-dynamicpv-s9ls: <nil>
STEP: delete the pod
Oct 22 06:44:58.391: INFO: Waiting for pod pod-subpath-test-dynamicpv-s9ls to disappear
Oct 22 06:44:58.502: INFO: Pod pod-subpath-test-dynamicpv-s9ls no longer exists
STEP: Deleting pod pod-subpath-test-dynamicpv-s9ls
Oct 22 06:44:58.502: INFO: Deleting pod "pod-subpath-test-dynamicpv-s9ls" in namespace "provisioning-2162"
STEP: Creating pod pod-subpath-test-dynamicpv-s9ls
STEP: Creating a pod to test subpath
Oct 22 06:44:58.718: INFO: Waiting up to 5m0s for pod "pod-subpath-test-dynamicpv-s9ls" in namespace "provisioning-2162" to be "Succeeded or Failed"
Oct 22 06:44:58.825: INFO: Pod "pod-subpath-test-dynamicpv-s9ls": Phase="Pending", Reason="", readiness=false. Elapsed: 106.900647ms
Oct 22 06:45:00.934: INFO: Pod "pod-subpath-test-dynamicpv-s9ls": Phase="Pending", Reason="", readiness=false. Elapsed: 2.21609082s
Oct 22 06:45:03.042: INFO: Pod "pod-subpath-test-dynamicpv-s9ls": Phase="Succeeded", Reason="", readiness=false. Elapsed: 4.324317827s
STEP: Saw pod success
Oct 22 06:45:03.042: INFO: Pod "pod-subpath-test-dynamicpv-s9ls" satisfied condition "Succeeded or Failed"
Oct 22 06:45:03.149: INFO: Trying to get logs from node k8s-agentpool1-22475134-vmss000001 pod pod-subpath-test-dynamicpv-s9ls container test-container-subpath-dynamicpv-s9ls: <nil>
STEP: delete the pod
Oct 22 06:45:03.375: INFO: Waiting for pod pod-subpath-test-dynamicpv-s9ls to disappear
Oct 22 06:45:03.483: INFO: Pod pod-subpath-test-dynamicpv-s9ls no longer exists
STEP: Deleting pod pod-subpath-test-dynamicpv-s9ls
Oct 22 06:45:03.483: INFO: Deleting pod "pod-subpath-test-dynamicpv-s9ls" in namespace "provisioning-2162"
... skipping 29 lines ...
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (default fs)] subPath
  /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50
    should support existing directories when readOnly specified in the volumeSource
    /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/subpath.go:395
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should support existing directories when readOnly specified in the volumeSource","total":18,"completed":5,"skipped":596,"failed":0}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Pre-provisioned PV (ext4)] volumes
  /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:51
Oct 22 06:46:16.019: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping
... skipping 37 lines ...
Oct 22 06:45:18.204: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comt4kcw] to have phase Bound
Oct 22 06:45:18.315: INFO: PersistentVolumeClaim test.csi.azure.comt4kcw found but phase is Pending instead of Bound.
Oct 22 06:45:20.424: INFO: PersistentVolumeClaim test.csi.azure.comt4kcw found but phase is Pending instead of Bound.
Oct 22 06:45:22.534: INFO: PersistentVolumeClaim test.csi.azure.comt4kcw found and phase=Bound (4.329199241s)
STEP: Creating pod exec-volume-test-dynamicpv-zxs8
STEP: Creating a pod to test exec-volume-test
Oct 22 06:45:22.876: INFO: Waiting up to 5m0s for pod "exec-volume-test-dynamicpv-zxs8" in namespace "volume-9223" to be "Succeeded or Failed"
Oct 22 06:45:22.985: INFO: Pod "exec-volume-test-dynamicpv-zxs8": Phase="Pending", Reason="", readiness=false. Elapsed: 108.673071ms
Oct 22 06:45:25.095: INFO: Pod "exec-volume-test-dynamicpv-zxs8": Phase="Pending", Reason="", readiness=false. Elapsed: 2.21860029s
Oct 22 06:45:27.205: INFO: Pod "exec-volume-test-dynamicpv-zxs8": Phase="Pending", Reason="", readiness=false. Elapsed: 4.32925394s
Oct 22 06:45:29.319: INFO: Pod "exec-volume-test-dynamicpv-zxs8": Phase="Pending", Reason="", readiness=false. Elapsed: 6.443068817s
Oct 22 06:45:31.428: INFO: Pod "exec-volume-test-dynamicpv-zxs8": Phase="Pending", Reason="", readiness=false. Elapsed: 8.552136134s
Oct 22 06:45:33.538: INFO: Pod "exec-volume-test-dynamicpv-zxs8": Phase="Pending", Reason="", readiness=false. Elapsed: 10.662190455s
... skipping 8 lines ...
Oct 22 06:45:52.528: INFO: Pod "exec-volume-test-dynamicpv-zxs8": Phase="Pending", Reason="", readiness=false. Elapsed: 29.651268884s
Oct 22 06:45:54.637: INFO: Pod "exec-volume-test-dynamicpv-zxs8": Phase="Pending", Reason="", readiness=false. Elapsed: 31.760937568s
Oct 22 06:45:56.747: INFO: Pod "exec-volume-test-dynamicpv-zxs8": Phase="Pending", Reason="", readiness=false. Elapsed: 33.87124469s
Oct 22 06:45:58.856: INFO: Pod "exec-volume-test-dynamicpv-zxs8": Phase="Pending", Reason="", readiness=false. Elapsed: 35.980232817s
Oct 22 06:46:00.967: INFO: Pod "exec-volume-test-dynamicpv-zxs8": Phase="Succeeded", Reason="", readiness=false. Elapsed: 38.090503981s
STEP: Saw pod success
Oct 22 06:46:00.967: INFO: Pod "exec-volume-test-dynamicpv-zxs8" satisfied condition "Succeeded or Failed"
Oct 22 06:46:01.076: INFO: Trying to get logs from node k8s-agentpool1-22475134-vmss000001 pod exec-volume-test-dynamicpv-zxs8 container exec-container-dynamicpv-zxs8: <nil>
STEP: delete the pod
Oct 22 06:46:01.310: INFO: Waiting for pod exec-volume-test-dynamicpv-zxs8 to disappear
Oct 22 06:46:01.419: INFO: Pod exec-volume-test-dynamicpv-zxs8 no longer exists
STEP: Deleting pod exec-volume-test-dynamicpv-zxs8
Oct 22 06:46:01.419: INFO: Deleting pod "exec-volume-test-dynamicpv-zxs8" in namespace "volume-9223"
... skipping 21 lines ...
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (default fs)] volumes
  /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50
    should allow exec of files on the volume
    /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/volumes.go:196
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] volumes should allow exec of files on the volume","total":18,"completed":2,"skipped":191,"failed":2,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy (OnRootMismatch)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed in first pod, new pod with same fsgroup skips ownership changes to the volume contents","External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] volumeMode should not mount / map unused volumes in a pod [LinuxOnly]"]}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath 
  should support existing single file [LinuxOnly]
  /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/subpath.go:219
... skipping 16 lines ...
Oct 22 06:45:31.906: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comdvtj7] to have phase Bound
Oct 22 06:45:32.014: INFO: PersistentVolumeClaim test.csi.azure.comdvtj7 found but phase is Pending instead of Bound.
Oct 22 06:45:34.124: INFO: PersistentVolumeClaim test.csi.azure.comdvtj7 found but phase is Pending instead of Bound.
Oct 22 06:45:36.236: INFO: PersistentVolumeClaim test.csi.azure.comdvtj7 found and phase=Bound (4.330251859s)
STEP: Creating pod pod-subpath-test-dynamicpv-dfp7
STEP: Creating a pod to test subpath
Oct 22 06:45:36.570: INFO: Waiting up to 5m0s for pod "pod-subpath-test-dynamicpv-dfp7" in namespace "provisioning-7956" to be "Succeeded or Failed"
Oct 22 06:45:36.679: INFO: Pod "pod-subpath-test-dynamicpv-dfp7": Phase="Pending", Reason="", readiness=false. Elapsed: 108.962401ms
Oct 22 06:45:38.788: INFO: Pod "pod-subpath-test-dynamicpv-dfp7": Phase="Pending", Reason="", readiness=false. Elapsed: 2.218456779s
Oct 22 06:45:40.898: INFO: Pod "pod-subpath-test-dynamicpv-dfp7": Phase="Pending", Reason="", readiness=false. Elapsed: 4.328047033s
Oct 22 06:45:43.007: INFO: Pod "pod-subpath-test-dynamicpv-dfp7": Phase="Pending", Reason="", readiness=false. Elapsed: 6.437803152s
Oct 22 06:45:45.117: INFO: Pod "pod-subpath-test-dynamicpv-dfp7": Phase="Pending", Reason="", readiness=false. Elapsed: 8.547503023s
Oct 22 06:45:47.231: INFO: Pod "pod-subpath-test-dynamicpv-dfp7": Phase="Pending", Reason="", readiness=false. Elapsed: 10.661608111s
Oct 22 06:45:49.345: INFO: Pod "pod-subpath-test-dynamicpv-dfp7": Phase="Pending", Reason="", readiness=false. Elapsed: 12.775722705s
Oct 22 06:45:51.472: INFO: Pod "pod-subpath-test-dynamicpv-dfp7": Phase="Pending", Reason="", readiness=false. Elapsed: 14.902547345s
Oct 22 06:45:53.581: INFO: Pod "pod-subpath-test-dynamicpv-dfp7": Phase="Pending", Reason="", readiness=false. Elapsed: 17.011486342s
Oct 22 06:45:55.690: INFO: Pod "pod-subpath-test-dynamicpv-dfp7": Phase="Pending", Reason="", readiness=false. Elapsed: 19.120456394s
Oct 22 06:45:57.800: INFO: Pod "pod-subpath-test-dynamicpv-dfp7": Phase="Succeeded", Reason="", readiness=false. Elapsed: 21.230353457s
STEP: Saw pod success
Oct 22 06:45:57.800: INFO: Pod "pod-subpath-test-dynamicpv-dfp7" satisfied condition "Succeeded or Failed"
Oct 22 06:45:57.909: INFO: Trying to get logs from node k8s-agentpool1-22475134-vmss000000 pod pod-subpath-test-dynamicpv-dfp7 container test-container-subpath-dynamicpv-dfp7: <nil>
STEP: delete the pod
Oct 22 06:45:58.141: INFO: Waiting for pod pod-subpath-test-dynamicpv-dfp7 to disappear
Oct 22 06:45:58.250: INFO: Pod pod-subpath-test-dynamicpv-dfp7 no longer exists
STEP: Deleting pod pod-subpath-test-dynamicpv-dfp7
Oct 22 06:45:58.250: INFO: Deleting pod "pod-subpath-test-dynamicpv-dfp7" in namespace "provisioning-7956"
... skipping 29 lines ...
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (default fs)] subPath
  /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50
    should support existing single file [LinuxOnly]
    /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/subpath.go:219
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should support existing single file [LinuxOnly]","total":19,"completed":5,"skipped":687,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy (Always)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed in first pod, new pod with same fsgroup applied to the volume contents"]}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy 
  (OnRootMismatch)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed in first pod, new pod with different fsgroup applied to the volume contents
  /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/fsgroupchangepolicy.go:208
... skipping 90 lines ...
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy
  /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50
    (OnRootMismatch)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed in first pod, new pod with different fsgroup applied to the volume contents
    /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/fsgroupchangepolicy.go:208
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy (OnRootMismatch)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed in first pod, new pod with different fsgroup applied to the volume contents","total":17,"completed":5,"skipped":717,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (filesystem volmode)] volumeMode should not mount / map unused volumes in a pod [LinuxOnly]"]}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Pre-provisioned PV (default fs)] subPath
  /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:51
Oct 22 06:47:59.175: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping
... skipping 68 lines ...
Oct 22 06:46:43.927: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.com6xqxc] to have phase Bound
Oct 22 06:46:44.035: INFO: PersistentVolumeClaim test.csi.azure.com6xqxc found but phase is Pending instead of Bound.
Oct 22 06:46:46.143: INFO: PersistentVolumeClaim test.csi.azure.com6xqxc found but phase is Pending instead of Bound.
Oct 22 06:46:48.253: INFO: PersistentVolumeClaim test.csi.azure.com6xqxc found and phase=Bound (4.326350778s)
STEP: Creating pod pod-subpath-test-dynamicpv-xrtq
STEP: Creating a pod to test subpath
Oct 22 06:46:48.584: INFO: Waiting up to 5m0s for pod "pod-subpath-test-dynamicpv-xrtq" in namespace "provisioning-6097" to be "Succeeded or Failed"
Oct 22 06:46:48.704: INFO: Pod "pod-subpath-test-dynamicpv-xrtq": Phase="Pending", Reason="", readiness=false. Elapsed: 119.264062ms
Oct 22 06:46:50.814: INFO: Pod "pod-subpath-test-dynamicpv-xrtq": Phase="Pending", Reason="", readiness=false. Elapsed: 2.229846448s
Oct 22 06:46:52.924: INFO: Pod "pod-subpath-test-dynamicpv-xrtq": Phase="Pending", Reason="", readiness=false. Elapsed: 4.339428007s
Oct 22 06:46:55.033: INFO: Pod "pod-subpath-test-dynamicpv-xrtq": Phase="Pending", Reason="", readiness=false. Elapsed: 6.44882132s
Oct 22 06:46:57.144: INFO: Pod "pod-subpath-test-dynamicpv-xrtq": Phase="Pending", Reason="", readiness=false. Elapsed: 8.559951204s
Oct 22 06:46:59.254: INFO: Pod "pod-subpath-test-dynamicpv-xrtq": Phase="Pending", Reason="", readiness=false. Elapsed: 10.669493779s
... skipping 8 lines ...
Oct 22 06:47:18.242: INFO: Pod "pod-subpath-test-dynamicpv-xrtq": Phase="Pending", Reason="", readiness=false. Elapsed: 29.657669367s
Oct 22 06:47:20.352: INFO: Pod "pod-subpath-test-dynamicpv-xrtq": Phase="Pending", Reason="", readiness=false. Elapsed: 31.768108063s
Oct 22 06:47:22.462: INFO: Pod "pod-subpath-test-dynamicpv-xrtq": Phase="Pending", Reason="", readiness=false. Elapsed: 33.877480734s
Oct 22 06:47:24.571: INFO: Pod "pod-subpath-test-dynamicpv-xrtq": Phase="Pending", Reason="", readiness=false. Elapsed: 35.987150719s
Oct 22 06:47:26.681: INFO: Pod "pod-subpath-test-dynamicpv-xrtq": Phase="Succeeded", Reason="", readiness=false. Elapsed: 38.097065803s
STEP: Saw pod success
Oct 22 06:47:26.681: INFO: Pod "pod-subpath-test-dynamicpv-xrtq" satisfied condition "Succeeded or Failed"
Oct 22 06:47:26.789: INFO: Trying to get logs from node k8s-agentpool1-22475134-vmss000000 pod pod-subpath-test-dynamicpv-xrtq container test-container-subpath-dynamicpv-xrtq: <nil>
STEP: delete the pod
Oct 22 06:47:27.032: INFO: Waiting for pod pod-subpath-test-dynamicpv-xrtq to disappear
Oct 22 06:47:27.140: INFO: Pod pod-subpath-test-dynamicpv-xrtq no longer exists
STEP: Deleting pod pod-subpath-test-dynamicpv-xrtq
Oct 22 06:47:27.140: INFO: Deleting pod "pod-subpath-test-dynamicpv-xrtq" in namespace "provisioning-6097"
... skipping 23 lines ...
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (default fs)] subPath
  /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50
    should support readOnly directory specified in the volumeMount
    /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/subpath.go:365
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should support readOnly directory specified in the volumeMount","total":18,"completed":3,"skipped":268,"failed":2,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy (OnRootMismatch)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed in first pod, new pod with same fsgroup skips ownership changes to the volume contents","External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] volumeMode should not mount / map unused volumes in a pod [LinuxOnly]"]}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Inline-volume (ntfs)][Feature:Windows] volumes
  /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:51
Oct 22 06:48:09.012: INFO: Driver "test.csi.azure.com" does not support volume type "InlineVolume" - skipping
... skipping 36 lines ...
Oct 22 06:43:33.424: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil
Oct 22 06:43:33.535: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.combkzh9] to have phase Bound
Oct 22 06:43:33.643: INFO: PersistentVolumeClaim test.csi.azure.combkzh9 found but phase is Pending instead of Bound.
Oct 22 06:43:35.754: INFO: PersistentVolumeClaim test.csi.azure.combkzh9 found but phase is Pending instead of Bound.
Oct 22 06:43:37.862: INFO: PersistentVolumeClaim test.csi.azure.combkzh9 found and phase=Bound (4.327037809s)
STEP: [init] starting a pod to use the claim
Oct 22 06:43:38.189: INFO: Waiting up to 15m0s for pod "pvc-snapshottable-tester-d7ddv" in namespace "snapshotting-5267" to be "Succeeded or Failed"
Oct 22 06:43:38.299: INFO: Pod "pvc-snapshottable-tester-d7ddv": Phase="Pending", Reason="", readiness=false. Elapsed: 109.030264ms
Oct 22 06:43:40.409: INFO: Pod "pvc-snapshottable-tester-d7ddv": Phase="Pending", Reason="", readiness=false. Elapsed: 2.219892569s
Oct 22 06:43:42.528: INFO: Pod "pvc-snapshottable-tester-d7ddv": Phase="Pending", Reason="", readiness=false. Elapsed: 4.338582769s
Oct 22 06:43:44.637: INFO: Pod "pvc-snapshottable-tester-d7ddv": Phase="Pending", Reason="", readiness=false. Elapsed: 6.4473422s
Oct 22 06:43:46.747: INFO: Pod "pvc-snapshottable-tester-d7ddv": Phase="Pending", Reason="", readiness=false. Elapsed: 8.557227532s
Oct 22 06:43:48.856: INFO: Pod "pvc-snapshottable-tester-d7ddv": Phase="Pending", Reason="", readiness=false. Elapsed: 10.666893654s
... skipping 7 lines ...
Oct 22 06:44:05.731: INFO: Pod "pvc-snapshottable-tester-d7ddv": Phase="Pending", Reason="", readiness=false. Elapsed: 27.541957581s
Oct 22 06:44:07.842: INFO: Pod "pvc-snapshottable-tester-d7ddv": Phase="Pending", Reason="", readiness=false. Elapsed: 29.652739535s
Oct 22 06:44:09.952: INFO: Pod "pvc-snapshottable-tester-d7ddv": Phase="Pending", Reason="", readiness=false. Elapsed: 31.762785256s
Oct 22 06:44:12.061: INFO: Pod "pvc-snapshottable-tester-d7ddv": Phase="Pending", Reason="", readiness=false. Elapsed: 33.871855833s
Oct 22 06:44:14.171: INFO: Pod "pvc-snapshottable-tester-d7ddv": Phase="Succeeded", Reason="", readiness=false. Elapsed: 35.981467541s
STEP: Saw pod success
Oct 22 06:44:14.171: INFO: Pod "pvc-snapshottable-tester-d7ddv" satisfied condition "Succeeded or Failed"
Oct 22 06:44:14.394: INFO: Pod pvc-snapshottable-tester-d7ddv has the following logs: 
Oct 22 06:44:14.394: INFO: Deleting pod "pvc-snapshottable-tester-d7ddv" in namespace "snapshotting-5267"
Oct 22 06:44:14.507: INFO: Wait up to 5m0s for pod "pvc-snapshottable-tester-d7ddv" to be fully deleted
Oct 22 06:44:14.616: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.combkzh9] to have phase Bound
Oct 22 06:44:14.724: INFO: PersistentVolumeClaim test.csi.azure.combkzh9 found and phase=Bound (108.770409ms)
STEP: [init] checking the claim
... skipping 18 lines ...
[It] should check snapshot fields, check restore correctly works after modifying source data, check deletion
  /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/snapshottable.go:248
STEP: checking the snapshot
STEP: checking the SnapshotContent
STEP: Modifying source data test
STEP: modifying the data in the source PVC
Oct 22 06:44:26.607: INFO: Waiting up to 15m0s for pod "pvc-snapshottable-data-tester-d5zbz" in namespace "snapshotting-5267" to be "Succeeded or Failed"
Oct 22 06:44:26.716: INFO: Pod "pvc-snapshottable-data-tester-d5zbz": Phase="Pending", Reason="", readiness=false. Elapsed: 108.278904ms
Oct 22 06:44:28.825: INFO: Pod "pvc-snapshottable-data-tester-d5zbz": Phase="Pending", Reason="", readiness=false. Elapsed: 2.21786792s
Oct 22 06:44:30.941: INFO: Pod "pvc-snapshottable-data-tester-d5zbz": Phase="Pending", Reason="", readiness=false. Elapsed: 4.333415849s
Oct 22 06:44:33.050: INFO: Pod "pvc-snapshottable-data-tester-d5zbz": Phase="Pending", Reason="", readiness=false. Elapsed: 6.442957704s
Oct 22 06:44:35.159: INFO: Pod "pvc-snapshottable-data-tester-d5zbz": Phase="Pending", Reason="", readiness=false. Elapsed: 8.551848261s
Oct 22 06:44:37.269: INFO: Pod "pvc-snapshottable-data-tester-d5zbz": Phase="Pending", Reason="", readiness=false. Elapsed: 10.66135184s
... skipping 56 lines ...
Oct 22 06:46:37.550: INFO: Pod "pvc-snapshottable-data-tester-d5zbz": Phase="Pending", Reason="", readiness=false. Elapsed: 2m10.942373332s
Oct 22 06:46:39.660: INFO: Pod "pvc-snapshottable-data-tester-d5zbz": Phase="Pending", Reason="", readiness=false. Elapsed: 2m13.052105374s
Oct 22 06:46:41.808: INFO: Pod "pvc-snapshottable-data-tester-d5zbz": Phase="Pending", Reason="", readiness=false. Elapsed: 2m15.200387963s
Oct 22 06:46:43.917: INFO: Pod "pvc-snapshottable-data-tester-d5zbz": Phase="Pending", Reason="", readiness=false. Elapsed: 2m17.309704697s
Oct 22 06:46:46.027: INFO: Pod "pvc-snapshottable-data-tester-d5zbz": Phase="Succeeded", Reason="", readiness=false. Elapsed: 2m19.419381521s
STEP: Saw pod success
Oct 22 06:46:46.027: INFO: Pod "pvc-snapshottable-data-tester-d5zbz" satisfied condition "Succeeded or Failed"
Oct 22 06:46:46.256: INFO: Pod pvc-snapshottable-data-tester-d5zbz has the following logs: 
Oct 22 06:46:46.256: INFO: Deleting pod "pvc-snapshottable-data-tester-d5zbz" in namespace "snapshotting-5267"
Oct 22 06:46:46.380: INFO: Wait up to 5m0s for pod "pvc-snapshottable-data-tester-d5zbz" to be fully deleted
STEP: creating a pvc from the snapshot
STEP: starting a pod to use the claim
Oct 22 06:47:10.930: INFO: Running '/usr/local/bin/kubectl --server=https://kubetest-5p3qtork.westeurope.cloudapp.azure.com --kubeconfig=/root/tmp1541861128/kubeconfig/kubeconfig.westeurope.json --namespace=snapshotting-5267 exec restored-pvc-tester-nplsb --namespace=snapshotting-5267 -- cat /mnt/test/data'
... skipping 31 lines ...
Oct 22 06:47:37.169: INFO: volumesnapshotcontents snapcontent-c32c3e47-e8ba-4aef-85a9-b15a6a2e0d5c has been found and is not deleted
Oct 22 06:47:38.279: INFO: volumesnapshotcontents snapcontent-c32c3e47-e8ba-4aef-85a9-b15a6a2e0d5c has been found and is not deleted
Oct 22 06:47:39.389: INFO: volumesnapshotcontents snapcontent-c32c3e47-e8ba-4aef-85a9-b15a6a2e0d5c has been found and is not deleted
Oct 22 06:47:40.498: INFO: volumesnapshotcontents snapcontent-c32c3e47-e8ba-4aef-85a9-b15a6a2e0d5c has been found and is not deleted
Oct 22 06:47:41.609: INFO: volumesnapshotcontents snapcontent-c32c3e47-e8ba-4aef-85a9-b15a6a2e0d5c has been found and is not deleted
Oct 22 06:47:42.718: INFO: volumesnapshotcontents snapcontent-c32c3e47-e8ba-4aef-85a9-b15a6a2e0d5c has been found and is not deleted
Oct 22 06:47:43.719: INFO: WaitUntil failed after reaching the timeout 30s
[AfterEach] volume snapshot controller
  /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/snapshottable.go:226
Oct 22 06:47:43.833: INFO: Pod restored-pvc-tester-nplsb has the following logs: 
Oct 22 06:47:43.833: INFO: Deleting pod "restored-pvc-tester-nplsb" in namespace "snapshotting-5267"
Oct 22 06:47:43.944: INFO: Wait up to 5m0s for pod "restored-pvc-tester-nplsb" to be fully deleted
Oct 22 06:48:16.162: INFO: deleting claim "snapshotting-5267"/"pvc-h8fw7"
... skipping 31 lines ...
    /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/snapshottable.go:108
      
      /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/snapshottable.go:230
        should check snapshot fields, check restore correctly works after modifying source data, check deletion
        /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/snapshottable.go:248
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic Snapshot (retain policy)] snapshottable[Feature:VolumeSnapshotDataSource] volume snapshot controller  should check snapshot fields, check restore correctly works after modifying source data, check deletion","total":12,"completed":3,"skipped":699,"failed":0}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Pre-provisioned PV (default fs)] volumes
  /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:51
Oct 22 06:48:34.427: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping
... skipping 101 lines ...
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (default fs)] volumes
  /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50
    should store data
    /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/volumes.go:159
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] volumes should store data","total":18,"completed":6,"skipped":768,"failed":0}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath 
  should support existing directory
  /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/subpath.go:205
... skipping 16 lines ...
Oct 22 06:50:03.091: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comnn5vk] to have phase Bound
Oct 22 06:50:03.223: INFO: PersistentVolumeClaim test.csi.azure.comnn5vk found but phase is Pending instead of Bound.
Oct 22 06:50:05.331: INFO: PersistentVolumeClaim test.csi.azure.comnn5vk found but phase is Pending instead of Bound.
Oct 22 06:50:07.440: INFO: PersistentVolumeClaim test.csi.azure.comnn5vk found and phase=Bound (4.349261025s)
STEP: Creating pod pod-subpath-test-dynamicpv-llw4
STEP: Creating a pod to test subpath
Oct 22 06:50:07.764: INFO: Waiting up to 5m0s for pod "pod-subpath-test-dynamicpv-llw4" in namespace "provisioning-9989" to be "Succeeded or Failed"
Oct 22 06:50:07.871: INFO: Pod "pod-subpath-test-dynamicpv-llw4": Phase="Pending", Reason="", readiness=false. Elapsed: 107.286835ms
Oct 22 06:50:09.981: INFO: Pod "pod-subpath-test-dynamicpv-llw4": Phase="Pending", Reason="", readiness=false. Elapsed: 2.216666577s
Oct 22 06:50:12.089: INFO: Pod "pod-subpath-test-dynamicpv-llw4": Phase="Pending", Reason="", readiness=false. Elapsed: 4.325016138s
Oct 22 06:50:14.197: INFO: Pod "pod-subpath-test-dynamicpv-llw4": Phase="Pending", Reason="", readiness=false. Elapsed: 6.433233343s
Oct 22 06:50:16.307: INFO: Pod "pod-subpath-test-dynamicpv-llw4": Phase="Pending", Reason="", readiness=false. Elapsed: 8.542876849s
Oct 22 06:50:18.414: INFO: Pod "pod-subpath-test-dynamicpv-llw4": Phase="Pending", Reason="", readiness=false. Elapsed: 10.65003242s
Oct 22 06:50:20.523: INFO: Pod "pod-subpath-test-dynamicpv-llw4": Phase="Pending", Reason="", readiness=false. Elapsed: 12.75863474s
Oct 22 06:50:22.632: INFO: Pod "pod-subpath-test-dynamicpv-llw4": Phase="Pending", Reason="", readiness=false. Elapsed: 14.868043711s
Oct 22 06:50:24.739: INFO: Pod "pod-subpath-test-dynamicpv-llw4": Phase="Pending", Reason="", readiness=false. Elapsed: 16.975558411s
Oct 22 06:50:26.847: INFO: Pod "pod-subpath-test-dynamicpv-llw4": Phase="Pending", Reason="", readiness=false. Elapsed: 19.08332196s
Oct 22 06:50:28.955: INFO: Pod "pod-subpath-test-dynamicpv-llw4": Phase="Pending", Reason="", readiness=false. Elapsed: 21.191613659s
Oct 22 06:50:31.066: INFO: Pod "pod-subpath-test-dynamicpv-llw4": Phase="Succeeded", Reason="", readiness=false. Elapsed: 23.302571851s
STEP: Saw pod success
Oct 22 06:50:31.067: INFO: Pod "pod-subpath-test-dynamicpv-llw4" satisfied condition "Succeeded or Failed"
Oct 22 06:50:31.173: INFO: Trying to get logs from node k8s-agentpool1-22475134-vmss000001 pod pod-subpath-test-dynamicpv-llw4 container test-container-volume-dynamicpv-llw4: <nil>
STEP: delete the pod
Oct 22 06:50:31.406: INFO: Waiting for pod pod-subpath-test-dynamicpv-llw4 to disappear
Oct 22 06:50:31.513: INFO: Pod pod-subpath-test-dynamicpv-llw4 no longer exists
STEP: Deleting pod pod-subpath-test-dynamicpv-llw4
Oct 22 06:50:31.514: INFO: Deleting pod "pod-subpath-test-dynamicpv-llw4" in namespace "provisioning-9989"
... skipping 23 lines ...
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (default fs)] subPath
  /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50
    should support existing directory
    /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/subpath.go:205
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should support existing directory","total":18,"completed":7,"skipped":798,"failed":0}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: CSI Ephemeral-volume (default fs)] ephemeral
  /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:51
Oct 22 06:51:13.422: INFO: Driver "test.csi.azure.com" does not support volume type "CSIInlineVolume" - skipping
... skipping 67 lines ...
Oct 22 06:48:09.714: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil
Oct 22 06:48:09.835: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comcc4nv] to have phase Bound
Oct 22 06:48:09.944: INFO: PersistentVolumeClaim test.csi.azure.comcc4nv found but phase is Pending instead of Bound.
Oct 22 06:48:12.054: INFO: PersistentVolumeClaim test.csi.azure.comcc4nv found but phase is Pending instead of Bound.
Oct 22 06:48:14.165: INFO: PersistentVolumeClaim test.csi.azure.comcc4nv found and phase=Bound (4.32949012s)
STEP: [init] starting a pod to use the claim
Oct 22 06:48:14.492: INFO: Waiting up to 15m0s for pod "pvc-snapshottable-tester-zp267" in namespace "snapshotting-6996" to be "Succeeded or Failed"
Oct 22 06:48:14.601: INFO: Pod "pvc-snapshottable-tester-zp267": Phase="Pending", Reason="", readiness=false. Elapsed: 108.7133ms
Oct 22 06:48:16.711: INFO: Pod "pvc-snapshottable-tester-zp267": Phase="Pending", Reason="", readiness=false. Elapsed: 2.219245845s
Oct 22 06:48:18.821: INFO: Pod "pvc-snapshottable-tester-zp267": Phase="Pending", Reason="", readiness=false. Elapsed: 4.329235778s
Oct 22 06:48:20.932: INFO: Pod "pvc-snapshottable-tester-zp267": Phase="Pending", Reason="", readiness=false. Elapsed: 6.439980819s
Oct 22 06:48:23.044: INFO: Pod "pvc-snapshottable-tester-zp267": Phase="Pending", Reason="", readiness=false. Elapsed: 8.551438662s
Oct 22 06:48:25.154: INFO: Pod "pvc-snapshottable-tester-zp267": Phase="Pending", Reason="", readiness=false. Elapsed: 10.662308385s
Oct 22 06:48:27.267: INFO: Pod "pvc-snapshottable-tester-zp267": Phase="Pending", Reason="", readiness=false. Elapsed: 12.774632924s
Oct 22 06:48:29.377: INFO: Pod "pvc-snapshottable-tester-zp267": Phase="Pending", Reason="", readiness=false. Elapsed: 14.884848194s
Oct 22 06:48:31.488: INFO: Pod "pvc-snapshottable-tester-zp267": Phase="Pending", Reason="", readiness=false. Elapsed: 16.996020959s
Oct 22 06:48:33.597: INFO: Pod "pvc-snapshottable-tester-zp267": Phase="Pending", Reason="", readiness=false. Elapsed: 19.105042653s
Oct 22 06:48:35.706: INFO: Pod "pvc-snapshottable-tester-zp267": Phase="Succeeded", Reason="", readiness=false. Elapsed: 21.213876686s
STEP: Saw pod success
Oct 22 06:48:35.706: INFO: Pod "pvc-snapshottable-tester-zp267" satisfied condition "Succeeded or Failed"
Oct 22 06:48:35.928: INFO: Pod pvc-snapshottable-tester-zp267 has the following logs: 
Oct 22 06:48:35.928: INFO: Deleting pod "pvc-snapshottable-tester-zp267" in namespace "snapshotting-6996"
Oct 22 06:48:36.043: INFO: Wait up to 5m0s for pod "pvc-snapshottable-tester-zp267" to be fully deleted
Oct 22 06:48:36.151: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comcc4nv] to have phase Bound
Oct 22 06:48:36.260: INFO: PersistentVolumeClaim test.csi.azure.comcc4nv found and phase=Bound (108.727344ms)
STEP: [init] checking the claim
... skipping 45 lines ...
[It] should check snapshot fields, check restore correctly works after modifying source data, check deletion
  /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/snapshottable.go:248
STEP: checking the snapshot
STEP: checking the SnapshotContent
STEP: Modifying source data test
STEP: modifying the data in the source PVC
Oct 22 06:49:09.396: INFO: Waiting up to 15m0s for pod "pvc-snapshottable-data-tester-kcj4n" in namespace "snapshotting-6996" to be "Succeeded or Failed"
Oct 22 06:49:09.505: INFO: Pod "pvc-snapshottable-data-tester-kcj4n": Phase="Pending", Reason="", readiness=false. Elapsed: 108.274177ms
Oct 22 06:49:11.614: INFO: Pod "pvc-snapshottable-data-tester-kcj4n": Phase="Pending", Reason="", readiness=false. Elapsed: 2.217652113s
Oct 22 06:49:13.725: INFO: Pod "pvc-snapshottable-data-tester-kcj4n": Phase="Pending", Reason="", readiness=false. Elapsed: 4.328322623s
Oct 22 06:49:15.834: INFO: Pod "pvc-snapshottable-data-tester-kcj4n": Phase="Pending", Reason="", readiness=false. Elapsed: 6.437902768s
Oct 22 06:49:17.943: INFO: Pod "pvc-snapshottable-data-tester-kcj4n": Phase="Pending", Reason="", readiness=false. Elapsed: 8.547065144s
Oct 22 06:49:20.054: INFO: Pod "pvc-snapshottable-data-tester-kcj4n": Phase="Pending", Reason="", readiness=false. Elapsed: 10.657168518s
... skipping 22 lines ...
Oct 22 06:50:08.594: INFO: Pod "pvc-snapshottable-data-tester-kcj4n": Phase="Pending", Reason="", readiness=false. Elapsed: 59.198086094s
Oct 22 06:50:10.704: INFO: Pod "pvc-snapshottable-data-tester-kcj4n": Phase="Pending", Reason="", readiness=false. Elapsed: 1m1.307527868s
Oct 22 06:50:12.813: INFO: Pod "pvc-snapshottable-data-tester-kcj4n": Phase="Pending", Reason="", readiness=false. Elapsed: 1m3.416268386s
Oct 22 06:50:14.922: INFO: Pod "pvc-snapshottable-data-tester-kcj4n": Phase="Pending", Reason="", readiness=false. Elapsed: 1m5.52606163s
Oct 22 06:50:17.037: INFO: Pod "pvc-snapshottable-data-tester-kcj4n": Phase="Succeeded", Reason="", readiness=false. Elapsed: 1m7.640996716s
STEP: Saw pod success
Oct 22 06:50:17.037: INFO: Pod "pvc-snapshottable-data-tester-kcj4n" satisfied condition "Succeeded or Failed"
Oct 22 06:50:17.302: INFO: Pod pvc-snapshottable-data-tester-kcj4n has the following logs: 
Oct 22 06:50:17.302: INFO: Deleting pod "pvc-snapshottable-data-tester-kcj4n" in namespace "snapshotting-6996"
Oct 22 06:50:17.418: INFO: Wait up to 5m0s for pod "pvc-snapshottable-data-tester-kcj4n" to be fully deleted
STEP: creating a pvc from the snapshot
STEP: starting a pod to use the claim
Oct 22 06:50:59.966: INFO: Running '/usr/local/bin/kubectl --server=https://kubetest-5p3qtork.westeurope.cloudapp.azure.com --kubeconfig=/root/tmp1541861128/kubeconfig/kubeconfig.westeurope.json --namespace=snapshotting-6996 exec restored-pvc-tester-9v9mn --namespace=snapshotting-6996 -- cat /mnt/test/data'
... skipping 31 lines ...
Oct 22 06:51:26.699: INFO: volumesnapshotcontents pre-provisioned-snapcontent-4cc2a8d5-96d7-4393-9e56-43acc9383db0 has been found and is not deleted
Oct 22 06:51:27.813: INFO: volumesnapshotcontents pre-provisioned-snapcontent-4cc2a8d5-96d7-4393-9e56-43acc9383db0 has been found and is not deleted
Oct 22 06:51:28.923: INFO: volumesnapshotcontents pre-provisioned-snapcontent-4cc2a8d5-96d7-4393-9e56-43acc9383db0 has been found and is not deleted
Oct 22 06:51:30.032: INFO: volumesnapshotcontents pre-provisioned-snapcontent-4cc2a8d5-96d7-4393-9e56-43acc9383db0 has been found and is not deleted
Oct 22 06:51:31.142: INFO: volumesnapshotcontents pre-provisioned-snapcontent-4cc2a8d5-96d7-4393-9e56-43acc9383db0 has been found and is not deleted
Oct 22 06:51:32.252: INFO: volumesnapshotcontents pre-provisioned-snapcontent-4cc2a8d5-96d7-4393-9e56-43acc9383db0 has been found and is not deleted
Oct 22 06:51:33.253: INFO: WaitUntil failed after reaching the timeout 30s
[AfterEach] volume snapshot controller
  /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/snapshottable.go:226
Oct 22 06:51:33.367: INFO: Pod restored-pvc-tester-9v9mn has the following logs: 
Oct 22 06:51:33.367: INFO: Deleting pod "restored-pvc-tester-9v9mn" in namespace "snapshotting-6996"
Oct 22 06:51:33.480: INFO: Wait up to 5m0s for pod "restored-pvc-tester-9v9mn" to be fully deleted
Oct 22 06:52:05.698: INFO: deleting claim "snapshotting-6996"/"pvc-r9b6r"
... skipping 31 lines ...
    /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/snapshottable.go:108
      
      /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/snapshottable.go:230
        should check snapshot fields, check restore correctly works after modifying source data, check deletion
        /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/snapshottable.go:248
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Pre-provisioned Snapshot (retain policy)] snapshottable[Feature:VolumeSnapshotDataSource] volume snapshot controller  should check snapshot fields, check restore correctly works after modifying source data, check deletion","total":18,"completed":4,"skipped":430,"failed":2,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy (OnRootMismatch)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed in first pod, new pod with same fsgroup skips ownership changes to the volume contents","External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] volumeMode should not mount / map unused volumes in a pod [LinuxOnly]"]}

SSS
------------------------------
[BeforeEach] [Testpattern: Inline-volume (default fs)] subPath
  /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:51
Oct 22 06:52:23.900: INFO: Driver "test.csi.azure.com" does not support volume type "InlineVolume" - skipping
... skipping 106 lines ...
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/external/external.go:174
  [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral
  /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50
    should create read/write inline ephemeral volume
    /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/ephemeral.go:166
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral should create read/write inline ephemeral volume","total":18,"completed":5,"skipped":659,"failed":2,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy (OnRootMismatch)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed in first pod, new pod with same fsgroup skips ownership changes to the volume contents","External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] volumeMode should not mount / map unused volumes in a pod [LinuxOnly]"]}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
External Storage [Driver: test.csi.azure.com] [Testpattern: Generic Ephemeral-volume (default fs) (immediate-binding)] ephemeral 
  should create read-only inline ephemeral volume
  /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/ephemeral.go:149
... skipping 43 lines ...
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/external/external.go:174
  [Testpattern: Generic Ephemeral-volume (default fs) (immediate-binding)] ephemeral
  /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50
    should create read-only inline ephemeral volume
    /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/ephemeral.go:149
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Generic Ephemeral-volume (default fs) (immediate-binding)] ephemeral should create read-only inline ephemeral volume","total":18,"completed":6,"skipped":773,"failed":2,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy (OnRootMismatch)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed in first pod, new pod with same fsgroup skips ownership changes to the volume contents","External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] volumeMode should not mount / map unused volumes in a pod [LinuxOnly]"]}

SSSSSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Inline-volume (default fs)] subPath
  /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:51
Oct 22 06:55:36.355: INFO: Driver "test.csi.azure.com" does not support volume type "InlineVolume" - skipping
... skipping 140 lines ...
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (default fs)] provisioning
  /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50
    should provision storage with snapshot data source [Feature:VolumeSnapshotDataSource]
    /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/provisioning.go:201
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] provisioning should provision storage with snapshot data source [Feature:VolumeSnapshotDataSource]","total":18,"completed":7,"skipped":815,"failed":2,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy (OnRootMismatch)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed in first pod, new pod with same fsgroup skips ownership changes to the volume contents","External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] volumeMode should not mount / map unused volumes in a pod [LinuxOnly]"]}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Dynamic PV (default fs)] capacity
  /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:51
Oct 22 06:57:50.474: INFO: Driver test.csi.azure.com doesn't publish storage capacity -- skipping
... skipping 43 lines ...
Oct 22 06:48:35.459: INFO: Deleting pod "inline-volume-fzscn" in namespace "ephemeral-1169"
Oct 22 06:48:35.569: INFO: Wait up to 5m0s for pod "inline-volume-fzscn" to be fully deleted
Oct 22 06:48:35.676: INFO: Creating resource for dynamic PV
Oct 22 06:48:35.677: INFO: Using claimSize:5Gi, test suite supported size:{ }, driver(test.csi.azure.com) supported size:{ } 
STEP: creating a StorageClass ephemeral-1169-e2e-sc2h9pc
STEP: checking the requested inline volume exists in the pod running on node {Name: Selector:map[] Affinity:nil}
Oct 22 07:06:16.671: FAIL: waiting for second pod with inline volume
Unexpected error:
    <*errors.errorString | 0xc00024c250>: {
        s: "timed out waiting for the condition",
    }
    timed out waiting for the condition
occurred

... skipping 43 lines ...
Oct 22 07:08:07.770: INFO: At 2021-10-22 06:48:35 +0000 UTC - event for inline-volume-tester-qkkht: {default-scheduler } FailedScheduling: 0/3 nodes are available: 3 persistentvolumeclaim "inline-volume-tester-qkkht-my-volume-0" not found.
Oct 22 07:08:07.770: INFO: At 2021-10-22 06:48:35 +0000 UTC - event for inline-volume-tester-qkkht-my-volume-0: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "test.csi.azure.com" or manually created by system administrator
Oct 22 07:08:07.770: INFO: At 2021-10-22 06:48:35 +0000 UTC - event for inline-volume-tester-qkkht-my-volume-0: {test.csi.azure.com_k8s-agentpool1-22475134-vmss000001_643e5b82-d98f-4701-82bb-2ae23a5c6604 } Provisioning: External provisioner is provisioning volume for claim "ephemeral-1169/inline-volume-tester-qkkht-my-volume-0"
Oct 22 07:08:07.770: INFO: At 2021-10-22 06:48:36 +0000 UTC - event for inline-volume-tester-qkkht: {default-scheduler } FailedScheduling: 0/3 nodes are available: 3 pod has unbound immediate PersistentVolumeClaims.
Oct 22 07:08:07.770: INFO: At 2021-10-22 06:48:38 +0000 UTC - event for inline-volume-tester-qkkht: {default-scheduler } Scheduled: Successfully assigned ephemeral-1169/inline-volume-tester-qkkht to k8s-agentpool1-22475134-vmss000000
Oct 22 07:08:07.770: INFO: At 2021-10-22 06:48:38 +0000 UTC - event for inline-volume-tester-qkkht-my-volume-0: {test.csi.azure.com_k8s-agentpool1-22475134-vmss000001_643e5b82-d98f-4701-82bb-2ae23a5c6604 } ProvisioningSucceeded: Successfully provisioned volume pvc-e6c2dd85-04e5-44ed-9840-b9b2e2da6777
Oct 22 07:08:07.770: INFO: At 2021-10-22 06:48:50 +0000 UTC - event for inline-volume-tester-qkkht: {attachdetach-controller } FailedAttachVolume: AttachVolume.Attach failed for volume "pvc-e6c2dd85-04e5-44ed-9840-b9b2e2da6777" : rpc error: code = Unknown desc = Attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-e6c2dd85-04e5-44ed-9840-b9b2e2da6777" to instance "k8s-agentpool1-22475134-vmss000000" failed with Retriable: false, RetryAfter: 0s, HTTPStatusCode: 409, RawError: {

  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}
Oct 22 07:08:07.770: INFO: At 2021-10-22 06:50:41 +0000 UTC - event for inline-volume-tester-qkkht: {kubelet k8s-agentpool1-22475134-vmss000000} FailedMount: Unable to attach or mount volumes: unmounted volumes=[my-volume-0], unattached volumes=[my-volume-0 kube-api-access-hkmsk]: timed out waiting for the condition
... skipping 5 lines ...
Oct 22 07:08:07.770: INFO: At 2021-10-22 06:51:16 +0000 UTC - event for inline-volume-tester2-9rcj7: {default-scheduler } FailedScheduling: 0/3 nodes are available: 3 persistentvolumeclaim "inline-volume-tester2-9rcj7-my-volume-0" not found.
Oct 22 07:08:07.770: INFO: At 2021-10-22 06:51:16 +0000 UTC - event for inline-volume-tester2-9rcj7-my-volume-0: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "test.csi.azure.com" or manually created by system administrator
Oct 22 07:08:07.770: INFO: At 2021-10-22 06:51:16 +0000 UTC - event for inline-volume-tester2-9rcj7-my-volume-0: {test.csi.azure.com_k8s-agentpool1-22475134-vmss000001_643e5b82-d98f-4701-82bb-2ae23a5c6604 } Provisioning: External provisioner is provisioning volume for claim "ephemeral-1169/inline-volume-tester2-9rcj7-my-volume-0"
Oct 22 07:08:07.770: INFO: At 2021-10-22 06:51:17 +0000 UTC - event for inline-volume-tester2-9rcj7: {default-scheduler } FailedScheduling: 0/3 nodes are available: 3 pod has unbound immediate PersistentVolumeClaims.
Oct 22 07:08:07.770: INFO: At 2021-10-22 06:51:18 +0000 UTC - event for inline-volume-tester2-9rcj7-my-volume-0: {test.csi.azure.com_k8s-agentpool1-22475134-vmss000001_643e5b82-d98f-4701-82bb-2ae23a5c6604 } ProvisioningSucceeded: Successfully provisioned volume pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739
Oct 22 07:08:07.770: INFO: At 2021-10-22 06:51:19 +0000 UTC - event for inline-volume-tester2-9rcj7: {default-scheduler } Scheduled: Successfully assigned ephemeral-1169/inline-volume-tester2-9rcj7 to k8s-agentpool1-22475134-vmss000000
Oct 22 07:08:07.770: INFO: At 2021-10-22 06:51:20 +0000 UTC - event for inline-volume-tester2-9rcj7: {attachdetach-controller } FailedAttachVolume: AttachVolume.Attach failed for volume "pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739" : rpc error: code = Unknown desc = Attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739" to instance "k8s-agentpool1-22475134-vmss000000" failed with Retriable: false, RetryAfter: 0s, HTTPStatusCode: 409, RawError: {

  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}
Oct 22 07:08:07.770: INFO: At 2021-10-22 06:53:22 +0000 UTC - event for inline-volume-tester2-9rcj7: {kubelet k8s-agentpool1-22475134-vmss000000} FailedMount: Unable to attach or mount volumes: unmounted volumes=[my-volume-0], unattached volumes=[my-volume-0 kube-api-access-s2ffn]: timed out waiting for the condition
Oct 22 07:08:07.770: INFO: At 2021-10-22 06:57:54 +0000 UTC - event for inline-volume-tester2-9rcj7: {kubelet k8s-agentpool1-22475134-vmss000000} FailedMount: Unable to attach or mount volumes: unmounted volumes=[my-volume-0], unattached volumes=[kube-api-access-s2ffn my-volume-0]: timed out waiting for the condition
Oct 22 07:08:07.770: INFO: At 2021-10-22 07:06:17 +0000 UTC - event for inline-volume-tester-qkkht: {kubelet k8s-agentpool1-22475134-vmss000000} Killing: Stopping container csi-volume-tester
Oct 22 07:08:07.770: INFO: At 2021-10-22 07:07:51 +0000 UTC - event for inline-volume-tester2-9rcj7: {attachdetach-controller } FailedAttachVolume: AttachVolume.Attach failed for volume "pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739" : rpc error: code = Unknown desc = Attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739" to instance "k8s-agentpool1-22475134-vmss000000" failed with could not find disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739) in current disk list(len: 31) nor in diskMap(map[])
Oct 22 07:08:07.882: INFO: POD                          NODE                                PHASE    GRACE  CONDITIONS
Oct 22 07:08:07.882: INFO: inline-volume-tester2-9rcj7  k8s-agentpool1-22475134-vmss000000  Pending         [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2021-10-22 06:51:19 +0000 UTC  } {Ready False 0001-01-01 00:00:00 +0000 UTC 2021-10-22 06:51:19 +0000 UTC ContainersNotReady containers with unready status: [csi-volume-tester]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2021-10-22 06:51:19 +0000 UTC ContainersNotReady containers with unready status: [csi-volume-tester]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2021-10-22 06:51:19 +0000 UTC  }]
Oct 22 07:08:07.882: INFO: 
Oct 22 07:08:08.102: INFO: 
Logging node info for node k8s-agentpool1-22475134-vmss000000
Oct 22 07:08:08.214: INFO: Node Info: &Node{ObjectMeta:{k8s-agentpool1-22475134-vmss000000    72bf23fc-217b-4f1a-9ed6-d90fb28f8d04 13213 0 2021-10-22 06:15:58 +0000 UTC <nil> <nil> map[agentpool:agentpool1 beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_DS4_v2 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westeurope failure-domain.beta.kubernetes.io/zone:0 kubernetes.azure.com/cluster:kubetest-5p3qtork kubernetes.azure.com/role:agent kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-agentpool1-22475134-vmss000000 kubernetes.io/os:linux kubernetes.io/role:agent node-role.kubernetes.io/agent: node.kubernetes.io/instance-type:Standard_DS4_v2 storageprofile:managed storagetier:Premium_LRS topology.kubernetes.io/region:westeurope topology.kubernetes.io/zone:0 topology.test.csi.azure.com/zone:] map[csi.volume.kubernetes.io/nodeid:{"test.csi.azure.com":"k8s-agentpool1-22475134-vmss000000"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] []  [{kubelet Update v1 2021-10-22 06:15:58 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:agentpool":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.azure.com/cluster":{},"f:kubernetes.azure.com/role":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:storageprofile":{},"f:storagetier":{}}}} } {kubectl-label Update v1 2021-10-22 06:16:06 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/agent":{}}}} } {cloud-controller-manager Update v1 2021-10-22 06:16:12 +0000 UTC FieldsV1 {"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"10.244.0.0/24\"":{}}}} } {cloud-controller-manager Update v1 2021-10-22 06:16:12 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2021-10-22 06:16:15 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}}} } {cloud-node-manager Update v1 2021-10-22 06:17:01 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kubelet Update v1 2021-10-22 06:27:43 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.test.csi.azure.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status} {kube-controller-manager Update v1 2021-10-22 06:48:25 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status}]},Spec:NodeSpec{PodCIDR:10.244.0.0/24,DoNotUseExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool1-22475134-vmss/virtualMachines/0,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[10.244.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{8 0} {<nil>} 8 DecimalSI},ephemeral-storage: {{31036686336 0} {<nil>}  BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{29449355264 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{8 0} {<nil>} 8 DecimalSI},ephemeral-storage: {{27933017657 0} {<nil>} 27933017657 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{28662923264 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2021-10-22 06:17:09 +0000 UTC,LastTransitionTime:2021-10-22 06:17:09 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2021-10-22 07:07:00 +0000 UTC,LastTransitionTime:2021-10-22 06:15:55 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2021-10-22 07:07:00 +0000 UTC,LastTransitionTime:2021-10-22 06:15:55 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2021-10-22 07:07:00 +0000 UTC,LastTransitionTime:2021-10-22 06:15:55 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2021-10-22 07:07:00 +0000 UTC,LastTransitionTime:2021-10-22 06:16:12 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.240.0.4,},NodeAddress{Type:Hostname,Address:k8s-agentpool1-22475134-vmss000000,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:d2790c386f8b4816867c93ee5ec07afb,SystemUUID:99c573b5-645b-634d-bb56-59a354ec6405,BootID:87fd645e-5f99-4dfe-9c69-ecf7970fbbbf,KernelVersion:5.4.0-1059-azure,OSImage:Ubuntu 18.04.6 LTS,ContainerRuntimeVersion:docker://20.10.7+azure,KubeletVersion:v1.22.2,KubeProxyVersion:v1.22.2,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/azuredisk-csi@sha256:0743369a58a6620586dcf8671640b81f8dc66bc5986a5c2d3b23a4ce6a9886e8 k8sprow.azurecr.io/azuredisk-csi:v1.9.0-716f56a44514b799d2d6c6922145f22026e7d4e2],SizeBytes:239897501,},ContainerImage{Names:[mcr.microsoft.com/k8s/csi/azurefile-csi@sha256:91972069fc98aae86e3297b054cb0a347b256d0a81799337e1ec0c196c341065 mcr.microsoft.com/k8s/csi/azurefile-csi:v1.6.0],SizeBytes:223047280,},ContainerImage{Names:[mcr.microsoft.com/k8s/csi/azuredisk-csi@sha256:d1e335c6d52dc8b0f7db7650879e1673fda5c561761a52f9ba72a216d9ca0721 mcr.microsoft.com/k8s/csi/azuredisk-csi:v1.7.0],SizeBytes:212174459,},ContainerImage{Names:[mcr.microsoft.com/containernetworking/azure-npm@sha256:106f669f48e5e80c4ec0afb49858ead72cf4b901cd8664e7bf81f8d789e56e12 mcr.microsoft.com/containernetworking/azure-npm:v1.2.2_hotfix],SizeBytes:175230380,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-apiserver@sha256:6c37c0efdaf49d24dafcced601fb78750205670e4e41842ab3a71935ed23b753 mcr.microsoft.com/oss/kubernetes/kube-apiserver:v1.22.2],SizeBytes:128401987,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:758db666ac7028534dba72e7e9bb1e57bb81b8196f976f7a5cc351ef8b3529e1 k8s.gcr.io/e2e-test-images/agnhost:2.32],SizeBytes:125910179,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/secrets-store/driver@sha256:c0d040a1c4fbfceb65663e31c09ea40f4f78e356437610cbc3fbb4bb409bd6f1 mcr.microsoft.com/oss/kubernetes-csi/secrets-store/driver:v0.0.19],SizeBytes:123229697,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-controller-manager@sha256:76fa95d463bf8950324d68f9f53063bb73628adcc862708f4534d6f44546df63 mcr.microsoft.com/oss/kubernetes/kube-controller-manager:v1.22.2],SizeBytes:121930716,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-proxy@sha256:b1ec3f3090fc799799bee98619a33300d8ee9095219adc4bb688084718810d23 mcr.microsoft.com/oss/kubernetes/kube-proxy:v1.22.2],SizeBytes:103625747,},ContainerImage{Names:[mcr.microsoft.com/oss/azure/secrets-store/provider-azure@sha256:6f67f3d0c7cdde5702f8ce7f101b6519daa0237f0c34fecb7c058b6af8c22ad1 mcr.microsoft.com/oss/azure/secrets-store/provider-azure:0.0.12],SizeBytes:101061355,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/autoscaler/cluster-autoscaler@sha256:29ba45b98c8035adea245f124d08d54dffbd27068940edd879cfd86022db20f9 mcr.microsoft.com/oss/kubernetes/autoscaler/cluster-autoscaler:v1.22.0],SizeBytes:99406403,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-addon-manager@sha256:32e2836018c96e73533bd4642fe438e465b81dcbfa8b7b61935a6f4d0246c7ae mcr.microsoft.com/oss/kubernetes/kube-addon-manager:v9.1.3],SizeBytes:86832059,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-addon-manager@sha256:92c2c5aad9012ee32d2a43a74966cc0adc6ccb1705ad15abb10485ecf406d88b mcr.microsoft.com/oss/kubernetes/kube-addon-manager:v9.1.5],SizeBytes:84094027,},ContainerImage{Names:[mcr.microsoft.com/oss/nvidia/k8s-device-plugin@sha256:0f5b52bf28239234e831697d96db63ac03cde70fe68058f964504ab7564ee810 mcr.microsoft.com/oss/nvidia/k8s-device-plugin:1.0.0-beta6],SizeBytes:64160241,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/metrics-server@sha256:682d3a1a48abbe03df05e1084923ab668a6651fd80327f9694b0cf302cc49f90 mcr.microsoft.com/oss/kubernetes/metrics-server:v0.5.0],SizeBytes:63501445,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-provisioner@sha256:e9ddadc44ba87a4a27f67e54760a14f9986885b534b3dff170a14eae1e35d213 mcr.microsoft.com/oss/kubernetes-csi/csi-provisioner:v3.0.0],SizeBytes:56881280,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-resizer@sha256:c5bb71ceaac60b1a4b58739fa07b709f6248c452ff6272a384d2f7648895a750 mcr.microsoft.com/oss/kubernetes-csi/csi-resizer:v1.3.0],SizeBytes:54313772,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-snapshotter@sha256:e28e996e03519565907d7158e0e1a6613cec76220fa8625e0ebfc1aa1c3bacb2 mcr.microsoft.com/oss/kubernetes-csi/csi-snapshotter:v4.2.1],SizeBytes:54210936,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-attacher@sha256:6b41e7153ebdfdc1501aa65184624bc15fd33a52d93f88ec3a758d0f8c9b8c10 mcr.microsoft.com/oss/kubernetes-csi/csi-attacher:v3.3.0],SizeBytes:53842561,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-scheduler@sha256:0457773d5575596d4f898913b45293f77dea0d955bc2002a4dbf7ea827cfe4f0 mcr.microsoft.com/oss/kubernetes/kube-scheduler:v1.22.2],SizeBytes:52626428,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/snapshot-controller@sha256:bb61653b29713588c280999cbb25b472fef8794623164fd9177c270da1121db0 mcr.microsoft.com/oss/kubernetes-csi/snapshot-controller:v4.2.1],SizeBytes:51575245,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager@sha256:ea0065331ce28081b4a048bd18530bbe4fe675dacb3c9515a2022ef10abf95ff mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager:v1.1.1],SizeBytes:50901571,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager@sha256:c24707c5f947d9e384d0f111469ca3266a01c48de6557f306e718c23d699cadf mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager:v1.1.1],SizeBytes:50295363,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/ip-masq-agent@sha256:bddf19be25644c7176e8c501465dbad07d032288334f135877b984660b0d3ee6 mcr.microsoft.com/oss/kubernetes/ip-masq-agent:v2.5.0],SizeBytes:50146762,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/coredns@sha256:1c23b17c78ffbc90c068ee088b77456284b9649952fa127b0f86a0ccff575539 mcr.microsoft.com/oss/kubernetes/coredns:1.7.0],SizeBytes:45233936,},ContainerImage{Names:[mcr.microsoft.com/oss/etcd-io/etcd@sha256:cf587862e3f1b6fa4d9a2565520a34f164bdf72c50f37af8c3c668160593246e mcr.microsoft.com/oss/etcd-io/etcd:v3.3.25],SizeBytes:41832119,},ContainerImage{Names:[mcr.microsoft.com/k8s/aad-pod-identity/mic@sha256:bd9465be94966b9a917e1e3904fa5e63dd91772ccadf304e18ffd8e4ad8ccedd mcr.microsoft.com/k8s/aad-pod-identity/mic:1.6.1],SizeBytes:41374894,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/autoscaler/cluster-proportional-autoscaler@sha256:753e3ca79dc833feed24ee4107ff4f06a96abbd6cefba8d41d7dec6f4d3e75f0 mcr.microsoft.com/oss/kubernetes/autoscaler/cluster-proportional-autoscaler:1.7.1],SizeBytes:40051560,},ContainerImage{Names:[mcr.microsoft.com/k8s/aad-pod-identity/nmi@sha256:02128fefcdb7593ac53fc342e2c53a0fc6fabd813036bf60457bf43cc2940116 mcr.microsoft.com/k8s/aad-pod-identity/nmi:1.6.1],SizeBytes:38007982,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:921f301c44dda06a325164accf22e78ecc570b5c7d9d6ee4c66bd8cbb2b60b9a mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.16],SizeBytes:26970670,},ContainerImage{Names:[mcr.microsoft.com/k8s/kms/keyvault@sha256:1a27e175f8c125209e32d2957b5509fe20757bd8cb309ff9da598799b56326fb mcr.microsoft.com/k8s/kms/keyvault:v0.0.10],SizeBytes:23077387,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-node-driver-registrar@sha256:f1205ba7ae0b0783bb19442064f4d7d664f3dc0c9b0b5b33358e5eef5aedc554 mcr.microsoft.com/oss/kubernetes-csi/csi-node-driver-registrar:v2.3.0],SizeBytes:18746317,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/livenessprobe@sha256:3770688d4efa33f8f284cc2ef58d33efa4ff606147e0a8de20e67458e13fedc2 mcr.microsoft.com/oss/kubernetes-csi/livenessprobe:v2.4.0],SizeBytes:17183286,},ContainerImage{Names:[busybox@sha256:f7ca5a32c10d51aeda3b4d01c61c6061f497893d7f6628b92f822f7117182a57 busybox:latest],SizeBytes:1235724,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8],SizeBytes:1159025,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:1154361,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/pause@sha256:e3b8c20681593c21b344ad801fbb8abaf564427ee3a57a9fcfa3b455f917ce46 mcr.microsoft.com/oss/kubernetes/pause:3.4.1],SizeBytes:682696,},},VolumesInUse:[kubernetes.io/csi/test.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-07911b25-debe-4781-8c5f-b81e70b6bd7d kubernetes.io/csi/test.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-09b74d52-8b3c-431d-a110-aef9ac1d65b2 kubernetes.io/csi/test.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739 kubernetes.io/csi/test.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-13fe5e26-c069-46f7-836d-30618bd80606 kubernetes.io/csi/test.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-2cacef5a-9bc2-4a71-a22d-c72e842538c3 kubernetes.io/csi/test.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-3107aca1-7d54-4ac3-945a-25d6c0246a0c kubernetes.io/csi/test.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-3b7d143a-9eee-4ed1-b383-cd1d6673be53 kubernetes.io/csi/test.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-42ebe9ec-eca5-4926-8da8-4b21f93e4b2f kubernetes.io/csi/test.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-4468e1b3-f470-4e0c-9404-16d282d00932 kubernetes.io/csi/test.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-4a6ce6eb-10e0-401a-8257-fc05a4e4349f kubernetes.io/csi/test.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-4bf40635-c941-4c51-9db3-35374a82e0df kubernetes.io/csi/test.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-506f8b7e-8262-4a63-8f37-1a4145b662aa kubernetes.io/csi/test.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-625cb0a6-3ef0-4bb1-8a80-8d075355b851 kubernetes.io/csi/test.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-6e5bb4b7-9c65-4f65-9a30-72575fc9afac kubernetes.io/csi/test.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-7292229f-a145-4b90-84c0-8ad707acb8f1 kubernetes.io/csi/test.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-7598fafa-b8d8-45fc-bd09-7abb6e913a34 kubernetes.io/csi/test.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-7a1277c4-4dd2-4783-a3a2-70e6c7455fdc kubernetes.io/csi/test.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-84c19a86-6835-4758-8251-2d417a93c3c7 kubernetes.io/csi/test.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-90b4af34-20b0-494b-a46f-a3471b81f63a kubernetes.io/csi/test.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5 kubernetes.io/csi/test.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-9273691d-6bca-4b0d-a798-48cd39f4be66 kubernetes.io/csi/test.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-98953f15-d806-48a5-8627-b381bc559972 kubernetes.io/csi/test.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-a38ef8f3-7817-4091-b6e0-7741e79fd589 kubernetes.io/csi/test.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-a43477fa-15b0-4f50-92e3-0a58c87f24cd kubernetes.io/csi/test.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-aba6fcc4-93be-4a03-b913-b0ad6dfd64e9 kubernetes.io/csi/test.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-ac27c51e-aedd-4f4e-b24e-df2508a5e9b1 kubernetes.io/csi/test.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-acd944ee-f869-4788-b766-c6715eee7d96 kubernetes.io/csi/test.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-d330a831-95bd-45e4-b274-292d41f80459 kubernetes.io/csi/test.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-d92c57dc-1538-4651-bc16-6cc2c3ef92cc kubernetes.io/csi/test.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-df1a5c24-dd06-4ce9-9ef0-7c6c82516986 kubernetes.io/csi/test.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-e01638a5-80eb-44cb-902c-a1c302a75244 kubernetes.io/csi/test.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-e7fe0c82-6611-4efa-b38b-831ca3daa630 kubernetes.io/csi/test.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-ede47b18-01aa-4bba-b094-3bba45a8d78c],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/test.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-d330a831-95bd-45e4-b274-292d41f80459,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/test.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-a43477fa-15b0-4f50-92e3-0a58c87f24cd,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/test.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-df1a5c24-dd06-4ce9-9ef0-7c6c82516986,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/test.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-13fe5e26-c069-46f7-836d-30618bd80606,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/test.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-09b74d52-8b3c-431d-a110-aef9ac1d65b2,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/test.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-ac27c51e-aedd-4f4e-b24e-df2508a5e9b1,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/test.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-625cb0a6-3ef0-4bb1-8a80-8d075355b851,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/test.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-84c19a86-6835-4758-8251-2d417a93c3c7,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/test.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-4a6ce6eb-10e0-401a-8257-fc05a4e4349f,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/test.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-d92c57dc-1538-4651-bc16-6cc2c3ef92cc,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/test.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-a38ef8f3-7817-4091-b6e0-7741e79fd589,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/test.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-7a1277c4-4dd2-4783-a3a2-70e6c7455fdc,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/test.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-7598fafa-b8d8-45fc-bd09-7abb6e913a34,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/test.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-e01638a5-80eb-44cb-902c-a1c302a75244,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/test.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-2cacef5a-9bc2-4a71-a22d-c72e842538c3,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/test.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-98953f15-d806-48a5-8627-b381bc559972,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/test.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-7292229f-a145-4b90-84c0-8ad707acb8f1,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/test.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-acd944ee-f869-4788-b766-c6715eee7d96,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/test.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-6e5bb4b7-9c65-4f65-9a30-72575fc9afac,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/test.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-aba6fcc4-93be-4a03-b913-b0ad6dfd64e9,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/test.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-42ebe9ec-eca5-4926-8da8-4b21f93e4b2f,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/test.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-90b4af34-20b0-494b-a46f-a3471b81f63a,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/test.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-506f8b7e-8262-4a63-8f37-1a4145b662aa,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/test.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-4bf40635-c941-4c51-9db3-35374a82e0df,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/test.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-e7fe0c82-6611-4efa-b38b-831ca3daa630,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/test.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-4468e1b3-f470-4e0c-9404-16d282d00932,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/test.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-9273691d-6bca-4b0d-a798-48cd39f4be66,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/test.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-07911b25-debe-4781-8c5f-b81e70b6bd7d,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/test.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-3107aca1-7d54-4ac3-945a-25d6c0246a0c,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/test.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-ede47b18-01aa-4bba-b094-3bba45a8d78c,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/test.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-3b7d143a-9eee-4ed1-b383-cd1d6673be53,DevicePath:,},},Config:nil,},}
... skipping 97 lines ...
  [Testpattern: Generic Ephemeral-volume (default fs) (immediate-binding)] ephemeral
  /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50
    should support two pods which share the same volume [Measurement]
    /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/ephemeral.go:183

    Oct 22 07:06:16.671: waiting for second pod with inline volume
    Unexpected error:
        <*errors.errorString | 0xc00024c250>: {
            s: "timed out waiting for the condition",
        }
        timed out waiting for the condition
    occurred

    /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/ephemeral.go:201
------------------------------
{"msg":"FAILED External Storage [Driver: test.csi.azure.com] [Testpattern: Generic Ephemeral-volume (default fs) (immediate-binding)] ephemeral should support two pods which share the same volume","total":12,"completed":3,"skipped":942,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Generic Ephemeral-volume (default fs) (immediate-binding)] ephemeral should support two pods which share the same volume"]}

SSSSSS
------------------------------
Oct 22 07:08:10.795: INFO: Running AfterSuite actions on all nodes
Oct 22 07:08:10.795: INFO: Running Cleanup Action: k8s.io/kubernetes/test/e2e/storage.glob..func17.2
Oct 22 07:08:10.795: INFO: Running Cleanup Action: k8s.io/kubernetes/test/e2e/storage.glob..func8.2
... skipping 99 lines ...
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (filesystem volmode)] volumeLimits
  /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50
    should support volume limits [Serial]
    /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/volumelimits.go:123
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (filesystem volmode)] volumeLimits should support volume limits [Serial]","total":19,"completed":6,"skipped":738,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy (Always)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed in first pod, new pod with same fsgroup applied to the volume contents"]}

SSSSS
------------------------------
[BeforeEach] [Testpattern: Inline-volume (default fs)] subPath
  /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:51
Oct 22 07:15:12.740: INFO: Driver "test.csi.azure.com" does not support volume type "InlineVolume" - skipping
... skipping 78 lines ...
Oct 22 07:15:12.987: INFO: Running AfterSuite actions on node 1



Summarizing 6 Failures:

[Fail] External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy [Measurement] (Always)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed in first pod, new pod with same fsgroup applied to the volume contents 
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/fsgroupchangepolicy.go:132

[Fail] External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (filesystem volmode)] volumeMode [Measurement] should not mount / map unused volumes in a pod [LinuxOnly] 
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/volumemode.go:188

[Fail] External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath [Measurement] should be able to unmount after the subpath directory is deleted [LinuxOnly] 
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/subpath.go:465

[Fail] External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy [Measurement] (OnRootMismatch)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed in first pod, new pod with same fsgroup skips ownership changes to the volume contents 
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/fsgroupchangepolicy.go:250

[Fail] External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] volumeMode [Measurement] should not mount / map unused volumes in a pod [LinuxOnly] 
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/volumemode.go:188

[Fail] External Storage [Driver: test.csi.azure.com] [Testpattern: Generic Ephemeral-volume (default fs) (immediate-binding)] ephemeral [Measurement] should support two pods which share the same volume 
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/ephemeral.go:201

Ran 38 of 6664 Specs in 2866.121 seconds
FAIL! -- 32 Passed | 6 Failed | 0 Pending | 6626 Skipped 

Ginkgo ran 1 suite in 47m49.803542179s
Test Suite Failed
+ print_logs
+ sed -i s/disk.csi.azure.com/test.csi.azure.com/g deploy/example/storageclass-azuredisk-csi.yaml
+ bash ./hack/verify-examples.sh linux azurepubliccloud ephemeral test
begin to create deployment examples ...
storageclass.storage.k8s.io/managed-csi created
Applying config "deploy/example/deployment.yaml"
... skipping 73 lines ...
Platform: linux/amd64
Topology Key: topology.test.csi.azure.com/zone

Streaming logs below:
I1022 06:27:16.751113       1 azuredisk.go:139] driver userAgent: test.csi.azure.com/v1.9.0-716f56a44514b799d2d6c6922145f22026e7d4e2 gc/go1.17.2 (amd64-linux) e2e-test
I1022 06:27:16.752269       1 azure_disk_utils.go:137] reading cloud config from secret kube-system/azure-cloud-provider
W1022 06:27:16.776553       1 azure_disk_utils.go:144] InitializeCloudFromSecret: failed to get cloud config from secret kube-system/azure-cloud-provider: failed to get secret kube-system/azure-cloud-provider: secrets "azure-cloud-provider" not found
I1022 06:27:16.776576       1 azure_disk_utils.go:149] could not read cloud config from secret kube-system/azure-cloud-provider
I1022 06:27:16.776603       1 azure_disk_utils.go:159] use default AZURE_CREDENTIAL_FILE env var: /etc/kubernetes/azure.json
I1022 06:27:16.776622       1 azure_disk_utils.go:167] read cloud config from file: /etc/kubernetes/azure.json successfully
I1022 06:27:16.777276       1 azure_auth.go:234] Using AzurePublicCloud environment
I1022 06:27:16.777335       1 azure_auth.go:96] azure: using managed identity extension to retrieve access token
I1022 06:27:16.777357       1 azure_auth.go:102] azure: using User Assigned MSI ID to retrieve access token
... skipping 91 lines ...
I1022 06:27:29.705294       1 utils.go:95] GRPC call: /csi.v1.Controller/CreateVolume
I1022 06:27:29.705320       1 utils.go:96] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"requisite":[{"segments":{"topology.test.csi.azure.com/zone":""}}]},"capacity_range":{"required_bytes":5368709120},"name":"pvc-545e7003-a379-4bb4-a461-d761add02e77","parameters":{"csi.storage.k8s.io/pv/name":"pvc-545e7003-a379-4bb4-a461-d761add02e77","csi.storage.k8s.io/pvc/name":"test.csi.azure.com6t7zx","csi.storage.k8s.io/pvc/namespace":"provisioning-2124"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]}
I1022 06:27:29.847031       1 utils.go:95] GRPC call: /csi.v1.Controller/CreateVolume
I1022 06:27:29.847062       1 utils.go:96] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"requisite":[{"segments":{"topology.test.csi.azure.com/zone":""}}]},"capacity_range":{"required_bytes":5368709120},"name":"pvc-20082bbf-dc21-47a4-8a4b-e7a04b62cdac","parameters":{"csi.storage.k8s.io/pv/name":"pvc-20082bbf-dc21-47a4-8a4b-e7a04b62cdac","csi.storage.k8s.io/pvc/name":"test.csi.azure.comp2wmc","csi.storage.k8s.io/pvc/namespace":"provisioning-5043"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]}
I1022 06:27:29.946809       1 utils.go:95] GRPC call: /csi.v1.Controller/CreateVolume
I1022 06:27:29.946834       1 utils.go:96] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"requisite":[{"segments":{"topology.test.csi.azure.com/zone":""}}]},"capacity_range":{"required_bytes":5368709120},"name":"pvc-018d34da-7b68-4682-b04f-4d5c9fb75314","parameters":{"csi.storage.k8s.io/pv/name":"pvc-018d34da-7b68-4682-b04f-4d5c9fb75314","csi.storage.k8s.io/pvc/name":"test.csi.azure.comzm5jh","csi.storage.k8s.io/pvc/namespace":"fsgroupchangepolicy-1691"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]}
I1022 06:27:30.046718       1 azure_diskclient.go:135] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-20082bbf-dc21-47a4-8a4b-e7a04b62cdac, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-20082bbf-dc21-47a4-8a4b-e7a04b62cdac' under resource group 'kubetest-5p3qtork' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I1022 06:27:30.046755       1 azure_diskclient.go:135] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-9b8ef2c7-3d19-4ab3-80c1-fb0736e83e7d, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-9b8ef2c7-3d19-4ab3-80c1-fb0736e83e7d' under resource group 'kubetest-5p3qtork' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I1022 06:27:30.046969       1 controllerserver.go:271] begin to create azure disk(pvc-20082bbf-dc21-47a4-8a4b-e7a04b62cdac) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) diskZone() maxShares(0)
I1022 06:27:30.046997       1 azure_managedDiskController.go:88] azureDisk - creating new managed Name:pvc-20082bbf-dc21-47a4-8a4b-e7a04b62cdac StorageAccountType:StandardSSD_LRS Size:5
I1022 06:27:30.046973       1 controllerserver.go:271] begin to create azure disk(pvc-9b8ef2c7-3d19-4ab3-80c1-fb0736e83e7d) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) diskZone() maxShares(0)
I1022 06:27:30.047074       1 azure_managedDiskController.go:88] azureDisk - creating new managed Name:pvc-9b8ef2c7-3d19-4ab3-80c1-fb0736e83e7d StorageAccountType:StandardSSD_LRS Size:5
I1022 06:27:30.146460       1 azure_diskclient.go:135] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-545e7003-a379-4bb4-a461-d761add02e77, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-545e7003-a379-4bb4-a461-d761add02e77' under resource group 'kubetest-5p3qtork' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I1022 06:27:30.146533       1 controllerserver.go:271] begin to create azure disk(pvc-545e7003-a379-4bb4-a461-d761add02e77) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) diskZone() maxShares(0)
I1022 06:27:30.146549       1 azure_managedDiskController.go:88] azureDisk - creating new managed Name:pvc-545e7003-a379-4bb4-a461-d761add02e77 StorageAccountType:StandardSSD_LRS Size:5
I1022 06:27:30.146478       1 azure_diskclient.go:135] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-018d34da-7b68-4682-b04f-4d5c9fb75314, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-018d34da-7b68-4682-b04f-4d5c9fb75314' under resource group 'kubetest-5p3qtork' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I1022 06:27:30.146620       1 controllerserver.go:271] begin to create azure disk(pvc-018d34da-7b68-4682-b04f-4d5c9fb75314) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) diskZone() maxShares(0)
I1022 06:27:30.146642       1 azure_managedDiskController.go:88] azureDisk - creating new managed Name:pvc-018d34da-7b68-4682-b04f-4d5c9fb75314 StorageAccountType:StandardSSD_LRS Size:5
I1022 06:27:30.826318       1 utils.go:95] GRPC call: /csi.v1.Controller/CreateVolume
I1022 06:27:30.826345       1 utils.go:96] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"requisite":[{"segments":{"topology.test.csi.azure.com/zone":""}}]},"capacity_range":{"required_bytes":5368709120},"name":"pvc-d0f0cce2-e6a9-4010-b1a2-189c3f78f4a6","parameters":{"csi.storage.k8s.io/pv/name":"pvc-d0f0cce2-e6a9-4010-b1a2-189c3f78f4a6","csi.storage.k8s.io/pvc/name":"test.csi.azure.comrvcx6","csi.storage.k8s.io/pvc/namespace":"volume-expand-2387"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]}
I1022 06:27:30.845073       1 azure_diskclient.go:135] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-d0f0cce2-e6a9-4010-b1a2-189c3f78f4a6, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-d0f0cce2-e6a9-4010-b1a2-189c3f78f4a6' under resource group 'kubetest-5p3qtork' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I1022 06:27:30.845138       1 controllerserver.go:271] begin to create azure disk(pvc-d0f0cce2-e6a9-4010-b1a2-189c3f78f4a6) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) diskZone() maxShares(0)
I1022 06:27:30.845153       1 azure_managedDiskController.go:88] azureDisk - creating new managed Name:pvc-d0f0cce2-e6a9-4010-b1a2-189c3f78f4a6 StorageAccountType:StandardSSD_LRS Size:5
I1022 06:27:32.075206       1 utils.go:95] GRPC call: /csi.v1.Controller/CreateVolume
I1022 06:27:32.075232       1 utils.go:96] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"requisite":[{"segments":{"topology.test.csi.azure.com/zone":""}}]},"capacity_range":{"required_bytes":5368709120},"name":"pvc-d2b2d817-66c5-404d-969d-b833e1464a67","parameters":{"csi.storage.k8s.io/pv/name":"pvc-d2b2d817-66c5-404d-969d-b833e1464a67","csi.storage.k8s.io/pvc/name":"inline-volume-tester-ctxmq-my-volume-0","csi.storage.k8s.io/pvc/namespace":"ephemeral-8965"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]}
I1022 06:27:32.076029       1 utils.go:95] GRPC call: /csi.v1.Controller/CreateVolume
I1022 06:27:32.076051       1 utils.go:96] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"requisite":[{"segments":{"topology.test.csi.azure.com/zone":""}}]},"capacity_range":{"required_bytes":5368709120},"name":"pvc-20467431-462d-4b84-ad69-43d31f6a7484","parameters":{"csi.storage.k8s.io/pv/name":"pvc-20467431-462d-4b84-ad69-43d31f6a7484","csi.storage.k8s.io/pvc/name":"inline-volume-tester-b9kwc-my-volume-0","csi.storage.k8s.io/pvc/namespace":"ephemeral-6468"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]}
I1022 06:27:32.083556       1 utils.go:95] GRPC call: /csi.v1.Controller/CreateVolume
I1022 06:27:32.083580       1 utils.go:96] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"requisite":[{"segments":{"topology.test.csi.azure.com/zone":""}}]},"capacity_range":{"required_bytes":5368709120},"name":"pvc-81b30bc4-32d1-427d-a923-fc43cefb3a70","parameters":{"csi.storage.k8s.io/pv/name":"pvc-81b30bc4-32d1-427d-a923-fc43cefb3a70","csi.storage.k8s.io/pvc/name":"inline-volume-tester-b9kwc-my-volume-1","csi.storage.k8s.io/pvc/namespace":"ephemeral-6468"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]}
I1022 06:27:32.095084       1 azure_diskclient.go:135] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-d2b2d817-66c5-404d-969d-b833e1464a67, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-d2b2d817-66c5-404d-969d-b833e1464a67' under resource group 'kubetest-5p3qtork' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I1022 06:27:32.095228       1 controllerserver.go:271] begin to create azure disk(pvc-d2b2d817-66c5-404d-969d-b833e1464a67) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) diskZone() maxShares(0)
I1022 06:27:32.095241       1 azure_managedDiskController.go:88] azureDisk - creating new managed Name:pvc-d2b2d817-66c5-404d-969d-b833e1464a67 StorageAccountType:StandardSSD_LRS Size:5
I1022 06:27:32.096613       1 azure_diskclient.go:135] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-20467431-462d-4b84-ad69-43d31f6a7484, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-20467431-462d-4b84-ad69-43d31f6a7484' under resource group 'kubetest-5p3qtork' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I1022 06:27:32.096678       1 controllerserver.go:271] begin to create azure disk(pvc-20467431-462d-4b84-ad69-43d31f6a7484) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) diskZone() maxShares(0)
I1022 06:27:32.096689       1 azure_managedDiskController.go:88] azureDisk - creating new managed Name:pvc-20467431-462d-4b84-ad69-43d31f6a7484 StorageAccountType:StandardSSD_LRS Size:5
I1022 06:27:32.146607       1 azure_diskclient.go:135] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-81b30bc4-32d1-427d-a923-fc43cefb3a70, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-81b30bc4-32d1-427d-a923-fc43cefb3a70' under resource group 'kubetest-5p3qtork' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I1022 06:27:32.146670       1 controllerserver.go:271] begin to create azure disk(pvc-81b30bc4-32d1-427d-a923-fc43cefb3a70) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) diskZone() maxShares(0)
I1022 06:27:32.146680       1 azure_managedDiskController.go:88] azureDisk - creating new managed Name:pvc-81b30bc4-32d1-427d-a923-fc43cefb3a70 StorageAccountType:StandardSSD_LRS Size:5
I1022 06:27:32.543740       1 azure_managedDiskController.go:249] azureDisk - created new MD Name:pvc-545e7003-a379-4bb4-a461-d761add02e77 StorageAccountType:StandardSSD_LRS Size:5
I1022 06:27:32.543810       1 controllerserver.go:348] create azure disk(pvc-545e7003-a379-4bb4-a461-d761add02e77) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) tags(map[kubernetes.io-created-for-pv-name:pvc-545e7003-a379-4bb4-a461-d761add02e77 kubernetes.io-created-for-pvc-name:test.csi.azure.com6t7zx kubernetes.io-created-for-pvc-namespace:provisioning-2124]) successfully
I1022 06:27:32.543867       1 utils.go:102] GRPC response: {"volume":{"accessible_topology":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"capacity_bytes":5368709120,"content_source":{"Type":null},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-545e7003-a379-4bb4-a461-d761add02e77","csi.storage.k8s.io/pvc/name":"test.csi.azure.com6t7zx","csi.storage.k8s.io/pvc/namespace":"provisioning-2124","requestedsizegib":"5"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-545e7003-a379-4bb4-a461-d761add02e77"}}
I1022 06:27:32.570216       1 azure_managedDiskController.go:249] azureDisk - created new MD Name:pvc-20082bbf-dc21-47a4-8a4b-e7a04b62cdac StorageAccountType:StandardSSD_LRS Size:5
... skipping 57 lines ...
I1022 06:27:35.206386       1 controllerserver.go:461] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-d2b2d817-66c5-404d-969d-b833e1464a67" to node "k8s-agentpool1-22475134-vmss000001"
I1022 06:27:35.246760       1 utils.go:95] GRPC call: /csi.v1.Controller/ControllerPublishVolume
I1022 06:27:35.246780       1 utils.go:96] GRPC request: {"node_id":"k8s-agentpool1-22475134-vmss000000","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-81b30bc4-32d1-427d-a923-fc43cefb3a70","csi.storage.k8s.io/pvc/name":"inline-volume-tester-b9kwc-my-volume-1","csi.storage.k8s.io/pvc/namespace":"ephemeral-6468","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1634884037556-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-81b30bc4-32d1-427d-a923-fc43cefb3a70"}
I1022 06:27:35.287057       1 controllerserver.go:440] GetDiskLun returned: <nil>. Initiating attaching volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-81b30bc4-32d1-427d-a923-fc43cefb3a70" to node "k8s-agentpool1-22475134-vmss000000".
I1022 06:27:35.287081       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:27:35.287096       1 controllerserver.go:461] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-81b30bc4-32d1-427d-a923-fc43cefb3a70" to node "k8s-agentpool1-22475134-vmss000000"
I1022 06:27:35.368016       1 azure_controller_vmss.go:134] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-20082bbf-dc21-47a4-8a4b-e7a04b62cdac:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-20082bbf-dc21-47a4-8a4b-e7a04b62cdac  true false 0})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING)
I1022 06:27:35.368251       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000001" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:27:35.368299       1 azure_vmss.go:198] Couldn't find VMSS VM with nodeName k8s-agentpool1-22475134-vmss000001, refreshing the cache(vmss: k8s-agentpool1-22475134-vmss, rg: kubetest-5p3qtork)
I1022 06:27:35.423291       1 azure_controller_common.go:242] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-d2b2d817-66c5-404d-969d-b833e1464a67" lun 1 to node "k8s-agentpool1-22475134-vmss000001", diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-d2b2d817-66c5-404d-969d-b833e1464a67:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-d2b2d817-66c5-404d-969d-b833e1464a67  true false 1})]
I1022 06:27:35.423324       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000001" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:27:35.423340       1 azure_controller_vmss.go:122] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-d2b2d817-66c5-404d-969d-b833e1464a67:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-d2b2d817-66c5-404d-969d-b833e1464a67  true false 1})])
I1022 06:27:35.445567       1 azure_controller_vmss.go:134] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-9b8ef2c7-3d19-4ab3-80c1-fb0736e83e7d:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-9b8ef2c7-3d19-4ab3-80c1-fb0736e83e7d  true false 0})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING)
I1022 06:27:35.445742       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:27:35.445784       1 azure_vmss.go:198] Couldn't find VMSS VM with nodeName k8s-agentpool1-22475134-vmss000000, refreshing the cache(vmss: k8s-agentpool1-22475134-vmss, rg: kubetest-5p3qtork)
I1022 06:27:35.508526       1 azure_controller_common.go:242] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-545e7003-a379-4bb4-a461-d761add02e77" lun 1 to node "k8s-agentpool1-22475134-vmss000000", diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-018d34da-7b68-4682-b04f-4d5c9fb75314:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-018d34da-7b68-4682-b04f-4d5c9fb75314  true false 2}) /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-20467431-462d-4b84-ad69-43d31f6a7484:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-20467431-462d-4b84-ad69-43d31f6a7484  true false 3}) /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-545e7003-a379-4bb4-a461-d761add02e77:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-545e7003-a379-4bb4-a461-d761add02e77  true false 1}) /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-81b30bc4-32d1-427d-a923-fc43cefb3a70:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-81b30bc4-32d1-427d-a923-fc43cefb3a70  true false 4})]
I1022 06:27:35.508558       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:27:35.508593       1 azure_controller_vmss.go:122] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-018d34da-7b68-4682-b04f-4d5c9fb75314:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-018d34da-7b68-4682-b04f-4d5c9fb75314  true false 2}) /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-20467431-462d-4b84-ad69-43d31f6a7484:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-20467431-462d-4b84-ad69-43d31f6a7484  true false 3}) /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-545e7003-a379-4bb4-a461-d761add02e77:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-545e7003-a379-4bb4-a461-d761add02e77  true false 1}) /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-81b30bc4-32d1-427d-a923-fc43cefb3a70:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-81b30bc4-32d1-427d-a923-fc43cefb3a70  true false 4})])
I1022 06:27:35.589624       1 utils.go:95] GRPC call: /csi.v1.Controller/ControllerPublishVolume
I1022 06:27:35.589645       1 utils.go:96] GRPC request: {"node_id":"k8s-agentpool1-22475134-vmss000001","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-d0f0cce2-e6a9-4010-b1a2-189c3f78f4a6","csi.storage.k8s.io/pvc/name":"test.csi.azure.comrvcx6","csi.storage.k8s.io/pvc/namespace":"volume-expand-2387","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1634884037556-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-d0f0cce2-e6a9-4010-b1a2-189c3f78f4a6"}
I1022 06:27:35.613083       1 controllerserver.go:440] GetDiskLun returned: <nil>. Initiating attaching volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-d0f0cce2-e6a9-4010-b1a2-189c3f78f4a6" to node "k8s-agentpool1-22475134-vmss000001".
I1022 06:27:35.613106       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000001" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:27:35.613138       1 controllerserver.go:461] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-d0f0cce2-e6a9-4010-b1a2-189c3f78f4a6" to node "k8s-agentpool1-22475134-vmss000001"
I1022 06:27:35.743385       1 azure_controller_vmss.go:134] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-018d34da-7b68-4682-b04f-4d5c9fb75314:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-018d34da-7b68-4682-b04f-4d5c9fb75314  true false 2}) /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-20467431-462d-4b84-ad69-43d31f6a7484:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-20467431-462d-4b84-ad69-43d31f6a7484  true false 3}) /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-545e7003-a379-4bb4-a461-d761add02e77:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-545e7003-a379-4bb4-a461-d761add02e77  true false 1}) /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-81b30bc4-32d1-427d-a923-fc43cefb3a70:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-81b30bc4-32d1-427d-a923-fc43cefb3a70  true false 4})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING)
I1022 06:27:35.743531       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:27:35.743597       1 azure_vmss.go:198] Couldn't find VMSS VM with nodeName k8s-agentpool1-22475134-vmss000000, refreshing the cache(vmss: k8s-agentpool1-22475134-vmss, rg: kubetest-5p3qtork)
I1022 06:27:35.839853       1 azure_controller_common.go:242] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-018d34da-7b68-4682-b04f-4d5c9fb75314" lun 2 to node "k8s-agentpool1-22475134-vmss000000", diskMap: map[]
I1022 06:27:35.839905       1 controllerserver.go:465] Attach operation successful: volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-018d34da-7b68-4682-b04f-4d5c9fb75314" attached to node "k8s-agentpool1-22475134-vmss000000".
I1022 06:27:35.839934       1 controllerserver.go:485] attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-018d34da-7b68-4682-b04f-4d5c9fb75314" to node "k8s-agentpool1-22475134-vmss000000" successfully
I1022 06:27:35.839954       1 utils.go:102] GRPC response: {"publish_context":{"LUN":"2"}}
... skipping 4 lines ...
I1022 06:27:35.840181       1 utils.go:102] GRPC response: {"publish_context":{"LUN":"3"}}
I1022 06:27:35.840164       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:27:35.840256       1 azure_controller_common.go:242] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-81b30bc4-32d1-427d-a923-fc43cefb3a70" lun 4 to node "k8s-agentpool1-22475134-vmss000000", diskMap: map[]
I1022 06:27:35.840275       1 controllerserver.go:465] Attach operation successful: volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-81b30bc4-32d1-427d-a923-fc43cefb3a70" attached to node "k8s-agentpool1-22475134-vmss000000".
I1022 06:27:35.840292       1 controllerserver.go:485] attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-81b30bc4-32d1-427d-a923-fc43cefb3a70" to node "k8s-agentpool1-22475134-vmss000000" successfully
I1022 06:27:35.840321       1 utils.go:102] GRPC response: {"publish_context":{"LUN":"4"}}
I1022 06:27:36.344988       1 azure_controller_vmss.go:134] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-d2b2d817-66c5-404d-969d-b833e1464a67:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-d2b2d817-66c5-404d-969d-b833e1464a67  true false 1})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING)
I1022 06:27:36.345159       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000001" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:27:36.345204       1 azure_vmss.go:198] Couldn't find VMSS VM with nodeName k8s-agentpool1-22475134-vmss000001, refreshing the cache(vmss: k8s-agentpool1-22475134-vmss, rg: kubetest-5p3qtork)
I1022 06:27:36.389714       1 azure_controller_common.go:242] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-d0f0cce2-e6a9-4010-b1a2-189c3f78f4a6" lun 2 to node "k8s-agentpool1-22475134-vmss000001", diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-d0f0cce2-e6a9-4010-b1a2-189c3f78f4a6:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-d0f0cce2-e6a9-4010-b1a2-189c3f78f4a6  true false 2})]
I1022 06:27:36.389751       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000001" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:27:36.389769       1 azure_controller_vmss.go:122] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-d0f0cce2-e6a9-4010-b1a2-189c3f78f4a6:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-d0f0cce2-e6a9-4010-b1a2-189c3f78f4a6  true false 2})])
I1022 06:27:36.550924       1 azure_controller_vmss.go:134] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-d0f0cce2-e6a9-4010-b1a2-189c3f78f4a6:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-d0f0cce2-e6a9-4010-b1a2-189c3f78f4a6  true false 2})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING)
I1022 06:27:50.533161       1 controllerserver.go:465] Attach operation successful: volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-20082bbf-dc21-47a4-8a4b-e7a04b62cdac" attached to node "k8s-agentpool1-22475134-vmss000001".
I1022 06:27:50.533192       1 controllerserver.go:485] attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-20082bbf-dc21-47a4-8a4b-e7a04b62cdac" to node "k8s-agentpool1-22475134-vmss000001" successfully
I1022 06:27:50.533226       1 utils.go:102] GRPC response: {"publish_context":{"LUN":"0"}}
I1022 06:27:50.544543       1 utils.go:95] GRPC call: /csi.v1.Controller/ControllerPublishVolume
I1022 06:27:50.544564       1 utils.go:96] GRPC request: {"node_id":"k8s-agentpool1-22475134-vmss000001","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-20082bbf-dc21-47a4-8a4b-e7a04b62cdac","csi.storage.k8s.io/pvc/name":"test.csi.azure.comp2wmc","csi.storage.k8s.io/pvc/namespace":"provisioning-5043","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1634884037556-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-20082bbf-dc21-47a4-8a4b-e7a04b62cdac"}
I1022 06:27:50.556658       1 controllerserver.go:465] Attach operation successful: volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-9b8ef2c7-3d19-4ab3-80c1-fb0736e83e7d" attached to node "k8s-agentpool1-22475134-vmss000000".
... skipping 20 lines ...
I1022 06:27:51.706635       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000001" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:27:51.706692       1 azure_controller_common.go:457] azureDisk - find disk: lun 2 name "pvc-d0f0cce2-e6a9-4010-b1a2-189c3f78f4a6" uri "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-d0f0cce2-e6a9-4010-b1a2-189c3f78f4a6"
I1022 06:27:51.706703       1 controllerserver.go:455] Attach operation is successful. volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-d0f0cce2-e6a9-4010-b1a2-189c3f78f4a6" is already attached to node "k8s-agentpool1-22475134-vmss000001" at lun 2.
I1022 06:27:51.706714       1 utils.go:102] GRPC response: {"publish_context":{"LUN":"2"}}
I1022 06:28:14.085903       1 utils.go:95] GRPC call: /csi.v1.Controller/CreateVolume
I1022 06:28:14.085928       1 utils.go:96] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"requisite":[{"segments":{"topology.test.csi.azure.com/zone":""}}]},"capacity_range":{"required_bytes":5368709120},"name":"pvc-8f420b91-438e-4fc3-a9c6-b57ba1679195","parameters":{"csi.storage.k8s.io/pv/name":"pvc-8f420b91-438e-4fc3-a9c6-b57ba1679195","csi.storage.k8s.io/pvc/name":"inline-volume-tester2-rr2d2-my-volume-0","csi.storage.k8s.io/pvc/namespace":"ephemeral-8965"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]}
I1022 06:28:14.120579       1 azure_diskclient.go:135] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-8f420b91-438e-4fc3-a9c6-b57ba1679195, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-8f420b91-438e-4fc3-a9c6-b57ba1679195' under resource group 'kubetest-5p3qtork' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I1022 06:28:14.120653       1 controllerserver.go:271] begin to create azure disk(pvc-8f420b91-438e-4fc3-a9c6-b57ba1679195) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) diskZone() maxShares(0)
I1022 06:28:14.120678       1 azure_managedDiskController.go:88] azureDisk - creating new managed Name:pvc-8f420b91-438e-4fc3-a9c6-b57ba1679195 StorageAccountType:StandardSSD_LRS Size:5
I1022 06:28:16.405181       1 utils.go:95] GRPC call: /csi.v1.Controller/ControllerGetCapabilities
I1022 06:28:16.405205       1 utils.go:96] GRPC request: {}
I1022 06:28:16.405245       1 utils.go:102] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}},{"Type":{"Rpc":{"type":6}}},{"Type":{"Rpc":{"type":7}}},{"Type":{"Rpc":{"type":9}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":10}}},{"Type":{"Rpc":{"type":13}}}]}
I1022 06:28:16.406041       1 utils.go:95] GRPC call: /csi.v1.Controller/ControllerExpandVolume
I1022 06:28:16.406064       1 utils.go:96] GRPC request: {"capacity_range":{"required_bytes":6442450944},"volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-d0f0cce2-e6a9-4010-b1a2-189c3f78f4a6"}
I1022 06:28:16.429942       1 controllerserver.go:807] begin to expand azure disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-d0f0cce2-e6a9-4010-b1a2-189c3f78f4a6) with new size(6442450944)
I1022 06:28:16.461112       1 azure_managedDiskController.go:336] azureDisk - begin to resize disk(pvc-d0f0cce2-e6a9-4010-b1a2-189c3f78f4a6) with new size(6), old size({{5 0} {<nil>}  BinarySI})
E1022 06:28:16.461182       1 utils.go:100] GRPC error: rpc error: code = Internal desc = failed to resize disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-d0f0cce2-e6a9-4010-b1a2-189c3f78f4a6) with error(azureDisk - disk resize is only supported on Unattached disk, current disk state: Attached, already attached to /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool1-22475134-vmss/virtualMachines/k8s-agentpool1-22475134-vmss_1)
I1022 06:28:16.490227       1 azure_managedDiskController.go:249] azureDisk - created new MD Name:pvc-8f420b91-438e-4fc3-a9c6-b57ba1679195 StorageAccountType:StandardSSD_LRS Size:5
I1022 06:28:16.490291       1 controllerserver.go:348] create azure disk(pvc-8f420b91-438e-4fc3-a9c6-b57ba1679195) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) tags(map[kubernetes.io-created-for-pv-name:pvc-8f420b91-438e-4fc3-a9c6-b57ba1679195 kubernetes.io-created-for-pvc-name:inline-volume-tester2-rr2d2-my-volume-0 kubernetes.io-created-for-pvc-namespace:ephemeral-8965]) successfully
I1022 06:28:16.490716       1 utils.go:102] GRPC response: {"volume":{"accessible_topology":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"capacity_bytes":5368709120,"content_source":{"Type":null},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-8f420b91-438e-4fc3-a9c6-b57ba1679195","csi.storage.k8s.io/pvc/name":"inline-volume-tester2-rr2d2-my-volume-0","csi.storage.k8s.io/pvc/namespace":"ephemeral-8965","requestedsizegib":"5"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-8f420b91-438e-4fc3-a9c6-b57ba1679195"}}
I1022 06:28:17.179107       1 utils.go:95] GRPC call: /csi.v1.Controller/ControllerPublishVolume
I1022 06:28:17.179136       1 utils.go:96] GRPC request: {"node_id":"k8s-agentpool1-22475134-vmss000000","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-8f420b91-438e-4fc3-a9c6-b57ba1679195","csi.storage.k8s.io/pvc/name":"inline-volume-tester2-rr2d2-my-volume-0","csi.storage.k8s.io/pvc/namespace":"ephemeral-8965","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1634884037556-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-8f420b91-438e-4fc3-a9c6-b57ba1679195"}
I1022 06:28:17.224094       1 controllerserver.go:440] GetDiskLun returned: <nil>. Initiating attaching volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-8f420b91-438e-4fc3-a9c6-b57ba1679195" to node "k8s-agentpool1-22475134-vmss000000".
I1022 06:28:17.224135       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:28:17.224176       1 controllerserver.go:461] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-8f420b91-438e-4fc3-a9c6-b57ba1679195" to node "k8s-agentpool1-22475134-vmss000000"
I1022 06:28:17.224190       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:28:17.224201       1 azure_controller_common.go:242] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-8f420b91-438e-4fc3-a9c6-b57ba1679195" lun 5 to node "k8s-agentpool1-22475134-vmss000000", diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-8f420b91-438e-4fc3-a9c6-b57ba1679195:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-8f420b91-438e-4fc3-a9c6-b57ba1679195  true false 5})]
I1022 06:28:17.224222       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:28:17.224255       1 azure_controller_vmss.go:122] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-8f420b91-438e-4fc3-a9c6-b57ba1679195:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-8f420b91-438e-4fc3-a9c6-b57ba1679195  true false 5})])
I1022 06:28:17.401911       1 azure_controller_vmss.go:134] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-8f420b91-438e-4fc3-a9c6-b57ba1679195:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-8f420b91-438e-4fc3-a9c6-b57ba1679195  true false 5})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING)
I1022 06:28:17.468826       1 utils.go:95] GRPC call: /csi.v1.Controller/ControllerGetCapabilities
I1022 06:28:17.468847       1 utils.go:96] GRPC request: {}
I1022 06:28:17.468881       1 utils.go:102] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}},{"Type":{"Rpc":{"type":6}}},{"Type":{"Rpc":{"type":7}}},{"Type":{"Rpc":{"type":9}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":10}}},{"Type":{"Rpc":{"type":13}}}]}
I1022 06:28:17.469516       1 utils.go:95] GRPC call: /csi.v1.Controller/ControllerExpandVolume
I1022 06:28:17.469539       1 utils.go:96] GRPC request: {"capacity_range":{"required_bytes":6442450944},"volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-d0f0cce2-e6a9-4010-b1a2-189c3f78f4a6"}
I1022 06:28:17.493180       1 controllerserver.go:807] begin to expand azure disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-d0f0cce2-e6a9-4010-b1a2-189c3f78f4a6) with new size(6442450944)
I1022 06:28:17.522211       1 azure_managedDiskController.go:336] azureDisk - begin to resize disk(pvc-d0f0cce2-e6a9-4010-b1a2-189c3f78f4a6) with new size(6), old size({{5 0} {<nil>}  BinarySI})
E1022 06:28:17.522270       1 utils.go:100] GRPC error: rpc error: code = Internal desc = failed to resize disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-d0f0cce2-e6a9-4010-b1a2-189c3f78f4a6) with error(azureDisk - disk resize is only supported on Unattached disk, current disk state: Attached, already attached to /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool1-22475134-vmss/virtualMachines/k8s-agentpool1-22475134-vmss_1)
I1022 06:28:17.529535       1 utils.go:95] GRPC call: /csi.v1.Controller/ControllerGetCapabilities
I1022 06:28:17.529570       1 utils.go:96] GRPC request: {}
I1022 06:28:17.529597       1 utils.go:102] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}},{"Type":{"Rpc":{"type":6}}},{"Type":{"Rpc":{"type":7}}},{"Type":{"Rpc":{"type":9}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":10}}},{"Type":{"Rpc":{"type":13}}}]}
I1022 06:28:17.530172       1 utils.go:95] GRPC call: /csi.v1.Controller/ControllerExpandVolume
I1022 06:28:17.530194       1 utils.go:96] GRPC request: {"capacity_range":{"required_bytes":6442450944},"volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-d0f0cce2-e6a9-4010-b1a2-189c3f78f4a6"}
I1022 06:28:17.552375       1 controllerserver.go:807] begin to expand azure disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-d0f0cce2-e6a9-4010-b1a2-189c3f78f4a6) with new size(6442450944)
I1022 06:28:17.586287       1 azure_managedDiskController.go:336] azureDisk - begin to resize disk(pvc-d0f0cce2-e6a9-4010-b1a2-189c3f78f4a6) with new size(6), old size({{5 0} {<nil>}  BinarySI})
E1022 06:28:17.586323       1 utils.go:100] GRPC error: rpc error: code = Internal desc = failed to resize disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-d0f0cce2-e6a9-4010-b1a2-189c3f78f4a6) with error(azureDisk - disk resize is only supported on Unattached disk, current disk state: Attached, already attached to /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool1-22475134-vmss/virtualMachines/k8s-agentpool1-22475134-vmss_1)
I1022 06:28:18.422843       1 azure_vmss.go:319] Couldn't find VMSS VM with scaleSetName "k8s-agentpool1-22475134-vmss" and instanceID "0", refreshing the cache
I1022 06:28:19.532248       1 utils.go:95] GRPC call: /csi.v1.Controller/ControllerGetCapabilities
I1022 06:28:19.532290       1 utils.go:96] GRPC request: {}
I1022 06:28:19.532353       1 utils.go:102] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}},{"Type":{"Rpc":{"type":6}}},{"Type":{"Rpc":{"type":7}}},{"Type":{"Rpc":{"type":9}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":10}}},{"Type":{"Rpc":{"type":13}}}]}
I1022 06:28:19.532989       1 utils.go:95] GRPC call: /csi.v1.Controller/ControllerExpandVolume
I1022 06:28:19.533009       1 utils.go:96] GRPC request: {"capacity_range":{"required_bytes":6442450944},"volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-d0f0cce2-e6a9-4010-b1a2-189c3f78f4a6"}
I1022 06:28:19.557996       1 controllerserver.go:807] begin to expand azure disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-d0f0cce2-e6a9-4010-b1a2-189c3f78f4a6) with new size(6442450944)
I1022 06:28:19.588385       1 azure_managedDiskController.go:336] azureDisk - begin to resize disk(pvc-d0f0cce2-e6a9-4010-b1a2-189c3f78f4a6) with new size(6), old size({{5 0} {<nil>}  BinarySI})
E1022 06:28:19.588439       1 utils.go:100] GRPC error: rpc error: code = Internal desc = failed to resize disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-d0f0cce2-e6a9-4010-b1a2-189c3f78f4a6) with error(azureDisk - disk resize is only supported on Unattached disk, current disk state: Attached, already attached to /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool1-22475134-vmss/virtualMachines/k8s-agentpool1-22475134-vmss_1)
I1022 06:28:19.646816       1 utils.go:95] GRPC call: /csi.v1.Controller/ControllerGetCapabilities
I1022 06:28:19.646835       1 utils.go:96] GRPC request: {}
I1022 06:28:19.646863       1 utils.go:102] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}},{"Type":{"Rpc":{"type":6}}},{"Type":{"Rpc":{"type":7}}},{"Type":{"Rpc":{"type":9}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":10}}},{"Type":{"Rpc":{"type":13}}}]}
I1022 06:28:19.647286       1 utils.go:95] GRPC call: /csi.v1.Controller/ControllerExpandVolume
I1022 06:28:19.647306       1 utils.go:96] GRPC request: {"capacity_range":{"required_bytes":6442450944},"volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-d0f0cce2-e6a9-4010-b1a2-189c3f78f4a6"}
I1022 06:28:19.678928       1 controllerserver.go:807] begin to expand azure disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-d0f0cce2-e6a9-4010-b1a2-189c3f78f4a6) with new size(6442450944)
I1022 06:28:19.701915       1 azure_managedDiskController.go:336] azureDisk - begin to resize disk(pvc-d0f0cce2-e6a9-4010-b1a2-189c3f78f4a6) with new size(6), old size({{5 0} {<nil>}  BinarySI})
E1022 06:28:19.701965       1 utils.go:100] GRPC error: rpc error: code = Internal desc = failed to resize disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-d0f0cce2-e6a9-4010-b1a2-189c3f78f4a6) with error(azureDisk - disk resize is only supported on Unattached disk, current disk state: Attached, already attached to /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool1-22475134-vmss/virtualMachines/k8s-agentpool1-22475134-vmss_1)
I1022 06:28:24.059681       1 utils.go:95] GRPC call: /csi.v1.Controller/ControllerUnpublishVolume
I1022 06:28:24.059714       1 utils.go:96] GRPC request: {"node_id":"k8s-agentpool1-22475134-vmss000001","volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-d0f0cce2-e6a9-4010-b1a2-189c3f78f4a6"}
I1022 06:28:24.059852       1 controllerserver.go:524] Trying to detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-d0f0cce2-e6a9-4010-b1a2-189c3f78f4a6 from node k8s-agentpool1-22475134-vmss000001
I1022 06:28:24.059876       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000001" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:28:24.059919       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000001" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:28:24.059936       1 azure_controller_common.go:346] Trying to detach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-d0f0cce2-e6a9-4010-b1a2-189c3f78f4a6" from node "k8s-agentpool1-22475134-vmss000001", diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-d0f0cce2-e6a9-4010-b1a2-189c3f78f4a6:pvc-d0f0cce2-e6a9-4010-b1a2-189c3f78f4a6]
... skipping 26 lines ...
I1022 06:28:27.597758       1 utils.go:96] GRPC request: {}
I1022 06:28:27.597835       1 utils.go:102] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}},{"Type":{"Rpc":{"type":6}}},{"Type":{"Rpc":{"type":7}}},{"Type":{"Rpc":{"type":9}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":10}}},{"Type":{"Rpc":{"type":13}}}]}
I1022 06:28:27.598391       1 utils.go:95] GRPC call: /csi.v1.Controller/ControllerExpandVolume
I1022 06:28:27.598427       1 utils.go:96] GRPC request: {"capacity_range":{"required_bytes":6442450944},"volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-d0f0cce2-e6a9-4010-b1a2-189c3f78f4a6"}
I1022 06:28:27.632763       1 controllerserver.go:807] begin to expand azure disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-d0f0cce2-e6a9-4010-b1a2-189c3f78f4a6) with new size(6442450944)
I1022 06:28:27.655752       1 azure_managedDiskController.go:336] azureDisk - begin to resize disk(pvc-d0f0cce2-e6a9-4010-b1a2-189c3f78f4a6) with new size(6), old size({{5 0} {<nil>}  BinarySI})
E1022 06:28:27.655847       1 utils.go:100] GRPC error: rpc error: code = Internal desc = failed to resize disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-d0f0cce2-e6a9-4010-b1a2-189c3f78f4a6) with error(azureDisk - disk resize is only supported on Unattached disk, current disk state: Attached, already attached to /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool1-22475134-vmss/virtualMachines/k8s-agentpool1-22475134-vmss_1)
I1022 06:28:27.666677       1 utils.go:95] GRPC call: /csi.v1.Controller/ControllerGetCapabilities
I1022 06:28:27.666695       1 utils.go:96] GRPC request: {}
I1022 06:28:27.666728       1 utils.go:102] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}},{"Type":{"Rpc":{"type":6}}},{"Type":{"Rpc":{"type":7}}},{"Type":{"Rpc":{"type":9}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":10}}},{"Type":{"Rpc":{"type":13}}}]}
I1022 06:28:27.667395       1 utils.go:95] GRPC call: /csi.v1.Controller/ControllerExpandVolume
I1022 06:28:27.667415       1 utils.go:96] GRPC request: {"capacity_range":{"required_bytes":6442450944},"volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-d0f0cce2-e6a9-4010-b1a2-189c3f78f4a6"}
I1022 06:28:27.690203       1 controllerserver.go:807] begin to expand azure disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-d0f0cce2-e6a9-4010-b1a2-189c3f78f4a6) with new size(6442450944)
I1022 06:28:27.713814       1 azure_managedDiskController.go:336] azureDisk - begin to resize disk(pvc-d0f0cce2-e6a9-4010-b1a2-189c3f78f4a6) with new size(6), old size({{5 0} {<nil>}  BinarySI})
E1022 06:28:27.713905       1 utils.go:100] GRPC error: rpc error: code = Internal desc = failed to resize disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-d0f0cce2-e6a9-4010-b1a2-189c3f78f4a6) with error(azureDisk - disk resize is only supported on Unattached disk, current disk state: Attached, already attached to /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool1-22475134-vmss/virtualMachines/k8s-agentpool1-22475134-vmss_1)
I1022 06:28:34.154297       1 utils.go:95] GRPC call: /csi.v1.Controller/ControllerUnpublishVolume
I1022 06:28:34.154322       1 utils.go:96] GRPC request: {"node_id":"k8s-agentpool1-22475134-vmss000000","volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-9b8ef2c7-3d19-4ab3-80c1-fb0736e83e7d"}
I1022 06:28:34.154434       1 controllerserver.go:524] Trying to detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-9b8ef2c7-3d19-4ab3-80c1-fb0736e83e7d from node k8s-agentpool1-22475134-vmss000000
I1022 06:28:34.154449       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:28:34.154562       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:28:44.130319       1 utils.go:95] GRPC call: /csi.v1.Controller/ControllerUnpublishVolume
... skipping 95 lines ...
I1022 06:29:09.889628       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:29:09.889648       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:29:09.889659       1 azure_controller_common.go:242] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-d0f0cce2-e6a9-4010-b1a2-189c3f78f4a6" lun 0 to node "k8s-agentpool1-22475134-vmss000000", diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-d0f0cce2-e6a9-4010-b1a2-189c3f78f4a6:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-d0f0cce2-e6a9-4010-b1a2-189c3f78f4a6  true false 0})]
I1022 06:29:09.889677       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:29:09.889694       1 azure_controller_vmss.go:122] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-d0f0cce2-e6a9-4010-b1a2-189c3f78f4a6:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-d0f0cce2-e6a9-4010-b1a2-189c3f78f4a6  true false 0})])
I1022 06:29:09.889736       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:29:10.089381       1 azure_controller_vmss.go:134] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-d0f0cce2-e6a9-4010-b1a2-189c3f78f4a6:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-d0f0cce2-e6a9-4010-b1a2-189c3f78f4a6  true false 0})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING)
I1022 06:29:10.089580       1 azure_controller_common.go:346] Trying to detach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-018d34da-7b68-4682-b04f-4d5c9fb75314" from node "k8s-agentpool1-22475134-vmss000000", diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-018d34da-7b68-4682-b04f-4d5c9fb75314:pvc-018d34da-7b68-4682-b04f-4d5c9fb75314 /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-9b8ef2c7-3d19-4ab3-80c1-fb0736e83e7d:pvc-9b8ef2c7-3d19-4ab3-80c1-fb0736e83e7d]
I1022 06:29:10.089664       1 azure_vmss.go:198] Couldn't find VMSS VM with nodeName k8s-agentpool1-22475134-vmss000000, refreshing the cache(vmss: k8s-agentpool1-22475134-vmss, rg: kubetest-5p3qtork)
E1022 06:29:10.168861       1 azure_controller_vmss.go:183] detach azure disk on node(k8s-agentpool1-22475134-vmss000000): disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-018d34da-7b68-4682-b04f-4d5c9fb75314:pvc-018d34da-7b68-4682-b04f-4d5c9fb75314 /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-9b8ef2c7-3d19-4ab3-80c1-fb0736e83e7d:pvc-9b8ef2c7-3d19-4ab3-80c1-fb0736e83e7d]) not found
I1022 06:29:10.168890       1 azure_controller_vmss.go:213] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - detach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-018d34da-7b68-4682-b04f-4d5c9fb75314:pvc-018d34da-7b68-4682-b04f-4d5c9fb75314 /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-9b8ef2c7-3d19-4ab3-80c1-fb0736e83e7d:pvc-9b8ef2c7-3d19-4ab3-80c1-fb0736e83e7d])
I1022 06:29:17.770031       1 utils.go:95] GRPC call: /csi.v1.Controller/DeleteVolume
I1022 06:29:17.770055       1 utils.go:96] GRPC request: {"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-018d34da-7b68-4682-b04f-4d5c9fb75314"}
I1022 06:29:17.770158       1 controllerserver.go:389] deleting azure disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-018d34da-7b68-4682-b04f-4d5c9fb75314)
I1022 06:29:17.770199       1 controllerserver.go:391] delete azure disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-018d34da-7b68-4682-b04f-4d5c9fb75314) returned with failed to delete disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-018d34da-7b68-4682-b04f-4d5c9fb75314) since it's in attaching or detaching state
E1022 06:29:17.770220       1 utils.go:100] GRPC error: failed to delete disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-018d34da-7b68-4682-b04f-4d5c9fb75314) since it's in attaching or detaching state
I1022 06:29:22.720569       1 utils.go:95] GRPC call: /csi.v1.Controller/DeleteVolume
I1022 06:29:22.720595       1 utils.go:96] GRPC request: {"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-20082bbf-dc21-47a4-8a4b-e7a04b62cdac"}
I1022 06:29:22.720671       1 controllerserver.go:389] deleting azure disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-20082bbf-dc21-47a4-8a4b-e7a04b62cdac)
I1022 06:29:25.148191       1 azure_controller_vmss.go:225] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000001) - detach disk(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-d0f0cce2-e6a9-4010-b1a2-189c3f78f4a6:pvc-d0f0cce2-e6a9-4010-b1a2-189c3f78f4a6]) returned with <nil>
I1022 06:29:25.148256       1 azure_controller_common.go:369] azureDisk - detach disk(pvc-d0f0cce2-e6a9-4010-b1a2-189c3f78f4a6, /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-d0f0cce2-e6a9-4010-b1a2-189c3f78f4a6) succeeded
I1022 06:29:25.148267       1 controllerserver.go:533] detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-d0f0cce2-e6a9-4010-b1a2-189c3f78f4a6 from node k8s-agentpool1-22475134-vmss000001 successfully
... skipping 26 lines ...
I1022 06:29:27.915921       1 utils.go:102] GRPC response: {}
I1022 06:29:31.016247       1 azure_managedDiskController.go:286] azureDisk - deleted a managed disk: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-81b30bc4-32d1-427d-a923-fc43cefb3a70
I1022 06:29:31.016276       1 controllerserver.go:391] delete azure disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-81b30bc4-32d1-427d-a923-fc43cefb3a70) returned with <nil>
I1022 06:29:31.016295       1 utils.go:102] GRPC response: {}
I1022 06:29:33.467691       1 utils.go:95] GRPC call: /csi.v1.Controller/CreateVolume
I1022 06:29:33.467715       1 utils.go:96] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"requisite":[{"segments":{"topology.test.csi.azure.com/zone":""}}]},"capacity_range":{"required_bytes":5368709120},"name":"pvc-4028b709-1233-4cd4-8837-fd82aeedf83a","parameters":{"csi.storage.k8s.io/pv/name":"pvc-4028b709-1233-4cd4-8837-fd82aeedf83a","csi.storage.k8s.io/pvc/name":"test.csi.azure.comb6jzz","csi.storage.k8s.io/pvc/namespace":"snapshotting-2296"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]}
I1022 06:29:33.566706       1 azure_diskclient.go:135] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-4028b709-1233-4cd4-8837-fd82aeedf83a, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-4028b709-1233-4cd4-8837-fd82aeedf83a' under resource group 'kubetest-5p3qtork' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I1022 06:29:33.566787       1 controllerserver.go:271] begin to create azure disk(pvc-4028b709-1233-4cd4-8837-fd82aeedf83a) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) diskZone() maxShares(0)
I1022 06:29:33.566802       1 azure_managedDiskController.go:88] azureDisk - creating new managed Name:pvc-4028b709-1233-4cd4-8837-fd82aeedf83a StorageAccountType:StandardSSD_LRS Size:5
I1022 06:29:34.951449       1 utils.go:95] GRPC call: /csi.v1.Controller/DeleteVolume
I1022 06:29:34.951488       1 utils.go:96] GRPC request: {"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-9b8ef2c7-3d19-4ab3-80c1-fb0736e83e7d"}
I1022 06:29:34.951655       1 controllerserver.go:389] deleting azure disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-9b8ef2c7-3d19-4ab3-80c1-fb0736e83e7d)
I1022 06:29:35.905011       1 azure_managedDiskController.go:249] azureDisk - created new MD Name:pvc-4028b709-1233-4cd4-8837-fd82aeedf83a StorageAccountType:StandardSSD_LRS Size:5
I1022 06:29:35.905087       1 controllerserver.go:348] create azure disk(pvc-4028b709-1233-4cd4-8837-fd82aeedf83a) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) tags(map[kubernetes.io-created-for-pv-name:pvc-4028b709-1233-4cd4-8837-fd82aeedf83a kubernetes.io-created-for-pvc-name:test.csi.azure.comb6jzz kubernetes.io-created-for-pvc-namespace:snapshotting-2296]) successfully
I1022 06:29:35.905110       1 utils.go:102] GRPC response: {"volume":{"accessible_topology":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"capacity_bytes":5368709120,"content_source":{"Type":null},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-4028b709-1233-4cd4-8837-fd82aeedf83a","csi.storage.k8s.io/pvc/name":"test.csi.azure.comb6jzz","csi.storage.k8s.io/pvc/namespace":"snapshotting-2296","requestedsizegib":"5"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-4028b709-1233-4cd4-8837-fd82aeedf83a"}}
I1022 06:29:37.447719       1 utils.go:95] GRPC call: /csi.v1.Controller/CreateVolume
I1022 06:29:37.447747       1 utils.go:96] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"requisite":[{"segments":{"topology.test.csi.azure.com/zone":""}}]},"capacity_range":{"required_bytes":5368709120},"name":"pvc-3470e515-3208-4ae7-8a58-48ebecf49398","parameters":{"csi.storage.k8s.io/pv/name":"pvc-3470e515-3208-4ae7-8a58-48ebecf49398","csi.storage.k8s.io/pvc/name":"test.csi.azure.com6wbxd","csi.storage.k8s.io/pvc/namespace":"fsgroupchangepolicy-1195"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]}
I1022 06:29:37.476091       1 azure_diskclient.go:135] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-3470e515-3208-4ae7-8a58-48ebecf49398, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-3470e515-3208-4ae7-8a58-48ebecf49398' under resource group 'kubetest-5p3qtork' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I1022 06:29:37.476165       1 controllerserver.go:271] begin to create azure disk(pvc-3470e515-3208-4ae7-8a58-48ebecf49398) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) diskZone() maxShares(0)
I1022 06:29:37.476177       1 azure_managedDiskController.go:88] azureDisk - creating new managed Name:pvc-3470e515-3208-4ae7-8a58-48ebecf49398 StorageAccountType:StandardSSD_LRS Size:5
I1022 06:29:38.279002       1 utils.go:95] GRPC call: /csi.v1.Controller/ControllerPublishVolume
I1022 06:29:38.279027       1 utils.go:96] GRPC request: {"node_id":"k8s-agentpool1-22475134-vmss000000","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-4028b709-1233-4cd4-8837-fd82aeedf83a","csi.storage.k8s.io/pvc/name":"test.csi.azure.comb6jzz","csi.storage.k8s.io/pvc/namespace":"snapshotting-2296","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1634884037556-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-4028b709-1233-4cd4-8837-fd82aeedf83a"}
I1022 06:29:38.318953       1 controllerserver.go:440] GetDiskLun returned: <nil>. Initiating attaching volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-4028b709-1233-4cd4-8837-fd82aeedf83a" to node "k8s-agentpool1-22475134-vmss000000".
I1022 06:29:38.318978       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:29:38.319001       1 controllerserver.go:461] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-4028b709-1233-4cd4-8837-fd82aeedf83a" to node "k8s-agentpool1-22475134-vmss000000"
I1022 06:29:38.319014       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:29:38.319050       1 azure_controller_common.go:242] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-4028b709-1233-4cd4-8837-fd82aeedf83a" lun 1 to node "k8s-agentpool1-22475134-vmss000000", diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-4028b709-1233-4cd4-8837-fd82aeedf83a:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-4028b709-1233-4cd4-8837-fd82aeedf83a  true false 1})]
I1022 06:29:38.319105       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:29:38.319120       1 azure_controller_vmss.go:122] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-4028b709-1233-4cd4-8837-fd82aeedf83a:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-4028b709-1233-4cd4-8837-fd82aeedf83a  true false 1})])
I1022 06:29:38.472550       1 azure_controller_vmss.go:134] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-4028b709-1233-4cd4-8837-fd82aeedf83a:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-4028b709-1233-4cd4-8837-fd82aeedf83a  true false 1})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING)
I1022 06:29:39.434876       1 utils.go:95] GRPC call: /csi.v1.Controller/DeleteVolume
I1022 06:29:39.434904       1 utils.go:96] GRPC request: {"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-545e7003-a379-4bb4-a461-d761add02e77"}
I1022 06:29:39.435011       1 controllerserver.go:389] deleting azure disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-545e7003-a379-4bb4-a461-d761add02e77)
I1022 06:29:39.971650       1 azure_managedDiskController.go:249] azureDisk - created new MD Name:pvc-3470e515-3208-4ae7-8a58-48ebecf49398 StorageAccountType:StandardSSD_LRS Size:5
I1022 06:29:39.971704       1 controllerserver.go:348] create azure disk(pvc-3470e515-3208-4ae7-8a58-48ebecf49398) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) tags(map[kubernetes.io-created-for-pv-name:pvc-3470e515-3208-4ae7-8a58-48ebecf49398 kubernetes.io-created-for-pvc-name:test.csi.azure.com6wbxd kubernetes.io-created-for-pvc-namespace:fsgroupchangepolicy-1195]) successfully
I1022 06:29:39.971728       1 utils.go:102] GRPC response: {"volume":{"accessible_topology":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"capacity_bytes":5368709120,"content_source":{"Type":null},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-3470e515-3208-4ae7-8a58-48ebecf49398","csi.storage.k8s.io/pvc/name":"test.csi.azure.com6wbxd","csi.storage.k8s.io/pvc/namespace":"fsgroupchangepolicy-1195","requestedsizegib":"5"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-3470e515-3208-4ae7-8a58-48ebecf49398"}}
... skipping 7 lines ...
I1022 06:29:42.232522       1 azure_vmss.go:198] Couldn't find VMSS VM with nodeName k8s-agentpool1-22475134-vmss000000, refreshing the cache(vmss: k8s-agentpool1-22475134-vmss, rg: kubetest-5p3qtork)
I1022 06:29:42.296831       1 controllerserver.go:461] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-3470e515-3208-4ae7-8a58-48ebecf49398" to node "k8s-agentpool1-22475134-vmss000000"
I1022 06:29:42.296872       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:29:42.296893       1 azure_controller_common.go:242] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-3470e515-3208-4ae7-8a58-48ebecf49398" lun 2 to node "k8s-agentpool1-22475134-vmss000000", diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-3470e515-3208-4ae7-8a58-48ebecf49398:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-3470e515-3208-4ae7-8a58-48ebecf49398  true false 2})]
I1022 06:29:42.296910       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:29:42.296956       1 azure_controller_vmss.go:122] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-3470e515-3208-4ae7-8a58-48ebecf49398:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-3470e515-3208-4ae7-8a58-48ebecf49398  true false 2})])
I1022 06:29:42.472510       1 azure_controller_vmss.go:134] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-3470e515-3208-4ae7-8a58-48ebecf49398:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-3470e515-3208-4ae7-8a58-48ebecf49398  true false 2})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING)
I1022 06:29:44.632327       1 utils.go:95] GRPC call: /csi.v1.Controller/ControllerUnpublishVolume
I1022 06:29:44.632358       1 utils.go:96] GRPC request: {"node_id":"k8s-agentpool1-22475134-vmss000000","volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-d0f0cce2-e6a9-4010-b1a2-189c3f78f4a6"}
I1022 06:29:44.632464       1 controllerserver.go:524] Trying to detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-d0f0cce2-e6a9-4010-b1a2-189c3f78f4a6 from node k8s-agentpool1-22475134-vmss000000
I1022 06:29:44.632475       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:29:44.632503       1 azure_vmss.go:198] Couldn't find VMSS VM with nodeName k8s-agentpool1-22475134-vmss000000, refreshing the cache(vmss: k8s-agentpool1-22475134-vmss, rg: kubetest-5p3qtork)
I1022 06:29:44.654725       1 azure_managedDiskController.go:286] azureDisk - deleted a managed disk: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-545e7003-a379-4bb4-a461-d761add02e77
... skipping 2 lines ...
I1022 06:29:44.686440       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:29:44.686494       1 azure_controller_common.go:346] Trying to detach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-d0f0cce2-e6a9-4010-b1a2-189c3f78f4a6" from node "k8s-agentpool1-22475134-vmss000000", diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-d0f0cce2-e6a9-4010-b1a2-189c3f78f4a6:pvc-d0f0cce2-e6a9-4010-b1a2-189c3f78f4a6]
I1022 06:29:44.686560       1 azure_controller_vmss.go:174] azureDisk - detach disk: name "pvc-d0f0cce2-e6a9-4010-b1a2-189c3f78f4a6" uri "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-d0f0cce2-e6a9-4010-b1a2-189c3f78f4a6"
I1022 06:29:44.686603       1 azure_controller_vmss.go:213] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - detach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-d0f0cce2-e6a9-4010-b1a2-189c3f78f4a6:pvc-d0f0cce2-e6a9-4010-b1a2-189c3f78f4a6])
I1022 06:29:45.669557       1 utils.go:95] GRPC call: /csi.v1.Controller/CreateVolume
I1022 06:29:45.669585       1 utils.go:96] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"requisite":[{"segments":{"topology.test.csi.azure.com/zone":""}}]},"capacity_range":{"required_bytes":5368709120},"name":"pvc-f3e47329-1be6-465a-8415-51a95da5fb5b","parameters":{"csi.storage.k8s.io/pv/name":"pvc-f3e47329-1be6-465a-8415-51a95da5fb5b","csi.storage.k8s.io/pvc/name":"test.csi.azure.comvx7qt","csi.storage.k8s.io/pvc/namespace":"fsgroupchangepolicy-8367"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]}
I1022 06:29:45.731620       1 azure_diskclient.go:135] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-f3e47329-1be6-465a-8415-51a95da5fb5b, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-f3e47329-1be6-465a-8415-51a95da5fb5b' under resource group 'kubetest-5p3qtork' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I1022 06:29:45.731706       1 controllerserver.go:271] begin to create azure disk(pvc-f3e47329-1be6-465a-8415-51a95da5fb5b) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) diskZone() maxShares(0)
I1022 06:29:45.731722       1 azure_managedDiskController.go:88] azureDisk - creating new managed Name:pvc-f3e47329-1be6-465a-8415-51a95da5fb5b StorageAccountType:StandardSSD_LRS Size:5
I1022 06:29:48.090557       1 azure_managedDiskController.go:249] azureDisk - created new MD Name:pvc-f3e47329-1be6-465a-8415-51a95da5fb5b StorageAccountType:StandardSSD_LRS Size:5
I1022 06:29:48.090610       1 controllerserver.go:348] create azure disk(pvc-f3e47329-1be6-465a-8415-51a95da5fb5b) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) tags(map[kubernetes.io-created-for-pv-name:pvc-f3e47329-1be6-465a-8415-51a95da5fb5b kubernetes.io-created-for-pvc-name:test.csi.azure.comvx7qt kubernetes.io-created-for-pvc-namespace:fsgroupchangepolicy-8367]) successfully
I1022 06:29:48.090632       1 utils.go:102] GRPC response: {"volume":{"accessible_topology":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"capacity_bytes":5368709120,"content_source":{"Type":null},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-f3e47329-1be6-465a-8415-51a95da5fb5b","csi.storage.k8s.io/pvc/name":"test.csi.azure.comvx7qt","csi.storage.k8s.io/pvc/namespace":"fsgroupchangepolicy-8367","requestedsizegib":"5"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-f3e47329-1be6-465a-8415-51a95da5fb5b"}}
I1022 06:29:49.321165       1 utils.go:95] GRPC call: /csi.v1.Controller/CreateVolume
I1022 06:29:49.321193       1 utils.go:96] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"requisite":[{"segments":{"topology.test.csi.azure.com/zone":""}}]},"capacity_range":{"required_bytes":5368709120},"name":"pvc-bc294ea0-3146-4351-837c-8dba4447e4d2","parameters":{"csi.storage.k8s.io/pv/name":"pvc-bc294ea0-3146-4351-837c-8dba4447e4d2","csi.storage.k8s.io/pvc/name":"test.csi.azure.com2mwrs","csi.storage.k8s.io/pvc/namespace":"volume-expand-1548"},"volume_capabilities":[{"AccessType":{"Block":{}},"access_mode":{"mode":7}}]}
I1022 06:29:49.340577       1 azure_diskclient.go:135] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-bc294ea0-3146-4351-837c-8dba4447e4d2, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-bc294ea0-3146-4351-837c-8dba4447e4d2' under resource group 'kubetest-5p3qtork' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I1022 06:29:49.340647       1 controllerserver.go:271] begin to create azure disk(pvc-bc294ea0-3146-4351-837c-8dba4447e4d2) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) diskZone() maxShares(0)
I1022 06:29:49.340660       1 azure_managedDiskController.go:88] azureDisk - creating new managed Name:pvc-bc294ea0-3146-4351-837c-8dba4447e4d2 StorageAccountType:StandardSSD_LRS Size:5
I1022 06:29:50.391887       1 utils.go:95] GRPC call: /csi.v1.Controller/ControllerPublishVolume
I1022 06:29:50.391934       1 utils.go:96] GRPC request: {"node_id":"k8s-agentpool1-22475134-vmss000000","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-f3e47329-1be6-465a-8415-51a95da5fb5b","csi.storage.k8s.io/pvc/name":"test.csi.azure.comvx7qt","csi.storage.k8s.io/pvc/namespace":"fsgroupchangepolicy-8367","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1634884037556-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-f3e47329-1be6-465a-8415-51a95da5fb5b"}
I1022 06:29:50.419939       1 controllerserver.go:440] GetDiskLun returned: <nil>. Initiating attaching volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-f3e47329-1be6-465a-8415-51a95da5fb5b" to node "k8s-agentpool1-22475134-vmss000000".
I1022 06:29:50.419983       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
... skipping 7 lines ...
I1022 06:29:54.033167       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000001" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:29:54.033206       1 controllerserver.go:461] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-bc294ea0-3146-4351-837c-8dba4447e4d2" to node "k8s-agentpool1-22475134-vmss000001"
I1022 06:29:54.033221       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000001" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:29:54.033250       1 azure_controller_common.go:242] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-bc294ea0-3146-4351-837c-8dba4447e4d2" lun 0 to node "k8s-agentpool1-22475134-vmss000001", diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-bc294ea0-3146-4351-837c-8dba4447e4d2:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-bc294ea0-3146-4351-837c-8dba4447e4d2  true false 0})]
I1022 06:29:54.033325       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000001" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:29:54.033340       1 azure_controller_vmss.go:122] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-bc294ea0-3146-4351-837c-8dba4447e4d2:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-bc294ea0-3146-4351-837c-8dba4447e4d2  true false 0})])
I1022 06:29:54.201891       1 azure_controller_vmss.go:134] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-bc294ea0-3146-4351-837c-8dba4447e4d2:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-bc294ea0-3146-4351-837c-8dba4447e4d2  true false 0})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING)
I1022 06:30:04.692726       1 utils.go:95] GRPC call: /csi.v1.Controller/ControllerUnpublishVolume
I1022 06:30:04.692753       1 utils.go:96] GRPC request: {"node_id":"k8s-agentpool1-22475134-vmss000000","volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-8f420b91-438e-4fc3-a9c6-b57ba1679195"}
I1022 06:30:04.692851       1 controllerserver.go:524] Trying to detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-8f420b91-438e-4fc3-a9c6-b57ba1679195 from node k8s-agentpool1-22475134-vmss000000
I1022 06:30:04.692870       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:30:04.692910       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:30:17.825325       1 controllerserver.go:465] Attach operation successful: volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-3470e515-3208-4ae7-8a58-48ebecf49398" attached to node "k8s-agentpool1-22475134-vmss000000".
... skipping 31 lines ...
I1022 06:30:20.024262       1 controllerserver.go:524] Trying to detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-d0f0cce2-e6a9-4010-b1a2-189c3f78f4a6 from node k8s-agentpool1-22475134-vmss000000
I1022 06:30:20.024277       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:30:20.134604       1 azure_controller_common.go:242] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-f3e47329-1be6-465a-8415-51a95da5fb5b" lun 0 to node "k8s-agentpool1-22475134-vmss000000", diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-f3e47329-1be6-465a-8415-51a95da5fb5b:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-f3e47329-1be6-465a-8415-51a95da5fb5b  true false 0})]
I1022 06:30:20.134639       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:30:20.134697       1 azure_controller_vmss.go:122] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-f3e47329-1be6-465a-8415-51a95da5fb5b:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-f3e47329-1be6-465a-8415-51a95da5fb5b  true false 0})])
I1022 06:30:20.134706       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:30:20.333601       1 azure_controller_vmss.go:134] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-f3e47329-1be6-465a-8415-51a95da5fb5b:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-f3e47329-1be6-465a-8415-51a95da5fb5b  true false 0})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING)
I1022 06:30:20.333802       1 azure_controller_common.go:346] Trying to detach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-8f420b91-438e-4fc3-a9c6-b57ba1679195" from node "k8s-agentpool1-22475134-vmss000000", diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-8f420b91-438e-4fc3-a9c6-b57ba1679195:pvc-8f420b91-438e-4fc3-a9c6-b57ba1679195 /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-d0f0cce2-e6a9-4010-b1a2-189c3f78f4a6:pvc-d0f0cce2-e6a9-4010-b1a2-189c3f78f4a6]
I1022 06:30:20.333858       1 azure_vmss.go:198] Couldn't find VMSS VM with nodeName k8s-agentpool1-22475134-vmss000000, refreshing the cache(vmss: k8s-agentpool1-22475134-vmss, rg: kubetest-5p3qtork)
I1022 06:30:20.378872       1 azure_controller_vmss.go:174] azureDisk - detach disk: name "pvc-8f420b91-438e-4fc3-a9c6-b57ba1679195" uri "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-8f420b91-438e-4fc3-a9c6-b57ba1679195"
I1022 06:30:20.378913       1 azure_controller_vmss.go:213] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - detach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-8f420b91-438e-4fc3-a9c6-b57ba1679195:pvc-8f420b91-438e-4fc3-a9c6-b57ba1679195 /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-d0f0cce2-e6a9-4010-b1a2-189c3f78f4a6:pvc-d0f0cce2-e6a9-4010-b1a2-189c3f78f4a6])
I1022 06:30:21.771559       1 utils.go:95] GRPC call: /csi.v1.Controller/DeleteVolume
I1022 06:30:21.771586       1 utils.go:96] GRPC request: {"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-018d34da-7b68-4682-b04f-4d5c9fb75314"}
I1022 06:30:21.771680       1 controllerserver.go:389] deleting azure disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-018d34da-7b68-4682-b04f-4d5c9fb75314)
I1022 06:30:22.031066       1 azure_managedDiskController.go:286] azureDisk - deleted a managed disk: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-018d34da-7b68-4682-b04f-4d5c9fb75314
I1022 06:30:22.031091       1 controllerserver.go:391] delete azure disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-018d34da-7b68-4682-b04f-4d5c9fb75314) returned with <nil>
I1022 06:30:22.031124       1 utils.go:102] GRPC response: {}
I1022 06:30:23.981289       1 utils.go:95] GRPC call: /csi.v1.Controller/CreateVolume
I1022 06:30:23.981315       1 utils.go:96] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"requisite":[{"segments":{"topology.test.csi.azure.com/zone":""}}]},"capacity_range":{"required_bytes":5368709120},"name":"pvc-a8b788c1-65b1-446d-9813-2274459c86a0","parameters":{"csi.storage.k8s.io/pv/name":"pvc-a8b788c1-65b1-446d-9813-2274459c86a0","csi.storage.k8s.io/pvc/name":"test.csi.azure.comncjl2","csi.storage.k8s.io/pvc/namespace":"provisioning-4223"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]}
I1022 06:30:24.002011       1 azure_diskclient.go:135] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-a8b788c1-65b1-446d-9813-2274459c86a0, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-a8b788c1-65b1-446d-9813-2274459c86a0' under resource group 'kubetest-5p3qtork' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I1022 06:30:24.002086       1 controllerserver.go:271] begin to create azure disk(pvc-a8b788c1-65b1-446d-9813-2274459c86a0) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) diskZone() maxShares(0)
I1022 06:30:24.002117       1 azure_managedDiskController.go:88] azureDisk - creating new managed Name:pvc-a8b788c1-65b1-446d-9813-2274459c86a0 StorageAccountType:StandardSSD_LRS Size:5
I1022 06:30:26.615765       1 azure_managedDiskController.go:249] azureDisk - created new MD Name:pvc-a8b788c1-65b1-446d-9813-2274459c86a0 StorageAccountType:StandardSSD_LRS Size:5
I1022 06:30:26.615816       1 controllerserver.go:348] create azure disk(pvc-a8b788c1-65b1-446d-9813-2274459c86a0) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) tags(map[kubernetes.io-created-for-pv-name:pvc-a8b788c1-65b1-446d-9813-2274459c86a0 kubernetes.io-created-for-pvc-name:test.csi.azure.comncjl2 kubernetes.io-created-for-pvc-namespace:provisioning-4223]) successfully
I1022 06:30:26.615853       1 utils.go:102] GRPC response: {"volume":{"accessible_topology":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"capacity_bytes":5368709120,"content_source":{"Type":null},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-a8b788c1-65b1-446d-9813-2274459c86a0","csi.storage.k8s.io/pvc/name":"test.csi.azure.comncjl2","csi.storage.k8s.io/pvc/namespace":"provisioning-4223","requestedsizegib":"5"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-a8b788c1-65b1-446d-9813-2274459c86a0"}}
I1022 06:30:28.715822       1 utils.go:95] GRPC call: /csi.v1.Controller/ControllerPublishVolume
... skipping 2 lines ...
I1022 06:30:28.741395       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000001" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:30:28.741436       1 controllerserver.go:461] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-a8b788c1-65b1-446d-9813-2274459c86a0" to node "k8s-agentpool1-22475134-vmss000001"
I1022 06:30:28.741498       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000001" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:30:28.741527       1 azure_controller_common.go:242] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-a8b788c1-65b1-446d-9813-2274459c86a0" lun 2 to node "k8s-agentpool1-22475134-vmss000001", diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-a8b788c1-65b1-446d-9813-2274459c86a0:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-a8b788c1-65b1-446d-9813-2274459c86a0  true false 2})]
I1022 06:30:28.741550       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000001" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:30:28.741568       1 azure_controller_vmss.go:122] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-a8b788c1-65b1-446d-9813-2274459c86a0:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-a8b788c1-65b1-446d-9813-2274459c86a0  true false 2})])
I1022 06:30:28.949371       1 azure_controller_vmss.go:134] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-a8b788c1-65b1-446d-9813-2274459c86a0:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-a8b788c1-65b1-446d-9813-2274459c86a0  true false 2})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING)
I1022 06:30:32.857161       1 utils.go:95] GRPC call: /csi.v1.Controller/ControllerGetCapabilities
I1022 06:30:32.857192       1 utils.go:96] GRPC request: {}
I1022 06:30:32.857238       1 utils.go:102] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}},{"Type":{"Rpc":{"type":6}}},{"Type":{"Rpc":{"type":7}}},{"Type":{"Rpc":{"type":9}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":10}}},{"Type":{"Rpc":{"type":13}}}]}
I1022 06:30:32.857920       1 utils.go:95] GRPC call: /csi.v1.Controller/ControllerExpandVolume
I1022 06:30:32.857960       1 utils.go:96] GRPC request: {"capacity_range":{"required_bytes":6442450944},"volume_capability":{"AccessType":{"Block":{}},"access_mode":{"mode":7}},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-bc294ea0-3146-4351-837c-8dba4447e4d2"}
I1022 06:30:32.889971       1 controllerserver.go:807] begin to expand azure disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-bc294ea0-3146-4351-837c-8dba4447e4d2) with new size(6442450944)
I1022 06:30:32.939339       1 azure_managedDiskController.go:336] azureDisk - begin to resize disk(pvc-bc294ea0-3146-4351-837c-8dba4447e4d2) with new size(6), old size({{5 0} {<nil>}  BinarySI})
E1022 06:30:32.939424       1 utils.go:100] GRPC error: rpc error: code = Internal desc = failed to resize disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-bc294ea0-3146-4351-837c-8dba4447e4d2) with error(azureDisk - disk resize is only supported on Unattached disk, current disk state: Attached, already attached to /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool1-22475134-vmss/virtualMachines/k8s-agentpool1-22475134-vmss_1)
I1022 06:30:33.949545       1 utils.go:95] GRPC call: /csi.v1.Controller/ControllerGetCapabilities
I1022 06:30:33.949573       1 utils.go:96] GRPC request: {}
I1022 06:30:33.949611       1 utils.go:102] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}},{"Type":{"Rpc":{"type":6}}},{"Type":{"Rpc":{"type":7}}},{"Type":{"Rpc":{"type":9}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":10}}},{"Type":{"Rpc":{"type":13}}}]}
I1022 06:30:33.950210       1 utils.go:95] GRPC call: /csi.v1.Controller/ControllerExpandVolume
I1022 06:30:33.950230       1 utils.go:96] GRPC request: {"capacity_range":{"required_bytes":6442450944},"volume_capability":{"AccessType":{"Block":{}},"access_mode":{"mode":7}},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-bc294ea0-3146-4351-837c-8dba4447e4d2"}
I1022 06:30:33.973626       1 controllerserver.go:807] begin to expand azure disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-bc294ea0-3146-4351-837c-8dba4447e4d2) with new size(6442450944)
I1022 06:30:33.999875       1 azure_managedDiskController.go:336] azureDisk - begin to resize disk(pvc-bc294ea0-3146-4351-837c-8dba4447e4d2) with new size(6), old size({{5 0} {<nil>}  BinarySI})
E1022 06:30:33.999932       1 utils.go:100] GRPC error: rpc error: code = Internal desc = failed to resize disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-bc294ea0-3146-4351-837c-8dba4447e4d2) with error(azureDisk - disk resize is only supported on Unattached disk, current disk state: Attached, already attached to /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool1-22475134-vmss/virtualMachines/k8s-agentpool1-22475134-vmss_1)
I1022 06:30:34.017096       1 utils.go:95] GRPC call: /csi.v1.Controller/ControllerGetCapabilities
I1022 06:30:34.017117       1 utils.go:96] GRPC request: {}
I1022 06:30:34.017142       1 utils.go:102] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}},{"Type":{"Rpc":{"type":6}}},{"Type":{"Rpc":{"type":7}}},{"Type":{"Rpc":{"type":9}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":10}}},{"Type":{"Rpc":{"type":13}}}]}
I1022 06:30:34.017561       1 utils.go:95] GRPC call: /csi.v1.Controller/ControllerExpandVolume
I1022 06:30:34.017577       1 utils.go:96] GRPC request: {"capacity_range":{"required_bytes":6442450944},"volume_capability":{"AccessType":{"Block":{}},"access_mode":{"mode":7}},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-bc294ea0-3146-4351-837c-8dba4447e4d2"}
I1022 06:30:34.047768       1 controllerserver.go:807] begin to expand azure disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-bc294ea0-3146-4351-837c-8dba4447e4d2) with new size(6442450944)
I1022 06:30:34.073386       1 azure_managedDiskController.go:336] azureDisk - begin to resize disk(pvc-bc294ea0-3146-4351-837c-8dba4447e4d2) with new size(6), old size({{5 0} {<nil>}  BinarySI})
E1022 06:30:34.073438       1 utils.go:100] GRPC error: rpc error: code = Internal desc = failed to resize disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-bc294ea0-3146-4351-837c-8dba4447e4d2) with error(azureDisk - disk resize is only supported on Unattached disk, current disk state: Attached, already attached to /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool1-22475134-vmss/virtualMachines/k8s-agentpool1-22475134-vmss_1)
I1022 06:30:34.963008       1 utils.go:95] GRPC call: /csi.v1.Controller/ControllerUnpublishVolume
I1022 06:30:34.963034       1 utils.go:96] GRPC request: {"node_id":"k8s-agentpool1-22475134-vmss000001","volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-bc294ea0-3146-4351-837c-8dba4447e4d2"}
I1022 06:30:34.963131       1 controllerserver.go:524] Trying to detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-bc294ea0-3146-4351-837c-8dba4447e4d2 from node k8s-agentpool1-22475134-vmss000001
I1022 06:30:34.963145       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000001" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:30:34.963170       1 azure_vmss.go:198] Couldn't find VMSS VM with nodeName k8s-agentpool1-22475134-vmss000001, refreshing the cache(vmss: k8s-agentpool1-22475134-vmss, rg: kubetest-5p3qtork)
I1022 06:30:35.018747       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000001" is managed by VMSS since DisableAvailabilitySetNodes is set to true
... skipping 4 lines ...
I1022 06:30:36.011281       1 utils.go:96] GRPC request: {}
I1022 06:30:36.011323       1 utils.go:102] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}},{"Type":{"Rpc":{"type":6}}},{"Type":{"Rpc":{"type":7}}},{"Type":{"Rpc":{"type":9}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":10}}},{"Type":{"Rpc":{"type":13}}}]}
I1022 06:30:36.011863       1 utils.go:95] GRPC call: /csi.v1.Controller/ControllerExpandVolume
I1022 06:30:36.011879       1 utils.go:96] GRPC request: {"capacity_range":{"required_bytes":6442450944},"volume_capability":{"AccessType":{"Block":{}},"access_mode":{"mode":7}},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-bc294ea0-3146-4351-837c-8dba4447e4d2"}
I1022 06:30:36.035097       1 controllerserver.go:807] begin to expand azure disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-bc294ea0-3146-4351-837c-8dba4447e4d2) with new size(6442450944)
I1022 06:30:36.061201       1 azure_managedDiskController.go:336] azureDisk - begin to resize disk(pvc-bc294ea0-3146-4351-837c-8dba4447e4d2) with new size(6), old size({{5 0} {<nil>}  BinarySI})
E1022 06:30:36.061245       1 utils.go:100] GRPC error: rpc error: code = Internal desc = failed to resize disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-bc294ea0-3146-4351-837c-8dba4447e4d2) with error(azureDisk - disk resize is only supported on Unattached disk, current disk state: Attached, already attached to /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool1-22475134-vmss/virtualMachines/k8s-agentpool1-22475134-vmss_1)
I1022 06:30:44.078724       1 utils.go:95] GRPC call: /csi.v1.Controller/ControllerGetCapabilities
I1022 06:30:44.078750       1 utils.go:96] GRPC request: {}
I1022 06:30:44.078791       1 utils.go:102] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}},{"Type":{"Rpc":{"type":6}}},{"Type":{"Rpc":{"type":7}}},{"Type":{"Rpc":{"type":9}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":10}}},{"Type":{"Rpc":{"type":13}}}]}
I1022 06:30:44.079374       1 utils.go:95] GRPC call: /csi.v1.Controller/ControllerExpandVolume
I1022 06:30:44.079391       1 utils.go:96] GRPC request: {"capacity_range":{"required_bytes":6442450944},"volume_capability":{"AccessType":{"Block":{}},"access_mode":{"mode":7}},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-bc294ea0-3146-4351-837c-8dba4447e4d2"}
I1022 06:30:44.112286       1 controllerserver.go:807] begin to expand azure disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-bc294ea0-3146-4351-837c-8dba4447e4d2) with new size(6442450944)
I1022 06:30:44.158309       1 azure_managedDiskController.go:336] azureDisk - begin to resize disk(pvc-bc294ea0-3146-4351-837c-8dba4447e4d2) with new size(6), old size({{5 0} {<nil>}  BinarySI})
E1022 06:30:44.158362       1 utils.go:100] GRPC error: rpc error: code = Internal desc = failed to resize disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-bc294ea0-3146-4351-837c-8dba4447e4d2) with error(azureDisk - disk resize is only supported on Unattached disk, current disk state: Attached, already attached to /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool1-22475134-vmss/virtualMachines/k8s-agentpool1-22475134-vmss_1)
I1022 06:30:44.165577       1 utils.go:95] GRPC call: /csi.v1.Controller/ControllerGetCapabilities
I1022 06:30:44.165602       1 utils.go:96] GRPC request: {}
I1022 06:30:44.165636       1 utils.go:102] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}},{"Type":{"Rpc":{"type":6}}},{"Type":{"Rpc":{"type":7}}},{"Type":{"Rpc":{"type":9}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":10}}},{"Type":{"Rpc":{"type":13}}}]}
I1022 06:30:44.166207       1 utils.go:95] GRPC call: /csi.v1.Controller/ControllerExpandVolume
I1022 06:30:44.166227       1 utils.go:96] GRPC request: {"capacity_range":{"required_bytes":6442450944},"volume_capability":{"AccessType":{"Block":{}},"access_mode":{"mode":7}},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-bc294ea0-3146-4351-837c-8dba4447e4d2"}
I1022 06:30:44.223798       1 controllerserver.go:807] begin to expand azure disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-bc294ea0-3146-4351-837c-8dba4447e4d2) with new size(6442450944)
I1022 06:30:44.248538       1 azure_managedDiskController.go:336] azureDisk - begin to resize disk(pvc-bc294ea0-3146-4351-837c-8dba4447e4d2) with new size(6), old size({{5 0} {<nil>}  BinarySI})
E1022 06:30:44.248609       1 utils.go:100] GRPC error: rpc error: code = Internal desc = failed to resize disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-bc294ea0-3146-4351-837c-8dba4447e4d2) with error(azureDisk - disk resize is only supported on Unattached disk, current disk state: Attached, already attached to /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool1-22475134-vmss/virtualMachines/k8s-agentpool1-22475134-vmss_1)
I1022 06:30:45.055151       1 utils.go:95] GRPC call: /csi.v1.Controller/DeleteVolume
I1022 06:30:45.055293       1 utils.go:96] GRPC request: {"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-d0f0cce2-e6a9-4010-b1a2-189c3f78f4a6"}
I1022 06:30:45.055424       1 controllerserver.go:389] deleting azure disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-d0f0cce2-e6a9-4010-b1a2-189c3f78f4a6)
I1022 06:30:55.121937       1 utils.go:95] GRPC call: /csi.v1.Controller/ControllerUnpublishVolume
I1022 06:30:55.121960       1 utils.go:96] GRPC request: {"node_id":"k8s-agentpool1-22475134-vmss000000","volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-4028b709-1233-4cd4-8837-fd82aeedf83a"}
I1022 06:30:55.122064       1 controllerserver.go:524] Trying to detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-4028b709-1233-4cd4-8837-fd82aeedf83a from node k8s-agentpool1-22475134-vmss000000
... skipping 4 lines ...
I1022 06:30:55.821143       1 utils.go:102] GRPC response: {"name":"test.csi.azure.com","vendor_version":"v1.9.0-716f56a44514b799d2d6c6922145f22026e7d4e2"}
I1022 06:30:55.821636       1 utils.go:95] GRPC call: /csi.v1.Controller/CreateSnapshot
I1022 06:30:55.821657       1 utils.go:96] GRPC request: {"name":"snapshot-0c714dcf-a300-494b-bca3-761d95ce0ca6","source_volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-4028b709-1233-4cd4-8837-fd82aeedf83a"}
I1022 06:30:55.821786       1 controllerserver.go:909] begin to create snapshot(snapshot-0c714dcf-a300-494b-bca3-761d95ce0ca6, incremental: true) under rg(kubetest-5p3qtork)
I1022 06:30:58.194385       1 controllerserver.go:918] create snapshot(snapshot-0c714dcf-a300-494b-bca3-761d95ce0ca6) under rg(kubetest-5p3qtork) successfully
I1022 06:30:58.217933       1 utils.go:102] GRPC response: {"snapshot":{"creation_time":{"nanos":38028200,"seconds":1634884256},"ready_to_use":true,"size_bytes":5368709120,"snapshot_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/snapshots/snapshot-0c714dcf-a300-494b-bca3-761d95ce0ca6","source_volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-4028b709-1233-4cd4-8837-fd82aeedf83a"}}
I1022 06:31:00.055920       1 azure_armclient.go:304] Received error in WaitForCompletionRef: 'Future#WaitForCompletion: context has been cancelled: StatusCode=200 -- Original Error: context canceled'
I1022 06:31:00.055961       1 azure_armclient.go:638] Received error in delete.wait: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-d0f0cce2-e6a9-4010-b1a2-189c3f78f4a6, error: %!s(<nil>)
I1022 06:31:00.056004       1 controllerserver.go:391] delete azure disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-d0f0cce2-e6a9-4010-b1a2-189c3f78f4a6) returned with Retriable: true, RetryAfter: 0s, HTTPStatusCode: 0, RawError: Future#WaitForCompletion: context has been cancelled: StatusCode=200 -- Original Error: context canceled
E1022 06:31:00.056026       1 utils.go:100] GRPC error: Retriable: true, RetryAfter: 0s, HTTPStatusCode: 0, RawError: Future#WaitForCompletion: context has been cancelled: StatusCode=200 -- Original Error: context canceled
I1022 06:31:00.166486       1 utils.go:95] GRPC call: /csi.v1.Controller/ControllerGetCapabilities
I1022 06:31:00.166507       1 utils.go:96] GRPC request: {}
I1022 06:31:00.166549       1 utils.go:102] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}},{"Type":{"Rpc":{"type":6}}},{"Type":{"Rpc":{"type":7}}},{"Type":{"Rpc":{"type":9}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":10}}},{"Type":{"Rpc":{"type":13}}}]}
I1022 06:31:00.167070       1 utils.go:95] GRPC call: /csi.v1.Controller/ControllerExpandVolume
I1022 06:31:00.167084       1 utils.go:96] GRPC request: {"capacity_range":{"required_bytes":6442450944},"volume_capability":{"AccessType":{"Block":{}},"access_mode":{"mode":7}},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-bc294ea0-3146-4351-837c-8dba4447e4d2"}
I1022 06:31:00.297926       1 controllerserver.go:807] begin to expand azure disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-bc294ea0-3146-4351-837c-8dba4447e4d2) with new size(6442450944)
I1022 06:31:00.336445       1 azure_managedDiskController.go:336] azureDisk - begin to resize disk(pvc-bc294ea0-3146-4351-837c-8dba4447e4d2) with new size(6), old size({{5 0} {<nil>}  BinarySI})
E1022 06:31:00.336496       1 utils.go:100] GRPC error: rpc error: code = Internal desc = failed to resize disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-bc294ea0-3146-4351-837c-8dba4447e4d2) with error(azureDisk - disk resize is only supported on Unattached disk, current disk state: Attached, already attached to /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool1-22475134-vmss/virtualMachines/k8s-agentpool1-22475134-vmss_1)
I1022 06:31:00.342934       1 utils.go:95] GRPC call: /csi.v1.Controller/ControllerGetCapabilities
I1022 06:31:00.342962       1 utils.go:96] GRPC request: {}
I1022 06:31:00.342993       1 utils.go:102] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}},{"Type":{"Rpc":{"type":6}}},{"Type":{"Rpc":{"type":7}}},{"Type":{"Rpc":{"type":9}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":10}}},{"Type":{"Rpc":{"type":13}}}]}
I1022 06:31:00.343457       1 utils.go:95] GRPC call: /csi.v1.Controller/ControllerExpandVolume
I1022 06:31:00.343477       1 utils.go:96] GRPC request: {"capacity_range":{"required_bytes":6442450944},"volume_capability":{"AccessType":{"Block":{}},"access_mode":{"mode":7}},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-bc294ea0-3146-4351-837c-8dba4447e4d2"}
I1022 06:31:00.365784       1 controllerserver.go:807] begin to expand azure disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-bc294ea0-3146-4351-837c-8dba4447e4d2) with new size(6442450944)
I1022 06:31:00.391360       1 azure_managedDiskController.go:336] azureDisk - begin to resize disk(pvc-bc294ea0-3146-4351-837c-8dba4447e4d2) with new size(6), old size({{5 0} {<nil>}  BinarySI})
E1022 06:31:00.391409       1 utils.go:100] GRPC error: rpc error: code = Internal desc = failed to resize disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-bc294ea0-3146-4351-837c-8dba4447e4d2) with error(azureDisk - disk resize is only supported on Unattached disk, current disk state: Attached, already attached to /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool1-22475134-vmss/virtualMachines/k8s-agentpool1-22475134-vmss_1)
I1022 06:31:00.956420       1 utils.go:95] GRPC call: /csi.v1.Controller/ControllerGetCapabilities
I1022 06:31:00.956461       1 utils.go:96] GRPC request: {}
I1022 06:31:00.956522       1 utils.go:102] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}},{"Type":{"Rpc":{"type":6}}},{"Type":{"Rpc":{"type":7}}},{"Type":{"Rpc":{"type":9}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":10}}},{"Type":{"Rpc":{"type":13}}}]}
I1022 06:31:00.957189       1 utils.go:95] GRPC call: /csi.v1.Controller/ListSnapshots
I1022 06:31:00.957214       1 utils.go:96] GRPC request: {"snapshot_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/snapshots/snapshot-0c714dcf-a300-494b-bca3-761d95ce0ca6"}
I1022 06:31:00.992639       1 utils.go:102] GRPC response: {"entries":[{"snapshot":{"creation_time":{"nanos":38028200,"seconds":1634884256},"ready_to_use":true,"size_bytes":5368709120,"snapshot_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/snapshots/snapshot-0c714dcf-a300-494b-bca3-761d95ce0ca6","source_volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-4028b709-1233-4cd4-8837-fd82aeedf83a"}}]}
... skipping 2 lines ...
I1022 06:31:15.177367       1 controllerserver.go:524] Trying to detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-3470e515-3208-4ae7-8a58-48ebecf49398 from node k8s-agentpool1-22475134-vmss000000
I1022 06:31:15.177384       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:31:15.177421       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:32:04.057153       1 utils.go:95] GRPC call: /csi.v1.Controller/DeleteVolume
I1022 06:32:04.057177       1 utils.go:96] GRPC request: {"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-d0f0cce2-e6a9-4010-b1a2-189c3f78f4a6"}
I1022 06:32:04.057263       1 controllerserver.go:389] deleting azure disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-d0f0cce2-e6a9-4010-b1a2-189c3f78f4a6)
I1022 06:32:04.077845       1 azure_diskclient.go:135] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-d0f0cce2-e6a9-4010-b1a2-189c3f78f4a6, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-d0f0cce2-e6a9-4010-b1a2-189c3f78f4a6' under resource group 'kubetest-5p3qtork' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I1022 06:32:04.077913       1 azure_managedDiskController.go:268] azureDisk - disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-d0f0cce2-e6a9-4010-b1a2-189c3f78f4a6) is already deleted
I1022 06:32:04.077923       1 controllerserver.go:391] delete azure disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-d0f0cce2-e6a9-4010-b1a2-189c3f78f4a6) returned with <nil>
I1022 06:32:04.077935       1 utils.go:102] GRPC response: {}
I1022 06:32:04.353476       1 utils.go:95] GRPC call: /csi.v1.Controller/ControllerGetCapabilities
I1022 06:32:04.353502       1 utils.go:96] GRPC request: {}
I1022 06:32:04.353546       1 utils.go:102] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}},{"Type":{"Rpc":{"type":6}}},{"Type":{"Rpc":{"type":7}}},{"Type":{"Rpc":{"type":9}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":10}}},{"Type":{"Rpc":{"type":13}}}]}
I1022 06:32:04.354169       1 utils.go:95] GRPC call: /csi.v1.Controller/ControllerExpandVolume
I1022 06:32:04.354202       1 utils.go:96] GRPC request: {"capacity_range":{"required_bytes":6442450944},"volume_capability":{"AccessType":{"Block":{}},"access_mode":{"mode":7}},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-bc294ea0-3146-4351-837c-8dba4447e4d2"}
I1022 06:32:04.394498       1 controllerserver.go:807] begin to expand azure disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-bc294ea0-3146-4351-837c-8dba4447e4d2) with new size(6442450944)
I1022 06:32:04.418191       1 azure_managedDiskController.go:336] azureDisk - begin to resize disk(pvc-bc294ea0-3146-4351-837c-8dba4447e4d2) with new size(6), old size({{5 0} {<nil>}  BinarySI})
E1022 06:32:04.418229       1 utils.go:100] GRPC error: rpc error: code = Internal desc = failed to resize disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-bc294ea0-3146-4351-837c-8dba4447e4d2) with error(azureDisk - disk resize is only supported on Unattached disk, current disk state: Attached, already attached to /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool1-22475134-vmss/virtualMachines/k8s-agentpool1-22475134-vmss_1)
I1022 06:32:04.454922       1 utils.go:95] GRPC call: /csi.v1.Controller/ControllerGetCapabilities
I1022 06:32:04.454942       1 utils.go:96] GRPC request: {}
I1022 06:32:04.454970       1 utils.go:102] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}},{"Type":{"Rpc":{"type":6}}},{"Type":{"Rpc":{"type":7}}},{"Type":{"Rpc":{"type":9}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":10}}},{"Type":{"Rpc":{"type":13}}}]}
I1022 06:32:04.455344       1 utils.go:95] GRPC call: /csi.v1.Controller/ControllerExpandVolume
I1022 06:32:04.455364       1 utils.go:96] GRPC request: {"capacity_range":{"required_bytes":6442450944},"volume_capability":{"AccessType":{"Block":{}},"access_mode":{"mode":7}},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-bc294ea0-3146-4351-837c-8dba4447e4d2"}
I1022 06:32:04.476773       1 controllerserver.go:807] begin to expand azure disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-bc294ea0-3146-4351-837c-8dba4447e4d2) with new size(6442450944)
I1022 06:32:04.497639       1 azure_managedDiskController.go:336] azureDisk - begin to resize disk(pvc-bc294ea0-3146-4351-837c-8dba4447e4d2) with new size(6), old size({{5 0} {<nil>}  BinarySI})
E1022 06:32:04.497709       1 utils.go:100] GRPC error: rpc error: code = Internal desc = failed to resize disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-bc294ea0-3146-4351-837c-8dba4447e4d2) with error(azureDisk - disk resize is only supported on Unattached disk, current disk state: Attached, already attached to /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool1-22475134-vmss/virtualMachines/k8s-agentpool1-22475134-vmss_1)
I1022 06:32:06.563070       1 utils.go:95] GRPC call: /csi.v1.Controller/CreateVolume
I1022 06:32:06.563095       1 utils.go:96] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"requisite":[{"segments":{"topology.test.csi.azure.com/zone":""}}]},"capacity_range":{"required_bytes":5368709120},"name":"pvc-35f243f0-f6ed-410c-9d3f-999c356910d0","parameters":{"csi.storage.k8s.io/pv/name":"pvc-35f243f0-f6ed-410c-9d3f-999c356910d0","csi.storage.k8s.io/pvc/name":"test.csi.azure.comgmfvc","csi.storage.k8s.io/pvc/namespace":"volumemode-7968"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]}
I1022 06:32:06.582012       1 azure_diskclient.go:135] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-35f243f0-f6ed-410c-9d3f-999c356910d0, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-35f243f0-f6ed-410c-9d3f-999c356910d0' under resource group 'kubetest-5p3qtork' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I1022 06:32:06.582122       1 controllerserver.go:271] begin to create azure disk(pvc-35f243f0-f6ed-410c-9d3f-999c356910d0) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) diskZone() maxShares(0)
I1022 06:32:06.582134       1 azure_managedDiskController.go:88] azureDisk - creating new managed Name:pvc-35f243f0-f6ed-410c-9d3f-999c356910d0 StorageAccountType:StandardSSD_LRS Size:5
I1022 06:32:08.968099       1 azure_managedDiskController.go:249] azureDisk - created new MD Name:pvc-35f243f0-f6ed-410c-9d3f-999c356910d0 StorageAccountType:StandardSSD_LRS Size:5
I1022 06:32:08.968169       1 controllerserver.go:348] create azure disk(pvc-35f243f0-f6ed-410c-9d3f-999c356910d0) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) tags(map[kubernetes.io-created-for-pv-name:pvc-35f243f0-f6ed-410c-9d3f-999c356910d0 kubernetes.io-created-for-pvc-name:test.csi.azure.comgmfvc kubernetes.io-created-for-pvc-namespace:volumemode-7968]) successfully
I1022 06:32:08.968190       1 utils.go:102] GRPC response: {"volume":{"accessible_topology":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"capacity_bytes":5368709120,"content_source":{"Type":null},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-35f243f0-f6ed-410c-9d3f-999c356910d0","csi.storage.k8s.io/pvc/name":"test.csi.azure.comgmfvc","csi.storage.k8s.io/pvc/namespace":"volumemode-7968","requestedsizegib":"5"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-35f243f0-f6ed-410c-9d3f-999c356910d0"}}
I1022 06:32:11.264066       1 utils.go:95] GRPC call: /csi.v1.Controller/ControllerPublishVolume
... skipping 12 lines ...
I1022 06:36:20.456736       1 utils.go:95] GRPC call: /csi.v1.Controller/ControllerExpandVolume
I1022 06:36:20.456758       1 utils.go:96] GRPC request: {"capacity_range":{"required_bytes":6442450944},"volume_capability":{"AccessType":{"Block":{}},"access_mode":{"mode":7}},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-bc294ea0-3146-4351-837c-8dba4447e4d2"}
I1022 06:36:20.481662       1 controllerserver.go:807] begin to expand azure disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-bc294ea0-3146-4351-837c-8dba4447e4d2) with new size(6442450944)
I1022 06:36:20.507197       1 azure_managedDiskController.go:336] azureDisk - begin to resize disk(pvc-bc294ea0-3146-4351-837c-8dba4447e4d2) with new size(6), old size({{5 0} {<nil>}  BinarySI})
I1022 06:36:22.385845       1 utils.go:95] GRPC call: /csi.v1.Controller/CreateVolume
I1022 06:36:22.385869       1 utils.go:96] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"requisite":[{"segments":{"topology.test.csi.azure.com/zone":""}}]},"capacity_range":{"required_bytes":5368709120},"name":"pvc-6c25433f-b868-4830-abac-dc47802cc74d","parameters":{"csi.storage.k8s.io/pv/name":"pvc-6c25433f-b868-4830-abac-dc47802cc74d","csi.storage.k8s.io/pvc/name":"inline-volume-tester-f2zk9-my-volume-0","csi.storage.k8s.io/pvc/namespace":"ephemeral-1422"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]}
I1022 06:36:22.446631       1 azure_diskclient.go:135] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-6c25433f-b868-4830-abac-dc47802cc74d, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-6c25433f-b868-4830-abac-dc47802cc74d' under resource group 'kubetest-5p3qtork' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I1022 06:36:22.446721       1 controllerserver.go:271] begin to create azure disk(pvc-6c25433f-b868-4830-abac-dc47802cc74d) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) diskZone() maxShares(0)
I1022 06:36:22.446760       1 azure_managedDiskController.go:88] azureDisk - creating new managed Name:pvc-6c25433f-b868-4830-abac-dc47802cc74d StorageAccountType:StandardSSD_LRS Size:5
I1022 06:36:22.681763       1 azure_managedDiskController.go:358] azureDisk - resize disk(pvc-bc294ea0-3146-4351-837c-8dba4447e4d2) with new size(6) completed
I1022 06:36:22.681807       1 controllerserver.go:819] expand azure disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-bc294ea0-3146-4351-837c-8dba4447e4d2) successfully, currentSize(6442450944)
I1022 06:36:22.681819       1 utils.go:102] GRPC response: {"capacity_bytes":6442450944,"node_expansion_required":true}
I1022 06:36:24.872205       1 azure_managedDiskController.go:249] azureDisk - created new MD Name:pvc-6c25433f-b868-4830-abac-dc47802cc74d StorageAccountType:StandardSSD_LRS Size:5
... skipping 3 lines ...
I1022 06:36:25.504273       1 utils.go:96] GRPC request: {"node_id":"k8s-agentpool1-22475134-vmss000000","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-6c25433f-b868-4830-abac-dc47802cc74d","csi.storage.k8s.io/pvc/name":"inline-volume-tester-f2zk9-my-volume-0","csi.storage.k8s.io/pvc/namespace":"ephemeral-1422","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1634884037556-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-6c25433f-b868-4830-abac-dc47802cc74d"}
I1022 06:36:25.530426       1 controllerserver.go:440] GetDiskLun returned: <nil>. Initiating attaching volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-6c25433f-b868-4830-abac-dc47802cc74d" to node "k8s-agentpool1-22475134-vmss000000".
I1022 06:36:25.530489       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:36:25.530512       1 controllerserver.go:461] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-6c25433f-b868-4830-abac-dc47802cc74d" to node "k8s-agentpool1-22475134-vmss000000"
I1022 06:37:26.501390       1 utils.go:95] GRPC call: /csi.v1.Controller/CreateVolume
I1022 06:37:26.501418       1 utils.go:96] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"requisite":[{"segments":{"topology.test.csi.azure.com/zone":""}}]},"capacity_range":{"required_bytes":5368709120},"name":"pvc-9c0d9aa4-9841-4932-aa15-e9e57d5bbfc5","parameters":{"csi.storage.k8s.io/pv/name":"pvc-9c0d9aa4-9841-4932-aa15-e9e57d5bbfc5","csi.storage.k8s.io/pvc/name":"test.csi.azure.comv88b9","csi.storage.k8s.io/pvc/namespace":"provisioning-873"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]}
I1022 06:37:26.520482       1 azure_diskclient.go:135] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-9c0d9aa4-9841-4932-aa15-e9e57d5bbfc5, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-9c0d9aa4-9841-4932-aa15-e9e57d5bbfc5' under resource group 'kubetest-5p3qtork' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I1022 06:37:26.520561       1 controllerserver.go:271] begin to create azure disk(pvc-9c0d9aa4-9841-4932-aa15-e9e57d5bbfc5) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) diskZone() maxShares(0)
I1022 06:37:26.520574       1 azure_managedDiskController.go:88] azureDisk - creating new managed Name:pvc-9c0d9aa4-9841-4932-aa15-e9e57d5bbfc5 StorageAccountType:StandardSSD_LRS Size:5
I1022 06:37:28.921868       1 azure_managedDiskController.go:249] azureDisk - created new MD Name:pvc-9c0d9aa4-9841-4932-aa15-e9e57d5bbfc5 StorageAccountType:StandardSSD_LRS Size:5
I1022 06:37:28.921924       1 controllerserver.go:348] create azure disk(pvc-9c0d9aa4-9841-4932-aa15-e9e57d5bbfc5) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) tags(map[kubernetes.io-created-for-pv-name:pvc-9c0d9aa4-9841-4932-aa15-e9e57d5bbfc5 kubernetes.io-created-for-pvc-name:test.csi.azure.comv88b9 kubernetes.io-created-for-pvc-namespace:provisioning-873]) successfully
I1022 06:37:28.921947       1 utils.go:102] GRPC response: {"volume":{"accessible_topology":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"capacity_bytes":5368709120,"content_source":{"Type":null},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-9c0d9aa4-9841-4932-aa15-e9e57d5bbfc5","csi.storage.k8s.io/pvc/name":"test.csi.azure.comv88b9","csi.storage.k8s.io/pvc/namespace":"provisioning-873","requestedsizegib":"5"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-9c0d9aa4-9841-4932-aa15-e9e57d5bbfc5"}}
I1022 06:37:31.192340       1 utils.go:95] GRPC call: /csi.v1.Controller/ControllerPublishVolume
... skipping 53 lines ...
I1022 06:38:48.839122       1 utils.go:102] GRPC response: {}
I1022 06:38:48.839214       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000001" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:38:48.839236       1 azure_vmss.go:198] Couldn't find VMSS VM with nodeName k8s-agentpool1-22475134-vmss000001, refreshing the cache(vmss: k8s-agentpool1-22475134-vmss, rg: kubetest-5p3qtork)
I1022 06:38:48.921276       1 azure_controller_common.go:242] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-9c0d9aa4-9841-4932-aa15-e9e57d5bbfc5" lun 0 to node "k8s-agentpool1-22475134-vmss000001", diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-9c0d9aa4-9841-4932-aa15-e9e57d5bbfc5:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-9c0d9aa4-9841-4932-aa15-e9e57d5bbfc5  true false 0})]
I1022 06:38:48.921317       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000001" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:38:48.921337       1 azure_controller_vmss.go:122] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-9c0d9aa4-9841-4932-aa15-e9e57d5bbfc5:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-9c0d9aa4-9841-4932-aa15-e9e57d5bbfc5  true false 0})])
I1022 06:38:49.086095       1 azure_controller_vmss.go:134] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-9c0d9aa4-9841-4932-aa15-e9e57d5bbfc5:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-9c0d9aa4-9841-4932-aa15-e9e57d5bbfc5  true false 0})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING)
I1022 06:38:49.086307       1 azure_controller_common.go:346] Trying to detach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-a8b788c1-65b1-446d-9813-2274459c86a0" from node "k8s-agentpool1-22475134-vmss000001", diskMap: map[]
I1022 06:38:49.086352       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000001" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:38:49.086386       1 azure_vmss.go:198] Couldn't find VMSS VM with nodeName k8s-agentpool1-22475134-vmss000001, refreshing the cache(vmss: k8s-agentpool1-22475134-vmss, rg: kubetest-5p3qtork)
I1022 06:38:49.102376       1 azure_controller_vmss.go:225] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - detach disk(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-3470e515-3208-4ae7-8a58-48ebecf49398:pvc-3470e515-3208-4ae7-8a58-48ebecf49398 /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-4028b709-1233-4cd4-8837-fd82aeedf83a:pvc-4028b709-1233-4cd4-8837-fd82aeedf83a]) returned with <nil>
I1022 06:38:49.158762       1 azure_controller_common.go:369] azureDisk - detach disk(pvc-a8b788c1-65b1-446d-9813-2274459c86a0, /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-a8b788c1-65b1-446d-9813-2274459c86a0) succeeded
I1022 06:38:49.158788       1 controllerserver.go:533] detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-a8b788c1-65b1-446d-9813-2274459c86a0 from node k8s-agentpool1-22475134-vmss000001 successfully
... skipping 24 lines ...
I1022 06:39:04.512451       1 utils.go:102] GRPC response: {}
I1022 06:39:04.512517       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:39:04.512548       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:39:04.512551       1 azure_controller_common.go:242] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-35f243f0-f6ed-410c-9d3f-999c356910d0" lun 2 to node "k8s-agentpool1-22475134-vmss000000", diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-35f243f0-f6ed-410c-9d3f-999c356910d0:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-35f243f0-f6ed-410c-9d3f-999c356910d0  true false 2}) /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-6c25433f-b868-4830-abac-dc47802cc74d:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-6c25433f-b868-4830-abac-dc47802cc74d  true false 3}) /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-bc294ea0-3146-4351-837c-8dba4447e4d2:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-bc294ea0-3146-4351-837c-8dba4447e4d2  true false 1})]
I1022 06:39:04.512601       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:39:04.512626       1 azure_controller_vmss.go:122] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-35f243f0-f6ed-410c-9d3f-999c356910d0:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-35f243f0-f6ed-410c-9d3f-999c356910d0  true false 2}) /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-6c25433f-b868-4830-abac-dc47802cc74d:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-6c25433f-b868-4830-abac-dc47802cc74d  true false 3}) /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-bc294ea0-3146-4351-837c-8dba4447e4d2:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-bc294ea0-3146-4351-837c-8dba4447e4d2  true false 1})])
I1022 06:39:04.845700       1 azure_controller_vmss.go:134] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-35f243f0-f6ed-410c-9d3f-999c356910d0:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-35f243f0-f6ed-410c-9d3f-999c356910d0  true false 2}) /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-6c25433f-b868-4830-abac-dc47802cc74d:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-6c25433f-b868-4830-abac-dc47802cc74d  true false 3}) /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-bc294ea0-3146-4351-837c-8dba4447e4d2:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-bc294ea0-3146-4351-837c-8dba4447e4d2  true false 1})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING)
I1022 06:39:04.845923       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:39:04.845970       1 azure_vmss.go:198] Couldn't find VMSS VM with nodeName k8s-agentpool1-22475134-vmss000000, refreshing the cache(vmss: k8s-agentpool1-22475134-vmss, rg: kubetest-5p3qtork)
I1022 06:39:05.117931       1 azure_controller_common.go:242] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-6c25433f-b868-4830-abac-dc47802cc74d" lun 3 to node "k8s-agentpool1-22475134-vmss000000", diskMap: map[]
I1022 06:39:05.117995       1 controllerserver.go:465] Attach operation successful: volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-6c25433f-b868-4830-abac-dc47802cc74d" attached to node "k8s-agentpool1-22475134-vmss000000".
I1022 06:39:05.118024       1 controllerserver.go:485] attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-6c25433f-b868-4830-abac-dc47802cc74d" to node "k8s-agentpool1-22475134-vmss000000" successfully
I1022 06:39:05.118058       1 utils.go:102] GRPC response: {"publish_context":{"LUN":"3"}}
... skipping 26 lines ...
I1022 06:39:16.527647       1 controllerserver.go:389] deleting azure disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-a8b788c1-65b1-446d-9813-2274459c86a0)
I1022 06:39:21.776249       1 azure_managedDiskController.go:286] azureDisk - deleted a managed disk: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-a8b788c1-65b1-446d-9813-2274459c86a0
I1022 06:39:21.776274       1 controllerserver.go:391] delete azure disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-a8b788c1-65b1-446d-9813-2274459c86a0) returned with <nil>
I1022 06:39:21.776289       1 utils.go:102] GRPC response: {}
I1022 06:39:26.662470       1 utils.go:95] GRPC call: /csi.v1.Controller/CreateVolume
I1022 06:39:26.662495       1 utils.go:96] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"requisite":[{"segments":{"topology.test.csi.azure.com/zone":""}}]},"capacity_range":{"required_bytes":5368709120},"name":"pvc-751af155-010a-4be3-9b70-54bbabd05ac7","parameters":{"csi.storage.k8s.io/pv/name":"pvc-751af155-010a-4be3-9b70-54bbabd05ac7","csi.storage.k8s.io/pvc/name":"pvc-fhwcj","csi.storage.k8s.io/pvc/namespace":"provisioning-7877"},"volume_capabilities":[{"AccessType":{"Block":{}},"access_mode":{"mode":7}}]}
I1022 06:39:26.681113       1 azure_diskclient.go:135] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-751af155-010a-4be3-9b70-54bbabd05ac7, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-751af155-010a-4be3-9b70-54bbabd05ac7' under resource group 'kubetest-5p3qtork' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I1022 06:39:26.681187       1 controllerserver.go:271] begin to create azure disk(pvc-751af155-010a-4be3-9b70-54bbabd05ac7) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) diskZone() maxShares(0)
I1022 06:39:26.681228       1 azure_managedDiskController.go:88] azureDisk - creating new managed Name:pvc-751af155-010a-4be3-9b70-54bbabd05ac7 StorageAccountType:StandardSSD_LRS Size:5
I1022 06:39:29.175685       1 azure_managedDiskController.go:249] azureDisk - created new MD Name:pvc-751af155-010a-4be3-9b70-54bbabd05ac7 StorageAccountType:StandardSSD_LRS Size:5
I1022 06:39:29.175743       1 controllerserver.go:348] create azure disk(pvc-751af155-010a-4be3-9b70-54bbabd05ac7) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) tags(map[kubernetes.io-created-for-pv-name:pvc-751af155-010a-4be3-9b70-54bbabd05ac7 kubernetes.io-created-for-pvc-name:pvc-fhwcj kubernetes.io-created-for-pvc-namespace:provisioning-7877]) successfully
I1022 06:39:29.175782       1 utils.go:102] GRPC response: {"volume":{"accessible_topology":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"capacity_bytes":5368709120,"content_source":{"Type":null},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-751af155-010a-4be3-9b70-54bbabd05ac7","csi.storage.k8s.io/pvc/name":"pvc-fhwcj","csi.storage.k8s.io/pvc/namespace":"provisioning-7877","requestedsizegib":"5"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-751af155-010a-4be3-9b70-54bbabd05ac7"}}
I1022 06:39:31.482305       1 azure_controller_vmss.go:225] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - detach disk(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-35f243f0-f6ed-410c-9d3f-999c356910d0:pvc-35f243f0-f6ed-410c-9d3f-999c356910d0]) returned with <nil>
... skipping 21 lines ...
I1022 06:39:31.624123       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000001" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:39:31.624138       1 controllerserver.go:461] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-751af155-010a-4be3-9b70-54bbabd05ac7" to node "k8s-agentpool1-22475134-vmss000001"
I1022 06:39:31.624150       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000001" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:39:31.624160       1 azure_controller_common.go:242] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-751af155-010a-4be3-9b70-54bbabd05ac7" lun 1 to node "k8s-agentpool1-22475134-vmss000001", diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-751af155-010a-4be3-9b70-54bbabd05ac7:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-751af155-010a-4be3-9b70-54bbabd05ac7  true false 1})]
I1022 06:39:31.624196       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000001" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:39:31.624209       1 azure_controller_vmss.go:122] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-751af155-010a-4be3-9b70-54bbabd05ac7:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-751af155-010a-4be3-9b70-54bbabd05ac7  true false 1})])
I1022 06:39:31.811389       1 azure_controller_vmss.go:134] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-4028b709-1233-4cd4-8837-fd82aeedf83a:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-4028b709-1233-4cd4-8837-fd82aeedf83a  true false 2})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING)
I1022 06:39:31.835933       1 azure_controller_vmss.go:134] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-751af155-010a-4be3-9b70-54bbabd05ac7:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-751af155-010a-4be3-9b70-54bbabd05ac7  true false 1})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING)
I1022 06:39:45.909812       1 utils.go:95] GRPC call: /csi.v1.Controller/DeleteVolume
I1022 06:39:45.909836       1 utils.go:96] GRPC request: {"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-3470e515-3208-4ae7-8a58-48ebecf49398"}
I1022 06:39:45.909906       1 controllerserver.go:389] deleting azure disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-3470e515-3208-4ae7-8a58-48ebecf49398)
I1022 06:39:46.951517       1 controllerserver.go:465] Attach operation successful: volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-4028b709-1233-4cd4-8837-fd82aeedf83a" attached to node "k8s-agentpool1-22475134-vmss000000".
I1022 06:39:46.951544       1 controllerserver.go:485] attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-4028b709-1233-4cd4-8837-fd82aeedf83a" to node "k8s-agentpool1-22475134-vmss000000" successfully
I1022 06:39:46.951580       1 utils.go:102] GRPC response: {"publish_context":{"LUN":"2"}}
... skipping 8 lines ...
I1022 06:39:51.245149       1 utils.go:102] GRPC response: {}
I1022 06:39:52.815092       1 azure_managedDiskController.go:286] azureDisk - deleted a managed disk: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-d2b2d817-66c5-404d-969d-b833e1464a67
I1022 06:39:52.815117       1 controllerserver.go:391] delete azure disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-d2b2d817-66c5-404d-969d-b833e1464a67) returned with <nil>
I1022 06:39:52.815134       1 utils.go:102] GRPC response: {}
I1022 06:39:56.127748       1 utils.go:95] GRPC call: /csi.v1.Controller/CreateVolume
I1022 06:39:56.127772       1 utils.go:96] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"requisite":[{"segments":{"topology.test.csi.azure.com/zone":""}}]},"capacity_range":{"required_bytes":5368709120},"name":"pvc-1943503f-e1dc-4825-81a8-b38b96df76b0","parameters":{"csi.storage.k8s.io/pv/name":"pvc-1943503f-e1dc-4825-81a8-b38b96df76b0","csi.storage.k8s.io/pvc/name":"test.csi.azure.comgrsx8","csi.storage.k8s.io/pvc/namespace":"volumemode-2028"},"volume_capabilities":[{"AccessType":{"Block":{}},"access_mode":{"mode":7}}]}
I1022 06:39:56.146433       1 azure_diskclient.go:135] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-1943503f-e1dc-4825-81a8-b38b96df76b0, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-1943503f-e1dc-4825-81a8-b38b96df76b0' under resource group 'kubetest-5p3qtork' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I1022 06:39:56.146506       1 controllerserver.go:271] begin to create azure disk(pvc-1943503f-e1dc-4825-81a8-b38b96df76b0) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) diskZone() maxShares(0)
I1022 06:39:56.146530       1 azure_managedDiskController.go:88] azureDisk - creating new managed Name:pvc-1943503f-e1dc-4825-81a8-b38b96df76b0 StorageAccountType:StandardSSD_LRS Size:5
I1022 06:39:58.535156       1 utils.go:95] GRPC call: /csi.v1.Identity/GetPluginInfo
I1022 06:39:58.535180       1 utils.go:96] GRPC request: {}
I1022 06:39:58.535214       1 utils.go:102] GRPC response: {"name":"test.csi.azure.com","vendor_version":"v1.9.0-716f56a44514b799d2d6c6922145f22026e7d4e2"}
I1022 06:39:58.535549       1 utils.go:95] GRPC call: /csi.v1.Controller/CreateSnapshot
I1022 06:39:58.535567       1 utils.go:96] GRPC request: {"name":"snapshot-36cd98cb-be8f-4188-9dd7-3716e8f5ff89","source_volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-751af155-010a-4be3-9b70-54bbabd05ac7"}
I1022 06:39:58.535626       1 controllerserver.go:909] begin to create snapshot(snapshot-36cd98cb-be8f-4188-9dd7-3716e8f5ff89, incremental: true) under rg(kubetest-5p3qtork)
I1022 06:39:58.537014       1 azure_managedDiskController.go:249] azureDisk - created new MD Name:pvc-1943503f-e1dc-4825-81a8-b38b96df76b0 StorageAccountType:StandardSSD_LRS Size:5
I1022 06:39:58.537092       1 controllerserver.go:348] create azure disk(pvc-1943503f-e1dc-4825-81a8-b38b96df76b0) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) tags(map[kubernetes.io-created-for-pv-name:pvc-1943503f-e1dc-4825-81a8-b38b96df76b0 kubernetes.io-created-for-pvc-name:test.csi.azure.comgrsx8 kubernetes.io-created-for-pvc-namespace:volumemode-2028]) successfully
I1022 06:39:58.537130       1 utils.go:102] GRPC response: {"volume":{"accessible_topology":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"capacity_bytes":5368709120,"content_source":{"Type":null},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-1943503f-e1dc-4825-81a8-b38b96df76b0","csi.storage.k8s.io/pvc/name":"test.csi.azure.comgrsx8","csi.storage.k8s.io/pvc/namespace":"volumemode-2028","requestedsizegib":"5"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-1943503f-e1dc-4825-81a8-b38b96df76b0"}}
I1022 06:39:59.078995       1 utils.go:95] GRPC call: /csi.v1.Controller/CreateVolume
I1022 06:39:59.079023       1 utils.go:96] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"requisite":[{"segments":{"topology.test.csi.azure.com/zone":""}}]},"capacity_range":{"required_bytes":5368709120},"name":"pvc-8cb3be6a-7e14-407c-a360-45a8e702464a","parameters":{"csi.storage.k8s.io/pv/name":"pvc-8cb3be6a-7e14-407c-a360-45a8e702464a","csi.storage.k8s.io/pvc/name":"inline-volume-tester-nh4hq-my-volume-0","csi.storage.k8s.io/pvc/namespace":"ephemeral-6709"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]}
I1022 06:39:59.097765       1 azure_diskclient.go:135] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-8cb3be6a-7e14-407c-a360-45a8e702464a, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-8cb3be6a-7e14-407c-a360-45a8e702464a' under resource group 'kubetest-5p3qtork' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I1022 06:39:59.097845       1 controllerserver.go:271] begin to create azure disk(pvc-8cb3be6a-7e14-407c-a360-45a8e702464a) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) diskZone() maxShares(0)
I1022 06:39:59.097859       1 azure_managedDiskController.go:88] azureDisk - creating new managed Name:pvc-8cb3be6a-7e14-407c-a360-45a8e702464a StorageAccountType:StandardSSD_LRS Size:5
I1022 06:39:59.131576       1 utils.go:95] GRPC call: /csi.v1.Controller/ControllerUnpublishVolume
I1022 06:39:59.131595       1 utils.go:96] GRPC request: {"node_id":"k8s-agentpool1-22475134-vmss000001","volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-751af155-010a-4be3-9b70-54bbabd05ac7"}
I1022 06:39:59.131673       1 controllerserver.go:524] Trying to detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-751af155-010a-4be3-9b70-54bbabd05ac7 from node k8s-agentpool1-22475134-vmss000001
I1022 06:39:59.131685       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000001" is managed by VMSS since DisableAvailabilitySetNodes is set to true
... skipping 21 lines ...
I1022 06:40:02.580707       1 utils.go:96] GRPC request: {"node_id":"k8s-agentpool1-22475134-vmss000001","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-8cb3be6a-7e14-407c-a360-45a8e702464a","csi.storage.k8s.io/pvc/name":"inline-volume-tester-nh4hq-my-volume-0","csi.storage.k8s.io/pvc/namespace":"ephemeral-6709","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1634884037556-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-8cb3be6a-7e14-407c-a360-45a8e702464a"}
I1022 06:40:02.603786       1 controllerserver.go:440] GetDiskLun returned: <nil>. Initiating attaching volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-8cb3be6a-7e14-407c-a360-45a8e702464a" to node "k8s-agentpool1-22475134-vmss000001".
I1022 06:40:02.603830       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000001" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:40:02.603869       1 controllerserver.go:461] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-8cb3be6a-7e14-407c-a360-45a8e702464a" to node "k8s-agentpool1-22475134-vmss000001"
I1022 06:40:03.271457       1 utils.go:95] GRPC call: /csi.v1.Controller/CreateVolume
I1022 06:40:03.271484       1 utils.go:96] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"requisite":[{"segments":{"topology.test.csi.azure.com/zone":""}}]},"capacity_range":{"required_bytes":5368709120},"name":"pvc-bfc35524-8d50-41f1-8c03-f2f48aade7e0","parameters":{"csi.storage.k8s.io/pv/name":"pvc-bfc35524-8d50-41f1-8c03-f2f48aade7e0","csi.storage.k8s.io/pvc/name":"pvc-2dxzs","csi.storage.k8s.io/pvc/namespace":"provisioning-7877"},"volume_capabilities":[{"AccessType":{"Block":{}},"access_mode":{"mode":7}}],"volume_content_source":{"Type":{"Snapshot":{"snapshot_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/snapshots/snapshot-36cd98cb-be8f-4188-9dd7-3716e8f5ff89"}}}}
I1022 06:40:03.290783       1 azure_diskclient.go:135] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-bfc35524-8d50-41f1-8c03-f2f48aade7e0, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-bfc35524-8d50-41f1-8c03-f2f48aade7e0' under resource group 'kubetest-5p3qtork' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I1022 06:40:03.290897       1 controllerserver.go:271] begin to create azure disk(pvc-bfc35524-8d50-41f1-8c03-f2f48aade7e0) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) diskZone() maxShares(0)
I1022 06:40:03.290910       1 azure_managedDiskController.go:88] azureDisk - creating new managed Name:pvc-bfc35524-8d50-41f1-8c03-f2f48aade7e0 StorageAccountType:StandardSSD_LRS Size:5
I1022 06:40:05.643582       1 azure_managedDiskController.go:249] azureDisk - created new MD Name:pvc-bfc35524-8d50-41f1-8c03-f2f48aade7e0 StorageAccountType:StandardSSD_LRS Size:5
I1022 06:40:05.643637       1 controllerserver.go:348] create azure disk(pvc-bfc35524-8d50-41f1-8c03-f2f48aade7e0) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) tags(map[kubernetes.io-created-for-pv-name:pvc-bfc35524-8d50-41f1-8c03-f2f48aade7e0 kubernetes.io-created-for-pvc-name:pvc-2dxzs kubernetes.io-created-for-pvc-namespace:provisioning-7877]) successfully
I1022 06:40:05.643698       1 utils.go:102] GRPC response: {"volume":{"accessible_topology":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"capacity_bytes":5368709120,"content_source":{"Type":{"Snapshot":{"snapshot_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/snapshots/snapshot-36cd98cb-be8f-4188-9dd7-3716e8f5ff89"}}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-bfc35524-8d50-41f1-8c03-f2f48aade7e0","csi.storage.k8s.io/pvc/name":"pvc-2dxzs","csi.storage.k8s.io/pvc/namespace":"provisioning-7877","requestedsizegib":"5"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-bfc35524-8d50-41f1-8c03-f2f48aade7e0"}}
I1022 06:40:08.622114       1 utils.go:95] GRPC call: /csi.v1.Controller/ControllerPublishVolume
... skipping 2 lines ...
I1022 06:40:08.660420       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:40:08.660463       1 controllerserver.go:461] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-bfc35524-8d50-41f1-8c03-f2f48aade7e0" to node "k8s-agentpool1-22475134-vmss000000"
I1022 06:40:08.660478       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:40:08.660488       1 azure_controller_common.go:242] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-bfc35524-8d50-41f1-8c03-f2f48aade7e0" lun 4 to node "k8s-agentpool1-22475134-vmss000000", diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-bfc35524-8d50-41f1-8c03-f2f48aade7e0:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-bfc35524-8d50-41f1-8c03-f2f48aade7e0  true false 4})]
I1022 06:40:08.660521       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:40:08.660556       1 azure_controller_vmss.go:122] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-bfc35524-8d50-41f1-8c03-f2f48aade7e0:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-bfc35524-8d50-41f1-8c03-f2f48aade7e0  true false 4})])
I1022 06:40:08.877255       1 azure_controller_vmss.go:134] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-bfc35524-8d50-41f1-8c03-f2f48aade7e0:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-bfc35524-8d50-41f1-8c03-f2f48aade7e0  true false 4})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING)
I1022 06:40:09.238784       1 utils.go:95] GRPC call: /csi.v1.Controller/ControllerUnpublishVolume
I1022 06:40:09.238809       1 utils.go:96] GRPC request: {"node_id":"k8s-agentpool1-22475134-vmss000000","volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-f3e47329-1be6-465a-8415-51a95da5fb5b"}
I1022 06:40:09.238904       1 controllerserver.go:524] Trying to detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-f3e47329-1be6-465a-8415-51a95da5fb5b from node k8s-agentpool1-22475134-vmss000000
I1022 06:40:09.238919       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:40:09.238940       1 azure_vmss.go:198] Couldn't find VMSS VM with nodeName k8s-agentpool1-22475134-vmss000000, refreshing the cache(vmss: k8s-agentpool1-22475134-vmss, rg: kubetest-5p3qtork)
I1022 06:40:09.298955       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
... skipping 46 lines ...
I1022 06:41:19.700375       1 utils.go:96] GRPC request: {"node_id":"k8s-agentpool1-22475134-vmss000000","volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-6c25433f-b868-4830-abac-dc47802cc74d"}
I1022 06:41:19.700516       1 controllerserver.go:524] Trying to detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-6c25433f-b868-4830-abac-dc47802cc74d from node k8s-agentpool1-22475134-vmss000000
I1022 06:41:19.700529       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:41:19.700567       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:41:27.151436       1 utils.go:95] GRPC call: /csi.v1.Controller/CreateVolume
I1022 06:41:27.151461       1 utils.go:96] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"requisite":[{"segments":{"topology.test.csi.azure.com/zone":""}}]},"capacity_range":{"required_bytes":5368709120},"name":"pvc-23114395-d906-41ff-a057-97298a933d12","parameters":{"csi.storage.k8s.io/pv/name":"pvc-23114395-d906-41ff-a057-97298a933d12","csi.storage.k8s.io/pvc/name":"pvc-cnph7","csi.storage.k8s.io/pvc/namespace":"snapshotting-2296"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}],"volume_content_source":{"Type":{"Snapshot":{"snapshot_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/snapshots/snapshot-0c714dcf-a300-494b-bca3-761d95ce0ca6"}}}}
I1022 06:41:27.171138       1 azure_diskclient.go:135] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-23114395-d906-41ff-a057-97298a933d12, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-23114395-d906-41ff-a057-97298a933d12' under resource group 'kubetest-5p3qtork' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I1022 06:41:27.171210       1 controllerserver.go:271] begin to create azure disk(pvc-23114395-d906-41ff-a057-97298a933d12) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) diskZone() maxShares(0)
I1022 06:41:27.171226       1 azure_managedDiskController.go:88] azureDisk - creating new managed Name:pvc-23114395-d906-41ff-a057-97298a933d12 StorageAccountType:StandardSSD_LRS Size:5
I1022 06:41:29.526852       1 azure_managedDiskController.go:249] azureDisk - created new MD Name:pvc-23114395-d906-41ff-a057-97298a933d12 StorageAccountType:StandardSSD_LRS Size:5
I1022 06:41:29.526925       1 controllerserver.go:348] create azure disk(pvc-23114395-d906-41ff-a057-97298a933d12) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) tags(map[kubernetes.io-created-for-pv-name:pvc-23114395-d906-41ff-a057-97298a933d12 kubernetes.io-created-for-pvc-name:pvc-cnph7 kubernetes.io-created-for-pvc-namespace:snapshotting-2296]) successfully
I1022 06:41:29.526961       1 utils.go:102] GRPC response: {"volume":{"accessible_topology":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"capacity_bytes":5368709120,"content_source":{"Type":{"Snapshot":{"snapshot_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/snapshots/snapshot-0c714dcf-a300-494b-bca3-761d95ce0ca6"}}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-23114395-d906-41ff-a057-97298a933d12","csi.storage.k8s.io/pvc/name":"pvc-cnph7","csi.storage.k8s.io/pvc/namespace":"snapshotting-2296","requestedsizegib":"5"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-23114395-d906-41ff-a057-97298a933d12"}}
I1022 06:41:29.808679       1 utils.go:95] GRPC call: /csi.v1.Controller/ControllerUnpublishVolume
... skipping 21 lines ...
I1022 06:41:40.834778       1 utils.go:102] GRPC response: {}
I1022 06:41:40.834863       1 azure_controller_common.go:346] Trying to detach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-f3e47329-1be6-465a-8415-51a95da5fb5b" from node "k8s-agentpool1-22475134-vmss000000", diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-4028b709-1233-4cd4-8837-fd82aeedf83a:pvc-4028b709-1233-4cd4-8837-fd82aeedf83a /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-6c25433f-b868-4830-abac-dc47802cc74d:pvc-6c25433f-b868-4830-abac-dc47802cc74d /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-f3e47329-1be6-465a-8415-51a95da5fb5b:pvc-f3e47329-1be6-465a-8415-51a95da5fb5b]
I1022 06:41:40.834927       1 azure_vmss.go:198] Couldn't find VMSS VM with nodeName k8s-agentpool1-22475134-vmss000000, refreshing the cache(vmss: k8s-agentpool1-22475134-vmss, rg: kubetest-5p3qtork)
I1022 06:41:40.880916       1 azure_controller_vmss.go:174] azureDisk - detach disk: name "pvc-6c25433f-b868-4830-abac-dc47802cc74d" uri "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-6c25433f-b868-4830-abac-dc47802cc74d"
I1022 06:41:40.880983       1 azure_controller_vmss.go:174] azureDisk - detach disk: name "pvc-4028b709-1233-4cd4-8837-fd82aeedf83a" uri "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-4028b709-1233-4cd4-8837-fd82aeedf83a"
I1022 06:41:40.881019       1 azure_controller_vmss.go:213] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - detach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-4028b709-1233-4cd4-8837-fd82aeedf83a:pvc-4028b709-1233-4cd4-8837-fd82aeedf83a /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-6c25433f-b868-4830-abac-dc47802cc74d:pvc-6c25433f-b868-4830-abac-dc47802cc74d /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-f3e47329-1be6-465a-8415-51a95da5fb5b:pvc-f3e47329-1be6-465a-8415-51a95da5fb5b])
I1022 06:41:41.063386       1 azure_controller_vmss.go:134] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-1943503f-e1dc-4825-81a8-b38b96df76b0:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-1943503f-e1dc-4825-81a8-b38b96df76b0  true false 0}) /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-8cb3be6a-7e14-407c-a360-45a8e702464a:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-8cb3be6a-7e14-407c-a360-45a8e702464a  true false 1})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING)
I1022 06:41:41.063562       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000001" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:41:41.063591       1 azure_vmss.go:198] Couldn't find VMSS VM with nodeName k8s-agentpool1-22475134-vmss000001, refreshing the cache(vmss: k8s-agentpool1-22475134-vmss, rg: kubetest-5p3qtork)
I1022 06:41:41.115277       1 azure_controller_common.go:242] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-8cb3be6a-7e14-407c-a360-45a8e702464a" lun 1 to node "k8s-agentpool1-22475134-vmss000001", diskMap: map[]
I1022 06:41:41.115311       1 controllerserver.go:465] Attach operation successful: volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-8cb3be6a-7e14-407c-a360-45a8e702464a" attached to node "k8s-agentpool1-22475134-vmss000001".
I1022 06:41:41.115321       1 controllerserver.go:485] attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-8cb3be6a-7e14-407c-a360-45a8e702464a" to node "k8s-agentpool1-22475134-vmss000001" successfully
I1022 06:41:41.115334       1 utils.go:102] GRPC response: {"publish_context":{"LUN":"1"}}
... skipping 2 lines ...
I1022 06:41:45.497964       1 controllerserver.go:389] deleting azure disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-bc294ea0-3146-4351-837c-8dba4447e4d2)
I1022 06:41:50.938858       1 azure_managedDiskController.go:286] azureDisk - deleted a managed disk: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-bc294ea0-3146-4351-837c-8dba4447e4d2
I1022 06:41:50.938884       1 controllerserver.go:391] delete azure disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-bc294ea0-3146-4351-837c-8dba4447e4d2) returned with <nil>
I1022 06:41:50.938920       1 utils.go:102] GRPC response: {}
I1022 06:41:55.368178       1 utils.go:95] GRPC call: /csi.v1.Controller/CreateVolume
I1022 06:41:55.368221       1 utils.go:96] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"requisite":[{"segments":{"topology.test.csi.azure.com/zone":""}}]},"capacity_range":{"required_bytes":5368709120},"name":"pvc-2d720a3b-8cf5-4acd-a1d5-7527d7349bd6","parameters":{"csi.storage.k8s.io/pv/name":"pvc-2d720a3b-8cf5-4acd-a1d5-7527d7349bd6","csi.storage.k8s.io/pvc/name":"test.csi.azure.com827mb","csi.storage.k8s.io/pvc/namespace":"volume-2238"},"volume_capabilities":[{"AccessType":{"Block":{}},"access_mode":{"mode":7}}]}
I1022 06:41:55.389206       1 azure_diskclient.go:135] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-2d720a3b-8cf5-4acd-a1d5-7527d7349bd6, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-2d720a3b-8cf5-4acd-a1d5-7527d7349bd6' under resource group 'kubetest-5p3qtork' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I1022 06:41:55.389259       1 controllerserver.go:271] begin to create azure disk(pvc-2d720a3b-8cf5-4acd-a1d5-7527d7349bd6) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) diskZone() maxShares(0)
I1022 06:41:55.389293       1 azure_managedDiskController.go:88] azureDisk - creating new managed Name:pvc-2d720a3b-8cf5-4acd-a1d5-7527d7349bd6 StorageAccountType:StandardSSD_LRS Size:5
I1022 06:41:56.177137       1 controllerserver.go:465] Attach operation successful: volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-1943503f-e1dc-4825-81a8-b38b96df76b0" attached to node "k8s-agentpool1-22475134-vmss000001".
I1022 06:41:56.177214       1 controllerserver.go:485] attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-1943503f-e1dc-4825-81a8-b38b96df76b0" to node "k8s-agentpool1-22475134-vmss000001" successfully
I1022 06:41:56.177235       1 utils.go:102] GRPC response: {"publish_context":{"LUN":"0"}}
I1022 06:41:56.280130       1 azure_controller_vmss.go:225] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - detach disk(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-4028b709-1233-4cd4-8837-fd82aeedf83a:pvc-4028b709-1233-4cd4-8837-fd82aeedf83a /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-6c25433f-b868-4830-abac-dc47802cc74d:pvc-6c25433f-b868-4830-abac-dc47802cc74d /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-f3e47329-1be6-465a-8415-51a95da5fb5b:pvc-f3e47329-1be6-465a-8415-51a95da5fb5b]) returned with <nil>
... skipping 22 lines ...
I1022 06:41:56.363337       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:41:56.364991       1 utils.go:95] GRPC call: /csi.v1.Controller/ControllerUnpublishVolume
I1022 06:41:56.365010       1 utils.go:96] GRPC request: {"node_id":"k8s-agentpool1-22475134-vmss000000","volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-4028b709-1233-4cd4-8837-fd82aeedf83a"}
I1022 06:41:56.365083       1 controllerserver.go:524] Trying to detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-4028b709-1233-4cd4-8837-fd82aeedf83a from node k8s-agentpool1-22475134-vmss000000
I1022 06:41:56.365090       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:41:56.365120       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:41:56.547188       1 azure_controller_vmss.go:134] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-23114395-d906-41ff-a057-97298a933d12:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-23114395-d906-41ff-a057-97298a933d12  true false 0})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING)
I1022 06:41:56.547351       1 azure_controller_common.go:346] Trying to detach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-6c25433f-b868-4830-abac-dc47802cc74d" from node "k8s-agentpool1-22475134-vmss000000", diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-4028b709-1233-4cd4-8837-fd82aeedf83a:pvc-4028b709-1233-4cd4-8837-fd82aeedf83a /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-6c25433f-b868-4830-abac-dc47802cc74d:pvc-6c25433f-b868-4830-abac-dc47802cc74d]
I1022 06:41:56.547387       1 azure_vmss.go:198] Couldn't find VMSS VM with nodeName k8s-agentpool1-22475134-vmss000000, refreshing the cache(vmss: k8s-agentpool1-22475134-vmss, rg: kubetest-5p3qtork)
E1022 06:41:56.612689       1 azure_controller_vmss.go:183] detach azure disk on node(k8s-agentpool1-22475134-vmss000000): disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-4028b709-1233-4cd4-8837-fd82aeedf83a:pvc-4028b709-1233-4cd4-8837-fd82aeedf83a /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-6c25433f-b868-4830-abac-dc47802cc74d:pvc-6c25433f-b868-4830-abac-dc47802cc74d]) not found
I1022 06:41:56.612718       1 azure_controller_vmss.go:213] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - detach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-4028b709-1233-4cd4-8837-fd82aeedf83a:pvc-4028b709-1233-4cd4-8837-fd82aeedf83a /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-6c25433f-b868-4830-abac-dc47802cc74d:pvc-6c25433f-b868-4830-abac-dc47802cc74d])
I1022 06:41:57.747155       1 azure_managedDiskController.go:249] azureDisk - created new MD Name:pvc-2d720a3b-8cf5-4acd-a1d5-7527d7349bd6 StorageAccountType:StandardSSD_LRS Size:5
I1022 06:41:57.747195       1 controllerserver.go:348] create azure disk(pvc-2d720a3b-8cf5-4acd-a1d5-7527d7349bd6) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) tags(map[kubernetes.io-created-for-pv-name:pvc-2d720a3b-8cf5-4acd-a1d5-7527d7349bd6 kubernetes.io-created-for-pvc-name:test.csi.azure.com827mb kubernetes.io-created-for-pvc-namespace:volume-2238]) successfully
... skipping 8 lines ...
I1022 06:42:00.166249       1 controllerserver.go:461] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-2d720a3b-8cf5-4acd-a1d5-7527d7349bd6" to node "k8s-agentpool1-22475134-vmss000000"
I1022 06:42:03.547861       1 azure_managedDiskController.go:286] azureDisk - deleted a managed disk: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-9c0d9aa4-9841-4932-aa15-e9e57d5bbfc5
I1022 06:42:03.547888       1 controllerserver.go:391] delete azure disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-9c0d9aa4-9841-4932-aa15-e9e57d5bbfc5) returned with <nil>
I1022 06:42:03.547940       1 utils.go:102] GRPC response: {}
I1022 06:42:05.732005       1 utils.go:95] GRPC call: /csi.v1.Controller/CreateVolume
I1022 06:42:05.732029       1 utils.go:96] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"requisite":[{"segments":{"topology.test.csi.azure.com/zone":""}}]},"capacity_range":{"required_bytes":5368709120},"name":"pvc-1b1ac790-3b7b-4f7f-a54b-bd639857f2f7","parameters":{"csi.storage.k8s.io/pv/name":"pvc-1b1ac790-3b7b-4f7f-a54b-bd639857f2f7","csi.storage.k8s.io/pvc/name":"test.csi.azure.comhvs4f","csi.storage.k8s.io/pvc/namespace":"volume-expand-2071"},"volume_capabilities":[{"AccessType":{"Block":{}},"access_mode":{"mode":7}}]}
I1022 06:42:05.766073       1 azure_diskclient.go:135] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-1b1ac790-3b7b-4f7f-a54b-bd639857f2f7, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-1b1ac790-3b7b-4f7f-a54b-bd639857f2f7' under resource group 'kubetest-5p3qtork' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I1022 06:42:05.766151       1 controllerserver.go:271] begin to create azure disk(pvc-1b1ac790-3b7b-4f7f-a54b-bd639857f2f7) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) diskZone() maxShares(0)
I1022 06:42:05.766180       1 azure_managedDiskController.go:88] azureDisk - creating new managed Name:pvc-1b1ac790-3b7b-4f7f-a54b-bd639857f2f7 StorageAccountType:StandardSSD_LRS Size:5
I1022 06:42:08.095331       1 utils.go:95] GRPC call: /csi.v1.Controller/DeleteVolume
I1022 06:42:08.095355       1 utils.go:96] GRPC request: {"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-f3e47329-1be6-465a-8415-51a95da5fb5b"}
I1022 06:42:08.095430       1 controllerserver.go:389] deleting azure disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-f3e47329-1be6-465a-8415-51a95da5fb5b)
I1022 06:42:08.200326       1 azure_managedDiskController.go:249] azureDisk - created new MD Name:pvc-1b1ac790-3b7b-4f7f-a54b-bd639857f2f7 StorageAccountType:StandardSSD_LRS Size:5
... skipping 13 lines ...
I1022 06:42:11.942962       1 controllerserver.go:533] detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-4028b709-1233-4cd4-8837-fd82aeedf83a from node k8s-agentpool1-22475134-vmss000000 successfully
I1022 06:42:11.943002       1 utils.go:102] GRPC response: {}
I1022 06:42:11.943068       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:42:11.943119       1 azure_controller_common.go:242] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-2d720a3b-8cf5-4acd-a1d5-7527d7349bd6" lun 1 to node "k8s-agentpool1-22475134-vmss000000", diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-2d720a3b-8cf5-4acd-a1d5-7527d7349bd6:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-2d720a3b-8cf5-4acd-a1d5-7527d7349bd6  true false 1})]
I1022 06:42:11.943159       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:42:11.943194       1 azure_controller_vmss.go:122] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-2d720a3b-8cf5-4acd-a1d5-7527d7349bd6:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-2d720a3b-8cf5-4acd-a1d5-7527d7349bd6  true false 1})])
I1022 06:42:12.097897       1 azure_controller_vmss.go:134] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-2d720a3b-8cf5-4acd-a1d5-7527d7349bd6:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-2d720a3b-8cf5-4acd-a1d5-7527d7349bd6  true false 1})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING)
I1022 06:42:13.275262       1 azure_managedDiskController.go:286] azureDisk - deleted a managed disk: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-f3e47329-1be6-465a-8415-51a95da5fb5b
I1022 06:42:13.275287       1 controllerserver.go:391] delete azure disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-f3e47329-1be6-465a-8415-51a95da5fb5b) returned with <nil>
I1022 06:42:13.275319       1 utils.go:102] GRPC response: {}
I1022 06:42:14.921347       1 utils.go:95] GRPC call: /csi.v1.Controller/DeleteVolume
I1022 06:42:14.921375       1 utils.go:96] GRPC request: {"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-6c25433f-b868-4830-abac-dc47802cc74d"}
I1022 06:42:14.921447       1 controllerserver.go:389] deleting azure disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-6c25433f-b868-4830-abac-dc47802cc74d)
... skipping 16 lines ...
I1022 06:42:22.235014       1 utils.go:102] GRPC response: {"publish_context":{"LUN":"1"}}
I1022 06:42:23.385696       1 azure_managedDiskController.go:286] azureDisk - deleted a managed disk: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-8f420b91-438e-4fc3-a9c6-b57ba1679195
I1022 06:42:23.385722       1 controllerserver.go:391] delete azure disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-8f420b91-438e-4fc3-a9c6-b57ba1679195) returned with <nil>
I1022 06:42:23.385756       1 utils.go:102] GRPC response: {}
I1022 06:42:27.815695       1 utils.go:95] GRPC call: /csi.v1.Controller/CreateVolume
I1022 06:42:27.815736       1 utils.go:96] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"requisite":[{"segments":{"topology.test.csi.azure.com/zone":""}}]},"capacity_range":{"required_bytes":5368709120},"name":"pvc-36d4a527-6ebf-4355-aa89-9dc4be2d5626","parameters":{"csi.storage.k8s.io/pv/name":"pvc-36d4a527-6ebf-4355-aa89-9dc4be2d5626","csi.storage.k8s.io/pvc/name":"test.csi.azure.comhw7mn","csi.storage.k8s.io/pvc/namespace":"snapshotting-8809"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]}
I1022 06:42:27.835161       1 azure_diskclient.go:135] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-36d4a527-6ebf-4355-aa89-9dc4be2d5626, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-36d4a527-6ebf-4355-aa89-9dc4be2d5626' under resource group 'kubetest-5p3qtork' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I1022 06:42:27.835231       1 controllerserver.go:271] begin to create azure disk(pvc-36d4a527-6ebf-4355-aa89-9dc4be2d5626) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) diskZone() maxShares(0)
I1022 06:42:27.835243       1 azure_managedDiskController.go:88] azureDisk - creating new managed Name:pvc-36d4a527-6ebf-4355-aa89-9dc4be2d5626 StorageAccountType:StandardSSD_LRS Size:5
I1022 06:42:30.181800       1 azure_managedDiskController.go:249] azureDisk - created new MD Name:pvc-36d4a527-6ebf-4355-aa89-9dc4be2d5626 StorageAccountType:StandardSSD_LRS Size:5
I1022 06:42:30.181873       1 controllerserver.go:348] create azure disk(pvc-36d4a527-6ebf-4355-aa89-9dc4be2d5626) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) tags(map[kubernetes.io-created-for-pv-name:pvc-36d4a527-6ebf-4355-aa89-9dc4be2d5626 kubernetes.io-created-for-pvc-name:test.csi.azure.comhw7mn kubernetes.io-created-for-pvc-namespace:snapshotting-8809]) successfully
I1022 06:42:30.181894       1 utils.go:102] GRPC response: {"volume":{"accessible_topology":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"capacity_bytes":5368709120,"content_source":{"Type":null},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-36d4a527-6ebf-4355-aa89-9dc4be2d5626","csi.storage.k8s.io/pvc/name":"test.csi.azure.comhw7mn","csi.storage.k8s.io/pvc/namespace":"snapshotting-8809","requestedsizegib":"5"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-36d4a527-6ebf-4355-aa89-9dc4be2d5626"}}
I1022 06:42:32.496073       1 utils.go:95] GRPC call: /csi.v1.Controller/ControllerPublishVolume
... skipping 2 lines ...
I1022 06:42:32.567051       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:42:32.567091       1 controllerserver.go:461] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-36d4a527-6ebf-4355-aa89-9dc4be2d5626" to node "k8s-agentpool1-22475134-vmss000000"
I1022 06:42:32.567123       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:42:32.567134       1 azure_controller_common.go:242] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-36d4a527-6ebf-4355-aa89-9dc4be2d5626" lun 2 to node "k8s-agentpool1-22475134-vmss000000", diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-36d4a527-6ebf-4355-aa89-9dc4be2d5626:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-36d4a527-6ebf-4355-aa89-9dc4be2d5626  true false 2})]
I1022 06:42:32.567166       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:42:32.567203       1 azure_controller_vmss.go:122] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-36d4a527-6ebf-4355-aa89-9dc4be2d5626:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-36d4a527-6ebf-4355-aa89-9dc4be2d5626  true false 2})])
I1022 06:42:32.736274       1 azure_controller_vmss.go:134] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-36d4a527-6ebf-4355-aa89-9dc4be2d5626:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-36d4a527-6ebf-4355-aa89-9dc4be2d5626  true false 2})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING)
I1022 06:42:40.361484       1 utils.go:95] GRPC call: /csi.v1.Controller/ControllerUnpublishVolume
I1022 06:42:40.361511       1 utils.go:96] GRPC request: {"node_id":"k8s-agentpool1-22475134-vmss000000","volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-bfc35524-8d50-41f1-8c03-f2f48aade7e0"}
I1022 06:42:40.361618       1 controllerserver.go:524] Trying to detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-bfc35524-8d50-41f1-8c03-f2f48aade7e0 from node k8s-agentpool1-22475134-vmss000000
I1022 06:42:40.361636       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:42:40.361656       1 azure_vmss.go:198] Couldn't find VMSS VM with nodeName k8s-agentpool1-22475134-vmss000000, refreshing the cache(vmss: k8s-agentpool1-22475134-vmss, rg: kubetest-5p3qtork)
I1022 06:42:40.450737       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
... skipping 13 lines ...
I1022 06:42:46.138428       1 utils.go:102] GRPC response: {}
I1022 06:42:46.303818       1 azure_managedDiskController.go:286] azureDisk - deleted a managed disk: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-1b1ac790-3b7b-4f7f-a54b-bd639857f2f7
I1022 06:42:46.303860       1 controllerserver.go:391] delete azure disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-1b1ac790-3b7b-4f7f-a54b-bd639857f2f7) returned with <nil>
I1022 06:42:46.303875       1 utils.go:102] GRPC response: {}
I1022 06:42:52.550869       1 utils.go:95] GRPC call: /csi.v1.Controller/CreateVolume
I1022 06:42:52.550895       1 utils.go:96] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"requisite":[{"segments":{"topology.test.csi.azure.com/zone":""}}]},"capacity_range":{"required_bytes":5368709120},"name":"pvc-775fddc6-3d43-48ec-acb8-25cbf1f5427f","parameters":{"csi.storage.k8s.io/pv/name":"pvc-775fddc6-3d43-48ec-acb8-25cbf1f5427f","csi.storage.k8s.io/pvc/name":"test.csi.azure.comchg7p","csi.storage.k8s.io/pvc/namespace":"fsgroupchangepolicy-7047"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]}
I1022 06:42:52.571368       1 azure_diskclient.go:135] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-775fddc6-3d43-48ec-acb8-25cbf1f5427f, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-775fddc6-3d43-48ec-acb8-25cbf1f5427f' under resource group 'kubetest-5p3qtork' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I1022 06:42:52.571458       1 controllerserver.go:271] begin to create azure disk(pvc-775fddc6-3d43-48ec-acb8-25cbf1f5427f) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) diskZone() maxShares(0)
I1022 06:42:52.571471       1 azure_managedDiskController.go:88] azureDisk - creating new managed Name:pvc-775fddc6-3d43-48ec-acb8-25cbf1f5427f StorageAccountType:StandardSSD_LRS Size:5
I1022 06:42:54.923101       1 azure_managedDiskController.go:249] azureDisk - created new MD Name:pvc-775fddc6-3d43-48ec-acb8-25cbf1f5427f StorageAccountType:StandardSSD_LRS Size:5
I1022 06:42:54.923160       1 controllerserver.go:348] create azure disk(pvc-775fddc6-3d43-48ec-acb8-25cbf1f5427f) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) tags(map[kubernetes.io-created-for-pv-name:pvc-775fddc6-3d43-48ec-acb8-25cbf1f5427f kubernetes.io-created-for-pvc-name:test.csi.azure.comchg7p kubernetes.io-created-for-pvc-namespace:fsgroupchangepolicy-7047]) successfully
I1022 06:42:54.923183       1 utils.go:102] GRPC response: {"volume":{"accessible_topology":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"capacity_bytes":5368709120,"content_source":{"Type":null},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-775fddc6-3d43-48ec-acb8-25cbf1f5427f","csi.storage.k8s.io/pvc/name":"test.csi.azure.comchg7p","csi.storage.k8s.io/pvc/namespace":"fsgroupchangepolicy-7047","requestedsizegib":"5"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-775fddc6-3d43-48ec-acb8-25cbf1f5427f"}}
I1022 06:42:55.706963       1 azure_controller_vmss.go:225] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - detach disk(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-bfc35524-8d50-41f1-8c03-f2f48aade7e0:pvc-bfc35524-8d50-41f1-8c03-f2f48aade7e0]) returned with <nil>
... skipping 6 lines ...
I1022 06:42:57.313355       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000001" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:42:57.313393       1 controllerserver.go:461] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-775fddc6-3d43-48ec-acb8-25cbf1f5427f" to node "k8s-agentpool1-22475134-vmss000001"
I1022 06:42:57.313407       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000001" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:42:57.313418       1 azure_controller_common.go:242] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-775fddc6-3d43-48ec-acb8-25cbf1f5427f" lun 2 to node "k8s-agentpool1-22475134-vmss000001", diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-775fddc6-3d43-48ec-acb8-25cbf1f5427f:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-775fddc6-3d43-48ec-acb8-25cbf1f5427f  true false 2})]
I1022 06:42:57.313449       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000001" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:42:57.313482       1 azure_controller_vmss.go:122] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-775fddc6-3d43-48ec-acb8-25cbf1f5427f:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-775fddc6-3d43-48ec-acb8-25cbf1f5427f  true false 2})])
I1022 06:42:57.513883       1 azure_controller_vmss.go:134] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-775fddc6-3d43-48ec-acb8-25cbf1f5427f:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-775fddc6-3d43-48ec-acb8-25cbf1f5427f  true false 2})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING)
I1022 06:43:00.428062       1 utils.go:95] GRPC call: /csi.v1.Controller/ControllerUnpublishVolume
I1022 06:43:00.428113       1 utils.go:96] GRPC request: {"node_id":"k8s-agentpool1-22475134-vmss000001","volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-1943503f-e1dc-4825-81a8-b38b96df76b0"}
I1022 06:43:00.428230       1 controllerserver.go:524] Trying to detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-1943503f-e1dc-4825-81a8-b38b96df76b0 from node k8s-agentpool1-22475134-vmss000001
I1022 06:43:00.428244       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000001" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:43:00.428286       1 azure_vmss.go:198] Couldn't find VMSS VM with nodeName k8s-agentpool1-22475134-vmss000001, refreshing the cache(vmss: k8s-agentpool1-22475134-vmss, rg: kubetest-5p3qtork)
I1022 06:43:00.449692       1 utils.go:95] GRPC call: /csi.v1.Controller/ControllerUnpublishVolume
... skipping 101 lines ...
I1022 06:43:31.280027       1 azure_controller_vmss.go:213] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000001) - detach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-1943503f-e1dc-4825-81a8-b38b96df76b0:pvc-1943503f-e1dc-4825-81a8-b38b96df76b0])
I1022 06:43:31.669664       1 utils.go:95] GRPC call: /csi.v1.Controller/DeleteVolume
I1022 06:43:31.669693       1 utils.go:96] GRPC request: {"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-2d720a3b-8cf5-4acd-a1d5-7527d7349bd6"}
I1022 06:43:31.669797       1 controllerserver.go:389] deleting azure disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-2d720a3b-8cf5-4acd-a1d5-7527d7349bd6)
I1022 06:43:33.491532       1 utils.go:95] GRPC call: /csi.v1.Controller/CreateVolume
I1022 06:43:33.491557       1 utils.go:96] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"requisite":[{"segments":{"topology.test.csi.azure.com/zone":""}}]},"capacity_range":{"required_bytes":5368709120},"name":"pvc-6894fdc9-9e02-446f-92bb-2b026c63a266","parameters":{"csi.storage.k8s.io/pv/name":"pvc-6894fdc9-9e02-446f-92bb-2b026c63a266","csi.storage.k8s.io/pvc/name":"test.csi.azure.combkzh9","csi.storage.k8s.io/pvc/namespace":"snapshotting-5267"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]}
I1022 06:43:33.509969       1 azure_diskclient.go:135] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-6894fdc9-9e02-446f-92bb-2b026c63a266, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-6894fdc9-9e02-446f-92bb-2b026c63a266' under resource group 'kubetest-5p3qtork' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I1022 06:43:33.510055       1 controllerserver.go:271] begin to create azure disk(pvc-6894fdc9-9e02-446f-92bb-2b026c63a266) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) diskZone() maxShares(0)
I1022 06:43:33.510099       1 azure_managedDiskController.go:88] azureDisk - creating new managed Name:pvc-6894fdc9-9e02-446f-92bb-2b026c63a266 StorageAccountType:StandardSSD_LRS Size:5
I1022 06:43:35.895974       1 azure_managedDiskController.go:249] azureDisk - created new MD Name:pvc-6894fdc9-9e02-446f-92bb-2b026c63a266 StorageAccountType:StandardSSD_LRS Size:5
I1022 06:43:35.896015       1 controllerserver.go:348] create azure disk(pvc-6894fdc9-9e02-446f-92bb-2b026c63a266) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) tags(map[kubernetes.io-created-for-pv-name:pvc-6894fdc9-9e02-446f-92bb-2b026c63a266 kubernetes.io-created-for-pvc-name:test.csi.azure.combkzh9 kubernetes.io-created-for-pvc-namespace:snapshotting-5267]) successfully
I1022 06:43:35.896039       1 utils.go:102] GRPC response: {"volume":{"accessible_topology":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"capacity_bytes":5368709120,"content_source":{"Type":null},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-6894fdc9-9e02-446f-92bb-2b026c63a266","csi.storage.k8s.io/pvc/name":"test.csi.azure.combkzh9","csi.storage.k8s.io/pvc/namespace":"snapshotting-5267","requestedsizegib":"5"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-6894fdc9-9e02-446f-92bb-2b026c63a266"}}
I1022 06:43:36.884626       1 azure_managedDiskController.go:286] azureDisk - deleted a managed disk: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-2d720a3b-8cf5-4acd-a1d5-7527d7349bd6
... skipping 3 lines ...
I1022 06:43:38.185890       1 utils.go:96] GRPC request: {"node_id":"k8s-agentpool1-22475134-vmss000000","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-6894fdc9-9e02-446f-92bb-2b026c63a266","csi.storage.k8s.io/pvc/name":"test.csi.azure.combkzh9","csi.storage.k8s.io/pvc/namespace":"snapshotting-5267","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1634884037556-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-6894fdc9-9e02-446f-92bb-2b026c63a266"}
I1022 06:43:38.210047       1 controllerserver.go:440] GetDiskLun returned: <nil>. Initiating attaching volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-6894fdc9-9e02-446f-92bb-2b026c63a266" to node "k8s-agentpool1-22475134-vmss000000".
I1022 06:43:38.210072       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:43:38.210109       1 controllerserver.go:461] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-6894fdc9-9e02-446f-92bb-2b026c63a266" to node "k8s-agentpool1-22475134-vmss000000"
I1022 06:43:43.077112       1 utils.go:95] GRPC call: /csi.v1.Controller/CreateVolume
I1022 06:43:43.077138       1 utils.go:96] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"requisite":[{"segments":{"topology.test.csi.azure.com/zone":""}}]},"capacity_range":{"required_bytes":5368709120},"name":"pvc-d6c052b6-f54f-44e8-a567-4abeecf87be1","parameters":{"csi.storage.k8s.io/pv/name":"pvc-d6c052b6-f54f-44e8-a567-4abeecf87be1","csi.storage.k8s.io/pvc/name":"test.csi.azure.comvnkfv","csi.storage.k8s.io/pvc/namespace":"volume-expand-4084"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]}
I1022 06:43:43.100403       1 azure_diskclient.go:135] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-d6c052b6-f54f-44e8-a567-4abeecf87be1, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-d6c052b6-f54f-44e8-a567-4abeecf87be1' under resource group 'kubetest-5p3qtork' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I1022 06:43:43.100490       1 controllerserver.go:271] begin to create azure disk(pvc-d6c052b6-f54f-44e8-a567-4abeecf87be1) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) diskZone() maxShares(0)
I1022 06:43:43.100501       1 azure_managedDiskController.go:88] azureDisk - creating new managed Name:pvc-d6c052b6-f54f-44e8-a567-4abeecf87be1 StorageAccountType:StandardSSD_LRS Size:5
I1022 06:43:45.488018       1 azure_managedDiskController.go:249] azureDisk - created new MD Name:pvc-d6c052b6-f54f-44e8-a567-4abeecf87be1 StorageAccountType:StandardSSD_LRS Size:5
I1022 06:43:45.488071       1 controllerserver.go:348] create azure disk(pvc-d6c052b6-f54f-44e8-a567-4abeecf87be1) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) tags(map[kubernetes.io-created-for-pv-name:pvc-d6c052b6-f54f-44e8-a567-4abeecf87be1 kubernetes.io-created-for-pvc-name:test.csi.azure.comvnkfv kubernetes.io-created-for-pvc-namespace:volume-expand-4084]) successfully
I1022 06:43:45.488114       1 utils.go:102] GRPC response: {"volume":{"accessible_topology":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"capacity_bytes":5368709120,"content_source":{"Type":null},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-d6c052b6-f54f-44e8-a567-4abeecf87be1","csi.storage.k8s.io/pvc/name":"test.csi.azure.comvnkfv","csi.storage.k8s.io/pvc/namespace":"volume-expand-4084","requestedsizegib":"5"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-d6c052b6-f54f-44e8-a567-4abeecf87be1"}}
I1022 06:43:46.455887       1 azure_controller_vmss.go:225] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - detach disk(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-23114395-d906-41ff-a057-97298a933d12:pvc-23114395-d906-41ff-a057-97298a933d12 /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-36d4a527-6ebf-4355-aa89-9dc4be2d5626:pvc-36d4a527-6ebf-4355-aa89-9dc4be2d5626]) returned with <nil>
... skipping 16 lines ...
I1022 06:43:46.548388       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:43:46.548420       1 azure_controller_vmss.go:122] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-6894fdc9-9e02-446f-92bb-2b026c63a266:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-6894fdc9-9e02-446f-92bb-2b026c63a266  true false 0})])
I1022 06:43:46.575085       1 azure_controller_vmss.go:225] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000001) - detach disk(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-1943503f-e1dc-4825-81a8-b38b96df76b0:pvc-1943503f-e1dc-4825-81a8-b38b96df76b0]) returned with <nil>
I1022 06:43:46.575140       1 azure_controller_common.go:369] azureDisk - detach disk(pvc-1943503f-e1dc-4825-81a8-b38b96df76b0, /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-1943503f-e1dc-4825-81a8-b38b96df76b0) succeeded
I1022 06:43:46.575158       1 controllerserver.go:533] detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-1943503f-e1dc-4825-81a8-b38b96df76b0 from node k8s-agentpool1-22475134-vmss000001 successfully
I1022 06:43:46.575168       1 utils.go:102] GRPC response: {}
I1022 06:43:46.693157       1 azure_controller_vmss.go:134] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-6894fdc9-9e02-446f-92bb-2b026c63a266:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-6894fdc9-9e02-446f-92bb-2b026c63a266  true false 0})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING)
I1022 06:43:46.693363       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:43:46.693415       1 azure_vmss.go:198] Couldn't find VMSS VM with nodeName k8s-agentpool1-22475134-vmss000000, refreshing the cache(vmss: k8s-agentpool1-22475134-vmss, rg: kubetest-5p3qtork)
I1022 06:43:46.743455       1 azure_controller_common.go:242] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-36d4a527-6ebf-4355-aa89-9dc4be2d5626" lun 1 to node "k8s-agentpool1-22475134-vmss000000", diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-36d4a527-6ebf-4355-aa89-9dc4be2d5626:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-36d4a527-6ebf-4355-aa89-9dc4be2d5626  true false 1})]
I1022 06:43:46.743483       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:43:46.743497       1 azure_controller_vmss.go:122] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-36d4a527-6ebf-4355-aa89-9dc4be2d5626:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-36d4a527-6ebf-4355-aa89-9dc4be2d5626  true false 1})])
I1022 06:43:46.911254       1 azure_controller_vmss.go:134] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-36d4a527-6ebf-4355-aa89-9dc4be2d5626:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-36d4a527-6ebf-4355-aa89-9dc4be2d5626  true false 1})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING)
I1022 06:43:49.701109       1 utils.go:95] GRPC call: /csi.v1.Controller/DeleteVolume
I1022 06:43:49.701139       1 utils.go:96] GRPC request: {"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-23114395-d906-41ff-a057-97298a933d12"}
I1022 06:43:49.701217       1 controllerserver.go:389] deleting azure disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-23114395-d906-41ff-a057-97298a933d12)
I1022 06:43:55.570344       1 azure_managedDiskController.go:286] azureDisk - deleted a managed disk: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-23114395-d906-41ff-a057-97298a933d12
I1022 06:43:55.570371       1 controllerserver.go:391] delete azure disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-23114395-d906-41ff-a057-97298a933d12) returned with <nil>
I1022 06:43:55.570389       1 utils.go:102] GRPC response: {}
... skipping 24 lines ...
I1022 06:44:00.801482       1 azure_controller_vmss.go:213] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000001) - detach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-775fddc6-3d43-48ec-acb8-25cbf1f5427f:pvc-775fddc6-3d43-48ec-acb8-25cbf1f5427f])
I1022 06:44:02.836296       1 azure_managedDiskController.go:286] azureDisk - deleted a managed disk: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-8cb3be6a-7e14-407c-a360-45a8e702464a
I1022 06:44:02.836329       1 controllerserver.go:391] delete azure disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-8cb3be6a-7e14-407c-a360-45a8e702464a) returned with <nil>
I1022 06:44:02.836366       1 utils.go:102] GRPC response: {}
I1022 06:44:14.669177       1 utils.go:95] GRPC call: /csi.v1.Controller/CreateVolume
I1022 06:44:14.669202       1 utils.go:96] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"requisite":[{"segments":{"topology.test.csi.azure.com/zone":""}}]},"capacity_range":{"required_bytes":5368709120},"name":"pvc-2ead2c64-3a60-46c4-940f-249658572375","parameters":{"csi.storage.k8s.io/pv/name":"pvc-2ead2c64-3a60-46c4-940f-249658572375","csi.storage.k8s.io/pvc/name":"pvc-vdspj","csi.storage.k8s.io/pvc/namespace":"snapshotting-8809"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}],"volume_content_source":{"Type":{"Snapshot":{"snapshot_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/snapshots/snapshot-1048deec-dd9e-4141-ad47-2619f103e1f1"}}}}
I1022 06:44:14.687172       1 azure_diskclient.go:135] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-2ead2c64-3a60-46c4-940f-249658572375, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-2ead2c64-3a60-46c4-940f-249658572375' under resource group 'kubetest-5p3qtork' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I1022 06:44:14.687249       1 controllerserver.go:271] begin to create azure disk(pvc-2ead2c64-3a60-46c4-940f-249658572375) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) diskZone() maxShares(0)
I1022 06:44:14.687263       1 azure_managedDiskController.go:88] azureDisk - creating new managed Name:pvc-2ead2c64-3a60-46c4-940f-249658572375 StorageAccountType:StandardSSD_LRS Size:5
I1022 06:44:17.031048       1 azure_managedDiskController.go:249] azureDisk - created new MD Name:pvc-2ead2c64-3a60-46c4-940f-249658572375 StorageAccountType:StandardSSD_LRS Size:5
I1022 06:44:17.031124       1 controllerserver.go:348] create azure disk(pvc-2ead2c64-3a60-46c4-940f-249658572375) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) tags(map[kubernetes.io-created-for-pv-name:pvc-2ead2c64-3a60-46c4-940f-249658572375 kubernetes.io-created-for-pvc-name:pvc-vdspj kubernetes.io-created-for-pvc-namespace:snapshotting-8809]) successfully
I1022 06:44:17.031169       1 utils.go:102] GRPC response: {"volume":{"accessible_topology":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"capacity_bytes":5368709120,"content_source":{"Type":{"Snapshot":{"snapshot_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/snapshots/snapshot-1048deec-dd9e-4141-ad47-2619f103e1f1"}}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-2ead2c64-3a60-46c4-940f-249658572375","csi.storage.k8s.io/pvc/name":"pvc-vdspj","csi.storage.k8s.io/pvc/namespace":"snapshotting-8809","requestedsizegib":"5"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-2ead2c64-3a60-46c4-940f-249658572375"}}
I1022 06:44:18.379232       1 utils.go:95] GRPC call: /csi.v1.Controller/DeleteVolume
... skipping 5 lines ...
I1022 06:44:19.791805       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:44:19.791826       1 controllerserver.go:461] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-2ead2c64-3a60-46c4-940f-249658572375" to node "k8s-agentpool1-22475134-vmss000000"
I1022 06:44:19.791857       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:44:19.791921       1 azure_controller_common.go:242] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-2ead2c64-3a60-46c4-940f-249658572375" lun 2 to node "k8s-agentpool1-22475134-vmss000000", diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-2ead2c64-3a60-46c4-940f-249658572375:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-2ead2c64-3a60-46c4-940f-249658572375  true false 2})]
I1022 06:44:19.791940       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:44:19.791955       1 azure_controller_vmss.go:122] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-2ead2c64-3a60-46c4-940f-249658572375:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-2ead2c64-3a60-46c4-940f-249658572375  true false 2})])
I1022 06:44:19.984840       1 azure_controller_vmss.go:134] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-2ead2c64-3a60-46c4-940f-249658572375:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-2ead2c64-3a60-46c4-940f-249658572375  true false 2})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING)
I1022 06:44:20.953124       1 utils.go:95] GRPC call: /csi.v1.Controller/ControllerUnpublishVolume
I1022 06:44:20.953152       1 utils.go:96] GRPC request: {"node_id":"k8s-agentpool1-22475134-vmss000000","volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-36d4a527-6ebf-4355-aa89-9dc4be2d5626"}
I1022 06:44:20.953256       1 controllerserver.go:524] Trying to detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-36d4a527-6ebf-4355-aa89-9dc4be2d5626 from node k8s-agentpool1-22475134-vmss000000
I1022 06:44:20.953274       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:44:20.953296       1 azure_vmss.go:198] Couldn't find VMSS VM with nodeName k8s-agentpool1-22475134-vmss000000, refreshing the cache(vmss: k8s-agentpool1-22475134-vmss, rg: kubetest-5p3qtork)
I1022 06:44:20.967224       1 utils.go:95] GRPC call: /csi.v1.Controller/ControllerUnpublishVolume
... skipping 19 lines ...
I1022 06:44:23.593083       1 controllerserver.go:391] delete azure disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-d6c052b6-f54f-44e8-a567-4abeecf87be1) returned with <nil>
I1022 06:44:23.593100       1 utils.go:102] GRPC response: {}
I1022 06:44:24.050301       1 controllerserver.go:918] create snapshot(snapshot-c32c3e47-e8ba-4aef-85a9-b15a6a2e0d5c) under rg(kubetest-5p3qtork) successfully
I1022 06:44:24.073727       1 utils.go:102] GRPC response: {"snapshot":{"creation_time":{"nanos":896745100,"seconds":1634885061},"ready_to_use":true,"size_bytes":5368709120,"snapshot_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/snapshots/snapshot-c32c3e47-e8ba-4aef-85a9-b15a6a2e0d5c","source_volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-6894fdc9-9e02-446f-92bb-2b026c63a266"}}
I1022 06:44:30.015078       1 utils.go:95] GRPC call: /csi.v1.Controller/CreateVolume
I1022 06:44:30.015108       1 utils.go:96] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"requisite":[{"segments":{"topology.test.csi.azure.com/zone":""}}]},"capacity_range":{"required_bytes":5368709120},"name":"pvc-2d20a1a1-33a8-41eb-a444-7d7f46969228","parameters":{"csi.storage.k8s.io/pv/name":"pvc-2d20a1a1-33a8-41eb-a444-7d7f46969228","csi.storage.k8s.io/pvc/name":"test.csi.azure.comztpm2","csi.storage.k8s.io/pvc/namespace":"provisioning-2162"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]}
I1022 06:44:30.048478       1 azure_diskclient.go:135] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-2d20a1a1-33a8-41eb-a444-7d7f46969228, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-2d20a1a1-33a8-41eb-a444-7d7f46969228' under resource group 'kubetest-5p3qtork' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I1022 06:44:30.048549       1 controllerserver.go:271] begin to create azure disk(pvc-2d20a1a1-33a8-41eb-a444-7d7f46969228) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) diskZone() maxShares(0)
I1022 06:44:30.048562       1 azure_managedDiskController.go:88] azureDisk - creating new managed Name:pvc-2d20a1a1-33a8-41eb-a444-7d7f46969228 StorageAccountType:StandardSSD_LRS Size:5
I1022 06:44:31.020538       1 utils.go:95] GRPC call: /csi.v1.Controller/DeleteVolume
I1022 06:44:31.020562       1 utils.go:96] GRPC request: {"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-775fddc6-3d43-48ec-acb8-25cbf1f5427f"}
I1022 06:44:31.020646       1 controllerserver.go:389] deleting azure disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-775fddc6-3d43-48ec-acb8-25cbf1f5427f)
I1022 06:44:32.394337       1 azure_managedDiskController.go:249] azureDisk - created new MD Name:pvc-2d20a1a1-33a8-41eb-a444-7d7f46969228 StorageAccountType:StandardSSD_LRS Size:5
... skipping 6 lines ...
I1022 06:44:34.805601       1 azure_vmss.go:198] Couldn't find VMSS VM with nodeName k8s-agentpool1-22475134-vmss000001, refreshing the cache(vmss: k8s-agentpool1-22475134-vmss, rg: kubetest-5p3qtork)
I1022 06:44:34.887462       1 controllerserver.go:461] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-2d20a1a1-33a8-41eb-a444-7d7f46969228" to node "k8s-agentpool1-22475134-vmss000001"
I1022 06:44:34.887527       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000001" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:44:34.887541       1 azure_controller_common.go:242] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-2d20a1a1-33a8-41eb-a444-7d7f46969228" lun 0 to node "k8s-agentpool1-22475134-vmss000001", diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-2d20a1a1-33a8-41eb-a444-7d7f46969228:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-2d20a1a1-33a8-41eb-a444-7d7f46969228  true false 0})]
I1022 06:44:34.887555       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000001" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:44:34.887566       1 azure_controller_vmss.go:122] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-2d20a1a1-33a8-41eb-a444-7d7f46969228:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-2d20a1a1-33a8-41eb-a444-7d7f46969228  true false 0})])
I1022 06:44:35.050515       1 azure_controller_vmss.go:134] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-2d20a1a1-33a8-41eb-a444-7d7f46969228:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-2d20a1a1-33a8-41eb-a444-7d7f46969228  true false 0})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING)
I1022 06:44:35.094326       1 controllerserver.go:465] Attach operation successful: volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-2ead2c64-3a60-46c4-940f-249658572375" attached to node "k8s-agentpool1-22475134-vmss000000".
I1022 06:44:35.094352       1 controllerserver.go:485] attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-2ead2c64-3a60-46c4-940f-249658572375" to node "k8s-agentpool1-22475134-vmss000000" successfully
I1022 06:44:35.094385       1 utils.go:102] GRPC response: {"publish_context":{"LUN":"2"}}
I1022 06:44:36.213736       1 azure_managedDiskController.go:286] azureDisk - deleted a managed disk: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-775fddc6-3d43-48ec-acb8-25cbf1f5427f
I1022 06:44:36.213764       1 controllerserver.go:391] delete azure disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-775fddc6-3d43-48ec-acb8-25cbf1f5427f) returned with <nil>
I1022 06:44:36.213796       1 utils.go:102] GRPC response: {}
... skipping 7 lines ...
I1022 06:44:36.347145       1 azure_controller_vmss.go:213] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - detach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-6894fdc9-9e02-446f-92bb-2b026c63a266:pvc-6894fdc9-9e02-446f-92bb-2b026c63a266])
I1022 06:44:40.507408       1 utils.go:95] GRPC call: /csi.v1.Controller/DeleteSnapshot
I1022 06:44:40.507449       1 utils.go:96] GRPC request: {"snapshot_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/snapshots/snapshot-1048deec-dd9e-4141-ad47-2619f103e1f1"}
I1022 06:44:40.507579       1 controllerserver.go:956] begin to delete snapshot(snapshot-1048deec-dd9e-4141-ad47-2619f103e1f1) under rg(kubetest-5p3qtork)
I1022 06:44:42.208908       1 utils.go:95] GRPC call: /csi.v1.Controller/CreateVolume
I1022 06:44:42.208949       1 utils.go:96] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"requisite":[{"segments":{"topology.test.csi.azure.com/zone":""}}]},"capacity_range":{"required_bytes":5368709120},"name":"pvc-16d55a1f-a769-40b5-8384-92940fcc54f1","parameters":{"csi.storage.k8s.io/pv/name":"pvc-16d55a1f-a769-40b5-8384-92940fcc54f1","csi.storage.k8s.io/pvc/name":"test.csi.azure.com4zrm6","csi.storage.k8s.io/pvc/namespace":"fsgroupchangepolicy-1902"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]}
I1022 06:44:42.231416       1 azure_diskclient.go:135] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-16d55a1f-a769-40b5-8384-92940fcc54f1, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-16d55a1f-a769-40b5-8384-92940fcc54f1' under resource group 'kubetest-5p3qtork' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I1022 06:44:42.231495       1 controllerserver.go:271] begin to create azure disk(pvc-16d55a1f-a769-40b5-8384-92940fcc54f1) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) diskZone() maxShares(0)
I1022 06:44:42.231507       1 azure_managedDiskController.go:88] azureDisk - creating new managed Name:pvc-16d55a1f-a769-40b5-8384-92940fcc54f1 StorageAccountType:StandardSSD_LRS Size:5
I1022 06:44:44.557031       1 azure_managedDiskController.go:249] azureDisk - created new MD Name:pvc-16d55a1f-a769-40b5-8384-92940fcc54f1 StorageAccountType:StandardSSD_LRS Size:5
I1022 06:44:44.557083       1 controllerserver.go:348] create azure disk(pvc-16d55a1f-a769-40b5-8384-92940fcc54f1) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) tags(map[kubernetes.io-created-for-pv-name:pvc-16d55a1f-a769-40b5-8384-92940fcc54f1 kubernetes.io-created-for-pvc-name:test.csi.azure.com4zrm6 kubernetes.io-created-for-pvc-namespace:fsgroupchangepolicy-1902]) successfully
I1022 06:44:44.557103       1 utils.go:102] GRPC response: {"volume":{"accessible_topology":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"capacity_bytes":5368709120,"content_source":{"Type":null},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-16d55a1f-a769-40b5-8384-92940fcc54f1","csi.storage.k8s.io/pvc/name":"test.csi.azure.com4zrm6","csi.storage.k8s.io/pvc/namespace":"fsgroupchangepolicy-1902","requestedsizegib":"5"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-16d55a1f-a769-40b5-8384-92940fcc54f1"}}
I1022 06:44:45.681265       1 controllerserver.go:961] delete snapshot(snapshot-1048deec-dd9e-4141-ad47-2619f103e1f1) under rg(kubetest-5p3qtork) successfully
... skipping 12 lines ...
I1022 06:44:51.669321       1 utils.go:102] GRPC response: {}
I1022 06:44:51.669415       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:44:51.669437       1 azure_vmss.go:198] Couldn't find VMSS VM with nodeName k8s-agentpool1-22475134-vmss000000, refreshing the cache(vmss: k8s-agentpool1-22475134-vmss, rg: kubetest-5p3qtork)
I1022 06:44:51.736217       1 azure_controller_common.go:242] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-16d55a1f-a769-40b5-8384-92940fcc54f1" lun 0 to node "k8s-agentpool1-22475134-vmss000000", diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-16d55a1f-a769-40b5-8384-92940fcc54f1:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-16d55a1f-a769-40b5-8384-92940fcc54f1  true false 0})]
I1022 06:44:51.736248       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:44:51.736281       1 azure_controller_vmss.go:122] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-16d55a1f-a769-40b5-8384-92940fcc54f1:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-16d55a1f-a769-40b5-8384-92940fcc54f1  true false 0})])
I1022 06:44:51.888397       1 azure_controller_vmss.go:134] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-16d55a1f-a769-40b5-8384-92940fcc54f1:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-16d55a1f-a769-40b5-8384-92940fcc54f1  true false 0})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING)
I1022 06:44:53.226571       1 utils.go:95] GRPC call: /csi.v1.Controller/ControllerPublishVolume
I1022 06:44:53.226597       1 utils.go:96] GRPC request: {"node_id":"k8s-agentpool1-22475134-vmss000000","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-6894fdc9-9e02-446f-92bb-2b026c63a266","csi.storage.k8s.io/pvc/name":"test.csi.azure.combkzh9","csi.storage.k8s.io/pvc/namespace":"snapshotting-5267","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1634884037556-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-6894fdc9-9e02-446f-92bb-2b026c63a266"}
I1022 06:44:53.257500       1 controllerserver.go:440] GetDiskLun returned: <nil>. Initiating attaching volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-6894fdc9-9e02-446f-92bb-2b026c63a266" to node "k8s-agentpool1-22475134-vmss000000".
I1022 06:44:53.257528       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:44:53.257548       1 azure_vmss.go:198] Couldn't find VMSS VM with nodeName k8s-agentpool1-22475134-vmss000000, refreshing the cache(vmss: k8s-agentpool1-22475134-vmss, rg: kubetest-5p3qtork)
I1022 06:44:53.350918       1 controllerserver.go:461] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-6894fdc9-9e02-446f-92bb-2b026c63a266" to node "k8s-agentpool1-22475134-vmss000000"
I1022 06:44:53.350958       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:44:53.350978       1 azure_controller_common.go:242] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-6894fdc9-9e02-446f-92bb-2b026c63a266" lun 1 to node "k8s-agentpool1-22475134-vmss000000", diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-6894fdc9-9e02-446f-92bb-2b026c63a266:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-6894fdc9-9e02-446f-92bb-2b026c63a266  true false 1})]
I1022 06:44:53.350994       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:44:53.351009       1 azure_controller_vmss.go:122] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-6894fdc9-9e02-446f-92bb-2b026c63a266:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-6894fdc9-9e02-446f-92bb-2b026c63a266  true false 1})])
I1022 06:44:53.538919       1 azure_controller_vmss.go:134] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-6894fdc9-9e02-446f-92bb-2b026c63a266:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-6894fdc9-9e02-446f-92bb-2b026c63a266  true false 1})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING)
I1022 06:45:11.262075       1 utils.go:95] GRPC call: /csi.v1.Controller/ControllerUnpublishVolume
I1022 06:45:11.262128       1 utils.go:96] GRPC request: {"node_id":"k8s-agentpool1-22475134-vmss000001","volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-2d20a1a1-33a8-41eb-a444-7d7f46969228"}
I1022 06:45:11.262220       1 controllerserver.go:524] Trying to detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-2d20a1a1-33a8-41eb-a444-7d7f46969228 from node k8s-agentpool1-22475134-vmss000001
I1022 06:45:11.262235       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000001" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:45:11.262277       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000001" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:45:11.262293       1 azure_controller_common.go:346] Trying to detach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-2d20a1a1-33a8-41eb-a444-7d7f46969228" from node "k8s-agentpool1-22475134-vmss000001", diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-2d20a1a1-33a8-41eb-a444-7d7f46969228:pvc-2d20a1a1-33a8-41eb-a444-7d7f46969228]
I1022 06:45:11.262331       1 azure_controller_vmss.go:174] azureDisk - detach disk: name "pvc-2d20a1a1-33a8-41eb-a444-7d7f46969228" uri "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-2d20a1a1-33a8-41eb-a444-7d7f46969228"
I1022 06:45:11.262345       1 azure_controller_vmss.go:213] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000001) - detach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-2d20a1a1-33a8-41eb-a444-7d7f46969228:pvc-2d20a1a1-33a8-41eb-a444-7d7f46969228])
I1022 06:45:18.165238       1 utils.go:95] GRPC call: /csi.v1.Controller/CreateVolume
I1022 06:45:18.165264       1 utils.go:96] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"requisite":[{"segments":{"topology.test.csi.azure.com/zone":""}}]},"capacity_range":{"required_bytes":5368709120},"name":"pvc-7fb42b9e-a2c0-493e-a731-9cc913162e55","parameters":{"csi.storage.k8s.io/pv/name":"pvc-7fb42b9e-a2c0-493e-a731-9cc913162e55","csi.storage.k8s.io/pvc/name":"test.csi.azure.comt4kcw","csi.storage.k8s.io/pvc/namespace":"volume-9223"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]}
I1022 06:45:18.204536       1 azure_diskclient.go:135] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-7fb42b9e-a2c0-493e-a731-9cc913162e55, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-7fb42b9e-a2c0-493e-a731-9cc913162e55' under resource group 'kubetest-5p3qtork' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I1022 06:45:18.204631       1 controllerserver.go:271] begin to create azure disk(pvc-7fb42b9e-a2c0-493e-a731-9cc913162e55) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) diskZone() maxShares(0)
I1022 06:45:18.204645       1 azure_managedDiskController.go:88] azureDisk - creating new managed Name:pvc-7fb42b9e-a2c0-493e-a731-9cc913162e55 StorageAccountType:StandardSSD_LRS Size:5
I1022 06:45:19.891372       1 azure_vmss.go:319] Couldn't find VMSS VM with scaleSetName "k8s-agentpool1-22475134-vmss" and instanceID "0", refreshing the cache
I1022 06:45:20.273894       1 utils.go:95] GRPC call: /csi.v1.Controller/DeleteVolume
I1022 06:45:20.273973       1 utils.go:96] GRPC request: {"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-36d4a527-6ebf-4355-aa89-9dc4be2d5626"}
I1022 06:45:20.274087       1 controllerserver.go:389] deleting azure disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-36d4a527-6ebf-4355-aa89-9dc4be2d5626)
... skipping 15 lines ...
I1022 06:45:22.895073       1 controllerserver.go:461] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-7fb42b9e-a2c0-493e-a731-9cc913162e55" to node "k8s-agentpool1-22475134-vmss000001"
I1022 06:45:26.390699       1 azure_managedDiskController.go:286] azureDisk - deleted a managed disk: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-36d4a527-6ebf-4355-aa89-9dc4be2d5626
I1022 06:45:26.390729       1 controllerserver.go:391] delete azure disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-36d4a527-6ebf-4355-aa89-9dc4be2d5626) returned with <nil>
I1022 06:45:26.390761       1 utils.go:102] GRPC response: {}
I1022 06:45:31.862161       1 utils.go:95] GRPC call: /csi.v1.Controller/CreateVolume
I1022 06:45:31.862187       1 utils.go:96] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"requisite":[{"segments":{"topology.test.csi.azure.com/zone":""}}]},"capacity_range":{"required_bytes":5368709120},"name":"pvc-f85949af-4e03-4a6f-b472-17ae34d61122","parameters":{"csi.storage.k8s.io/pv/name":"pvc-f85949af-4e03-4a6f-b472-17ae34d61122","csi.storage.k8s.io/pvc/name":"test.csi.azure.comdvtj7","csi.storage.k8s.io/pvc/namespace":"provisioning-7956"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]}
I1022 06:45:31.977061       1 azure_diskclient.go:135] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-f85949af-4e03-4a6f-b472-17ae34d61122, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-f85949af-4e03-4a6f-b472-17ae34d61122' under resource group 'kubetest-5p3qtork' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I1022 06:45:31.977182       1 controllerserver.go:271] begin to create azure disk(pvc-f85949af-4e03-4a6f-b472-17ae34d61122) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) diskZone() maxShares(0)
I1022 06:45:31.977197       1 azure_managedDiskController.go:88] azureDisk - creating new managed Name:pvc-f85949af-4e03-4a6f-b472-17ae34d61122 StorageAccountType:StandardSSD_LRS Size:5
I1022 06:45:34.343193       1 azure_managedDiskController.go:249] azureDisk - created new MD Name:pvc-f85949af-4e03-4a6f-b472-17ae34d61122 StorageAccountType:StandardSSD_LRS Size:5
I1022 06:45:34.343250       1 controllerserver.go:348] create azure disk(pvc-f85949af-4e03-4a6f-b472-17ae34d61122) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) tags(map[kubernetes.io-created-for-pv-name:pvc-f85949af-4e03-4a6f-b472-17ae34d61122 kubernetes.io-created-for-pvc-name:test.csi.azure.comdvtj7 kubernetes.io-created-for-pvc-namespace:provisioning-7956]) successfully
I1022 06:45:34.343393       1 utils.go:102] GRPC response: {"volume":{"accessible_topology":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"capacity_bytes":5368709120,"content_source":{"Type":null},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-f85949af-4e03-4a6f-b472-17ae34d61122","csi.storage.k8s.io/pvc/name":"test.csi.azure.comdvtj7","csi.storage.k8s.io/pvc/namespace":"provisioning-7956","requestedsizegib":"5"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-f85949af-4e03-4a6f-b472-17ae34d61122"}}
I1022 06:45:36.561303       1 utils.go:95] GRPC call: /csi.v1.Controller/ControllerPublishVolume
... skipping 21 lines ...
I1022 06:45:36.742394       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000001" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:45:36.742417       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:45:36.742438       1 azure_vmss.go:198] Couldn't find VMSS VM with nodeName k8s-agentpool1-22475134-vmss000000, refreshing the cache(vmss: k8s-agentpool1-22475134-vmss, rg: kubetest-5p3qtork)
I1022 06:45:36.787067       1 azure_controller_common.go:242] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-f85949af-4e03-4a6f-b472-17ae34d61122" lun 2 to node "k8s-agentpool1-22475134-vmss000000", diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-f85949af-4e03-4a6f-b472-17ae34d61122:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-f85949af-4e03-4a6f-b472-17ae34d61122  true false 2})]
I1022 06:45:36.787096       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:45:36.787129       1 azure_controller_vmss.go:122] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-f85949af-4e03-4a6f-b472-17ae34d61122:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-f85949af-4e03-4a6f-b472-17ae34d61122  true false 2})])
I1022 06:45:36.942435       1 azure_controller_vmss.go:134] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-f85949af-4e03-4a6f-b472-17ae34d61122:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-f85949af-4e03-4a6f-b472-17ae34d61122  true false 2})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING)
I1022 06:45:36.963030       1 azure_controller_vmss.go:134] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-7fb42b9e-a2c0-493e-a731-9cc913162e55:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-7fb42b9e-a2c0-493e-a731-9cc913162e55  true false 0})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING)
I1022 06:45:36.963233       1 azure_controller_common.go:346] Trying to detach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-2d20a1a1-33a8-41eb-a444-7d7f46969228" from node "k8s-agentpool1-22475134-vmss000001", diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-2d20a1a1-33a8-41eb-a444-7d7f46969228:pvc-2d20a1a1-33a8-41eb-a444-7d7f46969228]
I1022 06:45:36.963572       1 azure_vmss.go:198] Couldn't find VMSS VM with nodeName k8s-agentpool1-22475134-vmss000001, refreshing the cache(vmss: k8s-agentpool1-22475134-vmss, rg: kubetest-5p3qtork)
E1022 06:45:37.063902       1 azure_controller_vmss.go:183] detach azure disk on node(k8s-agentpool1-22475134-vmss000001): disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-2d20a1a1-33a8-41eb-a444-7d7f46969228:pvc-2d20a1a1-33a8-41eb-a444-7d7f46969228]) not found
I1022 06:45:37.063928       1 azure_controller_vmss.go:213] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000001) - detach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-2d20a1a1-33a8-41eb-a444-7d7f46969228:pvc-2d20a1a1-33a8-41eb-a444-7d7f46969228])
I1022 06:45:37.137975       1 controllerserver.go:465] Attach operation successful: volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-16d55a1f-a769-40b5-8384-92940fcc54f1" attached to node "k8s-agentpool1-22475134-vmss000000".
I1022 06:45:37.138000       1 controllerserver.go:485] attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-16d55a1f-a769-40b5-8384-92940fcc54f1" to node "k8s-agentpool1-22475134-vmss000000" successfully
... skipping 58 lines ...
I1022 06:46:11.867948       1 azure_controller_vmss.go:213] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000001) - detach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-7fb42b9e-a2c0-493e-a731-9cc913162e55:pvc-7fb42b9e-a2c0-493e-a731-9cc913162e55])
I1022 06:46:12.012682       1 azure_managedDiskController.go:286] azureDisk - deleted a managed disk: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-2d20a1a1-33a8-41eb-a444-7d7f46969228
I1022 06:46:12.012709       1 controllerserver.go:391] delete azure disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-2d20a1a1-33a8-41eb-a444-7d7f46969228) returned with <nil>
I1022 06:46:12.012743       1 utils.go:102] GRPC response: {}
I1022 06:46:16.755469       1 utils.go:95] GRPC call: /csi.v1.Controller/CreateVolume
I1022 06:46:16.755496       1 utils.go:96] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"requisite":[{"segments":{"topology.test.csi.azure.com/zone":""}}]},"capacity_range":{"required_bytes":5368709120},"name":"pvc-1d31be02-fc07-4482-97ed-0d5e45609233","parameters":{"csi.storage.k8s.io/pv/name":"pvc-1d31be02-fc07-4482-97ed-0d5e45609233","csi.storage.k8s.io/pvc/name":"test.csi.azure.com8rw8g","csi.storage.k8s.io/pvc/namespace":"volume-2350"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]}
I1022 06:46:16.774083       1 azure_diskclient.go:135] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-1d31be02-fc07-4482-97ed-0d5e45609233, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-1d31be02-fc07-4482-97ed-0d5e45609233' under resource group 'kubetest-5p3qtork' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I1022 06:46:16.774159       1 controllerserver.go:271] begin to create azure disk(pvc-1d31be02-fc07-4482-97ed-0d5e45609233) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) diskZone() maxShares(0)
I1022 06:46:16.774172       1 azure_managedDiskController.go:88] azureDisk - creating new managed Name:pvc-1d31be02-fc07-4482-97ed-0d5e45609233 StorageAccountType:StandardSSD_LRS Size:5
I1022 06:46:19.193525       1 azure_managedDiskController.go:249] azureDisk - created new MD Name:pvc-1d31be02-fc07-4482-97ed-0d5e45609233 StorageAccountType:StandardSSD_LRS Size:5
I1022 06:46:19.193578       1 controllerserver.go:348] create azure disk(pvc-1d31be02-fc07-4482-97ed-0d5e45609233) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) tags(map[kubernetes.io-created-for-pv-name:pvc-1d31be02-fc07-4482-97ed-0d5e45609233 kubernetes.io-created-for-pvc-name:test.csi.azure.com8rw8g kubernetes.io-created-for-pvc-namespace:volume-2350]) successfully
I1022 06:46:19.193600       1 utils.go:102] GRPC response: {"volume":{"accessible_topology":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"capacity_bytes":5368709120,"content_source":{"Type":null},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-1d31be02-fc07-4482-97ed-0d5e45609233","csi.storage.k8s.io/pvc/name":"test.csi.azure.com8rw8g","csi.storage.k8s.io/pvc/namespace":"volume-2350","requestedsizegib":"5"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-1d31be02-fc07-4482-97ed-0d5e45609233"}}
I1022 06:46:21.434675       1 utils.go:95] GRPC call: /csi.v1.Controller/ControllerPublishVolume
... skipping 24 lines ...
I1022 06:46:37.932484       1 utils.go:102] GRPC response: {}
I1022 06:46:40.974354       1 azure_managedDiskController.go:286] azureDisk - deleted a managed disk: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-1943503f-e1dc-4825-81a8-b38b96df76b0
I1022 06:46:40.974378       1 controllerserver.go:391] delete azure disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-1943503f-e1dc-4825-81a8-b38b96df76b0) returned with <nil>
I1022 06:46:40.974409       1 utils.go:102] GRPC response: {}
I1022 06:46:43.882700       1 utils.go:95] GRPC call: /csi.v1.Controller/CreateVolume
I1022 06:46:43.882724       1 utils.go:96] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"requisite":[{"segments":{"topology.test.csi.azure.com/zone":""}}]},"capacity_range":{"required_bytes":5368709120},"name":"pvc-3205115f-50fd-40ed-84c1-a7eddd360123","parameters":{"csi.storage.k8s.io/pv/name":"pvc-3205115f-50fd-40ed-84c1-a7eddd360123","csi.storage.k8s.io/pvc/name":"test.csi.azure.com6xqxc","csi.storage.k8s.io/pvc/namespace":"provisioning-6097"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]}
I1022 06:46:43.901819       1 azure_diskclient.go:135] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-3205115f-50fd-40ed-84c1-a7eddd360123, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-3205115f-50fd-40ed-84c1-a7eddd360123' under resource group 'kubetest-5p3qtork' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I1022 06:46:43.901900       1 controllerserver.go:271] begin to create azure disk(pvc-3205115f-50fd-40ed-84c1-a7eddd360123) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) diskZone() maxShares(0)
I1022 06:46:43.901931       1 azure_managedDiskController.go:88] azureDisk - creating new managed Name:pvc-3205115f-50fd-40ed-84c1-a7eddd360123 StorageAccountType:StandardSSD_LRS Size:5
I1022 06:46:46.244650       1 azure_managedDiskController.go:249] azureDisk - created new MD Name:pvc-3205115f-50fd-40ed-84c1-a7eddd360123 StorageAccountType:StandardSSD_LRS Size:5
I1022 06:46:46.244703       1 controllerserver.go:348] create azure disk(pvc-3205115f-50fd-40ed-84c1-a7eddd360123) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) tags(map[kubernetes.io-created-for-pv-name:pvc-3205115f-50fd-40ed-84c1-a7eddd360123 kubernetes.io-created-for-pvc-name:test.csi.azure.com6xqxc kubernetes.io-created-for-pvc-namespace:provisioning-6097]) successfully
I1022 06:46:46.244742       1 utils.go:102] GRPC response: {"volume":{"accessible_topology":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"capacity_bytes":5368709120,"content_source":{"Type":null},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-3205115f-50fd-40ed-84c1-a7eddd360123","csi.storage.k8s.io/pvc/name":"test.csi.azure.com6xqxc","csi.storage.k8s.io/pvc/namespace":"provisioning-6097","requestedsizegib":"5"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-3205115f-50fd-40ed-84c1-a7eddd360123"}}
I1022 06:46:46.569558       1 utils.go:95] GRPC call: /csi.v1.Controller/CreateVolume
I1022 06:46:46.569598       1 utils.go:96] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"requisite":[{"segments":{"topology.test.csi.azure.com/zone":""}}]},"capacity_range":{"required_bytes":5368709120},"name":"pvc-84b462f9-4ddf-46af-81d8-d8eb861e965a","parameters":{"csi.storage.k8s.io/pv/name":"pvc-84b462f9-4ddf-46af-81d8-d8eb861e965a","csi.storage.k8s.io/pvc/name":"pvc-h8fw7","csi.storage.k8s.io/pvc/namespace":"snapshotting-5267"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}],"volume_content_source":{"Type":{"Snapshot":{"snapshot_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/snapshots/snapshot-c32c3e47-e8ba-4aef-85a9-b15a6a2e0d5c"}}}}
I1022 06:46:46.642480       1 azure_diskclient.go:135] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-84b462f9-4ddf-46af-81d8-d8eb861e965a, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-84b462f9-4ddf-46af-81d8-d8eb861e965a' under resource group 'kubetest-5p3qtork' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I1022 06:46:46.642556       1 controllerserver.go:271] begin to create azure disk(pvc-84b462f9-4ddf-46af-81d8-d8eb861e965a) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) diskZone() maxShares(0)
I1022 06:46:46.642573       1 azure_managedDiskController.go:88] azureDisk - creating new managed Name:pvc-84b462f9-4ddf-46af-81d8-d8eb861e965a StorageAccountType:StandardSSD_LRS Size:5
I1022 06:46:47.398729       1 azure_controller_vmss.go:225] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - detach disk(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-16d55a1f-a769-40b5-8384-92940fcc54f1:pvc-16d55a1f-a769-40b5-8384-92940fcc54f1]) returned with <nil>
I1022 06:46:47.398801       1 azure_controller_common.go:369] azureDisk - detach disk(pvc-16d55a1f-a769-40b5-8384-92940fcc54f1, /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-16d55a1f-a769-40b5-8384-92940fcc54f1) succeeded
I1022 06:46:47.398822       1 controllerserver.go:533] detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-16d55a1f-a769-40b5-8384-92940fcc54f1 from node k8s-agentpool1-22475134-vmss000000 successfully
I1022 06:46:47.398854       1 utils.go:102] GRPC response: {}
I1022 06:46:47.398952       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:46:47.398978       1 azure_vmss.go:198] Couldn't find VMSS VM with nodeName k8s-agentpool1-22475134-vmss000000, refreshing the cache(vmss: k8s-agentpool1-22475134-vmss, rg: kubetest-5p3qtork)
I1022 06:46:47.497796       1 azure_controller_common.go:242] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-1d31be02-fc07-4482-97ed-0d5e45609233" lun 0 to node "k8s-agentpool1-22475134-vmss000000", diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-1d31be02-fc07-4482-97ed-0d5e45609233:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-1d31be02-fc07-4482-97ed-0d5e45609233  true false 0})]
I1022 06:46:47.497830       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:46:47.497865       1 azure_controller_vmss.go:122] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-1d31be02-fc07-4482-97ed-0d5e45609233:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-1d31be02-fc07-4482-97ed-0d5e45609233  true false 0})])
I1022 06:46:47.680900       1 azure_controller_vmss.go:134] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-1d31be02-fc07-4482-97ed-0d5e45609233:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-1d31be02-fc07-4482-97ed-0d5e45609233  true false 0})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING)
I1022 06:46:48.636304       1 utils.go:95] GRPC call: /csi.v1.Controller/ControllerPublishVolume
I1022 06:46:48.636346       1 utils.go:96] GRPC request: {"node_id":"k8s-agentpool1-22475134-vmss000001","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-16d55a1f-a769-40b5-8384-92940fcc54f1","csi.storage.k8s.io/pvc/name":"test.csi.azure.com4zrm6","csi.storage.k8s.io/pvc/namespace":"fsgroupchangepolicy-1902","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1634884037556-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-16d55a1f-a769-40b5-8384-92940fcc54f1"}
I1022 06:46:48.636716       1 utils.go:95] GRPC call: /csi.v1.Controller/ControllerPublishVolume
I1022 06:46:48.636738       1 utils.go:96] GRPC request: {"node_id":"k8s-agentpool1-22475134-vmss000000","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-3205115f-50fd-40ed-84c1-a7eddd360123","csi.storage.k8s.io/pvc/name":"test.csi.azure.com6xqxc","csi.storage.k8s.io/pvc/namespace":"provisioning-6097","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1634884037556-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-3205115f-50fd-40ed-84c1-a7eddd360123"}
I1022 06:46:48.659428       1 controllerserver.go:440] GetDiskLun returned: <nil>. Initiating attaching volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-16d55a1f-a769-40b5-8384-92940fcc54f1" to node "k8s-agentpool1-22475134-vmss000001".
I1022 06:46:48.659453       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000001" is managed by VMSS since DisableAvailabilitySetNodes is set to true
... skipping 7 lines ...
I1022 06:46:48.731605       1 azure_vmss.go:198] Couldn't find VMSS VM with nodeName k8s-agentpool1-22475134-vmss000000, refreshing the cache(vmss: k8s-agentpool1-22475134-vmss, rg: kubetest-5p3qtork)
I1022 06:46:48.774841       1 controllerserver.go:461] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-3205115f-50fd-40ed-84c1-a7eddd360123" to node "k8s-agentpool1-22475134-vmss000000"
I1022 06:46:48.774887       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:46:48.774902       1 azure_controller_common.go:242] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-3205115f-50fd-40ed-84c1-a7eddd360123" lun 2 to node "k8s-agentpool1-22475134-vmss000000", diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-3205115f-50fd-40ed-84c1-a7eddd360123:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-3205115f-50fd-40ed-84c1-a7eddd360123  true false 2})]
I1022 06:46:48.774915       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:46:48.774927       1 azure_controller_vmss.go:122] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-3205115f-50fd-40ed-84c1-a7eddd360123:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-3205115f-50fd-40ed-84c1-a7eddd360123  true false 2})])
I1022 06:46:48.844869       1 azure_controller_vmss.go:134] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-16d55a1f-a769-40b5-8384-92940fcc54f1:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-16d55a1f-a769-40b5-8384-92940fcc54f1  true false 0})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING)
I1022 06:46:48.960960       1 azure_managedDiskController.go:249] azureDisk - created new MD Name:pvc-84b462f9-4ddf-46af-81d8-d8eb861e965a StorageAccountType:StandardSSD_LRS Size:5
I1022 06:46:48.961005       1 controllerserver.go:348] create azure disk(pvc-84b462f9-4ddf-46af-81d8-d8eb861e965a) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) tags(map[kubernetes.io-created-for-pv-name:pvc-84b462f9-4ddf-46af-81d8-d8eb861e965a kubernetes.io-created-for-pvc-name:pvc-h8fw7 kubernetes.io-created-for-pvc-namespace:snapshotting-5267]) successfully
I1022 06:46:48.961023       1 utils.go:102] GRPC response: {"volume":{"accessible_topology":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"capacity_bytes":5368709120,"content_source":{"Type":{"Snapshot":{"snapshot_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/snapshots/snapshot-c32c3e47-e8ba-4aef-85a9-b15a6a2e0d5c"}}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-84b462f9-4ddf-46af-81d8-d8eb861e965a","csi.storage.k8s.io/pvc/name":"pvc-h8fw7","csi.storage.k8s.io/pvc/namespace":"snapshotting-5267","requestedsizegib":"5"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-84b462f9-4ddf-46af-81d8-d8eb861e965a"}}
I1022 06:46:48.977770       1 azure_controller_vmss.go:134] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-3205115f-50fd-40ed-84c1-a7eddd360123:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-3205115f-50fd-40ed-84c1-a7eddd360123  true false 2})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING)
I1022 06:46:50.853313       1 utils.go:95] GRPC call: /csi.v1.Controller/ControllerPublishVolume
I1022 06:46:50.853356       1 utils.go:96] GRPC request: {"node_id":"k8s-agentpool1-22475134-vmss000000","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-84b462f9-4ddf-46af-81d8-d8eb861e965a","csi.storage.k8s.io/pvc/name":"pvc-h8fw7","csi.storage.k8s.io/pvc/namespace":"snapshotting-5267","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1634884037556-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-84b462f9-4ddf-46af-81d8-d8eb861e965a"}
I1022 06:46:50.877607       1 controllerserver.go:440] GetDiskLun returned: <nil>. Initiating attaching volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-84b462f9-4ddf-46af-81d8-d8eb861e965a" to node "k8s-agentpool1-22475134-vmss000000".
I1022 06:46:50.877650       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:46:50.877708       1 azure_vmss.go:198] Couldn't find VMSS VM with nodeName k8s-agentpool1-22475134-vmss000000, refreshing the cache(vmss: k8s-agentpool1-22475134-vmss, rg: kubetest-5p3qtork)
I1022 06:46:50.920962       1 controllerserver.go:461] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-84b462f9-4ddf-46af-81d8-d8eb861e965a" to node "k8s-agentpool1-22475134-vmss000000"
I1022 06:46:50.921010       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:46:50.921024       1 azure_controller_common.go:242] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-84b462f9-4ddf-46af-81d8-d8eb861e965a" lun 3 to node "k8s-agentpool1-22475134-vmss000000", diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-84b462f9-4ddf-46af-81d8-d8eb861e965a:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-84b462f9-4ddf-46af-81d8-d8eb861e965a  true false 3})]
I1022 06:46:50.921037       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:46:50.921050       1 azure_controller_vmss.go:122] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-84b462f9-4ddf-46af-81d8-d8eb861e965a:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-84b462f9-4ddf-46af-81d8-d8eb861e965a  true false 3})])
I1022 06:46:51.136503       1 azure_controller_vmss.go:134] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-84b462f9-4ddf-46af-81d8-d8eb861e965a:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-84b462f9-4ddf-46af-81d8-d8eb861e965a  true false 3})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING)
I1022 06:46:51.968054       1 utils.go:95] GRPC call: /csi.v1.Controller/ControllerUnpublishVolume
I1022 06:46:51.968078       1 utils.go:96] GRPC request: {"node_id":"k8s-agentpool1-22475134-vmss000000","volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-6894fdc9-9e02-446f-92bb-2b026c63a266"}
I1022 06:46:51.968230       1 controllerserver.go:524] Trying to detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-6894fdc9-9e02-446f-92bb-2b026c63a266 from node k8s-agentpool1-22475134-vmss000000
I1022 06:46:51.968240       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:46:51.968265       1 azure_vmss.go:198] Couldn't find VMSS VM with nodeName k8s-agentpool1-22475134-vmss000000, refreshing the cache(vmss: k8s-agentpool1-22475134-vmss, rg: kubetest-5p3qtork)
I1022 06:46:52.022994       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
... skipping 31 lines ...
I1022 06:47:06.338544       1 utils.go:102] GRPC response: {"publish_context":{"LUN":"3"}}
I1022 06:47:06.804542       1 azure_managedDiskController.go:286] azureDisk - deleted a managed disk: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-f85949af-4e03-4a6f-b472-17ae34d61122
I1022 06:47:06.804572       1 controllerserver.go:391] delete azure disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-f85949af-4e03-4a6f-b472-17ae34d61122) returned with <nil>
I1022 06:47:06.804604       1 utils.go:102] GRPC response: {}
I1022 06:47:11.649075       1 utils.go:95] GRPC call: /csi.v1.Controller/CreateVolume
I1022 06:47:11.649099       1 utils.go:96] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"requisite":[{"segments":{"topology.test.csi.azure.com/zone":""}}]},"capacity_range":{"required_bytes":5368709120},"name":"pvc-7b73b031-7bdf-4e53-ab88-3fc4fa2c4fda","parameters":{"csi.storage.k8s.io/pv/name":"pvc-7b73b031-7bdf-4e53-ab88-3fc4fa2c4fda","csi.storage.k8s.io/pvc/name":"test.csi.azure.comp8dl6","csi.storage.k8s.io/pvc/namespace":"volumelimits-928"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]}
I1022 06:47:11.668234       1 azure_diskclient.go:135] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-7b73b031-7bdf-4e53-ab88-3fc4fa2c4fda, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-7b73b031-7bdf-4e53-ab88-3fc4fa2c4fda' under resource group 'kubetest-5p3qtork' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I1022 06:47:11.668305       1 controllerserver.go:271] begin to create azure disk(pvc-7b73b031-7bdf-4e53-ab88-3fc4fa2c4fda) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) diskZone() maxShares(0)
I1022 06:47:11.668318       1 azure_managedDiskController.go:88] azureDisk - creating new managed Name:pvc-7b73b031-7bdf-4e53-ab88-3fc4fa2c4fda StorageAccountType:StandardSSD_LRS Size:5
I1022 06:47:14.034102       1 azure_managedDiskController.go:249] azureDisk - created new MD Name:pvc-7b73b031-7bdf-4e53-ab88-3fc4fa2c4fda StorageAccountType:StandardSSD_LRS Size:5
I1022 06:47:14.034159       1 controllerserver.go:348] create azure disk(pvc-7b73b031-7bdf-4e53-ab88-3fc4fa2c4fda) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) tags(map[kubernetes.io-created-for-pv-name:pvc-7b73b031-7bdf-4e53-ab88-3fc4fa2c4fda kubernetes.io-created-for-pvc-name:test.csi.azure.comp8dl6 kubernetes.io-created-for-pvc-namespace:volumelimits-928]) successfully
I1022 06:47:14.034197       1 utils.go:102] GRPC response: {"volume":{"accessible_topology":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"capacity_bytes":5368709120,"content_source":{"Type":null},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-7b73b031-7bdf-4e53-ab88-3fc4fa2c4fda","csi.storage.k8s.io/pvc/name":"test.csi.azure.comp8dl6","csi.storage.k8s.io/pvc/namespace":"volumelimits-928","requestedsizegib":"5"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-7b73b031-7bdf-4e53-ab88-3fc4fa2c4fda"}}
I1022 06:47:16.302527       1 utils.go:95] GRPC call: /csi.v1.Controller/CreateVolume
I1022 06:47:16.302554       1 utils.go:96] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"requisite":[{"segments":{"topology.test.csi.azure.com/zone":""}}]},"capacity_range":{"required_bytes":5368709120},"name":"pvc-7292229f-a145-4b90-84c0-8ad707acb8f1","parameters":{"csi.storage.k8s.io/pv/name":"pvc-7292229f-a145-4b90-84c0-8ad707acb8f1","csi.storage.k8s.io/pvc/name":"pvc-5hfc4","csi.storage.k8s.io/pvc/namespace":"volumelimits-928"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]}
I1022 06:47:16.325542       1 azure_diskclient.go:135] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-7292229f-a145-4b90-84c0-8ad707acb8f1, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-7292229f-a145-4b90-84c0-8ad707acb8f1' under resource group 'kubetest-5p3qtork' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I1022 06:47:16.325596       1 controllerserver.go:271] begin to create azure disk(pvc-7292229f-a145-4b90-84c0-8ad707acb8f1) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) diskZone() maxShares(0)
I1022 06:47:16.325639       1 azure_managedDiskController.go:88] azureDisk - creating new managed Name:pvc-7292229f-a145-4b90-84c0-8ad707acb8f1 StorageAccountType:StandardSSD_LRS Size:5
I1022 06:47:16.411292       1 utils.go:95] GRPC call: /csi.v1.Controller/CreateVolume
I1022 06:47:16.411314       1 utils.go:96] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"requisite":[{"segments":{"topology.test.csi.azure.com/zone":""}}]},"capacity_range":{"required_bytes":5368709120},"name":"pvc-7598fafa-b8d8-45fc-bd09-7abb6e913a34","parameters":{"csi.storage.k8s.io/pv/name":"pvc-7598fafa-b8d8-45fc-bd09-7abb6e913a34","csi.storage.k8s.io/pvc/name":"pvc-pxw5k","csi.storage.k8s.io/pvc/namespace":"volumelimits-928"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]}
I1022 06:47:16.432247       1 azure_diskclient.go:135] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-7598fafa-b8d8-45fc-bd09-7abb6e913a34, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-7598fafa-b8d8-45fc-bd09-7abb6e913a34' under resource group 'kubetest-5p3qtork' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I1022 06:47:16.432317       1 controllerserver.go:271] begin to create azure disk(pvc-7598fafa-b8d8-45fc-bd09-7abb6e913a34) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) diskZone() maxShares(0)
I1022 06:47:16.432330       1 azure_managedDiskController.go:88] azureDisk - creating new managed Name:pvc-7598fafa-b8d8-45fc-bd09-7abb6e913a34 StorageAccountType:StandardSSD_LRS Size:5
I1022 06:47:16.519765       1 utils.go:95] GRPC call: /csi.v1.Controller/CreateVolume
I1022 06:47:16.519823       1 utils.go:96] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"requisite":[{"segments":{"topology.test.csi.azure.com/zone":""}}]},"capacity_range":{"required_bytes":5368709120},"name":"pvc-625cb0a6-3ef0-4bb1-8a80-8d075355b851","parameters":{"csi.storage.k8s.io/pv/name":"pvc-625cb0a6-3ef0-4bb1-8a80-8d075355b851","csi.storage.k8s.io/pvc/name":"pvc-gxtth","csi.storage.k8s.io/pvc/namespace":"volumelimits-928"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]}
I1022 06:47:16.572056       1 azure_diskclient.go:135] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-625cb0a6-3ef0-4bb1-8a80-8d075355b851, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-625cb0a6-3ef0-4bb1-8a80-8d075355b851' under resource group 'kubetest-5p3qtork' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I1022 06:47:16.572145       1 controllerserver.go:271] begin to create azure disk(pvc-625cb0a6-3ef0-4bb1-8a80-8d075355b851) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) diskZone() maxShares(0)
I1022 06:47:16.572184       1 azure_managedDiskController.go:88] azureDisk - creating new managed Name:pvc-625cb0a6-3ef0-4bb1-8a80-8d075355b851 StorageAccountType:StandardSSD_LRS Size:5
I1022 06:47:16.629294       1 utils.go:95] GRPC call: /csi.v1.Controller/CreateVolume
I1022 06:47:16.629316       1 utils.go:96] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"requisite":[{"segments":{"topology.test.csi.azure.com/zone":""}}]},"capacity_range":{"required_bytes":5368709120},"name":"pvc-84c19a86-6835-4758-8251-2d417a93c3c7","parameters":{"csi.storage.k8s.io/pv/name":"pvc-84c19a86-6835-4758-8251-2d417a93c3c7","csi.storage.k8s.io/pvc/name":"pvc-l7r25","csi.storage.k8s.io/pvc/namespace":"volumelimits-928"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]}
I1022 06:47:16.655144       1 azure_diskclient.go:135] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-84c19a86-6835-4758-8251-2d417a93c3c7, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-84c19a86-6835-4758-8251-2d417a93c3c7' under resource group 'kubetest-5p3qtork' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I1022 06:47:16.655210       1 controllerserver.go:271] begin to create azure disk(pvc-84c19a86-6835-4758-8251-2d417a93c3c7) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) diskZone() maxShares(0)
I1022 06:47:16.655221       1 azure_managedDiskController.go:88] azureDisk - creating new managed Name:pvc-84c19a86-6835-4758-8251-2d417a93c3c7 StorageAccountType:StandardSSD_LRS Size:5
I1022 06:47:16.744432       1 utils.go:95] GRPC call: /csi.v1.Controller/CreateVolume
I1022 06:47:16.744453       1 utils.go:96] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"requisite":[{"segments":{"topology.test.csi.azure.com/zone":""}}]},"capacity_range":{"required_bytes":5368709120},"name":"pvc-a43477fa-15b0-4f50-92e3-0a58c87f24cd","parameters":{"csi.storage.k8s.io/pv/name":"pvc-a43477fa-15b0-4f50-92e3-0a58c87f24cd","csi.storage.k8s.io/pvc/name":"pvc-7q85h","csi.storage.k8s.io/pvc/namespace":"volumelimits-928"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]}
I1022 06:47:16.763194       1 azure_diskclient.go:135] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-a43477fa-15b0-4f50-92e3-0a58c87f24cd, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-a43477fa-15b0-4f50-92e3-0a58c87f24cd' under resource group 'kubetest-5p3qtork' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I1022 06:47:16.763269       1 controllerserver.go:271] begin to create azure disk(pvc-a43477fa-15b0-4f50-92e3-0a58c87f24cd) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) diskZone() maxShares(0)
I1022 06:47:16.763283       1 azure_managedDiskController.go:88] azureDisk - creating new managed Name:pvc-a43477fa-15b0-4f50-92e3-0a58c87f24cd StorageAccountType:StandardSSD_LRS Size:5
I1022 06:47:16.855970       1 utils.go:95] GRPC call: /csi.v1.Controller/CreateVolume
I1022 06:47:16.855990       1 utils.go:96] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"requisite":[{"segments":{"topology.test.csi.azure.com/zone":""}}]},"capacity_range":{"required_bytes":5368709120},"name":"pvc-d92c57dc-1538-4651-bc16-6cc2c3ef92cc","parameters":{"csi.storage.k8s.io/pv/name":"pvc-d92c57dc-1538-4651-bc16-6cc2c3ef92cc","csi.storage.k8s.io/pvc/name":"pvc-nmhhg","csi.storage.k8s.io/pvc/namespace":"volumelimits-928"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]}
I1022 06:47:16.897585       1 azure_diskclient.go:135] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-d92c57dc-1538-4651-bc16-6cc2c3ef92cc, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-d92c57dc-1538-4651-bc16-6cc2c3ef92cc' under resource group 'kubetest-5p3qtork' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I1022 06:47:16.897664       1 controllerserver.go:271] begin to create azure disk(pvc-d92c57dc-1538-4651-bc16-6cc2c3ef92cc) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) diskZone() maxShares(0)
I1022 06:47:16.897673       1 azure_managedDiskController.go:88] azureDisk - creating new managed Name:pvc-d92c57dc-1538-4651-bc16-6cc2c3ef92cc StorageAccountType:StandardSSD_LRS Size:5
I1022 06:47:16.976112       1 utils.go:95] GRPC call: /csi.v1.Controller/CreateVolume
I1022 06:47:16.976133       1 utils.go:96] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"requisite":[{"segments":{"topology.test.csi.azure.com/zone":""}}]},"capacity_range":{"required_bytes":5368709120},"name":"pvc-aba6fcc4-93be-4a03-b913-b0ad6dfd64e9","parameters":{"csi.storage.k8s.io/pv/name":"pvc-aba6fcc4-93be-4a03-b913-b0ad6dfd64e9","csi.storage.k8s.io/pvc/name":"pvc-7hlb8","csi.storage.k8s.io/pvc/namespace":"volumelimits-928"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]}
I1022 06:47:17.032504       1 azure_diskclient.go:135] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-aba6fcc4-93be-4a03-b913-b0ad6dfd64e9, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-aba6fcc4-93be-4a03-b913-b0ad6dfd64e9' under resource group 'kubetest-5p3qtork' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I1022 06:47:17.032561       1 controllerserver.go:271] begin to create azure disk(pvc-aba6fcc4-93be-4a03-b913-b0ad6dfd64e9) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) diskZone() maxShares(0)
I1022 06:47:17.032575       1 azure_managedDiskController.go:88] azureDisk - creating new managed Name:pvc-aba6fcc4-93be-4a03-b913-b0ad6dfd64e9 StorageAccountType:StandardSSD_LRS Size:5
I1022 06:47:17.077890       1 utils.go:95] GRPC call: /csi.v1.Controller/CreateVolume
I1022 06:47:17.077936       1 utils.go:96] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"requisite":[{"segments":{"topology.test.csi.azure.com/zone":""}}]},"capacity_range":{"required_bytes":5368709120},"name":"pvc-90b4af34-20b0-494b-a46f-a3471b81f63a","parameters":{"csi.storage.k8s.io/pv/name":"pvc-90b4af34-20b0-494b-a46f-a3471b81f63a","csi.storage.k8s.io/pvc/name":"pvc-8mtq5","csi.storage.k8s.io/pvc/namespace":"volumelimits-928"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]}
I1022 06:47:17.097603       1 azure_diskclient.go:135] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-90b4af34-20b0-494b-a46f-a3471b81f63a, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-90b4af34-20b0-494b-a46f-a3471b81f63a' under resource group 'kubetest-5p3qtork' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I1022 06:47:17.097679       1 controllerserver.go:271] begin to create azure disk(pvc-90b4af34-20b0-494b-a46f-a3471b81f63a) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) diskZone() maxShares(0)
I1022 06:47:17.097692       1 azure_managedDiskController.go:88] azureDisk - creating new managed Name:pvc-90b4af34-20b0-494b-a46f-a3471b81f63a StorageAccountType:StandardSSD_LRS Size:5
I1022 06:47:17.190568       1 utils.go:95] GRPC call: /csi.v1.Controller/CreateVolume
I1022 06:47:17.190590       1 utils.go:96] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"requisite":[{"segments":{"topology.test.csi.azure.com/zone":""}}]},"capacity_range":{"required_bytes":5368709120},"name":"pvc-6e5bb4b7-9c65-4f65-9a30-72575fc9afac","parameters":{"csi.storage.k8s.io/pv/name":"pvc-6e5bb4b7-9c65-4f65-9a30-72575fc9afac","csi.storage.k8s.io/pvc/name":"pvc-4nvg4","csi.storage.k8s.io/pvc/namespace":"volumelimits-928"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]}
I1022 06:47:17.209849       1 azure_diskclient.go:135] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-6e5bb4b7-9c65-4f65-9a30-72575fc9afac, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-6e5bb4b7-9c65-4f65-9a30-72575fc9afac' under resource group 'kubetest-5p3qtork' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I1022 06:47:17.209914       1 controllerserver.go:271] begin to create azure disk(pvc-6e5bb4b7-9c65-4f65-9a30-72575fc9afac) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) diskZone() maxShares(0)
I1022 06:47:17.209925       1 azure_managedDiskController.go:88] azureDisk - creating new managed Name:pvc-6e5bb4b7-9c65-4f65-9a30-72575fc9afac StorageAccountType:StandardSSD_LRS Size:5
I1022 06:47:17.298698       1 utils.go:95] GRPC call: /csi.v1.Controller/CreateVolume
I1022 06:47:17.298720       1 utils.go:96] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"requisite":[{"segments":{"topology.test.csi.azure.com/zone":""}}]},"capacity_range":{"required_bytes":5368709120},"name":"pvc-a38ef8f3-7817-4091-b6e0-7741e79fd589","parameters":{"csi.storage.k8s.io/pv/name":"pvc-a38ef8f3-7817-4091-b6e0-7741e79fd589","csi.storage.k8s.io/pvc/name":"pvc-r8496","csi.storage.k8s.io/pvc/namespace":"volumelimits-928"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]}
I1022 06:47:17.317163       1 azure_diskclient.go:135] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-a38ef8f3-7817-4091-b6e0-7741e79fd589, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-a38ef8f3-7817-4091-b6e0-7741e79fd589' under resource group 'kubetest-5p3qtork' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I1022 06:47:17.317258       1 controllerserver.go:271] begin to create azure disk(pvc-a38ef8f3-7817-4091-b6e0-7741e79fd589) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) diskZone() maxShares(0)
I1022 06:47:17.317278       1 azure_managedDiskController.go:88] azureDisk - creating new managed Name:pvc-a38ef8f3-7817-4091-b6e0-7741e79fd589 StorageAccountType:StandardSSD_LRS Size:5
I1022 06:47:17.407688       1 utils.go:95] GRPC call: /csi.v1.Controller/CreateVolume
I1022 06:47:17.407711       1 utils.go:96] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"requisite":[{"segments":{"topology.test.csi.azure.com/zone":""}}]},"capacity_range":{"required_bytes":5368709120},"name":"pvc-4468e1b3-f470-4e0c-9404-16d282d00932","parameters":{"csi.storage.k8s.io/pv/name":"pvc-4468e1b3-f470-4e0c-9404-16d282d00932","csi.storage.k8s.io/pvc/name":"pvc-5zlhj","csi.storage.k8s.io/pvc/namespace":"volumelimits-928"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]}
I1022 06:47:17.446483       1 azure_diskclient.go:135] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-4468e1b3-f470-4e0c-9404-16d282d00932, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-4468e1b3-f470-4e0c-9404-16d282d00932' under resource group 'kubetest-5p3qtork' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I1022 06:47:17.446544       1 controllerserver.go:271] begin to create azure disk(pvc-4468e1b3-f470-4e0c-9404-16d282d00932) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) diskZone() maxShares(0)
I1022 06:47:17.446561       1 azure_managedDiskController.go:88] azureDisk - creating new managed Name:pvc-4468e1b3-f470-4e0c-9404-16d282d00932 StorageAccountType:StandardSSD_LRS Size:5
I1022 06:47:17.447229       1 azure_controller_vmss.go:225] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - detach disk(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-6894fdc9-9e02-446f-92bb-2b026c63a266:pvc-6894fdc9-9e02-446f-92bb-2b026c63a266]) returned with <nil>
I1022 06:47:17.447283       1 azure_controller_common.go:369] azureDisk - detach disk(pvc-6894fdc9-9e02-446f-92bb-2b026c63a266, /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-6894fdc9-9e02-446f-92bb-2b026c63a266) succeeded
I1022 06:47:17.447294       1 controllerserver.go:533] detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-6894fdc9-9e02-446f-92bb-2b026c63a266 from node k8s-agentpool1-22475134-vmss000000 successfully
I1022 06:47:17.447303       1 utils.go:102] GRPC response: {}
I1022 06:47:17.516472       1 utils.go:95] GRPC call: /csi.v1.Controller/CreateVolume
I1022 06:47:17.516492       1 utils.go:96] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"requisite":[{"segments":{"topology.test.csi.azure.com/zone":""}}]},"capacity_range":{"required_bytes":5368709120},"name":"pvc-e7fe0c82-6611-4efa-b38b-831ca3daa630","parameters":{"csi.storage.k8s.io/pv/name":"pvc-e7fe0c82-6611-4efa-b38b-831ca3daa630","csi.storage.k8s.io/pvc/name":"pvc-9p79g","csi.storage.k8s.io/pvc/namespace":"volumelimits-928"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]}
I1022 06:47:17.565189       1 azure_diskclient.go:135] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-e7fe0c82-6611-4efa-b38b-831ca3daa630, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-e7fe0c82-6611-4efa-b38b-831ca3daa630' under resource group 'kubetest-5p3qtork' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I1022 06:47:17.565250       1 controllerserver.go:271] begin to create azure disk(pvc-e7fe0c82-6611-4efa-b38b-831ca3daa630) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) diskZone() maxShares(0)
I1022 06:47:17.565262       1 azure_managedDiskController.go:88] azureDisk - creating new managed Name:pvc-e7fe0c82-6611-4efa-b38b-831ca3daa630 StorageAccountType:StandardSSD_LRS Size:5
I1022 06:47:17.631358       1 utils.go:95] GRPC call: /csi.v1.Controller/CreateVolume
I1022 06:47:17.631378       1 utils.go:96] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"requisite":[{"segments":{"topology.test.csi.azure.com/zone":""}}]},"capacity_range":{"required_bytes":5368709120},"name":"pvc-3b7d143a-9eee-4ed1-b383-cd1d6673be53","parameters":{"csi.storage.k8s.io/pv/name":"pvc-3b7d143a-9eee-4ed1-b383-cd1d6673be53","csi.storage.k8s.io/pvc/name":"pvc-ksc24","csi.storage.k8s.io/pvc/namespace":"volumelimits-928"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]}
I1022 06:47:17.651362       1 azure_diskclient.go:135] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-3b7d143a-9eee-4ed1-b383-cd1d6673be53, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-3b7d143a-9eee-4ed1-b383-cd1d6673be53' under resource group 'kubetest-5p3qtork' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I1022 06:47:17.651432       1 controllerserver.go:271] begin to create azure disk(pvc-3b7d143a-9eee-4ed1-b383-cd1d6673be53) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) diskZone() maxShares(0)
I1022 06:47:17.651442       1 azure_managedDiskController.go:88] azureDisk - creating new managed Name:pvc-3b7d143a-9eee-4ed1-b383-cd1d6673be53 StorageAccountType:StandardSSD_LRS Size:5
I1022 06:47:17.739260       1 utils.go:95] GRPC call: /csi.v1.Controller/CreateVolume
I1022 06:47:17.739284       1 utils.go:96] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"requisite":[{"segments":{"topology.test.csi.azure.com/zone":""}}]},"capacity_range":{"required_bytes":5368709120},"name":"pvc-9273691d-6bca-4b0d-a798-48cd39f4be66","parameters":{"csi.storage.k8s.io/pv/name":"pvc-9273691d-6bca-4b0d-a798-48cd39f4be66","csi.storage.k8s.io/pvc/name":"pvc-fnp8c","csi.storage.k8s.io/pvc/namespace":"volumelimits-928"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]}
I1022 06:47:17.758136       1 azure_diskclient.go:135] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-9273691d-6bca-4b0d-a798-48cd39f4be66, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-9273691d-6bca-4b0d-a798-48cd39f4be66' under resource group 'kubetest-5p3qtork' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I1022 06:47:17.758259       1 controllerserver.go:271] begin to create azure disk(pvc-9273691d-6bca-4b0d-a798-48cd39f4be66) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) diskZone() maxShares(0)
I1022 06:47:17.758276       1 azure_managedDiskController.go:88] azureDisk - creating new managed Name:pvc-9273691d-6bca-4b0d-a798-48cd39f4be66 StorageAccountType:StandardSSD_LRS Size:5
I1022 06:47:17.844458       1 utils.go:95] GRPC call: /csi.v1.Controller/CreateVolume
I1022 06:47:17.844482       1 utils.go:96] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"requisite":[{"segments":{"topology.test.csi.azure.com/zone":""}}]},"capacity_range":{"required_bytes":5368709120},"name":"pvc-acd944ee-f869-4788-b766-c6715eee7d96","parameters":{"csi.storage.k8s.io/pv/name":"pvc-acd944ee-f869-4788-b766-c6715eee7d96","csi.storage.k8s.io/pvc/name":"pvc-8cr6x","csi.storage.k8s.io/pvc/namespace":"volumelimits-928"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]}
I1022 06:47:17.862576       1 azure_diskclient.go:135] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-acd944ee-f869-4788-b766-c6715eee7d96, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-acd944ee-f869-4788-b766-c6715eee7d96' under resource group 'kubetest-5p3qtork' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I1022 06:47:17.862701       1 controllerserver.go:271] begin to create azure disk(pvc-acd944ee-f869-4788-b766-c6715eee7d96) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) diskZone() maxShares(0)
I1022 06:47:17.862729       1 azure_managedDiskController.go:88] azureDisk - creating new managed Name:pvc-acd944ee-f869-4788-b766-c6715eee7d96 StorageAccountType:StandardSSD_LRS Size:5
I1022 06:47:17.962479       1 utils.go:95] GRPC call: /csi.v1.Controller/CreateVolume
I1022 06:47:17.962508       1 utils.go:96] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"requisite":[{"segments":{"topology.test.csi.azure.com/zone":""}}]},"capacity_range":{"required_bytes":5368709120},"name":"pvc-4bf40635-c941-4c51-9db3-35374a82e0df","parameters":{"csi.storage.k8s.io/pv/name":"pvc-4bf40635-c941-4c51-9db3-35374a82e0df","csi.storage.k8s.io/pvc/name":"pvc-74ggk","csi.storage.k8s.io/pvc/namespace":"volumelimits-928"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]}
I1022 06:47:17.982280       1 azure_diskclient.go:135] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-4bf40635-c941-4c51-9db3-35374a82e0df, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-4bf40635-c941-4c51-9db3-35374a82e0df' under resource group 'kubetest-5p3qtork' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I1022 06:47:17.982327       1 controllerserver.go:271] begin to create azure disk(pvc-4bf40635-c941-4c51-9db3-35374a82e0df) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) diskZone() maxShares(0)
I1022 06:47:17.982354       1 azure_managedDiskController.go:88] azureDisk - creating new managed Name:pvc-4bf40635-c941-4c51-9db3-35374a82e0df StorageAccountType:StandardSSD_LRS Size:5
I1022 06:47:18.074727       1 utils.go:95] GRPC call: /csi.v1.Controller/CreateVolume
I1022 06:47:18.074746       1 utils.go:96] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"requisite":[{"segments":{"topology.test.csi.azure.com/zone":""}}]},"capacity_range":{"required_bytes":5368709120},"name":"pvc-ac27c51e-aedd-4f4e-b24e-df2508a5e9b1","parameters":{"csi.storage.k8s.io/pv/name":"pvc-ac27c51e-aedd-4f4e-b24e-df2508a5e9b1","csi.storage.k8s.io/pvc/name":"pvc-k2wqn","csi.storage.k8s.io/pvc/namespace":"volumelimits-928"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]}
I1022 06:47:18.094622       1 azure_diskclient.go:135] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-ac27c51e-aedd-4f4e-b24e-df2508a5e9b1, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-ac27c51e-aedd-4f4e-b24e-df2508a5e9b1' under resource group 'kubetest-5p3qtork' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I1022 06:47:18.094709       1 controllerserver.go:271] begin to create azure disk(pvc-ac27c51e-aedd-4f4e-b24e-df2508a5e9b1) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) diskZone() maxShares(0)
I1022 06:47:18.094719       1 azure_managedDiskController.go:88] azureDisk - creating new managed Name:pvc-ac27c51e-aedd-4f4e-b24e-df2508a5e9b1 StorageAccountType:StandardSSD_LRS Size:5
I1022 06:47:18.182913       1 utils.go:95] GRPC call: /csi.v1.Controller/CreateVolume
I1022 06:47:18.182935       1 utils.go:96] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"requisite":[{"segments":{"topology.test.csi.azure.com/zone":""}}]},"capacity_range":{"required_bytes":5368709120},"name":"pvc-df1a5c24-dd06-4ce9-9ef0-7c6c82516986","parameters":{"csi.storage.k8s.io/pv/name":"pvc-df1a5c24-dd06-4ce9-9ef0-7c6c82516986","csi.storage.k8s.io/pvc/name":"pvc-c6wnc","csi.storage.k8s.io/pvc/namespace":"volumelimits-928"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]}
I1022 06:47:18.235652       1 azure_diskclient.go:135] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-df1a5c24-dd06-4ce9-9ef0-7c6c82516986, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-df1a5c24-dd06-4ce9-9ef0-7c6c82516986' under resource group 'kubetest-5p3qtork' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I1022 06:47:18.235725       1 controllerserver.go:271] begin to create azure disk(pvc-df1a5c24-dd06-4ce9-9ef0-7c6c82516986) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) diskZone() maxShares(0)
I1022 06:47:18.235737       1 azure_managedDiskController.go:88] azureDisk - creating new managed Name:pvc-df1a5c24-dd06-4ce9-9ef0-7c6c82516986 StorageAccountType:StandardSSD_LRS Size:5
I1022 06:47:18.292744       1 utils.go:95] GRPC call: /csi.v1.Controller/CreateVolume
I1022 06:47:18.292763       1 utils.go:96] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"requisite":[{"segments":{"topology.test.csi.azure.com/zone":""}}]},"capacity_range":{"required_bytes":5368709120},"name":"pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5","parameters":{"csi.storage.k8s.io/pv/name":"pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5","csi.storage.k8s.io/pvc/name":"pvc-tq8kk","csi.storage.k8s.io/pvc/namespace":"volumelimits-928"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]}
I1022 06:47:18.312754       1 azure_diskclient.go:135] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5' under resource group 'kubetest-5p3qtork' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I1022 06:47:18.312841       1 controllerserver.go:271] begin to create azure disk(pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) diskZone() maxShares(0)
I1022 06:47:18.312870       1 azure_managedDiskController.go:88] azureDisk - creating new managed Name:pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5 StorageAccountType:StandardSSD_LRS Size:5
I1022 06:47:18.403316       1 utils.go:95] GRPC call: /csi.v1.Controller/CreateVolume
I1022 06:47:18.403337       1 utils.go:96] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"requisite":[{"segments":{"topology.test.csi.azure.com/zone":""}}]},"capacity_range":{"required_bytes":5368709120},"name":"pvc-42ebe9ec-eca5-4926-8da8-4b21f93e4b2f","parameters":{"csi.storage.k8s.io/pv/name":"pvc-42ebe9ec-eca5-4926-8da8-4b21f93e4b2f","csi.storage.k8s.io/pvc/name":"pvc-n8qjn","csi.storage.k8s.io/pvc/namespace":"volumelimits-928"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]}
I1022 06:47:18.449895       1 azure_diskclient.go:135] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-42ebe9ec-eca5-4926-8da8-4b21f93e4b2f, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-42ebe9ec-eca5-4926-8da8-4b21f93e4b2f' under resource group 'kubetest-5p3qtork' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I1022 06:47:18.449981       1 controllerserver.go:271] begin to create azure disk(pvc-42ebe9ec-eca5-4926-8da8-4b21f93e4b2f) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) diskZone() maxShares(0)
I1022 06:47:18.449992       1 azure_managedDiskController.go:88] azureDisk - creating new managed Name:pvc-42ebe9ec-eca5-4926-8da8-4b21f93e4b2f StorageAccountType:StandardSSD_LRS Size:5
I1022 06:47:18.511094       1 utils.go:95] GRPC call: /csi.v1.Controller/CreateVolume
I1022 06:47:18.511115       1 utils.go:96] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"requisite":[{"segments":{"topology.test.csi.azure.com/zone":""}}]},"capacity_range":{"required_bytes":5368709120},"name":"pvc-98953f15-d806-48a5-8627-b381bc559972","parameters":{"csi.storage.k8s.io/pv/name":"pvc-98953f15-d806-48a5-8627-b381bc559972","csi.storage.k8s.io/pvc/name":"pvc-t67h2","csi.storage.k8s.io/pvc/namespace":"volumelimits-928"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]}
I1022 06:47:18.530011       1 azure_diskclient.go:135] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-98953f15-d806-48a5-8627-b381bc559972, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-98953f15-d806-48a5-8627-b381bc559972' under resource group 'kubetest-5p3qtork' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I1022 06:47:18.530072       1 controllerserver.go:271] begin to create azure disk(pvc-98953f15-d806-48a5-8627-b381bc559972) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) diskZone() maxShares(0)
I1022 06:47:18.530084       1 azure_managedDiskController.go:88] azureDisk - creating new managed Name:pvc-98953f15-d806-48a5-8627-b381bc559972 StorageAccountType:StandardSSD_LRS Size:5
I1022 06:47:18.620176       1 utils.go:95] GRPC call: /csi.v1.Controller/CreateVolume
I1022 06:47:18.620196       1 utils.go:96] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"requisite":[{"segments":{"topology.test.csi.azure.com/zone":""}}]},"capacity_range":{"required_bytes":5368709120},"name":"pvc-ede47b18-01aa-4bba-b094-3bba45a8d78c","parameters":{"csi.storage.k8s.io/pv/name":"pvc-ede47b18-01aa-4bba-b094-3bba45a8d78c","csi.storage.k8s.io/pvc/name":"pvc-qnvsj","csi.storage.k8s.io/pvc/namespace":"volumelimits-928"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]}
I1022 06:47:18.640744       1 azure_diskclient.go:135] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-ede47b18-01aa-4bba-b094-3bba45a8d78c, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-ede47b18-01aa-4bba-b094-3bba45a8d78c' under resource group 'kubetest-5p3qtork' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I1022 06:47:18.640870       1 controllerserver.go:271] begin to create azure disk(pvc-ede47b18-01aa-4bba-b094-3bba45a8d78c) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) diskZone() maxShares(0)
I1022 06:47:18.640886       1 azure_managedDiskController.go:88] azureDisk - creating new managed Name:pvc-ede47b18-01aa-4bba-b094-3bba45a8d78c StorageAccountType:StandardSSD_LRS Size:5
I1022 06:47:18.673442       1 azure_managedDiskController.go:249] azureDisk - created new MD Name:pvc-7292229f-a145-4b90-84c0-8ad707acb8f1 StorageAccountType:StandardSSD_LRS Size:5
I1022 06:47:18.673472       1 controllerserver.go:348] create azure disk(pvc-7292229f-a145-4b90-84c0-8ad707acb8f1) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) tags(map[kubernetes.io-created-for-pv-name:pvc-7292229f-a145-4b90-84c0-8ad707acb8f1 kubernetes.io-created-for-pvc-name:pvc-5hfc4 kubernetes.io-created-for-pvc-namespace:volumelimits-928]) successfully
I1022 06:47:18.673507       1 utils.go:102] GRPC response: {"volume":{"accessible_topology":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"capacity_bytes":5368709120,"content_source":{"Type":null},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-7292229f-a145-4b90-84c0-8ad707acb8f1","csi.storage.k8s.io/pvc/name":"pvc-5hfc4","csi.storage.k8s.io/pvc/namespace":"volumelimits-928","requestedsizegib":"5"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-7292229f-a145-4b90-84c0-8ad707acb8f1"}}
I1022 06:47:18.727334       1 utils.go:95] GRPC call: /csi.v1.Controller/CreateVolume
I1022 06:47:18.727362       1 utils.go:96] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"requisite":[{"segments":{"topology.test.csi.azure.com/zone":""}}]},"capacity_range":{"required_bytes":5368709120},"name":"pvc-4a6ce6eb-10e0-401a-8257-fc05a4e4349f","parameters":{"csi.storage.k8s.io/pv/name":"pvc-4a6ce6eb-10e0-401a-8257-fc05a4e4349f","csi.storage.k8s.io/pvc/name":"pvc-69l44","csi.storage.k8s.io/pvc/namespace":"volumelimits-928"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]}
I1022 06:47:18.742996       1 azure_managedDiskController.go:249] azureDisk - created new MD Name:pvc-7598fafa-b8d8-45fc-bd09-7abb6e913a34 StorageAccountType:StandardSSD_LRS Size:5
I1022 06:47:18.743034       1 controllerserver.go:348] create azure disk(pvc-7598fafa-b8d8-45fc-bd09-7abb6e913a34) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) tags(map[kubernetes.io-created-for-pv-name:pvc-7598fafa-b8d8-45fc-bd09-7abb6e913a34 kubernetes.io-created-for-pvc-name:pvc-pxw5k kubernetes.io-created-for-pvc-namespace:volumelimits-928]) successfully
I1022 06:47:18.743057       1 utils.go:102] GRPC response: {"volume":{"accessible_topology":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"capacity_bytes":5368709120,"content_source":{"Type":null},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-7598fafa-b8d8-45fc-bd09-7abb6e913a34","csi.storage.k8s.io/pvc/name":"pvc-pxw5k","csi.storage.k8s.io/pvc/namespace":"volumelimits-928","requestedsizegib":"5"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-7598fafa-b8d8-45fc-bd09-7abb6e913a34"}}
I1022 06:47:18.744395       1 azure_diskclient.go:135] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-4a6ce6eb-10e0-401a-8257-fc05a4e4349f, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-4a6ce6eb-10e0-401a-8257-fc05a4e4349f' under resource group 'kubetest-5p3qtork' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I1022 06:47:18.744487       1 controllerserver.go:271] begin to create azure disk(pvc-4a6ce6eb-10e0-401a-8257-fc05a4e4349f) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) diskZone() maxShares(0)
I1022 06:47:18.744527       1 azure_managedDiskController.go:88] azureDisk - creating new managed Name:pvc-4a6ce6eb-10e0-401a-8257-fc05a4e4349f StorageAccountType:StandardSSD_LRS Size:5
I1022 06:47:18.838226       1 utils.go:95] GRPC call: /csi.v1.Controller/CreateVolume
I1022 06:47:18.838250       1 utils.go:96] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"requisite":[{"segments":{"topology.test.csi.azure.com/zone":""}}]},"capacity_range":{"required_bytes":5368709120},"name":"pvc-e01638a5-80eb-44cb-902c-a1c302a75244","parameters":{"csi.storage.k8s.io/pv/name":"pvc-e01638a5-80eb-44cb-902c-a1c302a75244","csi.storage.k8s.io/pvc/name":"pvc-4bt7m","csi.storage.k8s.io/pvc/namespace":"volumelimits-928"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]}
I1022 06:47:18.861075       1 azure_diskclient.go:135] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-e01638a5-80eb-44cb-902c-a1c302a75244, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-e01638a5-80eb-44cb-902c-a1c302a75244' under resource group 'kubetest-5p3qtork' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I1022 06:47:18.861169       1 controllerserver.go:271] begin to create azure disk(pvc-e01638a5-80eb-44cb-902c-a1c302a75244) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) diskZone() maxShares(0)
I1022 06:47:18.861184       1 azure_managedDiskController.go:88] azureDisk - creating new managed Name:pvc-e01638a5-80eb-44cb-902c-a1c302a75244 StorageAccountType:StandardSSD_LRS Size:5
I1022 06:47:18.949939       1 utils.go:95] GRPC call: /csi.v1.Controller/CreateVolume
I1022 06:47:18.949960       1 utils.go:96] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"requisite":[{"segments":{"topology.test.csi.azure.com/zone":""}}]},"capacity_range":{"required_bytes":5368709120},"name":"pvc-2cacef5a-9bc2-4a71-a22d-c72e842538c3","parameters":{"csi.storage.k8s.io/pv/name":"pvc-2cacef5a-9bc2-4a71-a22d-c72e842538c3","csi.storage.k8s.io/pvc/name":"pvc-sq2kg","csi.storage.k8s.io/pvc/namespace":"volumelimits-928"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]}
I1022 06:47:18.967881       1 azure_diskclient.go:135] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-2cacef5a-9bc2-4a71-a22d-c72e842538c3, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-2cacef5a-9bc2-4a71-a22d-c72e842538c3' under resource group 'kubetest-5p3qtork' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I1022 06:47:18.967950       1 controllerserver.go:271] begin to create azure disk(pvc-2cacef5a-9bc2-4a71-a22d-c72e842538c3) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) diskZone() maxShares(0)
I1022 06:47:18.967963       1 azure_managedDiskController.go:88] azureDisk - creating new managed Name:pvc-2cacef5a-9bc2-4a71-a22d-c72e842538c3 StorageAccountType:StandardSSD_LRS Size:5
I1022 06:47:18.977296       1 azure_managedDiskController.go:249] azureDisk - created new MD Name:pvc-625cb0a6-3ef0-4bb1-8a80-8d075355b851 StorageAccountType:StandardSSD_LRS Size:5
I1022 06:47:18.977341       1 controllerserver.go:348] create azure disk(pvc-625cb0a6-3ef0-4bb1-8a80-8d075355b851) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) tags(map[kubernetes.io-created-for-pv-name:pvc-625cb0a6-3ef0-4bb1-8a80-8d075355b851 kubernetes.io-created-for-pvc-name:pvc-gxtth kubernetes.io-created-for-pvc-namespace:volumelimits-928]) successfully
I1022 06:47:18.977357       1 utils.go:102] GRPC response: {"volume":{"accessible_topology":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"capacity_bytes":5368709120,"content_source":{"Type":null},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-625cb0a6-3ef0-4bb1-8a80-8d075355b851","csi.storage.k8s.io/pvc/name":"pvc-gxtth","csi.storage.k8s.io/pvc/namespace":"volumelimits-928","requestedsizegib":"5"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-625cb0a6-3ef0-4bb1-8a80-8d075355b851"}}
I1022 06:47:18.992375       1 azure_managedDiskController.go:249] azureDisk - created new MD Name:pvc-84c19a86-6835-4758-8251-2d417a93c3c7 StorageAccountType:StandardSSD_LRS Size:5
I1022 06:47:18.992440       1 controllerserver.go:348] create azure disk(pvc-84c19a86-6835-4758-8251-2d417a93c3c7) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) tags(map[kubernetes.io-created-for-pv-name:pvc-84c19a86-6835-4758-8251-2d417a93c3c7 kubernetes.io-created-for-pvc-name:pvc-l7r25 kubernetes.io-created-for-pvc-namespace:volumelimits-928]) successfully
I1022 06:47:18.992476       1 utils.go:102] GRPC response: {"volume":{"accessible_topology":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"capacity_bytes":5368709120,"content_source":{"Type":null},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-84c19a86-6835-4758-8251-2d417a93c3c7","csi.storage.k8s.io/pvc/name":"pvc-l7r25","csi.storage.k8s.io/pvc/namespace":"volumelimits-928","requestedsizegib":"5"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-84c19a86-6835-4758-8251-2d417a93c3c7"}}
I1022 06:47:19.055467       1 utils.go:95] GRPC call: /csi.v1.Controller/CreateVolume
I1022 06:47:19.055490       1 utils.go:96] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"requisite":[{"segments":{"topology.test.csi.azure.com/zone":""}}]},"capacity_range":{"required_bytes":5368709120},"name":"pvc-506f8b7e-8262-4a63-8f37-1a4145b662aa","parameters":{"csi.storage.k8s.io/pv/name":"pvc-506f8b7e-8262-4a63-8f37-1a4145b662aa","csi.storage.k8s.io/pvc/name":"pvc-pc7vv","csi.storage.k8s.io/pvc/namespace":"volumelimits-928"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]}
I1022 06:47:19.094371       1 azure_diskclient.go:135] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-506f8b7e-8262-4a63-8f37-1a4145b662aa, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-506f8b7e-8262-4a63-8f37-1a4145b662aa' under resource group 'kubetest-5p3qtork' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I1022 06:47:19.094436       1 controllerserver.go:271] begin to create azure disk(pvc-506f8b7e-8262-4a63-8f37-1a4145b662aa) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) diskZone() maxShares(0)
I1022 06:47:19.094445       1 azure_managedDiskController.go:88] azureDisk - creating new managed Name:pvc-506f8b7e-8262-4a63-8f37-1a4145b662aa StorageAccountType:StandardSSD_LRS Size:5
I1022 06:47:19.108226       1 azure_managedDiskController.go:249] azureDisk - created new MD Name:pvc-a43477fa-15b0-4f50-92e3-0a58c87f24cd StorageAccountType:StandardSSD_LRS Size:5
I1022 06:47:19.108273       1 controllerserver.go:348] create azure disk(pvc-a43477fa-15b0-4f50-92e3-0a58c87f24cd) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) tags(map[kubernetes.io-created-for-pv-name:pvc-a43477fa-15b0-4f50-92e3-0a58c87f24cd kubernetes.io-created-for-pvc-name:pvc-7q85h kubernetes.io-created-for-pvc-namespace:volumelimits-928]) successfully
I1022 06:47:19.108294       1 utils.go:102] GRPC response: {"volume":{"accessible_topology":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"capacity_bytes":5368709120,"content_source":{"Type":null},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-a43477fa-15b0-4f50-92e3-0a58c87f24cd","csi.storage.k8s.io/pvc/name":"pvc-7q85h","csi.storage.k8s.io/pvc/namespace":"volumelimits-928","requestedsizegib":"5"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-a43477fa-15b0-4f50-92e3-0a58c87f24cd"}}
I1022 06:47:19.165789       1 utils.go:95] GRPC call: /csi.v1.Controller/CreateVolume
I1022 06:47:19.165811       1 utils.go:96] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"requisite":[{"segments":{"topology.test.csi.azure.com/zone":""}}]},"capacity_range":{"required_bytes":5368709120},"name":"pvc-09b74d52-8b3c-431d-a110-aef9ac1d65b2","parameters":{"csi.storage.k8s.io/pv/name":"pvc-09b74d52-8b3c-431d-a110-aef9ac1d65b2","csi.storage.k8s.io/pvc/name":"pvc-5n2wz","csi.storage.k8s.io/pvc/namespace":"volumelimits-928"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]}
I1022 06:47:19.183350       1 azure_diskclient.go:135] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-09b74d52-8b3c-431d-a110-aef9ac1d65b2, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-09b74d52-8b3c-431d-a110-aef9ac1d65b2' under resource group 'kubetest-5p3qtork' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I1022 06:47:19.183441       1 controllerserver.go:271] begin to create azure disk(pvc-09b74d52-8b3c-431d-a110-aef9ac1d65b2) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) diskZone() maxShares(0)
I1022 06:47:19.183450       1 azure_managedDiskController.go:88] azureDisk - creating new managed Name:pvc-09b74d52-8b3c-431d-a110-aef9ac1d65b2 StorageAccountType:StandardSSD_LRS Size:5
I1022 06:47:19.278974       1 utils.go:95] GRPC call: /csi.v1.Controller/CreateVolume
I1022 06:47:19.279017       1 utils.go:96] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"requisite":[{"segments":{"topology.test.csi.azure.com/zone":""}}]},"capacity_range":{"required_bytes":5368709120},"name":"pvc-13fe5e26-c069-46f7-836d-30618bd80606","parameters":{"csi.storage.k8s.io/pv/name":"pvc-13fe5e26-c069-46f7-836d-30618bd80606","csi.storage.k8s.io/pvc/name":"pvc-jgjh9","csi.storage.k8s.io/pvc/namespace":"volumelimits-928"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]}
I1022 06:47:19.315908       1 azure_diskclient.go:135] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-13fe5e26-c069-46f7-836d-30618bd80606, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-13fe5e26-c069-46f7-836d-30618bd80606' under resource group 'kubetest-5p3qtork' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I1022 06:47:19.315999       1 controllerserver.go:271] begin to create azure disk(pvc-13fe5e26-c069-46f7-836d-30618bd80606) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) diskZone() maxShares(0)
I1022 06:47:19.316013       1 azure_managedDiskController.go:88] azureDisk - creating new managed Name:pvc-13fe5e26-c069-46f7-836d-30618bd80606 StorageAccountType:StandardSSD_LRS Size:5
I1022 06:47:19.391362       1 utils.go:95] GRPC call: /csi.v1.Controller/CreateVolume
I1022 06:47:19.391383       1 utils.go:96] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"requisite":[{"segments":{"topology.test.csi.azure.com/zone":""}}]},"capacity_range":{"required_bytes":5368709120},"name":"pvc-07911b25-debe-4781-8c5f-b81e70b6bd7d","parameters":{"csi.storage.k8s.io/pv/name":"pvc-07911b25-debe-4781-8c5f-b81e70b6bd7d","csi.storage.k8s.io/pvc/name":"pvc-ckxtv","csi.storage.k8s.io/pvc/namespace":"volumelimits-928"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]}
I1022 06:47:19.410263       1 azure_diskclient.go:135] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-07911b25-debe-4781-8c5f-b81e70b6bd7d, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-07911b25-debe-4781-8c5f-b81e70b6bd7d' under resource group 'kubetest-5p3qtork' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I1022 06:47:19.410328       1 controllerserver.go:271] begin to create azure disk(pvc-07911b25-debe-4781-8c5f-b81e70b6bd7d) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) diskZone() maxShares(0)
I1022 06:47:19.410339       1 azure_managedDiskController.go:88] azureDisk - creating new managed Name:pvc-07911b25-debe-4781-8c5f-b81e70b6bd7d StorageAccountType:StandardSSD_LRS Size:5
I1022 06:47:19.425668       1 azure_managedDiskController.go:249] azureDisk - created new MD Name:pvc-d92c57dc-1538-4651-bc16-6cc2c3ef92cc StorageAccountType:StandardSSD_LRS Size:5
I1022 06:47:19.425716       1 controllerserver.go:348] create azure disk(pvc-d92c57dc-1538-4651-bc16-6cc2c3ef92cc) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) tags(map[kubernetes.io-created-for-pv-name:pvc-d92c57dc-1538-4651-bc16-6cc2c3ef92cc kubernetes.io-created-for-pvc-name:pvc-nmhhg kubernetes.io-created-for-pvc-namespace:volumelimits-928]) successfully
I1022 06:47:19.425757       1 utils.go:102] GRPC response: {"volume":{"accessible_topology":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"capacity_bytes":5368709120,"content_source":{"Type":null},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-d92c57dc-1538-4651-bc16-6cc2c3ef92cc","csi.storage.k8s.io/pvc/name":"pvc-nmhhg","csi.storage.k8s.io/pvc/namespace":"volumelimits-928","requestedsizegib":"5"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-d92c57dc-1538-4651-bc16-6cc2c3ef92cc"}}
I1022 06:47:19.546787       1 utils.go:95] GRPC call: /csi.v1.Controller/CreateVolume
I1022 06:47:19.546814       1 utils.go:96] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"requisite":[{"segments":{"topology.test.csi.azure.com/zone":""}}]},"capacity_range":{"required_bytes":5368709120},"name":"pvc-7a1277c4-4dd2-4783-a3a2-70e6c7455fdc","parameters":{"csi.storage.k8s.io/pv/name":"pvc-7a1277c4-4dd2-4783-a3a2-70e6c7455fdc","csi.storage.k8s.io/pvc/name":"pvc-wg49s","csi.storage.k8s.io/pvc/namespace":"volumelimits-928"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]}
I1022 06:47:19.565334       1 azure_diskclient.go:135] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-7a1277c4-4dd2-4783-a3a2-70e6c7455fdc, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-7a1277c4-4dd2-4783-a3a2-70e6c7455fdc' under resource group 'kubetest-5p3qtork' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I1022 06:47:19.565407       1 controllerserver.go:271] begin to create azure disk(pvc-7a1277c4-4dd2-4783-a3a2-70e6c7455fdc) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) diskZone() maxShares(0)
I1022 06:47:19.565421       1 azure_managedDiskController.go:88] azureDisk - creating new managed Name:pvc-7a1277c4-4dd2-4783-a3a2-70e6c7455fdc StorageAccountType:StandardSSD_LRS Size:5
I1022 06:47:19.646450       1 utils.go:95] GRPC call: /csi.v1.Controller/CreateVolume
I1022 06:47:19.646473       1 utils.go:96] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"requisite":[{"segments":{"topology.test.csi.azure.com/zone":""}}]},"capacity_range":{"required_bytes":5368709120},"name":"pvc-3107aca1-7d54-4ac3-945a-25d6c0246a0c","parameters":{"csi.storage.k8s.io/pv/name":"pvc-3107aca1-7d54-4ac3-945a-25d6c0246a0c","csi.storage.k8s.io/pvc/name":"pvc-kgfc4","csi.storage.k8s.io/pvc/namespace":"volumelimits-928"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]}
I1022 06:47:19.646592       1 azure_managedDiskController.go:249] azureDisk - created new MD Name:pvc-aba6fcc4-93be-4a03-b913-b0ad6dfd64e9 StorageAccountType:StandardSSD_LRS Size:5
I1022 06:47:19.646640       1 controllerserver.go:348] create azure disk(pvc-aba6fcc4-93be-4a03-b913-b0ad6dfd64e9) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) tags(map[kubernetes.io-created-for-pv-name:pvc-aba6fcc4-93be-4a03-b913-b0ad6dfd64e9 kubernetes.io-created-for-pvc-name:pvc-7hlb8 kubernetes.io-created-for-pvc-namespace:volumelimits-928]) successfully
I1022 06:47:19.646613       1 azure_managedDiskController.go:249] azureDisk - created new MD Name:pvc-6e5bb4b7-9c65-4f65-9a30-72575fc9afac StorageAccountType:StandardSSD_LRS Size:5
I1022 06:47:19.646956       1 controllerserver.go:348] create azure disk(pvc-6e5bb4b7-9c65-4f65-9a30-72575fc9afac) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) tags(map[kubernetes.io-created-for-pv-name:pvc-6e5bb4b7-9c65-4f65-9a30-72575fc9afac kubernetes.io-created-for-pvc-name:pvc-4nvg4 kubernetes.io-created-for-pvc-namespace:volumelimits-928]) successfully
I1022 06:47:19.646673       1 utils.go:102] GRPC response: {"volume":{"accessible_topology":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"capacity_bytes":5368709120,"content_source":{"Type":null},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-aba6fcc4-93be-4a03-b913-b0ad6dfd64e9","csi.storage.k8s.io/pvc/name":"pvc-7hlb8","csi.storage.k8s.io/pvc/namespace":"volumelimits-928","requestedsizegib":"5"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-aba6fcc4-93be-4a03-b913-b0ad6dfd64e9"}}
I1022 06:47:19.647002       1 utils.go:102] GRPC response: {"volume":{"accessible_topology":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"capacity_bytes":5368709120,"content_source":{"Type":null},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-6e5bb4b7-9c65-4f65-9a30-72575fc9afac","csi.storage.k8s.io/pvc/name":"pvc-4nvg4","csi.storage.k8s.io/pvc/namespace":"volumelimits-928","requestedsizegib":"5"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-6e5bb4b7-9c65-4f65-9a30-72575fc9afac"}}
I1022 06:47:19.664146       1 azure_diskclient.go:135] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-3107aca1-7d54-4ac3-945a-25d6c0246a0c, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-3107aca1-7d54-4ac3-945a-25d6c0246a0c' under resource group 'kubetest-5p3qtork' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I1022 06:47:19.664193       1 controllerserver.go:271] begin to create azure disk(pvc-3107aca1-7d54-4ac3-945a-25d6c0246a0c) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) diskZone() maxShares(0)
I1022 06:47:19.664204       1 azure_managedDiskController.go:88] azureDisk - creating new managed Name:pvc-3107aca1-7d54-4ac3-945a-25d6c0246a0c StorageAccountType:StandardSSD_LRS Size:5
I1022 06:47:19.746589       1 utils.go:95] GRPC call: /csi.v1.Controller/CreateVolume
I1022 06:47:19.746615       1 utils.go:96] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"requisite":[{"segments":{"topology.test.csi.azure.com/zone":""}}]},"capacity_range":{"required_bytes":5368709120},"name":"pvc-d330a831-95bd-45e4-b274-292d41f80459","parameters":{"csi.storage.k8s.io/pv/name":"pvc-d330a831-95bd-45e4-b274-292d41f80459","csi.storage.k8s.io/pvc/name":"pvc-9w5p8","csi.storage.k8s.io/pvc/namespace":"volumelimits-928"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]}
I1022 06:47:19.746806       1 azure_managedDiskController.go:249] azureDisk - created new MD Name:pvc-90b4af34-20b0-494b-a46f-a3471b81f63a StorageAccountType:StandardSSD_LRS Size:5
I1022 06:47:19.746846       1 controllerserver.go:348] create azure disk(pvc-90b4af34-20b0-494b-a46f-a3471b81f63a) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) tags(map[kubernetes.io-created-for-pv-name:pvc-90b4af34-20b0-494b-a46f-a3471b81f63a kubernetes.io-created-for-pvc-name:pvc-8mtq5 kubernetes.io-created-for-pvc-namespace:volumelimits-928]) successfully
I1022 06:47:19.746882       1 utils.go:102] GRPC response: {"volume":{"accessible_topology":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"capacity_bytes":5368709120,"content_source":{"Type":null},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-90b4af34-20b0-494b-a46f-a3471b81f63a","csi.storage.k8s.io/pvc/name":"pvc-8mtq5","csi.storage.k8s.io/pvc/namespace":"volumelimits-928","requestedsizegib":"5"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-90b4af34-20b0-494b-a46f-a3471b81f63a"}}
I1022 06:47:19.786963       1 azure_diskclient.go:135] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-d330a831-95bd-45e4-b274-292d41f80459, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-d330a831-95bd-45e4-b274-292d41f80459' under resource group 'kubetest-5p3qtork' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I1022 06:47:19.787042       1 controllerserver.go:271] begin to create azure disk(pvc-d330a831-95bd-45e4-b274-292d41f80459) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) diskZone() maxShares(0)
I1022 06:47:19.787058       1 azure_managedDiskController.go:88] azureDisk - creating new managed Name:pvc-d330a831-95bd-45e4-b274-292d41f80459 StorageAccountType:StandardSSD_LRS Size:5
I1022 06:47:19.807399       1 azure_managedDiskController.go:249] azureDisk - created new MD Name:pvc-4468e1b3-f470-4e0c-9404-16d282d00932 StorageAccountType:StandardSSD_LRS Size:5
I1022 06:47:19.807464       1 controllerserver.go:348] create azure disk(pvc-4468e1b3-f470-4e0c-9404-16d282d00932) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) tags(map[kubernetes.io-created-for-pv-name:pvc-4468e1b3-f470-4e0c-9404-16d282d00932 kubernetes.io-created-for-pvc-name:pvc-5zlhj kubernetes.io-created-for-pvc-namespace:volumelimits-928]) successfully
I1022 06:47:19.807498       1 utils.go:102] GRPC response: {"volume":{"accessible_topology":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"capacity_bytes":5368709120,"content_source":{"Type":null},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-4468e1b3-f470-4e0c-9404-16d282d00932","csi.storage.k8s.io/pvc/name":"pvc-5zlhj","csi.storage.k8s.io/pvc/namespace":"volumelimits-928","requestedsizegib":"5"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-4468e1b3-f470-4e0c-9404-16d282d00932"}}
I1022 06:47:19.888716       1 azure_managedDiskController.go:249] azureDisk - created new MD Name:pvc-a38ef8f3-7817-4091-b6e0-7741e79fd589 StorageAccountType:StandardSSD_LRS Size:5
... skipping 119 lines ...
I1022 06:48:03.659210       1 utils.go:102] GRPC response: {}
I1022 06:48:03.666774       1 controllerserver.go:461] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-1d31be02-fc07-4482-97ed-0d5e45609233" to node "k8s-agentpool1-22475134-vmss000001"
I1022 06:48:03.666804       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000001" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:48:03.666821       1 azure_controller_common.go:242] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-1d31be02-fc07-4482-97ed-0d5e45609233" lun 0 to node "k8s-agentpool1-22475134-vmss000001", diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-1d31be02-fc07-4482-97ed-0d5e45609233:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-1d31be02-fc07-4482-97ed-0d5e45609233  true false 0})]
I1022 06:48:03.666859       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000001" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:48:03.666871       1 azure_controller_vmss.go:122] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-1d31be02-fc07-4482-97ed-0d5e45609233:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-1d31be02-fc07-4482-97ed-0d5e45609233  true false 0})])
I1022 06:48:03.897490       1 azure_controller_vmss.go:134] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-1d31be02-fc07-4482-97ed-0d5e45609233:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-1d31be02-fc07-4482-97ed-0d5e45609233  true false 0})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING)
I1022 06:48:09.788747       1 utils.go:95] GRPC call: /csi.v1.Controller/CreateVolume
I1022 06:48:09.788779       1 utils.go:96] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"requisite":[{"segments":{"topology.test.csi.azure.com/zone":""}}]},"capacity_range":{"required_bytes":5368709120},"name":"pvc-cb861b94-be9c-4a97-a6fe-4793c8f2d550","parameters":{"csi.storage.k8s.io/pv/name":"pvc-cb861b94-be9c-4a97-a6fe-4793c8f2d550","csi.storage.k8s.io/pvc/name":"test.csi.azure.comcc4nv","csi.storage.k8s.io/pvc/namespace":"snapshotting-6996"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]}
I1022 06:48:09.823800       1 azure_diskclient.go:135] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-cb861b94-be9c-4a97-a6fe-4793c8f2d550, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-cb861b94-be9c-4a97-a6fe-4793c8f2d550' under resource group 'kubetest-5p3qtork' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I1022 06:48:09.823880       1 controllerserver.go:271] begin to create azure disk(pvc-cb861b94-be9c-4a97-a6fe-4793c8f2d550) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) diskZone() maxShares(0)
I1022 06:48:09.823935       1 azure_managedDiskController.go:88] azureDisk - creating new managed Name:pvc-cb861b94-be9c-4a97-a6fe-4793c8f2d550 StorageAccountType:StandardSSD_LRS Size:5
I1022 06:48:12.178618       1 azure_managedDiskController.go:249] azureDisk - created new MD Name:pvc-cb861b94-be9c-4a97-a6fe-4793c8f2d550 StorageAccountType:StandardSSD_LRS Size:5
I1022 06:48:12.178653       1 controllerserver.go:348] create azure disk(pvc-cb861b94-be9c-4a97-a6fe-4793c8f2d550) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) tags(map[kubernetes.io-created-for-pv-name:pvc-cb861b94-be9c-4a97-a6fe-4793c8f2d550 kubernetes.io-created-for-pvc-name:test.csi.azure.comcc4nv kubernetes.io-created-for-pvc-namespace:snapshotting-6996]) successfully
I1022 06:48:12.178676       1 utils.go:102] GRPC response: {"volume":{"accessible_topology":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"capacity_bytes":5368709120,"content_source":{"Type":null},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-cb861b94-be9c-4a97-a6fe-4793c8f2d550","csi.storage.k8s.io/pvc/name":"test.csi.azure.comcc4nv","csi.storage.k8s.io/pvc/namespace":"snapshotting-6996","requestedsizegib":"5"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-cb861b94-be9c-4a97-a6fe-4793c8f2d550"}}
I1022 06:48:14.543405       1 utils.go:95] GRPC call: /csi.v1.Controller/ControllerPublishVolume
... skipping 2 lines ...
I1022 06:48:14.566211       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:48:14.566252       1 controllerserver.go:461] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-cb861b94-be9c-4a97-a6fe-4793c8f2d550" to node "k8s-agentpool1-22475134-vmss000000"
I1022 06:48:14.566289       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:48:14.566315       1 azure_controller_common.go:242] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-cb861b94-be9c-4a97-a6fe-4793c8f2d550" lun 0 to node "k8s-agentpool1-22475134-vmss000000", diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-cb861b94-be9c-4a97-a6fe-4793c8f2d550:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-cb861b94-be9c-4a97-a6fe-4793c8f2d550  true false 0})]
I1022 06:48:14.566347       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:48:14.566379       1 azure_controller_vmss.go:122] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-cb861b94-be9c-4a97-a6fe-4793c8f2d550:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-cb861b94-be9c-4a97-a6fe-4793c8f2d550  true false 0})])
I1022 06:48:14.724724       1 azure_controller_vmss.go:134] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-cb861b94-be9c-4a97-a6fe-4793c8f2d550:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-cb861b94-be9c-4a97-a6fe-4793c8f2d550  true false 0})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING)
I1022 06:48:16.664673       1 utils.go:95] GRPC call: /csi.v1.Controller/DeleteSnapshot
I1022 06:48:16.664700       1 utils.go:96] GRPC request: {"snapshot_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/snapshots/snapshot-c32c3e47-e8ba-4aef-85a9-b15a6a2e0d5c"}
I1022 06:48:16.664828       1 controllerserver.go:956] begin to delete snapshot(snapshot-c32c3e47-e8ba-4aef-85a9-b15a6a2e0d5c) under rg(kubetest-5p3qtork)
I1022 06:48:19.003650       1 controllerserver.go:465] Attach operation successful: volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-1d31be02-fc07-4482-97ed-0d5e45609233" attached to node "k8s-agentpool1-22475134-vmss000001".
I1022 06:48:19.003680       1 controllerserver.go:485] attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-1d31be02-fc07-4482-97ed-0d5e45609233" to node "k8s-agentpool1-22475134-vmss000001" successfully
I1022 06:48:19.003714       1 utils.go:102] GRPC response: {"publish_context":{"LUN":"0"}}
... skipping 48 lines ...
I1022 06:48:35.344786       1 utils.go:96] GRPC request: {"node_id":"k8s-agentpool1-22475134-vmss000000","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-90b4af34-20b0-494b-a46f-a3471b81f63a","csi.storage.k8s.io/pvc/name":"pvc-8mtq5","csi.storage.k8s.io/pvc/namespace":"volumelimits-928","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1634884037556-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-90b4af34-20b0-494b-a46f-a3471b81f63a"}
I1022 06:48:35.376553       1 controllerserver.go:440] GetDiskLun returned: <nil>. Initiating attaching volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-90b4af34-20b0-494b-a46f-a3471b81f63a" to node "k8s-agentpool1-22475134-vmss000000".
I1022 06:48:35.376582       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:48:35.376606       1 controllerserver.go:461] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-90b4af34-20b0-494b-a46f-a3471b81f63a" to node "k8s-agentpool1-22475134-vmss000000"
I1022 06:48:35.863254       1 utils.go:95] GRPC call: /csi.v1.Controller/CreateVolume
I1022 06:48:35.863281       1 utils.go:96] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"requisite":[{"segments":{"topology.test.csi.azure.com/zone":""}}]},"capacity_range":{"required_bytes":5368709120},"name":"pvc-e6c2dd85-04e5-44ed-9840-b9b2e2da6777","parameters":{"csi.storage.k8s.io/pv/name":"pvc-e6c2dd85-04e5-44ed-9840-b9b2e2da6777","csi.storage.k8s.io/pvc/name":"inline-volume-tester-qkkht-my-volume-0","csi.storage.k8s.io/pvc/namespace":"ephemeral-1169"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]}
I1022 06:48:35.882433       1 azure_diskclient.go:135] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-e6c2dd85-04e5-44ed-9840-b9b2e2da6777, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-e6c2dd85-04e5-44ed-9840-b9b2e2da6777' under resource group 'kubetest-5p3qtork' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I1022 06:48:35.882490       1 controllerserver.go:271] begin to create azure disk(pvc-e6c2dd85-04e5-44ed-9840-b9b2e2da6777) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) diskZone() maxShares(0)
I1022 06:48:35.882501       1 azure_managedDiskController.go:88] azureDisk - creating new managed Name:pvc-e6c2dd85-04e5-44ed-9840-b9b2e2da6777 StorageAccountType:StandardSSD_LRS Size:5
I1022 06:48:36.144613       1 utils.go:95] GRPC call: /csi.v1.Controller/ControllerPublishVolume
I1022 06:48:36.144636       1 utils.go:96] GRPC request: {"node_id":"k8s-agentpool1-22475134-vmss000000","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-42ebe9ec-eca5-4926-8da8-4b21f93e4b2f","csi.storage.k8s.io/pvc/name":"pvc-n8qjn","csi.storage.k8s.io/pvc/namespace":"volumelimits-928","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1634884037556-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-42ebe9ec-eca5-4926-8da8-4b21f93e4b2f"}
I1022 06:48:36.177321       1 controllerserver.go:440] GetDiskLun returned: <nil>. Initiating attaching volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-42ebe9ec-eca5-4926-8da8-4b21f93e4b2f" to node "k8s-agentpool1-22475134-vmss000000".
I1022 06:48:36.177385       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
... skipping 17 lines ...
I1022 06:48:37.981810       1 azure_controller_common.go:242] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-4468e1b3-f470-4e0c-9404-16d282d00932" lun 3 to node "k8s-agentpool1-22475134-vmss000000", diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-42ebe9ec-eca5-4926-8da8-4b21f93e4b2f:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-42ebe9ec-eca5-4926-8da8-4b21f93e4b2f  true false 8}) /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-4468e1b3-f470-4e0c-9404-16d282d00932:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-4468e1b3-f470-4e0c-9404-16d282d00932  true false 3}) /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-6e5bb4b7-9c65-4f65-9a30-72575fc9afac:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-6e5bb4b7-9c65-4f65-9a30-72575fc9afac  true false 1}) /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-7292229f-a145-4b90-84c0-8ad707acb8f1:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-7292229f-a145-4b90-84c0-8ad707acb8f1  true false 6}) /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-90b4af34-20b0-494b-a46f-a3471b81f63a:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-90b4af34-20b0-494b-a46f-a3471b81f63a  true false 7}) /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-a38ef8f3-7817-4091-b6e0-7741e79fd589:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-a38ef8f3-7817-4091-b6e0-7741e79fd589  true false 2}) /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-d330a831-95bd-45e4-b274-292d41f80459:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-d330a831-95bd-45e4-b274-292d41f80459  true false 5}) /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-d92c57dc-1538-4651-bc16-6cc2c3ef92cc:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-d92c57dc-1538-4651-bc16-6cc2c3ef92cc  true false 4})]
I1022 06:48:37.981870       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:48:37.981928       1 azure_controller_vmss.go:122] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-42ebe9ec-eca5-4926-8da8-4b21f93e4b2f:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-42ebe9ec-eca5-4926-8da8-4b21f93e4b2f  true false 8}) /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-4468e1b3-f470-4e0c-9404-16d282d00932:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-4468e1b3-f470-4e0c-9404-16d282d00932  true false 3}) /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-6e5bb4b7-9c65-4f65-9a30-72575fc9afac:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-6e5bb4b7-9c65-4f65-9a30-72575fc9afac  true false 1}) /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-7292229f-a145-4b90-84c0-8ad707acb8f1:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-7292229f-a145-4b90-84c0-8ad707acb8f1  true false 6}) /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-90b4af34-20b0-494b-a46f-a3471b81f63a:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-90b4af34-20b0-494b-a46f-a3471b81f63a  true false 7}) /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-a38ef8f3-7817-4091-b6e0-7741e79fd589:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-a38ef8f3-7817-4091-b6e0-7741e79fd589  true false 2}) /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-d330a831-95bd-45e4-b274-292d41f80459:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-d330a831-95bd-45e4-b274-292d41f80459  true false 5}) /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-d92c57dc-1538-4651-bc16-6cc2c3ef92cc:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-d92c57dc-1538-4651-bc16-6cc2c3ef92cc  true false 4})])
I1022 06:48:38.207916       1 azure_managedDiskController.go:249] azureDisk - created new MD Name:pvc-e6c2dd85-04e5-44ed-9840-b9b2e2da6777 StorageAccountType:StandardSSD_LRS Size:5
I1022 06:48:38.207967       1 controllerserver.go:348] create azure disk(pvc-e6c2dd85-04e5-44ed-9840-b9b2e2da6777) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) tags(map[kubernetes.io-created-for-pv-name:pvc-e6c2dd85-04e5-44ed-9840-b9b2e2da6777 kubernetes.io-created-for-pvc-name:inline-volume-tester-qkkht-my-volume-0 kubernetes.io-created-for-pvc-namespace:ephemeral-1169]) successfully
I1022 06:48:38.207999       1 utils.go:102] GRPC response: {"volume":{"accessible_topology":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"capacity_bytes":5368709120,"content_source":{"Type":null},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-e6c2dd85-04e5-44ed-9840-b9b2e2da6777","csi.storage.k8s.io/pvc/name":"inline-volume-tester-qkkht-my-volume-0","csi.storage.k8s.io/pvc/namespace":"ephemeral-1169","requestedsizegib":"5"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-e6c2dd85-04e5-44ed-9840-b9b2e2da6777"}}
I1022 06:48:38.293831       1 azure_controller_vmss.go:134] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-42ebe9ec-eca5-4926-8da8-4b21f93e4b2f:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-42ebe9ec-eca5-4926-8da8-4b21f93e4b2f  true false 8}) /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-4468e1b3-f470-4e0c-9404-16d282d00932:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-4468e1b3-f470-4e0c-9404-16d282d00932  true false 3}) /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-6e5bb4b7-9c65-4f65-9a30-72575fc9afac:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-6e5bb4b7-9c65-4f65-9a30-72575fc9afac  true false 1}) /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-7292229f-a145-4b90-84c0-8ad707acb8f1:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-7292229f-a145-4b90-84c0-8ad707acb8f1  true false 6}) /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-90b4af34-20b0-494b-a46f-a3471b81f63a:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-90b4af34-20b0-494b-a46f-a3471b81f63a  true false 7}) /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-a38ef8f3-7817-4091-b6e0-7741e79fd589:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-a38ef8f3-7817-4091-b6e0-7741e79fd589  true false 2}) /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-d330a831-95bd-45e4-b274-292d41f80459:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-d330a831-95bd-45e4-b274-292d41f80459  true false 5}) /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-d92c57dc-1538-4651-bc16-6cc2c3ef92cc:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-d92c57dc-1538-4651-bc16-6cc2c3ef92cc  true false 4})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING)
I1022 06:48:38.294076       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:48:38.294101       1 azure_vmss.go:198] Couldn't find VMSS VM with nodeName k8s-agentpool1-22475134-vmss000000, refreshing the cache(vmss: k8s-agentpool1-22475134-vmss, rg: kubetest-5p3qtork)
I1022 06:48:38.360163       1 azure_controller_common.go:242] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-d92c57dc-1538-4651-bc16-6cc2c3ef92cc" lun 4 to node "k8s-agentpool1-22475134-vmss000000", diskMap: map[]
I1022 06:48:38.360232       1 controllerserver.go:465] Attach operation successful: volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-d92c57dc-1538-4651-bc16-6cc2c3ef92cc" attached to node "k8s-agentpool1-22475134-vmss000000".
I1022 06:48:38.360261       1 controllerserver.go:485] attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-d92c57dc-1538-4651-bc16-6cc2c3ef92cc" to node "k8s-agentpool1-22475134-vmss000000" successfully
I1022 06:48:38.360274       1 utils.go:102] GRPC response: {"publish_context":{"LUN":"4"}}
... skipping 53 lines ...
I1022 06:48:41.173396       1 controllerserver.go:461] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-3107aca1-7d54-4ac3-945a-25d6c0246a0c" to node "k8s-agentpool1-22475134-vmss000000"
I1022 06:48:41.343896       1 utils.go:95] GRPC call: /csi.v1.Controller/ControllerPublishVolume
I1022 06:48:41.343918       1 utils.go:96] GRPC request: {"node_id":"k8s-agentpool1-22475134-vmss000000","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-acd944ee-f869-4788-b766-c6715eee7d96","csi.storage.k8s.io/pvc/name":"pvc-8cr6x","csi.storage.k8s.io/pvc/namespace":"volumelimits-928","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1634884037556-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-acd944ee-f869-4788-b766-c6715eee7d96"}
I1022 06:48:41.369437       1 controllerserver.go:440] GetDiskLun returned: <nil>. Initiating attaching volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-acd944ee-f869-4788-b766-c6715eee7d96" to node "k8s-agentpool1-22475134-vmss000000".
I1022 06:48:41.369493       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:48:41.369539       1 controllerserver.go:461] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-acd944ee-f869-4788-b766-c6715eee7d96" to node "k8s-agentpool1-22475134-vmss000000"
I1022 06:48:41.461208       1 azure_controller_vmss.go:134] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-2cacef5a-9bc2-4a71-a22d-c72e842538c3:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-2cacef5a-9bc2-4a71-a22d-c72e842538c3  true false 9})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING)
I1022 06:48:41.461483       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:48:41.461542       1 azure_vmss.go:198] Couldn't find VMSS VM with nodeName k8s-agentpool1-22475134-vmss000000, refreshing the cache(vmss: k8s-agentpool1-22475134-vmss, rg: kubetest-5p3qtork)
I1022 06:48:41.543425       1 utils.go:95] GRPC call: /csi.v1.Controller/ControllerPublishVolume
I1022 06:48:41.543447       1 utils.go:96] GRPC request: {"node_id":"k8s-agentpool1-22475134-vmss000000","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-506f8b7e-8262-4a63-8f37-1a4145b662aa","csi.storage.k8s.io/pvc/name":"pvc-pc7vv","csi.storage.k8s.io/pvc/namespace":"volumelimits-928","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1634884037556-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-506f8b7e-8262-4a63-8f37-1a4145b662aa"}
I1022 06:48:41.550063       1 azure_controller_common.go:242] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-aba6fcc4-93be-4a03-b913-b0ad6dfd64e9" lun 10 to node "k8s-agentpool1-22475134-vmss000000", diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-3107aca1-7d54-4ac3-945a-25d6c0246a0c:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-3107aca1-7d54-4ac3-945a-25d6c0246a0c  true false 12}) /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-4a6ce6eb-10e0-401a-8257-fc05a4e4349f:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-4a6ce6eb-10e0-401a-8257-fc05a4e4349f  true false 11}) /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-aba6fcc4-93be-4a03-b913-b0ad6dfd64e9:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-aba6fcc4-93be-4a03-b913-b0ad6dfd64e9  true false 10}) /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-acd944ee-f869-4788-b766-c6715eee7d96:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-acd944ee-f869-4788-b766-c6715eee7d96  true false 13})]
I1022 06:48:41.550098       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
... skipping 3 lines ...
I1022 06:48:41.565378       1 controllerserver.go:461] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-506f8b7e-8262-4a63-8f37-1a4145b662aa" to node "k8s-agentpool1-22475134-vmss000000"
I1022 06:48:41.744070       1 utils.go:95] GRPC call: /csi.v1.Controller/ControllerPublishVolume
I1022 06:48:41.744094       1 utils.go:96] GRPC request: {"node_id":"k8s-agentpool1-22475134-vmss000000","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-a43477fa-15b0-4f50-92e3-0a58c87f24cd","csi.storage.k8s.io/pvc/name":"pvc-7q85h","csi.storage.k8s.io/pvc/namespace":"volumelimits-928","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1634884037556-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-a43477fa-15b0-4f50-92e3-0a58c87f24cd"}
I1022 06:48:41.766438       1 controllerserver.go:440] GetDiskLun returned: <nil>. Initiating attaching volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-a43477fa-15b0-4f50-92e3-0a58c87f24cd" to node "k8s-agentpool1-22475134-vmss000000".
I1022 06:48:41.766467       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:48:41.766545       1 controllerserver.go:461] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-a43477fa-15b0-4f50-92e3-0a58c87f24cd" to node "k8s-agentpool1-22475134-vmss000000"
I1022 06:48:41.785716       1 azure_controller_vmss.go:134] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-3107aca1-7d54-4ac3-945a-25d6c0246a0c:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-3107aca1-7d54-4ac3-945a-25d6c0246a0c  true false 12}) /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-4a6ce6eb-10e0-401a-8257-fc05a4e4349f:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-4a6ce6eb-10e0-401a-8257-fc05a4e4349f  true false 11}) /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-aba6fcc4-93be-4a03-b913-b0ad6dfd64e9:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-aba6fcc4-93be-4a03-b913-b0ad6dfd64e9  true false 10}) /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-acd944ee-f869-4788-b766-c6715eee7d96:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-acd944ee-f869-4788-b766-c6715eee7d96  true false 13})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING)
I1022 06:48:41.785989       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:48:41.786021       1 azure_vmss.go:198] Couldn't find VMSS VM with nodeName k8s-agentpool1-22475134-vmss000000, refreshing the cache(vmss: k8s-agentpool1-22475134-vmss, rg: kubetest-5p3qtork)
I1022 06:48:41.831841       1 azure_controller_common.go:242] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-4a6ce6eb-10e0-401a-8257-fc05a4e4349f" lun 11 to node "k8s-agentpool1-22475134-vmss000000", diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-506f8b7e-8262-4a63-8f37-1a4145b662aa:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-506f8b7e-8262-4a63-8f37-1a4145b662aa  true false 14}) /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-a43477fa-15b0-4f50-92e3-0a58c87f24cd:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-a43477fa-15b0-4f50-92e3-0a58c87f24cd  true false 15})]
I1022 06:48:41.831908       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:48:41.831951       1 azure_controller_vmss.go:122] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-506f8b7e-8262-4a63-8f37-1a4145b662aa:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-506f8b7e-8262-4a63-8f37-1a4145b662aa  true false 14}) /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-a43477fa-15b0-4f50-92e3-0a58c87f24cd:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-a43477fa-15b0-4f50-92e3-0a58c87f24cd  true false 15})])
I1022 06:48:41.943922       1 utils.go:95] GRPC call: /csi.v1.Controller/ControllerPublishVolume
I1022 06:48:41.943946       1 utils.go:96] GRPC request: {"node_id":"k8s-agentpool1-22475134-vmss000000","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-84c19a86-6835-4758-8251-2d417a93c3c7","csi.storage.k8s.io/pvc/name":"pvc-l7r25","csi.storage.k8s.io/pvc/namespace":"volumelimits-928","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1634884037556-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-84c19a86-6835-4758-8251-2d417a93c3c7"}
I1022 06:48:41.972233       1 controllerserver.go:440] GetDiskLun returned: <nil>. Initiating attaching volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-84c19a86-6835-4758-8251-2d417a93c3c7" to node "k8s-agentpool1-22475134-vmss000000".
I1022 06:48:41.972259       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:48:41.972298       1 controllerserver.go:461] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-84c19a86-6835-4758-8251-2d417a93c3c7" to node "k8s-agentpool1-22475134-vmss000000"
I1022 06:48:42.059155       1 azure_controller_vmss.go:134] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-506f8b7e-8262-4a63-8f37-1a4145b662aa:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-506f8b7e-8262-4a63-8f37-1a4145b662aa  true false 14}) /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-a43477fa-15b0-4f50-92e3-0a58c87f24cd:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-a43477fa-15b0-4f50-92e3-0a58c87f24cd  true false 15})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING)
I1022 06:48:42.059390       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:48:42.059418       1 azure_vmss.go:198] Couldn't find VMSS VM with nodeName k8s-agentpool1-22475134-vmss000000, refreshing the cache(vmss: k8s-agentpool1-22475134-vmss, rg: kubetest-5p3qtork)
I1022 06:48:42.105735       1 azure_controller_common.go:242] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-3107aca1-7d54-4ac3-945a-25d6c0246a0c" lun 12 to node "k8s-agentpool1-22475134-vmss000000", diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-84c19a86-6835-4758-8251-2d417a93c3c7:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-84c19a86-6835-4758-8251-2d417a93c3c7  true false 16})]
I1022 06:48:42.105773       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:48:42.105822       1 azure_controller_vmss.go:122] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-84c19a86-6835-4758-8251-2d417a93c3c7:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-84c19a86-6835-4758-8251-2d417a93c3c7  true false 16})])
I1022 06:48:42.144533       1 utils.go:95] GRPC call: /csi.v1.Controller/ControllerPublishVolume
I1022 06:48:42.144560       1 utils.go:96] GRPC request: {"node_id":"k8s-agentpool1-22475134-vmss000000","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-3b7d143a-9eee-4ed1-b383-cd1d6673be53","csi.storage.k8s.io/pvc/name":"pvc-ksc24","csi.storage.k8s.io/pvc/namespace":"volumelimits-928","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1634884037556-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-3b7d143a-9eee-4ed1-b383-cd1d6673be53"}
I1022 06:48:42.166951       1 controllerserver.go:440] GetDiskLun returned: <nil>. Initiating attaching volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-3b7d143a-9eee-4ed1-b383-cd1d6673be53" to node "k8s-agentpool1-22475134-vmss000000".
I1022 06:48:42.166986       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:48:42.167016       1 controllerserver.go:461] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-3b7d143a-9eee-4ed1-b383-cd1d6673be53" to node "k8s-agentpool1-22475134-vmss000000"
I1022 06:48:42.309549       1 azure_controller_vmss.go:134] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-84c19a86-6835-4758-8251-2d417a93c3c7:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-84c19a86-6835-4758-8251-2d417a93c3c7  true false 16})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING)
I1022 06:48:42.309811       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:48:42.309862       1 azure_vmss.go:198] Couldn't find VMSS VM with nodeName k8s-agentpool1-22475134-vmss000000, refreshing the cache(vmss: k8s-agentpool1-22475134-vmss, rg: kubetest-5p3qtork)
I1022 06:48:42.343728       1 utils.go:95] GRPC call: /csi.v1.Controller/ControllerPublishVolume
I1022 06:48:42.343750       1 utils.go:96] GRPC request: {"node_id":"k8s-agentpool1-22475134-vmss000000","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-7a1277c4-4dd2-4783-a3a2-70e6c7455fdc","csi.storage.k8s.io/pvc/name":"pvc-wg49s","csi.storage.k8s.io/pvc/namespace":"volumelimits-928","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1634884037556-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-7a1277c4-4dd2-4783-a3a2-70e6c7455fdc"}
I1022 06:48:42.369168       1 azure_controller_common.go:242] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-acd944ee-f869-4788-b766-c6715eee7d96" lun 13 to node "k8s-agentpool1-22475134-vmss000000", diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-3b7d143a-9eee-4ed1-b383-cd1d6673be53:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-3b7d143a-9eee-4ed1-b383-cd1d6673be53  true false 17})]
I1022 06:48:42.369199       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:48:42.369247       1 azure_controller_vmss.go:122] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-3b7d143a-9eee-4ed1-b383-cd1d6673be53:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-3b7d143a-9eee-4ed1-b383-cd1d6673be53  true false 17})])
I1022 06:48:42.369796       1 controllerserver.go:440] GetDiskLun returned: <nil>. Initiating attaching volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-7a1277c4-4dd2-4783-a3a2-70e6c7455fdc" to node "k8s-agentpool1-22475134-vmss000000".
I1022 06:48:42.369823       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:48:42.369848       1 controllerserver.go:461] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-7a1277c4-4dd2-4783-a3a2-70e6c7455fdc" to node "k8s-agentpool1-22475134-vmss000000"
I1022 06:48:42.535359       1 azure_controller_vmss.go:134] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-3b7d143a-9eee-4ed1-b383-cd1d6673be53:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-3b7d143a-9eee-4ed1-b383-cd1d6673be53  true false 17})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING)
I1022 06:48:42.535566       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:48:42.535590       1 azure_vmss.go:198] Couldn't find VMSS VM with nodeName k8s-agentpool1-22475134-vmss000000, refreshing the cache(vmss: k8s-agentpool1-22475134-vmss, rg: kubetest-5p3qtork)
I1022 06:48:42.543218       1 utils.go:95] GRPC call: /csi.v1.Controller/ControllerPublishVolume
I1022 06:48:42.543239       1 utils.go:96] GRPC request: {"node_id":"k8s-agentpool1-22475134-vmss000000","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-ac27c51e-aedd-4f4e-b24e-df2508a5e9b1","csi.storage.k8s.io/pvc/name":"pvc-k2wqn","csi.storage.k8s.io/pvc/namespace":"volumelimits-928","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1634884037556-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-ac27c51e-aedd-4f4e-b24e-df2508a5e9b1"}
I1022 06:48:42.593880       1 controllerserver.go:440] GetDiskLun returned: <nil>. Initiating attaching volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-ac27c51e-aedd-4f4e-b24e-df2508a5e9b1" to node "k8s-agentpool1-22475134-vmss000000".
I1022 06:48:42.593905       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:48:42.600100       1 azure_controller_common.go:242] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-506f8b7e-8262-4a63-8f37-1a4145b662aa" lun 14 to node "k8s-agentpool1-22475134-vmss000000", diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-7a1277c4-4dd2-4783-a3a2-70e6c7455fdc:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-7a1277c4-4dd2-4783-a3a2-70e6c7455fdc  true false 18})]
I1022 06:48:42.600126       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:48:42.600167       1 azure_controller_vmss.go:122] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-7a1277c4-4dd2-4783-a3a2-70e6c7455fdc:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-7a1277c4-4dd2-4783-a3a2-70e6c7455fdc  true false 18})])
I1022 06:48:42.600200       1 controllerserver.go:461] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-ac27c51e-aedd-4f4e-b24e-df2508a5e9b1" to node "k8s-agentpool1-22475134-vmss000000"
I1022 06:48:42.821330       1 azure_controller_vmss.go:134] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-7a1277c4-4dd2-4783-a3a2-70e6c7455fdc:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-7a1277c4-4dd2-4783-a3a2-70e6c7455fdc  true false 18})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING)
I1022 06:48:42.821557       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:48:42.821594       1 azure_vmss.go:198] Couldn't find VMSS VM with nodeName k8s-agentpool1-22475134-vmss000000, refreshing the cache(vmss: k8s-agentpool1-22475134-vmss, rg: kubetest-5p3qtork)
I1022 06:48:42.943888       1 utils.go:95] GRPC call: /csi.v1.Controller/ControllerPublishVolume
I1022 06:48:42.943916       1 utils.go:96] GRPC request: {"node_id":"k8s-agentpool1-22475134-vmss000000","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-09b74d52-8b3c-431d-a110-aef9ac1d65b2","csi.storage.k8s.io/pvc/name":"pvc-5n2wz","csi.storage.k8s.io/pvc/namespace":"volumelimits-928","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1634884037556-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-09b74d52-8b3c-431d-a110-aef9ac1d65b2"}
I1022 06:48:42.968139       1 controllerserver.go:440] GetDiskLun returned: <nil>. Initiating attaching volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-09b74d52-8b3c-431d-a110-aef9ac1d65b2" to node "k8s-agentpool1-22475134-vmss000000".
I1022 06:48:42.968167       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
... skipping 8 lines ...
I1022 06:48:43.340236       1 controllerserver.go:461] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-7598fafa-b8d8-45fc-bd09-7abb6e913a34" to node "k8s-agentpool1-22475134-vmss000000"
I1022 06:48:43.344285       1 utils.go:95] GRPC call: /csi.v1.Controller/ControllerPublishVolume
I1022 06:48:43.344308       1 utils.go:96] GRPC request: {"node_id":"k8s-agentpool1-22475134-vmss000000","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-13fe5e26-c069-46f7-836d-30618bd80606","csi.storage.k8s.io/pvc/name":"pvc-jgjh9","csi.storage.k8s.io/pvc/namespace":"volumelimits-928","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1634884037556-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-13fe5e26-c069-46f7-836d-30618bd80606"}
I1022 06:48:43.380761       1 controllerserver.go:440] GetDiskLun returned: <nil>. Initiating attaching volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-13fe5e26-c069-46f7-836d-30618bd80606" to node "k8s-agentpool1-22475134-vmss000000".
I1022 06:48:43.380812       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:48:43.380873       1 controllerserver.go:461] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-13fe5e26-c069-46f7-836d-30618bd80606" to node "k8s-agentpool1-22475134-vmss000000"
I1022 06:48:43.679757       1 azure_controller_vmss.go:134] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-ac27c51e-aedd-4f4e-b24e-df2508a5e9b1:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-ac27c51e-aedd-4f4e-b24e-df2508a5e9b1  true false 19})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING)
I1022 06:48:43.679928       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:48:43.679964       1 azure_vmss.go:198] Couldn't find VMSS VM with nodeName k8s-agentpool1-22475134-vmss000000, refreshing the cache(vmss: k8s-agentpool1-22475134-vmss, rg: kubetest-5p3qtork)
I1022 06:48:43.943402       1 utils.go:95] GRPC call: /csi.v1.Controller/ControllerPublishVolume
I1022 06:48:43.943429       1 utils.go:96] GRPC request: {"node_id":"k8s-agentpool1-22475134-vmss000000","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-7292229f-a145-4b90-84c0-8ad707acb8f1","csi.storage.k8s.io/pvc/name":"pvc-5hfc4","csi.storage.k8s.io/pvc/namespace":"volumelimits-928","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1634884037556-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-7292229f-a145-4b90-84c0-8ad707acb8f1"}
I1022 06:48:43.987409       1 controllerserver.go:440] GetDiskLun returned: <nil>. Initiating attaching volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-7292229f-a145-4b90-84c0-8ad707acb8f1" to node "k8s-agentpool1-22475134-vmss000000".
I1022 06:48:43.987450       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
... skipping 7 lines ...
I1022 06:48:44.744513       1 utils.go:96] GRPC request: {"node_id":"k8s-agentpool1-22475134-vmss000000","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-a38ef8f3-7817-4091-b6e0-7741e79fd589","csi.storage.k8s.io/pvc/name":"pvc-r8496","csi.storage.k8s.io/pvc/namespace":"volumelimits-928","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1634884037556-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-a38ef8f3-7817-4091-b6e0-7741e79fd589"}
I1022 06:48:44.768900       1 controllerserver.go:440] GetDiskLun returned: <nil>. Initiating attaching volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-a38ef8f3-7817-4091-b6e0-7741e79fd589" to node "k8s-agentpool1-22475134-vmss000000".
I1022 06:48:44.768983       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:48:44.769008       1 azure_controller_common.go:457] azureDisk - find disk: lun 2 name "pvc-a38ef8f3-7817-4091-b6e0-7741e79fd589" uri "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-a38ef8f3-7817-4091-b6e0-7741e79fd589"
I1022 06:48:44.769019       1 controllerserver.go:455] Attach operation is successful. volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-a38ef8f3-7817-4091-b6e0-7741e79fd589" is already attached to node "k8s-agentpool1-22475134-vmss000000" at lun 2.
I1022 06:48:44.769035       1 utils.go:102] GRPC response: {"publish_context":{"LUN":"2"}}
I1022 06:48:44.809493       1 azure_controller_vmss.go:134] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-09b74d52-8b3c-431d-a110-aef9ac1d65b2:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-09b74d52-8b3c-431d-a110-aef9ac1d65b2  true false 22}) /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-13fe5e26-c069-46f7-836d-30618bd80606:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-13fe5e26-c069-46f7-836d-30618bd80606  true false 21}) /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-7598fafa-b8d8-45fc-bd09-7abb6e913a34:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-7598fafa-b8d8-45fc-bd09-7abb6e913a34  true false 20})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING)
I1022 06:48:44.809668       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:48:44.809755       1 azure_vmss.go:198] Couldn't find VMSS VM with nodeName k8s-agentpool1-22475134-vmss000000, refreshing the cache(vmss: k8s-agentpool1-22475134-vmss, rg: kubetest-5p3qtork)
I1022 06:48:44.896887       1 azure_controller_common.go:242] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-3b7d143a-9eee-4ed1-b383-cd1d6673be53" lun 17 to node "k8s-agentpool1-22475134-vmss000000", diskMap: map[]
I1022 06:48:44.896924       1 controllerserver.go:465] Attach operation successful: volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-3b7d143a-9eee-4ed1-b383-cd1d6673be53" attached to node "k8s-agentpool1-22475134-vmss000000".
I1022 06:48:44.897006       1 controllerserver.go:485] attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-3b7d143a-9eee-4ed1-b383-cd1d6673be53" to node "k8s-agentpool1-22475134-vmss000000" successfully
I1022 06:48:44.897037       1 utils.go:102] GRPC response: {"publish_context":{"LUN":"17"}}
... skipping 28 lines ...
I1022 06:48:44.965921       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:48:44.965964       1 controllerserver.go:461] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-9273691d-6bca-4b0d-a798-48cd39f4be66" to node "k8s-agentpool1-22475134-vmss000000"
I1022 06:48:44.965977       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:48:44.965986       1 azure_controller_common.go:242] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-9273691d-6bca-4b0d-a798-48cd39f4be66" lun 23 to node "k8s-agentpool1-22475134-vmss000000", diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-9273691d-6bca-4b0d-a798-48cd39f4be66:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-9273691d-6bca-4b0d-a798-48cd39f4be66  true false 23})]
I1022 06:48:44.966017       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:48:44.966056       1 azure_controller_vmss.go:122] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-9273691d-6bca-4b0d-a798-48cd39f4be66:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-9273691d-6bca-4b0d-a798-48cd39f4be66  true false 23})])
I1022 06:48:45.134391       1 azure_controller_vmss.go:134] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-9273691d-6bca-4b0d-a798-48cd39f4be66:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-9273691d-6bca-4b0d-a798-48cd39f4be66  true false 23})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING)
I1022 06:48:45.144043       1 utils.go:95] GRPC call: /csi.v1.Controller/ControllerPublishVolume
I1022 06:48:45.144066       1 utils.go:96] GRPC request: {"node_id":"k8s-agentpool1-22475134-vmss000000","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-e7fe0c82-6611-4efa-b38b-831ca3daa630","csi.storage.k8s.io/pvc/name":"pvc-9p79g","csi.storage.k8s.io/pvc/namespace":"volumelimits-928","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1634884037556-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-e7fe0c82-6611-4efa-b38b-831ca3daa630"}
I1022 06:48:45.176538       1 controllerserver.go:440] GetDiskLun returned: <nil>. Initiating attaching volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-e7fe0c82-6611-4efa-b38b-831ca3daa630" to node "k8s-agentpool1-22475134-vmss000000".
I1022 06:48:45.176569       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:48:45.176588       1 azure_vmss.go:198] Couldn't find VMSS VM with nodeName k8s-agentpool1-22475134-vmss000000, refreshing the cache(vmss: k8s-agentpool1-22475134-vmss, rg: kubetest-5p3qtork)
I1022 06:48:45.244077       1 controllerserver.go:461] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-e7fe0c82-6611-4efa-b38b-831ca3daa630" to node "k8s-agentpool1-22475134-vmss000000"
... skipping 3 lines ...
I1022 06:48:45.244242       1 azure_controller_vmss.go:122] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-e7fe0c82-6611-4efa-b38b-831ca3daa630:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-e7fe0c82-6611-4efa-b38b-831ca3daa630  true false 24})])
I1022 06:48:45.344172       1 utils.go:95] GRPC call: /csi.v1.Controller/ControllerPublishVolume
I1022 06:48:45.344200       1 utils.go:96] GRPC request: {"node_id":"k8s-agentpool1-22475134-vmss000000","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-df1a5c24-dd06-4ce9-9ef0-7c6c82516986","csi.storage.k8s.io/pvc/name":"pvc-c6wnc","csi.storage.k8s.io/pvc/namespace":"volumelimits-928","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1634884037556-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-df1a5c24-dd06-4ce9-9ef0-7c6c82516986"}
I1022 06:48:45.396842       1 controllerserver.go:440] GetDiskLun returned: <nil>. Initiating attaching volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-df1a5c24-dd06-4ce9-9ef0-7c6c82516986" to node "k8s-agentpool1-22475134-vmss000000".
I1022 06:48:45.396869       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:48:45.396961       1 controllerserver.go:461] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-df1a5c24-dd06-4ce9-9ef0-7c6c82516986" to node "k8s-agentpool1-22475134-vmss000000"
I1022 06:48:45.434526       1 azure_controller_vmss.go:134] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-e7fe0c82-6611-4efa-b38b-831ca3daa630:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-e7fe0c82-6611-4efa-b38b-831ca3daa630  true false 24})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING)
I1022 06:48:45.434708       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:48:45.434762       1 azure_vmss.go:198] Couldn't find VMSS VM with nodeName k8s-agentpool1-22475134-vmss000000, refreshing the cache(vmss: k8s-agentpool1-22475134-vmss, rg: kubetest-5p3qtork)
I1022 06:48:45.495927       1 azure_controller_common.go:242] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-df1a5c24-dd06-4ce9-9ef0-7c6c82516986" lun 25 to node "k8s-agentpool1-22475134-vmss000000", diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-df1a5c24-dd06-4ce9-9ef0-7c6c82516986:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-df1a5c24-dd06-4ce9-9ef0-7c6c82516986  true false 25})]
I1022 06:48:45.495966       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:48:45.496023       1 azure_controller_vmss.go:122] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-df1a5c24-dd06-4ce9-9ef0-7c6c82516986:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-df1a5c24-dd06-4ce9-9ef0-7c6c82516986  true false 25})])
I1022 06:48:45.743734       1 utils.go:95] GRPC call: /csi.v1.Controller/ControllerPublishVolume
I1022 06:48:45.743764       1 utils.go:96] GRPC request: {"node_id":"k8s-agentpool1-22475134-vmss000000","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-625cb0a6-3ef0-4bb1-8a80-8d075355b851","csi.storage.k8s.io/pvc/name":"pvc-gxtth","csi.storage.k8s.io/pvc/namespace":"volumelimits-928","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1634884037556-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-625cb0a6-3ef0-4bb1-8a80-8d075355b851"}
I1022 06:48:45.766625       1 controllerserver.go:440] GetDiskLun returned: <nil>. Initiating attaching volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-625cb0a6-3ef0-4bb1-8a80-8d075355b851" to node "k8s-agentpool1-22475134-vmss000000".
I1022 06:48:45.766672       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:48:45.766725       1 controllerserver.go:461] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-625cb0a6-3ef0-4bb1-8a80-8d075355b851" to node "k8s-agentpool1-22475134-vmss000000"
I1022 06:48:45.785611       1 azure_controller_vmss.go:134] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-df1a5c24-dd06-4ce9-9ef0-7c6c82516986:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-df1a5c24-dd06-4ce9-9ef0-7c6c82516986  true false 25})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING)
I1022 06:48:45.785765       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:48:45.785832       1 azure_vmss.go:198] Couldn't find VMSS VM with nodeName k8s-agentpool1-22475134-vmss000000, refreshing the cache(vmss: k8s-agentpool1-22475134-vmss, rg: kubetest-5p3qtork)
I1022 06:48:45.853229       1 azure_controller_common.go:242] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-625cb0a6-3ef0-4bb1-8a80-8d075355b851" lun 26 to node "k8s-agentpool1-22475134-vmss000000", diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-625cb0a6-3ef0-4bb1-8a80-8d075355b851:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-625cb0a6-3ef0-4bb1-8a80-8d075355b851  true false 26})]
I1022 06:48:45.853263       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:48:45.853317       1 azure_controller_vmss.go:122] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-625cb0a6-3ef0-4bb1-8a80-8d075355b851:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-625cb0a6-3ef0-4bb1-8a80-8d075355b851  true false 26})])
I1022 06:48:45.943634       1 utils.go:95] GRPC call: /csi.v1.Controller/ControllerPublishVolume
I1022 06:48:45.943663       1 utils.go:96] GRPC request: {"node_id":"k8s-agentpool1-22475134-vmss000000","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-98953f15-d806-48a5-8627-b381bc559972","csi.storage.k8s.io/pvc/name":"pvc-t67h2","csi.storage.k8s.io/pvc/namespace":"volumelimits-928","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1634884037556-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-98953f15-d806-48a5-8627-b381bc559972"}
I1022 06:48:45.965929       1 controllerserver.go:440] GetDiskLun returned: <nil>. Initiating attaching volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-98953f15-d806-48a5-8627-b381bc559972" to node "k8s-agentpool1-22475134-vmss000000".
I1022 06:48:45.965974       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:48:45.966030       1 controllerserver.go:461] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-98953f15-d806-48a5-8627-b381bc559972" to node "k8s-agentpool1-22475134-vmss000000"
I1022 06:48:46.134449       1 azure_controller_vmss.go:134] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-625cb0a6-3ef0-4bb1-8a80-8d075355b851:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-625cb0a6-3ef0-4bb1-8a80-8d075355b851  true false 26})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING)
I1022 06:48:46.134678       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:48:46.134727       1 azure_vmss.go:198] Couldn't find VMSS VM with nodeName k8s-agentpool1-22475134-vmss000000, refreshing the cache(vmss: k8s-agentpool1-22475134-vmss, rg: kubetest-5p3qtork)
I1022 06:48:46.142558       1 utils.go:95] GRPC call: /csi.v1.Controller/ControllerPublishVolume
I1022 06:48:46.142580       1 utils.go:96] GRPC request: {"node_id":"k8s-agentpool1-22475134-vmss000000","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-07911b25-debe-4781-8c5f-b81e70b6bd7d","csi.storage.k8s.io/pvc/name":"pvc-ckxtv","csi.storage.k8s.io/pvc/namespace":"volumelimits-928","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1634884037556-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-07911b25-debe-4781-8c5f-b81e70b6bd7d"}
I1022 06:48:46.175039       1 controllerserver.go:440] GetDiskLun returned: <nil>. Initiating attaching volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-07911b25-debe-4781-8c5f-b81e70b6bd7d" to node "k8s-agentpool1-22475134-vmss000000".
I1022 06:48:46.175078       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:48:46.183754       1 azure_controller_common.go:242] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-98953f15-d806-48a5-8627-b381bc559972" lun 27 to node "k8s-agentpool1-22475134-vmss000000", diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-98953f15-d806-48a5-8627-b381bc559972:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-98953f15-d806-48a5-8627-b381bc559972  true false 27})]
I1022 06:48:46.183779       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:48:46.183844       1 controllerserver.go:461] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-07911b25-debe-4781-8c5f-b81e70b6bd7d" to node "k8s-agentpool1-22475134-vmss000000"
I1022 06:48:46.183874       1 azure_controller_vmss.go:122] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-98953f15-d806-48a5-8627-b381bc559972:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-98953f15-d806-48a5-8627-b381bc559972  true false 27})])
I1022 06:48:46.343429       1 utils.go:95] GRPC call: /csi.v1.Controller/ControllerPublishVolume
I1022 06:48:46.343452       1 utils.go:96] GRPC request: {"node_id":"k8s-agentpool1-22475134-vmss000000","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-4bf40635-c941-4c51-9db3-35374a82e0df","csi.storage.k8s.io/pvc/name":"pvc-74ggk","csi.storage.k8s.io/pvc/namespace":"volumelimits-928","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1634884037556-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-4bf40635-c941-4c51-9db3-35374a82e0df"}
I1022 06:48:46.364712       1 azure_controller_vmss.go:134] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-98953f15-d806-48a5-8627-b381bc559972:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-98953f15-d806-48a5-8627-b381bc559972  true false 27})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING)
I1022 06:48:46.364884       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:48:46.364946       1 azure_vmss.go:198] Couldn't find VMSS VM with nodeName k8s-agentpool1-22475134-vmss000000, refreshing the cache(vmss: k8s-agentpool1-22475134-vmss, rg: kubetest-5p3qtork)
I1022 06:48:46.371160       1 controllerserver.go:440] GetDiskLun returned: <nil>. Initiating attaching volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-4bf40635-c941-4c51-9db3-35374a82e0df" to node "k8s-agentpool1-22475134-vmss000000".
I1022 06:48:46.371186       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:48:46.412266       1 azure_controller_common.go:242] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-07911b25-debe-4781-8c5f-b81e70b6bd7d" lun 28 to node "k8s-agentpool1-22475134-vmss000000", diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-07911b25-debe-4781-8c5f-b81e70b6bd7d:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-07911b25-debe-4781-8c5f-b81e70b6bd7d  true false 28})]
I1022 06:48:46.412296       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:48:46.412363       1 controllerserver.go:461] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-4bf40635-c941-4c51-9db3-35374a82e0df" to node "k8s-agentpool1-22475134-vmss000000"
I1022 06:48:46.412368       1 azure_controller_vmss.go:122] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-07911b25-debe-4781-8c5f-b81e70b6bd7d:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-07911b25-debe-4781-8c5f-b81e70b6bd7d  true false 28})])
I1022 06:48:46.543984       1 utils.go:95] GRPC call: /csi.v1.Controller/ControllerPublishVolume
I1022 06:48:46.544009       1 utils.go:96] GRPC request: {"node_id":"k8s-agentpool1-22475134-vmss000000","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-ede47b18-01aa-4bba-b094-3bba45a8d78c","csi.storage.k8s.io/pvc/name":"pvc-qnvsj","csi.storage.k8s.io/pvc/namespace":"volumelimits-928","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1634884037556-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-ede47b18-01aa-4bba-b094-3bba45a8d78c"}
I1022 06:48:46.567965       1 controllerserver.go:440] GetDiskLun returned: <nil>. Initiating attaching volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-ede47b18-01aa-4bba-b094-3bba45a8d78c" to node "k8s-agentpool1-22475134-vmss000000".
I1022 06:48:46.567991       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:48:46.568062       1 controllerserver.go:461] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-ede47b18-01aa-4bba-b094-3bba45a8d78c" to node "k8s-agentpool1-22475134-vmss000000"
I1022 06:48:46.601593       1 azure_controller_vmss.go:134] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-07911b25-debe-4781-8c5f-b81e70b6bd7d:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-07911b25-debe-4781-8c5f-b81e70b6bd7d  true false 28})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING)
I1022 06:48:46.601778       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:48:46.601801       1 azure_vmss.go:198] Couldn't find VMSS VM with nodeName k8s-agentpool1-22475134-vmss000000, refreshing the cache(vmss: k8s-agentpool1-22475134-vmss, rg: kubetest-5p3qtork)
I1022 06:48:46.713250       1 azure_controller_common.go:242] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-4bf40635-c941-4c51-9db3-35374a82e0df" lun 29 to node "k8s-agentpool1-22475134-vmss000000", diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-4bf40635-c941-4c51-9db3-35374a82e0df:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-4bf40635-c941-4c51-9db3-35374a82e0df  true false 29}) /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-ede47b18-01aa-4bba-b094-3bba45a8d78c:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-ede47b18-01aa-4bba-b094-3bba45a8d78c  true false 30})]
I1022 06:48:46.713284       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:48:46.713344       1 azure_controller_vmss.go:122] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-4bf40635-c941-4c51-9db3-35374a82e0df:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-4bf40635-c941-4c51-9db3-35374a82e0df  true false 29}) /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-ede47b18-01aa-4bba-b094-3bba45a8d78c:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-ede47b18-01aa-4bba-b094-3bba45a8d78c  true false 30})])
I1022 06:48:46.743484       1 utils.go:95] GRPC call: /csi.v1.Controller/ControllerPublishVolume
I1022 06:48:46.743507       1 utils.go:96] GRPC request: {"node_id":"k8s-agentpool1-22475134-vmss000000","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-e01638a5-80eb-44cb-902c-a1c302a75244","csi.storage.k8s.io/pvc/name":"pvc-4bt7m","csi.storage.k8s.io/pvc/namespace":"volumelimits-928","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1634884037556-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-e01638a5-80eb-44cb-902c-a1c302a75244"}
I1022 06:48:46.766276       1 controllerserver.go:440] GetDiskLun returned: <nil>. Initiating attaching volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-e01638a5-80eb-44cb-902c-a1c302a75244" to node "k8s-agentpool1-22475134-vmss000000".
I1022 06:48:46.766322       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:48:46.766371       1 controllerserver.go:461] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-e01638a5-80eb-44cb-902c-a1c302a75244" to node "k8s-agentpool1-22475134-vmss000000"
I1022 06:48:46.898092       1 azure_controller_vmss.go:134] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-4bf40635-c941-4c51-9db3-35374a82e0df:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-4bf40635-c941-4c51-9db3-35374a82e0df  true false 29}) /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-ede47b18-01aa-4bba-b094-3bba45a8d78c:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-ede47b18-01aa-4bba-b094-3bba45a8d78c  true false 30})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING)
I1022 06:48:46.898292       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:48:46.898331       1 azure_vmss.go:198] Couldn't find VMSS VM with nodeName k8s-agentpool1-22475134-vmss000000, refreshing the cache(vmss: k8s-agentpool1-22475134-vmss, rg: kubetest-5p3qtork)
I1022 06:48:46.942667       1 utils.go:95] GRPC call: /csi.v1.Controller/ControllerPublishVolume
I1022 06:48:46.942688       1 utils.go:96] GRPC request: {"node_id":"k8s-agentpool1-22475134-vmss000000","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5","csi.storage.k8s.io/pvc/name":"pvc-tq8kk","csi.storage.k8s.io/pvc/namespace":"volumelimits-928","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1634884037556-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5"}
I1022 06:48:46.950483       1 azure_controller_common.go:242] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-ede47b18-01aa-4bba-b094-3bba45a8d78c" lun 30 to node "k8s-agentpool1-22475134-vmss000000", diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-e01638a5-80eb-44cb-902c-a1c302a75244:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-e01638a5-80eb-44cb-902c-a1c302a75244  true false 31})]
I1022 06:48:46.950515       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:48:46.950564       1 azure_controller_vmss.go:122] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-e01638a5-80eb-44cb-902c-a1c302a75244:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-e01638a5-80eb-44cb-902c-a1c302a75244  true false 31})])
I1022 06:48:46.965674       1 controllerserver.go:440] GetDiskLun returned: <nil>. Initiating attaching volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5" to node "k8s-agentpool1-22475134-vmss000000".
I1022 06:48:46.965714       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:48:46.965779       1 controllerserver.go:461] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5" to node "k8s-agentpool1-22475134-vmss000000"
I1022 06:48:47.118646       1 azure_controller_vmss.go:134] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-e01638a5-80eb-44cb-902c-a1c302a75244:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-e01638a5-80eb-44cb-902c-a1c302a75244  true false 31})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING)
I1022 06:48:47.118810       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:48:47.118876       1 azure_vmss.go:198] Couldn't find VMSS VM with nodeName k8s-agentpool1-22475134-vmss000000, refreshing the cache(vmss: k8s-agentpool1-22475134-vmss, rg: kubetest-5p3qtork)
I1022 06:48:47.185821       1 azure_controller_common.go:242] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-e01638a5-80eb-44cb-902c-a1c302a75244" lun 31 to node "k8s-agentpool1-22475134-vmss000000", diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5  true false 32})]
I1022 06:48:47.185851       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:48:47.185898       1 azure_controller_vmss.go:122] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5  true false 32})])
I1022 06:48:47.263406       1 azure_armclient.go:341] Received error in sendAsync.send: resourceID: https://management.azure.com/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool1-22475134-vmss/virtualMachines/0?api-version=2020-12-01, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 409, RawError: {

  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}
I1022 06:48:47.263426       1 azure_armclient.go:594] Received error in put.send: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool1-22475134-vmss/virtualMachines/0, error: %!s(<nil>)
E1022 06:48:47.263485       1 azure_controller_vmss.go:125] azureDisk - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5  true false 32})]) on rg(kubetest-5p3qtork) vm(k8s-agentpool1-22475134-vmss000000) failed, err: &{false 409 0001-01-01 00:00:00 +0000 UTC {

  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}}
I1022 06:48:47.263530       1 azure_controller_vmss.go:134] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5  true false 32})], &{%!s(bool=false) %!s(int=409) 0001-01-01 00:00:00 +0000 UTC {
  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}}) returned with %!v(MISSING)
E1022 06:48:47.263585       1 controllerserver.go:481] Attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-e01638a5-80eb-44cb-902c-a1c302a75244" to instance "k8s-agentpool1-22475134-vmss000000" failed with Retriable: false, RetryAfter: 0s, HTTPStatusCode: 409, RawError: {

  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}
E1022 06:48:47.263634       1 utils.go:100] GRPC error: Attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-e01638a5-80eb-44cb-902c-a1c302a75244" to instance "k8s-agentpool1-22475134-vmss000000" failed with Retriable: false, RetryAfter: 0s, HTTPStatusCode: 409, RawError: {

  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}
I1022 06:48:47.263662       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:48:47.263685       1 azure_vmss.go:198] Couldn't find VMSS VM with nodeName k8s-agentpool1-22475134-vmss000000, refreshing the cache(vmss: k8s-agentpool1-22475134-vmss, rg: kubetest-5p3qtork)
E1022 06:48:47.343596       1 controllerserver.go:481] Attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5" to instance "k8s-agentpool1-22475134-vmss000000" failed with could not find disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5) in current disk list(len: 32) nor in diskMap(map[])
E1022 06:48:47.343653       1 utils.go:100] GRPC error: Attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5" to instance "k8s-agentpool1-22475134-vmss000000" failed with could not find disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5) in current disk list(len: 32) nor in diskMap(map[])
I1022 06:48:47.743747       1 utils.go:95] GRPC call: /csi.v1.Controller/ControllerPublishVolume
I1022 06:48:47.743774       1 utils.go:96] GRPC request: {"node_id":"k8s-agentpool1-22475134-vmss000000","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-3b7d143a-9eee-4ed1-b383-cd1d6673be53","csi.storage.k8s.io/pvc/name":"pvc-ksc24","csi.storage.k8s.io/pvc/namespace":"volumelimits-928","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1634884037556-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-3b7d143a-9eee-4ed1-b383-cd1d6673be53"}
I1022 06:48:47.768519       1 controllerserver.go:440] GetDiskLun returned: <nil>. Initiating attaching volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-3b7d143a-9eee-4ed1-b383-cd1d6673be53" to node "k8s-agentpool1-22475134-vmss000000".
I1022 06:48:47.768553       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:48:47.768585       1 azure_controller_common.go:457] azureDisk - find disk: lun 17 name "pvc-3b7d143a-9eee-4ed1-b383-cd1d6673be53" uri "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-3b7d143a-9eee-4ed1-b383-cd1d6673be53"
I1022 06:48:47.768634       1 controllerserver.go:455] Attach operation is successful. volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-3b7d143a-9eee-4ed1-b383-cd1d6673be53" is already attached to node "k8s-agentpool1-22475134-vmss000000" at lun 17.
... skipping 11 lines ...
I1022 06:48:48.965905       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:48:48.965958       1 controllerserver.go:461] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-e6c2dd85-04e5-44ed-9840-b9b2e2da6777" to node "k8s-agentpool1-22475134-vmss000000"
I1022 06:48:48.966005       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:48:48.966035       1 azure_controller_common.go:242] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-e6c2dd85-04e5-44ed-9840-b9b2e2da6777" lun 32 to node "k8s-agentpool1-22475134-vmss000000", diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-e6c2dd85-04e5-44ed-9840-b9b2e2da6777:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-e6c2dd85-04e5-44ed-9840-b9b2e2da6777  true false 32})]
I1022 06:48:48.966056       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:48:48.966086       1 azure_controller_vmss.go:122] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-e6c2dd85-04e5-44ed-9840-b9b2e2da6777:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-e6c2dd85-04e5-44ed-9840-b9b2e2da6777  true false 32})])
I1022 06:48:49.027212       1 azure_armclient.go:341] Received error in sendAsync.send: resourceID: https://management.azure.com/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool1-22475134-vmss/virtualMachines/0?api-version=2020-12-01, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 409, RawError: {

  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}
I1022 06:48:49.027236       1 azure_armclient.go:594] Received error in put.send: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool1-22475134-vmss/virtualMachines/0, error: %!s(<nil>)
E1022 06:48:49.027275       1 azure_controller_vmss.go:125] azureDisk - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-e6c2dd85-04e5-44ed-9840-b9b2e2da6777:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-e6c2dd85-04e5-44ed-9840-b9b2e2da6777  true false 32})]) on rg(kubetest-5p3qtork) vm(k8s-agentpool1-22475134-vmss000000) failed, err: &{false 409 0001-01-01 00:00:00 +0000 UTC {

  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}}
I1022 06:48:49.027321       1 azure_controller_vmss.go:134] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-e6c2dd85-04e5-44ed-9840-b9b2e2da6777:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-e6c2dd85-04e5-44ed-9840-b9b2e2da6777  true false 32})], &{%!s(bool=false) %!s(int=409) 0001-01-01 00:00:00 +0000 UTC {
  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}}) returned with %!v(MISSING)
E1022 06:48:49.027395       1 controllerserver.go:481] Attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-e6c2dd85-04e5-44ed-9840-b9b2e2da6777" to instance "k8s-agentpool1-22475134-vmss000000" failed with Retriable: false, RetryAfter: 0s, HTTPStatusCode: 409, RawError: {

  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}
E1022 06:48:49.027437       1 utils.go:100] GRPC error: Attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-e6c2dd85-04e5-44ed-9840-b9b2e2da6777" to instance "k8s-agentpool1-22475134-vmss000000" failed with Retriable: false, RetryAfter: 0s, HTTPStatusCode: 409, RawError: {

  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}
I1022 06:48:49.143985       1 utils.go:95] GRPC call: /csi.v1.Controller/ControllerPublishVolume
... skipping 10 lines ...
I1022 06:48:49.367988       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:48:49.368041       1 controllerserver.go:461] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5" to node "k8s-agentpool1-22475134-vmss000000"
I1022 06:48:49.368071       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:48:49.368083       1 azure_controller_common.go:242] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5" lun 32 to node "k8s-agentpool1-22475134-vmss000000", diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5  true false 32})]
I1022 06:48:49.368107       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:48:49.368192       1 azure_controller_vmss.go:122] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5  true false 32})])
I1022 06:48:49.427074       1 azure_armclient.go:341] Received error in sendAsync.send: resourceID: https://management.azure.com/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool1-22475134-vmss/virtualMachines/0?api-version=2020-12-01, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 409, RawError: {

  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}
I1022 06:48:49.427094       1 azure_armclient.go:594] Received error in put.send: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool1-22475134-vmss/virtualMachines/0, error: %!s(<nil>)
E1022 06:48:49.427147       1 azure_controller_vmss.go:125] azureDisk - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5  true false 32})]) on rg(kubetest-5p3qtork) vm(k8s-agentpool1-22475134-vmss000000) failed, err: &{false 409 0001-01-01 00:00:00 +0000 UTC {

  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}}
I1022 06:48:49.427194       1 azure_controller_vmss.go:134] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5  true false 32})], &{%!s(bool=false) %!s(int=409) 0001-01-01 00:00:00 +0000 UTC {
  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}}) returned with %!v(MISSING)
E1022 06:48:49.427221       1 controllerserver.go:481] Attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5" to instance "k8s-agentpool1-22475134-vmss000000" failed with Retriable: false, RetryAfter: 0s, HTTPStatusCode: 409, RawError: {

  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}
E1022 06:48:49.427260       1 utils.go:100] GRPC error: Attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5" to instance "k8s-agentpool1-22475134-vmss000000" failed with Retriable: false, RetryAfter: 0s, HTTPStatusCode: 409, RawError: {

  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}
I1022 06:48:49.950224       1 utils.go:95] GRPC call: /csi.v1.Controller/ControllerPublishVolume
... skipping 3 lines ...
I1022 06:48:49.973060       1 azure_vmss.go:198] Couldn't find VMSS VM with nodeName k8s-agentpool1-22475134-vmss000000, refreshing the cache(vmss: k8s-agentpool1-22475134-vmss, rg: kubetest-5p3qtork)
I1022 06:48:50.020553       1 controllerserver.go:461] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-e6c2dd85-04e5-44ed-9840-b9b2e2da6777" to node "k8s-agentpool1-22475134-vmss000000"
I1022 06:48:50.020609       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:48:50.020629       1 azure_controller_common.go:242] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-e6c2dd85-04e5-44ed-9840-b9b2e2da6777" lun 32 to node "k8s-agentpool1-22475134-vmss000000", diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-e6c2dd85-04e5-44ed-9840-b9b2e2da6777:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-e6c2dd85-04e5-44ed-9840-b9b2e2da6777  true false 32})]
I1022 06:48:50.020644       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:48:50.020693       1 azure_controller_vmss.go:122] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-e6c2dd85-04e5-44ed-9840-b9b2e2da6777:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-e6c2dd85-04e5-44ed-9840-b9b2e2da6777  true false 32})])
I1022 06:48:50.084500       1 azure_armclient.go:341] Received error in sendAsync.send: resourceID: https://management.azure.com/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool1-22475134-vmss/virtualMachines/0?api-version=2020-12-01, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 409, RawError: {

  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}
I1022 06:48:50.084527       1 azure_armclient.go:594] Received error in put.send: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool1-22475134-vmss/virtualMachines/0, error: %!s(<nil>)
E1022 06:48:50.084604       1 azure_controller_vmss.go:125] azureDisk - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-e6c2dd85-04e5-44ed-9840-b9b2e2da6777:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-e6c2dd85-04e5-44ed-9840-b9b2e2da6777  true false 32})]) on rg(kubetest-5p3qtork) vm(k8s-agentpool1-22475134-vmss000000) failed, err: &{false 409 0001-01-01 00:00:00 +0000 UTC {

  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}}
I1022 06:48:50.084630       1 azure_controller_vmss.go:134] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-e6c2dd85-04e5-44ed-9840-b9b2e2da6777:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-e6c2dd85-04e5-44ed-9840-b9b2e2da6777  true false 32})], &{%!s(bool=false) %!s(int=409) 0001-01-01 00:00:00 +0000 UTC {
  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}}) returned with %!v(MISSING)
E1022 06:48:50.084703       1 controllerserver.go:481] Attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-e6c2dd85-04e5-44ed-9840-b9b2e2da6777" to instance "k8s-agentpool1-22475134-vmss000000" failed with Retriable: false, RetryAfter: 0s, HTTPStatusCode: 409, RawError: {

  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}
E1022 06:48:50.084743       1 utils.go:100] GRPC error: Attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-e6c2dd85-04e5-44ed-9840-b9b2e2da6777" to instance "k8s-agentpool1-22475134-vmss000000" failed with Retriable: false, RetryAfter: 0s, HTTPStatusCode: 409, RawError: {

  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}
I1022 06:48:50.344525       1 utils.go:95] GRPC call: /csi.v1.Controller/ControllerPublishVolume
... skipping 3 lines ...
I1022 06:48:50.367417       1 azure_vmss.go:198] Couldn't find VMSS VM with nodeName k8s-agentpool1-22475134-vmss000000, refreshing the cache(vmss: k8s-agentpool1-22475134-vmss, rg: kubetest-5p3qtork)
I1022 06:48:50.467471       1 controllerserver.go:461] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5" to node "k8s-agentpool1-22475134-vmss000000"
I1022 06:48:50.467520       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:48:50.467535       1 azure_controller_common.go:242] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5" lun 32 to node "k8s-agentpool1-22475134-vmss000000", diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5  true false 32})]
I1022 06:48:50.467549       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:48:50.467591       1 azure_controller_vmss.go:122] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5  true false 32})])
I1022 06:48:50.546433       1 azure_armclient.go:341] Received error in sendAsync.send: resourceID: https://management.azure.com/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool1-22475134-vmss/virtualMachines/0?api-version=2020-12-01, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 409, RawError: {

  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}
I1022 06:48:50.546459       1 azure_armclient.go:594] Received error in put.send: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool1-22475134-vmss/virtualMachines/0, error: %!s(<nil>)
E1022 06:48:50.546535       1 azure_controller_vmss.go:125] azureDisk - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5  true false 32})]) on rg(kubetest-5p3qtork) vm(k8s-agentpool1-22475134-vmss000000) failed, err: &{false 409 0001-01-01 00:00:00 +0000 UTC {

  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}}
I1022 06:48:50.546591       1 azure_controller_vmss.go:134] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5  true false 32})], &{%!s(bool=false) %!s(int=409) 0001-01-01 00:00:00 +0000 UTC {
  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}}) returned with %!v(MISSING)
E1022 06:48:50.546633       1 controllerserver.go:481] Attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5" to instance "k8s-agentpool1-22475134-vmss000000" failed with Retriable: false, RetryAfter: 0s, HTTPStatusCode: 409, RawError: {

  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}
E1022 06:48:50.546662       1 utils.go:100] GRPC error: Attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5" to instance "k8s-agentpool1-22475134-vmss000000" failed with Retriable: false, RetryAfter: 0s, HTTPStatusCode: 409, RawError: {

  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}
I1022 06:48:50.951505       1 utils.go:95] GRPC call: /csi.v1.Controller/ControllerPublishVolume
... skipping 3 lines ...
I1022 06:48:50.997508       1 azure_vmss.go:198] Couldn't find VMSS VM with nodeName k8s-agentpool1-22475134-vmss000000, refreshing the cache(vmss: k8s-agentpool1-22475134-vmss, rg: kubetest-5p3qtork)
I1022 06:48:51.043229       1 controllerserver.go:461] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-e6c2dd85-04e5-44ed-9840-b9b2e2da6777" to node "k8s-agentpool1-22475134-vmss000000"
I1022 06:48:51.043283       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:48:51.043300       1 azure_controller_common.go:242] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-e6c2dd85-04e5-44ed-9840-b9b2e2da6777" lun 32 to node "k8s-agentpool1-22475134-vmss000000", diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-e6c2dd85-04e5-44ed-9840-b9b2e2da6777:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-e6c2dd85-04e5-44ed-9840-b9b2e2da6777  true false 32})]
I1022 06:48:51.043315       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:48:51.043361       1 azure_controller_vmss.go:122] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-e6c2dd85-04e5-44ed-9840-b9b2e2da6777:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-e6c2dd85-04e5-44ed-9840-b9b2e2da6777  true false 32})])
I1022 06:48:51.117207       1 azure_armclient.go:341] Received error in sendAsync.send: resourceID: https://management.azure.com/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool1-22475134-vmss/virtualMachines/0?api-version=2020-12-01, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 409, RawError: {

  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}
I1022 06:48:51.117228       1 azure_armclient.go:594] Received error in put.send: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool1-22475134-vmss/virtualMachines/0, error: %!s(<nil>)
E1022 06:48:51.117291       1 azure_controller_vmss.go:125] azureDisk - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-e6c2dd85-04e5-44ed-9840-b9b2e2da6777:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-e6c2dd85-04e5-44ed-9840-b9b2e2da6777  true false 32})]) on rg(kubetest-5p3qtork) vm(k8s-agentpool1-22475134-vmss000000) failed, err: &{false 409 0001-01-01 00:00:00 +0000 UTC {

  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}}
I1022 06:48:51.117315       1 azure_controller_vmss.go:134] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-e6c2dd85-04e5-44ed-9840-b9b2e2da6777:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-e6c2dd85-04e5-44ed-9840-b9b2e2da6777  true false 32})], &{%!s(bool=false) %!s(int=409) 0001-01-01 00:00:00 +0000 UTC {
  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}}) returned with %!v(MISSING)
E1022 06:48:51.117377       1 controllerserver.go:481] Attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-e6c2dd85-04e5-44ed-9840-b9b2e2da6777" to instance "k8s-agentpool1-22475134-vmss000000" failed with Retriable: false, RetryAfter: 0s, HTTPStatusCode: 409, RawError: {

  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}
E1022 06:48:51.117418       1 utils.go:100] GRPC error: Attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-e6c2dd85-04e5-44ed-9840-b9b2e2da6777" to instance "k8s-agentpool1-22475134-vmss000000" failed with Retriable: false, RetryAfter: 0s, HTTPStatusCode: 409, RawError: {

  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}
I1022 06:48:52.345586       1 utils.go:95] GRPC call: /csi.v1.Controller/ControllerPublishVolume
... skipping 3 lines ...
I1022 06:48:52.370103       1 azure_vmss.go:198] Couldn't find VMSS VM with nodeName k8s-agentpool1-22475134-vmss000000, refreshing the cache(vmss: k8s-agentpool1-22475134-vmss, rg: kubetest-5p3qtork)
I1022 06:48:52.439847       1 controllerserver.go:461] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5" to node "k8s-agentpool1-22475134-vmss000000"
I1022 06:48:52.439897       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:48:52.439935       1 azure_controller_common.go:242] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5" lun 32 to node "k8s-agentpool1-22475134-vmss000000", diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5  true false 32})]
I1022 06:48:52.439950       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:48:52.440018       1 azure_controller_vmss.go:122] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5  true false 32})])
I1022 06:48:52.514900       1 azure_armclient.go:341] Received error in sendAsync.send: resourceID: https://management.azure.com/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool1-22475134-vmss/virtualMachines/0?api-version=2020-12-01, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 409, RawError: {

  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}
I1022 06:48:52.514919       1 azure_armclient.go:594] Received error in put.send: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool1-22475134-vmss/virtualMachines/0, error: %!s(<nil>)
E1022 06:48:52.514968       1 azure_controller_vmss.go:125] azureDisk - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5  true false 32})]) on rg(kubetest-5p3qtork) vm(k8s-agentpool1-22475134-vmss000000) failed, err: &{false 409 0001-01-01 00:00:00 +0000 UTC {

  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}}
I1022 06:48:52.514989       1 azure_controller_vmss.go:134] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5  true false 32})], &{%!s(bool=false) %!s(int=409) 0001-01-01 00:00:00 +0000 UTC {
  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}}) returned with %!v(MISSING)
E1022 06:48:52.515049       1 controllerserver.go:481] Attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5" to instance "k8s-agentpool1-22475134-vmss000000" failed with Retriable: false, RetryAfter: 0s, HTTPStatusCode: 409, RawError: {

  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}
E1022 06:48:52.515075       1 utils.go:100] GRPC error: Attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5" to instance "k8s-agentpool1-22475134-vmss000000" failed with Retriable: false, RetryAfter: 0s, HTTPStatusCode: 409, RawError: {

  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}
I1022 06:48:52.827890       1 utils.go:95] GRPC call: /csi.v1.Controller/ControllerUnpublishVolume
... skipping 11 lines ...
I1022 06:48:55.152891       1 azure_vmss.go:198] Couldn't find VMSS VM with nodeName k8s-agentpool1-22475134-vmss000000, refreshing the cache(vmss: k8s-agentpool1-22475134-vmss, rg: kubetest-5p3qtork)
I1022 06:48:55.199799       1 controllerserver.go:461] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-e6c2dd85-04e5-44ed-9840-b9b2e2da6777" to node "k8s-agentpool1-22475134-vmss000000"
I1022 06:48:55.199850       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:48:55.199867       1 azure_controller_common.go:242] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-e6c2dd85-04e5-44ed-9840-b9b2e2da6777" lun 32 to node "k8s-agentpool1-22475134-vmss000000", diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-e6c2dd85-04e5-44ed-9840-b9b2e2da6777:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-e6c2dd85-04e5-44ed-9840-b9b2e2da6777  true false 32})]
I1022 06:48:55.199882       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:48:55.199947       1 azure_controller_vmss.go:122] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-e6c2dd85-04e5-44ed-9840-b9b2e2da6777:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-e6c2dd85-04e5-44ed-9840-b9b2e2da6777  true false 32})])
I1022 06:48:55.259474       1 azure_armclient.go:341] Received error in sendAsync.send: resourceID: https://management.azure.com/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool1-22475134-vmss/virtualMachines/0?api-version=2020-12-01, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 409, RawError: {

  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}
I1022 06:48:55.259514       1 azure_armclient.go:594] Received error in put.send: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool1-22475134-vmss/virtualMachines/0, error: %!s(<nil>)
E1022 06:48:55.259566       1 azure_controller_vmss.go:125] azureDisk - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-e6c2dd85-04e5-44ed-9840-b9b2e2da6777:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-e6c2dd85-04e5-44ed-9840-b9b2e2da6777  true false 32})]) on rg(kubetest-5p3qtork) vm(k8s-agentpool1-22475134-vmss000000) failed, err: &{false 409 0001-01-01 00:00:00 +0000 UTC {

  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}}
I1022 06:48:55.259598       1 azure_controller_vmss.go:134] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-e6c2dd85-04e5-44ed-9840-b9b2e2da6777:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-e6c2dd85-04e5-44ed-9840-b9b2e2da6777  true false 32})], &{%!s(bool=false) %!s(int=409) 0001-01-01 00:00:00 +0000 UTC {
  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}}) returned with %!v(MISSING)
E1022 06:48:55.259632       1 controllerserver.go:481] Attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-e6c2dd85-04e5-44ed-9840-b9b2e2da6777" to instance "k8s-agentpool1-22475134-vmss000000" failed with Retriable: false, RetryAfter: 0s, HTTPStatusCode: 409, RawError: {

  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}
E1022 06:48:55.259653       1 utils.go:100] GRPC error: Attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-e6c2dd85-04e5-44ed-9840-b9b2e2da6777" to instance "k8s-agentpool1-22475134-vmss000000" failed with Retriable: false, RetryAfter: 0s, HTTPStatusCode: 409, RawError: {

  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}
I1022 06:49:00.525801       1 utils.go:95] GRPC call: /csi.v1.Controller/ControllerPublishVolume
... skipping 3 lines ...
I1022 06:49:00.549110       1 azure_vmss.go:198] Couldn't find VMSS VM with nodeName k8s-agentpool1-22475134-vmss000000, refreshing the cache(vmss: k8s-agentpool1-22475134-vmss, rg: kubetest-5p3qtork)
I1022 06:49:00.639469       1 controllerserver.go:461] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5" to node "k8s-agentpool1-22475134-vmss000000"
I1022 06:49:00.639523       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:49:00.639542       1 azure_controller_common.go:242] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5" lun 32 to node "k8s-agentpool1-22475134-vmss000000", diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5  true false 32})]
I1022 06:49:00.639558       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:49:00.639678       1 azure_controller_vmss.go:122] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5  true false 32})])
I1022 06:49:00.714816       1 azure_armclient.go:341] Received error in sendAsync.send: resourceID: https://management.azure.com/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool1-22475134-vmss/virtualMachines/0?api-version=2020-12-01, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 409, RawError: {

  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}
I1022 06:49:00.714841       1 azure_armclient.go:594] Received error in put.send: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool1-22475134-vmss/virtualMachines/0, error: %!s(<nil>)
E1022 06:49:00.714922       1 azure_controller_vmss.go:125] azureDisk - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5  true false 32})]) on rg(kubetest-5p3qtork) vm(k8s-agentpool1-22475134-vmss000000) failed, err: &{false 409 0001-01-01 00:00:00 +0000 UTC {

  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}}
I1022 06:49:00.714972       1 azure_controller_vmss.go:134] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5  true false 32})], &{%!s(bool=false) %!s(int=409) 0001-01-01 00:00:00 +0000 UTC {
  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}}) returned with %!v(MISSING)
E1022 06:49:00.715023       1 controllerserver.go:481] Attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5" to instance "k8s-agentpool1-22475134-vmss000000" failed with Retriable: false, RetryAfter: 0s, HTTPStatusCode: 409, RawError: {

  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}
E1022 06:49:00.715054       1 utils.go:100] GRPC error: Attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5" to instance "k8s-agentpool1-22475134-vmss000000" failed with Retriable: false, RetryAfter: 0s, HTTPStatusCode: 409, RawError: {

  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}
I1022 06:49:02.912014       1 utils.go:95] GRPC call: /csi.v1.Controller/ControllerUnpublishVolume
... skipping 56 lines ...
I1022 06:49:33.582164       1 utils.go:102] GRPC response: {"publish_context":{"LUN":"14"}}
I1022 06:49:33.632579       1 controllerserver.go:465] Attach operation successful: volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-4468e1b3-f470-4e0c-9404-16d282d00932" attached to node "k8s-agentpool1-22475134-vmss000000".
I1022 06:49:33.632602       1 controllerserver.go:485] attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-4468e1b3-f470-4e0c-9404-16d282d00932" to node "k8s-agentpool1-22475134-vmss000000" successfully
I1022 06:49:33.632615       1 utils.go:102] GRPC response: {"publish_context":{"LUN":"3"}}
I1022 06:49:33.638538       1 utils.go:95] GRPC call: /csi.v1.Controller/ControllerPublishVolume
I1022 06:49:33.638559       1 utils.go:96] GRPC request: {"node_id":"k8s-agentpool1-22475134-vmss000000","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-4468e1b3-f470-4e0c-9404-16d282d00932","csi.storage.k8s.io/pvc/name":"pvc-5zlhj","csi.storage.k8s.io/pvc/namespace":"volumelimits-928","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1634884037556-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-4468e1b3-f470-4e0c-9404-16d282d00932"}
I1022 06:49:33.694865       1 azure_armclient.go:341] Received error in sendAsync.send: resourceID: https://management.azure.com/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool1-22475134-vmss/virtualMachines/0?api-version=2020-12-01, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 409, RawError: {

  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}
I1022 06:49:33.694887       1 azure_armclient.go:594] Received error in put.send: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool1-22475134-vmss/virtualMachines/0, error: %!s(<nil>)
E1022 06:49:33.694938       1 azure_controller_vmss.go:125] azureDisk - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5  true false 32}) /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-e6c2dd85-04e5-44ed-9840-b9b2e2da6777:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-e6c2dd85-04e5-44ed-9840-b9b2e2da6777  true false 0})]) on rg(kubetest-5p3qtork) vm(k8s-agentpool1-22475134-vmss000000) failed, err: &{false 409 0001-01-01 00:00:00 +0000 UTC {

  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}}
I1022 06:49:33.694962       1 azure_controller_vmss.go:134] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5  true false 32}) /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-e6c2dd85-04e5-44ed-9840-b9b2e2da6777:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-e6c2dd85-04e5-44ed-9840-b9b2e2da6777  true false 0})], &{%!s(bool=false) %!s(int=409) 0001-01-01 00:00:00 +0000 UTC {
  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}}) returned with %!v(MISSING)
E1022 06:49:33.695020       1 controllerserver.go:481] Attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-e6c2dd85-04e5-44ed-9840-b9b2e2da6777" to instance "k8s-agentpool1-22475134-vmss000000" failed with Retriable: false, RetryAfter: 0s, HTTPStatusCode: 409, RawError: {

  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}
E1022 06:49:33.695046       1 utils.go:100] GRPC error: Attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-e6c2dd85-04e5-44ed-9840-b9b2e2da6777" to instance "k8s-agentpool1-22475134-vmss000000" failed with Retriable: false, RetryAfter: 0s, HTTPStatusCode: 409, RawError: {

  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}
I1022 06:49:33.695113       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:49:33.695136       1 azure_vmss.go:198] Couldn't find VMSS VM with nodeName k8s-agentpool1-22475134-vmss000000, refreshing the cache(vmss: k8s-agentpool1-22475134-vmss, rg: kubetest-5p3qtork)
I1022 06:49:33.723996       1 controllerserver.go:440] GetDiskLun returned: <nil>. Initiating attaching volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-4468e1b3-f470-4e0c-9404-16d282d00932" to node "k8s-agentpool1-22475134-vmss000000".
I1022 06:49:33.724023       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
E1022 06:49:33.765554       1 controllerserver.go:481] Attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5" to instance "k8s-agentpool1-22475134-vmss000000" failed with could not find disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5) in current disk list(len: 31) nor in diskMap(map[])
E1022 06:49:33.765600       1 utils.go:100] GRPC error: Attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5" to instance "k8s-agentpool1-22475134-vmss000000" failed with could not find disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5) in current disk list(len: 31) nor in diskMap(map[])
I1022 06:49:33.765603       1 azure_controller_common.go:457] azureDisk - find disk: lun 3 name "pvc-4468e1b3-f470-4e0c-9404-16d282d00932" uri "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-4468e1b3-f470-4e0c-9404-16d282d00932"
I1022 06:49:33.765646       1 controllerserver.go:455] Attach operation is successful. volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-4468e1b3-f470-4e0c-9404-16d282d00932" is already attached to node "k8s-agentpool1-22475134-vmss000000" at lun 3.
I1022 06:49:33.765682       1 utils.go:102] GRPC response: {"publish_context":{"LUN":"3"}}
I1022 06:49:34.013508       1 controllerserver.go:465] Attach operation successful: volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-a43477fa-15b0-4f50-92e3-0a58c87f24cd" attached to node "k8s-agentpool1-22475134-vmss000000".
I1022 06:49:34.013571       1 controllerserver.go:485] attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-a43477fa-15b0-4f50-92e3-0a58c87f24cd" to node "k8s-agentpool1-22475134-vmss000000" successfully
I1022 06:49:34.013605       1 utils.go:102] GRPC response: {"publish_context":{"LUN":"15"}}
... skipping 148 lines ...
I1022 06:49:38.308086       1 utils.go:96] GRPC request: {"node_id":"k8s-agentpool1-22475134-vmss000000","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-3107aca1-7d54-4ac3-945a-25d6c0246a0c","csi.storage.k8s.io/pvc/name":"pvc-kgfc4","csi.storage.k8s.io/pvc/namespace":"volumelimits-928","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1634884037556-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-3107aca1-7d54-4ac3-945a-25d6c0246a0c"}
I1022 06:49:38.331809       1 controllerserver.go:440] GetDiskLun returned: <nil>. Initiating attaching volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-3107aca1-7d54-4ac3-945a-25d6c0246a0c" to node "k8s-agentpool1-22475134-vmss000000".
I1022 06:49:38.331844       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:49:38.331871       1 azure_controller_common.go:457] azureDisk - find disk: lun 12 name "pvc-3107aca1-7d54-4ac3-945a-25d6c0246a0c" uri "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-3107aca1-7d54-4ac3-945a-25d6c0246a0c"
I1022 06:49:38.331883       1 controllerserver.go:455] Attach operation is successful. volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-3107aca1-7d54-4ac3-945a-25d6c0246a0c" is already attached to node "k8s-agentpool1-22475134-vmss000000" at lun 12.
I1022 06:49:38.331906       1 utils.go:102] GRPC response: {"publish_context":{"LUN":"12"}}
I1022 06:49:38.457162       1 azure_controller_vmss.go:134] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-cb861b94-be9c-4a97-a6fe-4793c8f2d550:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-cb861b94-be9c-4a97-a6fe-4793c8f2d550  true false 0})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING)
I1022 06:49:38.707791       1 utils.go:95] GRPC call: /csi.v1.Controller/ControllerPublishVolume
I1022 06:49:38.707818       1 utils.go:96] GRPC request: {"node_id":"k8s-agentpool1-22475134-vmss000000","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-acd944ee-f869-4788-b766-c6715eee7d96","csi.storage.k8s.io/pvc/name":"pvc-8cr6x","csi.storage.k8s.io/pvc/namespace":"volumelimits-928","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1634884037556-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-acd944ee-f869-4788-b766-c6715eee7d96"}
I1022 06:49:38.739979       1 controllerserver.go:440] GetDiskLun returned: <nil>. Initiating attaching volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-acd944ee-f869-4788-b766-c6715eee7d96" to node "k8s-agentpool1-22475134-vmss000000".
I1022 06:49:38.740007       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:49:38.740034       1 azure_controller_common.go:457] azureDisk - find disk: lun 13 name "pvc-acd944ee-f869-4788-b766-c6715eee7d96" uri "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-acd944ee-f869-4788-b766-c6715eee7d96"
I1022 06:49:38.740044       1 controllerserver.go:455] Attach operation is successful. volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-acd944ee-f869-4788-b766-c6715eee7d96" is already attached to node "k8s-agentpool1-22475134-vmss000000" at lun 13.
... skipping 7 lines ...
I1022 06:49:49.733695       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:49:49.733770       1 controllerserver.go:461] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-e6c2dd85-04e5-44ed-9840-b9b2e2da6777" to node "k8s-agentpool1-22475134-vmss000000"
I1022 06:49:49.733784       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:49:49.733814       1 azure_controller_common.go:242] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-e6c2dd85-04e5-44ed-9840-b9b2e2da6777" lun 0 to node "k8s-agentpool1-22475134-vmss000000", diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-e6c2dd85-04e5-44ed-9840-b9b2e2da6777:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-e6c2dd85-04e5-44ed-9840-b9b2e2da6777  true false 0})]
I1022 06:49:49.733864       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:49:49.733898       1 azure_controller_vmss.go:122] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-e6c2dd85-04e5-44ed-9840-b9b2e2da6777:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-e6c2dd85-04e5-44ed-9840-b9b2e2da6777  true false 0})])
I1022 06:49:49.899903       1 azure_controller_vmss.go:134] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-e6c2dd85-04e5-44ed-9840-b9b2e2da6777:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-e6c2dd85-04e5-44ed-9840-b9b2e2da6777  true false 0})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING)
I1022 06:49:53.117421       1 utils.go:95] GRPC call: /csi.v1.Controller/DeleteVolume
I1022 06:49:53.117447       1 utils.go:96] GRPC request: {"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-1d31be02-fc07-4482-97ed-0d5e45609233"}
I1022 06:49:53.117522       1 controllerserver.go:389] deleting azure disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-1d31be02-fc07-4482-97ed-0d5e45609233)
I1022 06:49:58.331272       1 azure_managedDiskController.go:286] azureDisk - deleted a managed disk: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-1d31be02-fc07-4482-97ed-0d5e45609233
I1022 06:49:58.331295       1 controllerserver.go:391] delete azure disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-1d31be02-fc07-4482-97ed-0d5e45609233) returned with <nil>
I1022 06:49:58.331328       1 utils.go:102] GRPC response: {}
I1022 06:50:03.048101       1 utils.go:95] GRPC call: /csi.v1.Controller/CreateVolume
I1022 06:50:03.048128       1 utils.go:96] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"requisite":[{"segments":{"topology.test.csi.azure.com/zone":""}}]},"capacity_range":{"required_bytes":5368709120},"name":"pvc-edaeaefb-87ad-46d5-aa2b-745e67d33883","parameters":{"csi.storage.k8s.io/pv/name":"pvc-edaeaefb-87ad-46d5-aa2b-745e67d33883","csi.storage.k8s.io/pvc/name":"test.csi.azure.comnn5vk","csi.storage.k8s.io/pvc/namespace":"provisioning-9989"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]}
I1022 06:50:03.101045       1 azure_diskclient.go:135] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-edaeaefb-87ad-46d5-aa2b-745e67d33883, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-edaeaefb-87ad-46d5-aa2b-745e67d33883' under resource group 'kubetest-5p3qtork' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I1022 06:50:03.101125       1 controllerserver.go:271] begin to create azure disk(pvc-edaeaefb-87ad-46d5-aa2b-745e67d33883) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) diskZone() maxShares(0)
I1022 06:50:03.101138       1 azure_managedDiskController.go:88] azureDisk - creating new managed Name:pvc-edaeaefb-87ad-46d5-aa2b-745e67d33883 StorageAccountType:StandardSSD_LRS Size:5
I1022 06:50:04.999299       1 controllerserver.go:465] Attach operation successful: volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-e6c2dd85-04e5-44ed-9840-b9b2e2da6777" attached to node "k8s-agentpool1-22475134-vmss000000".
I1022 06:50:04.999330       1 controllerserver.go:485] attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-e6c2dd85-04e5-44ed-9840-b9b2e2da6777" to node "k8s-agentpool1-22475134-vmss000000" successfully
I1022 06:50:04.999366       1 utils.go:102] GRPC response: {"publish_context":{"LUN":"0"}}
I1022 06:50:05.453966       1 azure_managedDiskController.go:249] azureDisk - created new MD Name:pvc-edaeaefb-87ad-46d5-aa2b-745e67d33883 StorageAccountType:StandardSSD_LRS Size:5
... skipping 6 lines ...
I1022 06:50:05.803817       1 azure_vmss.go:198] Couldn't find VMSS VM with nodeName k8s-agentpool1-22475134-vmss000000, refreshing the cache(vmss: k8s-agentpool1-22475134-vmss, rg: kubetest-5p3qtork)
I1022 06:50:05.882357       1 controllerserver.go:461] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5" to node "k8s-agentpool1-22475134-vmss000000"
I1022 06:50:05.882409       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:50:05.882428       1 azure_controller_common.go:242] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5" lun 32 to node "k8s-agentpool1-22475134-vmss000000", diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5  true false 32})]
I1022 06:50:05.882445       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:50:05.882511       1 azure_controller_vmss.go:122] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5  true false 32})])
I1022 06:50:05.942263       1 azure_armclient.go:341] Received error in sendAsync.send: resourceID: https://management.azure.com/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool1-22475134-vmss/virtualMachines/0?api-version=2020-12-01, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 409, RawError: {

  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}
I1022 06:50:05.942285       1 azure_armclient.go:594] Received error in put.send: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool1-22475134-vmss/virtualMachines/0, error: %!s(<nil>)
E1022 06:50:05.942345       1 azure_controller_vmss.go:125] azureDisk - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5  true false 32})]) on rg(kubetest-5p3qtork) vm(k8s-agentpool1-22475134-vmss000000) failed, err: &{false 409 0001-01-01 00:00:00 +0000 UTC {

  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}}
I1022 06:50:05.942389       1 azure_controller_vmss.go:134] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5  true false 32})], &{%!s(bool=false) %!s(int=409) 0001-01-01 00:00:00 +0000 UTC {
  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}}) returned with %!v(MISSING)
E1022 06:50:05.942449       1 controllerserver.go:481] Attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5" to instance "k8s-agentpool1-22475134-vmss000000" failed with Retriable: false, RetryAfter: 0s, HTTPStatusCode: 409, RawError: {

  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}
E1022 06:50:05.942501       1 utils.go:100] GRPC error: Attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5" to instance "k8s-agentpool1-22475134-vmss000000" failed with Retriable: false, RetryAfter: 0s, HTTPStatusCode: 409, RawError: {

  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}
I1022 06:50:07.772451       1 utils.go:95] GRPC call: /csi.v1.Controller/ControllerPublishVolume
... skipping 2 lines ...
I1022 06:50:07.795073       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000001" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:50:07.795093       1 controllerserver.go:461] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-edaeaefb-87ad-46d5-aa2b-745e67d33883" to node "k8s-agentpool1-22475134-vmss000001"
I1022 06:50:07.795126       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000001" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:50:07.795155       1 azure_controller_common.go:242] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-edaeaefb-87ad-46d5-aa2b-745e67d33883" lun 1 to node "k8s-agentpool1-22475134-vmss000001", diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-edaeaefb-87ad-46d5-aa2b-745e67d33883:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-edaeaefb-87ad-46d5-aa2b-745e67d33883  true false 1})]
I1022 06:50:07.795194       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000001" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:50:07.795208       1 azure_controller_vmss.go:122] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-edaeaefb-87ad-46d5-aa2b-745e67d33883:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-edaeaefb-87ad-46d5-aa2b-745e67d33883  true false 1})])
I1022 06:50:07.965017       1 azure_controller_vmss.go:134] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-edaeaefb-87ad-46d5-aa2b-745e67d33883:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-edaeaefb-87ad-46d5-aa2b-745e67d33883  true false 1})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING)
I1022 06:50:17.605518       1 utils.go:95] GRPC call: /csi.v1.Controller/CreateVolume
I1022 06:50:17.605583       1 utils.go:96] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"requisite":[{"segments":{"topology.test.csi.azure.com/zone":""}}]},"capacity_range":{"required_bytes":5368709120},"name":"pvc-37d949ec-a812-46a9-9be8-cfa70d417f53","parameters":{"csi.storage.k8s.io/pv/name":"pvc-37d949ec-a812-46a9-9be8-cfa70d417f53","csi.storage.k8s.io/pvc/name":"pvc-r9b6r","csi.storage.k8s.io/pvc/namespace":"snapshotting-6996"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}],"volume_content_source":{"Type":{"Snapshot":{"snapshot_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/snapshots/snapshot-a970bf72-0e85-4af8-a5dd-5e12a4747e8e"}}}}
I1022 06:50:17.634706       1 azure_diskclient.go:135] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-37d949ec-a812-46a9-9be8-cfa70d417f53, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-37d949ec-a812-46a9-9be8-cfa70d417f53' under resource group 'kubetest-5p3qtork' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I1022 06:50:17.634778       1 controllerserver.go:271] begin to create azure disk(pvc-37d949ec-a812-46a9-9be8-cfa70d417f53) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) diskZone() maxShares(0)
I1022 06:50:17.634793       1 azure_managedDiskController.go:88] azureDisk - creating new managed Name:pvc-37d949ec-a812-46a9-9be8-cfa70d417f53 StorageAccountType:StandardSSD_LRS Size:5
I1022 06:50:18.046987       1 controllerserver.go:465] Attach operation successful: volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-edaeaefb-87ad-46d5-aa2b-745e67d33883" attached to node "k8s-agentpool1-22475134-vmss000001".
I1022 06:50:18.047018       1 controllerserver.go:485] attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-edaeaefb-87ad-46d5-aa2b-745e67d33883" to node "k8s-agentpool1-22475134-vmss000001" successfully
I1022 06:50:18.047054       1 utils.go:102] GRPC response: {"publish_context":{"LUN":"1"}}
I1022 06:50:19.993234       1 azure_managedDiskController.go:249] azureDisk - created new MD Name:pvc-37d949ec-a812-46a9-9be8-cfa70d417f53 StorageAccountType:StandardSSD_LRS Size:5
... skipping 6 lines ...
I1022 06:50:23.030748       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000001" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:50:23.030774       1 controllerserver.go:461] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-37d949ec-a812-46a9-9be8-cfa70d417f53" to node "k8s-agentpool1-22475134-vmss000001"
I1022 06:50:23.030790       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000001" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:50:23.030802       1 azure_controller_common.go:242] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-37d949ec-a812-46a9-9be8-cfa70d417f53" lun 2 to node "k8s-agentpool1-22475134-vmss000001", diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-37d949ec-a812-46a9-9be8-cfa70d417f53:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-37d949ec-a812-46a9-9be8-cfa70d417f53  true false 2})]
I1022 06:50:23.030840       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000001" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:50:23.030857       1 azure_controller_vmss.go:122] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-37d949ec-a812-46a9-9be8-cfa70d417f53:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-37d949ec-a812-46a9-9be8-cfa70d417f53  true false 2})])
I1022 06:50:23.231961       1 azure_controller_vmss.go:134] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-37d949ec-a812-46a9-9be8-cfa70d417f53:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-37d949ec-a812-46a9-9be8-cfa70d417f53  true false 2})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING)
I1022 06:50:23.404645       1 utils.go:95] GRPC call: /csi.v1.Controller/ControllerUnpublishVolume
I1022 06:50:23.404673       1 utils.go:96] GRPC request: {"node_id":"k8s-agentpool1-22475134-vmss000001","volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-cb861b94-be9c-4a97-a6fe-4793c8f2d550"}
I1022 06:50:23.404765       1 controllerserver.go:524] Trying to detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-cb861b94-be9c-4a97-a6fe-4793c8f2d550 from node k8s-agentpool1-22475134-vmss000001
I1022 06:50:23.404776       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000001" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:50:23.404796       1 azure_vmss.go:198] Couldn't find VMSS VM with nodeName k8s-agentpool1-22475134-vmss000001, refreshing the cache(vmss: k8s-agentpool1-22475134-vmss, rg: kubetest-5p3qtork)
I1022 06:50:23.495261       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000001" is managed by VMSS since DisableAvailabilitySetNodes is set to true
... skipping 45 lines ...
I1022 06:51:09.993992       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:51:09.994083       1 controllerserver.go:461] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5" to node "k8s-agentpool1-22475134-vmss000000"
I1022 06:51:09.994116       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:51:09.994129       1 azure_controller_common.go:242] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5" lun 32 to node "k8s-agentpool1-22475134-vmss000000", diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5  true false 32})]
I1022 06:51:09.994170       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:51:09.994203       1 azure_controller_vmss.go:122] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5  true false 32})])
I1022 06:51:10.056166       1 azure_armclient.go:341] Received error in sendAsync.send: resourceID: https://management.azure.com/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool1-22475134-vmss/virtualMachines/0?api-version=2020-12-01, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 409, RawError: {

  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}
I1022 06:51:10.056204       1 azure_armclient.go:594] Received error in put.send: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool1-22475134-vmss/virtualMachines/0, error: %!s(<nil>)
E1022 06:51:10.056256       1 azure_controller_vmss.go:125] azureDisk - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5  true false 32})]) on rg(kubetest-5p3qtork) vm(k8s-agentpool1-22475134-vmss000000) failed, err: &{false 409 0001-01-01 00:00:00 +0000 UTC {

  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}}
I1022 06:51:10.056298       1 azure_controller_vmss.go:134] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5  true false 32})], &{%!s(bool=false) %!s(int=409) 0001-01-01 00:00:00 +0000 UTC {
  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}}) returned with %!v(MISSING)
E1022 06:51:10.056349       1 controllerserver.go:481] Attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5" to instance "k8s-agentpool1-22475134-vmss000000" failed with Retriable: false, RetryAfter: 0s, HTTPStatusCode: 409, RawError: {

  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}
E1022 06:51:10.056395       1 utils.go:100] GRPC error: Attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5" to instance "k8s-agentpool1-22475134-vmss000000" failed with Retriable: false, RetryAfter: 0s, HTTPStatusCode: 409, RawError: {

  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}
I1022 06:51:16.335433       1 utils.go:95] GRPC call: /csi.v1.Controller/CreateVolume
I1022 06:51:16.335459       1 utils.go:96] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"requisite":[{"segments":{"topology.test.csi.azure.com/zone":""}}]},"capacity_range":{"required_bytes":5368709120},"name":"pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739","parameters":{"csi.storage.k8s.io/pv/name":"pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739","csi.storage.k8s.io/pvc/name":"inline-volume-tester2-9rcj7-my-volume-0","csi.storage.k8s.io/pvc/namespace":"ephemeral-1169"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]}
I1022 06:51:16.354701       1 azure_diskclient.go:135] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739' under resource group 'kubetest-5p3qtork' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I1022 06:51:16.354782       1 controllerserver.go:271] begin to create azure disk(pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) diskZone() maxShares(0)
I1022 06:51:16.354795       1 azure_managedDiskController.go:88] azureDisk - creating new managed Name:pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739 StorageAccountType:StandardSSD_LRS Size:5
I1022 06:51:18.691533       1 azure_managedDiskController.go:249] azureDisk - created new MD Name:pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739 StorageAccountType:StandardSSD_LRS Size:5
I1022 06:51:18.691593       1 controllerserver.go:348] create azure disk(pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) tags(map[kubernetes.io-created-for-pv-name:pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739 kubernetes.io-created-for-pvc-name:inline-volume-tester2-9rcj7-my-volume-0 kubernetes.io-created-for-pvc-namespace:ephemeral-1169]) successfully
I1022 06:51:18.691615       1 utils.go:102] GRPC response: {"volume":{"accessible_topology":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"capacity_bytes":5368709120,"content_source":{"Type":null},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739","csi.storage.k8s.io/pvc/name":"inline-volume-tester2-9rcj7-my-volume-0","csi.storage.k8s.io/pvc/namespace":"ephemeral-1169","requestedsizegib":"5"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739"}}
I1022 06:51:20.003782       1 utils.go:95] GRPC call: /csi.v1.Controller/ControllerPublishVolume
... skipping 3 lines ...
I1022 06:51:20.026940       1 azure_vmss.go:198] Couldn't find VMSS VM with nodeName k8s-agentpool1-22475134-vmss000000, refreshing the cache(vmss: k8s-agentpool1-22475134-vmss, rg: kubetest-5p3qtork)
I1022 06:51:20.092668       1 controllerserver.go:461] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739" to node "k8s-agentpool1-22475134-vmss000000"
I1022 06:51:20.092730       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:51:20.092750       1 azure_controller_common.go:242] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739" lun 32 to node "k8s-agentpool1-22475134-vmss000000", diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739  true false 32})]
I1022 06:51:20.092767       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:51:20.092796       1 azure_controller_vmss.go:122] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739  true false 32})])
I1022 06:51:20.168390       1 azure_armclient.go:341] Received error in sendAsync.send: resourceID: https://management.azure.com/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool1-22475134-vmss/virtualMachines/0?api-version=2020-12-01, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 409, RawError: {

  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}
I1022 06:51:20.168418       1 azure_armclient.go:594] Received error in put.send: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool1-22475134-vmss/virtualMachines/0, error: %!s(<nil>)
E1022 06:51:20.168477       1 azure_controller_vmss.go:125] azureDisk - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739  true false 32})]) on rg(kubetest-5p3qtork) vm(k8s-agentpool1-22475134-vmss000000) failed, err: &{false 409 0001-01-01 00:00:00 +0000 UTC {

  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}}
I1022 06:51:20.168520       1 azure_controller_vmss.go:134] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739  true false 32})], &{%!s(bool=false) %!s(int=409) 0001-01-01 00:00:00 +0000 UTC {
  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}}) returned with %!v(MISSING)
E1022 06:51:20.168568       1 controllerserver.go:481] Attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739" to instance "k8s-agentpool1-22475134-vmss000000" failed with Retriable: false, RetryAfter: 0s, HTTPStatusCode: 409, RawError: {

  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}
E1022 06:51:20.168599       1 utils.go:100] GRPC error: Attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739" to instance "k8s-agentpool1-22475134-vmss000000" failed with Retriable: false, RetryAfter: 0s, HTTPStatusCode: 409, RawError: {

  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}
I1022 06:51:20.176368       1 utils.go:95] GRPC call: /csi.v1.Controller/ControllerPublishVolume
... skipping 3 lines ...
I1022 06:51:20.238382       1 azure_vmss.go:198] Couldn't find VMSS VM with nodeName k8s-agentpool1-22475134-vmss000000, refreshing the cache(vmss: k8s-agentpool1-22475134-vmss, rg: kubetest-5p3qtork)
I1022 06:51:20.289632       1 controllerserver.go:461] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739" to node "k8s-agentpool1-22475134-vmss000000"
I1022 06:51:20.289686       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:51:20.289705       1 azure_controller_common.go:242] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739" lun 32 to node "k8s-agentpool1-22475134-vmss000000", diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739  true false 32})]
I1022 06:51:20.289721       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:51:20.289753       1 azure_controller_vmss.go:122] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739  true false 32})])
I1022 06:51:20.372725       1 azure_armclient.go:341] Received error in sendAsync.send: resourceID: https://management.azure.com/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool1-22475134-vmss/virtualMachines/0?api-version=2020-12-01, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 409, RawError: {

  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}
I1022 06:51:20.372747       1 azure_armclient.go:594] Received error in put.send: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool1-22475134-vmss/virtualMachines/0, error: %!s(<nil>)
E1022 06:51:20.372837       1 azure_controller_vmss.go:125] azureDisk - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739  true false 32})]) on rg(kubetest-5p3qtork) vm(k8s-agentpool1-22475134-vmss000000) failed, err: &{false 409 0001-01-01 00:00:00 +0000 UTC {

  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}}
I1022 06:51:20.372881       1 azure_controller_vmss.go:134] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739  true false 32})], &{%!s(bool=false) %!s(int=409) 0001-01-01 00:00:00 +0000 UTC {
  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}}) returned with %!v(MISSING)
E1022 06:51:20.372909       1 controllerserver.go:481] Attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739" to instance "k8s-agentpool1-22475134-vmss000000" failed with Retriable: false, RetryAfter: 0s, HTTPStatusCode: 409, RawError: {

  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}
E1022 06:51:20.373011       1 utils.go:100] GRPC error: Attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739" to instance "k8s-agentpool1-22475134-vmss000000" failed with Retriable: false, RetryAfter: 0s, HTTPStatusCode: 409, RawError: {

  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}
I1022 06:51:21.148084       1 azure_vmss.go:319] Couldn't find VMSS VM with scaleSetName "k8s-agentpool1-22475134-vmss" and instanceID "0", refreshing the cache
... skipping 3 lines ...
I1022 06:51:21.270011       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:51:21.270075       1 controllerserver.go:461] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739" to node "k8s-agentpool1-22475134-vmss000000"
I1022 06:51:21.270107       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:51:21.270135       1 azure_controller_common.go:242] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739" lun 32 to node "k8s-agentpool1-22475134-vmss000000", diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739  true false 32})]
I1022 06:51:21.270159       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:51:21.270190       1 azure_controller_vmss.go:122] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739  true false 32})])
I1022 06:51:21.379214       1 azure_armclient.go:341] Received error in sendAsync.send: resourceID: https://management.azure.com/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool1-22475134-vmss/virtualMachines/0?api-version=2020-12-01, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 409, RawError: {

  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}
I1022 06:51:21.379245       1 azure_armclient.go:594] Received error in put.send: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool1-22475134-vmss/virtualMachines/0, error: %!s(<nil>)
E1022 06:51:21.379311       1 azure_controller_vmss.go:125] azureDisk - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739  true false 32})]) on rg(kubetest-5p3qtork) vm(k8s-agentpool1-22475134-vmss000000) failed, err: &{false 409 0001-01-01 00:00:00 +0000 UTC {

  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}}
I1022 06:51:21.379342       1 azure_controller_vmss.go:134] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739  true false 32})], &{%!s(bool=false) %!s(int=409) 0001-01-01 00:00:00 +0000 UTC {
  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}}) returned with %!v(MISSING)
E1022 06:51:21.379411       1 controllerserver.go:481] Attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739" to instance "k8s-agentpool1-22475134-vmss000000" failed with Retriable: false, RetryAfter: 0s, HTTPStatusCode: 409, RawError: {

  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}
E1022 06:51:21.379461       1 utils.go:100] GRPC error: Attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739" to instance "k8s-agentpool1-22475134-vmss000000" failed with Retriable: false, RetryAfter: 0s, HTTPStatusCode: 409, RawError: {

  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}
I1022 06:51:25.402255       1 utils.go:95] GRPC call: /csi.v1.Controller/ControllerPublishVolume
... skipping 3 lines ...
I1022 06:51:25.425659       1 azure_vmss.go:198] Couldn't find VMSS VM with nodeName k8s-agentpool1-22475134-vmss000000, refreshing the cache(vmss: k8s-agentpool1-22475134-vmss, rg: kubetest-5p3qtork)
I1022 06:51:25.479659       1 controllerserver.go:461] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739" to node "k8s-agentpool1-22475134-vmss000000"
I1022 06:51:25.479732       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:51:25.479750       1 azure_controller_common.go:242] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739" lun 32 to node "k8s-agentpool1-22475134-vmss000000", diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739  true false 32})]
I1022 06:51:25.479780       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:51:25.479823       1 azure_controller_vmss.go:122] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739  true false 32})])
I1022 06:51:25.539728       1 azure_armclient.go:341] Received error in sendAsync.send: resourceID: https://management.azure.com/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool1-22475134-vmss/virtualMachines/0?api-version=2020-12-01, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 409, RawError: {

  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}
I1022 06:51:25.539749       1 azure_armclient.go:594] Received error in put.send: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool1-22475134-vmss/virtualMachines/0, error: %!s(<nil>)
E1022 06:51:25.539825       1 azure_controller_vmss.go:125] azureDisk - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739  true false 32})]) on rg(kubetest-5p3qtork) vm(k8s-agentpool1-22475134-vmss000000) failed, err: &{false 409 0001-01-01 00:00:00 +0000 UTC {

  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}}
I1022 06:51:25.539866       1 azure_controller_vmss.go:134] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739  true false 32})], &{%!s(bool=false) %!s(int=409) 0001-01-01 00:00:00 +0000 UTC {
  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}}) returned with %!v(MISSING)
E1022 06:51:25.539915       1 controllerserver.go:481] Attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739" to instance "k8s-agentpool1-22475134-vmss000000" failed with Retriable: false, RetryAfter: 0s, HTTPStatusCode: 409, RawError: {

  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}
E1022 06:51:25.539947       1 utils.go:100] GRPC error: Attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739" to instance "k8s-agentpool1-22475134-vmss000000" failed with Retriable: false, RetryAfter: 0s, HTTPStatusCode: 409, RawError: {

  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}
I1022 06:51:33.552582       1 utils.go:95] GRPC call: /csi.v1.Controller/ControllerPublishVolume
... skipping 3 lines ...
I1022 06:51:33.626688       1 azure_vmss.go:198] Couldn't find VMSS VM with nodeName k8s-agentpool1-22475134-vmss000000, refreshing the cache(vmss: k8s-agentpool1-22475134-vmss, rg: kubetest-5p3qtork)
I1022 06:51:33.679733       1 controllerserver.go:461] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739" to node "k8s-agentpool1-22475134-vmss000000"
I1022 06:51:33.679784       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:51:33.679803       1 azure_controller_common.go:242] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739" lun 32 to node "k8s-agentpool1-22475134-vmss000000", diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739  true false 32})]
I1022 06:51:33.679818       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:51:33.679868       1 azure_controller_vmss.go:122] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739  true false 32})])
I1022 06:51:33.760748       1 azure_armclient.go:341] Received error in sendAsync.send: resourceID: https://management.azure.com/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool1-22475134-vmss/virtualMachines/0?api-version=2020-12-01, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 409, RawError: {

  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}
I1022 06:51:33.760771       1 azure_armclient.go:594] Received error in put.send: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool1-22475134-vmss/virtualMachines/0, error: %!s(<nil>)
E1022 06:51:33.760831       1 azure_controller_vmss.go:125] azureDisk - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739  true false 32})]) on rg(kubetest-5p3qtork) vm(k8s-agentpool1-22475134-vmss000000) failed, err: &{false 409 0001-01-01 00:00:00 +0000 UTC {

  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}}
I1022 06:51:33.760880       1 azure_controller_vmss.go:134] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739  true false 32})], &{%!s(bool=false) %!s(int=409) 0001-01-01 00:00:00 +0000 UTC {
  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}}) returned with %!v(MISSING)
E1022 06:51:33.760911       1 controllerserver.go:481] Attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739" to instance "k8s-agentpool1-22475134-vmss000000" failed with Retriable: false, RetryAfter: 0s, HTTPStatusCode: 409, RawError: {

  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}
E1022 06:51:33.760982       1 utils.go:100] GRPC error: Attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739" to instance "k8s-agentpool1-22475134-vmss000000" failed with Retriable: false, RetryAfter: 0s, HTTPStatusCode: 409, RawError: {

  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}
I1022 06:51:49.786836       1 utils.go:95] GRPC call: /csi.v1.Controller/ControllerPublishVolume
... skipping 3 lines ...
I1022 06:51:49.815686       1 azure_vmss.go:198] Couldn't find VMSS VM with nodeName k8s-agentpool1-22475134-vmss000000, refreshing the cache(vmss: k8s-agentpool1-22475134-vmss, rg: kubetest-5p3qtork)
I1022 06:51:49.888067       1 controllerserver.go:461] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739" to node "k8s-agentpool1-22475134-vmss000000"
I1022 06:51:49.888118       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:51:49.888151       1 azure_controller_common.go:242] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739" lun 32 to node "k8s-agentpool1-22475134-vmss000000", diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739  true false 32})]
I1022 06:51:49.888166       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:51:49.888212       1 azure_controller_vmss.go:122] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739  true false 32})])
I1022 06:51:49.975732       1 azure_armclient.go:341] Received error in sendAsync.send: resourceID: https://management.azure.com/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool1-22475134-vmss/virtualMachines/0?api-version=2020-12-01, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 409, RawError: {

  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}
I1022 06:51:49.975773       1 azure_armclient.go:594] Received error in put.send: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool1-22475134-vmss/virtualMachines/0, error: %!s(<nil>)
E1022 06:51:49.975814       1 azure_controller_vmss.go:125] azureDisk - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739  true false 32})]) on rg(kubetest-5p3qtork) vm(k8s-agentpool1-22475134-vmss000000) failed, err: &{false 409 0001-01-01 00:00:00 +0000 UTC {

  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}}
I1022 06:51:49.975849       1 azure_controller_vmss.go:134] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739  true false 32})], &{%!s(bool=false) %!s(int=409) 0001-01-01 00:00:00 +0000 UTC {
  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}}) returned with %!v(MISSING)
E1022 06:51:49.975886       1 controllerserver.go:481] Attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739" to instance "k8s-agentpool1-22475134-vmss000000" failed with Retriable: false, RetryAfter: 0s, HTTPStatusCode: 409, RawError: {

  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}
E1022 06:51:49.975916       1 utils.go:100] GRPC error: Attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739" to instance "k8s-agentpool1-22475134-vmss000000" failed with Retriable: false, RetryAfter: 0s, HTTPStatusCode: 409, RawError: {

  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}
I1022 06:52:06.203421       1 utils.go:95] GRPC call: /csi.v1.Controller/DeleteSnapshot
... skipping 22 lines ...
I1022 06:52:22.015876       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:52:22.015947       1 controllerserver.go:461] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739" to node "k8s-agentpool1-22475134-vmss000000"
I1022 06:52:22.015978       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:52:22.016007       1 azure_controller_common.go:242] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739" lun 32 to node "k8s-agentpool1-22475134-vmss000000", diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739  true false 32})]
I1022 06:52:22.016047       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:52:22.016078       1 azure_controller_vmss.go:122] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739  true false 32})])
I1022 06:52:22.077997       1 azure_armclient.go:341] Received error in sendAsync.send: resourceID: https://management.azure.com/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool1-22475134-vmss/virtualMachines/0?api-version=2020-12-01, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 409, RawError: {

  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}
I1022 06:52:22.078019       1 azure_armclient.go:594] Received error in put.send: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool1-22475134-vmss/virtualMachines/0, error: %!s(<nil>)
E1022 06:52:22.078076       1 azure_controller_vmss.go:125] azureDisk - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739  true false 32})]) on rg(kubetest-5p3qtork) vm(k8s-agentpool1-22475134-vmss000000) failed, err: &{false 409 0001-01-01 00:00:00 +0000 UTC {

  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}}
I1022 06:52:22.078100       1 azure_controller_vmss.go:134] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739  true false 32})], &{%!s(bool=false) %!s(int=409) 0001-01-01 00:00:00 +0000 UTC {
  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}}) returned with %!v(MISSING)
E1022 06:52:22.078166       1 controllerserver.go:481] Attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739" to instance "k8s-agentpool1-22475134-vmss000000" failed with Retriable: false, RetryAfter: 0s, HTTPStatusCode: 409, RawError: {

  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}
E1022 06:52:22.078194       1 utils.go:100] GRPC error: Attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739" to instance "k8s-agentpool1-22475134-vmss000000" failed with Retriable: false, RetryAfter: 0s, HTTPStatusCode: 409, RawError: {

  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}
I1022 06:52:27.024824       1 utils.go:95] GRPC call: /csi.v1.Controller/CreateVolume
I1022 06:52:27.024851       1 utils.go:96] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"requisite":[{"segments":{"topology.test.csi.azure.com/zone":""}}]},"capacity_range":{"required_bytes":5368709120},"name":"pvc-8588d2f3-db3f-433a-a39a-2071db111ae9","parameters":{"csi.storage.k8s.io/pv/name":"pvc-8588d2f3-db3f-433a-a39a-2071db111ae9","csi.storage.k8s.io/pvc/name":"inline-volume-tester-x75h7-my-volume-0","csi.storage.k8s.io/pvc/namespace":"ephemeral-7502"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]}
I1022 06:52:27.043380       1 azure_diskclient.go:135] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-8588d2f3-db3f-433a-a39a-2071db111ae9, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-8588d2f3-db3f-433a-a39a-2071db111ae9' under resource group 'kubetest-5p3qtork' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I1022 06:52:27.043484       1 controllerserver.go:271] begin to create azure disk(pvc-8588d2f3-db3f-433a-a39a-2071db111ae9) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) diskZone() maxShares(0)
I1022 06:52:27.043496       1 azure_managedDiskController.go:88] azureDisk - creating new managed Name:pvc-8588d2f3-db3f-433a-a39a-2071db111ae9 StorageAccountType:StandardSSD_LRS Size:5
I1022 06:52:29.400446       1 azure_managedDiskController.go:249] azureDisk - created new MD Name:pvc-8588d2f3-db3f-433a-a39a-2071db111ae9 StorageAccountType:StandardSSD_LRS Size:5
I1022 06:52:29.400503       1 controllerserver.go:348] create azure disk(pvc-8588d2f3-db3f-433a-a39a-2071db111ae9) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) tags(map[kubernetes.io-created-for-pv-name:pvc-8588d2f3-db3f-433a-a39a-2071db111ae9 kubernetes.io-created-for-pvc-name:inline-volume-tester-x75h7-my-volume-0 kubernetes.io-created-for-pvc-namespace:ephemeral-7502]) successfully
I1022 06:52:29.400526       1 utils.go:102] GRPC response: {"volume":{"accessible_topology":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"capacity_bytes":5368709120,"content_source":{"Type":null},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-8588d2f3-db3f-433a-a39a-2071db111ae9","csi.storage.k8s.io/pvc/name":"inline-volume-tester-x75h7-my-volume-0","csi.storage.k8s.io/pvc/namespace":"ephemeral-7502","requestedsizegib":"5"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-8588d2f3-db3f-433a-a39a-2071db111ae9"}}
I1022 06:52:29.485145       1 azure_controller_vmss.go:225] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000001) - detach disk(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-37d949ec-a812-46a9-9be8-cfa70d417f53:pvc-37d949ec-a812-46a9-9be8-cfa70d417f53]) returned with <nil>
... skipping 7 lines ...
I1022 06:52:30.151314       1 azure_vmss.go:198] Couldn't find VMSS VM with nodeName k8s-agentpool1-22475134-vmss000001, refreshing the cache(vmss: k8s-agentpool1-22475134-vmss, rg: kubetest-5p3qtork)
I1022 06:52:30.261912       1 controllerserver.go:461] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-8588d2f3-db3f-433a-a39a-2071db111ae9" to node "k8s-agentpool1-22475134-vmss000001"
I1022 06:52:30.262014       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000001" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:52:30.262034       1 azure_controller_common.go:242] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-8588d2f3-db3f-433a-a39a-2071db111ae9" lun 0 to node "k8s-agentpool1-22475134-vmss000001", diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-8588d2f3-db3f-433a-a39a-2071db111ae9:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-8588d2f3-db3f-433a-a39a-2071db111ae9  true false 0})]
I1022 06:52:30.262050       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000001" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:52:30.262062       1 azure_controller_vmss.go:122] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-8588d2f3-db3f-433a-a39a-2071db111ae9:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-8588d2f3-db3f-433a-a39a-2071db111ae9  true false 0})])
I1022 06:52:30.423627       1 azure_controller_vmss.go:134] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-8588d2f3-db3f-433a-a39a-2071db111ae9:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-8588d2f3-db3f-433a-a39a-2071db111ae9  true false 0})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING)
I1022 06:52:35.297237       1 utils.go:95] GRPC call: /csi.v1.Controller/DeleteVolume
I1022 06:52:35.297267       1 utils.go:96] GRPC request: {"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-37d949ec-a812-46a9-9be8-cfa70d417f53"}
I1022 06:52:35.297405       1 controllerserver.go:389] deleting azure disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-37d949ec-a812-46a9-9be8-cfa70d417f53)
I1022 06:52:40.544263       1 azure_managedDiskController.go:286] azureDisk - deleted a managed disk: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-37d949ec-a812-46a9-9be8-cfa70d417f53
I1022 06:52:40.544290       1 controllerserver.go:391] delete azure disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-37d949ec-a812-46a9-9be8-cfa70d417f53) returned with <nil>
I1022 06:52:40.544306       1 utils.go:102] GRPC response: {}
... skipping 6 lines ...
I1022 06:53:18.107133       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:53:18.107184       1 controllerserver.go:461] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5" to node "k8s-agentpool1-22475134-vmss000000"
I1022 06:53:18.107199       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:53:18.107230       1 azure_controller_common.go:242] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5" lun 32 to node "k8s-agentpool1-22475134-vmss000000", diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5  true false 32})]
I1022 06:53:18.107282       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:53:18.107310       1 azure_controller_vmss.go:122] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5  true false 32})])
I1022 06:53:18.174122       1 azure_armclient.go:341] Received error in sendAsync.send: resourceID: https://management.azure.com/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool1-22475134-vmss/virtualMachines/0?api-version=2020-12-01, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 409, RawError: {

  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}
I1022 06:53:18.174143       1 azure_armclient.go:594] Received error in put.send: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool1-22475134-vmss/virtualMachines/0, error: %!s(<nil>)
E1022 06:53:18.174197       1 azure_controller_vmss.go:125] azureDisk - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5  true false 32})]) on rg(kubetest-5p3qtork) vm(k8s-agentpool1-22475134-vmss000000) failed, err: &{false 409 0001-01-01 00:00:00 +0000 UTC {

  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}}
I1022 06:53:18.174228       1 azure_controller_vmss.go:134] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5  true false 32})], &{%!s(bool=false) %!s(int=409) 0001-01-01 00:00:00 +0000 UTC {
  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}}) returned with %!v(MISSING)
E1022 06:53:18.174288       1 controllerserver.go:481] Attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5" to instance "k8s-agentpool1-22475134-vmss000000" failed with Retriable: false, RetryAfter: 0s, HTTPStatusCode: 409, RawError: {

  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}
E1022 06:53:18.174316       1 utils.go:100] GRPC error: Attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5" to instance "k8s-agentpool1-22475134-vmss000000" failed with Retriable: false, RetryAfter: 0s, HTTPStatusCode: 409, RawError: {

  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}
I1022 06:53:21.537325       1 azure_vmss.go:319] Couldn't find VMSS VM with scaleSetName "k8s-agentpool1-22475134-vmss" and instanceID "0", refreshing the cache
... skipping 11 lines ...
I1022 06:53:26.182160       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:53:26.182217       1 controllerserver.go:461] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739" to node "k8s-agentpool1-22475134-vmss000000"
I1022 06:53:26.182247       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:53:26.182258       1 azure_controller_common.go:242] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739" lun 32 to node "k8s-agentpool1-22475134-vmss000000", diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739  true false 32})]
I1022 06:53:26.182312       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:53:26.182352       1 azure_controller_vmss.go:122] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739  true false 32})])
I1022 06:53:26.248464       1 azure_armclient.go:341] Received error in sendAsync.send: resourceID: https://management.azure.com/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool1-22475134-vmss/virtualMachines/0?api-version=2020-12-01, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 409, RawError: {

  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}
I1022 06:53:26.248498       1 azure_armclient.go:594] Received error in put.send: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool1-22475134-vmss/virtualMachines/0, error: %!s(<nil>)
E1022 06:53:26.248555       1 azure_controller_vmss.go:125] azureDisk - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739  true false 32})]) on rg(kubetest-5p3qtork) vm(k8s-agentpool1-22475134-vmss000000) failed, err: &{false 409 0001-01-01 00:00:00 +0000 UTC {

  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}}
I1022 06:53:26.248578       1 azure_controller_vmss.go:134] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739  true false 32})], &{%!s(bool=false) %!s(int=409) 0001-01-01 00:00:00 +0000 UTC {
  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}}) returned with %!v(MISSING)
E1022 06:53:26.248650       1 controllerserver.go:481] Attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739" to instance "k8s-agentpool1-22475134-vmss000000" failed with Retriable: false, RetryAfter: 0s, HTTPStatusCode: 409, RawError: {

  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}
E1022 06:53:26.248677       1 utils.go:100] GRPC error: Attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739" to instance "k8s-agentpool1-22475134-vmss000000" failed with Retriable: false, RetryAfter: 0s, HTTPStatusCode: 409, RawError: {

  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}
I1022 06:53:45.000144       1 azure_controller_vmss.go:225] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000001) - detach disk(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-8588d2f3-db3f-433a-a39a-2071db111ae9:pvc-8588d2f3-db3f-433a-a39a-2071db111ae9]) returned with <nil>
... skipping 5 lines ...
I1022 06:53:53.700114       1 controllerserver.go:389] deleting azure disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-8588d2f3-db3f-433a-a39a-2071db111ae9)
I1022 06:53:59.047552       1 azure_managedDiskController.go:286] azureDisk - deleted a managed disk: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-8588d2f3-db3f-433a-a39a-2071db111ae9
I1022 06:53:59.047581       1 controllerserver.go:391] delete azure disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-8588d2f3-db3f-433a-a39a-2071db111ae9) returned with <nil>
I1022 06:53:59.047637       1 utils.go:102] GRPC response: {}
I1022 06:54:01.674195       1 utils.go:95] GRPC call: /csi.v1.Controller/CreateVolume
I1022 06:54:01.674221       1 utils.go:96] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"requisite":[{"segments":{"topology.test.csi.azure.com/zone":""}}]},"capacity_range":{"required_bytes":5368709120},"name":"pvc-a38a5509-dba8-45b3-8f4e-e9375f2f74db","parameters":{"csi.storage.k8s.io/pv/name":"pvc-a38a5509-dba8-45b3-8f4e-e9375f2f74db","csi.storage.k8s.io/pvc/name":"inline-volume-tester-dmgdv-my-volume-0","csi.storage.k8s.io/pvc/namespace":"ephemeral-9813"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]}
I1022 06:54:01.697559       1 azure_diskclient.go:135] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-a38a5509-dba8-45b3-8f4e-e9375f2f74db, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-a38a5509-dba8-45b3-8f4e-e9375f2f74db' under resource group 'kubetest-5p3qtork' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I1022 06:54:01.697628       1 controllerserver.go:271] begin to create azure disk(pvc-a38a5509-dba8-45b3-8f4e-e9375f2f74db) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) diskZone() maxShares(0)
I1022 06:54:01.697655       1 azure_managedDiskController.go:88] azureDisk - creating new managed Name:pvc-a38a5509-dba8-45b3-8f4e-e9375f2f74db StorageAccountType:StandardSSD_LRS Size:5
I1022 06:54:04.274429       1 azure_managedDiskController.go:249] azureDisk - created new MD Name:pvc-a38a5509-dba8-45b3-8f4e-e9375f2f74db StorageAccountType:StandardSSD_LRS Size:5
I1022 06:54:04.274469       1 controllerserver.go:348] create azure disk(pvc-a38a5509-dba8-45b3-8f4e-e9375f2f74db) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) tags(map[kubernetes.io-created-for-pv-name:pvc-a38a5509-dba8-45b3-8f4e-e9375f2f74db kubernetes.io-created-for-pvc-name:inline-volume-tester-dmgdv-my-volume-0 kubernetes.io-created-for-pvc-namespace:ephemeral-9813]) successfully
I1022 06:54:04.274514       1 utils.go:102] GRPC response: {"volume":{"accessible_topology":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"capacity_bytes":5368709120,"content_source":{"Type":null},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-a38a5509-dba8-45b3-8f4e-e9375f2f74db","csi.storage.k8s.io/pvc/name":"inline-volume-tester-dmgdv-my-volume-0","csi.storage.k8s.io/pvc/namespace":"ephemeral-9813","requestedsizegib":"5"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-a38a5509-dba8-45b3-8f4e-e9375f2f74db"}}
I1022 06:54:05.138742       1 utils.go:95] GRPC call: /csi.v1.Controller/ControllerPublishVolume
... skipping 3 lines ...
I1022 06:54:05.163478       1 azure_vmss.go:198] Couldn't find VMSS VM with nodeName k8s-agentpool1-22475134-vmss000001, refreshing the cache(vmss: k8s-agentpool1-22475134-vmss, rg: kubetest-5p3qtork)
I1022 06:54:05.251559       1 controllerserver.go:461] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-a38a5509-dba8-45b3-8f4e-e9375f2f74db" to node "k8s-agentpool1-22475134-vmss000001"
I1022 06:54:05.251613       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000001" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:54:05.251631       1 azure_controller_common.go:242] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-a38a5509-dba8-45b3-8f4e-e9375f2f74db" lun 0 to node "k8s-agentpool1-22475134-vmss000001", diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-a38a5509-dba8-45b3-8f4e-e9375f2f74db:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-a38a5509-dba8-45b3-8f4e-e9375f2f74db  true false 0})]
I1022 06:54:05.251668       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000001" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:54:05.251706       1 azure_controller_vmss.go:122] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-a38a5509-dba8-45b3-8f4e-e9375f2f74db:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-a38a5509-dba8-45b3-8f4e-e9375f2f74db  true false 0})])
I1022 06:54:05.463187       1 azure_controller_vmss.go:134] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-a38a5509-dba8-45b3-8f4e-e9375f2f74db:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-a38a5509-dba8-45b3-8f4e-e9375f2f74db  true false 0})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING)
I1022 06:54:15.550273       1 controllerserver.go:465] Attach operation successful: volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-a38a5509-dba8-45b3-8f4e-e9375f2f74db" attached to node "k8s-agentpool1-22475134-vmss000001".
I1022 06:54:15.550305       1 controllerserver.go:485] attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-a38a5509-dba8-45b3-8f4e-e9375f2f74db" to node "k8s-agentpool1-22475134-vmss000001" successfully
I1022 06:54:15.550338       1 utils.go:102] GRPC response: {"publish_context":{"LUN":"0"}}
I1022 06:54:21.756970       1 azure_vmss.go:319] Couldn't find VMSS VM with scaleSetName "k8s-agentpool1-22475134-vmss" and instanceID "1", refreshing the cache
I1022 06:55:05.481159       1 utils.go:95] GRPC call: /csi.v1.Controller/ControllerUnpublishVolume
I1022 06:55:05.481189       1 utils.go:96] GRPC request: {"node_id":"k8s-agentpool1-22475134-vmss000001","volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-a38a5509-dba8-45b3-8f4e-e9375f2f74db"}
... skipping 16 lines ...
I1022 06:55:34.283254       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:55:34.283327       1 controllerserver.go:461] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739" to node "k8s-agentpool1-22475134-vmss000000"
I1022 06:55:34.283342       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:55:34.283371       1 azure_controller_common.go:242] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739" lun 32 to node "k8s-agentpool1-22475134-vmss000000", diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739  true false 32})]
I1022 06:55:34.283410       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:55:34.283440       1 azure_controller_vmss.go:122] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739  true false 32})])
I1022 06:55:34.356673       1 azure_armclient.go:341] Received error in sendAsync.send: resourceID: https://management.azure.com/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool1-22475134-vmss/virtualMachines/0?api-version=2020-12-01, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 409, RawError: {

  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}
I1022 06:55:34.356692       1 azure_armclient.go:594] Received error in put.send: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool1-22475134-vmss/virtualMachines/0, error: %!s(<nil>)
E1022 06:55:34.356742       1 azure_controller_vmss.go:125] azureDisk - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739  true false 32})]) on rg(kubetest-5p3qtork) vm(k8s-agentpool1-22475134-vmss000000) failed, err: &{false 409 0001-01-01 00:00:00 +0000 UTC {

  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}}
I1022 06:55:34.356808       1 azure_controller_vmss.go:134] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739  true false 32})], &{%!s(bool=false) %!s(int=409) 0001-01-01 00:00:00 +0000 UTC {
  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}}) returned with %!v(MISSING)
E1022 06:55:34.356840       1 controllerserver.go:481] Attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739" to instance "k8s-agentpool1-22475134-vmss000000" failed with Retriable: false, RetryAfter: 0s, HTTPStatusCode: 409, RawError: {

  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}
E1022 06:55:34.356884       1 utils.go:100] GRPC error: Attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739" to instance "k8s-agentpool1-22475134-vmss000000" failed with Retriable: false, RetryAfter: 0s, HTTPStatusCode: 409, RawError: {

  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}
I1022 06:55:34.867173       1 azure_managedDiskController.go:286] azureDisk - deleted a managed disk: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-a38a5509-dba8-45b3-8f4e-e9375f2f74db
I1022 06:55:34.867217       1 controllerserver.go:391] delete azure disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-a38a5509-dba8-45b3-8f4e-e9375f2f74db) returned with <nil>
I1022 06:55:34.867249       1 utils.go:102] GRPC response: {}
I1022 06:55:37.323278       1 utils.go:95] GRPC call: /csi.v1.Controller/CreateVolume
I1022 06:55:37.323302       1 utils.go:96] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"requisite":[{"segments":{"topology.test.csi.azure.com/zone":""}}]},"capacity_range":{"required_bytes":5368709120},"name":"pvc-ffb53fd3-51c8-4820-ab0f-048c95ab3652","parameters":{"csi.storage.k8s.io/pv/name":"pvc-ffb53fd3-51c8-4820-ab0f-048c95ab3652","csi.storage.k8s.io/pvc/name":"pvc-bljw2","csi.storage.k8s.io/pvc/namespace":"provisioning-8468"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]}
I1022 06:55:37.342829       1 azure_diskclient.go:135] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-ffb53fd3-51c8-4820-ab0f-048c95ab3652, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-ffb53fd3-51c8-4820-ab0f-048c95ab3652' under resource group 'kubetest-5p3qtork' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I1022 06:55:37.342908       1 controllerserver.go:271] begin to create azure disk(pvc-ffb53fd3-51c8-4820-ab0f-048c95ab3652) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) diskZone() maxShares(0)
I1022 06:55:37.342945       1 azure_managedDiskController.go:88] azureDisk - creating new managed Name:pvc-ffb53fd3-51c8-4820-ab0f-048c95ab3652 StorageAccountType:StandardSSD_LRS Size:5
I1022 06:55:39.795643       1 azure_managedDiskController.go:249] azureDisk - created new MD Name:pvc-ffb53fd3-51c8-4820-ab0f-048c95ab3652 StorageAccountType:StandardSSD_LRS Size:5
I1022 06:55:39.795679       1 controllerserver.go:348] create azure disk(pvc-ffb53fd3-51c8-4820-ab0f-048c95ab3652) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) tags(map[kubernetes.io-created-for-pv-name:pvc-ffb53fd3-51c8-4820-ab0f-048c95ab3652 kubernetes.io-created-for-pvc-name:pvc-bljw2 kubernetes.io-created-for-pvc-namespace:provisioning-8468]) successfully
I1022 06:55:39.795756       1 utils.go:102] GRPC response: {"volume":{"accessible_topology":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"capacity_bytes":5368709120,"content_source":{"Type":null},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-ffb53fd3-51c8-4820-ab0f-048c95ab3652","csi.storage.k8s.io/pvc/name":"pvc-bljw2","csi.storage.k8s.io/pvc/namespace":"provisioning-8468","requestedsizegib":"5"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-ffb53fd3-51c8-4820-ab0f-048c95ab3652"}}
I1022 06:55:42.208771       1 utils.go:95] GRPC call: /csi.v1.Controller/ControllerPublishVolume
... skipping 3 lines ...
I1022 06:55:42.231363       1 azure_vmss.go:198] Couldn't find VMSS VM with nodeName k8s-agentpool1-22475134-vmss000001, refreshing the cache(vmss: k8s-agentpool1-22475134-vmss, rg: kubetest-5p3qtork)
I1022 06:55:42.312808       1 controllerserver.go:461] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-ffb53fd3-51c8-4820-ab0f-048c95ab3652" to node "k8s-agentpool1-22475134-vmss000001"
I1022 06:55:42.312864       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000001" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:55:42.312881       1 azure_controller_common.go:242] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-ffb53fd3-51c8-4820-ab0f-048c95ab3652" lun 0 to node "k8s-agentpool1-22475134-vmss000001", diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-ffb53fd3-51c8-4820-ab0f-048c95ab3652:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-ffb53fd3-51c8-4820-ab0f-048c95ab3652  true false 0})]
I1022 06:55:42.312896       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000001" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:55:42.312907       1 azure_controller_vmss.go:122] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-ffb53fd3-51c8-4820-ab0f-048c95ab3652:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-ffb53fd3-51c8-4820-ab0f-048c95ab3652  true false 0})])
I1022 06:55:42.517282       1 azure_controller_vmss.go:134] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-ffb53fd3-51c8-4820-ab0f-048c95ab3652:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-ffb53fd3-51c8-4820-ab0f-048c95ab3652  true false 0})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING)
I1022 06:55:52.636259       1 controllerserver.go:465] Attach operation successful: volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-ffb53fd3-51c8-4820-ab0f-048c95ab3652" attached to node "k8s-agentpool1-22475134-vmss000001".
I1022 06:55:52.636292       1 controllerserver.go:485] attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-ffb53fd3-51c8-4820-ab0f-048c95ab3652" to node "k8s-agentpool1-22475134-vmss000001" successfully
I1022 06:55:52.636325       1 utils.go:102] GRPC response: {"publish_context":{"LUN":"0"}}
I1022 06:56:10.626961       1 utils.go:95] GRPC call: /csi.v1.Identity/GetPluginInfo
I1022 06:56:10.626990       1 utils.go:96] GRPC request: {}
I1022 06:56:10.627123       1 utils.go:102] GRPC response: {"name":"test.csi.azure.com","vendor_version":"v1.9.0-716f56a44514b799d2d6c6922145f22026e7d4e2"}
I1022 06:56:10.627475       1 utils.go:95] GRPC call: /csi.v1.Controller/CreateSnapshot
I1022 06:56:10.627494       1 utils.go:96] GRPC request: {"name":"snapshot-a3cc0482-9ff3-4916-8ff3-6f2f830ccbc2","source_volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-ffb53fd3-51c8-4820-ab0f-048c95ab3652"}
I1022 06:56:10.627565       1 controllerserver.go:909] begin to create snapshot(snapshot-a3cc0482-9ff3-4916-8ff3-6f2f830ccbc2, incremental: true) under rg(kubetest-5p3qtork)
I1022 06:56:13.007265       1 controllerserver.go:918] create snapshot(snapshot-a3cc0482-9ff3-4916-8ff3-6f2f830ccbc2) under rg(kubetest-5p3qtork) successfully
I1022 06:56:13.030962       1 utils.go:102] GRPC response: {"snapshot":{"creation_time":{"nanos":848870900,"seconds":1634885770},"ready_to_use":true,"size_bytes":5368709120,"snapshot_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/snapshots/snapshot-a3cc0482-9ff3-4916-8ff3-6f2f830ccbc2","source_volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-ffb53fd3-51c8-4820-ab0f-048c95ab3652"}}
I1022 06:56:15.368326       1 utils.go:95] GRPC call: /csi.v1.Controller/CreateVolume
I1022 06:56:15.368355       1 utils.go:96] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"requisite":[{"segments":{"topology.test.csi.azure.com/zone":""}}]},"capacity_range":{"required_bytes":5368709120},"name":"pvc-73e8f02f-b4d5-42cd-af19-1cd55fe6e383","parameters":{"csi.storage.k8s.io/pv/name":"pvc-73e8f02f-b4d5-42cd-af19-1cd55fe6e383","csi.storage.k8s.io/pvc/name":"pvc-74ttj","csi.storage.k8s.io/pvc/namespace":"provisioning-8468"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}],"volume_content_source":{"Type":{"Snapshot":{"snapshot_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/snapshots/snapshot-a3cc0482-9ff3-4916-8ff3-6f2f830ccbc2"}}}}
I1022 06:56:15.387389       1 azure_diskclient.go:135] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-73e8f02f-b4d5-42cd-af19-1cd55fe6e383, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-73e8f02f-b4d5-42cd-af19-1cd55fe6e383' under resource group 'kubetest-5p3qtork' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I1022 06:56:15.387474       1 controllerserver.go:271] begin to create azure disk(pvc-73e8f02f-b4d5-42cd-af19-1cd55fe6e383) account type(StandardSSD_LRS) rg(kubetest-5p3qtork) location() size(5) diskZone() maxShares(0)
I1022 06:56:15.387502       1 azure_managedDiskController.go:88] azureDisk - creating new managed Name:pvc-73e8f02f-b4d5-42cd-af19-1cd55fe6e383 StorageAccountType:StandardSSD_LRS Size:5
I1022 06:56:16.000371       1 utils.go:95] GRPC call: /csi.v1.Controller/ControllerUnpublishVolume
I1022 06:56:16.000398       1 utils.go:96] GRPC request: {"node_id":"k8s-agentpool1-22475134-vmss000001","volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-ffb53fd3-51c8-4820-ab0f-048c95ab3652"}
I1022 06:56:16.000492       1 controllerserver.go:524] Trying to detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-ffb53fd3-51c8-4820-ab0f-048c95ab3652 from node k8s-agentpool1-22475134-vmss000001
I1022 06:56:16.000508       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000001" is managed by VMSS since DisableAvailabilitySetNodes is set to true
... skipping 21 lines ...
I1022 06:56:31.392610       1 controllerserver.go:524] Trying to detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-ffb53fd3-51c8-4820-ab0f-048c95ab3652 from node k8s-agentpool1-22475134-vmss000001
I1022 06:56:31.392626       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000001" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:56:31.448144       1 azure_controller_common.go:242] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-73e8f02f-b4d5-42cd-af19-1cd55fe6e383" lun 0 to node "k8s-agentpool1-22475134-vmss000001", diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-73e8f02f-b4d5-42cd-af19-1cd55fe6e383:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-73e8f02f-b4d5-42cd-af19-1cd55fe6e383  true false 0})]
I1022 06:56:31.448218       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000001" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:56:31.448253       1 azure_controller_vmss.go:122] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-73e8f02f-b4d5-42cd-af19-1cd55fe6e383:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-73e8f02f-b4d5-42cd-af19-1cd55fe6e383  true false 0})])
I1022 06:56:31.448297       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000001" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:56:31.642096       1 azure_controller_vmss.go:134] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-73e8f02f-b4d5-42cd-af19-1cd55fe6e383:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-73e8f02f-b4d5-42cd-af19-1cd55fe6e383  true false 0})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING)
I1022 06:56:31.642328       1 azure_controller_common.go:346] Trying to detach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-ffb53fd3-51c8-4820-ab0f-048c95ab3652" from node "k8s-agentpool1-22475134-vmss000001", diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-ffb53fd3-51c8-4820-ab0f-048c95ab3652:pvc-ffb53fd3-51c8-4820-ab0f-048c95ab3652]
I1022 06:56:31.642414       1 azure_vmss.go:198] Couldn't find VMSS VM with nodeName k8s-agentpool1-22475134-vmss000001, refreshing the cache(vmss: k8s-agentpool1-22475134-vmss, rg: kubetest-5p3qtork)
E1022 06:56:31.760598       1 azure_controller_vmss.go:183] detach azure disk on node(k8s-agentpool1-22475134-vmss000001): disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-ffb53fd3-51c8-4820-ab0f-048c95ab3652:pvc-ffb53fd3-51c8-4820-ab0f-048c95ab3652]) not found
I1022 06:56:31.760627       1 azure_controller_vmss.go:213] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000001) - detach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-ffb53fd3-51c8-4820-ab0f-048c95ab3652:pvc-ffb53fd3-51c8-4820-ab0f-048c95ab3652])
I1022 06:56:41.738054       1 controllerserver.go:465] Attach operation successful: volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-73e8f02f-b4d5-42cd-af19-1cd55fe6e383" attached to node "k8s-agentpool1-22475134-vmss000001".
I1022 06:56:41.738083       1 controllerserver.go:485] attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-73e8f02f-b4d5-42cd-af19-1cd55fe6e383" to node "k8s-agentpool1-22475134-vmss000001" successfully
... skipping 22 lines ...
I1022 06:57:18.174655       1 azure_controller_common.go:242] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5" lun 32 to node "k8s-agentpool1-22475134-vmss000000", diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5  true false 32})]
I1022 06:57:18.174711       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:57:18.174740       1 azure_controller_vmss.go:122] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5  true false 32})])
I1022 06:57:18.175609       1 controllerserver.go:440] GetDiskLun returned: <nil>. Initiating attaching volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739" to node "k8s-agentpool1-22475134-vmss000000".
I1022 06:57:18.175638       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:57:18.175707       1 controllerserver.go:461] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739" to node "k8s-agentpool1-22475134-vmss000000"
I1022 06:57:18.252037       1 azure_armclient.go:341] Received error in sendAsync.send: resourceID: https://management.azure.com/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool1-22475134-vmss/virtualMachines/0?api-version=2020-12-01, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 409, RawError: {

  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}
I1022 06:57:18.252076       1 azure_armclient.go:594] Received error in put.send: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool1-22475134-vmss/virtualMachines/0, error: %!s(<nil>)
E1022 06:57:18.252128       1 azure_controller_vmss.go:125] azureDisk - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5  true false 32})]) on rg(kubetest-5p3qtork) vm(k8s-agentpool1-22475134-vmss000000) failed, err: &{false 409 0001-01-01 00:00:00 +0000 UTC {

  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}}
I1022 06:57:18.252151       1 azure_controller_vmss.go:134] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5  true false 32})], &{%!s(bool=false) %!s(int=409) 0001-01-01 00:00:00 +0000 UTC {
  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}}) returned with %!v(MISSING)
E1022 06:57:18.252192       1 controllerserver.go:481] Attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5" to instance "k8s-agentpool1-22475134-vmss000000" failed with Retriable: false, RetryAfter: 0s, HTTPStatusCode: 409, RawError: {

  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}
E1022 06:57:18.252221       1 utils.go:100] GRPC error: Attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5" to instance "k8s-agentpool1-22475134-vmss000000" failed with Retriable: false, RetryAfter: 0s, HTTPStatusCode: 409, RawError: {

  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}
I1022 06:57:18.252280       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:57:18.252302       1 azure_vmss.go:198] Couldn't find VMSS VM with nodeName k8s-agentpool1-22475134-vmss000000, refreshing the cache(vmss: k8s-agentpool1-22475134-vmss, rg: kubetest-5p3qtork)
I1022 06:57:18.323040       1 azure_controller_common.go:242] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739" lun 32 to node "k8s-agentpool1-22475134-vmss000000", diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739  true false 32})]
I1022 06:57:18.323074       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:57:18.323139       1 azure_controller_vmss.go:122] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739  true false 32})])
I1022 06:57:18.389040       1 azure_armclient.go:341] Received error in sendAsync.send: resourceID: https://management.azure.com/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool1-22475134-vmss/virtualMachines/0?api-version=2020-12-01, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 409, RawError: {

  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}
I1022 06:57:18.389063       1 azure_armclient.go:594] Received error in put.send: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool1-22475134-vmss/virtualMachines/0, error: %!s(<nil>)
E1022 06:57:18.389116       1 azure_controller_vmss.go:125] azureDisk - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739  true false 32})]) on rg(kubetest-5p3qtork) vm(k8s-agentpool1-22475134-vmss000000) failed, err: &{false 409 0001-01-01 00:00:00 +0000 UTC {

  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}}
I1022 06:57:18.389137       1 azure_controller_vmss.go:134] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739  true false 32})], &{%!s(bool=false) %!s(int=409) 0001-01-01 00:00:00 +0000 UTC {
  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}}) returned with %!v(MISSING)
E1022 06:57:18.389182       1 controllerserver.go:481] Attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739" to instance "k8s-agentpool1-22475134-vmss000000" failed with Retriable: false, RetryAfter: 0s, HTTPStatusCode: 409, RawError: {

  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}
E1022 06:57:18.389233       1 utils.go:100] GRPC error: Attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739" to instance "k8s-agentpool1-22475134-vmss000000" failed with Retriable: false, RetryAfter: 0s, HTTPStatusCode: 409, RawError: {

  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}
I1022 06:57:22.216205       1 azure_vmss.go:319] Couldn't find VMSS VM with scaleSetName "k8s-agentpool1-22475134-vmss" and instanceID "0", refreshing the cache
... skipping 10 lines ...
I1022 06:57:34.210456       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:57:34.210508       1 controllerserver.go:461] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5" to node "k8s-agentpool1-22475134-vmss000000"
I1022 06:57:34.210541       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:57:34.210554       1 azure_controller_common.go:242] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5" lun 32 to node "k8s-agentpool1-22475134-vmss000000", diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5  true false 32})]
I1022 06:57:34.210578       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:57:34.210674       1 azure_controller_vmss.go:122] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5  true false 32})])
I1022 06:57:34.287326       1 azure_armclient.go:341] Received error in sendAsync.send: resourceID: https://management.azure.com/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool1-22475134-vmss/virtualMachines/0?api-version=2020-12-01, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 409, RawError: {

  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}
I1022 06:57:34.287345       1 azure_armclient.go:594] Received error in put.send: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool1-22475134-vmss/virtualMachines/0, error: %!s(<nil>)
E1022 06:57:34.287392       1 azure_controller_vmss.go:125] azureDisk - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5  true false 32})]) on rg(kubetest-5p3qtork) vm(k8s-agentpool1-22475134-vmss000000) failed, err: &{false 409 0001-01-01 00:00:00 +0000 UTC {

  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}}
I1022 06:57:34.287433       1 azure_controller_vmss.go:134] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5  true false 32})], &{%!s(bool=false) %!s(int=409) 0001-01-01 00:00:00 +0000 UTC {
  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}}) returned with %!v(MISSING)
E1022 06:57:34.287494       1 controllerserver.go:481] Attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5" to instance "k8s-agentpool1-22475134-vmss000000" failed with Retriable: false, RetryAfter: 0s, HTTPStatusCode: 409, RawError: {

  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}
E1022 06:57:34.287520       1 utils.go:100] GRPC error: Attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5" to instance "k8s-agentpool1-22475134-vmss000000" failed with Retriable: false, RetryAfter: 0s, HTTPStatusCode: 409, RawError: {

  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}
I1022 06:57:37.763648       1 azure_managedDiskController.go:286] azureDisk - deleted a managed disk: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-73e8f02f-b4d5-42cd-af19-1cd55fe6e383
... skipping 17 lines ...
I1022 06:59:50.428448       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:59:50.463474       1 controllerserver.go:461] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739" to node "k8s-agentpool1-22475134-vmss000000"
I1022 06:59:50.463521       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:59:50.463536       1 azure_controller_common.go:242] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739" lun 32 to node "k8s-agentpool1-22475134-vmss000000", diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739  true false 32})]
I1022 06:59:50.463566       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 06:59:50.463610       1 azure_controller_vmss.go:122] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739  true false 32})])
I1022 06:59:50.577107       1 azure_armclient.go:341] Received error in sendAsync.send: resourceID: https://management.azure.com/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool1-22475134-vmss/virtualMachines/0?api-version=2020-12-01, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 409, RawError: {

  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}
I1022 06:59:50.577156       1 azure_armclient.go:594] Received error in put.send: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool1-22475134-vmss/virtualMachines/0, error: %!s(<nil>)
E1022 06:59:50.577240       1 azure_controller_vmss.go:125] azureDisk - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739  true false 32})]) on rg(kubetest-5p3qtork) vm(k8s-agentpool1-22475134-vmss000000) failed, err: &{false 409 0001-01-01 00:00:00 +0000 UTC {

  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}}
I1022 06:59:50.577285       1 azure_controller_vmss.go:134] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739  true false 32})], &{%!s(bool=false) %!s(int=409) 0001-01-01 00:00:00 +0000 UTC {
  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}}) returned with %!v(MISSING)
E1022 06:59:50.577336       1 controllerserver.go:481] Attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739" to instance "k8s-agentpool1-22475134-vmss000000" failed with Retriable: false, RetryAfter: 0s, HTTPStatusCode: 409, RawError: {

  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}
E1022 06:59:50.577407       1 utils.go:100] GRPC error: Attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-0ed7c98b-0d8e-4dbd-827b-e2b908252739" to instance "k8s-agentpool1-22475134-vmss000000" failed with Retriable: false, RetryAfter: 0s, HTTPStatusCode: 409, RawError: {

  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}
I1022 07:00:22.768639       1 azure_vmss.go:319] Couldn't find VMSS VM with scaleSetName "k8s-agentpool1-22475134-vmss" and instanceID "0", refreshing the cache
... skipping 3 lines ...
I1022 07:02:34.352666       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 07:02:34.352742       1 controllerserver.go:461] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5" to node "k8s-agentpool1-22475134-vmss000000"
I1022 07:02:34.352776       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 07:02:34.352810       1 azure_controller_common.go:242] Trying to attach volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/disks/pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5" lun 32 to node "k8s-agentpool1-22475134-vmss000000", diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5  true false 32})]
I1022 07:02:34.352836       1 azure_vmss_cache.go:320] Assuming node "k8s-agentpool1-22475134-vmss000000" is managed by VMSS since DisableAvailabilitySetNodes is set to true
I1022 07:02:34.352885       1 azure_controller_vmss.go:122] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5  true false 32})])
I1022 07:02:34.431475       1 azure_armclient.go:341] Received error in sendAsync.send: resourceID: https://management.azure.com/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool1-22475134-vmss/virtualMachines/0?api-version=2020-12-01, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 409, RawError: {

  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}
I1022 07:02:34.431509       1 azure_armclient.go:594] Received error in put.send: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5p3qtork/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool1-22475134-vmss/virtualMachines/0, error: %!s(<nil>)
E1022 07:02:34.431562       1 azure_controller_vmss.go:125] azureDisk - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5  true false 32})]) on rg(kubetest-5p3qtork) vm(k8s-agentpool1-22475134-vmss000000) failed, err: &{false 409 0001-01-01 00:00:00 +0000 UTC {

  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 32.",
    "target": "dataDisks"
  }
}}
I1022 07:02:34.431581       1 azure_controller_vmss.go:134] azureDisk - update(kubetest-5p3qtork): vm(k8s-agentpool1-22475134-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-5p3qtork/providers/microsoft.compute/disks/pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-917e2ae4-a0cf-421a-9e18-73362dc98cd5  true false 32})], &{%!s(bool=false) %!s(int=409) 0001-01-01 00:00:00 +0000 UTC {
  "error": {

    "code": "OperationNotAllowed",
    "message": "The maximum number of data disks allowed to be attached to a VM o