pull-azuredisk-csi-driver-external-e2e-single-az-mainv2 #1620928613039738880 Results

Error lines from build-log.txt

Raw build-log.txt
... skipping 115 lines ...

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 11345  100 11345    0     0   167k      0 --:--:-- --:--:-- --:--:--  167k
Downloading https://get.helm.sh/helm-v3.11.0-linux-amd64.tar.gz
Verifying checksum... Done.
Preparing to install helm into /usr/local/bin
helm installed into /usr/local/bin/helm
docker pull k8sprow.azurecr.io/azuredisk-csi:latest-v2-3d84ccaf7bb075abd5912e16e39b09b4d5f88f6c || make container-all push-manifest
Error response from daemon: manifest for k8sprow.azurecr.io/azuredisk-csi:latest-v2-3d84ccaf7bb075abd5912e16e39b09b4d5f88f6c not found: manifest unknown: manifest tagged by "latest-v2-3d84ccaf7bb075abd5912e16e39b09b4d5f88f6c" is not found
make[2]: Entering directory '/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver'
CGO_ENABLED=0 GOOS=windows go build -a -ldflags "-X sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.driverVersion=latest-v2-3d84ccaf7bb075abd5912e16e39b09b4d5f88f6c -X sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.gitCommit=3d84ccaf7bb075abd5912e16e39b09b4d5f88f6c -X sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.buildDate=2023-02-01T23:44:14Z -extldflags "-static"" -tags azurediskv2 -mod vendor -o _output/amd64/azurediskpluginv2.exe ./pkg/azurediskplugin
docker buildx rm container-builder || true
ERROR: no builder "container-builder" found
docker buildx create --use --name=container-builder
container-builder
# enable qemu for arm64 build
# https://github.com/docker/buildx/issues/464#issuecomment-741507760
docker run --privileged --rm tonistiigi/binfmt --uninstall qemu-aarch64
Unable to find image 'tonistiigi/binfmt:latest' locally
... skipping 243 lines ...

#9 exporting to image
#9 pushing layers 1.6s done
#9 pushing manifest for k8sprow.azurecr.io/azuredisk-csi:latest-v2-3d84ccaf7bb075abd5912e16e39b09b4d5f88f6c-linux-amd64@sha256:7eb090a7a6f1be03ed4aaaf44a4f3188b32180d44fcb34e753c9dcfd5782431e
#9 pushing manifest for k8sprow.azurecr.io/azuredisk-csi:latest-v2-3d84ccaf7bb075abd5912e16e39b09b4d5f88f6c-linux-amd64@sha256:7eb090a7a6f1be03ed4aaaf44a4f3188b32180d44fcb34e753c9dcfd5782431e 0.4s done
#9 DONE 5.9s
WARNING: failed to get git remote url: fatal: No remote configured to list refs from.
make[3]: Leaving directory '/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver'
make[3]: Entering directory '/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver'
CGO_ENABLED=0 GOOS=linux GOARCH=arm64 go build -a -ldflags "-X sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.driverVersion=latest-v2-3d84ccaf7bb075abd5912e16e39b09b4d5f88f6c -X sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.gitCommit=3d84ccaf7bb075abd5912e16e39b09b4d5f88f6c -X sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.buildDate=2023-02-01T23:47:28Z -extldflags "-static"" -tags azurediskv2 -mod vendor -o _output/arm64/azurediskpluginv2 ./pkg/azurediskplugin
make[3]: Leaving directory '/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver'
make[3]: Entering directory '/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver'
docker buildx build . \
... skipping 117 lines ...
#8 exporting config sha256:aa8fe61e591ecdbf3a2125ffacf6afebf06d61032cfb1c759aa6d49632b3fc9c 0.0s done
#8 pushing layers
#8 pushing layers 1.4s done
#8 pushing manifest for k8sprow.azurecr.io/azuredisk-csi:latest-v2-3d84ccaf7bb075abd5912e16e39b09b4d5f88f6c-linux-arm64@sha256:b13014fb4a794662c94f796b6ff0f48c66e77eeba763591e8de1f87880bd11c0
#8 pushing manifest for k8sprow.azurecr.io/azuredisk-csi:latest-v2-3d84ccaf7bb075abd5912e16e39b09b4d5f88f6c-linux-arm64@sha256:b13014fb4a794662c94f796b6ff0f48c66e77eeba763591e8de1f87880bd11c0 0.4s done
#8 DONE 5.7s
WARNING: failed to get git remote url: fatal: No remote configured to list refs from.
make[3]: Leaving directory '/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver'
for osversion in 1809 20H2 ltsc2022; do \
	OSVERSION=${osversion} make container-windows; \
done
make[3]: Entering directory '/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver'
docker buildx build . \
... skipping 79 lines ...
#10 exporting config sha256:e9b76590dd9fd7bd65a7f72974e645b6144a9f5c95d444f6d4757559c854dc48 0.0s done
#10 pushing layers
#10 pushing layers 1.3s done
#10 pushing manifest for k8sprow.azurecr.io/azuredisk-csi:latest-v2-3d84ccaf7bb075abd5912e16e39b09b4d5f88f6c-windows-1809-amd64@sha256:be7ed22fcb2e48995d5f1aca87681106477a8201ca039fc8767da4dcb5a41e41
#10 pushing manifest for k8sprow.azurecr.io/azuredisk-csi:latest-v2-3d84ccaf7bb075abd5912e16e39b09b4d5f88f6c-windows-1809-amd64@sha256:be7ed22fcb2e48995d5f1aca87681106477a8201ca039fc8767da4dcb5a41e41 0.3s done
#10 DONE 5.5s
WARNING: failed to get git remote url: fatal: No remote configured to list refs from.
make[3]: Leaving directory '/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver'
make[3]: Entering directory '/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver'
docker buildx build . \
	--pull \
	--output=type=registry \
	--platform="windows/amd64" \
... skipping 66 lines ...
#10 exporting config sha256:b0c8cea03a84fe0a55e296cb8743cf67b4ec02f4c39a977296950197b49a6677 0.0s done
#10 pushing layers
#10 pushing layers 1.2s done
#10 pushing manifest for k8sprow.azurecr.io/azuredisk-csi:latest-v2-3d84ccaf7bb075abd5912e16e39b09b4d5f88f6c-windows-20H2-amd64@sha256:8964b54d3305d64abed7dd5cebad30af893894aacbe60f7deff582959ff5ecef
#10 pushing manifest for k8sprow.azurecr.io/azuredisk-csi:latest-v2-3d84ccaf7bb075abd5912e16e39b09b4d5f88f6c-windows-20H2-amd64@sha256:8964b54d3305d64abed7dd5cebad30af893894aacbe60f7deff582959ff5ecef 0.5s done
#10 DONE 5.5s
WARNING: failed to get git remote url: fatal: No remote configured to list refs from.
make[3]: Leaving directory '/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver'
make[3]: Entering directory '/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver'
docker buildx build . \
	--pull \
	--output=type=registry \
	--platform="windows/amd64" \
... skipping 72 lines ...
#10 exporting config sha256:983eea807d9e5a39a4acac767fd68e503a51d6ef6e00b4bd2d0af81e1c2bcec6 0.0s done
#10 pushing layers
#10 pushing layers 1.3s done
#10 pushing manifest for k8sprow.azurecr.io/azuredisk-csi:latest-v2-3d84ccaf7bb075abd5912e16e39b09b4d5f88f6c-windows-ltsc2022-amd64@sha256:dd312561113284e80a2833cb86b5ee10e3c8d9aa47208fdba0554b2f4e70ea26
#10 pushing manifest for k8sprow.azurecr.io/azuredisk-csi:latest-v2-3d84ccaf7bb075abd5912e16e39b09b4d5f88f6c-windows-ltsc2022-amd64@sha256:dd312561113284e80a2833cb86b5ee10e3c8d9aa47208fdba0554b2f4e70ea26 0.3s done
#10 DONE 5.4s
WARNING: failed to get git remote url: fatal: No remote configured to list refs from.
make[3]: Leaving directory '/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver'
docker manifest create --amend k8sprow.azurecr.io/azuredisk-csi:latest-v2-3d84ccaf7bb075abd5912e16e39b09b4d5f88f6c  k8sprow.azurecr.io/azuredisk-csi:latest-v2-3d84ccaf7bb075abd5912e16e39b09b4d5f88f6c-linux-amd64  k8sprow.azurecr.io/azuredisk-csi:latest-v2-3d84ccaf7bb075abd5912e16e39b09b4d5f88f6c-linux-arm64  k8sprow.azurecr.io/azuredisk-csi:latest-v2-3d84ccaf7bb075abd5912e16e39b09b4d5f88f6c-windows-1809-amd64  k8sprow.azurecr.io/azuredisk-csi:latest-v2-3d84ccaf7bb075abd5912e16e39b09b4d5f88f6c-windows-20H2-amd64  k8sprow.azurecr.io/azuredisk-csi:latest-v2-3d84ccaf7bb075abd5912e16e39b09b4d5f88f6c-windows-ltsc2022-amd64
Created manifest list k8sprow.azurecr.io/azuredisk-csi:latest-v2-3d84ccaf7bb075abd5912e16e39b09b4d5f88f6c
# add "os.version" field to windows images (based on https://github.com/kubernetes/kubernetes/blob/master/build/pause/Makefile)
set -x; \
for arch in amd64; do \
... skipping 74 lines ...
         }
      }
   ]
}
make[2]: Leaving directory '/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver'
docker pull k8sprow.azurecr.io/azdiskschedulerextender-csi:latest-v2-3d84ccaf7bb075abd5912e16e39b09b4d5f88f6c || make azdiskschedulerextender-all push-manifest-azdiskschedulerextender
Error response from daemon: manifest for k8sprow.azurecr.io/azdiskschedulerextender-csi:latest-v2-3d84ccaf7bb075abd5912e16e39b09b4d5f88f6c not found: manifest unknown: manifest tagged by "latest-v2-3d84ccaf7bb075abd5912e16e39b09b4d5f88f6c" is not found
make[2]: Entering directory '/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver'
docker buildx rm container-builder || true
container-builder removed
docker buildx create --use --name=container-builder
container-builder
# enable qemu for arm64 build
... skipping 158 lines ...

#8 exporting to image
#8 pushing layers 1.4s done
#8 pushing manifest for k8sprow.azurecr.io/azdiskschedulerextender-csi:latest-v2-3d84ccaf7bb075abd5912e16e39b09b4d5f88f6c-linux-amd64@sha256:4bff4caaaae5961461d622ff050b643c854002e33a178104434417efe655029e
#8 pushing manifest for k8sprow.azurecr.io/azdiskschedulerextender-csi:latest-v2-3d84ccaf7bb075abd5912e16e39b09b4d5f88f6c-linux-amd64@sha256:4bff4caaaae5961461d622ff050b643c854002e33a178104434417efe655029e 0.3s done
#8 DONE 4.5s
WARNING: failed to get git remote url: fatal: No remote configured to list refs from.
make[3]: Leaving directory '/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver'
make[3]: Entering directory '/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver'
CGO_ENABLED=0 GOOS=linux GOARCH=arm64 go build -a -ldflags "-X sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.schedulerVersion=latest-v2-3d84ccaf7bb075abd5912e16e39b09b4d5f88f6c -X sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.gitCommit=3d84ccaf7bb075abd5912e16e39b09b4d5f88f6c -X sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.buildDate=2023-02-01T23:51:24Z -X sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.DriverName= -extldflags "-static"" -tags azurediskv2 -mod vendor -o _output/arm64/azdiskschedulerextender ./pkg/azdiskschedulerextender
make[3]: Leaving directory '/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver'
make[3]: Entering directory '/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver'
docker buildx build . \
... skipping 38 lines ...
#7 exporting config sha256:96e30f53afa89092b6255b8c2893a050cec695ded7e6c121bbc8ccddc617e9b4 0.0s done
#7 pushing layers
#7 pushing layers 1.3s done
#7 pushing manifest for k8sprow.azurecr.io/azdiskschedulerextender-csi:latest-v2-3d84ccaf7bb075abd5912e16e39b09b4d5f88f6c-linux-arm64@sha256:9451e2f13d47015772a0d939db61f267e56c6321d3b620968eabf5cc9fdd539d
#7 pushing manifest for k8sprow.azurecr.io/azdiskschedulerextender-csi:latest-v2-3d84ccaf7bb075abd5912e16e39b09b4d5f88f6c-linux-arm64@sha256:9451e2f13d47015772a0d939db61f267e56c6321d3b620968eabf5cc9fdd539d 0.4s done
#7 DONE 4.1s
WARNING: failed to get git remote url: fatal: No remote configured to list refs from.
make[3]: Leaving directory '/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver'
docker manifest create --amend k8sprow.azurecr.io/azdiskschedulerextender-csi:latest-v2-3d84ccaf7bb075abd5912e16e39b09b4d5f88f6c  k8sprow.azurecr.io/azdiskschedulerextender-csi:latest-v2-3d84ccaf7bb075abd5912e16e39b09b4d5f88f6c-linux-amd64  k8sprow.azurecr.io/azdiskschedulerextender-csi:latest-v2-3d84ccaf7bb075abd5912e16e39b09b4d5f88f6c-linux-arm64
Created manifest list k8sprow.azurecr.io/azdiskschedulerextender-csi:latest-v2-3d84ccaf7bb075abd5912e16e39b09b4d5f88f6c
docker manifest push --purge k8sprow.azurecr.io/azdiskschedulerextender-csi:latest-v2-3d84ccaf7bb075abd5912e16e39b09b4d5f88f6c
sha256:ee65352a662bec74a07cdbd5196a2635780b4b72e7a3fec9dedce9f36e607540
docker manifest inspect k8sprow.azurecr.io/azdiskschedulerextender-csi:latest-v2-3d84ccaf7bb075abd5912e16e39b09b4d5f88f6c
... skipping 825 lines ...
                    type: string
                type: object
                oneOf:
                - required: ["persistentVolumeClaimName"]
                - required: ["volumeSnapshotContentName"]
              volumeSnapshotClassName:
                description: 'VolumeSnapshotClassName is the name of the VolumeSnapshotClass requested by the VolumeSnapshot. VolumeSnapshotClassName may be left nil to indicate that the default SnapshotClass should be used. A given cluster may have multiple default Volume SnapshotClasses: one default per CSI Driver. If a VolumeSnapshot does not specify a SnapshotClass, VolumeSnapshotSource will be checked to figure out what the associated CSI Driver is, and the default VolumeSnapshotClass associated with that CSI Driver will be used. If more than one VolumeSnapshotClass exist for a given CSI Driver and more than one have been marked as default, CreateSnapshot will fail and generate an event. Empty string is not allowed for this field.'
                type: string
            required:
            - source
            type: object
          status:
            description: status represents the current information of a snapshot. Consumers must verify binding between VolumeSnapshot and VolumeSnapshotContent objects is successful (by validating that both VolumeSnapshot and VolumeSnapshotContent point at each other) before using this object.
... skipping 2 lines ...
                description: 'boundVolumeSnapshotContentName is the name of the VolumeSnapshotContent object to which this VolumeSnapshot object intends to bind to. If not specified, it indicates that the VolumeSnapshot object has not been successfully bound to a VolumeSnapshotContent object yet. NOTE: To avoid possible security issues, consumers must verify binding between VolumeSnapshot and VolumeSnapshotContent objects is successful (by validating that both VolumeSnapshot and VolumeSnapshotContent point at each other) before using this object.'
                type: string
              creationTime:
                description: creationTime is the timestamp when the point-in-time snapshot is taken by the underlying storage system. In dynamic snapshot creation case, this field will be filled in by the snapshot controller with the "creation_time" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "creation_time" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. If not specified, it may indicate that the creation time of the snapshot is unknown.
                format: date-time
                type: string
              error:
                description: error is the last observed error during snapshot creation, if any. This field could be helpful to upper level controllers(i.e., application controller) to decide whether they should continue on waiting for the snapshot to be created based on the type of error reported. The snapshot controller will keep retrying when an error occurs during the snapshot creation. Upon success, this error field will be cleared.
                properties:
                  message:
                    description: 'message is a string detailing the encountered error during snapshot creation if specified. NOTE: message may be logged, and it should not contain sensitive information.'
                    type: string
                  time:
                    description: time is the timestamp when the error was encountered.
                    format: date-time
                    type: string
                type: object
              readyToUse:
                description: readyToUse indicates if the snapshot is ready to be used to restore a volume. In dynamic snapshot creation case, this field will be filled in by the snapshot controller with the "ready_to_use" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "ready_to_use" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it, otherwise, this field will be set to "True". If not specified, it means the readiness of a snapshot is unknown.
                type: boolean
              restoreSize:
                type: string
                description: restoreSize represents the minimum size of volume required to create a volume from this snapshot. In dynamic snapshot creation case, this field will be filled in by the snapshot controller with the "size_bytes" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "size_bytes" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. When restoring a volume from this snapshot, the size of the volume MUST NOT be smaller than the restoreSize if it is specified, otherwise the restoration will fail. If not specified, it indicates that the size is unknown.
                pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
                x-kubernetes-int-or-string: true
            type: object
        required:
        - spec
        type: object
... skipping 60 lines ...
                    type: string
                  volumeSnapshotContentName:
                    description: volumeSnapshotContentName specifies the name of a pre-existing VolumeSnapshotContent object representing an existing volume snapshot. This field should be set if the snapshot already exists and only needs a representation in Kubernetes. This field is immutable.
                    type: string
                type: object
              volumeSnapshotClassName:
                description: 'VolumeSnapshotClassName is the name of the VolumeSnapshotClass requested by the VolumeSnapshot. VolumeSnapshotClassName may be left nil to indicate that the default SnapshotClass should be used. A given cluster may have multiple default Volume SnapshotClasses: one default per CSI Driver. If a VolumeSnapshot does not specify a SnapshotClass, VolumeSnapshotSource will be checked to figure out what the associated CSI Driver is, and the default VolumeSnapshotClass associated with that CSI Driver will be used. If more than one VolumeSnapshotClass exist for a given CSI Driver and more than one have been marked as default, CreateSnapshot will fail and generate an event. Empty string is not allowed for this field.'
                type: string
            required:
            - source
            type: object
          status:
            description: status represents the current information of a snapshot. Consumers must verify binding between VolumeSnapshot and VolumeSnapshotContent objects is successful (by validating that both VolumeSnapshot and VolumeSnapshotContent point at each other) before using this object.
... skipping 2 lines ...
                description: 'boundVolumeSnapshotContentName is the name of the VolumeSnapshotContent object to which this VolumeSnapshot object intends to bind to. If not specified, it indicates that the VolumeSnapshot object has not been successfully bound to a VolumeSnapshotContent object yet. NOTE: To avoid possible security issues, consumers must verify binding between VolumeSnapshot and VolumeSnapshotContent objects is successful (by validating that both VolumeSnapshot and VolumeSnapshotContent point at each other) before using this object.'
                type: string
              creationTime:
                description: creationTime is the timestamp when the point-in-time snapshot is taken by the underlying storage system. In dynamic snapshot creation case, this field will be filled in by the snapshot controller with the "creation_time" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "creation_time" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. If not specified, it may indicate that the creation time of the snapshot is unknown.
                format: date-time
                type: string
              error:
                description: error is the last observed error during snapshot creation, if any. This field could be helpful to upper level controllers(i.e., application controller) to decide whether they should continue on waiting for the snapshot to be created based on the type of error reported. The snapshot controller will keep retrying when an error occurs during the snapshot creation. Upon success, this error field will be cleared.
                properties:
                  message:
                    description: 'message is a string detailing the encountered error during snapshot creation if specified. NOTE: message may be logged, and it should not contain sensitive information.'
                    type: string
                  time:
                    description: time is the timestamp when the error was encountered.
                    format: date-time
                    type: string
                type: object
              readyToUse:
                description: readyToUse indicates if the snapshot is ready to be used to restore a volume. In dynamic snapshot creation case, this field will be filled in by the snapshot controller with the "ready_to_use" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "ready_to_use" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it, otherwise, this field will be set to "True". If not specified, it means the readiness of a snapshot is unknown.
                type: boolean
              restoreSize:
                type: string
                description: restoreSize represents the minimum size of volume required to create a volume from this snapshot. In dynamic snapshot creation case, this field will be filled in by the snapshot controller with the "size_bytes" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "size_bytes" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. When restoring a volume from this snapshot, the size of the volume MUST NOT be smaller than the restoreSize if it is specified, otherwise the restoration will fail. If not specified, it indicates that the size is unknown.
                pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
                x-kubernetes-int-or-string: true
            type: object
        required:
        - spec
        type: object
... skipping 254 lines ...
            description: status represents the current information of a snapshot.
            properties:
              creationTime:
                description: creationTime is the timestamp when the point-in-time snapshot is taken by the underlying storage system. In dynamic snapshot creation case, this field will be filled in by the CSI snapshotter sidecar with the "creation_time" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "creation_time" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. If not specified, it indicates the creation time is unknown. The format of this field is a Unix nanoseconds time encoded as an int64. On Unix, the command `date +%s%N` returns the current time in nanoseconds since 1970-01-01 00:00:00 UTC.
                format: int64
                type: integer
              error:
                description: error is the last observed error during snapshot creation, if any. Upon success after retry, this error field will be cleared.
                properties:
                  message:
                    description: 'message is a string detailing the encountered error during snapshot creation if specified. NOTE: message may be logged, and it should not contain sensitive information.'
                    type: string
                  time:
                    description: time is the timestamp when the error was encountered.
                    format: date-time
                    type: string
                type: object
              readyToUse:
                description: readyToUse indicates if a snapshot is ready to be used to restore a volume. In dynamic snapshot creation case, this field will be filled in by the CSI snapshotter sidecar with the "ready_to_use" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "ready_to_use" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it, otherwise, this field will be set to "True". If not specified, it means the readiness of a snapshot is unknown.
                type: boolean
              restoreSize:
                description: restoreSize represents the complete size of the snapshot in bytes. In dynamic snapshot creation case, this field will be filled in by the CSI snapshotter sidecar with the "size_bytes" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "size_bytes" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. When restoring a volume from this snapshot, the size of the volume MUST NOT be smaller than the restoreSize if it is specified, otherwise the restoration will fail. If not specified, it indicates that the size is unknown.
                format: int64
                minimum: 0
                type: integer
              snapshotHandle:
                description: snapshotHandle is the CSI "snapshot_id" of a snapshot on the underlying storage system. If not specified, it indicates that dynamic snapshot creation has either failed or it is still in progress.
                type: string
            type: object
        required:
        - spec
        type: object
    served: true
... skipping 108 lines ...
            description: status represents the current information of a snapshot.
            properties:
              creationTime:
                description: creationTime is the timestamp when the point-in-time snapshot is taken by the underlying storage system. In dynamic snapshot creation case, this field will be filled in by the CSI snapshotter sidecar with the "creation_time" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "creation_time" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. If not specified, it indicates the creation time is unknown. The format of this field is a Unix nanoseconds time encoded as an int64. On Unix, the command `date +%s%N` returns the current time in nanoseconds since 1970-01-01 00:00:00 UTC.
                format: int64
                type: integer
              error:
                description: error is the last observed error during snapshot creation, if any. Upon success after retry, this error field will be cleared.
                properties:
                  message:
                    description: 'message is a string detailing the encountered error during snapshot creation if specified. NOTE: message may be logged, and it should not contain sensitive information.'
                    type: string
                  time:
                    description: time is the timestamp when the error was encountered.
                    format: date-time
                    type: string
                type: object
              readyToUse:
                description: readyToUse indicates if a snapshot is ready to be used to restore a volume. In dynamic snapshot creation case, this field will be filled in by the CSI snapshotter sidecar with the "ready_to_use" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "ready_to_use" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it, otherwise, this field will be set to "True". If not specified, it means the readiness of a snapshot is unknown.
                type: boolean
              restoreSize:
                description: restoreSize represents the complete size of the snapshot in bytes. In dynamic snapshot creation case, this field will be filled in by the CSI snapshotter sidecar with the "size_bytes" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "size_bytes" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. When restoring a volume from this snapshot, the size of the volume MUST NOT be smaller than the restoreSize if it is specified, otherwise the restoration will fail. If not specified, it indicates that the size is unknown.
                format: int64
                minimum: 0
                type: integer
              snapshotHandle:
                description: snapshotHandle is the CSI "snapshot_id" of a snapshot on the underlying storage system. If not specified, it indicates that dynamic snapshot creation has either failed or it is still in progress.
                type: string
            type: object
        required:
        - spec
        type: object
    served: true
... skipping 359 lines ...
            - volumeName
            - volume_context
            - volume_id
            type: object
          status:
            description: status represents the current state of AzVolumeAttachment.
              includes error, state, and attachment status Required
            properties:
              detail:
                description: Status summarizes the current attachment state of the
                  volume attachment Nil Status indicates that the volume has not yet
                  been attached to the node
                properties:
... skipping 7 lines ...
                  role:
                    description: The current attachment role.
                    type: string
                required:
                - role
                type: object
              error:
                description: Error occurred during attach/detach of volume
                properties:
                  code:
                    type: string
                  message:
                    type: string
                  parameters:
... skipping 90 lines ...
            - volumeName
            - volume_context
            - volume_id
            type: object
          status:
            description: status represents the current state of AzVolumeAttachment.
              includes error, state, and attachment status
            properties:
              annotation:
                additionalProperties:
                  type: string
                description: Annotations contains additional resource information
                  to guide driver actions
... skipping 13 lines ...
                  role:
                    description: The current attachment role.
                    type: string
                required:
                - role
                type: object
              error:
                description: Error occurred during attach/detach of volume
                properties:
                  code:
                    type: string
                  message:
                    type: string
                  parameters:
... skipping 169 lines ...
            - maxMountReplicaCount
            - volumeCapability
            - volumeName
            type: object
          status:
            description: status represents the current state of AzVolume. includes
              error, state, and volume status
            properties:
              detail:
                description: Current status detail of the AzVolume Nil detail indicates
                  that the volume has not been created
                properties:
                  accessible_topology:
... skipping 28 lines ...
                    type: string
                required:
                - capacity_bytes
                - node_expansion_required
                - volume_id
                type: object
              error:
                description: Error occurred during creation/deletion of volume
                properties:
                  code:
                    type: string
                  message:
                    type: string
                  parameters:
... skipping 154 lines ...
            - maxMountReplicaCount
            - volumeCapability
            - volumeName
            type: object
          status:
            description: status represents the current state of AzVolume. includes
              error, state, and volume status
            properties:
              annotation:
                additionalProperties:
                  type: string
                description: Annotations contains additional resource information
                  to guide driver actions
... skipping 34 lines ...
                    type: string
                required:
                - capacity_bytes
                - node_expansion_required
                - volume_id
                type: object
              error:
                description: Error occurred during creation/deletion of volume
                properties:
                  code:
                    type: string
                  message:
                    type: string
                  parameters:
... skipping 1069 lines ...
          image: "mcr.microsoft.com/oss/kubernetes-csi/csi-resizer:v1.5.0"
          args:
            - "-csi-address=$(ADDRESS)"
            - "-v=2"
            - "-leader-election"
            - "--leader-election-namespace=kube-system"
            - '-handle-volume-inuse-error=false'
            - '-feature-gates=RecoverVolumeExpansionFailure=true'
            - "-timeout=240s"
          env:
            - name: ADDRESS
              value: /csi/csi.sock
          volumeMounts:
... skipping 290 lines ...
      	If true, adds the file directory to the header of the log messages (DEPRECATED: will be removed in a future release, see https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/2845-deprecate-klog-specific-flags-in-k8s-components)
    -allow_dynamic_housekeeping
      	Whether to allow the housekeeping interval to be dynamic (default true)
    -allowed-not-ready-nodes int
      	If greater than zero, framework will allow for that many non-ready nodes when checking for all ready nodes. If -1, no waiting will be performed for ready nodes or daemonset pods.
    -alsologtostderr
      	log to standard error as well as files (DEPRECATED: will be removed in a future release, see https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/2845-deprecate-klog-specific-flags-in-k8s-components)
    -application_metrics_count_limit int
      	Max number of application metrics to store (per container) (default 100)
    -boot_id_file string
      	Comma-separated list of files to check for boot-id. Use the first one that exists. (default "/proc/sys/kernel/random/boot_id")
    -cert-dir string
      	Path to the directory containing the certs. Default is empty, which doesn't use certs.
    -clean-start
      	If true, purge all namespaces except default and system before running tests. This serves to Cleanup test namespaces from failed/interrupted e2e runs in a long-lived cluster.
    -cloud-config-file string
      	Cloud config file.  Only required if provider is azure or vsphere.
    -cloud-provider-gce-l7lb-src-cidrs value
      	CIDRs opened in GCE firewall for L7 LB traffic proxy & health checks (default 130.211.0.0/22,35.191.0.0/16)
    -cloud-provider-gce-lb-src-cidrs value
      	CIDRs opened in GCE firewall for L4 LB traffic proxy & health checks (default 130.211.0.0/22,209.85.152.0/22,209.85.204.0/22,35.191.0.0/16)
... skipping 79 lines ...
      	If set, ginkgo will emit node output to files when running in parallel.
    -ginkgo.dryRun
      	If set, ginkgo will walk the test hierarchy without actually running anything.  Best paired with -v.
    -ginkgo.failFast
      	If set, ginkgo will stop running a test suite after a failure occurs.
    -ginkgo.failOnPending
      	If set, ginkgo will mark the test suite as failed if any specs are pending.
    -ginkgo.flakeAttempts int
      	Make up to this many attempts to run each spec. Please note that if any of the attempts succeed, the suite will not be failed. But any failures will still be recorded. (default 1)
    -ginkgo.focus string
      	If set, ginkgo will only run specs that match this regular expression.
    -ginkgo.noColor
      	If set, suppress color output in default reporter.
    -ginkgo.noisyPendings
      	If set, default reporter will shout about pending tests. (default true)
... skipping 75 lines ...
      	If non-empty, use this log file (DEPRECATED: will be removed in a future release, see https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/2845-deprecate-klog-specific-flags-in-k8s-components)
    -log_file_max_size uint
      	Defines the maximum size a log file can grow to. Unit is megabytes. If the value is 0, the maximum file size is unlimited. (DEPRECATED: will be removed in a future release, see https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/2845-deprecate-klog-specific-flags-in-k8s-components) (default 1800)
    -logexporter-gcs-path string
      	Path to the GCS artifacts directory to dump logs from nodes. Logexporter gets enabled if this is non-empty.
    -logtostderr
      	log to standard error instead of files (DEPRECATED: will be removed in a future release, see https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/2845-deprecate-klog-specific-flags-in-k8s-components) (default true)
    -machine_id_file string
      	Comma-separated list of files to check for machine-id. Use the first one that exists. (default "/etc/machine-id,/var/lib/dbus/machine-id")
    -master-os-distro string
      	The OS distribution of cluster master (debian, ubuntu, gci, coreos, or custom). (default "debian")
    -master-tag string
      	Network tags used on master instances. Valid only for gce, gke
... skipping 169 lines ...
      	If true, adds the file directory to the header of the log messages (DEPRECATED: will be removed in a future release, see https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/2845-deprecate-klog-specific-flags-in-k8s-components)
    -allow_dynamic_housekeeping
      	Whether to allow the housekeeping interval to be dynamic (default true)
    -allowed-not-ready-nodes int
      	If greater than zero, framework will allow for that many non-ready nodes when checking for all ready nodes. If -1, no waiting will be performed for ready nodes or daemonset pods.
    -alsologtostderr
      	log to standard error as well as files (DEPRECATED: will be removed in a future release, see https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/2845-deprecate-klog-specific-flags-in-k8s-components)
    -application_metrics_count_limit int
      	Max number of application metrics to store (per container) (default 100)
    -boot_id_file string
      	Comma-separated list of files to check for boot-id. Use the first one that exists. (default "/proc/sys/kernel/random/boot_id")
    -cert-dir string
      	Path to the directory containing the certs. Default is empty, which doesn't use certs.
    -clean-start
      	If true, purge all namespaces except default and system before running tests. This serves to Cleanup test namespaces from failed/interrupted e2e runs in a long-lived cluster.
    -cloud-config-file string
      	Cloud config file.  Only required if provider is azure or vsphere.
    -cloud-provider-gce-l7lb-src-cidrs value
      	CIDRs opened in GCE firewall for L7 LB traffic proxy & health checks (default 130.211.0.0/22,35.191.0.0/16)
    -cloud-provider-gce-lb-src-cidrs value
      	CIDRs opened in GCE firewall for L4 LB traffic proxy & health checks (default 130.211.0.0/22,209.85.152.0/22,209.85.204.0/22,35.191.0.0/16)
... skipping 79 lines ...
      	If set, ginkgo will emit node output to files when running in parallel.
    -ginkgo.dryRun
      	If set, ginkgo will walk the test hierarchy without actually running anything.  Best paired with -v.
    -ginkgo.failFast
      	If set, ginkgo will stop running a test suite after a failure occurs.
    -ginkgo.failOnPending
      	If set, ginkgo will mark the test suite as failed if any specs are pending.
    -ginkgo.flakeAttempts int
      	Make up to this many attempts to run each spec. Please note that if any of the attempts succeed, the suite will not be failed. But any failures will still be recorded. (default 1)
    -ginkgo.focus string
      	If set, ginkgo will only run specs that match this regular expression.
    -ginkgo.noColor
      	If set, suppress color output in default reporter.
    -ginkgo.noisyPendings
      	If set, default reporter will shout about pending tests. (default true)
... skipping 75 lines ...
      	If non-empty, use this log file (DEPRECATED: will be removed in a future release, see https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/2845-deprecate-klog-specific-flags-in-k8s-components)
    -log_file_max_size uint
      	Defines the maximum size a log file can grow to. Unit is megabytes. If the value is 0, the maximum file size is unlimited. (DEPRECATED: will be removed in a future release, see https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/2845-deprecate-klog-specific-flags-in-k8s-components) (default 1800)
    -logexporter-gcs-path string
      	Path to the GCS artifacts directory to dump logs from nodes. Logexporter gets enabled if this is non-empty.
    -logtostderr
      	log to standard error instead of files (DEPRECATED: will be removed in a future release, see https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/2845-deprecate-klog-specific-flags-in-k8s-components) (default true)
    -machine_id_file string
      	Comma-separated list of files to check for machine-id. Use the first one that exists. (default "/etc/machine-id,/var/lib/dbus/machine-id")
    -master-os-distro string
      	The OS distribution of cluster master (debian, ubuntu, gci, coreos, or custom). (default "debian")
    -master-tag string
      	Network tags used on master instances. Valid only for gce, gke
... skipping 169 lines ...
      	If true, adds the file directory to the header of the log messages (DEPRECATED: will be removed in a future release, see https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/2845-deprecate-klog-specific-flags-in-k8s-components)
    -allow_dynamic_housekeeping
      	Whether to allow the housekeeping interval to be dynamic (default true)
    -allowed-not-ready-nodes int
      	If greater than zero, framework will allow for that many non-ready nodes when checking for all ready nodes. If -1, no waiting will be performed for ready nodes or daemonset pods.
    -alsologtostderr
      	log to standard error as well as files (DEPRECATED: will be removed in a future release, see https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/2845-deprecate-klog-specific-flags-in-k8s-components)
    -application_metrics_count_limit int
      	Max number of application metrics to store (per container) (default 100)
    -boot_id_file string
      	Comma-separated list of files to check for boot-id. Use the first one that exists. (default "/proc/sys/kernel/random/boot_id")
    -cert-dir string
      	Path to the directory containing the certs. Default is empty, which doesn't use certs.
    -clean-start
      	If true, purge all namespaces except default and system before running tests. This serves to Cleanup test namespaces from failed/interrupted e2e runs in a long-lived cluster.
    -cloud-config-file string
      	Cloud config file.  Only required if provider is azure or vsphere.
    -cloud-provider-gce-l7lb-src-cidrs value
      	CIDRs opened in GCE firewall for L7 LB traffic proxy & health checks (default 130.211.0.0/22,35.191.0.0/16)
    -cloud-provider-gce-lb-src-cidrs value
      	CIDRs opened in GCE firewall for L4 LB traffic proxy & health checks (default 130.211.0.0/22,209.85.152.0/22,209.85.204.0/22,35.191.0.0/16)
... skipping 79 lines ...
      	If set, ginkgo will emit node output to files when running in parallel.
    -ginkgo.dryRun
      	If set, ginkgo will walk the test hierarchy without actually running anything.  Best paired with -v.
    -ginkgo.failFast
      	If set, ginkgo will stop running a test suite after a failure occurs.
    -ginkgo.failOnPending
      	If set, ginkgo will mark the test suite as failed if any specs are pending.
    -ginkgo.flakeAttempts int
      	Make up to this many attempts to run each spec. Please note that if any of the attempts succeed, the suite will not be failed. But any failures will still be recorded. (default 1)
    -ginkgo.focus string
      	If set, ginkgo will only run specs that match this regular expression.
    -ginkgo.noColor
      	If set, suppress color output in default reporter.
    -ginkgo.noisyPendings
      	If set, default reporter will shout about pending tests. (default true)
... skipping 75 lines ...
      	If non-empty, use this log file (DEPRECATED: will be removed in a future release, see https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/2845-deprecate-klog-specific-flags-in-k8s-components)
    -log_file_max_size uint
      	Defines the maximum size a log file can grow to. Unit is megabytes. If the value is 0, the maximum file size is unlimited. (DEPRECATED: will be removed in a future release, see https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/2845-deprecate-klog-specific-flags-in-k8s-components) (default 1800)
    -logexporter-gcs-path string
      	Path to the GCS artifacts directory to dump logs from nodes. Logexporter gets enabled if this is non-empty.
    -logtostderr
      	log to standard error instead of files (DEPRECATED: will be removed in a future release, see https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/2845-deprecate-klog-specific-flags-in-k8s-components) (default true)
    -machine_id_file string
      	Comma-separated list of files to check for machine-id. Use the first one that exists. (default "/etc/machine-id,/var/lib/dbus/machine-id")
    -master-os-distro string
      	The OS distribution of cluster master (debian, ubuntu, gci, coreos, or custom). (default "debian")
    -master-tag string
      	Network tags used on master instances. Valid only for gce, gke
... skipping 169 lines ...
      	If true, adds the file directory to the header of the log messages (DEPRECATED: will be removed in a future release, see https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/2845-deprecate-klog-specific-flags-in-k8s-components)
    -allow_dynamic_housekeeping
      	Whether to allow the housekeeping interval to be dynamic (default true)
    -allowed-not-ready-nodes int
      	If greater than zero, framework will allow for that many non-ready nodes when checking for all ready nodes. If -1, no waiting will be performed for ready nodes or daemonset pods.
    -alsologtostderr
      	log to standard error as well as files (DEPRECATED: will be removed in a future release, see https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/2845-deprecate-klog-specific-flags-in-k8s-components)
    -application_metrics_count_limit int
      	Max number of application metrics to store (per container) (default 100)
    -boot_id_file string
      	Comma-separated list of files to check for boot-id. Use the first one that exists. (default "/proc/sys/kernel/random/boot_id")
    -cert-dir string
      	Path to the directory containing the certs. Default is empty, which doesn't use certs.
    -clean-start
      	If true, purge all namespaces except default and system before running tests. This serves to Cleanup test namespaces from failed/interrupted e2e runs in a long-lived cluster.
    -cloud-config-file string
      	Cloud config file.  Only required if provider is azure or vsphere.
    -cloud-provider-gce-l7lb-src-cidrs value
      	CIDRs opened in GCE firewall for L7 LB traffic proxy & health checks (default 130.211.0.0/22,35.191.0.0/16)
    -cloud-provider-gce-lb-src-cidrs value
      	CIDRs opened in GCE firewall for L4 LB traffic proxy & health checks (default 130.211.0.0/22,209.85.152.0/22,209.85.204.0/22,35.191.0.0/16)
... skipping 79 lines ...
      	If set, ginkgo will emit node output to files when running in parallel.
    -ginkgo.dryRun
      	If set, ginkgo will walk the test hierarchy without actually running anything.  Best paired with -v.
    -ginkgo.failFast
      	If set, ginkgo will stop running a test suite after a failure occurs.
    -ginkgo.failOnPending
      	If set, ginkgo will mark the test suite as failed if any specs are pending.
    -ginkgo.flakeAttempts int
      	Make up to this many attempts to run each spec. Please note that if any of the attempts succeed, the suite will not be failed. But any failures will still be recorded. (default 1)
    -ginkgo.focus string
      	If set, ginkgo will only run specs that match this regular expression.
    -ginkgo.noColor
      	If set, suppress color output in default reporter.
    -ginkgo.noisyPendings
      	If set, default reporter will shout about pending tests. (default true)
... skipping 75 lines ...
      	If non-empty, use this log file (DEPRECATED: will be removed in a future release, see https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/2845-deprecate-klog-specific-flags-in-k8s-components)
    -log_file_max_size uint
      	Defines the maximum size a log file can grow to. Unit is megabytes. If the value is 0, the maximum file size is unlimited. (DEPRECATED: will be removed in a future release, see https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/2845-deprecate-klog-specific-flags-in-k8s-components) (default 1800)
    -logexporter-gcs-path string
      	Path to the GCS artifacts directory to dump logs from nodes. Logexporter gets enabled if this is non-empty.
    -logtostderr
      	log to standard error instead of files (DEPRECATED: will be removed in a future release, see https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/2845-deprecate-klog-specific-flags-in-k8s-components) (default true)
    -machine_id_file string
      	Comma-separated list of files to check for machine-id. Use the first one that exists. (default "/etc/machine-id,/var/lib/dbus/machine-id")
    -master-os-distro string
      	The OS distribution of cluster master (debian, ubuntu, gci, coreos, or custom). (default "debian")
    -master-tag string
      	Network tags used on master instances. Valid only for gce, gke
... skipping 169 lines ...
      	If true, adds the file directory to the header of the log messages (DEPRECATED: will be removed in a future release, see https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/2845-deprecate-klog-specific-flags-in-k8s-components)
    -allow_dynamic_housekeeping
      	Whether to allow the housekeeping interval to be dynamic (default true)
    -allowed-not-ready-nodes int
      	If greater than zero, framework will allow for that many non-ready nodes when checking for all ready nodes. If -1, no waiting will be performed for ready nodes or daemonset pods.
    -alsologtostderr
      	log to standard error as well as files (DEPRECATED: will be removed in a future release, see https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/2845-deprecate-klog-specific-flags-in-k8s-components)
    -application_metrics_count_limit int
      	Max number of application metrics to store (per container) (default 100)
    -boot_id_file string
      	Comma-separated list of files to check for boot-id. Use the first one that exists. (default "/proc/sys/kernel/random/boot_id")
    -cert-dir string
      	Path to the directory containing the certs. Default is empty, which doesn't use certs.
    -clean-start
      	If true, purge all namespaces except default and system before running tests. This serves to Cleanup test namespaces from failed/interrupted e2e runs in a long-lived cluster.
    -cloud-config-file string
      	Cloud config file.  Only required if provider is azure or vsphere.
    -cloud-provider-gce-l7lb-src-cidrs value
      	CIDRs opened in GCE firewall for L7 LB traffic proxy & health checks (default 130.211.0.0/22,35.191.0.0/16)
    -cloud-provider-gce-lb-src-cidrs value
      	CIDRs opened in GCE firewall for L4 LB traffic proxy & health checks (default 130.211.0.0/22,209.85.152.0/22,209.85.204.0/22,35.191.0.0/16)
... skipping 79 lines ...
      	If set, ginkgo will emit node output to files when running in parallel.
    -ginkgo.dryRun
      	If set, ginkgo will walk the test hierarchy without actually running anything.  Best paired with -v.
    -ginkgo.failFast
      	If set, ginkgo will stop running a test suite after a failure occurs.
    -ginkgo.failOnPending
      	If set, ginkgo will mark the test suite as failed if any specs are pending.
    -ginkgo.flakeAttempts int
      	Make up to this many attempts to run each spec. Please note that if any of the attempts succeed, the suite will not be failed. But any failures will still be recorded. (default 1)
    -ginkgo.focus string
      	If set, ginkgo will only run specs that match this regular expression.
    -ginkgo.noColor
      	If set, suppress color output in default reporter.
    -ginkgo.noisyPendings
      	If set, default reporter will shout about pending tests. (default true)
... skipping 75 lines ...
      	If non-empty, use this log file (DEPRECATED: will be removed in a future release, see https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/2845-deprecate-klog-specific-flags-in-k8s-components)
    -log_file_max_size uint
      	Defines the maximum size a log file can grow to. Unit is megabytes. If the value is 0, the maximum file size is unlimited. (DEPRECATED: will be removed in a future release, see https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/2845-deprecate-klog-specific-flags-in-k8s-components) (default 1800)
    -logexporter-gcs-path string
      	Path to the GCS artifacts directory to dump logs from nodes. Logexporter gets enabled if this is non-empty.
    -logtostderr
      	log to standard error instead of files (DEPRECATED: will be removed in a future release, see https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/2845-deprecate-klog-specific-flags-in-k8s-components) (default true)
    -machine_id_file string
      	Comma-separated list of files to check for machine-id. Use the first one that exists. (default "/etc/machine-id,/var/lib/dbus/machine-id")
    -master-os-distro string
      	The OS distribution of cluster master (debian, ubuntu, gci, coreos, or custom). (default "debian")
    -master-tag string
      	Network tags used on master instances. Valid only for gce, gke
... skipping 169 lines ...
      	If true, adds the file directory to the header of the log messages (DEPRECATED: will be removed in a future release, see https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/2845-deprecate-klog-specific-flags-in-k8s-components)
    -allow_dynamic_housekeeping
      	Whether to allow the housekeeping interval to be dynamic (default true)
    -allowed-not-ready-nodes int
      	If greater than zero, framework will allow for that many non-ready nodes when checking for all ready nodes. If -1, no waiting will be performed for ready nodes or daemonset pods.
    -alsologtostderr
      	log to standard error as well as files (DEPRECATED: will be removed in a future release, see https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/2845-deprecate-klog-specific-flags-in-k8s-components)
    -application_metrics_count_limit int
      	Max number of application metrics to store (per container) (default 100)
    -boot_id_file string
      	Comma-separated list of files to check for boot-id. Use the first one that exists. (default "/proc/sys/kernel/random/boot_id")
    -cert-dir string
      	Path to the directory containing the certs. Default is empty, which doesn't use certs.
    -clean-start
      	If true, purge all namespaces except default and system before running tests. This serves to Cleanup test namespaces from failed/interrupted e2e runs in a long-lived cluster.
    -cloud-config-file string
      	Cloud config file.  Only required if provider is azure or vsphere.
    -cloud-provider-gce-l7lb-src-cidrs value
      	CIDRs opened in GCE firewall for L7 LB traffic proxy & health checks (default 130.211.0.0/22,35.191.0.0/16)
    -cloud-provider-gce-lb-src-cidrs value
      	CIDRs opened in GCE firewall for L4 LB traffic proxy & health checks (default 130.211.0.0/22,209.85.152.0/22,209.85.204.0/22,35.191.0.0/16)
... skipping 79 lines ...
      	If set, ginkgo will emit node output to files when running in parallel.
    -ginkgo.dryRun
      	If set, ginkgo will walk the test hierarchy without actually running anything.  Best paired with -v.
    -ginkgo.failFast
      	If set, ginkgo will stop running a test suite after a failure occurs.
    -ginkgo.failOnPending
      	If set, ginkgo will mark the test suite as failed if any specs are pending.
    -ginkgo.flakeAttempts int
      	Make up to this many attempts to run each spec. Please note that if any of the attempts succeed, the suite will not be failed. But any failures will still be recorded. (default 1)
    -ginkgo.focus string
      	If set, ginkgo will only run specs that match this regular expression.
    -ginkgo.noColor
      	If set, suppress color output in default reporter.
    -ginkgo.noisyPendings
      	If set, default reporter will shout about pending tests. (default true)
... skipping 75 lines ...
      	If non-empty, use this log file (DEPRECATED: will be removed in a future release, see https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/2845-deprecate-klog-specific-flags-in-k8s-components)
    -log_file_max_size uint
      	Defines the maximum size a log file can grow to. Unit is megabytes. If the value is 0, the maximum file size is unlimited. (DEPRECATED: will be removed in a future release, see https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/2845-deprecate-klog-specific-flags-in-k8s-components) (default 1800)
    -logexporter-gcs-path string
      	Path to the GCS artifacts directory to dump logs from nodes. Logexporter gets enabled if this is non-empty.
    -logtostderr
      	log to standard error instead of files (DEPRECATED: will be removed in a future release, see https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/2845-deprecate-klog-specific-flags-in-k8s-components) (default true)
    -machine_id_file string
      	Comma-separated list of files to check for machine-id. Use the first one that exists. (default "/etc/machine-id,/var/lib/dbus/machine-id")
    -master-os-distro string
      	The OS distribution of cluster master (debian, ubuntu, gci, coreos, or custom). (default "debian")
    -master-tag string
      	Network tags used on master instances. Valid only for gce, gke
... skipping 169 lines ...
      	If true, adds the file directory to the header of the log messages (DEPRECATED: will be removed in a future release, see https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/2845-deprecate-klog-specific-flags-in-k8s-components)
    -allow_dynamic_housekeeping
      	Whether to allow the housekeeping interval to be dynamic (default true)
    -allowed-not-ready-nodes int
      	If greater than zero, framework will allow for that many non-ready nodes when checking for all ready nodes. If -1, no waiting will be performed for ready nodes or daemonset pods.
    -alsologtostderr
      	log to standard error as well as files (DEPRECATED: will be removed in a future release, see https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/2845-deprecate-klog-specific-flags-in-k8s-components)
    -application_metrics_count_limit int
      	Max number of application metrics to store (per container) (default 100)
    -boot_id_file string
      	Comma-separated list of files to check for boot-id. Use the first one that exists. (default "/proc/sys/kernel/random/boot_id")
    -cert-dir string
      	Path to the directory containing the certs. Default is empty, which doesn't use certs.
    -clean-start
      	If true, purge all namespaces except default and system before running tests. This serves to Cleanup test namespaces from failed/interrupted e2e runs in a long-lived cluster.
    -cloud-config-file string
      	Cloud config file.  Only required if provider is azure or vsphere.
    -cloud-provider-gce-l7lb-src-cidrs value
      	CIDRs opened in GCE firewall for L7 LB traffic proxy & health checks (default 130.211.0.0/22,35.191.0.0/16)
    -cloud-provider-gce-lb-src-cidrs value
      	CIDRs opened in GCE firewall for L4 LB traffic proxy & health checks (default 130.211.0.0/22,209.85.152.0/22,209.85.204.0/22,35.191.0.0/16)
... skipping 79 lines ...
      	If set, ginkgo will emit node output to files when running in parallel.
    -ginkgo.dryRun
      	If set, ginkgo will walk the test hierarchy without actually running anything.  Best paired with -v.
    -ginkgo.failFast
      	If set, ginkgo will stop running a test suite after a failure occurs.
    -ginkgo.failOnPending
      	If set, ginkgo will mark the test suite as failed if any specs are pending.
    -ginkgo.flakeAttempts int
      	Make up to this many attempts to run each spec. Please note that if any of the attempts succeed, the suite will not be failed. But any failures will still be recorded. (default 1)
    -ginkgo.focus string
      	If set, ginkgo will only run specs that match this regular expression.
    -ginkgo.noColor
      	If set, suppress color output in default reporter.
    -ginkgo.noisyPendings
      	If set, default reporter will shout about pending tests. (default true)
... skipping 75 lines ...
      	If non-empty, use this log file (DEPRECATED: will be removed in a future release, see https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/2845-deprecate-klog-specific-flags-in-k8s-components)
    -log_file_max_size uint
      	Defines the maximum size a log file can grow to. Unit is megabytes. If the value is 0, the maximum file size is unlimited. (DEPRECATED: will be removed in a future release, see https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/2845-deprecate-klog-specific-flags-in-k8s-components) (default 1800)
    -logexporter-gcs-path string
      	Path to the GCS artifacts directory to dump logs from nodes. Logexporter gets enabled if this is non-empty.
    -logtostderr
      	log to standard error instead of files (DEPRECATED: will be removed in a future release, see https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/2845-deprecate-klog-specific-flags-in-k8s-components) (default true)
    -machine_id_file string
      	Comma-separated list of files to check for machine-id. Use the first one that exists. (default "/etc/machine-id,/var/lib/dbus/machine-id")
    -master-os-distro string
      	The OS distribution of cluster master (debian, ubuntu, gci, coreos, or custom). (default "debian")
    -master-tag string
      	Network tags used on master instances. Valid only for gce, gke
... skipping 162 lines ...
    -volume-dir string
      	An alias for --kubelet-root-dir, kept for backwards compatibility. (default "/var/lib/kubelet")

** End **
Ginkgo ran 1 suite in 1.256576342s

Test Suite Failed
[38;5;228mYou're using deprecated Ginkgo functionality:[0m
[38;5;228m=============================================[0m
  [38;5;11m--progress is deprecated .  The functionality provided by --progress was confusing and is no longer needed.  Use --show-node-events instead to see node entry and exit events included in the timeline of failed and verbose specs.  Or you can run with -vv to always see all node events.  Lastly, --poll-progress-after and the PollProgressAfter decorator now provide a better mechanism for debugging specs that tend to get stuck.[0m

[38;5;243mTo silence deprecations that can be silenced set the following environment variable:[0m
  [38;5;243mACK_GINKGO_DEPRECATIONS=2.8.0[0m

+ print_logs
+ sed -i s/disk.csi.azure.com/test.csi.azure.com/g deploy/example/storageclass-azuredisk-csi.yaml
... skipping 105 lines ...
I0201 23:53:22.414313       1 reflector.go:257] Listing and watching *v1.CustomResourceDefinition from k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132
I0201 23:53:22.414759       1 reflector.go:221] Starting reflector *v1beta2.AzVolumeAttachment (30s) from sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132
I0201 23:53:22.414771       1 reflector.go:257] Listing and watching *v1beta2.AzVolumeAttachment from sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132
I0201 23:53:22.514386       1 shared_informer.go:303] caches populated
I0201 23:53:22.514428       1 azuredisk_v2.go:225] driver userAgent: test.csi.azure.com/latest-v2-3d84ccaf7bb075abd5912e16e39b09b4d5f88f6c e2e-test
I0201 23:53:22.514439       1 azure_disk_utils.go:564] reading cloud config from secret kube-system/azure-cloud-provider
I0201 23:53:22.517082       1 azure_disk_utils.go:571] InitializeCloudFromSecret: failed to get cloud config from secret kube-system/azure-cloud-provider: failed to get secret kube-system/azure-cloud-provider: secrets "azure-cloud-provider" not found
I0201 23:53:22.517103       1 azure_disk_utils.go:576] could not read cloud config from secret kube-system/azure-cloud-provider
I0201 23:53:22.517109       1 azure_disk_utils.go:586] use default AZURE_CREDENTIAL_FILE env var: /etc/kubernetes/azure.json
I0201 23:53:22.517139       1 azure_disk_utils.go:594] read cloud config from file: /etc/kubernetes/azure.json successfully
I0201 23:53:22.517844       1 azure_auth.go:253] Using AzurePublicCloud environment
I0201 23:53:22.517864       1 azure_auth.go:104] azure: using managed identity extension to retrieve access token
I0201 23:53:22.517869       1 azure_auth.go:110] azure: using User Assigned MSI ID to retrieve access token
... skipping 197 lines ...
I0201 23:53:22.702649       1 shared_state.go:489]  "msg"="Storing pod kube-apiserver-k8s-master-34418057-0 and claim [] to podToClaimsMap map." "disk.csi.azure.com/request-id"="9c568bef-a28b-11ed-97d0-6045bd8e6fd7"
I0201 23:53:22.702662       1 pod.go:91]  "msg"="Creating replicas for pod kube-system/kube-apiserver-k8s-master-34418057-0." "disk.csi.azure.com/request-id"="9c568bef-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi/azure.com/pod-name"="kube-system/kube-apiserver-k8s-master-34418057-0"
I0201 23:53:22.702674       1 shared_state.go:313]  "msg"="Getting requested volumes for pod (kube-system/kube-apiserver-k8s-master-34418057-0)." "disk.csi.azure.com/request-id"="9c568bef-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi/azure.com/pod-name"="kube-system/kube-apiserver-k8s-master-34418057-0"
I0201 23:53:22.702685       1 pod.go:99]  "msg"="Pod kube-system/kube-apiserver-k8s-master-34418057-0 has 0 volumes. Volumes: []" "disk.csi.azure.com/request-id"="9c568bef-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi/azure.com/pod-name"="kube-system/kube-apiserver-k8s-master-34418057-0"
I0201 23:53:22.702702       1 pod.go:89]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="9c568bef-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcilePod).createReplicas" "disk.csi/azure.com/pod-name"="kube-system/kube-apiserver-k8s-master-34418057-0" "latency"=34100
I0201 23:53:22.702713       1 shared_state.go:419]  "msg"="Adding pod csi-test-controller-5dfc76c46c-pcgmr to shared map with keyName kube-system/csi-test-controller-5dfc76c46c-pcgmr." "disk.csi.azure.com/request-id"="9c568bef-a28b-11ed-97d0-6045bd8e6fd7"
I0201 23:53:22.703019       1 shared_state.go:425]  "msg"="Pod spec of pod csi-test-controller-5dfc76c46c-pcgmr is: {Volumes:[{Name:socket-dir VolumeSource:{HostPath:nil EmptyDir:&EmptyDirVolumeSource{Medium:,SizeLimit:<nil>,} GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:azure-cred VolumeSource:{HostPath:&HostPathVolumeSource{Path:/etc/kubernetes/,Type:*DirectoryOrCreate,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:csi-test-controller-config VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:&ConfigMapVolumeSource{LocalObjectReference:LocalObjectReference{Name:csi-test-controller-config,},Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,} VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:kube-api-access-d6rgf VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:&ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,} PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:csi-provisioner Image:mcr.microsoft.com/oss/kubernetes-csi/csi-provisioner:v3.2.0 Command:[] Args:[--feature-gates=Topology=true --csi-address=$(ADDRESS) --v=2 --timeout=30s --leader-election --leader-election-namespace=kube-system --worker-threads=100 --extra-create-metadata=true --strict-topology=true --kube-api-qps=50 --kube-api-burst=100] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:ADDRESS Value:/csi/csi.sock ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:524288000 scale:0} d:{Dec:<nil>} s:500Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}] Claims:[]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-d6rgf ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:Always SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:csi-attacher Image:mcr.microsoft.com/oss/kubernetes-csi/csi-attacher:v3.5.0 Command:[] Args:[-v=2 -csi-address=$(ADDRESS) -timeout=600s -leader-election --leader-election-namespace=kube-system -worker-threads=500 -kube-api-qps=50 -kube-api-burst=100] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:ADDRESS Value:/csi/csi.sock ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:524288000 scale:0} d:{Dec:<nil>} s:500Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}] Claims:[]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-d6rgf ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:Always SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:csi-snapshotter Image:mcr.microsoft.com/oss/kubernetes-csi/csi-snapshotter:v5.0.1 Command:[] Args:[-csi-address=$(ADDRESS) -leader-election --leader-election-namespace=kube-system -v=2] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:ADDRESS Value:/csi/csi.sock ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:104857600 scale:0} d:{Dec:<nil>} s:100Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}] Claims:[]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-d6rgf ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:Always SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:csi-resizer Image:mcr.microsoft.com/oss/kubernetes-csi/csi-resizer:v1.5.0 Command:[] Args:[-csi-address=$(ADDRESS) -v=2 -leader-election --leader-election-namespace=kube-system -handle-volume-inuse-error=false -feature-gates=RecoverVolumeExpansionFailure=true -timeout=240s] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:ADDRESS Value:/csi/csi.sock ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:524288000 scale:0} d:{Dec:<nil>} s:500Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}] Claims:[]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-d6rgf ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:Always SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:liveness-probe Image:mcr.microsoft.com/oss/kubernetes-csi/livenessprobe:v2.7.0 Command:[] Args:[--csi-address=/csi/csi.sock --probe-timeout=3s --health-port=29602 --v=2] WorkingDir: Ports:[] EnvFrom:[] Env:[] Resources:{Limits:map[memory:{i:{value:104857600 scale:0} d:{Dec:<nil>} s:100Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}] Claims:[]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-d6rgf ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:Always SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:azuredisk Image:k8sprow.azurecr.io/azuredisk-csi:latest-v2-3d84ccaf7bb075abd5912e16e39b09b4d5f88f6c Command:[] Args:[--v=5 --config=/etc/csi-test-controller/config.yaml] WorkingDir: Ports:[{Name:healthz HostPort:29602 ContainerPort:29602 Protocol:TCP HostIP:} {Name:metrics HostPort:29604 ContainerPort:29604 Protocol:TCP HostIP:}] EnvFrom:[] Env:[{Name:AZURE_CREDENTIAL_FILE Value: ValueFrom:&EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:&ConfigMapKeySelector{LocalObjectReference:LocalObjectReference{Name:azure-cred-file,},Key:path,Optional:*true,},SecretKeyRef:nil,}} {Name:CSI_ENDPOINT Value:unix:///csi/csi.sock ValueFrom:nil} {Name:AZURE_GO_SDK_LOG_LEVEL Value: ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:524288000 scale:0} d:{Dec:<nil>} s:500Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}] Claims:[]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:azure-cred ReadOnly:false MountPath:/etc/kubernetes/ SubPath: MountPropagation:<nil> SubPathExpr:} {Name:csi-test-controller-config ReadOnly:false MountPath:/etc/csi-test-controller SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-d6rgf ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/healthz,Port:{1 0 healthz},Host:,Scheme:HTTP,HTTPHeaders:[]HTTPHeader{},},TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:30,TimeoutSeconds:10,PeriodSeconds:30,SuccessThreshold:1,FailureThreshold:5,TerminationGracePeriodSeconds:nil,} ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:Always SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy:Always TerminationGracePeriodSeconds:0xc000bade80 ActiveDeadlineSeconds:<nil> DNSPolicy:ClusterFirst NodeSelector:map[kubernetes.io/os:linux] ServiceAccountName:csi-azuredisk-controller-sa DeprecatedServiceAccount:csi-azuredisk-controller-sa AutomountServiceAccountToken:<nil> NodeName:k8s-agentpool1-34418057-vmss000000 HostNetwork:true HostPID:false HostIPC:false ShareProcessNamespace:<nil> SecurityContext:&PodSecurityContext{SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,SupplementalGroups:[],FSGroup:nil,RunAsGroup:nil,Sysctls:[]Sysctl{},WindowsOptions:nil,FSGroupChangePolicy:nil,SeccompProfile:nil,} ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName:default-scheduler Tolerations:[{Key:node-role.kubernetes.io/master Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>} {Key:node-role.kubernetes.io/controlplane Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>} {Key:node-role.kubernetes.io/control-plane Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>} {Key:node.kubernetes.io/not-ready Operator:Exists Value: Effect:NoExecute TolerationSeconds:0xc000bade88} {Key:node.kubernetes.io/unreachable Operator:Exists Value: Effect:NoExecute TolerationSeconds:0xc000bade90}] HostAliases:[] PriorityClassName:system-cluster-critical Priority:0xc000bade98 DNSConfig:nil ReadinessGates:[] RuntimeClassName:<nil> EnableServiceLinks:0xc000bade9c PreemptionPolicy:0xc000250730 Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN:<nil> OS:nil HostUsers:<nil> SchedulingGates:[] ResourceClaims:[]}. With volumes: [{Name:socket-dir VolumeSource:{HostPath:nil EmptyDir:&EmptyDirVolumeSource{Medium:,SizeLimit:<nil>,} GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:azure-cred VolumeSource:{HostPath:&HostPathVolumeSource{Path:/etc/kubernetes/,Type:*DirectoryOrCreate,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:csi-test-controller-config VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:&ConfigMapVolumeSource{LocalObjectReference:LocalObjectReference{Name:csi-test-controller-config,},Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,} VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:kube-api-access-d6rgf VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:&ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,} PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}]" "disk.csi.azure.com/request-id"="9c568bef-a28b-11ed-97d0-6045bd8e6fd7"
I0201 23:53:22.703062       1 shared_state.go:463]  "msg"="Pod csi-test-controller-5dfc76c46c-pcgmr: Skipping Volume {socket-dir {nil &EmptyDirVolumeSource{Medium:,SizeLimit:<nil>,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}}. No persistent volume exists." "disk.csi.azure.com/request-id"="9c568bef-a28b-11ed-97d0-6045bd8e6fd7"
I0201 23:53:22.703079       1 shared_state.go:463]  "msg"="Pod csi-test-controller-5dfc76c46c-pcgmr: Skipping Volume {azure-cred {&HostPathVolumeSource{Path:/etc/kubernetes/,Type:*DirectoryOrCreate,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}}. No persistent volume exists." "disk.csi.azure.com/request-id"="9c568bef-a28b-11ed-97d0-6045bd8e6fd7"
I0201 23:53:22.703098       1 shared_state.go:463]  "msg"="Pod csi-test-controller-5dfc76c46c-pcgmr: Skipping Volume {csi-test-controller-config {nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil &ConfigMapVolumeSource{LocalObjectReference:LocalObjectReference{Name:csi-test-controller-config,},Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,} nil nil nil nil nil nil nil nil nil nil}}. No persistent volume exists." "disk.csi.azure.com/request-id"="9c568bef-a28b-11ed-97d0-6045bd8e6fd7"
I0201 23:53:22.703128       1 shared_state.go:463]  "msg"="Pod csi-test-controller-5dfc76c46c-pcgmr: Skipping Volume {kube-api-access-d6rgf {nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil &ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,} nil nil nil nil nil}}. No persistent volume exists." "disk.csi.azure.com/request-id"="9c568bef-a28b-11ed-97d0-6045bd8e6fd7"
I0201 23:53:22.703138       1 shared_state.go:489]  "msg"="Storing pod csi-test-controller-5dfc76c46c-pcgmr and claim [] to podToClaimsMap map." "disk.csi.azure.com/request-id"="9c568bef-a28b-11ed-97d0-6045bd8e6fd7"
I0201 23:53:22.703156       1 pod.go:91]  "msg"="Creating replicas for pod kube-system/csi-test-controller-5dfc76c46c-pcgmr." "disk.csi.azure.com/request-id"="9c568bef-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi/azure.com/pod-name"="kube-system/csi-test-controller-5dfc76c46c-pcgmr"
... skipping 272 lines ...
I0201 23:53:22.731050       1 utils.go:79] GRPC request: {}
I0201 23:53:22.731072       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}},{"Type":{"Rpc":{"type":7}}},{"Type":{"Rpc":{"type":9}}},{"Type":{"Rpc":{"type":13}}}]}
I0201 23:53:22.731474       1 utils.go:78] GRPC call: /csi.v1.Controller/ControllerGetCapabilities
I0201 23:53:22.731491       1 utils.go:79] GRPC request: {}
I0201 23:53:22.731516       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}},{"Type":{"Rpc":{"type":7}}},{"Type":{"Rpc":{"type":9}}},{"Type":{"Rpc":{"type":13}}}]}
I0201 23:53:22.752929       1 azuredisk_v2.go:435]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="9c568bef-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).StartControllersAndDieOnExit.func1" "latency"=210409809
I0201 23:53:22.803786       1 node.go:90] AzDiskControllerManager "msg"="Node is now available. Will requeue failed replica creation requests." "controller"="node" "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000000" "namespace"="azure-disk-csi" "partition"="csi-azuredisk-controller"
I0201 23:53:22.803891       1 shared_state.go:1203]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="9c7e6de4-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*SharedState).tryCreateFailedReplicas" "latency"=10600
I0201 23:53:22.803796       1 node.go:90] AzDiskControllerManager "msg"="Node is now available. Will requeue failed replica creation requests." "controller"="node" "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000002" "namespace"="azure-disk-csi" "partition"="csi-azuredisk-controller"
I0201 23:53:22.803905       1 node.go:90] AzDiskControllerManager "msg"="Node is now available. Will requeue failed replica creation requests." "controller"="node" "disk.csi.azure.com/node-name"="k8s-master-34418057-0" "namespace"="azure-disk-csi" "partition"="csi-azuredisk-controller"
I0201 23:53:22.803925       1 shared_state.go:1203]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="9c7e6fd1-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*SharedState).tryCreateFailedReplicas" "latency"=3400
I0201 23:53:22.803936       1 azure_disk_utils.go:1048]  "msg"="Getting AzVolumeAttachment list for node (k8s-agentpool1-34418057-vmss000001)" 
I0201 23:53:22.803988       1 azure_disk_utils.go:1068]  "msg"="Label selector is: disk.csi.azure.com/node-name=k8s-agentpool1-34418057-vmss000001." 
I0201 23:53:22.804009       1 shared_state.go:1778] Added node(k8s-agentpool1-34418057-vmss000001) to availableAttachmentsMap with capacity: 16
I0201 23:53:22.804036       1 node.go:90] AzDiskControllerManager "msg"="Node is now available. Will requeue failed replica creation requests." "controller"="node" "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000001" "namespace"="azure-disk-csi" "partition"="csi-azuredisk-controller"
I0201 23:53:22.803948       1 shared_state.go:1203]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="9c7e7048-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*SharedState).tryCreateFailedReplicas" "latency"=7200
I0201 23:53:22.804068       1 shared_state.go:1203]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="9c7e7501-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*SharedState).tryCreateFailedReplicas" "latency"=6000
I0201 23:53:22.846240       1 shared_state.go:419]  "msg"="Adding pod kube-proxy-ft8gc to shared map with keyName kube-system/kube-proxy-ft8gc." 
I0201 23:53:22.846682       1 shared_state.go:425]  "msg"="Pod spec of pod kube-proxy-ft8gc is: {Volumes:[{Name:ssl-certs-host VolumeSource:{HostPath:&HostPathVolumeSource{Path:/usr/share/ca-certificates,Type:*,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:kubeconfig VolumeSource:{HostPath:&HostPathVolumeSource{Path:/var/lib/kubelet/kubeconfig,Type:*,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:etc-kubernetes VolumeSource:{HostPath:&HostPathVolumeSource{Path:/etc/kubernetes,Type:*,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:iptableslock VolumeSource:{HostPath:&HostPathVolumeSource{Path:/run/xtables.lock,Type:*,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:kernelmodules VolumeSource:{HostPath:&HostPathVolumeSource{Path:/lib/modules/,Type:*,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:kube-proxy-config-volume VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:&ConfigMapVolumeSource{LocalObjectReference:LocalObjectReference{Name:kube-proxy-config,},Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,} VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:kube-api-access-wbxwd VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:&ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,} PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:kube-proxy Image:mcr.microsoft.com/oss/kubernetes/kube-proxy:v1.23.15 Command:[kube-proxy --config=/var/lib/kube-proxy/config.yaml] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[] Resources:{Limits:map[] Requests:map[cpu:{i:{value:100 scale:-3} d:{Dec:<nil>} s:100m Format:DecimalSI}] Claims:[]} VolumeMounts:[{Name:ssl-certs-host ReadOnly:true MountPath:/etc/ssl/certs SubPath: MountPropagation:<nil> SubPathExpr:} {Name:etc-kubernetes ReadOnly:true MountPath:/etc/kubernetes SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kubeconfig ReadOnly:true MountPath:/var/lib/kubelet/kubeconfig SubPath: MountPropagation:<nil> SubPathExpr:} {Name:iptableslock ReadOnly:false MountPath:/run/xtables.lock SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kernelmodules ReadOnly:true MountPath:/lib/modules/ SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-proxy-config-volume ReadOnly:true MountPath:/var/lib/kube-proxy/config.yaml SubPath:config.yaml MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-wbxwd ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:Always SecurityContext:&SecurityContext{Capabilities:nil,Privileged:*true,SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,ReadOnlyRootFilesystem:nil,AllowPrivilegeEscalation:nil,RunAsGroup:nil,ProcMount:nil,WindowsOptions:nil,SeccompProfile:nil,} Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy:Always TerminationGracePeriodSeconds:0xc000d96718 ActiveDeadlineSeconds:<nil> DNSPolicy:ClusterFirst NodeSelector:map[kubernetes.io/os:linux] ServiceAccountName:default DeprecatedServiceAccount:default AutomountServiceAccountToken:<nil> NodeName:k8s-agentpool1-34418057-vmss000002 HostNetwork:true HostPID:false HostIPC:false ShareProcessNamespace:<nil> SecurityContext:&PodSecurityContext{SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,SupplementalGroups:[],FSGroup:nil,RunAsGroup:nil,Sysctls:[]Sysctl{},WindowsOptions:nil,FSGroupChangePolicy:nil,SeccompProfile:nil,} ImagePullSecrets:[] Hostname: Subdomain: Affinity:&Affinity{NodeAffinity:&NodeAffinity{RequiredDuringSchedulingIgnoredDuringExecution:&NodeSelector{NodeSelectorTerms:[]NodeSelectorTerm{NodeSelectorTerm{MatchExpressions:[]NodeSelectorRequirement{},MatchFields:[]NodeSelectorRequirement{NodeSelectorRequirement{Key:metadata.name,Operator:In,Values:[k8s-agentpool1-34418057-vmss000002],},},},},},PreferredDuringSchedulingIgnoredDuringExecution:[]PreferredSchedulingTerm{},},PodAffinity:nil,PodAntiAffinity:nil,} SchedulerName:default-scheduler Tolerations:[{Key:node-role.kubernetes.io/master Operator:Equal Value:true Effect:NoSchedule TolerationSeconds:<nil>} {Key: Operator:Exists Value: Effect:NoExecute TolerationSeconds:<nil>} {Key: Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>} {Key:CriticalAddonsOnly Operator:Exists Value: Effect: TolerationSeconds:<nil>} {Key:node.kubernetes.io/not-ready Operator:Exists Value: Effect:NoExecute TolerationSeconds:<nil>} {Key:node.kubernetes.io/unreachable Operator:Exists Value: Effect:NoExecute TolerationSeconds:<nil>} {Key:node.kubernetes.io/disk-pressure Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>} {Key:node.kubernetes.io/memory-pressure Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>} {Key:node.kubernetes.io/pid-pressure Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>} {Key:node.kubernetes.io/unschedulable Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>} {Key:node.kubernetes.io/network-unavailable Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>}] HostAliases:[] PriorityClassName:system-node-critical Priority:0xc000d96720 DNSConfig:nil ReadinessGates:[] RuntimeClassName:<nil> EnableServiceLinks:0xc000d96724 PreemptionPolicy:0xc0008b8e00 Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN:<nil> OS:nil HostUsers:<nil> SchedulingGates:[] ResourceClaims:[]}. With volumes: [{Name:ssl-certs-host VolumeSource:{HostPath:&HostPathVolumeSource{Path:/usr/share/ca-certificates,Type:*,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:kubeconfig VolumeSource:{HostPath:&HostPathVolumeSource{Path:/var/lib/kubelet/kubeconfig,Type:*,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:etc-kubernetes VolumeSource:{HostPath:&HostPathVolumeSource{Path:/etc/kubernetes,Type:*,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:iptableslock VolumeSource:{HostPath:&HostPathVolumeSource{Path:/run/xtables.lock,Type:*,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:kernelmodules VolumeSource:{HostPath:&HostPathVolumeSource{Path:/lib/modules/,Type:*,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:kube-proxy-config-volume VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:&ConfigMapVolumeSource{LocalObjectReference:LocalObjectReference{Name:kube-proxy-config,},Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,} VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:kube-api-access-wbxwd VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:&ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,} PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}]" 
I0201 23:53:22.846743       1 shared_state.go:463]  "msg"="Pod kube-proxy-ft8gc: Skipping Volume {ssl-certs-host {&HostPathVolumeSource{Path:/usr/share/ca-certificates,Type:*,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}}. No persistent volume exists." 
I0201 23:53:22.846763       1 shared_state.go:463]  "msg"="Pod kube-proxy-ft8gc: Skipping Volume {kubeconfig {&HostPathVolumeSource{Path:/var/lib/kubelet/kubeconfig,Type:*,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}}. No persistent volume exists." 
... skipping 39 lines ...
I0201 23:53:23.046357       1 shared_state.go:489]  "msg"="Storing pod metrics-server-5b8986d847-qjdvt and claim [] to podToClaimsMap map." 
I0201 23:53:23.046380       1 pod.go:91]  "msg"="Creating replicas for pod kube-system/metrics-server-5b8986d847-qjdvt." "disk.csi.azure.com/request-id"="9ca36ead-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi/azure.com/pod-name"="kube-system/metrics-server-5b8986d847-qjdvt"
I0201 23:53:23.046395       1 shared_state.go:313]  "msg"="Getting requested volumes for pod (kube-system/metrics-server-5b8986d847-qjdvt)." "disk.csi.azure.com/request-id"="9ca36ead-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi/azure.com/pod-name"="kube-system/metrics-server-5b8986d847-qjdvt"
I0201 23:53:23.046413       1 pod.go:99]  "msg"="Pod kube-system/metrics-server-5b8986d847-qjdvt has 0 volumes. Volumes: []" "disk.csi.azure.com/request-id"="9ca36ead-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi/azure.com/pod-name"="kube-system/metrics-server-5b8986d847-qjdvt"
I0201 23:53:23.046439       1 pod.go:89]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="9ca36ead-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcilePod).createReplicas" "disk.csi/azure.com/pod-name"="kube-system/metrics-server-5b8986d847-qjdvt" "latency"=59801
I0201 23:53:23.146358       1 shared_state.go:419]  "msg"="Adding pod csi-test-controller-5dfc76c46c-pcgmr to shared map with keyName kube-system/csi-test-controller-5dfc76c46c-pcgmr." 
I0201 23:53:23.146828       1 shared_state.go:425]  "msg"="Pod spec of pod csi-test-controller-5dfc76c46c-pcgmr is: {Volumes:[{Name:socket-dir VolumeSource:{HostPath:nil EmptyDir:&EmptyDirVolumeSource{Medium:,SizeLimit:<nil>,} GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:azure-cred VolumeSource:{HostPath:&HostPathVolumeSource{Path:/etc/kubernetes/,Type:*DirectoryOrCreate,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:csi-test-controller-config VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:&ConfigMapVolumeSource{LocalObjectReference:LocalObjectReference{Name:csi-test-controller-config,},Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,} VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:kube-api-access-d6rgf VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:&ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,} PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:csi-provisioner Image:mcr.microsoft.com/oss/kubernetes-csi/csi-provisioner:v3.2.0 Command:[] Args:[--feature-gates=Topology=true --csi-address=$(ADDRESS) --v=2 --timeout=30s --leader-election --leader-election-namespace=kube-system --worker-threads=100 --extra-create-metadata=true --strict-topology=true --kube-api-qps=50 --kube-api-burst=100] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:ADDRESS Value:/csi/csi.sock ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:524288000 scale:0} d:{Dec:<nil>} s:500Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}] Claims:[]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-d6rgf ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:Always SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:csi-attacher Image:mcr.microsoft.com/oss/kubernetes-csi/csi-attacher:v3.5.0 Command:[] Args:[-v=2 -csi-address=$(ADDRESS) -timeout=600s -leader-election --leader-election-namespace=kube-system -worker-threads=500 -kube-api-qps=50 -kube-api-burst=100] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:ADDRESS Value:/csi/csi.sock ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:524288000 scale:0} d:{Dec:<nil>} s:500Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}] Claims:[]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-d6rgf ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:Always SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:csi-snapshotter Image:mcr.microsoft.com/oss/kubernetes-csi/csi-snapshotter:v5.0.1 Command:[] Args:[-csi-address=$(ADDRESS) -leader-election --leader-election-namespace=kube-system -v=2] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:ADDRESS Value:/csi/csi.sock ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:104857600 scale:0} d:{Dec:<nil>} s:100Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}] Claims:[]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-d6rgf ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:Always SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:csi-resizer Image:mcr.microsoft.com/oss/kubernetes-csi/csi-resizer:v1.5.0 Command:[] Args:[-csi-address=$(ADDRESS) -v=2 -leader-election --leader-election-namespace=kube-system -handle-volume-inuse-error=false -feature-gates=RecoverVolumeExpansionFailure=true -timeout=240s] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:ADDRESS Value:/csi/csi.sock ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:524288000 scale:0} d:{Dec:<nil>} s:500Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}] Claims:[]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-d6rgf ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:Always SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:liveness-probe Image:mcr.microsoft.com/oss/kubernetes-csi/livenessprobe:v2.7.0 Command:[] Args:[--csi-address=/csi/csi.sock --probe-timeout=3s --health-port=29602 --v=2] WorkingDir: Ports:[] EnvFrom:[] Env:[] Resources:{Limits:map[memory:{i:{value:104857600 scale:0} d:{Dec:<nil>} s:100Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}] Claims:[]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-d6rgf ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:Always SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:azuredisk Image:k8sprow.azurecr.io/azuredisk-csi:latest-v2-3d84ccaf7bb075abd5912e16e39b09b4d5f88f6c Command:[] Args:[--v=5 --config=/etc/csi-test-controller/config.yaml] WorkingDir: Ports:[{Name:healthz HostPort:29602 ContainerPort:29602 Protocol:TCP HostIP:} {Name:metrics HostPort:29604 ContainerPort:29604 Protocol:TCP HostIP:}] EnvFrom:[] Env:[{Name:AZURE_CREDENTIAL_FILE Value: ValueFrom:&EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:&ConfigMapKeySelector{LocalObjectReference:LocalObjectReference{Name:azure-cred-file,},Key:path,Optional:*true,},SecretKeyRef:nil,}} {Name:CSI_ENDPOINT Value:unix:///csi/csi.sock ValueFrom:nil} {Name:AZURE_GO_SDK_LOG_LEVEL Value: ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:524288000 scale:0} d:{Dec:<nil>} s:500Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}] Claims:[]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:azure-cred ReadOnly:false MountPath:/etc/kubernetes/ SubPath: MountPropagation:<nil> SubPathExpr:} {Name:csi-test-controller-config ReadOnly:false MountPath:/etc/csi-test-controller SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-d6rgf ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/healthz,Port:{1 0 healthz},Host:,Scheme:HTTP,HTTPHeaders:[]HTTPHeader{},},TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:30,TimeoutSeconds:10,PeriodSeconds:30,SuccessThreshold:1,FailureThreshold:5,TerminationGracePeriodSeconds:nil,} ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:Always SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy:Always TerminationGracePeriodSeconds:0xc000c218f0 ActiveDeadlineSeconds:<nil> DNSPolicy:ClusterFirst NodeSelector:map[kubernetes.io/os:linux] ServiceAccountName:csi-azuredisk-controller-sa DeprecatedServiceAccount:csi-azuredisk-controller-sa AutomountServiceAccountToken:<nil> NodeName:k8s-agentpool1-34418057-vmss000000 HostNetwork:true HostPID:false HostIPC:false ShareProcessNamespace:<nil> SecurityContext:&PodSecurityContext{SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,SupplementalGroups:[],FSGroup:nil,RunAsGroup:nil,Sysctls:[]Sysctl{},WindowsOptions:nil,FSGroupChangePolicy:nil,SeccompProfile:nil,} ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName:default-scheduler Tolerations:[{Key:node-role.kubernetes.io/master Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>} {Key:node-role.kubernetes.io/controlplane Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>} {Key:node-role.kubernetes.io/control-plane Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>} {Key:node.kubernetes.io/not-ready Operator:Exists Value: Effect:NoExecute TolerationSeconds:0xc000c218f8} {Key:node.kubernetes.io/unreachable Operator:Exists Value: Effect:NoExecute TolerationSeconds:0xc000c21900}] HostAliases:[] PriorityClassName:system-cluster-critical Priority:0xc000c21908 DNSConfig:nil ReadinessGates:[] RuntimeClassName:<nil> EnableServiceLinks:0xc000c2190c PreemptionPolicy:0xc000cb7d60 Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN:<nil> OS:nil HostUsers:<nil> SchedulingGates:[] ResourceClaims:[]}. With volumes: [{Name:socket-dir VolumeSource:{HostPath:nil EmptyDir:&EmptyDirVolumeSource{Medium:,SizeLimit:<nil>,} GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:azure-cred VolumeSource:{HostPath:&HostPathVolumeSource{Path:/etc/kubernetes/,Type:*DirectoryOrCreate,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:csi-test-controller-config VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:&ConfigMapVolumeSource{LocalObjectReference:LocalObjectReference{Name:csi-test-controller-config,},Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,} VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:kube-api-access-d6rgf VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:&ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,} PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}]" 
I0201 23:53:23.146875       1 shared_state.go:463]  "msg"="Pod csi-test-controller-5dfc76c46c-pcgmr: Skipping Volume {socket-dir {nil &EmptyDirVolumeSource{Medium:,SizeLimit:<nil>,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}}. No persistent volume exists." 
I0201 23:53:23.146896       1 shared_state.go:463]  "msg"="Pod csi-test-controller-5dfc76c46c-pcgmr: Skipping Volume {azure-cred {&HostPathVolumeSource{Path:/etc/kubernetes/,Type:*DirectoryOrCreate,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}}. No persistent volume exists." 
I0201 23:53:23.146916       1 shared_state.go:463]  "msg"="Pod csi-test-controller-5dfc76c46c-pcgmr: Skipping Volume {csi-test-controller-config {nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil &ConfigMapVolumeSource{LocalObjectReference:LocalObjectReference{Name:csi-test-controller-config,},Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,} nil nil nil nil nil nil nil nil nil nil}}. No persistent volume exists." 
I0201 23:53:23.146948       1 shared_state.go:463]  "msg"="Pod csi-test-controller-5dfc76c46c-pcgmr: Skipping Volume {kube-api-access-d6rgf {nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil &ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,} nil nil nil nil nil}}. No persistent volume exists." 
I0201 23:53:23.146959       1 shared_state.go:489]  "msg"="Storing pod csi-test-controller-5dfc76c46c-pcgmr and claim [] to podToClaimsMap map." 
I0201 23:53:23.146989       1 pod.go:91]  "msg"="Creating replicas for pod kube-system/csi-test-controller-5dfc76c46c-pcgmr." "disk.csi.azure.com/request-id"="9cb2c873-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi/azure.com/pod-name"="kube-system/csi-test-controller-5dfc76c46c-pcgmr"
... skipping 292 lines ...
I0201 23:53:29.406515       1 shared_state.go:489]  "msg"="Storing pod deployment-azuredisk-69468c8c97-snkqp and claim [] to podToClaimsMap map." 
I0201 23:53:29.430746       1 utils.go:78] GRPC call: /csi.v1.Controller/CreateVolume
I0201 23:53:29.430768       1 utils.go:79] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"requisite":[{"segments":{"topology.test.csi.azure.com/zone":""}}]},"capacity_range":{"required_bytes":10737418240},"name":"pvc-add98bb7-ddb5-4554-95ee-1310712bec9c","parameters":{"csi.storage.k8s.io/pv/name":"pvc-add98bb7-ddb5-4554-95ee-1310712bec9c","csi.storage.k8s.io/pvc/name":"pvc-azuredisk","csi.storage.k8s.io/pvc/namespace":"default","skuName":"StandardSSD_LRS"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]}
I0201 23:53:29.431084       1 crdprovisioner.go:308]  "msg"="Creating AzVolume CRI" "csi.storage.k8s.io/pv/name"="pvc-add98bb7-ddb5-4554-95ee-1310712bec9c" "disk.csi.azure.com/request-id"="a071a80b-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/volume-name"="pvc-add98bb7-ddb5-4554-95ee-1310712bec9c"
I0201 23:53:29.444056       1 crdprovisioner.go:316]  "msg"="Successfully created AzVolume CRI" "csi.storage.k8s.io/pv/name"="pvc-add98bb7-ddb5-4554-95ee-1310712bec9c" "disk.csi.azure.com/request-id"="a071a80b-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/volume-name"="pvc-add98bb7-ddb5-4554-95ee-1310712bec9c"
I0201 23:53:29.444085       1 conditionwatcher.go:99] Adding a condition function for azvolume (pvc-add98bb7-ddb5-4554-95ee-1310712bec9c)
I0201 23:53:29.444291       1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0201 23:53:29.449737       1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0201 23:53:29.452125       1 azure_disk_utils.go:1103]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate" "csi.storage.k8s.io/pv/name"="pvc-add98bb7-ddb5-4554-95ee-1310712bec9c" "disk.csi.azure.com/request-id"="a071a80b-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-add98bb7-ddb5-4554-95ee-1310712bec9c" "latency"=7639684 "number_of_net_retry"=0 "number_of_retry"=0
I0201 23:53:29.452151       1 azvolume.go:158]  "msg"="Creating Volume..." "csi.storage.k8s.io/pv/name"="pvc-add98bb7-ddb5-4554-95ee-1310712bec9c" "disk.csi.azure.com/request-id"="a071a80b-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/volume-name"="pvc-add98bb7-ddb5-4554-95ee-1310712bec9c"
I0201 23:53:29.571674       1 util.go:124] Send.sendRequest got response with ContentLength 253, StatusCode 404 and responseBody length 253
I0201 23:53:29.571709       1 azure_diskclient.go:139] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-dwxhf0qv/providers/Microsoft.Compute/disks/pvc-add98bb7-ddb5-4554-95ee-1310712bec9c, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-add98bb7-ddb5-4554-95ee-1310712bec9c' under resource group 'kubetest-dwxhf0qv' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I0201 23:53:29.571876       1 cloudprovisioner.go:246] begin to create disk(pvc-add98bb7-ddb5-4554-95ee-1310712bec9c) account type(StandardSSD_LRS) rg(kubetest-dwxhf0qv) location() size(10) selectedAvailabilityZone() maxShares(0)
I0201 23:53:29.648098       1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 3502
I0201 23:53:29.648269       1 azure_managedDiskController.go:94] azureDisk - creating new managed Name:pvc-add98bb7-ddb5-4554-95ee-1310712bec9c StorageAccountType:StandardSSD_LRS Size:10
I0201 23:53:30.590899       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0201 23:53:32.154674       1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1193
I0201 23:53:32.154857       1 azure_managedDiskController.go:280] azureDisk - created new MD Name:pvc-add98bb7-ddb5-4554-95ee-1310712bec9c StorageAccountType:StandardSSD_LRS Size:10
I0201 23:53:32.154912       1 cloudprovisioner.go:320]  "msg"="create disk(pvc-add98bb7-ddb5-4554-95ee-1310712bec9c) account type(StandardSSD_LRS) rg(kubetest-dwxhf0qv) location() size(10) tags(map[kubernetes.io-created-for-pv-name:pvc-add98bb7-ddb5-4554-95ee-1310712bec9c kubernetes.io-created-for-pvc-name:pvc-azuredisk kubernetes.io-created-for-pvc-namespace:default]) successfully" "csi.storage.k8s.io/pv/name"="pvc-add98bb7-ddb5-4554-95ee-1310712bec9c" "disk.csi.azure.com/request-id"="a071a80b-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/volume-name"="pvc-add98bb7-ddb5-4554-95ee-1310712bec9c"
I0201 23:53:32.154949       1 cloudprovisioner.go:142]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-add98bb7-ddb5-4554-95ee-1310712bec9c" "disk.csi.azure.com/request-id"="a071a80b-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-add98bb7-ddb5-4554-95ee-1310712bec9c" "latency"=2702735425
I0201 23:53:32.165500       1 azure_disk_utils.go:1103]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "csi.storage.k8s.io/pv/name"="pvc-add98bb7-ddb5-4554-95ee-1310712bec9c" "disk.csi.azure.com/request-id"="a071a80b-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-add98bb7-ddb5-4554-95ee-1310712bec9c" "latency"=10498116 "number_of_net_retry"=0 "number_of_retry"=0
I0201 23:53:32.165539       1 azvolume.go:165]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-add98bb7-ddb5-4554-95ee-1310712bec9c" "disk.csi.azure.com/request-id"="a071a80b-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "disk.csi.azure.com/volume-name"="pvc-add98bb7-ddb5-4554-95ee-1310712bec9c" "latency"=2713352342
I0201 23:53:32.165559       1 workflow.go:155]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-add98bb7-ddb5-4554-95ee-1310712bec9c" "disk.csi.azure.com/request-id"="a071a80b-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate" "disk.csi.azure.com/volume-name"="pvc-add98bb7-ddb5-4554-95ee-1310712bec9c" "latency"=2721091027
I0201 23:53:32.166351       1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0201 23:53:32.166405       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).CreateVolume" "csi.storage.k8s.io/pv/name"="pvc-add98bb7-ddb5-4554-95ee-1310712bec9c" "disk.csi.azure.com/request-id"="a071a80b-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-add98bb7-ddb5-4554-95ee-1310712bec9c" "latency"=2722257541
I0201 23:53:32.166438       1 crdprovisioner.go:228]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-add98bb7-ddb5-4554-95ee-1310712bec9c" "disk.csi.azure.com/request-id"="a071a80b-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-add98bb7-ddb5-4554-95ee-1310712bec9c" "latency"=2735376585
I0201 23:53:32.166487       1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=2.735427885 request="azuredisk_csi_driver_controller_create_volume" resource_group="kubetest-dwxhf0qv" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="test.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-dwxhf0qv/providers/Microsoft.Compute/disks/pvc-add98bb7-ddb5-4554-95ee-1310712bec9c" result_code="succeeded"
I0201 23:53:32.166504       1 utils.go:85] GRPC response: {"volume":{"accessible_topology":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"capacity_bytes":10737418240,"content_source":{"Type":{"Volume":{}}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-add98bb7-ddb5-4554-95ee-1310712bec9c","csi.storage.k8s.io/pvc/name":"pvc-azuredisk","csi.storage.k8s.io/pvc/namespace":"default","requestedsizegib":"10","skuName":"StandardSSD_LRS"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-dwxhf0qv/providers/Microsoft.Compute/disks/pvc-add98bb7-ddb5-4554-95ee-1310712bec9c"}}
I0201 23:53:32.547534       1 utils.go:78] GRPC call: /csi.v1.Controller/ControllerPublishVolume
I0201 23:53:32.547558       1 utils.go:79] GRPC request: {"node_id":"k8s-agentpool1-34418057-vmss000000","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-add98bb7-ddb5-4554-95ee-1310712bec9c","csi.storage.k8s.io/pvc/name":"pvc-azuredisk","csi.storage.k8s.io/pvc/namespace":"default","requestedsizegib":"10","skuName":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1675295602525-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-dwxhf0qv/providers/Microsoft.Compute/disks/pvc-add98bb7-ddb5-4554-95ee-1310712bec9c"}
I0201 23:53:32.547916       1 azure_disk_utils.go:1068]  "msg"="Label selector is: disk.csi.azure.com/node-name=k8s-agentpool1-34418057-vmss000000,disk.csi.azure.com/volume-name!=pvc-add98bb7-ddb5-4554-95ee-1310712bec9c." "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000000" "disk.csi.azure.com/request-id"="a24d3a2e-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-add98bb7-ddb5-4554-95ee-1310712bec9c"
I0201 23:53:32.553826       1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-add98bb7-ddb5-4554-95ee-1310712bec9c-k8s-agentpool1-34418057-vmss000000-attachment)
I0201 23:53:32.558804       1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0201 23:53:32.560253       1 azure_disk_utils.go:1103]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000000" "disk.csi.azure.com/request-id"="a24d3a2e-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-add98bb7-ddb5-4554-95ee-1310712bec9c" "latency"=6300669 "number_of_net_retry"=0 "number_of_retry"=0
I0201 23:53:32.560282       1 attach_detach.go:192]  "msg"="Attaching volume" "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000000" "disk.csi.azure.com/request-id"="a24d3a2e-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-add98bb7-ddb5-4554-95ee-1310712bec9c"
I0201 23:53:32.584213       1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1193
I0201 23:53:32.596549       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0201 23:53:32.644203       1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 4987
I0201 23:53:32.745060       1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 29044
I0201 23:53:32.748672       1 cloudprovisioner.go:405]  "msg"="GetDiskLun returned: -1. Initiating attaching volume \"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-dwxhf0qv/providers/Microsoft.Compute/disks/pvc-add98bb7-ddb5-4554-95ee-1310712bec9c\" to node \"k8s-agentpool1-34418057-vmss000000\"." "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000000" "disk.csi.azure.com/request-id"="a24d3a2e-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-add98bb7-ddb5-4554-95ee-1310712bec9c"
I0201 23:53:32.748710       1 cloudprovisioner.go:424]  "msg"="Trying to attach volume \"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-dwxhf0qv/providers/Microsoft.Compute/disks/pvc-add98bb7-ddb5-4554-95ee-1310712bec9c\" to node \"k8s-agentpool1-34418057-vmss000000\"." "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000000" "disk.csi.azure.com/request-id"="a24d3a2e-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-add98bb7-ddb5-4554-95ee-1310712bec9c"
I0201 23:53:33.749591       1 batch.go:224] "cloud-provider-azure: Delayed processing of batch due to start delay" type="batch" operation="attach_disk" key="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e|kubetest-dwxhf0qv|k8s-agentpool1-34418057-vmss000000" delay="1s"
I0201 23:53:33.749710       1 azure_controller_vmss.go:110] azureDisk - update(kubetest-dwxhf0qv): vm(k8s-agentpool1-34418057-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-dwxhf0qv/providers/Microsoft.Compute/disks/pvc-add98bb7-ddb5-4554-95ee-1310712bec9c:AttachDiskOptions{diskName: "pvc-add98bb7-ddb5-4554-95ee-1310712bec9c", lun: 0}])
I0201 23:53:33.761251       1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0201 23:53:33.761344       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000000" "disk.csi.azure.com/request-id"="a24d3a2e-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-add98bb7-ddb5-4554-95ee-1310712bec9c" "latency"=1207438420
I0201 23:53:33.761384       1 crdprovisioner.go:727]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000000" "disk.csi.azure.com/request-id"="a24d3a2e-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLun" "disk.csi.azure.com/volume-name"="pvc-add98bb7-ddb5-4554-95ee-1310712bec9c" "latency"=1207579821
I0201 23:53:33.761408       1 crdprovisioner.go:486]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000000" "disk.csi.azure.com/request-id"="a24d3a2e-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).PublishVolume" "disk.csi.azure.com/volume-name"="pvc-add98bb7-ddb5-4554-95ee-1310712bec9c" "latency"=1213650788
I0201 23:53:33.761460       1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=1.213718589 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-dwxhf0qv" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="test.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-dwxhf0qv/providers/Microsoft.Compute/disks/pvc-add98bb7-ddb5-4554-95ee-1310712bec9c" node="k8s-agentpool1-34418057-vmss000000" result_code="succeeded"
I0201 23:53:33.761481       1 utils.go:85] GRPC response: {"publish_context":{"LUN":"0"}}
I0201 23:53:33.761585       1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-add98bb7-ddb5-4554-95ee-1310712bec9c-k8s-agentpool1-34418057-vmss000000-attachment)
I0201 23:53:33.762731       1 azure_disk_utils.go:1103]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3.4" "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000000" "disk.csi.azure.com/request-id"="a24d3a2e-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-add98bb7-ddb5-4554-95ee-1310712bec9c" "latency"=12856842 "number_of_net_retry"=0 "number_of_retry"=0
I0201 23:53:34.602914       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0201 23:53:34.689830       1 azure_controller_vmss.go:122] azureDisk - update(kubetest-dwxhf0qv): vm(k8s-agentpool1-34418057-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-dwxhf0qv/providers/Microsoft.Compute/disks/pvc-add98bb7-ddb5-4554-95ee-1310712bec9c:AttachDiskOptions{diskName: "pvc-add98bb7-ddb5-4554-95ee-1310712bec9c", lun: 0}], %!s(*retry.Error=<nil>)) returned with %!v(MISSING)
I0201 23:53:34.689875       1 azure_controller_common.go:344] azuredisk - trying to attach disks to node k8s-agentpool1-34418057-vmss000000: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-dwxhf0qv/providers/Microsoft.Compute/disks/pvc-add98bb7-ddb5-4554-95ee-1310712bec9c:AttachDiskOptions{diskName: "pvc-add98bb7-ddb5-4554-95ee-1310712bec9c", lun: 0}]
I0201 23:53:36.615225       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0201 23:53:38.625736       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0201 23:53:40.635548       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0201 23:53:42.641129       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0201 23:53:44.649165       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
... skipping 6 lines ...
I0201 23:53:49.847041       1 workflow.go:155]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000000" "disk.csi.azure.com/request-id"="a24d3a2e-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).PublishVolume" "disk.csi.azure.com/volume-name"="pvc-add98bb7-ddb5-4554-95ee-1310712bec9c" "latency"=17286708049
I0201 23:53:49.847124       1 azure_disk_utils.go:1165]  "msg"="Skip update. No update needed." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).updateVolumeAttachmentWithResult" "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000000" "disk.csi.azure.com/request-id"="a24d3a2e-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-add98bb7-ddb5-4554-95ee-1310712bec9c"
I0201 23:53:49.847203       1 azure_disk_utils.go:1103]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).updateVolumeAttachmentWithResult" "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000000" "disk.csi.azure.com/request-id"="a24d3a2e-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-add98bb7-ddb5-4554-95ee-1310712bec9c" "latency"=79801 "number_of_net_retry"=0 "number_of_retry"=0
I0201 23:53:49.863312       1 azure_disk_utils.go:1103]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3.4" "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000000" "disk.csi.azure.com/request-id"="a24d3a2e-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-add98bb7-ddb5-4554-95ee-1310712bec9c" "latency"=16051976 "number_of_net_retry"=0 "number_of_retry"=0
I0201 23:53:49.863355       1 attach_detach.go:198]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000000" "disk.csi.azure.com/request-id"="a24d3a2e-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3" "disk.csi.azure.com/volume-name"="pvc-add98bb7-ddb5-4554-95ee-1310712bec9c" "latency"=17303030927
I0201 23:53:49.863382       1 workflow.go:155]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000000" "disk.csi.azure.com/request-id"="a24d3a2e-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach" "disk.csi.azure.com/volume-name"="pvc-add98bb7-ddb5-4554-95ee-1310712bec9c" "latency"=17309505799
I0201 23:53:49.864524       1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0201 23:53:49.864587       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000000" "disk.csi.azure.com/request-id"="a3066edb-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-add98bb7-ddb5-4554-95ee-1310712bec9c" "latency"=16102918389
I0201 23:53:49.864618       1 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).ControllerPublishVolume.func2" "disk.csi.azure.com/request-id"="a3066edb-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=16103067490
I0201 23:53:49.864659       1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=17.31691758 request="azuredisk_csi_driver_attach_volume_latency" resource_group="kubetest-dwxhf0qv" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="test.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-dwxhf0qv/providers/Microsoft.Compute/disks/pvc-add98bb7-ddb5-4554-95ee-1310712bec9c" node="k8s-agentpool1-34418057-vmss000000" result_code="succeeded"
I0201 23:53:49.864696       1 controllerserver_v2.go:264]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="a3066edb-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).ControllerPublishVolume.func2" "latency"=16103173291
I0201 23:53:50.676198       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0201 23:53:52.451934       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
... skipping 24 lines ...
I0201 23:53:55.568502       1 shared_state.go:489]  "msg"="Storing pod statefulset-azuredisk-0 and claim [] to podToClaimsMap map." 
I0201 23:53:55.601218       1 utils.go:78] GRPC call: /csi.v1.Controller/CreateVolume
I0201 23:53:55.601246       1 utils.go:79] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"requisite":[{"segments":{"topology.test.csi.azure.com/zone":""}}]},"capacity_range":{"required_bytes":10737418240},"name":"pvc-f8b6277d-3fb9-4e7e-807d-220f803033b3","parameters":{"csi.storage.k8s.io/pv/name":"pvc-f8b6277d-3fb9-4e7e-807d-220f803033b3","csi.storage.k8s.io/pvc/name":"persistent-storage-statefulset-azuredisk-0","csi.storage.k8s.io/pvc/namespace":"default","skuName":"StandardSSD_LRS"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]}
I0201 23:53:55.601458       1 crdprovisioner.go:308]  "msg"="Creating AzVolume CRI" "csi.storage.k8s.io/pv/name"="pvc-f8b6277d-3fb9-4e7e-807d-220f803033b3" "disk.csi.azure.com/request-id"="b00af052-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/volume-name"="pvc-f8b6277d-3fb9-4e7e-807d-220f803033b3"
I0201 23:53:55.614729       1 crdprovisioner.go:316]  "msg"="Successfully created AzVolume CRI" "csi.storage.k8s.io/pv/name"="pvc-f8b6277d-3fb9-4e7e-807d-220f803033b3" "disk.csi.azure.com/request-id"="b00af052-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/volume-name"="pvc-f8b6277d-3fb9-4e7e-807d-220f803033b3"
I0201 23:53:55.614753       1 conditionwatcher.go:99] Adding a condition function for azvolume (pvc-f8b6277d-3fb9-4e7e-807d-220f803033b3)
I0201 23:53:55.615973       1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0201 23:53:55.623761       1 azure_disk_utils.go:1103]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate" "csi.storage.k8s.io/pv/name"="pvc-f8b6277d-3fb9-4e7e-807d-220f803033b3" "disk.csi.azure.com/request-id"="b00af052-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-f8b6277d-3fb9-4e7e-807d-220f803033b3" "latency"=7507083 "number_of_net_retry"=0 "number_of_retry"=0
I0201 23:53:55.623797       1 azvolume.go:158]  "msg"="Creating Volume..." "csi.storage.k8s.io/pv/name"="pvc-f8b6277d-3fb9-4e7e-807d-220f803033b3" "disk.csi.azure.com/request-id"="b00af052-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/volume-name"="pvc-f8b6277d-3fb9-4e7e-807d-220f803033b3"
I0201 23:53:55.623958       1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0201 23:53:55.665082       1 util.go:124] Send.sendRequest got response with ContentLength 253, StatusCode 404 and responseBody length 253
I0201 23:53:55.665108       1 azure_diskclient.go:139] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-dwxhf0qv/providers/Microsoft.Compute/disks/pvc-f8b6277d-3fb9-4e7e-807d-220f803033b3, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-f8b6277d-3fb9-4e7e-807d-220f803033b3' under resource group 'kubetest-dwxhf0qv' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I0201 23:53:55.665166       1 cloudprovisioner.go:246] begin to create disk(pvc-f8b6277d-3fb9-4e7e-807d-220f803033b3) account type(StandardSSD_LRS) rg(kubetest-dwxhf0qv) location() size(10) selectedAvailabilityZone() maxShares(0)
I0201 23:53:55.811997       1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 5068
I0201 23:53:55.812207       1 azure_managedDiskController.go:94] azureDisk - creating new managed Name:pvc-f8b6277d-3fb9-4e7e-807d-220f803033b3 StorageAccountType:StandardSSD_LRS Size:10
I0201 23:53:56.700104       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0201 23:53:58.187385       1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1222
I0201 23:53:58.187575       1 azure_managedDiskController.go:280] azureDisk - created new MD Name:pvc-f8b6277d-3fb9-4e7e-807d-220f803033b3 StorageAccountType:StandardSSD_LRS Size:10
I0201 23:53:58.187639       1 cloudprovisioner.go:320]  "msg"="create disk(pvc-f8b6277d-3fb9-4e7e-807d-220f803033b3) account type(StandardSSD_LRS) rg(kubetest-dwxhf0qv) location() size(10) tags(map[kubernetes.io-created-for-pv-name:pvc-f8b6277d-3fb9-4e7e-807d-220f803033b3 kubernetes.io-created-for-pvc-name:persistent-storage-statefulset-azuredisk-0 kubernetes.io-created-for-pvc-namespace:default]) successfully" "csi.storage.k8s.io/pv/name"="pvc-f8b6277d-3fb9-4e7e-807d-220f803033b3" "disk.csi.azure.com/request-id"="b00af052-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/volume-name"="pvc-f8b6277d-3fb9-4e7e-807d-220f803033b3"
I0201 23:53:58.187675       1 cloudprovisioner.go:142]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-f8b6277d-3fb9-4e7e-807d-220f803033b3" "disk.csi.azure.com/request-id"="b00af052-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-f8b6277d-3fb9-4e7e-807d-220f803033b3" "latency"=2563790623
I0201 23:53:58.200729       1 azure_disk_utils.go:1103]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "csi.storage.k8s.io/pv/name"="pvc-f8b6277d-3fb9-4e7e-807d-220f803033b3" "disk.csi.azure.com/request-id"="b00af052-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-f8b6277d-3fb9-4e7e-807d-220f803033b3" "latency"=12989856 "number_of_net_retry"=0 "number_of_retry"=0
I0201 23:53:58.200774       1 azvolume.go:165]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-f8b6277d-3fb9-4e7e-807d-220f803033b3" "disk.csi.azure.com/request-id"="b00af052-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "disk.csi.azure.com/volume-name"="pvc-f8b6277d-3fb9-4e7e-807d-220f803033b3" "latency"=2576922981
I0201 23:53:58.200800       1 workflow.go:155]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-f8b6277d-3fb9-4e7e-807d-220f803033b3" "disk.csi.azure.com/request-id"="b00af052-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate" "disk.csi.azure.com/volume-name"="pvc-f8b6277d-3fb9-4e7e-807d-220f803033b3" "latency"=2584570265
I0201 23:53:58.202146       1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0201 23:53:58.202200       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).CreateVolume" "csi.storage.k8s.io/pv/name"="pvc-f8b6277d-3fb9-4e7e-807d-220f803033b3" "disk.csi.azure.com/request-id"="b00af052-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-f8b6277d-3fb9-4e7e-807d-220f803033b3" "latency"=2587391697
I0201 23:53:58.202228       1 crdprovisioner.go:228]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-f8b6277d-3fb9-4e7e-807d-220f803033b3" "disk.csi.azure.com/request-id"="b00af052-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-f8b6277d-3fb9-4e7e-807d-220f803033b3" "latency"=2600796445
I0201 23:53:58.202262       1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=2.600844445 request="azuredisk_csi_driver_controller_create_volume" resource_group="kubetest-dwxhf0qv" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="test.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-dwxhf0qv/providers/Microsoft.Compute/disks/pvc-f8b6277d-3fb9-4e7e-807d-220f803033b3" result_code="succeeded"
I0201 23:53:58.202279       1 utils.go:85] GRPC response: {"volume":{"accessible_topology":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"capacity_bytes":10737418240,"content_source":{"Type":{"Volume":{}}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-f8b6277d-3fb9-4e7e-807d-220f803033b3","csi.storage.k8s.io/pvc/name":"persistent-storage-statefulset-azuredisk-0","csi.storage.k8s.io/pvc/namespace":"default","requestedsizegib":"10","skuName":"StandardSSD_LRS"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-dwxhf0qv/providers/Microsoft.Compute/disks/pvc-f8b6277d-3fb9-4e7e-807d-220f803033b3"}}
I0201 23:53:58.695128       1 utils.go:78] GRPC call: /csi.v1.Controller/ControllerPublishVolume
I0201 23:53:58.695148       1 utils.go:79] GRPC request: {"node_id":"k8s-agentpool1-34418057-vmss000002","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-f8b6277d-3fb9-4e7e-807d-220f803033b3","csi.storage.k8s.io/pvc/name":"persistent-storage-statefulset-azuredisk-0","csi.storage.k8s.io/pvc/namespace":"default","requestedsizegib":"10","skuName":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1675295602525-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-dwxhf0qv/providers/Microsoft.Compute/disks/pvc-f8b6277d-3fb9-4e7e-807d-220f803033b3"}
I0201 23:53:58.695379       1 azure_disk_utils.go:1068]  "msg"="Label selector is: disk.csi.azure.com/node-name=k8s-agentpool1-34418057-vmss000002,disk.csi.azure.com/volume-name!=pvc-f8b6277d-3fb9-4e7e-807d-220f803033b3." "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000002" "disk.csi.azure.com/request-id"="b1e305b9-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-f8b6277d-3fb9-4e7e-807d-220f803033b3"
I0201 23:53:58.700581       1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-f8b6277d-3fb9-4e7e-807d-220f803033b3-k8s-agentpool1-34418057-vmss000002-attachment)
I0201 23:53:58.705454       1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0201 23:53:58.708326       1 azure_disk_utils.go:1103]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000002" "disk.csi.azure.com/request-id"="b1e305b9-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-f8b6277d-3fb9-4e7e-807d-220f803033b3" "latency"=7672693 "number_of_net_retry"=0 "number_of_retry"=0
I0201 23:53:58.708357       1 attach_detach.go:192]  "msg"="Attaching volume" "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000002" "disk.csi.azure.com/request-id"="b1e305b9-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-f8b6277d-3fb9-4e7e-807d-220f803033b3"
I0201 23:53:58.709117       1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0201 23:53:58.713928       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0201 23:53:58.733597       1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1222
I0201 23:53:58.733850       1 cloudprovisioner.go:405]  "msg"="GetDiskLun returned: -1. Initiating attaching volume \"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-dwxhf0qv/providers/Microsoft.Compute/disks/pvc-f8b6277d-3fb9-4e7e-807d-220f803033b3\" to node \"k8s-agentpool1-34418057-vmss000002\"." "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000002" "disk.csi.azure.com/request-id"="b1e305b9-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-f8b6277d-3fb9-4e7e-807d-220f803033b3"
I0201 23:53:58.733889       1 cloudprovisioner.go:424]  "msg"="Trying to attach volume \"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-dwxhf0qv/providers/Microsoft.Compute/disks/pvc-f8b6277d-3fb9-4e7e-807d-220f803033b3\" to node \"k8s-agentpool1-34418057-vmss000002\"." "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000002" "disk.csi.azure.com/request-id"="b1e305b9-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-f8b6277d-3fb9-4e7e-807d-220f803033b3"
I0201 23:53:59.734796       1 batch.go:224] "cloud-provider-azure: Delayed processing of batch due to start delay" type="batch" operation="attach_disk" key="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e|kubetest-dwxhf0qv|k8s-agentpool1-34418057-vmss000002" delay="1s"
I0201 23:53:59.734985       1 azure_controller_vmss.go:110] azureDisk - update(kubetest-dwxhf0qv): vm(k8s-agentpool1-34418057-vmss000002) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-dwxhf0qv/providers/Microsoft.Compute/disks/pvc-f8b6277d-3fb9-4e7e-807d-220f803033b3:AttachDiskOptions{diskName: "pvc-f8b6277d-3fb9-4e7e-807d-220f803033b3", lun: 0}])
I0201 23:53:59.746128       1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0201 23:53:59.746191       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000002" "disk.csi.azure.com/request-id"="b1e305b9-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-f8b6277d-3fb9-4e7e-807d-220f803033b3" "latency"=1045547015
I0201 23:53:59.746216       1 crdprovisioner.go:727]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000002" "disk.csi.azure.com/request-id"="b1e305b9-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLun" "disk.csi.azure.com/volume-name"="pvc-f8b6277d-3fb9-4e7e-807d-220f803033b3" "latency"=1045652816
I0201 23:53:59.746248       1 crdprovisioner.go:486]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000002" "disk.csi.azure.com/request-id"="b1e305b9-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).PublishVolume" "disk.csi.azure.com/volume-name"="pvc-f8b6277d-3fb9-4e7e-807d-220f803033b3" "latency"=1050975180
I0201 23:53:59.746286       1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=1.051017681 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-dwxhf0qv" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="test.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-dwxhf0qv/providers/Microsoft.Compute/disks/pvc-f8b6277d-3fb9-4e7e-807d-220f803033b3" node="k8s-agentpool1-34418057-vmss000002" result_code="succeeded"
I0201 23:53:59.746215       1 azure_disk_utils.go:1103]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3.4" "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000002" "disk.csi.azure.com/request-id"="b1e305b9-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-f8b6277d-3fb9-4e7e-807d-220f803033b3" "latency"=11268236 "number_of_net_retry"=0 "number_of_retry"=0
I0201 23:53:59.746304       1 utils.go:85] GRPC response: {"publish_context":{"LUN":"0"}}
I0201 23:53:59.746399       1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-f8b6277d-3fb9-4e7e-807d-220f803033b3-k8s-agentpool1-34418057-vmss000002-attachment)
I0201 23:53:59.890186       1 azure_controller_vmss.go:122] azureDisk - update(kubetest-dwxhf0qv): vm(k8s-agentpool1-34418057-vmss000002) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-dwxhf0qv/providers/Microsoft.Compute/disks/pvc-f8b6277d-3fb9-4e7e-807d-220f803033b3:AttachDiskOptions{diskName: "pvc-f8b6277d-3fb9-4e7e-807d-220f803033b3", lun: 0}], %!s(*retry.Error=<nil>)) returned with %!v(MISSING)
I0201 23:53:59.890232       1 azure_controller_common.go:344] azuredisk - trying to attach disks to node k8s-agentpool1-34418057-vmss000002: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-dwxhf0qv/providers/Microsoft.Compute/disks/pvc-f8b6277d-3fb9-4e7e-807d-220f803033b3:AttachDiskOptions{diskName: "pvc-f8b6277d-3fb9-4e7e-807d-220f803033b3", lun: 0}]
I0201 23:54:00.723320       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0201 23:54:02.730740       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0201 23:54:04.738980       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0201 23:54:06.748848       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0201 23:54:08.756156       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0201 23:54:10.024694       1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-dwxhf0qv, k8s-agentpool1-34418057-vmss, k8s-agentpool1-34418057-vmss000002) successfully
I0201 23:54:10.024727       1 azure_vmss_cache.go:313] updateCache(k8s-agentpool1-34418057-vmss, kubetest-dwxhf0qv, k8s-agentpool1-34418057-vmss000002) for cacheKey(kubetest-dwxhf0qv/k8s-agentpool1-34418057-vmss) updated successfully
I0201 23:54:10.024793       1 cloudprovisioner.go:448]  "msg"="attach operation successful: volume \"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-dwxhf0qv/providers/Microsoft.Compute/disks/pvc-f8b6277d-3fb9-4e7e-807d-220f803033b3\" attached to node \"k8s-agentpool1-34418057-vmss000002\"." "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000002" "disk.csi.azure.com/request-id"="b1e305b9-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-f8b6277d-3fb9-4e7e-807d-220f803033b3"
I0201 23:54:10.024837       1 cloudprovisioner.go:440]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000002" "disk.csi.azure.com/request-id"="b1e305b9-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).PublishVolume.func3" "disk.csi.azure.com/volume-name"="pvc-f8b6277d-3fb9-4e7e-807d-220f803033b3" "latency"=11290877476
I0201 23:54:10.024863       1 workflow.go:155]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000002" "disk.csi.azure.com/request-id"="b1e305b9-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).PublishVolume" "disk.csi.azure.com/volume-name"="pvc-f8b6277d-3fb9-4e7e-807d-220f803033b3" "latency"=11316456184
I0201 23:54:10.024985       1 azure_disk_utils.go:1165]  "msg"="Skip update. No update needed." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).updateVolumeAttachmentWithResult" "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000002" "disk.csi.azure.com/request-id"="b1e305b9-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-f8b6277d-3fb9-4e7e-807d-220f803033b3"
I0201 23:54:10.025044       1 azure_disk_utils.go:1103]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).updateVolumeAttachmentWithResult" "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000002" "disk.csi.azure.com/request-id"="b1e305b9-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-f8b6277d-3fb9-4e7e-807d-220f803033b3" "latency"=126702 "number_of_net_retry"=0 "number_of_retry"=0
I0201 23:54:10.062247       1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0201 23:54:10.062331       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000002" "disk.csi.azure.com/request-id"="b2836787-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-f8b6277d-3fb9-4e7e-807d-220f803033b3" "latency"=10315851772
I0201 23:54:10.062368       1 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).ControllerPublishVolume.func2" "disk.csi.azure.com/request-id"="b2836787-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=10315995173
I0201 23:54:10.062412       1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=11.367141056 request="azuredisk_csi_driver_attach_volume_latency" resource_group="kubetest-dwxhf0qv" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="test.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-dwxhf0qv/providers/Microsoft.Compute/disks/pvc-f8b6277d-3fb9-4e7e-807d-220f803033b3" node="k8s-agentpool1-34418057-vmss000002" result_code="succeeded"
I0201 23:54:10.062447       1 controllerserver_v2.go:264]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="b2836787-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).ControllerPublishVolume.func2" "latency"=10316088774
I0201 23:54:10.063065       1 azure_disk_utils.go:1103]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3.4" "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000002" "disk.csi.azure.com/request-id"="b1e305b9-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-f8b6277d-3fb9-4e7e-807d-220f803033b3" "latency"=37968018 "number_of_net_retry"=0 "number_of_retry"=0
I0201 23:54:10.063106       1 attach_detach.go:198]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000002" "disk.csi.azure.com/request-id"="b1e305b9-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3" "disk.csi.azure.com/volume-name"="pvc-f8b6277d-3fb9-4e7e-807d-220f803033b3" "latency"=11354711704
... skipping 26 lines ...
I0201 23:54:21.987490       1 shared_state.go:489]  "msg"="Storing pod statefulset-azuredisk-nonroot-0 and claim [] to podToClaimsMap map." 
I0201 23:54:22.000572       1 utils.go:78] GRPC call: /csi.v1.Controller/CreateVolume
I0201 23:54:22.000592       1 utils.go:79] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"requisite":[{"segments":{"topology.test.csi.azure.com/zone":""}}]},"capacity_range":{"required_bytes":10737418240},"name":"pvc-7ee71f68-c1f4-46fe-bd27-2b23e13b12d8","parameters":{"csi.storage.k8s.io/pv/name":"pvc-7ee71f68-c1f4-46fe-bd27-2b23e13b12d8","csi.storage.k8s.io/pvc/name":"persistent-storage-statefulset-azuredisk-nonroot-0","csi.storage.k8s.io/pvc/namespace":"default","skuName":"StandardSSD_LRS"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]}
I0201 23:54:22.000814       1 crdprovisioner.go:308]  "msg"="Creating AzVolume CRI" "csi.storage.k8s.io/pv/name"="pvc-7ee71f68-c1f4-46fe-bd27-2b23e13b12d8" "disk.csi.azure.com/request-id"="bfc72936-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/volume-name"="pvc-7ee71f68-c1f4-46fe-bd27-2b23e13b12d8"
I0201 23:54:22.010998       1 crdprovisioner.go:316]  "msg"="Successfully created AzVolume CRI" "csi.storage.k8s.io/pv/name"="pvc-7ee71f68-c1f4-46fe-bd27-2b23e13b12d8" "disk.csi.azure.com/request-id"="bfc72936-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/volume-name"="pvc-7ee71f68-c1f4-46fe-bd27-2b23e13b12d8"
I0201 23:54:22.011019       1 conditionwatcher.go:99] Adding a condition function for azvolume (pvc-7ee71f68-c1f4-46fe-bd27-2b23e13b12d8)
I0201 23:54:22.016889       1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0201 23:54:22.017107       1 azure_disk_utils.go:1103]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate" "csi.storage.k8s.io/pv/name"="pvc-7ee71f68-c1f4-46fe-bd27-2b23e13b12d8" "disk.csi.azure.com/request-id"="bfc72936-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-7ee71f68-c1f4-46fe-bd27-2b23e13b12d8" "latency"=6990678 "number_of_net_retry"=0 "number_of_retry"=0
I0201 23:54:22.017144       1 azvolume.go:158]  "msg"="Creating Volume..." "csi.storage.k8s.io/pv/name"="pvc-7ee71f68-c1f4-46fe-bd27-2b23e13b12d8" "disk.csi.azure.com/request-id"="bfc72936-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/volume-name"="pvc-7ee71f68-c1f4-46fe-bd27-2b23e13b12d8"
I0201 23:54:22.035277       1 util.go:124] Send.sendRequest got response with ContentLength 253, StatusCode 404 and responseBody length 253
I0201 23:54:22.035311       1 azure_diskclient.go:139] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-dwxhf0qv/providers/Microsoft.Compute/disks/pvc-7ee71f68-c1f4-46fe-bd27-2b23e13b12d8, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-7ee71f68-c1f4-46fe-bd27-2b23e13b12d8' under resource group 'kubetest-dwxhf0qv' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I0201 23:54:22.035373       1 cloudprovisioner.go:246] begin to create disk(pvc-7ee71f68-c1f4-46fe-bd27-2b23e13b12d8) account type(StandardSSD_LRS) rg(kubetest-dwxhf0qv) location() size(10) selectedAvailabilityZone() maxShares(0)
I0201 23:54:22.086161       1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 6663
I0201 23:54:22.086391       1 azure_managedDiskController.go:94] azureDisk - creating new managed Name:pvc-7ee71f68-c1f4-46fe-bd27-2b23e13b12d8 StorageAccountType:StandardSSD_LRS Size:10
I0201 23:54:22.452363       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0201 23:54:22.452450       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0201 23:54:22.452514       1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0201 23:54:22.453520       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0201 23:54:22.455761       1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync
I0201 23:54:22.818905       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0201 23:54:24.423597       1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1230
I0201 23:54:24.423844       1 azure_managedDiskController.go:280] azureDisk - created new MD Name:pvc-7ee71f68-c1f4-46fe-bd27-2b23e13b12d8 StorageAccountType:StandardSSD_LRS Size:10
I0201 23:54:24.423915       1 cloudprovisioner.go:320]  "msg"="create disk(pvc-7ee71f68-c1f4-46fe-bd27-2b23e13b12d8) account type(StandardSSD_LRS) rg(kubetest-dwxhf0qv) location() size(10) tags(map[kubernetes.io-created-for-pv-name:pvc-7ee71f68-c1f4-46fe-bd27-2b23e13b12d8 kubernetes.io-created-for-pvc-name:persistent-storage-statefulset-azuredisk-nonroot-0 kubernetes.io-created-for-pvc-namespace:default]) successfully" "csi.storage.k8s.io/pv/name"="pvc-7ee71f68-c1f4-46fe-bd27-2b23e13b12d8" "disk.csi.azure.com/request-id"="bfc72936-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/volume-name"="pvc-7ee71f68-c1f4-46fe-bd27-2b23e13b12d8"
I0201 23:54:24.423959       1 cloudprovisioner.go:142]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-7ee71f68-c1f4-46fe-bd27-2b23e13b12d8" "disk.csi.azure.com/request-id"="bfc72936-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-7ee71f68-c1f4-46fe-bd27-2b23e13b12d8" "latency"=2406723485
I0201 23:54:24.432863       1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0201 23:54:24.432920       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).CreateVolume" "csi.storage.k8s.io/pv/name"="pvc-7ee71f68-c1f4-46fe-bd27-2b23e13b12d8" "disk.csi.azure.com/request-id"="bfc72936-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-7ee71f68-c1f4-46fe-bd27-2b23e13b12d8" "latency"=2421846753
I0201 23:54:24.432956       1 crdprovisioner.go:228]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-7ee71f68-c1f4-46fe-bd27-2b23e13b12d8" "disk.csi.azure.com/request-id"="bfc72936-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-7ee71f68-c1f4-46fe-bd27-2b23e13b12d8" "latency"=2432159068
I0201 23:54:24.433003       1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=2.4322223689999998 request="azuredisk_csi_driver_controller_create_volume" resource_group="kubetest-dwxhf0qv" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="test.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-dwxhf0qv/providers/Microsoft.Compute/disks/pvc-7ee71f68-c1f4-46fe-bd27-2b23e13b12d8" result_code="succeeded"
I0201 23:54:24.433021       1 utils.go:85] GRPC response: {"volume":{"accessible_topology":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"capacity_bytes":10737418240,"content_source":{"Type":{"Volume":{}}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-7ee71f68-c1f4-46fe-bd27-2b23e13b12d8","csi.storage.k8s.io/pvc/name":"persistent-storage-statefulset-azuredisk-nonroot-0","csi.storage.k8s.io/pvc/namespace":"default","requestedsizegib":"10","skuName":"StandardSSD_LRS"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-dwxhf0qv/providers/Microsoft.Compute/disks/pvc-7ee71f68-c1f4-46fe-bd27-2b23e13b12d8"}}
I0201 23:54:24.436449       1 azure_disk_utils.go:1103]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "csi.storage.k8s.io/pv/name"="pvc-7ee71f68-c1f4-46fe-bd27-2b23e13b12d8" "disk.csi.azure.com/request-id"="bfc72936-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-7ee71f68-c1f4-46fe-bd27-2b23e13b12d8" "latency"=12430239 "number_of_net_retry"=0 "number_of_retry"=0
I0201 23:54:24.436483       1 azvolume.go:165]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-7ee71f68-c1f4-46fe-bd27-2b23e13b12d8" "disk.csi.azure.com/request-id"="bfc72936-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "disk.csi.azure.com/volume-name"="pvc-7ee71f68-c1f4-46fe-bd27-2b23e13b12d8" "latency"=2419297725
... skipping 2 lines ...
I0201 23:54:25.093299       1 utils.go:78] GRPC call: /csi.v1.Controller/ControllerPublishVolume
I0201 23:54:25.093325       1 utils.go:79] GRPC request: {"node_id":"k8s-agentpool1-34418057-vmss000001","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-7ee71f68-c1f4-46fe-bd27-2b23e13b12d8","csi.storage.k8s.io/pvc/name":"persistent-storage-statefulset-azuredisk-nonroot-0","csi.storage.k8s.io/pvc/namespace":"default","requestedsizegib":"10","skuName":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1675295602525-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-dwxhf0qv/providers/Microsoft.Compute/disks/pvc-7ee71f68-c1f4-46fe-bd27-2b23e13b12d8"}
I0201 23:54:25.093983       1 azure_disk_utils.go:1068]  "msg"="Label selector is: disk.csi.azure.com/node-name=k8s-agentpool1-34418057-vmss000001,disk.csi.azure.com/volume-name!=pvc-7ee71f68-c1f4-46fe-bd27-2b23e13b12d8." "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000001" "disk.csi.azure.com/request-id"="c19f1c5b-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-7ee71f68-c1f4-46fe-bd27-2b23e13b12d8"
I0201 23:54:25.100164       1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-7ee71f68-c1f4-46fe-bd27-2b23e13b12d8-k8s-agentpool1-34418057-vmss000001-attachment)
I0201 23:54:25.105280       1 azure_disk_utils.go:1103]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000001" "disk.csi.azure.com/request-id"="c19f1c5b-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-7ee71f68-c1f4-46fe-bd27-2b23e13b12d8" "latency"=4961755 "number_of_net_retry"=0 "number_of_retry"=0
I0201 23:54:25.105310       1 attach_detach.go:192]  "msg"="Attaching volume" "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000001" "disk.csi.azure.com/request-id"="c19f1c5b-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-7ee71f68-c1f4-46fe-bd27-2b23e13b12d8"
I0201 23:54:25.105903       1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0201 23:54:25.168253       1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1230
I0201 23:54:25.168485       1 cloudprovisioner.go:405]  "msg"="GetDiskLun returned: -1. Initiating attaching volume \"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-dwxhf0qv/providers/Microsoft.Compute/disks/pvc-7ee71f68-c1f4-46fe-bd27-2b23e13b12d8\" to node \"k8s-agentpool1-34418057-vmss000001\"." "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000001" "disk.csi.azure.com/request-id"="c19f1c5b-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-7ee71f68-c1f4-46fe-bd27-2b23e13b12d8"
I0201 23:54:25.168518       1 cloudprovisioner.go:424]  "msg"="Trying to attach volume \"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-dwxhf0qv/providers/Microsoft.Compute/disks/pvc-7ee71f68-c1f4-46fe-bd27-2b23e13b12d8\" to node \"k8s-agentpool1-34418057-vmss000001\"." "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000001" "disk.csi.azure.com/request-id"="c19f1c5b-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-7ee71f68-c1f4-46fe-bd27-2b23e13b12d8"
I0201 23:54:26.169591       1 batch.go:224] "cloud-provider-azure: Delayed processing of batch due to start delay" type="batch" operation="attach_disk" key="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e|kubetest-dwxhf0qv|k8s-agentpool1-34418057-vmss000001" delay="1s"
I0201 23:54:26.169704       1 azure_controller_vmss.go:110] azureDisk - update(kubetest-dwxhf0qv): vm(k8s-agentpool1-34418057-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-dwxhf0qv/providers/Microsoft.Compute/disks/pvc-7ee71f68-c1f4-46fe-bd27-2b23e13b12d8:AttachDiskOptions{diskName: "pvc-7ee71f68-c1f4-46fe-bd27-2b23e13b12d8", lun: 0}])
I0201 23:54:26.186898       1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0201 23:54:26.186973       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000001" "disk.csi.azure.com/request-id"="c19f1c5b-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-7ee71f68-c1f4-46fe-bd27-2b23e13b12d8" "latency"=1086722595
I0201 23:54:26.187014       1 crdprovisioner.go:727]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000001" "disk.csi.azure.com/request-id"="c19f1c5b-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLun" "disk.csi.azure.com/volume-name"="pvc-7ee71f68-c1f4-46fe-bd27-2b23e13b12d8" "latency"=1086858296
I0201 23:54:26.187051       1 crdprovisioner.go:486]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000001" "disk.csi.azure.com/request-id"="c19f1c5b-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).PublishVolume" "disk.csi.azure.com/volume-name"="pvc-7ee71f68-c1f4-46fe-bd27-2b23e13b12d8" "latency"=1093287168
I0201 23:54:26.187105       1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=1.093370068 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-dwxhf0qv" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="test.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-dwxhf0qv/providers/Microsoft.Compute/disks/pvc-7ee71f68-c1f4-46fe-bd27-2b23e13b12d8" node="k8s-agentpool1-34418057-vmss000001" result_code="succeeded"
I0201 23:54:26.187126       1 utils.go:85] GRPC response: {"publish_context":{"LUN":"0"}}
I0201 23:54:26.187254       1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-7ee71f68-c1f4-46fe-bd27-2b23e13b12d8-k8s-agentpool1-34418057-vmss000001-attachment)
I0201 23:54:26.187746       1 azure_disk_utils.go:1103]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3.4" "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000001" "disk.csi.azure.com/request-id"="c19f1c5b-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-7ee71f68-c1f4-46fe-bd27-2b23e13b12d8" "latency"=17900199 "number_of_net_retry"=0 "number_of_retry"=0
I0201 23:54:26.450383       1 azure_controller_vmss.go:122] azureDisk - update(kubetest-dwxhf0qv): vm(k8s-agentpool1-34418057-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-dwxhf0qv/providers/Microsoft.Compute/disks/pvc-7ee71f68-c1f4-46fe-bd27-2b23e13b12d8:AttachDiskOptions{diskName: "pvc-7ee71f68-c1f4-46fe-bd27-2b23e13b12d8", lun: 0}], %!s(*retry.Error=<nil>)) returned with %!v(MISSING)
I0201 23:54:26.450440       1 azure_controller_common.go:344] azuredisk - trying to attach disks to node k8s-agentpool1-34418057-vmss000001: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-dwxhf0qv/providers/Microsoft.Compute/disks/pvc-7ee71f68-c1f4-46fe-bd27-2b23e13b12d8:AttachDiskOptions{diskName: "pvc-7ee71f68-c1f4-46fe-bd27-2b23e13b12d8", lun: 0}]
I0201 23:54:26.836125       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0201 23:54:28.844063       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0201 23:54:30.852826       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0201 23:54:32.860534       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0201 23:54:34.869724       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
... skipping 3 lines ...
I0201 23:54:36.612998       1 cloudprovisioner.go:440]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000001" "disk.csi.azure.com/request-id"="c19f1c5b-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).PublishVolume.func3" "disk.csi.azure.com/volume-name"="pvc-7ee71f68-c1f4-46fe-bd27-2b23e13b12d8" "latency"=11444426813
I0201 23:54:36.613035       1 workflow.go:155]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000001" "disk.csi.azure.com/request-id"="c19f1c5b-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).PublishVolume" "disk.csi.azure.com/volume-name"="pvc-7ee71f68-c1f4-46fe-bd27-2b23e13b12d8" "latency"=11507650118
I0201 23:54:36.613096       1 azure_disk_utils.go:1165]  "msg"="Skip update. No update needed." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).updateVolumeAttachmentWithResult" "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000001" "disk.csi.azure.com/request-id"="c19f1c5b-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-7ee71f68-c1f4-46fe-bd27-2b23e13b12d8"
I0201 23:54:36.613136       1 azure_disk_utils.go:1103]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).updateVolumeAttachmentWithResult" "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000001" "disk.csi.azure.com/request-id"="c19f1c5b-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-7ee71f68-c1f4-46fe-bd27-2b23e13b12d8" "latency"=110501 "number_of_net_retry"=0 "number_of_retry"=0
I0201 23:54:36.625667       1 azure_disk_utils.go:1103]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3.4" "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000001" "disk.csi.azure.com/request-id"="c19f1c5b-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-7ee71f68-c1f4-46fe-bd27-2b23e13b12d8" "latency"=12488940 "number_of_net_retry"=0 "number_of_retry"=0
I0201 23:54:36.625707       1 attach_detach.go:198]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000001" "disk.csi.azure.com/request-id"="c19f1c5b-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3" "disk.csi.azure.com/volume-name"="pvc-7ee71f68-c1f4-46fe-bd27-2b23e13b12d8" "latency"=11520357360
I0201 23:54:36.625723       1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0201 23:54:36.625729       1 workflow.go:155]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000001" "disk.csi.azure.com/request-id"="c19f1c5b-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach" "disk.csi.azure.com/volume-name"="pvc-7ee71f68-c1f4-46fe-bd27-2b23e13b12d8" "latency"=11525454716
I0201 23:54:36.625776       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000001" "disk.csi.azure.com/request-id"="c245f4f2-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-7ee71f68-c1f4-46fe-bd27-2b23e13b12d8" "latency"=10438445819
I0201 23:54:36.625801       1 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).ControllerPublishVolume.func2" "disk.csi.azure.com/request-id"="c245f4f2-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=10438591020
I0201 23:54:36.625837       1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=11.53210479 request="azuredisk_csi_driver_attach_volume_latency" resource_group="kubetest-dwxhf0qv" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="test.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-dwxhf0qv/providers/Microsoft.Compute/disks/pvc-7ee71f68-c1f4-46fe-bd27-2b23e13b12d8" node="k8s-agentpool1-34418057-vmss000001" result_code="succeeded"
I0201 23:54:36.625863       1 controllerserver_v2.go:264]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="c245f4f2-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).ControllerPublishVolume.func2" "latency"=10438661722
I0201 23:54:36.878472       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
... skipping 26 lines ...
I0201 23:54:45.321140       1 utils.go:79] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"requisite":[{"segments":{"topology.test.csi.azure.com/zone":""}}]},"capacity_range":{"required_bytes":10737418240},"name":"pvc-493cdb4e-6be0-4e6c-908c-a4614b19b21e","parameters":{"csi.storage.k8s.io/pv/name":"pvc-493cdb4e-6be0-4e6c-908c-a4614b19b21e","csi.storage.k8s.io/pvc/name":"nginx-azuredisk-ephemeral-azuredisk01","csi.storage.k8s.io/pvc/namespace":"default","skuName":"StandardSSD_LRS"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]}
I0201 23:54:45.321388       1 crdprovisioner.go:308]  "msg"="Creating AzVolume CRI" "csi.storage.k8s.io/pv/name"="pvc-493cdb4e-6be0-4e6c-908c-a4614b19b21e" "disk.csi.azure.com/request-id"="cdad98bc-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/volume-name"="pvc-493cdb4e-6be0-4e6c-908c-a4614b19b21e"
I0201 23:54:45.339835       1 crdprovisioner.go:316]  "msg"="Successfully created AzVolume CRI" "csi.storage.k8s.io/pv/name"="pvc-493cdb4e-6be0-4e6c-908c-a4614b19b21e" "disk.csi.azure.com/request-id"="cdad98bc-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/volume-name"="pvc-493cdb4e-6be0-4e6c-908c-a4614b19b21e"
I0201 23:54:45.339857       1 conditionwatcher.go:99] Adding a condition function for azvolume (pvc-493cdb4e-6be0-4e6c-908c-a4614b19b21e)
I0201 23:54:45.349608       1 azure_disk_utils.go:1103]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate" "csi.storage.k8s.io/pv/name"="pvc-493cdb4e-6be0-4e6c-908c-a4614b19b21e" "disk.csi.azure.com/request-id"="cdad98bc-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-493cdb4e-6be0-4e6c-908c-a4614b19b21e" "latency"=11453928 "number_of_net_retry"=0 "number_of_retry"=0
I0201 23:54:45.349658       1 azvolume.go:158]  "msg"="Creating Volume..." "csi.storage.k8s.io/pv/name"="pvc-493cdb4e-6be0-4e6c-908c-a4614b19b21e" "disk.csi.azure.com/request-id"="cdad98bc-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/volume-name"="pvc-493cdb4e-6be0-4e6c-908c-a4614b19b21e"
I0201 23:54:45.349729       1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0201 23:54:45.367892       1 util.go:124] Send.sendRequest got response with ContentLength 253, StatusCode 404 and responseBody length 253
I0201 23:54:45.367916       1 azure_diskclient.go:139] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-dwxhf0qv/providers/Microsoft.Compute/disks/pvc-493cdb4e-6be0-4e6c-908c-a4614b19b21e, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-493cdb4e-6be0-4e6c-908c-a4614b19b21e' under resource group 'kubetest-dwxhf0qv' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I0201 23:54:45.367957       1 cloudprovisioner.go:246] begin to create disk(pvc-493cdb4e-6be0-4e6c-908c-a4614b19b21e) account type(StandardSSD_LRS) rg(kubetest-dwxhf0qv) location() size(10) selectedAvailabilityZone() maxShares(0)
I0201 23:54:45.404269       1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 8266
I0201 23:54:45.404512       1 azure_managedDiskController.go:94] azureDisk - creating new managed Name:pvc-493cdb4e-6be0-4e6c-908c-a4614b19b21e StorageAccountType:StandardSSD_LRS Size:10
I0201 23:54:46.922224       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0201 23:54:47.677289       1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1216
I0201 23:54:47.677481       1 azure_managedDiskController.go:280] azureDisk - created new MD Name:pvc-493cdb4e-6be0-4e6c-908c-a4614b19b21e StorageAccountType:StandardSSD_LRS Size:10
I0201 23:54:47.677537       1 cloudprovisioner.go:320]  "msg"="create disk(pvc-493cdb4e-6be0-4e6c-908c-a4614b19b21e) account type(StandardSSD_LRS) rg(kubetest-dwxhf0qv) location() size(10) tags(map[kubernetes.io-created-for-pv-name:pvc-493cdb4e-6be0-4e6c-908c-a4614b19b21e kubernetes.io-created-for-pvc-name:nginx-azuredisk-ephemeral-azuredisk01 kubernetes.io-created-for-pvc-namespace:default]) successfully" "csi.storage.k8s.io/pv/name"="pvc-493cdb4e-6be0-4e6c-908c-a4614b19b21e" "disk.csi.azure.com/request-id"="cdad98bc-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/volume-name"="pvc-493cdb4e-6be0-4e6c-908c-a4614b19b21e"
I0201 23:54:47.677581       1 cloudprovisioner.go:142]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-493cdb4e-6be0-4e6c-908c-a4614b19b21e" "disk.csi.azure.com/request-id"="cdad98bc-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-493cdb4e-6be0-4e6c-908c-a4614b19b21e" "latency"=2327838991
I0201 23:54:47.690498       1 azure_disk_utils.go:1103]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "csi.storage.k8s.io/pv/name"="pvc-493cdb4e-6be0-4e6c-908c-a4614b19b21e" "disk.csi.azure.com/request-id"="cdad98bc-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-493cdb4e-6be0-4e6c-908c-a4614b19b21e" "latency"=12864043 "number_of_net_retry"=0 "number_of_retry"=0
I0201 23:54:47.690586       1 azvolume.go:165]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-493cdb4e-6be0-4e6c-908c-a4614b19b21e" "disk.csi.azure.com/request-id"="cdad98bc-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "disk.csi.azure.com/volume-name"="pvc-493cdb4e-6be0-4e6c-908c-a4614b19b21e" "latency"=2340882835
I0201 23:54:47.690620       1 workflow.go:155]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-493cdb4e-6be0-4e6c-908c-a4614b19b21e" "disk.csi.azure.com/request-id"="cdad98bc-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate" "disk.csi.azure.com/volume-name"="pvc-493cdb4e-6be0-4e6c-908c-a4614b19b21e" "latency"=2352491865
I0201 23:54:47.691304       1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0201 23:54:47.691357       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).CreateVolume" "csi.storage.k8s.io/pv/name"="pvc-493cdb4e-6be0-4e6c-908c-a4614b19b21e" "disk.csi.azure.com/request-id"="cdad98bc-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-493cdb4e-6be0-4e6c-908c-a4614b19b21e" "latency"=2351450853
I0201 23:54:47.691388       1 crdprovisioner.go:228]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-493cdb4e-6be0-4e6c-908c-a4614b19b21e" "disk.csi.azure.com/request-id"="cdad98bc-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-493cdb4e-6be0-4e6c-908c-a4614b19b21e" "latency"=2370037859
I0201 23:54:47.691431       1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=2.370079261 request="azuredisk_csi_driver_controller_create_volume" resource_group="kubetest-dwxhf0qv" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="test.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-dwxhf0qv/providers/Microsoft.Compute/disks/pvc-493cdb4e-6be0-4e6c-908c-a4614b19b21e" result_code="succeeded"
I0201 23:54:47.691442       1 utils.go:85] GRPC response: {"volume":{"accessible_topology":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"capacity_bytes":10737418240,"content_source":{"Type":{"Volume":{}}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-493cdb4e-6be0-4e6c-908c-a4614b19b21e","csi.storage.k8s.io/pvc/name":"nginx-azuredisk-ephemeral-azuredisk01","csi.storage.k8s.io/pvc/namespace":"default","requestedsizegib":"10","skuName":"StandardSSD_LRS"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-dwxhf0qv/providers/Microsoft.Compute/disks/pvc-493cdb4e-6be0-4e6c-908c-a4614b19b21e"}}
I0201 23:54:48.348478       1 utils.go:78] GRPC call: /csi.v1.Controller/ControllerPublishVolume
I0201 23:54:48.348501       1 utils.go:79] GRPC request: {"node_id":"k8s-agentpool1-34418057-vmss000000","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-493cdb4e-6be0-4e6c-908c-a4614b19b21e","csi.storage.k8s.io/pvc/name":"nginx-azuredisk-ephemeral-azuredisk01","csi.storage.k8s.io/pvc/namespace":"default","requestedsizegib":"10","skuName":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1675295602525-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-dwxhf0qv/providers/Microsoft.Compute/disks/pvc-493cdb4e-6be0-4e6c-908c-a4614b19b21e"}
I0201 23:54:48.348776       1 azure_disk_utils.go:1068]  "msg"="Label selector is: disk.csi.azure.com/node-name=k8s-agentpool1-34418057-vmss000000,disk.csi.azure.com/volume-name!=pvc-493cdb4e-6be0-4e6c-908c-a4614b19b21e." "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000000" "disk.csi.azure.com/request-id"="cf7b8750-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-493cdb4e-6be0-4e6c-908c-a4614b19b21e"
I0201 23:54:48.354976       1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-493cdb4e-6be0-4e6c-908c-a4614b19b21e-k8s-agentpool1-34418057-vmss000000-attachment)
I0201 23:54:48.355512       1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0201 23:54:48.361480       1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0201 23:54:48.362063       1 azure_disk_utils.go:1103]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000000" "disk.csi.azure.com/request-id"="cf7b8750-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-493cdb4e-6be0-4e6c-908c-a4614b19b21e" "latency"=6890376 "number_of_net_retry"=0 "number_of_retry"=0
I0201 23:54:48.362099       1 attach_detach.go:192]  "msg"="Attaching volume" "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000000" "disk.csi.azure.com/request-id"="cf7b8750-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-493cdb4e-6be0-4e6c-908c-a4614b19b21e"
I0201 23:54:48.387422       1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1216
I0201 23:54:48.387640       1 cloudprovisioner.go:405]  "msg"="GetDiskLun returned: -1. Initiating attaching volume \"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-dwxhf0qv/providers/Microsoft.Compute/disks/pvc-493cdb4e-6be0-4e6c-908c-a4614b19b21e\" to node \"k8s-agentpool1-34418057-vmss000000\"." "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000000" "disk.csi.azure.com/request-id"="cf7b8750-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-493cdb4e-6be0-4e6c-908c-a4614b19b21e"
I0201 23:54:48.387672       1 cloudprovisioner.go:424]  "msg"="Trying to attach volume \"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-dwxhf0qv/providers/Microsoft.Compute/disks/pvc-493cdb4e-6be0-4e6c-908c-a4614b19b21e\" to node \"k8s-agentpool1-34418057-vmss000000\"." "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000000" "disk.csi.azure.com/request-id"="cf7b8750-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-493cdb4e-6be0-4e6c-908c-a4614b19b21e"
I0201 23:54:48.930564       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0201 23:54:49.387910       1 batch.go:224] "cloud-provider-azure: Delayed processing of batch due to start delay" type="batch" operation="attach_disk" key="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e|kubetest-dwxhf0qv|k8s-agentpool1-34418057-vmss000000" delay="1s"
I0201 23:54:49.388014       1 azure_controller_vmss.go:110] azureDisk - update(kubetest-dwxhf0qv): vm(k8s-agentpool1-34418057-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-dwxhf0qv/providers/Microsoft.Compute/disks/pvc-493cdb4e-6be0-4e6c-908c-a4614b19b21e:AttachDiskOptions{diskName: "pvc-493cdb4e-6be0-4e6c-908c-a4614b19b21e", lun: 1}])
I0201 23:54:49.397098       1 azure_disk_utils.go:1103]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3.4" "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000000" "disk.csi.azure.com/request-id"="cf7b8750-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-493cdb4e-6be0-4e6c-908c-a4614b19b21e" "latency"=8919599 "number_of_net_retry"=0 "number_of_retry"=0
I0201 23:54:49.397999       1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0201 23:54:49.398061       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000000" "disk.csi.azure.com/request-id"="cf7b8750-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-493cdb4e-6be0-4e6c-908c-a4614b19b21e" "latency"=1043022301
I0201 23:54:49.398095       1 crdprovisioner.go:727]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000000" "disk.csi.azure.com/request-id"="cf7b8750-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLun" "disk.csi.azure.com/volume-name"="pvc-493cdb4e-6be0-4e6c-908c-a4614b19b21e" "latency"=1043138801
I0201 23:54:49.398117       1 crdprovisioner.go:486]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000000" "disk.csi.azure.com/request-id"="cf7b8750-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).PublishVolume" "disk.csi.azure.com/volume-name"="pvc-493cdb4e-6be0-4e6c-908c-a4614b19b21e" "latency"=1049448872
I0201 23:54:49.398162       1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=1.049516173 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-dwxhf0qv" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="test.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-dwxhf0qv/providers/Microsoft.Compute/disks/pvc-493cdb4e-6be0-4e6c-908c-a4614b19b21e" node="k8s-agentpool1-34418057-vmss000000" result_code="succeeded"
I0201 23:54:49.398178       1 utils.go:85] GRPC response: {"publish_context":{"LUN":"1"}}
I0201 23:54:49.398298       1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-493cdb4e-6be0-4e6c-908c-a4614b19b21e-k8s-agentpool1-34418057-vmss000000-attachment)
I0201 23:54:49.562415       1 azure_controller_vmss.go:122] azureDisk - update(kubetest-dwxhf0qv): vm(k8s-agentpool1-34418057-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-dwxhf0qv/providers/Microsoft.Compute/disks/pvc-493cdb4e-6be0-4e6c-908c-a4614b19b21e:AttachDiskOptions{diskName: "pvc-493cdb4e-6be0-4e6c-908c-a4614b19b21e", lun: 1}], %!s(*retry.Error=<nil>)) returned with %!v(MISSING)
I0201 23:54:49.562461       1 azure_controller_common.go:344] azuredisk - trying to attach disks to node k8s-agentpool1-34418057-vmss000000: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-dwxhf0qv/providers/Microsoft.Compute/disks/pvc-493cdb4e-6be0-4e6c-908c-a4614b19b21e:AttachDiskOptions{diskName: "pvc-493cdb4e-6be0-4e6c-908c-a4614b19b21e", lun: 1}]
I0201 23:54:50.940024       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0201 23:54:52.453005       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0201 23:54:52.453035       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0201 23:54:52.453094       1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0201 23:54:52.454177       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0201 23:54:52.456333       1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync
I0201 23:54:52.946404       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0201 23:54:54.955367       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0201 23:54:56.963446       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0201 23:54:58.973036       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
... skipping 3 lines ...
I0201 23:55:04.690555       1 azure_vmss_cache.go:313] updateCache(k8s-agentpool1-34418057-vmss, kubetest-dwxhf0qv, k8s-agentpool1-34418057-vmss000000) for cacheKey(kubetest-dwxhf0qv/k8s-agentpool1-34418057-vmss) updated successfully
I0201 23:55:04.690625       1 cloudprovisioner.go:448]  "msg"="attach operation successful: volume \"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-dwxhf0qv/providers/Microsoft.Compute/disks/pvc-493cdb4e-6be0-4e6c-908c-a4614b19b21e\" attached to node \"k8s-agentpool1-34418057-vmss000000\"." "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000000" "disk.csi.azure.com/request-id"="cf7b8750-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-493cdb4e-6be0-4e6c-908c-a4614b19b21e"
I0201 23:55:04.690711       1 cloudprovisioner.go:440]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000000" "disk.csi.azure.com/request-id"="cf7b8750-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).PublishVolume.func3" "disk.csi.azure.com/volume-name"="pvc-493cdb4e-6be0-4e6c-908c-a4614b19b21e" "latency"=16302944239
I0201 23:55:04.690740       1 workflow.go:155]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000000" "disk.csi.azure.com/request-id"="cf7b8750-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).PublishVolume" "disk.csi.azure.com/volume-name"="pvc-493cdb4e-6be0-4e6c-908c-a4614b19b21e" "latency"=16328580624
I0201 23:55:04.690849       1 azure_disk_utils.go:1165]  "msg"="Skip update. No update needed." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).updateVolumeAttachmentWithResult" "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000000" "disk.csi.azure.com/request-id"="cf7b8750-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-493cdb4e-6be0-4e6c-908c-a4614b19b21e"
I0201 23:55:04.690895       1 azure_disk_utils.go:1103]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).updateVolumeAttachmentWithResult" "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000000" "disk.csi.azure.com/request-id"="cf7b8750-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-493cdb4e-6be0-4e6c-908c-a4614b19b21e" "latency"=105101 "number_of_net_retry"=0 "number_of_retry"=0
I0201 23:55:04.697813       1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0201 23:55:04.697876       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000000" "disk.csi.azure.com/request-id"="d01bae84-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-493cdb4e-6be0-4e6c-908c-a4614b19b21e" "latency"=15299499979
I0201 23:55:04.697918       1 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).ControllerPublishVolume.func2" "disk.csi.azure.com/request-id"="d01bae84-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=15299654780
I0201 23:55:04.697960       1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=16.349304755 request="azuredisk_csi_driver_attach_volume_latency" resource_group="kubetest-dwxhf0qv" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="test.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-dwxhf0qv/providers/Microsoft.Compute/disks/pvc-493cdb4e-6be0-4e6c-908c-a4614b19b21e" node="k8s-agentpool1-34418057-vmss000000" result_code="succeeded"
I0201 23:55:04.697986       1 controllerserver_v2.go:264]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="d01bae84-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).ControllerPublishVolume.func2" "latency"=15299737381
I0201 23:55:04.699696       1 azure_disk_utils.go:1103]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3.4" "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000000" "disk.csi.azure.com/request-id"="cf7b8750-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-493cdb4e-6be0-4e6c-908c-a4614b19b21e" "latency"=8765097 "number_of_net_retry"=0 "number_of_retry"=0
I0201 23:55:04.699731       1 attach_detach.go:198]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000000" "disk.csi.azure.com/request-id"="cf7b8750-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3" "disk.csi.azure.com/volume-name"="pvc-493cdb4e-6be0-4e6c-908c-a4614b19b21e" "latency"=16337588524
... skipping 35 lines ...
I0201 23:55:09.344562       1 utils.go:79] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"requisite":[{"segments":{"topology.test.csi.azure.com/zone":""}}]},"capacity_range":{"required_bytes":10737418240},"name":"pvc-e98dff13-e112-4307-ab64-b8f8e60741eb","parameters":{"csi.storage.k8s.io/pv/name":"pvc-e98dff13-e112-4307-ab64-b8f8e60741eb","csi.storage.k8s.io/pvc/name":"daemonset-azuredisk-ephemeral-g9fp6-azuredisk","csi.storage.k8s.io/pvc/namespace":"default","skuName":"StandardSSD_LRS"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]}
I0201 23:55:09.344760       1 crdprovisioner.go:308]  "msg"="Creating AzVolume CRI" "csi.storage.k8s.io/pv/name"="pvc-e98dff13-e112-4307-ab64-b8f8e60741eb" "disk.csi.azure.com/request-id"="dbff4602-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/volume-name"="pvc-e98dff13-e112-4307-ab64-b8f8e60741eb"
I0201 23:55:09.349923       1 crdprovisioner.go:316]  "msg"="Successfully created AzVolume CRI" "csi.storage.k8s.io/pv/name"="pvc-ea5fb163-7d4f-4ef9-845b-1feb05c75649" "disk.csi.azure.com/request-id"="dbfde925-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/volume-name"="pvc-ea5fb163-7d4f-4ef9-845b-1feb05c75649"
I0201 23:55:09.349950       1 conditionwatcher.go:99] Adding a condition function for azvolume (pvc-ea5fb163-7d4f-4ef9-845b-1feb05c75649)
I0201 23:55:09.350161       1 crdprovisioner.go:316]  "msg"="Successfully created AzVolume CRI" "csi.storage.k8s.io/pv/name"="pvc-b3bc4651-4d67-4ce9-aa67-71b6587a3a7e" "disk.csi.azure.com/request-id"="dbfeab21-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/volume-name"="pvc-b3bc4651-4d67-4ce9-aa67-71b6587a3a7e"
I0201 23:55:09.350174       1 conditionwatcher.go:99] Adding a condition function for azvolume (pvc-b3bc4651-4d67-4ce9-aa67-71b6587a3a7e)
I0201 23:55:09.353227       1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0201 23:55:09.354881       1 crdprovisioner.go:316]  "msg"="Successfully created AzVolume CRI" "csi.storage.k8s.io/pv/name"="pvc-e98dff13-e112-4307-ab64-b8f8e60741eb" "disk.csi.azure.com/request-id"="dbff4602-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/volume-name"="pvc-e98dff13-e112-4307-ab64-b8f8e60741eb"
I0201 23:55:09.354901       1 conditionwatcher.go:99] Adding a condition function for azvolume (pvc-e98dff13-e112-4307-ab64-b8f8e60741eb)
I0201 23:55:09.354908       1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0201 23:55:09.355030       1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0201 23:55:09.365192       1 azure_disk_utils.go:1103]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate" "csi.storage.k8s.io/pv/name"="pvc-ea5fb163-7d4f-4ef9-845b-1feb05c75649" "disk.csi.azure.com/request-id"="dbfde925-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-ea5fb163-7d4f-4ef9-845b-1feb05c75649" "latency"=11887431 "number_of_net_retry"=0 "number_of_retry"=0
I0201 23:55:09.365231       1 azvolume.go:158]  "msg"="Creating Volume..." "csi.storage.k8s.io/pv/name"="pvc-ea5fb163-7d4f-4ef9-845b-1feb05c75649" "disk.csi.azure.com/request-id"="dbfde925-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/volume-name"="pvc-ea5fb163-7d4f-4ef9-845b-1feb05c75649"
I0201 23:55:09.365978       1 azure_disk_utils.go:1103]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate" "csi.storage.k8s.io/pv/name"="pvc-b3bc4651-4d67-4ce9-aa67-71b6587a3a7e" "disk.csi.azure.com/request-id"="dbfeab21-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-b3bc4651-4d67-4ce9-aa67-71b6587a3a7e" "latency"=10735218 "number_of_net_retry"=0 "number_of_retry"=0
I0201 23:55:09.366003       1 azvolume.go:158]  "msg"="Creating Volume..." "csi.storage.k8s.io/pv/name"="pvc-b3bc4651-4d67-4ce9-aa67-71b6587a3a7e" "disk.csi.azure.com/request-id"="dbfeab21-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/volume-name"="pvc-b3bc4651-4d67-4ce9-aa67-71b6587a3a7e"
I0201 23:55:09.367309       1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0201 23:55:09.369518       1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0201 23:55:09.372044       1 azure_disk_utils.go:1103]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate" "csi.storage.k8s.io/pv/name"="pvc-e98dff13-e112-4307-ab64-b8f8e60741eb" "disk.csi.azure.com/request-id"="dbff4602-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-e98dff13-e112-4307-ab64-b8f8e60741eb" "latency"=16692683 "number_of_net_retry"=0 "number_of_retry"=0
I0201 23:55:09.372073       1 azvolume.go:158]  "msg"="Creating Volume..." "csi.storage.k8s.io/pv/name"="pvc-e98dff13-e112-4307-ab64-b8f8e60741eb" "disk.csi.azure.com/request-id"="dbff4602-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/volume-name"="pvc-e98dff13-e112-4307-ab64-b8f8e60741eb"
I0201 23:55:09.372195       1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0201 23:55:09.384385       1 util.go:124] Send.sendRequest got response with ContentLength 253, StatusCode 404 and responseBody length 253
I0201 23:55:09.384409       1 azure_diskclient.go:139] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-dwxhf0qv/providers/Microsoft.Compute/disks/pvc-ea5fb163-7d4f-4ef9-845b-1feb05c75649, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-ea5fb163-7d4f-4ef9-845b-1feb05c75649' under resource group 'kubetest-dwxhf0qv' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I0201 23:55:09.384455       1 cloudprovisioner.go:246] begin to create disk(pvc-ea5fb163-7d4f-4ef9-845b-1feb05c75649) account type(StandardSSD_LRS) rg(kubetest-dwxhf0qv) location() size(10) selectedAvailabilityZone() maxShares(0)
I0201 23:55:09.403664       1 util.go:124] Send.sendRequest got response with ContentLength 253, StatusCode 404 and responseBody length 253
I0201 23:55:09.403690       1 azure_diskclient.go:139] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-dwxhf0qv/providers/Microsoft.Compute/disks/pvc-e98dff13-e112-4307-ab64-b8f8e60741eb, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-e98dff13-e112-4307-ab64-b8f8e60741eb' under resource group 'kubetest-dwxhf0qv' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I0201 23:55:09.403735       1 cloudprovisioner.go:246] begin to create disk(pvc-e98dff13-e112-4307-ab64-b8f8e60741eb) account type(StandardSSD_LRS) rg(kubetest-dwxhf0qv) location() size(10) selectedAvailabilityZone() maxShares(0)
I0201 23:55:09.430401       1 util.go:124] Send.sendRequest got response with ContentLength 253, StatusCode 404 and responseBody length 253
I0201 23:55:09.430430       1 azure_diskclient.go:139] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-dwxhf0qv/providers/Microsoft.Compute/disks/pvc-b3bc4651-4d67-4ce9-aa67-71b6587a3a7e, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-b3bc4651-4d67-4ce9-aa67-71b6587a3a7e' under resource group 'kubetest-dwxhf0qv' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I0201 23:55:09.430488       1 cloudprovisioner.go:246] begin to create disk(pvc-b3bc4651-4d67-4ce9-aa67-71b6587a3a7e) account type(StandardSSD_LRS) rg(kubetest-dwxhf0qv) location() size(10) selectedAvailabilityZone() maxShares(0)
I0201 23:55:09.457180       1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 9855
I0201 23:55:09.457485       1 azure_managedDiskController.go:94] azureDisk - creating new managed Name:pvc-e98dff13-e112-4307-ab64-b8f8e60741eb StorageAccountType:StandardSSD_LRS Size:10
I0201 23:55:09.467611       1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 9855
I0201 23:55:09.467887       1 azure_managedDiskController.go:94] azureDisk - creating new managed Name:pvc-ea5fb163-7d4f-4ef9-845b-1feb05c75649 StorageAccountType:StandardSSD_LRS Size:10
I0201 23:55:09.515359       1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 9855
... skipping 3 lines ...
I0201 23:55:11.797424       1 azure_managedDiskController.go:280] azureDisk - created new MD Name:pvc-e98dff13-e112-4307-ab64-b8f8e60741eb StorageAccountType:StandardSSD_LRS Size:10
I0201 23:55:11.797482       1 cloudprovisioner.go:320]  "msg"="create disk(pvc-e98dff13-e112-4307-ab64-b8f8e60741eb) account type(StandardSSD_LRS) rg(kubetest-dwxhf0qv) location() size(10) tags(map[kubernetes.io-created-for-pv-name:pvc-e98dff13-e112-4307-ab64-b8f8e60741eb kubernetes.io-created-for-pvc-name:daemonset-azuredisk-ephemeral-g9fp6-azuredisk kubernetes.io-created-for-pvc-namespace:default]) successfully" "csi.storage.k8s.io/pv/name"="pvc-e98dff13-e112-4307-ab64-b8f8e60741eb" "disk.csi.azure.com/request-id"="dbff4602-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/volume-name"="pvc-e98dff13-e112-4307-ab64-b8f8e60741eb"
I0201 23:55:11.797518       1 cloudprovisioner.go:142]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-e98dff13-e112-4307-ab64-b8f8e60741eb" "disk.csi.azure.com/request-id"="dbff4602-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-e98dff13-e112-4307-ab64-b8f8e60741eb" "latency"=2425379082
I0201 23:55:11.809675       1 azure_disk_utils.go:1103]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "csi.storage.k8s.io/pv/name"="pvc-e98dff13-e112-4307-ab64-b8f8e60741eb" "disk.csi.azure.com/request-id"="dbff4602-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-e98dff13-e112-4307-ab64-b8f8e60741eb" "latency"=12097333 "number_of_net_retry"=0 "number_of_retry"=0
I0201 23:55:11.809715       1 azvolume.go:165]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-e98dff13-e112-4307-ab64-b8f8e60741eb" "disk.csi.azure.com/request-id"="dbff4602-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "disk.csi.azure.com/volume-name"="pvc-e98dff13-e112-4307-ab64-b8f8e60741eb" "latency"=2437605416
I0201 23:55:11.809732       1 workflow.go:155]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-e98dff13-e112-4307-ab64-b8f8e60741eb" "disk.csi.azure.com/request-id"="dbff4602-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate" "disk.csi.azure.com/volume-name"="pvc-e98dff13-e112-4307-ab64-b8f8e60741eb" "latency"=2454404800
I0201 23:55:11.814881       1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0201 23:55:11.814932       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).CreateVolume" "csi.storage.k8s.io/pv/name"="pvc-e98dff13-e112-4307-ab64-b8f8e60741eb" "disk.csi.azure.com/request-id"="dbff4602-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-e98dff13-e112-4307-ab64-b8f8e60741eb" "latency"=2459981762
I0201 23:55:11.814955       1 crdprovisioner.go:228]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-e98dff13-e112-4307-ab64-b8f8e60741eb" "disk.csi.azure.com/request-id"="dbff4602-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-e98dff13-e112-4307-ab64-b8f8e60741eb" "latency"=2470224175
I0201 23:55:11.814987       1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=2.470263775 request="azuredisk_csi_driver_controller_create_volume" resource_group="kubetest-dwxhf0qv" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="test.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-dwxhf0qv/providers/Microsoft.Compute/disks/pvc-e98dff13-e112-4307-ab64-b8f8e60741eb" result_code="succeeded"
I0201 23:55:11.815009       1 utils.go:85] GRPC response: {"volume":{"accessible_topology":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"capacity_bytes":10737418240,"content_source":{"Type":{"Volume":{}}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-e98dff13-e112-4307-ab64-b8f8e60741eb","csi.storage.k8s.io/pvc/name":"daemonset-azuredisk-ephemeral-g9fp6-azuredisk","csi.storage.k8s.io/pvc/namespace":"default","requestedsizegib":"10","skuName":"StandardSSD_LRS"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-dwxhf0qv/providers/Microsoft.Compute/disks/pvc-e98dff13-e112-4307-ab64-b8f8e60741eb"}}
I0201 23:55:11.876392       1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1225
I0201 23:55:11.876600       1 azure_managedDiskController.go:280] azureDisk - created new MD Name:pvc-b3bc4651-4d67-4ce9-aa67-71b6587a3a7e StorageAccountType:StandardSSD_LRS Size:10
... skipping 3 lines ...
I0201 23:55:11.895198       1 azure_managedDiskController.go:280] azureDisk - created new MD Name:pvc-ea5fb163-7d4f-4ef9-845b-1feb05c75649 StorageAccountType:StandardSSD_LRS Size:10
I0201 23:55:11.895245       1 cloudprovisioner.go:320]  "msg"="create disk(pvc-ea5fb163-7d4f-4ef9-845b-1feb05c75649) account type(StandardSSD_LRS) rg(kubetest-dwxhf0qv) location() size(10) tags(map[kubernetes.io-created-for-pv-name:pvc-ea5fb163-7d4f-4ef9-845b-1feb05c75649 kubernetes.io-created-for-pvc-name:daemonset-azuredisk-ephemeral-8h5jw-azuredisk kubernetes.io-created-for-pvc-namespace:default]) successfully" "csi.storage.k8s.io/pv/name"="pvc-ea5fb163-7d4f-4ef9-845b-1feb05c75649" "disk.csi.azure.com/request-id"="dbfde925-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/volume-name"="pvc-ea5fb163-7d4f-4ef9-845b-1feb05c75649"
I0201 23:55:11.895277       1 cloudprovisioner.go:142]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-ea5fb163-7d4f-4ef9-845b-1feb05c75649" "disk.csi.azure.com/request-id"="dbfde925-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-ea5fb163-7d4f-4ef9-845b-1feb05c75649" "latency"=2529972332
I0201 23:55:11.945083       1 azure_disk_utils.go:1103]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "csi.storage.k8s.io/pv/name"="pvc-ea5fb163-7d4f-4ef9-845b-1feb05c75649" "disk.csi.azure.com/request-id"="dbfde925-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-ea5fb163-7d4f-4ef9-845b-1feb05c75649" "latency"=49752547 "number_of_net_retry"=0 "number_of_retry"=0
I0201 23:55:11.945128       1 azvolume.go:165]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-ea5fb163-7d4f-4ef9-845b-1feb05c75649" "disk.csi.azure.com/request-id"="dbfde925-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "disk.csi.azure.com/volume-name"="pvc-ea5fb163-7d4f-4ef9-845b-1feb05c75649" "latency"=2579852981
I0201 23:55:11.945149       1 workflow.go:155]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-ea5fb163-7d4f-4ef9-845b-1feb05c75649" "disk.csi.azure.com/request-id"="dbfde925-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate" "disk.csi.azure.com/volume-name"="pvc-ea5fb163-7d4f-4ef9-845b-1feb05c75649" "latency"=2591868413
I0201 23:55:11.945955       1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0201 23:55:11.946026       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).CreateVolume" "csi.storage.k8s.io/pv/name"="pvc-ea5fb163-7d4f-4ef9-845b-1feb05c75649" "disk.csi.azure.com/request-id"="dbfde925-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-ea5fb163-7d4f-4ef9-845b-1feb05c75649" "latency"=2596002558
I0201 23:55:11.946068       1 crdprovisioner.go:228]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-ea5fb163-7d4f-4ef9-845b-1feb05c75649" "disk.csi.azure.com/request-id"="dbfde925-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-ea5fb163-7d4f-4ef9-845b-1feb05c75649" "latency"=2610260115
I0201 23:55:11.946111       1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=2.610308615 request="azuredisk_csi_driver_controller_create_volume" resource_group="kubetest-dwxhf0qv" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="test.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-dwxhf0qv/providers/Microsoft.Compute/disks/pvc-ea5fb163-7d4f-4ef9-845b-1feb05c75649" result_code="succeeded"
I0201 23:55:11.946131       1 utils.go:85] GRPC response: {"volume":{"accessible_topology":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"capacity_bytes":10737418240,"content_source":{"Type":{"Volume":{}}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-ea5fb163-7d4f-4ef9-845b-1feb05c75649","csi.storage.k8s.io/pvc/name":"daemonset-azuredisk-ephemeral-8h5jw-azuredisk","csi.storage.k8s.io/pvc/namespace":"default","requestedsizegib":"10","skuName":"StandardSSD_LRS"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-dwxhf0qv/providers/Microsoft.Compute/disks/pvc-ea5fb163-7d4f-4ef9-845b-1feb05c75649"}}
I0201 23:55:11.951645       1 azure_disk_utils.go:1103]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "csi.storage.k8s.io/pv/name"="pvc-b3bc4651-4d67-4ce9-aa67-71b6587a3a7e" "disk.csi.azure.com/request-id"="dbfeab21-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-b3bc4651-4d67-4ce9-aa67-71b6587a3a7e" "latency"=74872223 "number_of_net_retry"=0 "number_of_retry"=0
I0201 23:55:11.951688       1 azvolume.go:165]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-b3bc4651-4d67-4ce9-aa67-71b6587a3a7e" "disk.csi.azure.com/request-id"="dbfeab21-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "disk.csi.azure.com/volume-name"="pvc-b3bc4651-4d67-4ce9-aa67-71b6587a3a7e" "latency"=2585646845
I0201 23:55:11.951711       1 workflow.go:155]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-b3bc4651-4d67-4ce9-aa67-71b6587a3a7e" "disk.csi.azure.com/request-id"="dbfeab21-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate" "disk.csi.azure.com/volume-name"="pvc-b3bc4651-4d67-4ce9-aa67-71b6587a3a7e" "latency"=2596481764
I0201 23:55:11.951757       1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0201 23:55:11.951800       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).CreateVolume" "csi.storage.k8s.io/pv/name"="pvc-b3bc4651-4d67-4ce9-aa67-71b6587a3a7e" "disk.csi.azure.com/request-id"="dbfeab21-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-b3bc4651-4d67-4ce9-aa67-71b6587a3a7e" "latency"=2601580120
I0201 23:55:11.951890       1 crdprovisioner.go:228]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-b3bc4651-4d67-4ce9-aa67-71b6587a3a7e" "disk.csi.azure.com/request-id"="dbfeab21-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-b3bc4651-4d67-4ce9-aa67-71b6587a3a7e" "latency"=2611120725
I0201 23:55:11.951926       1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=2.6111649249999997 request="azuredisk_csi_driver_controller_create_volume" resource_group="kubetest-dwxhf0qv" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="test.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-dwxhf0qv/providers/Microsoft.Compute/disks/pvc-b3bc4651-4d67-4ce9-aa67-71b6587a3a7e" result_code="succeeded"
I0201 23:55:11.951943       1 utils.go:85] GRPC response: {"volume":{"accessible_topology":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"capacity_bytes":10737418240,"content_source":{"Type":{"Volume":{}}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-b3bc4651-4d67-4ce9-aa67-71b6587a3a7e","csi.storage.k8s.io/pvc/name":"daemonset-azuredisk-ephemeral-pnrrk-azuredisk","csi.storage.k8s.io/pvc/namespace":"default","requestedsizegib":"10","skuName":"StandardSSD_LRS"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-dwxhf0qv/providers/Microsoft.Compute/disks/pvc-b3bc4651-4d67-4ce9-aa67-71b6587a3a7e"}}
I0201 23:55:12.417739       1 utils.go:78] GRPC call: /csi.v1.Controller/ControllerPublishVolume
I0201 23:55:12.417761       1 utils.go:79] GRPC request: {"node_id":"k8s-agentpool1-34418057-vmss000002","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-e98dff13-e112-4307-ab64-b8f8e60741eb","csi.storage.k8s.io/pvc/name":"daemonset-azuredisk-ephemeral-g9fp6-azuredisk","csi.storage.k8s.io/pvc/namespace":"default","requestedsizegib":"10","skuName":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1675295602525-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-dwxhf0qv/providers/Microsoft.Compute/disks/pvc-e98dff13-e112-4307-ab64-b8f8e60741eb"}
... skipping 6 lines ...
I0201 23:55:12.422394       1 azure_disk_utils.go:1068]  "msg"="Label selector is: disk.csi.azure.com/node-name=k8s-agentpool1-34418057-vmss000000,disk.csi.azure.com/volume-name!=pvc-b3bc4651-4d67-4ce9-aa67-71b6587a3a7e." "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000000" "disk.csi.azure.com/request-id"="ddd4dfab-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-b3bc4651-4d67-4ce9-aa67-71b6587a3a7e"
I0201 23:55:12.427476       1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-e98dff13-e112-4307-ab64-b8f8e60741eb-k8s-agentpool1-34418057-vmss000002-attachment)
I0201 23:55:12.433989       1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-b3bc4651-4d67-4ce9-aa67-71b6587a3a7e-k8s-agentpool1-34418057-vmss000000-attachment)
I0201 23:55:12.438454       1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-ea5fb163-7d4f-4ef9-845b-1feb05c75649-k8s-agentpool1-34418057-vmss000001-attachment)
I0201 23:55:12.438564       1 azure_disk_utils.go:1103]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000002" "disk.csi.azure.com/request-id"="ddd43331-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-e98dff13-e112-4307-ab64-b8f8e60741eb" "latency"=11651029 "number_of_net_retry"=0 "number_of_retry"=0
I0201 23:55:12.438597       1 attach_detach.go:192]  "msg"="Attaching volume" "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000002" "disk.csi.azure.com/request-id"="ddd43331-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-e98dff13-e112-4307-ab64-b8f8e60741eb"
I0201 23:55:12.438679       1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0201 23:55:12.439554       1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0201 23:55:12.443975       1 azure_disk_utils.go:1103]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000000" "disk.csi.azure.com/request-id"="ddd4dfab-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-b3bc4651-4d67-4ce9-aa67-71b6587a3a7e" "latency"=12044232 "number_of_net_retry"=0 "number_of_retry"=0
I0201 23:55:12.444006       1 attach_detach.go:192]  "msg"="Attaching volume" "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000000" "disk.csi.azure.com/request-id"="ddd4dfab-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-b3bc4651-4d67-4ce9-aa67-71b6587a3a7e"
I0201 23:55:12.444251       1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0201 23:55:12.446991       1 azure_disk_utils.go:1103]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000001" "disk.csi.azure.com/request-id"="ddd4abb9-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-ea5fb163-7d4f-4ef9-845b-1feb05c75649" "latency"=8006688 "number_of_net_retry"=0 "number_of_retry"=0
I0201 23:55:12.447018       1 attach_detach.go:192]  "msg"="Attaching volume" "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000001" "disk.csi.azure.com/request-id"="ddd4abb9-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-ea5fb163-7d4f-4ef9-845b-1feb05c75649"
I0201 23:55:12.448253       1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0201 23:55:12.470620       1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1225
I0201 23:55:12.470852       1 cloudprovisioner.go:405]  "msg"="GetDiskLun returned: -1. Initiating attaching volume \"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-dwxhf0qv/providers/Microsoft.Compute/disks/pvc-ea5fb163-7d4f-4ef9-845b-1feb05c75649\" to node \"k8s-agentpool1-34418057-vmss000001\"." "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000001" "disk.csi.azure.com/request-id"="ddd4abb9-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-ea5fb163-7d4f-4ef9-845b-1feb05c75649"
I0201 23:55:12.470890       1 cloudprovisioner.go:424]  "msg"="Trying to attach volume \"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-dwxhf0qv/providers/Microsoft.Compute/disks/pvc-ea5fb163-7d4f-4ef9-845b-1feb05c75649\" to node \"k8s-agentpool1-34418057-vmss000001\"." "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000001" "disk.csi.azure.com/request-id"="ddd4abb9-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-ea5fb163-7d4f-4ef9-845b-1feb05c75649"
I0201 23:55:12.478989       1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1225
I0201 23:55:12.479196       1 cloudprovisioner.go:405]  "msg"="GetDiskLun returned: -1. Initiating attaching volume \"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-dwxhf0qv/providers/Microsoft.Compute/disks/pvc-e98dff13-e112-4307-ab64-b8f8e60741eb\" to node \"k8s-agentpool1-34418057-vmss000002\"." "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000002" "disk.csi.azure.com/request-id"="ddd43331-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-e98dff13-e112-4307-ab64-b8f8e60741eb"
I0201 23:55:12.479224       1 cloudprovisioner.go:424]  "msg"="Trying to attach volume \"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-dwxhf0qv/providers/Microsoft.Compute/disks/pvc-e98dff13-e112-4307-ab64-b8f8e60741eb\" to node \"k8s-agentpool1-34418057-vmss000002\"." "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000002" "disk.csi.azure.com/request-id"="ddd43331-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-e98dff13-e112-4307-ab64-b8f8e60741eb"
... skipping 3 lines ...
I0201 23:55:13.044950       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0201 23:55:13.471402       1 batch.go:224] "cloud-provider-azure: Delayed processing of batch due to start delay" type="batch" operation="attach_disk" key="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e|kubetest-dwxhf0qv|k8s-agentpool1-34418057-vmss000001" delay="1s"
I0201 23:55:13.471549       1 azure_controller_vmss.go:110] azureDisk - update(kubetest-dwxhf0qv): vm(k8s-agentpool1-34418057-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-dwxhf0qv/providers/Microsoft.Compute/disks/pvc-ea5fb163-7d4f-4ef9-845b-1feb05c75649:AttachDiskOptions{diskName: "pvc-ea5fb163-7d4f-4ef9-845b-1feb05c75649", lun: 1}])
I0201 23:55:13.480135       1 batch.go:224] "cloud-provider-azure: Delayed processing of batch due to start delay" type="batch" operation="attach_disk" key="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e|kubetest-dwxhf0qv|k8s-agentpool1-34418057-vmss000002" delay="1s"
I0201 23:55:13.480223       1 azure_controller_vmss.go:110] azureDisk - update(kubetest-dwxhf0qv): vm(k8s-agentpool1-34418057-vmss000002) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-dwxhf0qv/providers/Microsoft.Compute/disks/pvc-e98dff13-e112-4307-ab64-b8f8e60741eb:AttachDiskOptions{diskName: "pvc-e98dff13-e112-4307-ab64-b8f8e60741eb", lun: 1}])
I0201 23:55:13.480738       1 azure_disk_utils.go:1103]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3.4" "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000001" "disk.csi.azure.com/request-id"="ddd4abb9-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-ea5fb163-7d4f-4ef9-845b-1feb05c75649" "latency"=9059699 "number_of_net_retry"=0 "number_of_retry"=0
I0201 23:55:13.481772       1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0201 23:55:13.481824       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000001" "disk.csi.azure.com/request-id"="ddd4abb9-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-ea5fb163-7d4f-4ef9-845b-1feb05c75649" "latency"=1043308578
I0201 23:55:13.481860       1 crdprovisioner.go:727]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000001" "disk.csi.azure.com/request-id"="ddd4abb9-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLun" "disk.csi.azure.com/volume-name"="pvc-ea5fb163-7d4f-4ef9-845b-1feb05c75649" "latency"=1043412078
I0201 23:55:13.481894       1 crdprovisioner.go:486]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000001" "disk.csi.azure.com/request-id"="ddd4abb9-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).PublishVolume" "disk.csi.azure.com/volume-name"="pvc-ea5fb163-7d4f-4ef9-845b-1feb05c75649" "latency"=1060915171
I0201 23:55:13.481940       1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=1.060978972 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-dwxhf0qv" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="test.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-dwxhf0qv/providers/Microsoft.Compute/disks/pvc-ea5fb163-7d4f-4ef9-845b-1feb05c75649" node="k8s-agentpool1-34418057-vmss000001" result_code="succeeded"
I0201 23:55:13.481956       1 utils.go:85] GRPC response: {"publish_context":{"LUN":"1"}}
I0201 23:55:13.482057       1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-ea5fb163-7d4f-4ef9-845b-1feb05c75649-k8s-agentpool1-34418057-vmss000001-attachment)
... skipping 4 lines ...
I0201 23:55:13.487899       1 crdprovisioner.go:486]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000001" "disk.csi.azure.com/request-id"="de776e0f-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).PublishVolume" "disk.csi.azure.com/volume-name"="pvc-ea5fb163-7d4f-4ef9-845b-1feb05c75649" "latency"=264502
I0201 23:55:13.487939       1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=0.000329304 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-dwxhf0qv" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="test.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-dwxhf0qv/providers/Microsoft.Compute/disks/pvc-ea5fb163-7d4f-4ef9-845b-1feb05c75649" node="k8s-agentpool1-34418057-vmss000001" result_code="succeeded"
I0201 23:55:13.487954       1 utils.go:85] GRPC response: {"publish_context":{"LUN":"1"}}
I0201 23:55:13.488027       1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-ea5fb163-7d4f-4ef9-845b-1feb05c75649-k8s-agentpool1-34418057-vmss000001-attachment)
I0201 23:55:13.492429       1 batch.go:224] "cloud-provider-azure: Delayed processing of batch due to start delay" type="batch" operation="attach_disk" key="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e|kubetest-dwxhf0qv|k8s-agentpool1-34418057-vmss000000" delay="1s"
I0201 23:55:13.492525       1 azure_controller_vmss.go:110] azureDisk - update(kubetest-dwxhf0qv): vm(k8s-agentpool1-34418057-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-dwxhf0qv/providers/Microsoft.Compute/disks/pvc-b3bc4651-4d67-4ce9-aa67-71b6587a3a7e:AttachDiskOptions{diskName: "pvc-b3bc4651-4d67-4ce9-aa67-71b6587a3a7e", lun: 2}])
I0201 23:55:13.498776       1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0201 23:55:13.498804       1 azure_disk_utils.go:1103]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3.4" "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000002" "disk.csi.azure.com/request-id"="ddd43331-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-e98dff13-e112-4307-ab64-b8f8e60741eb" "latency"=18396603 "number_of_net_retry"=0 "number_of_retry"=0
I0201 23:55:13.498844       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000002" "disk.csi.azure.com/request-id"="ddd43331-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-e98dff13-e112-4307-ab64-b8f8e60741eb" "latency"=1071304586
I0201 23:55:13.498869       1 crdprovisioner.go:727]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000002" "disk.csi.azure.com/request-id"="ddd43331-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLun" "disk.csi.azure.com/volume-name"="pvc-e98dff13-e112-4307-ab64-b8f8e60741eb" "latency"=1071406186
I0201 23:55:13.498895       1 crdprovisioner.go:486]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000002" "disk.csi.azure.com/request-id"="ddd43331-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).PublishVolume" "disk.csi.azure.com/volume-name"="pvc-e98dff13-e112-4307-ab64-b8f8e60741eb" "latency"=1081005392
I0201 23:55:13.498941       1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=1.081067193 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-dwxhf0qv" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="test.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-dwxhf0qv/providers/Microsoft.Compute/disks/pvc-e98dff13-e112-4307-ab64-b8f8e60741eb" node="k8s-agentpool1-34418057-vmss000002" result_code="succeeded"
I0201 23:55:13.498959       1 utils.go:85] GRPC response: {"publish_context":{"LUN":"1"}}
I0201 23:55:13.499081       1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-e98dff13-e112-4307-ab64-b8f8e60741eb-k8s-agentpool1-34418057-vmss000002-attachment)
I0201 23:55:13.500482       1 azure_disk_utils.go:1103]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3.4" "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000000" "disk.csi.azure.com/request-id"="ddd4dfab-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-b3bc4651-4d67-4ce9-aa67-71b6587a3a7e" "latency"=7772486 "number_of_net_retry"=0 "number_of_retry"=0
I0201 23:55:13.501721       1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0201 23:55:13.501778       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000000" "disk.csi.azure.com/request-id"="ddd4dfab-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-b3bc4651-4d67-4ce9-aa67-71b6587a3a7e" "latency"=1067725046
I0201 23:55:13.501836       1 crdprovisioner.go:727]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000000" "disk.csi.azure.com/request-id"="ddd4dfab-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLun" "disk.csi.azure.com/volume-name"="pvc-b3bc4651-4d67-4ce9-aa67-71b6587a3a7e" "latency"=1067829048
I0201 23:55:13.501868       1 crdprovisioner.go:486]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000000" "disk.csi.azure.com/request-id"="ddd4dfab-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).PublishVolume" "disk.csi.azure.com/volume-name"="pvc-b3bc4651-4d67-4ce9-aa67-71b6587a3a7e" "latency"=1079558276
I0201 23:55:13.501915       1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=1.079617976 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-dwxhf0qv" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="test.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-dwxhf0qv/providers/Microsoft.Compute/disks/pvc-b3bc4651-4d67-4ce9-aa67-71b6587a3a7e" node="k8s-agentpool1-34418057-vmss000000" result_code="succeeded"
I0201 23:55:13.501946       1 utils.go:85] GRPC response: {"publish_context":{"LUN":"2"}}
I0201 23:55:13.502070       1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-b3bc4651-4d67-4ce9-aa67-71b6587a3a7e-k8s-agentpool1-34418057-vmss000000-attachment)
... skipping 2 lines ...
I0201 23:55:13.510132       1 azure_disk_utils.go:1068]  "msg"="Label selector is: disk.csi.azure.com/node-name=k8s-agentpool1-34418057-vmss000000,disk.csi.azure.com/volume-name!=pvc-b3bc4651-4d67-4ce9-aa67-71b6587a3a7e." "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000000" "disk.csi.azure.com/request-id"="de7ad6a7-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-b3bc4651-4d67-4ce9-aa67-71b6587a3a7e"
I0201 23:55:13.510223       1 crdprovisioner.go:727]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000000" "disk.csi.azure.com/request-id"="de7ad6a7-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLun" "disk.csi.azure.com/volume-name"="pvc-b3bc4651-4d67-4ce9-aa67-71b6587a3a7e" "latency"=29400
I0201 23:55:13.510247       1 crdprovisioner.go:486]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000000" "disk.csi.azure.com/request-id"="de7ad6a7-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).PublishVolume" "disk.csi.azure.com/volume-name"="pvc-b3bc4651-4d67-4ce9-aa67-71b6587a3a7e" "latency"=280203
I0201 23:55:13.510296       1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=0.000357204 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-dwxhf0qv" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="test.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-dwxhf0qv/providers/Microsoft.Compute/disks/pvc-b3bc4651-4d67-4ce9-aa67-71b6587a3a7e" node="k8s-agentpool1-34418057-vmss000000" result_code="succeeded"
I0201 23:55:13.510335       1 utils.go:85] GRPC response: {"publish_context":{"LUN":"2"}}
I0201 23:55:13.510449       1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-b3bc4651-4d67-4ce9-aa67-71b6587a3a7e-k8s-agentpool1-34418057-vmss000000-attachment)
I0201 23:55:13.718477       1 azure_controller_vmss.go:122] azureDisk - update(kubetest-dwxhf0qv): vm(k8s-agentpool1-34418057-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-dwxhf0qv/providers/Microsoft.Compute/disks/pvc-b3bc4651-4d67-4ce9-aa67-71b6587a3a7e:AttachDiskOptions{diskName: "pvc-b3bc4651-4d67-4ce9-aa67-71b6587a3a7e", lun: 2}], %!s(*retry.Error=<nil>)) returned with %!v(MISSING)
I0201 23:55:13.718525       1 azure_controller_common.go:344] azuredisk - trying to attach disks to node k8s-agentpool1-34418057-vmss000000: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-dwxhf0qv/providers/Microsoft.Compute/disks/pvc-b3bc4651-4d67-4ce9-aa67-71b6587a3a7e:AttachDiskOptions{diskName: "pvc-b3bc4651-4d67-4ce9-aa67-71b6587a3a7e", lun: 2}]
I0201 23:55:14.430684       1 azure_controller_vmss.go:122] azureDisk - update(kubetest-dwxhf0qv): vm(k8s-agentpool1-34418057-vmss000002) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-dwxhf0qv/providers/Microsoft.Compute/disks/pvc-e98dff13-e112-4307-ab64-b8f8e60741eb:AttachDiskOptions{diskName: "pvc-e98dff13-e112-4307-ab64-b8f8e60741eb", lun: 1}], %!s(*retry.Error=<nil>)) returned with %!v(MISSING)
I0201 23:55:14.430727       1 azure_controller_common.go:344] azuredisk - trying to attach disks to node k8s-agentpool1-34418057-vmss000002: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-dwxhf0qv/providers/Microsoft.Compute/disks/pvc-e98dff13-e112-4307-ab64-b8f8e60741eb:AttachDiskOptions{diskName: "pvc-e98dff13-e112-4307-ab64-b8f8e60741eb", lun: 1}]
I0201 23:55:14.585251       1 azure_controller_vmss.go:122] azureDisk - update(kubetest-dwxhf0qv): vm(k8s-agentpool1-34418057-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-dwxhf0qv/providers/Microsoft.Compute/disks/pvc-ea5fb163-7d4f-4ef9-845b-1feb05c75649:AttachDiskOptions{diskName: "pvc-ea5fb163-7d4f-4ef9-845b-1feb05c75649", lun: 1}], %!s(*retry.Error=<nil>)) returned with %!v(MISSING)
I0201 23:55:14.585294       1 azure_controller_common.go:344] azuredisk - trying to attach disks to node k8s-agentpool1-34418057-vmss000001: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-dwxhf0qv/providers/Microsoft.Compute/disks/pvc-ea5fb163-7d4f-4ef9-845b-1feb05c75649:AttachDiskOptions{diskName: "pvc-ea5fb163-7d4f-4ef9-845b-1feb05c75649", lun: 1}]
I0201 23:55:15.053061       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0201 23:55:17.066066       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0201 23:55:19.075486       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0201 23:55:21.083662       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0201 23:55:22.453464       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0201 23:55:22.453582       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0201 23:55:22.453670       1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0201 23:55:22.453793       1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0201 23:55:22.453812       1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0201 23:55:22.453835       1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0201 23:55:22.453858       1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0201 23:55:22.454808       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0201 23:55:22.456960       1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync
I0201 23:55:23.091391       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0201 23:55:25.100248       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0201 23:55:27.113161       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0201 23:55:28.953365       1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-dwxhf0qv, k8s-agentpool1-34418057-vmss, k8s-agentpool1-34418057-vmss000000) successfully
I0201 23:55:28.953392       1 azure_vmss_cache.go:313] updateCache(k8s-agentpool1-34418057-vmss, kubetest-dwxhf0qv, k8s-agentpool1-34418057-vmss000000) for cacheKey(kubetest-dwxhf0qv/k8s-agentpool1-34418057-vmss) updated successfully
I0201 23:55:28.953462       1 cloudprovisioner.go:448]  "msg"="attach operation successful: volume \"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-dwxhf0qv/providers/Microsoft.Compute/disks/pvc-b3bc4651-4d67-4ce9-aa67-71b6587a3a7e\" attached to node \"k8s-agentpool1-34418057-vmss000000\"." "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000000" "disk.csi.azure.com/request-id"="ddd4dfab-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-b3bc4651-4d67-4ce9-aa67-71b6587a3a7e"
I0201 23:55:28.953518       1 cloudprovisioner.go:440]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000000" "disk.csi.azure.com/request-id"="ddd4dfab-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).PublishVolume.func3" "disk.csi.azure.com/volume-name"="pvc-b3bc4651-4d67-4ce9-aa67-71b6587a3a7e" "latency"=16461813435
I0201 23:55:28.953546       1 workflow.go:155]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000000" "disk.csi.azure.com/request-id"="ddd4dfab-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).PublishVolume" "disk.csi.azure.com/volume-name"="pvc-b3bc4651-4d67-4ce9-aa67-71b6587a3a7e" "latency"=16509487560
I0201 23:55:28.953607       1 azure_disk_utils.go:1165]  "msg"="Skip update. No update needed." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).updateVolumeAttachmentWithResult" "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000000" "disk.csi.azure.com/request-id"="ddd4dfab-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-b3bc4651-4d67-4ce9-aa67-71b6587a3a7e"
I0201 23:55:28.953646       1 azure_disk_utils.go:1103]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).updateVolumeAttachmentWithResult" "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000000" "disk.csi.azure.com/request-id"="ddd4dfab-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-b3bc4651-4d67-4ce9-aa67-71b6587a3a7e" "latency"=100201 "number_of_net_retry"=0 "number_of_retry"=0
I0201 23:55:28.981192       1 azure_disk_utils.go:1103]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3.4" "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000000" "disk.csi.azure.com/request-id"="ddd4dfab-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-b3bc4651-4d67-4ce9-aa67-71b6587a3a7e" "latency"=27264000 "number_of_net_retry"=0 "number_of_retry"=0
I0201 23:55:28.981308       1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0201 23:55:28.981371       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000000" "disk.csi.azure.com/request-id"="de7ae75e-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-b3bc4651-4d67-4ce9-aa67-71b6587a3a7e" "latency"=15470800232
I0201 23:55:28.981378       1 attach_detach.go:198]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000000" "disk.csi.azure.com/request-id"="ddd4dfab-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3" "disk.csi.azure.com/volume-name"="pvc-b3bc4651-4d67-4ce9-aa67-71b6587a3a7e" "latency"=16537210364
I0201 23:55:28.981415       1 workflow.go:155]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000000" "disk.csi.azure.com/request-id"="ddd4dfab-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach" "disk.csi.azure.com/volume-name"="pvc-b3bc4651-4d67-4ce9-aa67-71b6587a3a7e" "latency"=16549504899
I0201 23:55:28.981471       1 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).ControllerPublishVolume.func2" "disk.csi.azure.com/request-id"="de7ae75e-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=15470988335
I0201 23:55:28.981507       1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=15.471560341 request="azuredisk_csi_driver_attach_volume_latency" resource_group="kubetest-dwxhf0qv" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="test.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-dwxhf0qv/providers/Microsoft.Compute/disks/pvc-b3bc4651-4d67-4ce9-aa67-71b6587a3a7e" node="k8s-agentpool1-34418057-vmss000000" result_code="succeeded"
I0201 23:55:28.981538       1 controllerserver_v2.go:264]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="de7ae75e-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).ControllerPublishVolume.func2" "latency"=15471133936
I0201 23:55:28.981583       1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0201 23:55:28.981628       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000000" "disk.csi.azure.com/request-id"="de79a114-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-b3bc4651-4d67-4ce9-aa67-71b6587a3a7e" "latency"=15479492528
I0201 23:55:28.981699       1 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).ControllerPublishVolume.func2" "disk.csi.azure.com/request-id"="de79a114-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=15479605430
I0201 23:55:28.981731       1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=16.559444209 request="azuredisk_csi_driver_attach_volume_latency" resource_group="kubetest-dwxhf0qv" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="test.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-dwxhf0qv/providers/Microsoft.Compute/disks/pvc-b3bc4651-4d67-4ce9-aa67-71b6587a3a7e" node="k8s-agentpool1-34418057-vmss000000" result_code="succeeded"
I0201 23:55:28.981797       1 controllerserver_v2.go:264]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="de79a114-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).ControllerPublishVolume.func2" "latency"=15479747731
I0201 23:55:29.122605       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0201 23:55:29.625848       1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-dwxhf0qv, k8s-agentpool1-34418057-vmss, k8s-agentpool1-34418057-vmss000002) successfully
... skipping 3 lines ...
I0201 23:55:29.626007       1 workflow.go:155]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000002" "disk.csi.azure.com/request-id"="ddd43331-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).PublishVolume" "disk.csi.azure.com/volume-name"="pvc-e98dff13-e112-4307-ab64-b8f8e60741eb" "latency"=17187349609
I0201 23:55:29.626151       1 azure_disk_utils.go:1165]  "msg"="Skip update. No update needed." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).updateVolumeAttachmentWithResult" "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000002" "disk.csi.azure.com/request-id"="ddd43331-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-e98dff13-e112-4307-ab64-b8f8e60741eb"
I0201 23:55:29.626201       1 azure_disk_utils.go:1103]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).updateVolumeAttachmentWithResult" "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000002" "disk.csi.azure.com/request-id"="ddd43331-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-e98dff13-e112-4307-ab64-b8f8e60741eb" "latency"=136301 "number_of_net_retry"=0 "number_of_retry"=0
I0201 23:55:29.634919       1 azure_disk_utils.go:1103]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3.4" "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000002" "disk.csi.azure.com/request-id"="ddd43331-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-e98dff13-e112-4307-ab64-b8f8e60741eb" "latency"=8672696 "number_of_net_retry"=0 "number_of_retry"=0
I0201 23:55:29.634956       1 attach_detach.go:198]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000002" "disk.csi.azure.com/request-id"="ddd43331-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3" "disk.csi.azure.com/volume-name"="pvc-e98dff13-e112-4307-ab64-b8f8e60741eb" "latency"=17196315907
I0201 23:55:29.635026       1 workflow.go:155]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000002" "disk.csi.azure.com/request-id"="ddd43331-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach" "disk.csi.azure.com/volume-name"="pvc-e98dff13-e112-4307-ab64-b8f8e60741eb" "latency"=17208108738
I0201 23:55:29.635927       1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0201 23:55:29.635990       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000002" "disk.csi.azure.com/request-id"="de792bf7-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-e98dff13-e112-4307-ab64-b8f8e60741eb" "latency"=16136835953
I0201 23:55:29.636019       1 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).ControllerPublishVolume.func2" "disk.csi.azure.com/request-id"="de792bf7-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=16136967054
I0201 23:55:29.636047       1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=17.218178248 request="azuredisk_csi_driver_attach_volume_latency" resource_group="kubetest-dwxhf0qv" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="test.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-dwxhf0qv/providers/Microsoft.Compute/disks/pvc-e98dff13-e112-4307-ab64-b8f8e60741eb" node="k8s-agentpool1-34418057-vmss000002" result_code="succeeded"
I0201 23:55:29.636071       1 controllerserver_v2.go:264]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="de792bf7-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).ControllerPublishVolume.func2" "latency"=16137023355
I0201 23:55:29.768830       1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-dwxhf0qv, k8s-agentpool1-34418057-vmss, k8s-agentpool1-34418057-vmss000001) successfully
I0201 23:55:29.768861       1 azure_vmss_cache.go:313] updateCache(k8s-agentpool1-34418057-vmss, kubetest-dwxhf0qv, k8s-agentpool1-34418057-vmss000001) for cacheKey(kubetest-dwxhf0qv/k8s-agentpool1-34418057-vmss) updated successfully
I0201 23:55:29.768950       1 cloudprovisioner.go:448]  "msg"="attach operation successful: volume \"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-dwxhf0qv/providers/Microsoft.Compute/disks/pvc-ea5fb163-7d4f-4ef9-845b-1feb05c75649\" attached to node \"k8s-agentpool1-34418057-vmss000001\"." "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000001" "disk.csi.azure.com/request-id"="ddd4abb9-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-ea5fb163-7d4f-4ef9-845b-1feb05c75649"
I0201 23:55:29.768982       1 cloudprovisioner.go:440]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000001" "disk.csi.azure.com/request-id"="ddd4abb9-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).PublishVolume.func3" "disk.csi.azure.com/volume-name"="pvc-ea5fb163-7d4f-4ef9-845b-1feb05c75649" "latency"=17298048825
I0201 23:55:29.769067       1 workflow.go:155]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000001" "disk.csi.azure.com/request-id"="ddd4abb9-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).PublishVolume" "disk.csi.azure.com/volume-name"="pvc-ea5fb163-7d4f-4ef9-845b-1feb05c75649" "latency"=17321958888
I0201 23:55:29.769173       1 azure_disk_utils.go:1165]  "msg"="Skip update. No update needed." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).updateVolumeAttachmentWithResult" "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000001" "disk.csi.azure.com/request-id"="ddd4abb9-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-ea5fb163-7d4f-4ef9-845b-1feb05c75649"
I0201 23:55:29.769234       1 azure_disk_utils.go:1103]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).updateVolumeAttachmentWithResult" "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000001" "disk.csi.azure.com/request-id"="ddd4abb9-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-ea5fb163-7d4f-4ef9-845b-1feb05c75649" "latency"=181502 "number_of_net_retry"=0 "number_of_retry"=0
I0201 23:55:29.776363       1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0201 23:55:29.776413       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000001" "disk.csi.azure.com/request-id"="de777bd2-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-ea5fb163-7d4f-4ef9-845b-1feb05c75649" "latency"=16288312817
I0201 23:55:29.776437       1 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).ControllerPublishVolume.func2" "disk.csi.azure.com/request-id"="de777bd2-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=16288445518
I0201 23:55:29.776374       1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0201 23:55:29.776479       1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=16.288859524 request="azuredisk_csi_driver_attach_volume_latency" resource_group="kubetest-dwxhf0qv" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="test.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-dwxhf0qv/providers/Microsoft.Compute/disks/pvc-ea5fb163-7d4f-4ef9-845b-1feb05c75649" node="k8s-agentpool1-34418057-vmss000001" result_code="succeeded"
I0201 23:55:29.776505       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000001" "disk.csi.azure.com/request-id"="de7692fd-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-ea5fb163-7d4f-4ef9-845b-1feb05c75649" "latency"=16294382784
I0201 23:55:29.776511       1 controllerserver_v2.go:264]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="de777bd2-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).ControllerPublishVolume.func2" "latency"=16288525720
I0201 23:55:29.776534       1 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).ControllerPublishVolume.func2" "disk.csi.azure.com/request-id"="de7692fd-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=16294497885
I0201 23:55:29.776555       1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=17.355606359 request="azuredisk_csi_driver_attach_volume_latency" resource_group="kubetest-dwxhf0qv" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="test.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-dwxhf0qv/providers/Microsoft.Compute/disks/pvc-ea5fb163-7d4f-4ef9-845b-1feb05c75649" node="k8s-agentpool1-34418057-vmss000001" result_code="succeeded"
I0201 23:55:29.776570       1 controllerserver_v2.go:264]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="de7692fd-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).ControllerPublishVolume.func2" "latency"=16294552786
... skipping 62 lines ...
I0201 23:56:11.346488       1 utils.go:78] GRPC call: /csi.v1.Controller/ControllerUnpublishVolume
I0201 23:56:11.346507       1 utils.go:79] GRPC request: {"node_id":"k8s-agentpool1-34418057-vmss000000","volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-dwxhf0qv/providers/Microsoft.Compute/disks/pvc-add98bb7-ddb5-4554-95ee-1310712bec9c"}
I0201 23:56:11.346698       1 crdprovisioner.go:927]  "msg"="Requesting AzVolumeAttachment (pvc-add98bb7-ddb5-4554-95ee-1310712bec9c-k8s-agentpool1-34418057-vmss000000-attachment) detachment" "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000000" "disk.csi.azure.com/request-id"="a24d3a2e-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-add98bb7-ddb5-4554-95ee-1310712bec9c"
I0201 23:56:11.353775       1 replica.go:160]  "msg"="Garbage collection of AzVolumeAttachments for AzVolume (pvc-add98bb7-ddb5-4554-95ee-1310712bec9c) scheduled in 5m0s." "disk.csi.azure.com/request-id"="00f5199c-a28c-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/volume-name"="pvc-add98bb7-ddb5-4554-95ee-1310712bec9c"
I0201 23:56:11.354477       1 azure_disk_utils.go:1103]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).detachVolume" "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000000" "disk.csi.azure.com/request-id"="a24d3a2e-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-add98bb7-ddb5-4554-95ee-1310712bec9c" "latency"=7733685 "number_of_net_retry"=0 "number_of_retry"=0
I0201 23:56:11.354523       1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-add98bb7-ddb5-4554-95ee-1310712bec9c-k8s-agentpool1-34418057-vmss000000-attachment)
I0201 23:56:11.360081       1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0201 23:56:11.360083       1 azure_disk_utils.go:1103]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach" "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000000" "disk.csi.azure.com/request-id"="a24d3a2e-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-add98bb7-ddb5-4554-95ee-1310712bec9c" "latency"=6245669 "number_of_net_retry"=0 "number_of_retry"=0
I0201 23:56:11.360117       1 attach_detach.go:340]  "msg"="Detaching volume" "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000000" "disk.csi.azure.com/request-id"="a24d3a2e-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-add98bb7-ddb5-4554-95ee-1310712bec9c"
I0201 23:56:11.360206       1 cloudprovisioner.go:481]  "msg"="Trying to detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-dwxhf0qv/providers/Microsoft.Compute/disks/pvc-add98bb7-ddb5-4554-95ee-1310712bec9c from node k8s-agentpool1-34418057-vmss000000" "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000000" "disk.csi.azure.com/request-id"="a24d3a2e-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-add98bb7-ddb5-4554-95ee-1310712bec9c"
I0201 23:56:12.361037       1 batch.go:224] "cloud-provider-azure: Delayed processing of batch due to start delay" type="batch" operation="detach_disk" key="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e|kubetest-dwxhf0qv|k8s-agentpool1-34418057-vmss000000" delay="1s"
I0201 23:56:12.361086       1 azure_controller_common.go:461] azuredisk - trying to detach disks from node k8s-agentpool1-34418057-vmss000000: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-dwxhf0qv/providers/Microsoft.Compute/disks/pvc-add98bb7-ddb5-4554-95ee-1310712bec9c:pvc-add98bb7-ddb5-4554-95ee-1310712bec9c]
I0201 23:56:12.361134       1 azure_controller_vmss.go:193] azureDisk - detach disk: name pvc-add98bb7-ddb5-4554-95ee-1310712bec9c uri /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-dwxhf0qv/providers/Microsoft.Compute/disks/pvc-add98bb7-ddb5-4554-95ee-1310712bec9c
... skipping 2 lines ...
I0201 23:56:15.343145       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0201 23:56:17.355102       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0201 23:56:19.365140       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0201 23:56:21.371908       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0201 23:56:22.455380       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0201 23:56:22.455391       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0201 23:56:22.455549       1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0201 23:56:22.456492       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0201 23:56:22.458641       1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync
I0201 23:56:23.378903       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0201 23:56:25.389628       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0201 23:56:27.399013       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0201 23:56:29.408443       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
... skipping 4 lines ...
I0201 23:56:32.740151       1 azure_controller_common.go:473] azuredisk - successfully detached disks from node k8s-agentpool1-34418057-vmss000000: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-dwxhf0qv/providers/Microsoft.Compute/disks/pvc-add98bb7-ddb5-4554-95ee-1310712bec9c:pvc-add98bb7-ddb5-4554-95ee-1310712bec9c]
I0201 23:56:32.740197       1 azure_controller_common.go:434] azureDisk - detach disk(pvc-add98bb7-ddb5-4554-95ee-1310712bec9c, /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-dwxhf0qv/providers/Microsoft.Compute/disks/pvc-add98bb7-ddb5-4554-95ee-1310712bec9c) succeeded
I0201 23:56:32.740241       1 cloudprovisioner.go:491]  "msg"="detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-dwxhf0qv/providers/Microsoft.Compute/disks/pvc-add98bb7-ddb5-4554-95ee-1310712bec9c from node k8s-agentpool1-34418057-vmss000000 successfully" "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000000" "disk.csi.azure.com/request-id"="a24d3a2e-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-add98bb7-ddb5-4554-95ee-1310712bec9c"
I0201 23:56:32.740287       1 cloudprovisioner.go:471]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000000" "disk.csi.azure.com/request-id"="a24d3a2e-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).UnpublishVolume" "disk.csi.azure.com/volume-name"="pvc-add98bb7-ddb5-4554-95ee-1310712bec9c" "latency"=21380105554
I0201 23:56:32.755641       1 attach_detach.go:346]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000000" "disk.csi.azure.com/request-id"="a24d3a2e-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach.func3" "disk.csi.azure.com/volume-name"="pvc-add98bb7-ddb5-4554-95ee-1310712bec9c" "latency"=21395449324
I0201 23:56:32.755717       1 workflow.go:155]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000000" "disk.csi.azure.com/request-id"="a24d3a2e-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach" "disk.csi.azure.com/volume-name"="pvc-add98bb7-ddb5-4554-95ee-1310712bec9c" "latency"=21401877094
I0201 23:56:32.756281       1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0201 23:56:32.764348       1 azure_disk_utils.go:1103]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).removeFinalizer" "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000000" "disk.csi.azure.com/request-id"="a24d3a2e-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-add98bb7-ddb5-4554-95ee-1310712bec9c" "latency"=12235035 "number_of_net_retry"=0 "number_of_retry"=0
I0201 23:56:32.764396       1 attach_detach.go:379]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000000" "disk.csi.azure.com/request-id"="a24d3a2e-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).removeFinalizer" "disk.csi.azure.com/volume-name"="pvc-add98bb7-ddb5-4554-95ee-1310712bec9c" "latency"=12297335
I0201 23:56:32.764418       1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0201 23:56:32.764471       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForDetach" "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000000" "disk.csi.azure.com/request-id"="a24d3a2e-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-add98bb7-ddb5-4554-95ee-1310712bec9c" "latency"=21409890283
I0201 23:56:32.764519       1 crdprovisioner.go:938]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).detachVolume" "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000000" "disk.csi.azure.com/request-id"="a24d3a2e-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForDetach" "disk.csi.azure.com/volume-name"="pvc-add98bb7-ddb5-4554-95ee-1310712bec9c" "latency"=21409995484
I0201 23:56:32.764548       1 crdprovisioner.go:836]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000000" "disk.csi.azure.com/request-id"="00f40521-a28c-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).UnpublishVolume" "disk.csi.azure.com/volume-name"="pvc-add98bb7-ddb5-4554-95ee-1310712bec9c" "latency"=21417863971
I0201 23:56:32.764581       1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=21.417950671 request="azuredisk_csi_driver_controller_unpublish_volume" resource_group="kubetest-dwxhf0qv" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="test.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-dwxhf0qv/providers/Microsoft.Compute/disks/pvc-add98bb7-ddb5-4554-95ee-1310712bec9c" node="k8s-agentpool1-34418057-vmss000000" result_code="succeeded"
I0201 23:56:32.764590       1 utils.go:85] GRPC response: {}
I0201 23:56:33.426690       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0201 23:56:35.436952       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0201 23:56:36.867502       1 utils.go:78] GRPC call: /csi.v1.Controller/DeleteVolume
I0201 23:56:36.867520       1 utils.go:79] GRPC request: {"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-dwxhf0qv/providers/Microsoft.Compute/disks/pvc-add98bb7-ddb5-4554-95ee-1310712bec9c"}
I0201 23:56:36.867578       1 controllerserver_v2.go:201] deleting disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-dwxhf0qv/providers/Microsoft.Compute/disks/pvc-add98bb7-ddb5-4554-95ee-1310712bec9c)
I0201 23:56:36.888668       1 azure_disk_utils.go:1103]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).DeleteVolume" "csi.storage.k8s.io/pv/name"="pvc-add98bb7-ddb5-4554-95ee-1310712bec9c" "disk.csi.azure.com/request-id"="102a3603-a28c-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-add98bb7-ddb5-4554-95ee-1310712bec9c" "latency"=20984631 "number_of_net_retry"=0 "number_of_retry"=0
I0201 23:56:36.888694       1 conditionwatcher.go:99] Adding a condition function for azvolume (pvc-add98bb7-ddb5-4554-95ee-1310712bec9c)
I0201 23:56:36.890230       1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0201 23:56:36.901269       1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0201 23:56:36.910662       1 azure_disk_utils.go:1103]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerDelete" "csi.storage.k8s.io/pv/name"="pvc-add98bb7-ddb5-4554-95ee-1310712bec9c" "disk.csi.azure.com/request-id"="a071a80b-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-add98bb7-ddb5-4554-95ee-1310712bec9c" "latency"=9241702 "number_of_net_retry"=0 "number_of_retry"=0
I0201 23:56:36.910690       1 azvolume.go:246]  "msg"="Deleting Volume..." "csi.storage.k8s.io/pv/name"="pvc-add98bb7-ddb5-4554-95ee-1310712bec9c" "disk.csi.azure.com/request-id"="a071a80b-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/volume-name"="pvc-add98bb7-ddb5-4554-95ee-1310712bec9c"
I0201 23:56:36.910747       1 shared_state.go:1037]  "msg"="AzVolumeAttachment clean up requested by azvolume-controller for AzVolume (pvc-add98bb7-ddb5-4554-95ee-1310712bec9c)" "csi.storage.k8s.io/pv/name"="pvc-add98bb7-ddb5-4554-95ee-1310712bec9c" "disk.csi.azure.com/request-id"="a071a80b-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/volume-name"="pvc-add98bb7-ddb5-4554-95ee-1310712bec9c"
I0201 23:56:36.910767       1 azure_disk_utils.go:1039]  "msg"="Getting AzVolumeAttachment list for volume (pvc-add98bb7-ddb5-4554-95ee-1310712bec9c)" "csi.storage.k8s.io/pv/name"="pvc-add98bb7-ddb5-4554-95ee-1310712bec9c" "disk.csi.azure.com/request-id"="a071a80b-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/volume-name"="pvc-add98bb7-ddb5-4554-95ee-1310712bec9c"
I0201 23:56:36.910814       1 azure_disk_utils.go:1068]  "msg"="Label selector is: disk.csi.azure.com/volume-name=pvc-add98bb7-ddb5-4554-95ee-1310712bec9c." "csi.storage.k8s.io/pv/name"="pvc-add98bb7-ddb5-4554-95ee-1310712bec9c" "disk.csi.azure.com/request-id"="a071a80b-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/volume-name"="pvc-add98bb7-ddb5-4554-95ee-1310712bec9c"
I0201 23:56:36.910861       1 shared_state.go:1035]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-add98bb7-ddb5-4554-95ee-1310712bec9c" "disk.csi.azure.com/request-id"="a071a80b-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*SharedState).cleanUpAzVolumeAttachmentByVolume" "disk.csi.azure.com/volume-name"="pvc-add98bb7-ddb5-4554-95ee-1310712bec9c" "latency"=104501
I0201 23:56:36.911914       1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0201 23:56:36.926131       1 pod.go:65] AzDiskControllerManager "msg"="No need to reconcile. Pod was deleted." "controller"="pod" "disk.csi/azure.com/pod-name"="statefulset-azuredisk-0" "namespace"="azure-disk-csi" "partition"="csi-azuredisk-controller"
I0201 23:56:36.936208       1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1193
I0201 23:56:37.445418       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0201 23:56:38.004578       1 pod.go:65] AzDiskControllerManager "msg"="No need to reconcile. Pod was deleted." "controller"="pod" "disk.csi/azure.com/pod-name"="statefulset-azuredisk-nonroot-0" "namespace"="azure-disk-csi" "partition"="csi-azuredisk-controller"
I0201 23:56:38.362483       1 pod.go:65] AzDiskControllerManager "msg"="No need to reconcile. Pod was deleted." "controller"="pod" "disk.csi/azure.com/pod-name"="nginx-azuredisk-ephemeral" "namespace"="azure-disk-csi" "partition"="csi-azuredisk-controller"
I0201 23:56:38.405977       1 shared_state.go:1037]  "msg"="AzVolumeAttachment clean up requested by pv-controller for AzVolume (pvc-493cdb4e-6be0-4e6c-908c-a4614b19b21e)" "disk.csi.azure.com/request-id"="1114ee63-a28c-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/volume-name"="pvc-493cdb4e-6be0-4e6c-908c-a4614b19b21e"
... skipping 6 lines ...
I0201 23:56:40.360289       1 utils.go:78] GRPC call: /csi.v1.Controller/ControllerUnpublishVolume
I0201 23:56:40.360308       1 utils.go:79] GRPC request: {"node_id":"k8s-agentpool1-34418057-vmss000002","volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-dwxhf0qv/providers/Microsoft.Compute/disks/pvc-f8b6277d-3fb9-4e7e-807d-220f803033b3"}
I0201 23:56:40.360454       1 crdprovisioner.go:927]  "msg"="Requesting AzVolumeAttachment (pvc-f8b6277d-3fb9-4e7e-807d-220f803033b3-k8s-agentpool1-34418057-vmss000002-attachment) detachment" "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000002" "disk.csi.azure.com/request-id"="b1e305b9-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-f8b6277d-3fb9-4e7e-807d-220f803033b3"
I0201 23:56:40.367589       1 replica.go:160]  "msg"="Garbage collection of AzVolumeAttachments for AzVolume (pvc-f8b6277d-3fb9-4e7e-807d-220f803033b3) scheduled in 5m0s." "disk.csi.azure.com/request-id"="124041aa-a28c-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/volume-name"="pvc-f8b6277d-3fb9-4e7e-807d-220f803033b3"
I0201 23:56:40.368802       1 azure_disk_utils.go:1103]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).detachVolume" "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000002" "disk.csi.azure.com/request-id"="b1e305b9-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-f8b6277d-3fb9-4e7e-807d-220f803033b3" "latency"=8304791 "number_of_net_retry"=0 "number_of_retry"=0
I0201 23:56:40.368857       1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-f8b6277d-3fb9-4e7e-807d-220f803033b3-k8s-agentpool1-34418057-vmss000002-attachment)
I0201 23:56:40.368926       1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0201 23:56:40.374093       1 azure_disk_utils.go:1103]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach" "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000002" "disk.csi.azure.com/request-id"="b1e305b9-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-f8b6277d-3fb9-4e7e-807d-220f803033b3" "latency"=6379570 "number_of_net_retry"=0 "number_of_retry"=0
I0201 23:56:40.374124       1 attach_detach.go:340]  "msg"="Detaching volume" "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000002" "disk.csi.azure.com/request-id"="b1e305b9-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-f8b6277d-3fb9-4e7e-807d-220f803033b3"
I0201 23:56:40.374201       1 cloudprovisioner.go:481]  "msg"="Trying to detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-dwxhf0qv/providers/Microsoft.Compute/disks/pvc-f8b6277d-3fb9-4e7e-807d-220f803033b3 from node k8s-agentpool1-34418057-vmss000002" "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000002" "disk.csi.azure.com/request-id"="b1e305b9-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-f8b6277d-3fb9-4e7e-807d-220f803033b3"
I0201 23:56:40.375385       1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0201 23:56:41.375077       1 batch.go:224] "cloud-provider-azure: Delayed processing of batch due to start delay" type="batch" operation="detach_disk" key="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e|kubetest-dwxhf0qv|k8s-agentpool1-34418057-vmss000002" delay="1s"
I0201 23:56:41.375128       1 azure_controller_common.go:461] azuredisk - trying to detach disks from node k8s-agentpool1-34418057-vmss000002: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-dwxhf0qv/providers/Microsoft.Compute/disks/pvc-f8b6277d-3fb9-4e7e-807d-220f803033b3:pvc-f8b6277d-3fb9-4e7e-807d-220f803033b3]
I0201 23:56:41.375179       1 azure_controller_vmss.go:193] azureDisk - detach disk: name pvc-f8b6277d-3fb9-4e7e-807d-220f803033b3 uri /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-dwxhf0qv/providers/Microsoft.Compute/disks/pvc-f8b6277d-3fb9-4e7e-807d-220f803033b3
I0201 23:56:41.375187       1 azure_controller_vmss.go:239] azureDisk - update(kubetest-dwxhf0qv): vm(k8s-agentpool1-34418057-vmss000002) - detach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-dwxhf0qv/providers/Microsoft.Compute/disks/pvc-f8b6277d-3fb9-4e7e-807d-220f803033b3:pvc-f8b6277d-3fb9-4e7e-807d-220f803033b3])
I0201 23:56:41.462902       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0201 23:56:41.890870       1 utils.go:78] GRPC call: /csi.v1.Controller/ControllerUnpublishVolume
I0201 23:56:41.890889       1 utils.go:79] GRPC request: {"node_id":"k8s-agentpool1-34418057-vmss000000","volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-dwxhf0qv/providers/Microsoft.Compute/disks/pvc-493cdb4e-6be0-4e6c-908c-a4614b19b21e"}
I0201 23:56:41.891081       1 crdprovisioner.go:927]  "msg"="Requesting AzVolumeAttachment (pvc-493cdb4e-6be0-4e6c-908c-a4614b19b21e-k8s-agentpool1-34418057-vmss000000-attachment) detachment" "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000000" "disk.csi.azure.com/request-id"="cf7b8750-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-493cdb4e-6be0-4e6c-908c-a4614b19b21e"
I0201 23:56:41.897619       1 replica.go:160]  "msg"="Garbage collection of AzVolumeAttachments for AzVolume (pvc-493cdb4e-6be0-4e6c-908c-a4614b19b21e) scheduled in 5m0s." "disk.csi.azure.com/request-id"="1329b833-a28c-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/volume-name"="pvc-493cdb4e-6be0-4e6c-908c-a4614b19b21e"
I0201 23:56:41.898068       1 azure_disk_utils.go:1103]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).detachVolume" "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000000" "disk.csi.azure.com/request-id"="cf7b8750-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-493cdb4e-6be0-4e6c-908c-a4614b19b21e" "latency"=6932376 "number_of_net_retry"=0 "number_of_retry"=0
I0201 23:56:41.898110       1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-493cdb4e-6be0-4e6c-908c-a4614b19b21e-k8s-agentpool1-34418057-vmss000000-attachment)
I0201 23:56:41.905899       1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0201 23:56:41.906042       1 azure_disk_utils.go:1103]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach" "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000000" "disk.csi.azure.com/request-id"="cf7b8750-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-493cdb4e-6be0-4e6c-908c-a4614b19b21e" "latency"=8391692 "number_of_net_retry"=0 "number_of_retry"=0
I0201 23:56:41.906073       1 attach_detach.go:340]  "msg"="Detaching volume" "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000000" "disk.csi.azure.com/request-id"="cf7b8750-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-493cdb4e-6be0-4e6c-908c-a4614b19b21e"
I0201 23:56:41.906188       1 cloudprovisioner.go:481]  "msg"="Trying to detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-dwxhf0qv/providers/Microsoft.Compute/disks/pvc-493cdb4e-6be0-4e6c-908c-a4614b19b21e from node k8s-agentpool1-34418057-vmss000000" "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000000" "disk.csi.azure.com/request-id"="cf7b8750-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-493cdb4e-6be0-4e6c-908c-a4614b19b21e"
I0201 23:56:42.113103       1 azure_managedDiskController.go:317] azureDisk - deleted a managed disk: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-dwxhf0qv/providers/Microsoft.Compute/disks/pvc-add98bb7-ddb5-4554-95ee-1310712bec9c
I0201 23:56:42.113160       1 cloudprovisioner.go:337]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-add98bb7-ddb5-4554-95ee-1310712bec9c" "disk.csi.azure.com/request-id"="a071a80b-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).DeleteVolume" "disk.csi.azure.com/volume-name"="pvc-add98bb7-ddb5-4554-95ee-1310712bec9c" "latency"=5202241313
I0201 23:56:42.124410       1 azure_disk_utils.go:1103]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerDelete.func4" "csi.storage.k8s.io/pv/name"="pvc-add98bb7-ddb5-4554-95ee-1310712bec9c" "disk.csi.azure.com/request-id"="a071a80b-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-add98bb7-ddb5-4554-95ee-1310712bec9c" "latency"=11204623 "number_of_net_retry"=0 "number_of_retry"=0
I0201 23:56:42.124468       1 azvolume.go:256]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-add98bb7-ddb5-4554-95ee-1310712bec9c" "disk.csi.azure.com/request-id"="a071a80b-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerDelete.func4" "disk.csi.azure.com/volume-name"="pvc-add98bb7-ddb5-4554-95ee-1310712bec9c" "latency"=5213733939
I0201 23:56:42.124495       1 workflow.go:155]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-add98bb7-ddb5-4554-95ee-1310712bec9c" "disk.csi.azure.com/request-id"="a071a80b-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerDelete" "disk.csi.azure.com/volume-name"="pvc-add98bb7-ddb5-4554-95ee-1310712bec9c" "latency"=5223089941
I0201 23:56:42.125472       1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0201 23:56:42.125539       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).DeleteVolume" "csi.storage.k8s.io/pv/name"="pvc-add98bb7-ddb5-4554-95ee-1310712bec9c" "disk.csi.azure.com/request-id"="102a3603-a28c-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-add98bb7-ddb5-4554-95ee-1310712bec9c" "latency"=5224422257
I0201 23:56:42.125583       1 crdprovisioner.go:375]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-add98bb7-ddb5-4554-95ee-1310712bec9c" "disk.csi.azure.com/request-id"="102a3603-a28c-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).DeleteVolume" "disk.csi.azure.com/volume-name"="pvc-add98bb7-ddb5-4554-95ee-1310712bec9c" "latency"=5257923824
I0201 23:56:42.125598       1 controllerserver_v2.go:203] delete disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-dwxhf0qv/providers/Microsoft.Compute/disks/pvc-add98bb7-ddb5-4554-95ee-1310712bec9c) returned with <nil>
I0201 23:56:42.125643       1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=5.258029926 request="azuredisk_csi_driver_controller_delete_volume" resource_group="kubetest-dwxhf0qv" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="test.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-dwxhf0qv/providers/Microsoft.Compute/disks/pvc-add98bb7-ddb5-4554-95ee-1310712bec9c" result_code="succeeded"
I0201 23:56:42.125663       1 utils.go:85] GRPC response: {}
I0201 23:56:42.177168       1 shared_state.go:574]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="13546042-a28c-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*SharedState).deletePV" "latency"=15201
I0201 23:56:42.205481       1 utils.go:78] GRPC call: /csi.v1.Controller/ControllerUnpublishVolume
I0201 23:56:42.205499       1 utils.go:79] GRPC request: {"node_id":"k8s-agentpool1-34418057-vmss000001","volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-dwxhf0qv/providers/Microsoft.Compute/disks/pvc-7ee71f68-c1f4-46fe-bd27-2b23e13b12d8"}
I0201 23:56:42.205666       1 crdprovisioner.go:927]  "msg"="Requesting AzVolumeAttachment (pvc-7ee71f68-c1f4-46fe-bd27-2b23e13b12d8-k8s-agentpool1-34418057-vmss000001-attachment) detachment" "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000001" "disk.csi.azure.com/request-id"="c19f1c5b-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-7ee71f68-c1f4-46fe-bd27-2b23e13b12d8"
I0201 23:56:42.210830       1 azure_disk_utils.go:1103]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).detachVolume" "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000001" "disk.csi.azure.com/request-id"="c19f1c5b-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-7ee71f68-c1f4-46fe-bd27-2b23e13b12d8" "latency"=5120657 "number_of_net_retry"=0 "number_of_retry"=0
I0201 23:56:42.210878       1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-7ee71f68-c1f4-46fe-bd27-2b23e13b12d8-k8s-agentpool1-34418057-vmss000001-attachment)
I0201 23:56:42.211727       1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0201 23:56:42.211868       1 replica.go:160]  "msg"="Garbage collection of AzVolumeAttachments for AzVolume (pvc-7ee71f68-c1f4-46fe-bd27-2b23e13b12d8) scheduled in 5m0s." "disk.csi.azure.com/request-id"="1359abbe-a28c-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/volume-name"="pvc-7ee71f68-c1f4-46fe-bd27-2b23e13b12d8"
I0201 23:56:42.217213       1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0201 23:56:42.217845       1 azure_disk_utils.go:1103]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach" "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000001" "disk.csi.azure.com/request-id"="c19f1c5b-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-7ee71f68-c1f4-46fe-bd27-2b23e13b12d8" "latency"=5933965 "number_of_net_retry"=0 "number_of_retry"=0
I0201 23:56:42.217871       1 attach_detach.go:340]  "msg"="Detaching volume" "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000001" "disk.csi.azure.com/request-id"="c19f1c5b-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-7ee71f68-c1f4-46fe-bd27-2b23e13b12d8"
I0201 23:56:42.217931       1 cloudprovisioner.go:481]  "msg"="Trying to detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-dwxhf0qv/providers/Microsoft.Compute/disks/pvc-7ee71f68-c1f4-46fe-bd27-2b23e13b12d8 from node k8s-agentpool1-34418057-vmss000001" "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000001" "disk.csi.azure.com/request-id"="c19f1c5b-a28b-11ed-97d0-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-7ee71f68-c1f4-46fe-bd27-2b23e13b12d8"
print out csi-test-node logs ...
======================================================================================
dumping logs for kube-system/csi-test-node-524w9/test
... skipping 34 lines ...
I0201 23:53:14.685818       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumes?allowWatchBookmarks=true&resourceVersion=2339&timeout=8m20s&timeoutSeconds=500&watch=true 200 OK in 10 milliseconds
I0201 23:53:14.686239       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/apiextensions.k8s.io/v1/customresourcedefinitions?allowWatchBookmarks=true&resourceVersion=2154&timeout=6m3s&timeoutSeconds=363&watch=true 200 OK in 8 milliseconds
I0201 23:53:14.711963       1 shared_informer.go:303] caches populated
I0201 23:53:14.711993       1 azuredisk_v2.go:225] driver userAgent: test.csi.azure.com/latest-v2-3d84ccaf7bb075abd5912e16e39b09b4d5f88f6c e2e-test
I0201 23:53:14.712003       1 azure_disk_utils.go:564] reading cloud config from secret kube-system/azure-cloud-provider
I0201 23:53:14.713941       1 round_trippers.go:553] GET https://10.0.0.1:443/api/v1/namespaces/kube-system/secrets/azure-cloud-provider 404 Not Found in 1 milliseconds
I0201 23:53:14.714223       1 azure_disk_utils.go:571] InitializeCloudFromSecret: failed to get cloud config from secret kube-system/azure-cloud-provider: failed to get secret kube-system/azure-cloud-provider: secrets "azure-cloud-provider" not found
I0201 23:53:14.714233       1 azure_disk_utils.go:576] could not read cloud config from secret kube-system/azure-cloud-provider
I0201 23:53:14.714241       1 azure_disk_utils.go:586] use default AZURE_CREDENTIAL_FILE env var: /etc/kubernetes/azure.json
I0201 23:53:14.714266       1 azure_disk_utils.go:594] read cloud config from file: /etc/kubernetes/azure.json successfully
I0201 23:53:14.715562       1 azure_auth.go:253] Using AzurePublicCloud environment
I0201 23:53:14.715583       1 azure_auth.go:104] azure: using managed identity extension to retrieve access token
I0201 23:53:14.715590       1 azure_auth.go:110] azure: using User Assigned MSI ID to retrieve access token
... skipping 174 lines ...
I0201 23:53:15.276695       1 round_trippers.go:553] GET https://10.0.0.1:443/api/v1/nodes?allowWatchBookmarks=true&resourceVersion=2346&timeout=6m9s&timeoutSeconds=369&watch=true 200 OK in 1 milliseconds
I0201 23:53:15.277229       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/apiextensions.k8s.io/v1/customresourcedefinitions?allowWatchBookmarks=true&resourceVersion=2154&timeout=7m37s&timeoutSeconds=457&watch=true 200 OK in 1 milliseconds
I0201 23:53:15.293305       1 shared_informer.go:303] caches populated
I0201 23:53:15.293347       1 azuredisk_v2.go:225] driver userAgent: test.csi.azure.com/latest-v2-3d84ccaf7bb075abd5912e16e39b09b4d5f88f6c e2e-test
I0201 23:53:15.293358       1 azure_disk_utils.go:564] reading cloud config from secret kube-system/azure-cloud-provider
I0201 23:53:15.295932       1 round_trippers.go:553] GET https://10.0.0.1:443/api/v1/namespaces/kube-system/secrets/azure-cloud-provider 404 Not Found in 2 milliseconds
I0201 23:53:15.296201       1 azure_disk_utils.go:571] InitializeCloudFromSecret: failed to get cloud config from secret kube-system/azure-cloud-provider: failed to get secret kube-system/azure-cloud-provider: secrets "azure-cloud-provider" not found
I0201 23:53:15.296217       1 azure_disk_utils.go:576] could not read cloud config from secret kube-system/azure-cloud-provider
I0201 23:53:15.296223       1 azure_disk_utils.go:586] use default AZURE_CREDENTIAL_FILE env var: /etc/kubernetes/azure.json
I0201 23:53:15.296252       1 azure_disk_utils.go:594] read cloud config from file: /etc/kubernetes/azure.json successfully
I0201 23:53:15.296805       1 azure_auth.go:253] Using AzurePublicCloud environment
I0201 23:53:15.296820       1 azure_auth.go:104] azure: using managed identity extension to retrieve access token
I0201 23:53:15.296825       1 azure_auth.go:110] azure: using User Assigned MSI ID to retrieve access token
... skipping 53 lines ...
I0201 23:53:15.423063       1 round_trippers.go:553] POST https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes 201 Created in 3 milliseconds
I0201 23:53:15.427705       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-34418057-vmss000000/status 200 OK in 4 milliseconds
I0201 23:53:15.427965       1 azuredisk_v2.go:554]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000000" "disk.csi.azure.com/request-id"="981796d5-a28b-11ed-8c7a-6045bd8e6fd7" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).registerAzDriverNode" "latency"=8911998
I0201 23:53:15.428066       1 azuredisk_v2.go:578]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-02-01T23:53:15Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000000" "disk.csi.azure.com/request-id"="9818f670-a28b-11ed-8c7a-6045bd8e6fd7"
I0201 23:53:15.428210       1 server.go:117] Listening for connections on address: &net.UnixAddr{Name:"//csi/csi.sock", Net:"unix"}
I0201 23:53:15.430627       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-34418057-vmss000000/status 409 Conflict in 2 milliseconds
E0201 23:53:15.430731       1 azuredisk_v2.go:613]  "msg"="Failed to update AzDriverNode status after creation" "error"="Operation cannot be fulfilled on azdrivernodes.disk.csi.azure.com \"k8s-agentpool1-34418057-vmss000000\": the object has been modified; please apply your changes to the latest version and try again" "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000000" "disk.csi.azure.com/request-id"="9818f670-a28b-11ed-8c7a-6045bd8e6fd7"
E0201 23:53:15.430795       1 azuredisk_v2.go:565]  "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [Operation cannot be fulfilled on azdrivernodes.disk.csi.azure.com \"k8s-agentpool1-34418057-vmss000000\": the object has been modified; please apply your changes to the latest version and try again]" "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000000" "disk.csi.azure.com/request-id"="9818f670-a28b-11ed-8c7a-6045bd8e6fd7" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=2741430
I0201 23:53:15.430810       1 azuredisk_v2.go:528] Starting heartbeat loop with initial delay 15.631s and frequency 30s
I0201 23:53:15.531133       1 utils.go:78] GRPC call: /csi.v1.Identity/GetPluginInfo
I0201 23:53:15.531156       1 utils.go:79] GRPC request: {}
I0201 23:53:15.533583       1 utils.go:85] GRPC response: {"name":"test.csi.azure.com","vendor_version":"latest-v2-3d84ccaf7bb075abd5912e16e39b09b4d5f88f6c"}
I0201 23:53:16.382514       1 utils.go:78] GRPC call: /csi.v1.Identity/GetPluginInfo
I0201 23:53:16.382536       1 utils.go:79] GRPC request: {}
... skipping 16 lines ...
I0201 23:53:37.942864       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0201 23:53:37.942882       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-add98bb7-ddb5-4554-95ee-1310712bec9c/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-add98bb7-ddb5-4554-95ee-1310712bec9c","csi.storage.k8s.io/pvc/name":"pvc-azuredisk","csi.storage.k8s.io/pvc/namespace":"default","requestedsizegib":"10","skuName":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1675295602525-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-dwxhf0qv/providers/Microsoft.Compute/disks/pvc-add98bb7-ddb5-4554-95ee-1310712bec9c"}
I0201 23:53:37.943149       1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-add98bb7-ddb5-4554-95ee-1310712bec9c-k8s-agentpool1-34418057-vmss000000-attachment)
I0201 23:53:45.272279       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0201 23:53:45.272310       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0201 23:53:45.272339       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0201 23:53:45.272369       1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0201 23:53:45.276820       1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync
I0201 23:53:45.320000       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0201 23:53:49.864399       1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0201 23:53:49.864498       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000000" "disk.csi.azure.com/request-id"="a5847e7b-a28b-11ed-8c7a-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-add98bb7-ddb5-4554-95ee-1310712bec9c" "latency"=11921231458
I0201 23:53:49.864539       1 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="a5847e7b-a28b-11ed-8c7a-6045bd8e6fd7" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=11921417961
I0201 23:53:51.571358       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/
I0201 23:53:51.571432       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType StandardSSD_LRS
I0201 23:53:51.571490       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-add98bb7-ddb5-4554-95ee-1310712bec9c/globalmount with mount options([])
I0201 23:53:51.571509       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0])
... skipping 62 lines ...
I0201 23:54:58.925073       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I0201 23:54:58.925098       1 utils.go:79] GRPC request: {}
I0201 23:54:58.925142       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I0201 23:55:01.064756       1 azuredisk_v2.go:578]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-02-01T23:55:01Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000000" "disk.csi.azure.com/request-id"="d70fd80c-a28b-11ed-8c7a-6045bd8e6fd7"
I0201 23:55:01.072060       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-34418057-vmss000000/status 200 OK in 7 milliseconds
I0201 23:55:01.072197       1 azuredisk_v2.go:565]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000000" "disk.csi.azure.com/request-id"="d70fd80c-a28b-11ed-8c7a-6045bd8e6fd7" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=7469582
I0201 23:55:04.697564       1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0201 23:55:04.697646       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000000" "disk.csi.azure.com/request-id"="d09b0713-a28b-11ed-8c7a-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-493cdb4e-6be0-4e6c-908c-a4614b19b21e" "latency"=14464737594
I0201 23:55:04.697686       1 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="d09b0713-a28b-11ed-8c7a-6045bd8e6fd7" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=14464853696
I0201 23:55:05.291070       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdd under /dev/disk/azure/scsi1/
I0201 23:55:05.291118       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType StandardSSD_LRS
I0201 23:55:05.291147       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun1 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-493cdb4e-6be0-4e6c-908c-a4614b19b21e/globalmount with mount options([])
I0201 23:55:05.291155       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun1" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun1])
... skipping 48 lines ...
I0201 23:55:21.199592       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetVolumeStats
I0201 23:55:21.199608       1 utils.go:79] GRPC request: {"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-dwxhf0qv/providers/Microsoft.Compute/disks/pvc-add98bb7-ddb5-4554-95ee-1310712bec9c","volume_path":"/var/lib/kubelet/pods/fdab38fd-abad-4b58-ae1a-72a2b8fd8b98/volumes/kubernetes.io~csi/pvc-add98bb7-ddb5-4554-95ee-1310712bec9c/mount"}
I0201 23:55:21.199741       1 utils.go:85] GRPC response: {"usage":[{"available":10447216640,"total":10464022528,"unit":1,"used":28672},{"available":655348,"total":655360,"unit":2,"used":12}]}
I0201 23:55:28.925451       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I0201 23:55:28.925478       1 utils.go:79] GRPC request: {}
I0201 23:55:28.925524       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I0201 23:55:28.980703       1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0201 23:55:28.980772       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000000" "disk.csi.azure.com/request-id"="e27e12c6-a28b-11ed-8c7a-6045bd8e6fd7" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-b3bc4651-4d67-4ce9-aa67-71b6587a3a7e" "latency"=8738632727
I0201 23:55:28.980827       1 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="e27e12c6-a28b-11ed-8c7a-6045bd8e6fd7" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=8738743127
I0201 23:55:29.587356       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun2 by sde under /dev/disk/azure/scsi1/
I0201 23:55:29.587404       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun2. perfProfile none accountType StandardSSD_LRS
I0201 23:55:29.587430       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun2 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-b3bc4651-4d67-4ce9-aa67-71b6587a3a7e/globalmount with mount options([])
I0201 23:55:29.587439       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun2" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun2])
... skipping 138 lines ...
I0201 23:53:23.254764       1 round_trippers.go:553] GET https://10.0.0.1:443/api/v1/nodes?allowWatchBookmarks=true&resourceVersion=2400&timeout=5m48s&timeoutSeconds=348&watch=true 200 OK in 0 milliseconds
I0201 23:53:23.255466       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/apiextensions.k8s.io/v1/customresourcedefinitions?allowWatchBookmarks=true&resourceVersion=2154&timeout=7m10s&timeoutSeconds=430&watch=true 200 OK in 0 milliseconds
I0201 23:53:23.301701       1 shared_informer.go:303] caches populated
I0201 23:53:23.301732       1 azuredisk_v2.go:225] driver userAgent: test.csi.azure.com/latest-v2-3d84ccaf7bb075abd5912e16e39b09b4d5f88f6c e2e-test
I0201 23:53:23.301742       1 azure_disk_utils.go:564] reading cloud config from secret kube-system/azure-cloud-provider
I0201 23:53:23.303978       1 round_trippers.go:553] GET https://10.0.0.1:443/api/v1/namespaces/kube-system/secrets/azure-cloud-provider 404 Not Found in 2 milliseconds
I0201 23:53:23.304225       1 azure_disk_utils.go:571] InitializeCloudFromSecret: failed to get cloud config from secret kube-system/azure-cloud-provider: failed to get secret kube-system/azure-cloud-provider: secrets "azure-cloud-provider" not found
I0201 23:53:23.304243       1 azure_disk_utils.go:576] could not read cloud config from secret kube-system/azure-cloud-provider
I0201 23:53:23.304249       1 azure_disk_utils.go:586] use default AZURE_CREDENTIAL_FILE env var: /etc/kubernetes/azure.json
I0201 23:53:23.304292       1 azure_disk_utils.go:594] read cloud config from file: /etc/kubernetes/azure.json successfully
I0201 23:53:23.304954       1 azure_auth.go:253] Using AzurePublicCloud environment
I0201 23:53:23.304975       1 azure_auth.go:104] azure: using managed identity extension to retrieve access token
I0201 23:53:23.304980       1 azure_auth.go:110] azure: using User Assigned MSI ID to retrieve access token
... skipping 98 lines ...
I0201 23:54:29.712709       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0201 23:54:29.712724       1 utils.go:79] GRPC request: {}
I0201 23:54:29.712752       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I0201 23:54:29.713858       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0201 23:54:29.713874       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-7ee71f68-c1f4-46fe-bd27-2b23e13b12d8/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-7ee71f68-c1f4-46fe-bd27-2b23e13b12d8","csi.storage.k8s.io/pvc/name":"persistent-storage-statefulset-azuredisk-nonroot-0","csi.storage.k8s.io/pvc/namespace":"default","requestedsizegib":"10","skuName":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1675295602525-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-dwxhf0qv/providers/Microsoft.Compute/disks/pvc-7ee71f68-c1f4-46fe-bd27-2b23e13b12d8"}
I0201 23:54:29.714145       1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-7ee71f68-c1f4-46fe-bd27-2b23e13b12d8-k8s-agentpool1-34418057-vmss000001-attachment)
I0201 23:54:36.625600       1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0201 23:54:36.625708       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000001" "disk.csi.azure.com/request-id"="c4601f86-a28b-11ed-8a6e-6045bd8e677f" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-7ee71f68-c1f4-46fe-bd27-2b23e13b12d8" "latency"=6911464592
I0201 23:54:36.625740       1 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="c4601f86-a28b-11ed-8a6e-6045bd8e677f" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=6911614093
I0201 23:54:38.287655       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/
I0201 23:54:38.287717       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType StandardSSD_LRS
I0201 23:54:38.287756       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-7ee71f68-c1f4-46fe-bd27-2b23e13b12d8/globalmount with mount options([])
I0201 23:54:38.287771       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0])
... skipping 51 lines ...
I0201 23:55:20.444198       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0201 23:55:20.444214       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"1"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-ea5fb163-7d4f-4ef9-845b-1feb05c75649/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-ea5fb163-7d4f-4ef9-845b-1feb05c75649","csi.storage.k8s.io/pvc/name":"daemonset-azuredisk-ephemeral-8h5jw-azuredisk","csi.storage.k8s.io/pvc/namespace":"default","requestedsizegib":"10","skuName":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1675295602525-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-dwxhf0qv/providers/Microsoft.Compute/disks/pvc-ea5fb163-7d4f-4ef9-845b-1feb05c75649"}
I0201 23:55:20.444386       1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-ea5fb163-7d4f-4ef9-845b-1feb05c75649-k8s-agentpool1-34418057-vmss000001-attachment)
I0201 23:55:23.255233       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0201 23:55:23.255270       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0201 23:55:23.255339       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0201 23:55:23.255371       1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil>
I0201 23:55:23.256470       1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync
I0201 23:55:23.328103       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync
I0201 23:55:28.905147       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I0201 23:55:28.905173       1 utils.go:79] GRPC request: {}
I0201 23:55:28.905221       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I0201 23:55:29.776332       1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0201 23:55:29.776398       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000001" "disk.csi.azure.com/request-id"="e29cf1a6-a28b-11ed-8a6e-6045bd8e677f" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-ea5fb163-7d4f-4ef9-845b-1feb05c75649" "latency"=9331954225
I0201 23:55:29.776428       1 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="e29cf1a6-a28b-11ed-8a6e-6045bd8e677f" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9332054825
I0201 23:55:30.378602       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdd under /dev/disk/azure/scsi1/
I0201 23:55:30.378665       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType StandardSSD_LRS
I0201 23:55:30.378705       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun1 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-ea5fb163-7d4f-4ef9-845b-1feb05c75649/globalmount with mount options([])
I0201 23:55:30.378728       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun1" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun1])
... skipping 116 lines ...
I0201 23:53:12.262211       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumeattachments?limit=500&resourceVersion=0 200 OK in 384 milliseconds
I0201 23:53:12.263618       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumeattachments?allowWatchBookmarks=true&resourceVersion=2339&timeout=7m49s&timeoutSeconds=469&watch=true 200 OK in 1 milliseconds
I0201 23:53:12.278863       1 shared_informer.go:303] caches populated
I0201 23:53:12.278917       1 azuredisk_v2.go:225] driver userAgent: test.csi.azure.com/latest-v2-3d84ccaf7bb075abd5912e16e39b09b4d5f88f6c e2e-test
I0201 23:53:12.278926       1 azure_disk_utils.go:564] reading cloud config from secret kube-system/azure-cloud-provider
I0201 23:53:12.281569       1 round_trippers.go:553] GET https://10.0.0.1:443/api/v1/namespaces/kube-system/secrets/azure-cloud-provider 404 Not Found in 2 milliseconds
I0201 23:53:12.281788       1 azure_disk_utils.go:571] InitializeCloudFromSecret: failed to get cloud config from secret kube-system/azure-cloud-provider: failed to get secret kube-system/azure-cloud-provider: secrets "azure-cloud-provider" not found
I0201 23:53:12.281803       1 azure_disk_utils.go:576] could not read cloud config from secret kube-system/azure-cloud-provider
I0201 23:53:12.281809       1 azure_disk_utils.go:586] use default AZURE_CREDENTIAL_FILE env var: /etc/kubernetes/azure.json
I0201 23:53:12.281840       1 azure_disk_utils.go:594] read cloud config from file: /etc/kubernetes/azure.json successfully
I0201 23:53:12.282503       1 azure_auth.go:253] Using AzurePublicCloud environment
I0201 23:53:12.282521       1 azure_auth.go:104] azure: using managed identity extension to retrieve access token
I0201 23:53:12.282527       1 azure_auth.go:110] azure: using User Assigned MSI ID to retrieve access token
... skipping 90 lines ...
I0201 23:54:07.809371       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0201 23:54:07.809388       1 utils.go:79] GRPC request: {}
I0201 23:54:07.809424       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I0201 23:54:07.810393       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0201 23:54:07.810411       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-f8b6277d-3fb9-4e7e-807d-220f803033b3/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-f8b6277d-3fb9-4e7e-807d-220f803033b3","csi.storage.k8s.io/pvc/name":"persistent-storage-statefulset-azuredisk-0","csi.storage.k8s.io/pvc/namespace":"default","requestedsizegib":"10","skuName":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1675295602525-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-dwxhf0qv/providers/Microsoft.Compute/disks/pvc-f8b6277d-3fb9-4e7e-807d-220f803033b3"}
I0201 23:54:07.810698       1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-f8b6277d-3fb9-4e7e-807d-220f803033b3-k8s-agentpool1-34418057-vmss000002-attachment)
I0201 23:54:10.062753       1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0201 23:54:10.062849       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000002" "disk.csi.azure.com/request-id"="b751ebfa-a28b-11ed-9c0a-6045bd8e6de3" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-f8b6277d-3fb9-4e7e-807d-220f803033b3" "latency"=2252055873
I0201 23:54:10.062882       1 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="b751ebfa-a28b-11ed-9c0a-6045bd8e6de3" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=2252206374
I0201 23:54:11.840128       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/
I0201 23:54:11.840185       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType StandardSSD_LRS
I0201 23:54:11.840218       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-f8b6277d-3fb9-4e7e-807d-220f803033b3/globalmount with mount options([])
I0201 23:54:11.840235       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0])
... skipping 73 lines ...
I0201 23:55:21.432287       1 azuredisk_v2.go:578]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-02-01T23:55:21Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000002" "disk.csi.azure.com/request-id"="e333ac4f-a28b-11ed-9c0a-6045bd8e6de3"
I0201 23:55:21.438170       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-34418057-vmss000002/status 200 OK in 5 milliseconds
I0201 23:55:21.438427       1 azuredisk_v2.go:565]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000002" "disk.csi.azure.com/request-id"="e333ac4f-a28b-11ed-9c0a-6045bd8e6de3" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=6226262
I0201 23:55:28.591068       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I0201 23:55:28.591098       1 utils.go:79] GRPC request: {}
I0201 23:55:28.591146       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I0201 23:55:29.636403       1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil>
I0201 23:55:29.636479       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-34418057-vmss000002" "disk.csi.azure.com/request-id"="e16537b7-a28b-11ed-9c0a-6045bd8e6de3" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-e98dff13-e112-4307-ab64-b8f8e60741eb" "latency"=11234962363
I0201 23:55:29.636503       1 crdprovisioner.go:743]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="e16537b7-a28b-11ed-9c0a-6045bd8e6de3" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=11235066264
I0201 23:55:30.208203       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdd under /dev/disk/azure/scsi1/
I0201 23:55:30.208251       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType StandardSSD_LRS
I0201 23:55:30.208280       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun1 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-e98dff13-e112-4307-ab64-b8f8e60741eb/globalmount with mount options([])
I0201 23:55:30.208292       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun1" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun1])
... skipping 431 lines ...
# HELP go_gc_heap_objects_objects Number of objects, live or unswept, occupying heap memory.
# TYPE go_gc_heap_objects_objects gauge
go_gc_heap_objects_objects 70757
# HELP go_gc_heap_tiny_allocs_objects_total Count of small allocations that are packed together into blocks. These allocations are counted separately from other allocations because each individual allocation is not tracked by the runtime, only their block. Each block is already accounted for in allocs-by-size and frees-by-size.
# TYPE go_gc_heap_tiny_allocs_objects_total counter
go_gc_heap_tiny_allocs_objects_total 50992
# HELP go_gc_limiter_last_enabled_gc_cycle GC cycle the last time the GC CPU limiter was enabled. This metric is useful for diagnosing the root cause of an out-of-memory error, because the limiter trades memory for CPU time when the GC's CPU time gets too high. This is most likely to occur with use of SetMemoryLimit. The first GC cycle is cycle 1, so a value of 0 indicates that it was never enabled.
# TYPE go_gc_limiter_last_enabled_gc_cycle gauge
go_gc_limiter_last_enabled_gc_cycle 0
# HELP go_gc_pauses_seconds Distribution individual GC-related stop-the-world pause latencies.
# TYPE go_gc_pauses_seconds histogram
go_gc_pauses_seconds_bucket{le="9.999999999999999e-10"} 0
go_gc_pauses_seconds_bucket{le="9.999999999999999e-09"} 0
... skipping 175 lines ...
# HELP process_virtual_memory_max_bytes Maximum amount of virtual memory available in bytes.
# TYPE process_virtual_memory_max_bytes gauge
process_virtual_memory_max_bytes 1.8446744073709552e+19
# HELP registered_metric_total [ALPHA] The count of registered metrics broken by stability level and deprecation version.
# TYPE registered_metric_total counter
registered_metric_total{deprecated_version="",stability_level="ALPHA"} 16
make: *** [Makefile:344: e2e-test] Error 1
2023/02/01 23:56:58 process.go:155: Step 'make e2e-test' finished in 13m2.158611175s
2023/02/01 23:56:58 aksengine_helpers.go:425: downloading /root/tmp1561155475/log-dump.sh from https://raw.githubusercontent.com/kubernetes-sigs/cloud-provider-azure/master/hack/log-dump/log-dump.sh
2023/02/01 23:56:58 util.go:70: curl https://raw.githubusercontent.com/kubernetes-sigs/cloud-provider-azure/master/hack/log-dump/log-dump.sh
2023/02/01 23:56:58 process.go:153: Running: chmod +x /root/tmp1561155475/log-dump.sh
2023/02/01 23:56:58 process.go:155: Step 'chmod +x /root/tmp1561155475/log-dump.sh' finished in 1.328829ms
2023/02/01 23:56:58 aksengine_helpers.go:425: downloading /root/tmp1561155475/log-dump-daemonset.yaml from https://raw.githubusercontent.com/kubernetes-sigs/cloud-provider-azure/master/hack/log-dump/log-dump-daemonset.yaml
... skipping 72 lines ...
2023/02/01 23:57:56 process.go:153: Running: chmod +x /root/tmp1561155475/win-ci-logs-collector.sh
2023/02/01 23:57:56 process.go:155: Step 'chmod +x /root/tmp1561155475/win-ci-logs-collector.sh' finished in 1.168538ms
2023/02/01 23:57:56 process.go:153: Running: bash -c /root/tmp1561155475/win-ci-logs-collector.sh kubetest-dwxhf0qv.westeurope.cloudapp.azure.com /root/tmp1561155475 /root/.ssh/id_rsa
ssh key file /root/.ssh/id_rsa does not exist. Exiting.
2023/02/01 23:57:56 process.go:155: Step 'bash -c /root/tmp1561155475/win-ci-logs-collector.sh kubetest-dwxhf0qv.westeurope.cloudapp.azure.com /root/tmp1561155475 /root/.ssh/id_rsa' finished in 5.1162ms
2023/02/01 23:57:56 aksengine.go:1141: Deleting resource group: kubetest-dwxhf0qv.
2023/02/02 00:13:00 aksengine_helpers.go:326: Warning: failed to delete resource group "kubetest-dwxhf0qv" with error Future#WaitForCompletion: context has been cancelled: StatusCode=202 -- Original Error: context deadline exceeded
2023/02/02 00:13:00 process.go:96: Saved XML output to /logs/artifacts/junit_runner.xml.
2023/02/02 00:13:00 process.go:153: Running: bash -c . hack/lib/version.sh && KUBE_ROOT=. kube::version::get_version_vars && echo "${KUBE_GIT_VERSION-}"
2023/02/02 00:13:01 process.go:155: Step 'bash -c . hack/lib/version.sh && KUBE_ROOT=. kube::version::get_version_vars && echo "${KUBE_GIT_VERSION-}"' finished in 262.322613ms
2023/02/02 00:13:01 main.go:328: Something went wrong: encountered 1 errors: [error during make e2e-test: exit status 2]
+ EXIT_VALUE=1
+ set +o xtrace
Cleaning up after docker in docker.
================================================================================
Cleaning up after docker
3a3931d9b4ac
... skipping 4 lines ...
PR	edreed: [V2] build: disable provenance attestation to work around docker issue
Result	FAILURE
Tests	1 failed / 13 succeeded
Started	2023-02-01 23:36
Elapsed	36m28s
Revision	5ea95d0ebda60f445b259c9f22f01aee679e594a
Refs	1712
job-version	v1.27.0-alpha.1.193+d4750857760ae5
kubetest-version	v20230127-9396ca613c
revision	v1.27.0-alpha.1.193+d4750857760ae5
kubernetes-sigs/azuredisk-csi-driver PR #1712 pull-azuredisk-csi-driver-external-e2e-single-az-mainv2 #1620928613039738880

Test Failures

kubetest Test 13m2s

Passed

Error lines from build-log.txt
