kubernetes-sigs/azuredisk-csi-driver PR #1712 pull-azuredisk-csi-driver-external-e2e-single-az-mainv2 #1620551314154983424
Recent runs || View in Spyglass
Recent runs || View in Spyglass
| PR | edreed: [V2] build: disable provenance attestation to work around docker issue |
| Result | FAILURE |
| Tests | 1 failed / 13 succeeded |
| Started | |
| Elapsed | 29m40s |
| Revision | da7bb6ca0b2c627874174ac85b58dad975d8dfc9 |
| Refs |
1712 |
| job-version | v1.27.0-alpha.1.161+c4ebbeeb747cd3 |
| kubetest-version | v20230127-9396ca613c |
| revision | v1.27.0-alpha.1.161+c4ebbeeb747cd3 |
error during make e2e-test: exit status 2
from junit_runner.xml
Filter through log files | View test history on testgrid
kubetest Check APIReachability
kubetest Deferred TearDown
kubetest DumpClusterLogs
kubetest GetDeployer
kubetest IsUp
kubetest Prepare
kubetest TearDown
kubetest TearDown Previous
kubetest Timeout
kubetest Up
kubetest kubectl version
kubetest list nodes
kubetest test setup
... skipping 241 lines ... 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 11345 100 11345 0 0 170k 0 --:--:-- --:--:-- --:--:-- 170k Downloading https://get.helm.sh/helm-v3.11.0-linux-amd64.tar.gz Verifying checksum... Done. Preparing to install helm into /usr/local/bin helm installed into /usr/local/bin/helm docker pull k8sprow.azurecr.io/azuredisk-csi:latest-v2-930851ca0d19682b1fa9f893a9e8fa4374a3f5f9 || make container-all push-manifest Error response from daemon: manifest for k8sprow.azurecr.io/azuredisk-csi:latest-v2-930851ca0d19682b1fa9f893a9e8fa4374a3f5f9 not found: manifest unknown: manifest tagged by "latest-v2-930851ca0d19682b1fa9f893a9e8fa4374a3f5f9" is not found make[2]: Entering directory '/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver' CGO_ENABLED=0 GOOS=windows go build -a -ldflags "-X sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.driverVersion=latest-v2-930851ca0d19682b1fa9f893a9e8fa4374a3f5f9 -X sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.gitCommit=930851ca0d19682b1fa9f893a9e8fa4374a3f5f9 -X sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.buildDate=2023-01-31T22:41:59Z -extldflags "-static"" -tags azurediskv2 -mod vendor -o _output/amd64/azurediskpluginv2.exe ./pkg/azurediskplugin docker buildx rm container-builder || true ERROR: no builder "container-builder" found docker buildx create --use --name=container-builder --driver-opt image=moby/buildkit:v0.10.6 container-builder # enable qemu for arm64 build # https://github.com/docker/buildx/issues/464#issuecomment-741507760 docker run --privileged --rm tonistiigi/binfmt --uninstall qemu-aarch64 Unable to find image 'tonistiigi/binfmt:latest' locally ... skipping 240 lines ... #9 exporting to image #9 pushing layers 1.8s done #9 pushing manifest for k8sprow.azurecr.io/azuredisk-csi:latest-v2-930851ca0d19682b1fa9f893a9e8fa4374a3f5f9-linux-amd64@sha256:b2419e7515588eb5270f282b18612e648cee6c86136187749d617ecdbf503f8b #9 pushing manifest for k8sprow.azurecr.io/azuredisk-csi:latest-v2-930851ca0d19682b1fa9f893a9e8fa4374a3f5f9-linux-amd64@sha256:b2419e7515588eb5270f282b18612e648cee6c86136187749d617ecdbf503f8b 0.4s done #9 DONE 9.1s WARNING: failed to get git remote url: fatal: No remote configured to list refs from. make[3]: Leaving directory '/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver' make[3]: Entering directory '/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver' CGO_ENABLED=0 GOOS=linux GOARCH=arm64 go build -a -ldflags "-X sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.driverVersion=latest-v2-930851ca0d19682b1fa9f893a9e8fa4374a3f5f9 -X sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.gitCommit=930851ca0d19682b1fa9f893a9e8fa4374a3f5f9 -X sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.buildDate=2023-01-31T22:46:00Z -extldflags "-static"" -tags azurediskv2 -mod vendor -o _output/arm64/azurediskpluginv2 ./pkg/azurediskplugin make[3]: Leaving directory '/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver' make[3]: Entering directory '/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver' docker buildx build . \ ... skipping 114 lines ... #8 exporting config sha256:1b73fa002442c9a8c65afa92f7132bf78f0b5df603c6f08a7e05a39109694a08 0.0s done #8 pushing layers #8 pushing layers 1.6s done #8 pushing manifest for k8sprow.azurecr.io/azuredisk-csi:latest-v2-930851ca0d19682b1fa9f893a9e8fa4374a3f5f9-linux-arm64@sha256:9f4fc984daf8ac6ab08877ac11a63cf5c6994df7b4305028bcdbcd8c0a521b9f #8 pushing manifest for k8sprow.azurecr.io/azuredisk-csi:latest-v2-930851ca0d19682b1fa9f893a9e8fa4374a3f5f9-linux-arm64@sha256:9f4fc984daf8ac6ab08877ac11a63cf5c6994df7b4305028bcdbcd8c0a521b9f 0.4s done #8 DONE 5.7s WARNING: failed to get git remote url: fatal: No remote configured to list refs from. make[3]: Leaving directory '/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver' for osversion in 1809 20H2 ltsc2022; do \ OSVERSION=${osversion} make container-windows; \ done make[3]: Entering directory '/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver' docker buildx build . \ ... skipping 78 lines ... #10 exporting config sha256:a26d2f2349d405abe28f27be5d1a0a9bd62a1ab0b8f60d85a02e903597a85d55 0.0s done #10 pushing layers #10 pushing layers 1.3s done #10 pushing manifest for k8sprow.azurecr.io/azuredisk-csi:latest-v2-930851ca0d19682b1fa9f893a9e8fa4374a3f5f9-windows-1809-amd64@sha256:fed89cfa359d9d1cfaadda9680565cc867e5339190a7ce187ccc61af0a415fe6 #10 pushing manifest for k8sprow.azurecr.io/azuredisk-csi:latest-v2-930851ca0d19682b1fa9f893a9e8fa4374a3f5f9-windows-1809-amd64@sha256:fed89cfa359d9d1cfaadda9680565cc867e5339190a7ce187ccc61af0a415fe6 0.5s done #10 DONE 5.7s WARNING: failed to get git remote url: fatal: No remote configured to list refs from. make[3]: Leaving directory '/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver' make[3]: Entering directory '/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver' docker buildx build . \ --pull \ --output=type=registry \ --platform="windows/amd64" \ ... skipping 64 lines ... #10 exporting config sha256:4ea20632359096654b60187b9e51ee157c95fba1d9bfa427b2add69464fff9b2 0.0s done #10 pushing layers #10 pushing layers 1.3s done #10 pushing manifest for k8sprow.azurecr.io/azuredisk-csi:latest-v2-930851ca0d19682b1fa9f893a9e8fa4374a3f5f9-windows-20H2-amd64@sha256:7af8c69c7e75c745df2511bd343eb627947f3b02b5ad75ae86f0ef9a4127f03d #10 pushing manifest for k8sprow.azurecr.io/azuredisk-csi:latest-v2-930851ca0d19682b1fa9f893a9e8fa4374a3f5f9-windows-20H2-amd64@sha256:7af8c69c7e75c745df2511bd343eb627947f3b02b5ad75ae86f0ef9a4127f03d 0.3s done #10 DONE 5.5s WARNING: failed to get git remote url: fatal: No remote configured to list refs from. make[3]: Leaving directory '/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver' make[3]: Entering directory '/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver' docker buildx build . \ --pull \ --output=type=registry \ --platform="windows/amd64" \ ... skipping 75 lines ... #10 exporting config sha256:85c13f6d13312773acb1be92c600c25659f41598e30cc942fa0ef48688279284 0.0s done #10 pushing layers #10 pushing layers 1.4s done #10 pushing manifest for k8sprow.azurecr.io/azuredisk-csi:latest-v2-930851ca0d19682b1fa9f893a9e8fa4374a3f5f9-windows-ltsc2022-amd64@sha256:629acb0f547fbd4910ad1074e9fbf647062bdbca114a892bad58849dbc596537 #10 pushing manifest for k8sprow.azurecr.io/azuredisk-csi:latest-v2-930851ca0d19682b1fa9f893a9e8fa4374a3f5f9-windows-ltsc2022-amd64@sha256:629acb0f547fbd4910ad1074e9fbf647062bdbca114a892bad58849dbc596537 0.4s done #10 DONE 5.6s WARNING: failed to get git remote url: fatal: No remote configured to list refs from. make[3]: Leaving directory '/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver' docker manifest create --amend k8sprow.azurecr.io/azuredisk-csi:latest-v2-930851ca0d19682b1fa9f893a9e8fa4374a3f5f9 k8sprow.azurecr.io/azuredisk-csi:latest-v2-930851ca0d19682b1fa9f893a9e8fa4374a3f5f9-linux-amd64 k8sprow.azurecr.io/azuredisk-csi:latest-v2-930851ca0d19682b1fa9f893a9e8fa4374a3f5f9-linux-arm64 k8sprow.azurecr.io/azuredisk-csi:latest-v2-930851ca0d19682b1fa9f893a9e8fa4374a3f5f9-windows-1809-amd64 k8sprow.azurecr.io/azuredisk-csi:latest-v2-930851ca0d19682b1fa9f893a9e8fa4374a3f5f9-windows-20H2-amd64 k8sprow.azurecr.io/azuredisk-csi:latest-v2-930851ca0d19682b1fa9f893a9e8fa4374a3f5f9-windows-ltsc2022-amd64 Created manifest list k8sprow.azurecr.io/azuredisk-csi:latest-v2-930851ca0d19682b1fa9f893a9e8fa4374a3f5f9 # add "os.version" field to windows images (based on https://github.com/kubernetes/kubernetes/blob/master/build/pause/Makefile) set -x; \ for arch in amd64; do \ ... skipping 74 lines ... } } ] } make[2]: Leaving directory '/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver' docker pull k8sprow.azurecr.io/azdiskschedulerextender-csi:latest-v2-930851ca0d19682b1fa9f893a9e8fa4374a3f5f9 || make azdiskschedulerextender-all push-manifest-azdiskschedulerextender Error response from daemon: manifest for k8sprow.azurecr.io/azdiskschedulerextender-csi:latest-v2-930851ca0d19682b1fa9f893a9e8fa4374a3f5f9 not found: manifest unknown: manifest tagged by "latest-v2-930851ca0d19682b1fa9f893a9e8fa4374a3f5f9" is not found make[2]: Entering directory '/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver' docker buildx rm container-builder || true container-builder removed docker buildx create --use --name=container-builder --driver-opt image=moby/buildkit:v0.10.6 container-builder # enable qemu for arm64 build ... skipping 159 lines ... #8 exporting to image #8 pushing layers 1.4s done #8 pushing manifest for k8sprow.azurecr.io/azdiskschedulerextender-csi:latest-v2-930851ca0d19682b1fa9f893a9e8fa4374a3f5f9-linux-amd64@sha256:f3732dc37782ca0c4666e27a9f42c859df0b03d59d24cb1d61f4d893effa6508 #8 pushing manifest for k8sprow.azurecr.io/azdiskschedulerextender-csi:latest-v2-930851ca0d19682b1fa9f893a9e8fa4374a3f5f9-linux-amd64@sha256:f3732dc37782ca0c4666e27a9f42c859df0b03d59d24cb1d61f4d893effa6508 0.3s done #8 DONE 5.5s WARNING: failed to get git remote url: fatal: No remote configured to list refs from. make[3]: Leaving directory '/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver' make[3]: Entering directory '/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver' CGO_ENABLED=0 GOOS=linux GOARCH=arm64 go build -a -ldflags "-X sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.schedulerVersion=latest-v2-930851ca0d19682b1fa9f893a9e8fa4374a3f5f9 -X sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.gitCommit=930851ca0d19682b1fa9f893a9e8fa4374a3f5f9 -X sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.buildDate=2023-01-31T22:51:48Z -X sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.DriverName= -extldflags "-static"" -tags azurediskv2 -mod vendor -o _output/arm64/azdiskschedulerextender ./pkg/azdiskschedulerextender make[3]: Leaving directory '/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver' make[3]: Entering directory '/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver' docker buildx build . \ ... skipping 41 lines ... #7 exporting config sha256:89f7ee8befa56f5f978ab845ca3dda2b414d031626c8553439e67ec1a6d6ee17 0.1s done #7 pushing layers #7 pushing layers 1.3s done #7 pushing manifest for k8sprow.azurecr.io/azdiskschedulerextender-csi:latest-v2-930851ca0d19682b1fa9f893a9e8fa4374a3f5f9-linux-arm64@sha256:38cfc7ec21dd6208d95e36ab954441470209c6c6a35b69e6696252a6f43644d3 #7 pushing manifest for k8sprow.azurecr.io/azdiskschedulerextender-csi:latest-v2-930851ca0d19682b1fa9f893a9e8fa4374a3f5f9-linux-arm64@sha256:38cfc7ec21dd6208d95e36ab954441470209c6c6a35b69e6696252a6f43644d3 0.4s done #7 DONE 5.8s WARNING: failed to get git remote url: fatal: No remote configured to list refs from. make[3]: Leaving directory '/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver' docker manifest create --amend k8sprow.azurecr.io/azdiskschedulerextender-csi:latest-v2-930851ca0d19682b1fa9f893a9e8fa4374a3f5f9 k8sprow.azurecr.io/azdiskschedulerextender-csi:latest-v2-930851ca0d19682b1fa9f893a9e8fa4374a3f5f9-linux-amd64 k8sprow.azurecr.io/azdiskschedulerextender-csi:latest-v2-930851ca0d19682b1fa9f893a9e8fa4374a3f5f9-linux-arm64 Created manifest list k8sprow.azurecr.io/azdiskschedulerextender-csi:latest-v2-930851ca0d19682b1fa9f893a9e8fa4374a3f5f9 docker manifest push --purge k8sprow.azurecr.io/azdiskschedulerextender-csi:latest-v2-930851ca0d19682b1fa9f893a9e8fa4374a3f5f9 sha256:e935492813284cad9dedaf696e89125d63657502bf85f36b2857566e5e850467 docker manifest inspect k8sprow.azurecr.io/azdiskschedulerextender-csi:latest-v2-930851ca0d19682b1fa9f893a9e8fa4374a3f5f9 ... skipping 826 lines ... type: string type: object oneOf: - required: ["persistentVolumeClaimName"] - required: ["volumeSnapshotContentName"] volumeSnapshotClassName: description: 'VolumeSnapshotClassName is the name of the VolumeSnapshotClass requested by the VolumeSnapshot. VolumeSnapshotClassName may be left nil to indicate that the default SnapshotClass should be used. A given cluster may have multiple default Volume SnapshotClasses: one default per CSI Driver. If a VolumeSnapshot does not specify a SnapshotClass, VolumeSnapshotSource will be checked to figure out what the associated CSI Driver is, and the default VolumeSnapshotClass associated with that CSI Driver will be used. If more than one VolumeSnapshotClass exist for a given CSI Driver and more than one have been marked as default, CreateSnapshot will fail and generate an event. Empty string is not allowed for this field.' type: string required: - source type: object status: description: status represents the current information of a snapshot. Consumers must verify binding between VolumeSnapshot and VolumeSnapshotContent objects is successful (by validating that both VolumeSnapshot and VolumeSnapshotContent point at each other) before using this object. ... skipping 2 lines ... description: 'boundVolumeSnapshotContentName is the name of the VolumeSnapshotContent object to which this VolumeSnapshot object intends to bind to. If not specified, it indicates that the VolumeSnapshot object has not been successfully bound to a VolumeSnapshotContent object yet. NOTE: To avoid possible security issues, consumers must verify binding between VolumeSnapshot and VolumeSnapshotContent objects is successful (by validating that both VolumeSnapshot and VolumeSnapshotContent point at each other) before using this object.' type: string creationTime: description: creationTime is the timestamp when the point-in-time snapshot is taken by the underlying storage system. In dynamic snapshot creation case, this field will be filled in by the snapshot controller with the "creation_time" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "creation_time" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. If not specified, it may indicate that the creation time of the snapshot is unknown. format: date-time type: string error: description: error is the last observed error during snapshot creation, if any. This field could be helpful to upper level controllers(i.e., application controller) to decide whether they should continue on waiting for the snapshot to be created based on the type of error reported. The snapshot controller will keep retrying when an error occurs during the snapshot creation. Upon success, this error field will be cleared. properties: message: description: 'message is a string detailing the encountered error during snapshot creation if specified. NOTE: message may be logged, and it should not contain sensitive information.' type: string time: description: time is the timestamp when the error was encountered. format: date-time type: string type: object readyToUse: description: readyToUse indicates if the snapshot is ready to be used to restore a volume. In dynamic snapshot creation case, this field will be filled in by the snapshot controller with the "ready_to_use" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "ready_to_use" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it, otherwise, this field will be set to "True". If not specified, it means the readiness of a snapshot is unknown. type: boolean restoreSize: type: string description: restoreSize represents the minimum size of volume required to create a volume from this snapshot. In dynamic snapshot creation case, this field will be filled in by the snapshot controller with the "size_bytes" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "size_bytes" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. When restoring a volume from this snapshot, the size of the volume MUST NOT be smaller than the restoreSize if it is specified, otherwise the restoration will fail. If not specified, it indicates that the size is unknown. pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ x-kubernetes-int-or-string: true type: object required: - spec type: object ... skipping 60 lines ... type: string volumeSnapshotContentName: description: volumeSnapshotContentName specifies the name of a pre-existing VolumeSnapshotContent object representing an existing volume snapshot. This field should be set if the snapshot already exists and only needs a representation in Kubernetes. This field is immutable. type: string type: object volumeSnapshotClassName: description: 'VolumeSnapshotClassName is the name of the VolumeSnapshotClass requested by the VolumeSnapshot. VolumeSnapshotClassName may be left nil to indicate that the default SnapshotClass should be used. A given cluster may have multiple default Volume SnapshotClasses: one default per CSI Driver. If a VolumeSnapshot does not specify a SnapshotClass, VolumeSnapshotSource will be checked to figure out what the associated CSI Driver is, and the default VolumeSnapshotClass associated with that CSI Driver will be used. If more than one VolumeSnapshotClass exist for a given CSI Driver and more than one have been marked as default, CreateSnapshot will fail and generate an event. Empty string is not allowed for this field.' type: string required: - source type: object status: description: status represents the current information of a snapshot. Consumers must verify binding between VolumeSnapshot and VolumeSnapshotContent objects is successful (by validating that both VolumeSnapshot and VolumeSnapshotContent point at each other) before using this object. ... skipping 2 lines ... description: 'boundVolumeSnapshotContentName is the name of the VolumeSnapshotContent object to which this VolumeSnapshot object intends to bind to. If not specified, it indicates that the VolumeSnapshot object has not been successfully bound to a VolumeSnapshotContent object yet. NOTE: To avoid possible security issues, consumers must verify binding between VolumeSnapshot and VolumeSnapshotContent objects is successful (by validating that both VolumeSnapshot and VolumeSnapshotContent point at each other) before using this object.' type: string creationTime: description: creationTime is the timestamp when the point-in-time snapshot is taken by the underlying storage system. In dynamic snapshot creation case, this field will be filled in by the snapshot controller with the "creation_time" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "creation_time" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. If not specified, it may indicate that the creation time of the snapshot is unknown. format: date-time type: string error: description: error is the last observed error during snapshot creation, if any. This field could be helpful to upper level controllers(i.e., application controller) to decide whether they should continue on waiting for the snapshot to be created based on the type of error reported. The snapshot controller will keep retrying when an error occurs during the snapshot creation. Upon success, this error field will be cleared. properties: message: description: 'message is a string detailing the encountered error during snapshot creation if specified. NOTE: message may be logged, and it should not contain sensitive information.' type: string time: description: time is the timestamp when the error was encountered. format: date-time type: string type: object readyToUse: description: readyToUse indicates if the snapshot is ready to be used to restore a volume. In dynamic snapshot creation case, this field will be filled in by the snapshot controller with the "ready_to_use" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "ready_to_use" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it, otherwise, this field will be set to "True". If not specified, it means the readiness of a snapshot is unknown. type: boolean restoreSize: type: string description: restoreSize represents the minimum size of volume required to create a volume from this snapshot. In dynamic snapshot creation case, this field will be filled in by the snapshot controller with the "size_bytes" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "size_bytes" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. When restoring a volume from this snapshot, the size of the volume MUST NOT be smaller than the restoreSize if it is specified, otherwise the restoration will fail. If not specified, it indicates that the size is unknown. pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ x-kubernetes-int-or-string: true type: object required: - spec type: object ... skipping 254 lines ... description: status represents the current information of a snapshot. properties: creationTime: description: creationTime is the timestamp when the point-in-time snapshot is taken by the underlying storage system. In dynamic snapshot creation case, this field will be filled in by the CSI snapshotter sidecar with the "creation_time" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "creation_time" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. If not specified, it indicates the creation time is unknown. The format of this field is a Unix nanoseconds time encoded as an int64. On Unix, the command `date +%s%N` returns the current time in nanoseconds since 1970-01-01 00:00:00 UTC. format: int64 type: integer error: description: error is the last observed error during snapshot creation, if any. Upon success after retry, this error field will be cleared. properties: message: description: 'message is a string detailing the encountered error during snapshot creation if specified. NOTE: message may be logged, and it should not contain sensitive information.' type: string time: description: time is the timestamp when the error was encountered. format: date-time type: string type: object readyToUse: description: readyToUse indicates if a snapshot is ready to be used to restore a volume. In dynamic snapshot creation case, this field will be filled in by the CSI snapshotter sidecar with the "ready_to_use" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "ready_to_use" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it, otherwise, this field will be set to "True". If not specified, it means the readiness of a snapshot is unknown. type: boolean restoreSize: description: restoreSize represents the complete size of the snapshot in bytes. In dynamic snapshot creation case, this field will be filled in by the CSI snapshotter sidecar with the "size_bytes" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "size_bytes" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. When restoring a volume from this snapshot, the size of the volume MUST NOT be smaller than the restoreSize if it is specified, otherwise the restoration will fail. If not specified, it indicates that the size is unknown. format: int64 minimum: 0 type: integer snapshotHandle: description: snapshotHandle is the CSI "snapshot_id" of a snapshot on the underlying storage system. If not specified, it indicates that dynamic snapshot creation has either failed or it is still in progress. type: string type: object required: - spec type: object served: true ... skipping 108 lines ... description: status represents the current information of a snapshot. properties: creationTime: description: creationTime is the timestamp when the point-in-time snapshot is taken by the underlying storage system. In dynamic snapshot creation case, this field will be filled in by the CSI snapshotter sidecar with the "creation_time" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "creation_time" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. If not specified, it indicates the creation time is unknown. The format of this field is a Unix nanoseconds time encoded as an int64. On Unix, the command `date +%s%N` returns the current time in nanoseconds since 1970-01-01 00:00:00 UTC. format: int64 type: integer error: description: error is the last observed error during snapshot creation, if any. Upon success after retry, this error field will be cleared. properties: message: description: 'message is a string detailing the encountered error during snapshot creation if specified. NOTE: message may be logged, and it should not contain sensitive information.' type: string time: description: time is the timestamp when the error was encountered. format: date-time type: string type: object readyToUse: description: readyToUse indicates if a snapshot is ready to be used to restore a volume. In dynamic snapshot creation case, this field will be filled in by the CSI snapshotter sidecar with the "ready_to_use" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "ready_to_use" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it, otherwise, this field will be set to "True". If not specified, it means the readiness of a snapshot is unknown. type: boolean restoreSize: description: restoreSize represents the complete size of the snapshot in bytes. In dynamic snapshot creation case, this field will be filled in by the CSI snapshotter sidecar with the "size_bytes" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "size_bytes" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. When restoring a volume from this snapshot, the size of the volume MUST NOT be smaller than the restoreSize if it is specified, otherwise the restoration will fail. If not specified, it indicates that the size is unknown. format: int64 minimum: 0 type: integer snapshotHandle: description: snapshotHandle is the CSI "snapshot_id" of a snapshot on the underlying storage system. If not specified, it indicates that dynamic snapshot creation has either failed or it is still in progress. type: string type: object required: - spec type: object served: true ... skipping 359 lines ... - volumeName - volume_context - volume_id type: object status: description: status represents the current state of AzVolumeAttachment. includes error, state, and attachment status Required properties: detail: description: Status summarizes the current attachment state of the volume attachment Nil Status indicates that the volume has not yet been attached to the node properties: ... skipping 7 lines ... role: description: The current attachment role. type: string required: - role type: object error: description: Error occurred during attach/detach of volume properties: code: type: string message: type: string parameters: ... skipping 90 lines ... - volumeName - volume_context - volume_id type: object status: description: status represents the current state of AzVolumeAttachment. includes error, state, and attachment status properties: annotation: additionalProperties: type: string description: Annotations contains additional resource information to guide driver actions ... skipping 13 lines ... role: description: The current attachment role. type: string required: - role type: object error: description: Error occurred during attach/detach of volume properties: code: type: string message: type: string parameters: ... skipping 169 lines ... - maxMountReplicaCount - volumeCapability - volumeName type: object status: description: status represents the current state of AzVolume. includes error, state, and volume status properties: detail: description: Current status detail of the AzVolume Nil detail indicates that the volume has not been created properties: accessible_topology: ... skipping 28 lines ... type: string required: - capacity_bytes - node_expansion_required - volume_id type: object error: description: Error occurred during creation/deletion of volume properties: code: type: string message: type: string parameters: ... skipping 154 lines ... - maxMountReplicaCount - volumeCapability - volumeName type: object status: description: status represents the current state of AzVolume. includes error, state, and volume status properties: annotation: additionalProperties: type: string description: Annotations contains additional resource information to guide driver actions ... skipping 34 lines ... type: string required: - capacity_bytes - node_expansion_required - volume_id type: object error: description: Error occurred during creation/deletion of volume properties: code: type: string message: type: string parameters: ... skipping 1069 lines ... image: "mcr.microsoft.com/oss/kubernetes-csi/csi-resizer:v1.5.0" args: - "-csi-address=$(ADDRESS)" - "-v=2" - "-leader-election" - "--leader-election-namespace=kube-system" - '-handle-volume-inuse-error=false' - '-feature-gates=RecoverVolumeExpansionFailure=true' - "-timeout=240s" env: - name: ADDRESS value: /csi/csi.sock volumeMounts: ... skipping 380 lines ... I0131 22:55:00.770021 1 reflector.go:257] Listing and watching *v1beta2.AzVolumeAttachment from sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132 I0131 22:55:00.770571 1 reflector.go:221] Starting reflector *v1.CustomResourceDefinition (30s) from k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132 I0131 22:55:00.770582 1 reflector.go:257] Listing and watching *v1.CustomResourceDefinition from k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132 I0131 22:55:00.869643 1 shared_informer.go:303] caches populated I0131 22:55:00.869680 1 azuredisk_v2.go:225] driver userAgent: test.csi.azure.com/latest-v2-930851ca0d19682b1fa9f893a9e8fa4374a3f5f9 e2e-test I0131 22:55:00.869690 1 azure_disk_utils.go:564] reading cloud config from secret kube-system/azure-cloud-provider I0131 22:55:00.873608 1 azure_disk_utils.go:571] InitializeCloudFromSecret: failed to get cloud config from secret kube-system/azure-cloud-provider: failed to get secret kube-system/azure-cloud-provider: secrets "azure-cloud-provider" not found I0131 22:55:00.873629 1 azure_disk_utils.go:576] could not read cloud config from secret kube-system/azure-cloud-provider I0131 22:55:00.873635 1 azure_disk_utils.go:586] use default AZURE_CREDENTIAL_FILE env var: /etc/kubernetes/azure.json I0131 22:55:00.873661 1 azure_disk_utils.go:594] read cloud config from file: /etc/kubernetes/azure.json successfully I0131 22:55:00.874282 1 azure_auth.go:253] Using AzurePublicCloud environment I0131 22:55:00.874300 1 azure_auth.go:104] azure: using managed identity extension to retrieve access token I0131 22:55:00.874305 1 azure_auth.go:110] azure: using User Assigned MSI ID to retrieve access token ... skipping 369 lines ... I0131 22:55:01.125948 1 shared_state.go:489] "msg"="Storing pod cloud-node-manager-4n65c and claim [] to podToClaimsMap map." "disk.csi.azure.com/request-id"="4ac9e4dd-a1ba-11ed-8e8f-6045bdf48a25" I0131 22:55:01.125960 1 pod.go:91] "msg"="Creating replicas for pod kube-system/cloud-node-manager-4n65c." "disk.csi.azure.com/request-id"="4ac9e4dd-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi/azure.com/pod-name"="kube-system/cloud-node-manager-4n65c" I0131 22:55:01.125974 1 shared_state.go:313] "msg"="Getting requested volumes for pod (kube-system/cloud-node-manager-4n65c)." "disk.csi.azure.com/request-id"="4ac9e4dd-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi/azure.com/pod-name"="kube-system/cloud-node-manager-4n65c" I0131 22:55:01.125984 1 pod.go:99] "msg"="Pod kube-system/cloud-node-manager-4n65c has 0 volumes. Volumes: []" "disk.csi.azure.com/request-id"="4ac9e4dd-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi/azure.com/pod-name"="kube-system/cloud-node-manager-4n65c" I0131 22:55:01.125996 1 pod.go:89] "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="4ac9e4dd-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcilePod).createReplicas" "disk.csi/azure.com/pod-name"="kube-system/cloud-node-manager-4n65c" "latency"=34600 I0131 22:55:01.126010 1 shared_state.go:419] "msg"="Adding pod csi-test-controller-ccf775cc-m8cpv to shared map with keyName kube-system/csi-test-controller-ccf775cc-m8cpv." "disk.csi.azure.com/request-id"="4ac9e4dd-a1ba-11ed-8e8f-6045bdf48a25" I0131 22:55:01.126265 1 shared_state.go:425] "msg"="Pod spec of pod csi-test-controller-ccf775cc-m8cpv is: {Volumes:[{Name:socket-dir VolumeSource:{HostPath:nil EmptyDir:&EmptyDirVolumeSource{Medium:,SizeLimit:<nil>,} GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:azure-cred VolumeSource:{HostPath:&HostPathVolumeSource{Path:/etc/kubernetes/,Type:*DirectoryOrCreate,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:csi-test-controller-config VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:&ConfigMapVolumeSource{LocalObjectReference:LocalObjectReference{Name:csi-test-controller-config,},Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,} VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:kube-api-access-bzvts VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:&ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,} PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:csi-provisioner Image:mcr.microsoft.com/oss/kubernetes-csi/csi-provisioner:v3.2.0 Command:[] Args:[--feature-gates=Topology=true --csi-address=$(ADDRESS) --v=2 --timeout=30s --leader-election --leader-election-namespace=kube-system --worker-threads=100 --extra-create-metadata=true --strict-topology=true --kube-api-qps=50 --kube-api-burst=100] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:ADDRESS Value:/csi/csi.sock ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:524288000 scale:0} d:{Dec:<nil>} s:500Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}] Claims:[]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-bzvts ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:Always SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:csi-attacher Image:mcr.microsoft.com/oss/kubernetes-csi/csi-attacher:v3.5.0 Command:[] Args:[-v=2 -csi-address=$(ADDRESS) -timeout=600s -leader-election --leader-election-namespace=kube-system -worker-threads=500 -kube-api-qps=50 -kube-api-burst=100] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:ADDRESS Value:/csi/csi.sock ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:524288000 scale:0} d:{Dec:<nil>} s:500Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}] Claims:[]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-bzvts ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:Always SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:csi-snapshotter Image:mcr.microsoft.com/oss/kubernetes-csi/csi-snapshotter:v5.0.1 Command:[] Args:[-csi-address=$(ADDRESS) -leader-election --leader-election-namespace=kube-system -v=2] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:ADDRESS Value:/csi/csi.sock ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:104857600 scale:0} d:{Dec:<nil>} s:100Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}] Claims:[]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-bzvts ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:Always SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:csi-resizer Image:mcr.microsoft.com/oss/kubernetes-csi/csi-resizer:v1.5.0 Command:[] Args:[-csi-address=$(ADDRESS) -v=2 -leader-election --leader-election-namespace=kube-system -handle-volume-inuse-error=false -feature-gates=RecoverVolumeExpansionFailure=true -timeout=240s] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:ADDRESS Value:/csi/csi.sock ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:524288000 scale:0} d:{Dec:<nil>} s:500Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}] Claims:[]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-bzvts ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:Always SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:liveness-probe Image:mcr.microsoft.com/oss/kubernetes-csi/livenessprobe:v2.7.0 Command:[] Args:[--csi-address=/csi/csi.sock --probe-timeout=3s --health-port=29602 --v=2] WorkingDir: Ports:[] EnvFrom:[] Env:[] Resources:{Limits:map[memory:{i:{value:104857600 scale:0} d:{Dec:<nil>} s:100Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}] Claims:[]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-bzvts ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:Always SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:azuredisk Image:k8sprow.azurecr.io/azuredisk-csi:latest-v2-930851ca0d19682b1fa9f893a9e8fa4374a3f5f9 Command:[] Args:[--v=5 --config=/etc/csi-test-controller/config.yaml] WorkingDir: Ports:[{Name:healthz HostPort:29602 ContainerPort:29602 Protocol:TCP HostIP:} {Name:metrics HostPort:29604 ContainerPort:29604 Protocol:TCP HostIP:}] EnvFrom:[] Env:[{Name:AZURE_CREDENTIAL_FILE Value: ValueFrom:&EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:&ConfigMapKeySelector{LocalObjectReference:LocalObjectReference{Name:azure-cred-file,},Key:path,Optional:*true,},SecretKeyRef:nil,}} {Name:CSI_ENDPOINT Value:unix:///csi/csi.sock ValueFrom:nil} {Name:AZURE_GO_SDK_LOG_LEVEL Value: ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:524288000 scale:0} d:{Dec:<nil>} s:500Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}] Claims:[]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:azure-cred ReadOnly:false MountPath:/etc/kubernetes/ SubPath: MountPropagation:<nil> SubPathExpr:} {Name:csi-test-controller-config ReadOnly:false MountPath:/etc/csi-test-controller SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-bzvts ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/healthz,Port:{1 0 healthz},Host:,Scheme:HTTP,HTTPHeaders:[]HTTPHeader{},},TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:30,TimeoutSeconds:10,PeriodSeconds:30,SuccessThreshold:1,FailureThreshold:5,TerminationGracePeriodSeconds:nil,} ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:Always SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy:Always TerminationGracePeriodSeconds:0xc00066d2b0 ActiveDeadlineSeconds:<nil> DNSPolicy:ClusterFirst NodeSelector:map[kubernetes.io/os:linux] ServiceAccountName:csi-azuredisk-controller-sa DeprecatedServiceAccount:csi-azuredisk-controller-sa AutomountServiceAccountToken:<nil> NodeName:k8s-agentpool1-24870519-vmss000002 HostNetwork:true HostPID:false HostIPC:false ShareProcessNamespace:<nil> SecurityContext:&PodSecurityContext{SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,SupplementalGroups:[],FSGroup:nil,RunAsGroup:nil,Sysctls:[]Sysctl{},WindowsOptions:nil,FSGroupChangePolicy:nil,SeccompProfile:nil,} ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName:default-scheduler Tolerations:[{Key:node-role.kubernetes.io/master Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>} {Key:node-role.kubernetes.io/controlplane Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>} {Key:node-role.kubernetes.io/control-plane Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>} {Key:node.kubernetes.io/not-ready Operator:Exists Value: Effect:NoExecute TolerationSeconds:0xc00066d2b8} {Key:node.kubernetes.io/unreachable Operator:Exists Value: Effect:NoExecute TolerationSeconds:0xc00066d2c0}] HostAliases:[] PriorityClassName:system-cluster-critical Priority:0xc00066d2c8 DNSConfig:nil ReadinessGates:[] RuntimeClassName:<nil> EnableServiceLinks:0xc00066d2cc PreemptionPolicy:0xc000667410 Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN:<nil> OS:nil HostUsers:<nil> SchedulingGates:[] ResourceClaims:[]}. With volumes: [{Name:socket-dir VolumeSource:{HostPath:nil EmptyDir:&EmptyDirVolumeSource{Medium:,SizeLimit:<nil>,} GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:azure-cred VolumeSource:{HostPath:&HostPathVolumeSource{Path:/etc/kubernetes/,Type:*DirectoryOrCreate,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:csi-test-controller-config VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:&ConfigMapVolumeSource{LocalObjectReference:LocalObjectReference{Name:csi-test-controller-config,},Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,} VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:kube-api-access-bzvts VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:&ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,} PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}]" "disk.csi.azure.com/request-id"="4ac9e4dd-a1ba-11ed-8e8f-6045bdf48a25" I0131 22:55:01.126298 1 shared_state.go:463] "msg"="Pod csi-test-controller-ccf775cc-m8cpv: Skipping Volume {socket-dir {nil &EmptyDirVolumeSource{Medium:,SizeLimit:<nil>,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}}. No persistent volume exists." "disk.csi.azure.com/request-id"="4ac9e4dd-a1ba-11ed-8e8f-6045bdf48a25" I0131 22:55:01.126316 1 shared_state.go:463] "msg"="Pod csi-test-controller-ccf775cc-m8cpv: Skipping Volume {azure-cred {&HostPathVolumeSource{Path:/etc/kubernetes/,Type:*DirectoryOrCreate,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}}. No persistent volume exists." "disk.csi.azure.com/request-id"="4ac9e4dd-a1ba-11ed-8e8f-6045bdf48a25" I0131 22:55:01.126336 1 shared_state.go:463] "msg"="Pod csi-test-controller-ccf775cc-m8cpv: Skipping Volume {csi-test-controller-config {nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil &ConfigMapVolumeSource{LocalObjectReference:LocalObjectReference{Name:csi-test-controller-config,},Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,} nil nil nil nil nil nil nil nil nil nil}}. No persistent volume exists." "disk.csi.azure.com/request-id"="4ac9e4dd-a1ba-11ed-8e8f-6045bdf48a25" I0131 22:55:01.126365 1 shared_state.go:463] "msg"="Pod csi-test-controller-ccf775cc-m8cpv: Skipping Volume {kube-api-access-bzvts {nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil &ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,} nil nil nil nil nil}}. No persistent volume exists." "disk.csi.azure.com/request-id"="4ac9e4dd-a1ba-11ed-8e8f-6045bdf48a25" I0131 22:55:01.126381 1 shared_state.go:489] "msg"="Storing pod csi-test-controller-ccf775cc-m8cpv and claim [] to podToClaimsMap map." "disk.csi.azure.com/request-id"="4ac9e4dd-a1ba-11ed-8e8f-6045bdf48a25" I0131 22:55:01.126395 1 pod.go:91] "msg"="Creating replicas for pod kube-system/csi-test-controller-ccf775cc-m8cpv." "disk.csi.azure.com/request-id"="4ac9e4dd-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi/azure.com/pod-name"="kube-system/csi-test-controller-ccf775cc-m8cpv" ... skipping 113 lines ... I0131 22:55:01.311477 1 shared_state.go:463] "msg"="Pod cloud-node-manager-vwbl8: Skipping Volume {kube-api-access-kcvpl {nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil &ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,} nil nil nil nil nil}}. No persistent volume exists." I0131 22:55:01.311491 1 shared_state.go:489] "msg"="Storing pod cloud-node-manager-vwbl8 and claim [] to podToClaimsMap map." I0131 22:55:01.311526 1 pod.go:91] "msg"="Creating replicas for pod kube-system/cloud-node-manager-vwbl8." "disk.csi.azure.com/request-id"="4b0758d6-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi/azure.com/pod-name"="kube-system/cloud-node-manager-vwbl8" I0131 22:55:01.311547 1 shared_state.go:313] "msg"="Getting requested volumes for pod (kube-system/cloud-node-manager-vwbl8)." "disk.csi.azure.com/request-id"="4b0758d6-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi/azure.com/pod-name"="kube-system/cloud-node-manager-vwbl8" I0131 22:55:01.311564 1 pod.go:99] "msg"="Pod kube-system/cloud-node-manager-vwbl8 has 0 volumes. Volumes: []" "disk.csi.azure.com/request-id"="4b0758d6-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi/azure.com/pod-name"="kube-system/cloud-node-manager-vwbl8" I0131 22:55:01.311590 1 pod.go:89] "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="4b0758d6-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcilePod).createReplicas" "disk.csi/azure.com/pod-name"="kube-system/cloud-node-manager-vwbl8" "latency"=73601 I0131 22:55:01.327450 1 node.go:90] AzDiskControllerManager "msg"="Node is now available. Will requeue failed replica creation requests." "controller"="node" "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000002" "namespace"="azure-disk-csi" "partition"="csi-azuredisk-controller" I0131 22:55:01.327479 1 node.go:90] AzDiskControllerManager "msg"="Node is now available. Will requeue failed replica creation requests." "controller"="node" "disk.csi.azure.com/node-name"="k8s-master-24870519-0" "namespace"="azure-disk-csi" "partition"="csi-azuredisk-controller" I0131 22:55:01.327503 1 shared_state.go:1203] "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="4b09c88f-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*SharedState).tryCreateFailedReplicas" "latency"=8300 I0131 22:55:01.327518 1 node.go:90] AzDiskControllerManager "msg"="Node is now available. Will requeue failed replica creation requests." "controller"="node" "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000001" "namespace"="azure-disk-csi" "partition"="csi-azuredisk-controller" I0131 22:55:01.327536 1 azure_disk_utils.go:1048] "msg"="Getting AzVolumeAttachment list for node (k8s-agentpool1-24870519-vmss000000)" I0131 22:55:01.327575 1 shared_state.go:1203] "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="4b09cb82-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*SharedState).tryCreateFailedReplicas" "latency"=8000 I0131 22:55:01.327581 1 azure_disk_utils.go:1068] "msg"="Label selector is: disk.csi.azure.com/node-name=k8s-agentpool1-24870519-vmss000000." I0131 22:55:01.327525 1 shared_state.go:1203] "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="4b09c981-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*SharedState).tryCreateFailedReplicas" "latency"=8000 I0131 22:55:01.327601 1 shared_state.go:1778] Added node(k8s-agentpool1-24870519-vmss000000) to availableAttachmentsMap with capacity: 16 I0131 22:55:01.327630 1 node.go:90] AzDiskControllerManager "msg"="Node is now available. Will requeue failed replica creation requests." "controller"="node" "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000000" "namespace"="azure-disk-csi" "partition"="csi-azuredisk-controller" I0131 22:55:01.327656 1 shared_state.go:1203] "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="4b09cf0a-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*SharedState).tryCreateFailedReplicas" "latency"=3700 I0131 22:55:01.410974 1 shared_state.go:419] "msg"="Adding pod csi-test-controller-ccf775cc-m8cpv to shared map with keyName kube-system/csi-test-controller-ccf775cc-m8cpv." I0131 22:55:01.411451 1 shared_state.go:425] "msg"="Pod spec of pod csi-test-controller-ccf775cc-m8cpv is: {Volumes:[{Name:socket-dir VolumeSource:{HostPath:nil EmptyDir:&EmptyDirVolumeSource{Medium:,SizeLimit:<nil>,} GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:azure-cred VolumeSource:{HostPath:&HostPathVolumeSource{Path:/etc/kubernetes/,Type:*DirectoryOrCreate,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:csi-test-controller-config VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:&ConfigMapVolumeSource{LocalObjectReference:LocalObjectReference{Name:csi-test-controller-config,},Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,} VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:kube-api-access-bzvts VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:&ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,} PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:csi-provisioner Image:mcr.microsoft.com/oss/kubernetes-csi/csi-provisioner:v3.2.0 Command:[] Args:[--feature-gates=Topology=true --csi-address=$(ADDRESS) --v=2 --timeout=30s --leader-election --leader-election-namespace=kube-system --worker-threads=100 --extra-create-metadata=true --strict-topology=true --kube-api-qps=50 --kube-api-burst=100] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:ADDRESS Value:/csi/csi.sock ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:524288000 scale:0} d:{Dec:<nil>} s:500Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}] Claims:[]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-bzvts ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:Always SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:csi-attacher Image:mcr.microsoft.com/oss/kubernetes-csi/csi-attacher:v3.5.0 Command:[] Args:[-v=2 -csi-address=$(ADDRESS) -timeout=600s -leader-election --leader-election-namespace=kube-system -worker-threads=500 -kube-api-qps=50 -kube-api-burst=100] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:ADDRESS Value:/csi/csi.sock ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:524288000 scale:0} d:{Dec:<nil>} s:500Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}] Claims:[]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-bzvts ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:Always SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:csi-snapshotter Image:mcr.microsoft.com/oss/kubernetes-csi/csi-snapshotter:v5.0.1 Command:[] Args:[-csi-address=$(ADDRESS) -leader-election --leader-election-namespace=kube-system -v=2] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:ADDRESS Value:/csi/csi.sock ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:104857600 scale:0} d:{Dec:<nil>} s:100Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}] Claims:[]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-bzvts ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:Always SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:csi-resizer Image:mcr.microsoft.com/oss/kubernetes-csi/csi-resizer:v1.5.0 Command:[] Args:[-csi-address=$(ADDRESS) -v=2 -leader-election --leader-election-namespace=kube-system -handle-volume-inuse-error=false -feature-gates=RecoverVolumeExpansionFailure=true -timeout=240s] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:ADDRESS Value:/csi/csi.sock ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:524288000 scale:0} d:{Dec:<nil>} s:500Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}] Claims:[]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-bzvts ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:Always SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:liveness-probe Image:mcr.microsoft.com/oss/kubernetes-csi/livenessprobe:v2.7.0 Command:[] Args:[--csi-address=/csi/csi.sock --probe-timeout=3s --health-port=29602 --v=2] WorkingDir: Ports:[] EnvFrom:[] Env:[] Resources:{Limits:map[memory:{i:{value:104857600 scale:0} d:{Dec:<nil>} s:100Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}] Claims:[]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-bzvts ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:Always SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:azuredisk Image:k8sprow.azurecr.io/azuredisk-csi:latest-v2-930851ca0d19682b1fa9f893a9e8fa4374a3f5f9 Command:[] Args:[--v=5 --config=/etc/csi-test-controller/config.yaml] WorkingDir: Ports:[{Name:healthz HostPort:29602 ContainerPort:29602 Protocol:TCP HostIP:} {Name:metrics HostPort:29604 ContainerPort:29604 Protocol:TCP HostIP:}] EnvFrom:[] Env:[{Name:AZURE_CREDENTIAL_FILE Value: ValueFrom:&EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:&ConfigMapKeySelector{LocalObjectReference:LocalObjectReference{Name:azure-cred-file,},Key:path,Optional:*true,},SecretKeyRef:nil,}} {Name:CSI_ENDPOINT Value:unix:///csi/csi.sock ValueFrom:nil} {Name:AZURE_GO_SDK_LOG_LEVEL Value: ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:524288000 scale:0} d:{Dec:<nil>} s:500Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}] Claims:[]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:azure-cred ReadOnly:false MountPath:/etc/kubernetes/ SubPath: MountPropagation:<nil> SubPathExpr:} {Name:csi-test-controller-config ReadOnly:false MountPath:/etc/csi-test-controller SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-bzvts ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/healthz,Port:{1 0 healthz},Host:,Scheme:HTTP,HTTPHeaders:[]HTTPHeader{},},TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:30,TimeoutSeconds:10,PeriodSeconds:30,SuccessThreshold:1,FailureThreshold:5,TerminationGracePeriodSeconds:nil,} ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:Always SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy:Always TerminationGracePeriodSeconds:0xc000aa2848 ActiveDeadlineSeconds:<nil> DNSPolicy:ClusterFirst NodeSelector:map[kubernetes.io/os:linux] ServiceAccountName:csi-azuredisk-controller-sa DeprecatedServiceAccount:csi-azuredisk-controller-sa AutomountServiceAccountToken:<nil> NodeName:k8s-agentpool1-24870519-vmss000002 HostNetwork:true HostPID:false HostIPC:false ShareProcessNamespace:<nil> SecurityContext:&PodSecurityContext{SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,SupplementalGroups:[],FSGroup:nil,RunAsGroup:nil,Sysctls:[]Sysctl{},WindowsOptions:nil,FSGroupChangePolicy:nil,SeccompProfile:nil,} ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName:default-scheduler Tolerations:[{Key:node-role.kubernetes.io/master Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>} {Key:node-role.kubernetes.io/controlplane Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>} {Key:node-role.kubernetes.io/control-plane Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>} {Key:node.kubernetes.io/not-ready Operator:Exists Value: Effect:NoExecute TolerationSeconds:0xc000aa2850} {Key:node.kubernetes.io/unreachable Operator:Exists Value: Effect:NoExecute TolerationSeconds:0xc000aa2858}] HostAliases:[] PriorityClassName:system-cluster-critical Priority:0xc000aa2860 DNSConfig:nil ReadinessGates:[] RuntimeClassName:<nil> EnableServiceLinks:0xc000aa2864 PreemptionPolicy:0xc000c745b0 Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN:<nil> OS:nil HostUsers:<nil> SchedulingGates:[] ResourceClaims:[]}. With volumes: [{Name:socket-dir VolumeSource:{HostPath:nil EmptyDir:&EmptyDirVolumeSource{Medium:,SizeLimit:<nil>,} GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:azure-cred VolumeSource:{HostPath:&HostPathVolumeSource{Path:/etc/kubernetes/,Type:*DirectoryOrCreate,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:csi-test-controller-config VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:&ConfigMapVolumeSource{LocalObjectReference:LocalObjectReference{Name:csi-test-controller-config,},Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,} VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:kube-api-access-bzvts VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:&ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,} PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}]" I0131 22:55:01.411513 1 shared_state.go:463] "msg"="Pod csi-test-controller-ccf775cc-m8cpv: Skipping Volume {socket-dir {nil &EmptyDirVolumeSource{Medium:,SizeLimit:<nil>,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}}. No persistent volume exists." I0131 22:55:01.411543 1 shared_state.go:463] "msg"="Pod csi-test-controller-ccf775cc-m8cpv: Skipping Volume {azure-cred {&HostPathVolumeSource{Path:/etc/kubernetes/,Type:*DirectoryOrCreate,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}}. No persistent volume exists." I0131 22:55:01.411572 1 shared_state.go:463] "msg"="Pod csi-test-controller-ccf775cc-m8cpv: Skipping Volume {csi-test-controller-config {nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil &ConfigMapVolumeSource{LocalObjectReference:LocalObjectReference{Name:csi-test-controller-config,},Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,} nil nil nil nil nil nil nil nil nil nil}}. No persistent volume exists." I0131 22:55:01.411625 1 shared_state.go:463] "msg"="Pod csi-test-controller-ccf775cc-m8cpv: Skipping Volume {kube-api-access-bzvts {nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil &ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,} nil nil nil nil nil}}. No persistent volume exists." I0131 22:55:01.411647 1 shared_state.go:489] "msg"="Storing pod csi-test-controller-ccf775cc-m8cpv and claim [] to podToClaimsMap map." I0131 22:55:01.511317 1 shared_state.go:419] "msg"="Adding pod kube-proxy-jz9dt to shared map with keyName kube-system/kube-proxy-jz9dt." ... skipping 8 lines ... I0131 22:55:01.511862 1 shared_state.go:489] "msg"="Storing pod kube-proxy-jz9dt and claim [] to podToClaimsMap map." I0131 22:55:01.511905 1 pod.go:91] "msg"="Creating replicas for pod kube-system/kube-proxy-jz9dt." "disk.csi.azure.com/request-id"="4b25ebf7-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi/azure.com/pod-name"="kube-system/kube-proxy-jz9dt" I0131 22:55:01.511927 1 shared_state.go:313] "msg"="Getting requested volumes for pod (kube-system/kube-proxy-jz9dt)." "disk.csi.azure.com/request-id"="4b25ebf7-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi/azure.com/pod-name"="kube-system/kube-proxy-jz9dt" I0131 22:55:01.511943 1 pod.go:99] "msg"="Pod kube-system/kube-proxy-jz9dt has 0 volumes. Volumes: []" "disk.csi.azure.com/request-id"="4b25ebf7-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi/azure.com/pod-name"="kube-system/kube-proxy-jz9dt" I0131 22:55:01.511972 1 pod.go:89] "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="4b25ebf7-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcilePod).createReplicas" "disk.csi/azure.com/pod-name"="kube-system/kube-proxy-jz9dt" "latency"=73500 I0131 22:55:01.538155 1 shared_state.go:419] "msg"="Adding pod csi-test-controller-ccf775cc-m8cpv to shared map with keyName kube-system/csi-test-controller-ccf775cc-m8cpv." I0131 22:55:01.538706 1 shared_state.go:425] "msg"="Pod spec of pod csi-test-controller-ccf775cc-m8cpv is: {Volumes:[{Name:socket-dir VolumeSource:{HostPath:nil EmptyDir:&EmptyDirVolumeSource{Medium:,SizeLimit:<nil>,} GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:azure-cred VolumeSource:{HostPath:&HostPathVolumeSource{Path:/etc/kubernetes/,Type:*DirectoryOrCreate,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:csi-test-controller-config VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:&ConfigMapVolumeSource{LocalObjectReference:LocalObjectReference{Name:csi-test-controller-config,},Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,} VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:kube-api-access-bzvts VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:&ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,} PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:csi-provisioner Image:mcr.microsoft.com/oss/kubernetes-csi/csi-provisioner:v3.2.0 Command:[] Args:[--feature-gates=Topology=true --csi-address=$(ADDRESS) --v=2 --timeout=30s --leader-election --leader-election-namespace=kube-system --worker-threads=100 --extra-create-metadata=true --strict-topology=true --kube-api-qps=50 --kube-api-burst=100] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:ADDRESS Value:/csi/csi.sock ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:524288000 scale:0} d:{Dec:<nil>} s:500Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}] Claims:[]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-bzvts ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:Always SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:csi-attacher Image:mcr.microsoft.com/oss/kubernetes-csi/csi-attacher:v3.5.0 Command:[] Args:[-v=2 -csi-address=$(ADDRESS) -timeout=600s -leader-election --leader-election-namespace=kube-system -worker-threads=500 -kube-api-qps=50 -kube-api-burst=100] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:ADDRESS Value:/csi/csi.sock ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:524288000 scale:0} d:{Dec:<nil>} s:500Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}] Claims:[]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-bzvts ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:Always SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:csi-snapshotter Image:mcr.microsoft.com/oss/kubernetes-csi/csi-snapshotter:v5.0.1 Command:[] Args:[-csi-address=$(ADDRESS) -leader-election --leader-election-namespace=kube-system -v=2] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:ADDRESS Value:/csi/csi.sock ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:104857600 scale:0} d:{Dec:<nil>} s:100Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}] Claims:[]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-bzvts ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:Always SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:csi-resizer Image:mcr.microsoft.com/oss/kubernetes-csi/csi-resizer:v1.5.0 Command:[] Args:[-csi-address=$(ADDRESS) -v=2 -leader-election --leader-election-namespace=kube-system -handle-volume-inuse-error=false -feature-gates=RecoverVolumeExpansionFailure=true -timeout=240s] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:ADDRESS Value:/csi/csi.sock ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:524288000 scale:0} d:{Dec:<nil>} s:500Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}] Claims:[]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-bzvts ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:Always SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:liveness-probe Image:mcr.microsoft.com/oss/kubernetes-csi/livenessprobe:v2.7.0 Command:[] Args:[--csi-address=/csi/csi.sock --probe-timeout=3s --health-port=29602 --v=2] WorkingDir: Ports:[] EnvFrom:[] Env:[] Resources:{Limits:map[memory:{i:{value:104857600 scale:0} d:{Dec:<nil>} s:100Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}] Claims:[]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-bzvts ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:Always SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:azuredisk Image:k8sprow.azurecr.io/azuredisk-csi:latest-v2-930851ca0d19682b1fa9f893a9e8fa4374a3f5f9 Command:[] Args:[--v=5 --config=/etc/csi-test-controller/config.yaml] WorkingDir: Ports:[{Name:healthz HostPort:29602 ContainerPort:29602 Protocol:TCP HostIP:} {Name:metrics HostPort:29604 ContainerPort:29604 Protocol:TCP HostIP:}] EnvFrom:[] Env:[{Name:AZURE_CREDENTIAL_FILE Value: ValueFrom:&EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:&ConfigMapKeySelector{LocalObjectReference:LocalObjectReference{Name:azure-cred-file,},Key:path,Optional:*true,},SecretKeyRef:nil,}} {Name:CSI_ENDPOINT Value:unix:///csi/csi.sock ValueFrom:nil} {Name:AZURE_GO_SDK_LOG_LEVEL Value: ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:524288000 scale:0} d:{Dec:<nil>} s:500Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}] Claims:[]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:azure-cred ReadOnly:false MountPath:/etc/kubernetes/ SubPath: MountPropagation:<nil> SubPathExpr:} {Name:csi-test-controller-config ReadOnly:false MountPath:/etc/csi-test-controller SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-bzvts ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/healthz,Port:{1 0 healthz},Host:,Scheme:HTTP,HTTPHeaders:[]HTTPHeader{},},TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:30,TimeoutSeconds:10,PeriodSeconds:30,SuccessThreshold:1,FailureThreshold:5,TerminationGracePeriodSeconds:nil,} ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:Always SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy:Always TerminationGracePeriodSeconds:0xc000c0cc28 ActiveDeadlineSeconds:<nil> DNSPolicy:ClusterFirst NodeSelector:map[kubernetes.io/os:linux] ServiceAccountName:csi-azuredisk-controller-sa DeprecatedServiceAccount:csi-azuredisk-controller-sa AutomountServiceAccountToken:<nil> NodeName:k8s-agentpool1-24870519-vmss000002 HostNetwork:true HostPID:false HostIPC:false ShareProcessNamespace:<nil> SecurityContext:&PodSecurityContext{SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,SupplementalGroups:[],FSGroup:nil,RunAsGroup:nil,Sysctls:[]Sysctl{},WindowsOptions:nil,FSGroupChangePolicy:nil,SeccompProfile:nil,} ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName:default-scheduler Tolerations:[{Key:node-role.kubernetes.io/master Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>} {Key:node-role.kubernetes.io/controlplane Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>} {Key:node-role.kubernetes.io/control-plane Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>} {Key:node.kubernetes.io/not-ready Operator:Exists Value: Effect:NoExecute TolerationSeconds:0xc000c0cc30} {Key:node.kubernetes.io/unreachable Operator:Exists Value: Effect:NoExecute TolerationSeconds:0xc000c0cc38}] HostAliases:[] PriorityClassName:system-cluster-critical Priority:0xc000c0cc40 DNSConfig:nil ReadinessGates:[] RuntimeClassName:<nil> EnableServiceLinks:0xc000c0cc44 PreemptionPolicy:0xc000c75830 Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN:<nil> OS:nil HostUsers:<nil> SchedulingGates:[] ResourceClaims:[]}. With volumes: [{Name:socket-dir VolumeSource:{HostPath:nil EmptyDir:&EmptyDirVolumeSource{Medium:,SizeLimit:<nil>,} GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:azure-cred VolumeSource:{HostPath:&HostPathVolumeSource{Path:/etc/kubernetes/,Type:*DirectoryOrCreate,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:csi-test-controller-config VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:&ConfigMapVolumeSource{LocalObjectReference:LocalObjectReference{Name:csi-test-controller-config,},Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,} VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:kube-api-access-bzvts VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:&ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,} PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}]" I0131 22:55:01.538759 1 shared_state.go:463] "msg"="Pod csi-test-controller-ccf775cc-m8cpv: Skipping Volume {socket-dir {nil &EmptyDirVolumeSource{Medium:,SizeLimit:<nil>,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}}. No persistent volume exists." I0131 22:55:01.538780 1 shared_state.go:463] "msg"="Pod csi-test-controller-ccf775cc-m8cpv: Skipping Volume {azure-cred {&HostPathVolumeSource{Path:/etc/kubernetes/,Type:*DirectoryOrCreate,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}}. No persistent volume exists." I0131 22:55:01.538797 1 shared_state.go:463] "msg"="Pod csi-test-controller-ccf775cc-m8cpv: Skipping Volume {csi-test-controller-config {nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil &ConfigMapVolumeSource{LocalObjectReference:LocalObjectReference{Name:csi-test-controller-config,},Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,} nil nil nil nil nil nil nil nil nil nil}}. No persistent volume exists." I0131 22:55:01.538830 1 shared_state.go:463] "msg"="Pod csi-test-controller-ccf775cc-m8cpv: Skipping Volume {kube-api-access-bzvts {nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil &ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,} nil nil nil nil nil}}. No persistent volume exists." I0131 22:55:01.538844 1 shared_state.go:489] "msg"="Storing pod csi-test-controller-ccf775cc-m8cpv and claim [] to podToClaimsMap map." I0131 22:55:01.538866 1 pod.go:91] "msg"="Creating replicas for pod kube-system/csi-test-controller-ccf775cc-m8cpv." "disk.csi.azure.com/request-id"="4b2a097d-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi/azure.com/pod-name"="kube-system/csi-test-controller-ccf775cc-m8cpv" ... skipping 320 lines ... I0131 22:55:11.831153 1 shared_state.go:489] "msg"="Storing pod deployment-azuredisk-69468c8c97-57gjc and claim [] to podToClaimsMap map." I0131 22:55:11.852735 1 utils.go:78] GRPC call: /csi.v1.Controller/CreateVolume I0131 22:55:11.852755 1 utils.go:79] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"requisite":[{"segments":{"topology.test.csi.azure.com/zone":""}}]},"capacity_range":{"required_bytes":10737418240},"name":"pvc-ca962c87-9010-49bd-aee3-69e8c86cf241","parameters":{"csi.storage.k8s.io/pv/name":"pvc-ca962c87-9010-49bd-aee3-69e8c86cf241","csi.storage.k8s.io/pvc/name":"pvc-azuredisk","csi.storage.k8s.io/pvc/namespace":"default","skuName":"StandardSSD_LRS"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]} I0131 22:55:11.853048 1 crdprovisioner.go:308] "msg"="Creating AzVolume CRI" "csi.storage.k8s.io/pv/name"="pvc-ca962c87-9010-49bd-aee3-69e8c86cf241" "disk.csi.azure.com/request-id"="514fda4d-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/volume-name"="pvc-ca962c87-9010-49bd-aee3-69e8c86cf241" I0131 22:55:11.863689 1 crdprovisioner.go:316] "msg"="Successfully created AzVolume CRI" "csi.storage.k8s.io/pv/name"="pvc-ca962c87-9010-49bd-aee3-69e8c86cf241" "disk.csi.azure.com/request-id"="514fda4d-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/volume-name"="pvc-ca962c87-9010-49bd-aee3-69e8c86cf241" I0131 22:55:11.863719 1 conditionwatcher.go:99] Adding a condition function for azvolume (pvc-ca962c87-9010-49bd-aee3-69e8c86cf241) I0131 22:55:11.864672 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0131 22:55:11.872015 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0131 22:55:11.872518 1 azure_disk_utils.go:1103] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate" "csi.storage.k8s.io/pv/name"="pvc-ca962c87-9010-49bd-aee3-69e8c86cf241" "disk.csi.azure.com/request-id"="514fda4d-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-ca962c87-9010-49bd-aee3-69e8c86cf241" "latency"=7796940 "number_of_net_retry"=0 "number_of_retry"=0 I0131 22:55:11.872563 1 azvolume.go:158] "msg"="Creating Volume..." "csi.storage.k8s.io/pv/name"="pvc-ca962c87-9010-49bd-aee3-69e8c86cf241" "disk.csi.azure.com/request-id"="514fda4d-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/volume-name"="pvc-ca962c87-9010-49bd-aee3-69e8c86cf241" I0131 22:55:12.002544 1 util.go:124] Send.sendRequest got response with ContentLength 253, StatusCode 404 and responseBody length 253 I0131 22:55:12.002629 1 azure_diskclient.go:139] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-n9iqgmbq/providers/Microsoft.Compute/disks/pvc-ca962c87-9010-49bd-aee3-69e8c86cf241, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-ca962c87-9010-49bd-aee3-69e8c86cf241' under resource group 'kubetest-n9iqgmbq' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}} I0131 22:55:12.002965 1 cloudprovisioner.go:246] begin to create disk(pvc-ca962c87-9010-49bd-aee3-69e8c86cf241) account type(StandardSSD_LRS) rg(kubetest-n9iqgmbq) location() size(10) selectedAvailabilityZone() maxShares(0) I0131 22:55:12.048166 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 3502 I0131 22:55:12.048340 1 azure_managedDiskController.go:94] azureDisk - creating new managed Name:pvc-ca962c87-9010-49bd-aee3-69e8c86cf241 StorageAccountType:StandardSSD_LRS Size:10 I0131 22:55:13.053372 1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller I0131 22:55:14.541323 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1192 I0131 22:55:14.541507 1 azure_managedDiskController.go:280] azureDisk - created new MD Name:pvc-ca962c87-9010-49bd-aee3-69e8c86cf241 StorageAccountType:StandardSSD_LRS Size:10 I0131 22:55:14.541583 1 cloudprovisioner.go:320] "msg"="create disk(pvc-ca962c87-9010-49bd-aee3-69e8c86cf241) account type(StandardSSD_LRS) rg(kubetest-n9iqgmbq) location() size(10) tags(map[kubernetes.io-created-for-pv-name:pvc-ca962c87-9010-49bd-aee3-69e8c86cf241 kubernetes.io-created-for-pvc-name:pvc-azuredisk kubernetes.io-created-for-pvc-namespace:default]) successfully" "csi.storage.k8s.io/pv/name"="pvc-ca962c87-9010-49bd-aee3-69e8c86cf241" "disk.csi.azure.com/request-id"="514fda4d-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/volume-name"="pvc-ca962c87-9010-49bd-aee3-69e8c86cf241" I0131 22:55:14.541624 1 cloudprovisioner.go:142] "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-ca962c87-9010-49bd-aee3-69e8c86cf241" "disk.csi.azure.com/request-id"="514fda4d-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-ca962c87-9010-49bd-aee3-69e8c86cf241" "latency"=2668971410 I0131 22:55:14.555456 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0131 22:55:14.555515 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).CreateVolume" "csi.storage.k8s.io/pv/name"="pvc-ca962c87-9010-49bd-aee3-69e8c86cf241" "disk.csi.azure.com/request-id"="514fda4d-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-ca962c87-9010-49bd-aee3-69e8c86cf241" "latency"=2691739029 I0131 22:55:14.555546 1 crdprovisioner.go:228] "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-ca962c87-9010-49bd-aee3-69e8c86cf241" "disk.csi.azure.com/request-id"="514fda4d-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-ca962c87-9010-49bd-aee3-69e8c86cf241" "latency"=2702513985 I0131 22:55:14.555598 1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=2.702561985 request="azuredisk_csi_driver_controller_create_volume" resource_group="kubetest-n9iqgmbq" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="test.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-n9iqgmbq/providers/Microsoft.Compute/disks/pvc-ca962c87-9010-49bd-aee3-69e8c86cf241" result_code="succeeded" I0131 22:55:14.555622 1 utils.go:85] GRPC response: {"volume":{"accessible_topology":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"capacity_bytes":10737418240,"content_source":{"Type":{"Volume":{}}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-ca962c87-9010-49bd-aee3-69e8c86cf241","csi.storage.k8s.io/pvc/name":"pvc-azuredisk","csi.storage.k8s.io/pvc/namespace":"default","requestedsizegib":"10","skuName":"StandardSSD_LRS"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-n9iqgmbq/providers/Microsoft.Compute/disks/pvc-ca962c87-9010-49bd-aee3-69e8c86cf241"}} I0131 22:55:14.556433 1 azure_disk_utils.go:1103] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "csi.storage.k8s.io/pv/name"="pvc-ca962c87-9010-49bd-aee3-69e8c86cf241" "disk.csi.azure.com/request-id"="514fda4d-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-ca962c87-9010-49bd-aee3-69e8c86cf241" "latency"=14757377 "number_of_net_retry"=0 "number_of_retry"=0 I0131 22:55:14.556475 1 azvolume.go:165] "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-ca962c87-9010-49bd-aee3-69e8c86cf241" "disk.csi.azure.com/request-id"="514fda4d-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "disk.csi.azure.com/volume-name"="pvc-ca962c87-9010-49bd-aee3-69e8c86cf241" "latency"=2683878988 I0131 22:55:14.556503 1 workflow.go:155] "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-ca962c87-9010-49bd-aee3-69e8c86cf241" "disk.csi.azure.com/request-id"="514fda4d-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate" "disk.csi.azure.com/volume-name"="pvc-ca962c87-9010-49bd-aee3-69e8c86cf241" "latency"=2691853629 I0131 22:55:14.906796 1 utils.go:78] GRPC call: /csi.v1.Controller/ControllerPublishVolume I0131 22:55:14.906880 1 utils.go:79] GRPC request: {"node_id":"k8s-agentpool1-24870519-vmss000002","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-ca962c87-9010-49bd-aee3-69e8c86cf241","csi.storage.k8s.io/pvc/name":"pvc-azuredisk","csi.storage.k8s.io/pvc/namespace":"default","requestedsizegib":"10","skuName":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1675205701216-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-n9iqgmbq/providers/Microsoft.Compute/disks/pvc-ca962c87-9010-49bd-aee3-69e8c86cf241"} I0131 22:55:14.907630 1 azure_disk_utils.go:1068] "msg"="Label selector is: disk.csi.azure.com/node-name=k8s-agentpool1-24870519-vmss000002,disk.csi.azure.com/volume-name!=pvc-ca962c87-9010-49bd-aee3-69e8c86cf241." "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000002" "disk.csi.azure.com/request-id"="5321ebaf-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-ca962c87-9010-49bd-aee3-69e8c86cf241" I0131 22:55:14.915024 1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-ca962c87-9010-49bd-aee3-69e8c86cf241-k8s-agentpool1-24870519-vmss000002-attachment) I0131 22:55:14.915835 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0131 22:55:14.922086 1 azure_disk_utils.go:1103] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000002" "disk.csi.azure.com/request-id"="5321ebaf-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-ca962c87-9010-49bd-aee3-69e8c86cf241" "latency"=6371533 "number_of_net_retry"=0 "number_of_retry"=0 I0131 22:55:14.922123 1 attach_detach.go:192] "msg"="Attaching volume" "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000002" "disk.csi.azure.com/request-id"="5321ebaf-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-ca962c87-9010-49bd-aee3-69e8c86cf241" I0131 22:55:14.922725 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0131 22:55:14.947172 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1192 I0131 22:55:14.989037 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 4987 I0131 22:55:15.064267 1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller I0131 22:55:15.142016 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 29044 I0131 22:55:15.143993 1 cloudprovisioner.go:405] "msg"="GetDiskLun returned: -1. Initiating attaching volume \"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-n9iqgmbq/providers/Microsoft.Compute/disks/pvc-ca962c87-9010-49bd-aee3-69e8c86cf241\" to node \"k8s-agentpool1-24870519-vmss000002\"." "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000002" "disk.csi.azure.com/request-id"="5321ebaf-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-ca962c87-9010-49bd-aee3-69e8c86cf241" I0131 22:55:15.144046 1 cloudprovisioner.go:424] "msg"="Trying to attach volume \"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-n9iqgmbq/providers/Microsoft.Compute/disks/pvc-ca962c87-9010-49bd-aee3-69e8c86cf241\" to node \"k8s-agentpool1-24870519-vmss000002\"." "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000002" "disk.csi.azure.com/request-id"="5321ebaf-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-ca962c87-9010-49bd-aee3-69e8c86cf241" I0131 22:55:16.144195 1 batch.go:224] "cloud-provider-azure: Delayed processing of batch due to start delay" type="batch" operation="attach_disk" key="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e|kubetest-n9iqgmbq|k8s-agentpool1-24870519-vmss000002" delay="1s" I0131 22:55:16.144348 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-n9iqgmbq): vm(k8s-agentpool1-24870519-vmss000002) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-n9iqgmbq/providers/Microsoft.Compute/disks/pvc-ca962c87-9010-49bd-aee3-69e8c86cf241:AttachDiskOptions{diskName: "pvc-ca962c87-9010-49bd-aee3-69e8c86cf241", lun: 0}]) I0131 22:55:16.158413 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0131 22:55:16.158480 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000002" "disk.csi.azure.com/request-id"="5321ebaf-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-ca962c87-9010-49bd-aee3-69e8c86cf241" "latency"=1243383915 I0131 22:55:16.158513 1 crdprovisioner.go:727] "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000002" "disk.csi.azure.com/request-id"="5321ebaf-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLun" "disk.csi.azure.com/volume-name"="pvc-ca962c87-9010-49bd-aee3-69e8c86cf241" "latency"=1243499816 I0131 22:55:16.158551 1 crdprovisioner.go:486] "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000002" "disk.csi.azure.com/request-id"="5321ebaf-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).PublishVolume" "disk.csi.azure.com/volume-name"="pvc-ca962c87-9010-49bd-aee3-69e8c86cf241" "latency"=1251097955 I0131 22:55:16.158596 1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=1.251167455 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-n9iqgmbq" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="test.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-n9iqgmbq/providers/Microsoft.Compute/disks/pvc-ca962c87-9010-49bd-aee3-69e8c86cf241" node="k8s-agentpool1-24870519-vmss000002" result_code="succeeded" I0131 22:55:16.158611 1 utils.go:85] GRPC response: {"publish_context":{"LUN":"0"}} I0131 22:55:16.158733 1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-ca962c87-9010-49bd-aee3-69e8c86cf241-k8s-agentpool1-24870519-vmss000002-attachment) I0131 22:55:16.158738 1 azure_disk_utils.go:1103] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3.4" "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000002" "disk.csi.azure.com/request-id"="5321ebaf-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-ca962c87-9010-49bd-aee3-69e8c86cf241" "latency"=14186574 "number_of_net_retry"=0 "number_of_retry"=0 I0131 22:55:17.075200 1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller I0131 22:55:17.170982 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-n9iqgmbq): vm(k8s-agentpool1-24870519-vmss000002) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-n9iqgmbq/providers/Microsoft.Compute/disks/pvc-ca962c87-9010-49bd-aee3-69e8c86cf241:AttachDiskOptions{diskName: "pvc-ca962c87-9010-49bd-aee3-69e8c86cf241", lun: 0}], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I0131 22:55:17.171027 1 azure_controller_common.go:344] azuredisk - trying to attach disks to node k8s-agentpool1-24870519-vmss000002: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-n9iqgmbq/providers/Microsoft.Compute/disks/pvc-ca962c87-9010-49bd-aee3-69e8c86cf241:AttachDiskOptions{diskName: "pvc-ca962c87-9010-49bd-aee3-69e8c86cf241", lun: 0}] I0131 22:55:19.086234 1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller I0131 22:55:21.105791 1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller I0131 22:55:23.117190 1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller I0131 22:55:25.131615 1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller I0131 22:55:27.140341 1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller ... skipping 3 lines ... I0131 22:55:27.292357 1 cloudprovisioner.go:440] "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000002" "disk.csi.azure.com/request-id"="5321ebaf-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).PublishVolume.func3" "disk.csi.azure.com/volume-name"="pvc-ca962c87-9010-49bd-aee3-69e8c86cf241" "latency"=12148252259 I0131 22:55:27.292397 1 workflow.go:155] "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000002" "disk.csi.azure.com/request-id"="5321ebaf-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).PublishVolume" "disk.csi.azure.com/volume-name"="pvc-ca962c87-9010-49bd-aee3-69e8c86cf241" "latency"=12370205823 I0131 22:55:27.292488 1 azure_disk_utils.go:1165] "msg"="Skip update. No update needed." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).updateVolumeAttachmentWithResult" "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000002" "disk.csi.azure.com/request-id"="5321ebaf-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-ca962c87-9010-49bd-aee3-69e8c86cf241" I0131 22:55:27.292525 1 azure_disk_utils.go:1103] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).updateVolumeAttachmentWithResult" "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000002" "disk.csi.azure.com/request-id"="5321ebaf-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-ca962c87-9010-49bd-aee3-69e8c86cf241" "latency"=84100 "number_of_net_retry"=0 "number_of_retry"=0 I0131 22:55:27.303321 1 azure_disk_utils.go:1103] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3.4" "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000002" "disk.csi.azure.com/request-id"="5321ebaf-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-ca962c87-9010-49bd-aee3-69e8c86cf241" "latency"=10746757 "number_of_net_retry"=0 "number_of_retry"=0 I0131 22:55:27.303371 1 attach_detach.go:198] "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000002" "disk.csi.azure.com/request-id"="5321ebaf-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3" "disk.csi.azure.com/volume-name"="pvc-ca962c87-9010-49bd-aee3-69e8c86cf241" "latency"=12381196981 I0131 22:55:27.303391 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0131 22:55:27.303400 1 workflow.go:155] "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000002" "disk.csi.azure.com/request-id"="5321ebaf-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach" "disk.csi.azure.com/volume-name"="pvc-ca962c87-9010-49bd-aee3-69e8c86cf241" "latency"=12387722516 I0131 22:55:27.303443 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000002" "disk.csi.azure.com/request-id"="53e0d781-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-ca962c87-9010-49bd-aee3-69e8c86cf241" "latency"=11144629202 I0131 22:55:27.303479 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).ControllerPublishVolume.func2" "disk.csi.azure.com/request-id"="53e0d781-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=11144800703 I0131 22:55:27.303675 1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=12.396091559 request="azuredisk_csi_driver_attach_volume_latency" resource_group="kubetest-n9iqgmbq" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="test.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-n9iqgmbq/providers/Microsoft.Compute/disks/pvc-ca962c87-9010-49bd-aee3-69e8c86cf241" node="k8s-agentpool1-24870519-vmss000002" result_code="succeeded" I0131 22:55:27.303713 1 controllerserver_v2.go:264] "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="53e0d781-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).ControllerPublishVolume.func2" "latency"=11145040004 I0131 22:55:29.150779 1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller ... skipping 25 lines ... I0131 22:55:34.034420 1 shared_state.go:489] "msg"="Storing pod statefulset-azuredisk-0 and claim [] to podToClaimsMap map." I0131 22:55:34.052986 1 utils.go:78] GRPC call: /csi.v1.Controller/CreateVolume I0131 22:55:34.053006 1 utils.go:79] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"requisite":[{"segments":{"topology.test.csi.azure.com/zone":""}}]},"capacity_range":{"required_bytes":10737418240},"name":"pvc-feff1987-6fdb-4ff7-877a-8a3a988036e7","parameters":{"csi.storage.k8s.io/pv/name":"pvc-feff1987-6fdb-4ff7-877a-8a3a988036e7","csi.storage.k8s.io/pvc/name":"persistent-storage-statefulset-azuredisk-0","csi.storage.k8s.io/pvc/namespace":"default","skuName":"StandardSSD_LRS"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]} I0131 22:55:34.053222 1 crdprovisioner.go:308] "msg"="Creating AzVolume CRI" "csi.storage.k8s.io/pv/name"="pvc-feff1987-6fdb-4ff7-877a-8a3a988036e7" "disk.csi.azure.com/request-id"="5e8b543c-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/volume-name"="pvc-feff1987-6fdb-4ff7-877a-8a3a988036e7" I0131 22:55:34.062397 1 crdprovisioner.go:316] "msg"="Successfully created AzVolume CRI" "csi.storage.k8s.io/pv/name"="pvc-feff1987-6fdb-4ff7-877a-8a3a988036e7" "disk.csi.azure.com/request-id"="5e8b543c-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/volume-name"="pvc-feff1987-6fdb-4ff7-877a-8a3a988036e7" I0131 22:55:34.062416 1 conditionwatcher.go:99] Adding a condition function for azvolume (pvc-feff1987-6fdb-4ff7-877a-8a3a988036e7) I0131 22:55:34.064162 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0131 22:55:34.070895 1 azure_disk_utils.go:1103] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate" "csi.storage.k8s.io/pv/name"="pvc-feff1987-6fdb-4ff7-877a-8a3a988036e7" "disk.csi.azure.com/request-id"="5e8b543c-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-feff1987-6fdb-4ff7-877a-8a3a988036e7" "latency"=7046936 "number_of_net_retry"=0 "number_of_retry"=0 I0131 22:55:34.070930 1 azvolume.go:158] "msg"="Creating Volume..." "csi.storage.k8s.io/pv/name"="pvc-feff1987-6fdb-4ff7-877a-8a3a988036e7" "disk.csi.azure.com/request-id"="5e8b543c-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/volume-name"="pvc-feff1987-6fdb-4ff7-877a-8a3a988036e7" I0131 22:55:34.071004 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0131 22:55:34.089035 1 util.go:124] Send.sendRequest got response with ContentLength 253, StatusCode 404 and responseBody length 253 I0131 22:55:34.089061 1 azure_diskclient.go:139] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-n9iqgmbq/providers/Microsoft.Compute/disks/pvc-feff1987-6fdb-4ff7-877a-8a3a988036e7, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-feff1987-6fdb-4ff7-877a-8a3a988036e7' under resource group 'kubetest-n9iqgmbq' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}} I0131 22:55:34.089114 1 cloudprovisioner.go:246] begin to create disk(pvc-feff1987-6fdb-4ff7-877a-8a3a988036e7) account type(StandardSSD_LRS) rg(kubetest-n9iqgmbq) location() size(10) selectedAvailabilityZone() maxShares(0) I0131 22:55:34.138246 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 5067 I0131 22:55:34.138428 1 azure_managedDiskController.go:94] azureDisk - creating new managed Name:pvc-feff1987-6fdb-4ff7-877a-8a3a988036e7 StorageAccountType:StandardSSD_LRS Size:10 I0131 22:55:35.188321 1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller I0131 22:55:36.403603 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1222 I0131 22:55:36.403795 1 azure_managedDiskController.go:280] azureDisk - created new MD Name:pvc-feff1987-6fdb-4ff7-877a-8a3a988036e7 StorageAccountType:StandardSSD_LRS Size:10 I0131 22:55:36.403860 1 cloudprovisioner.go:320] "msg"="create disk(pvc-feff1987-6fdb-4ff7-877a-8a3a988036e7) account type(StandardSSD_LRS) rg(kubetest-n9iqgmbq) location() size(10) tags(map[kubernetes.io-created-for-pv-name:pvc-feff1987-6fdb-4ff7-877a-8a3a988036e7 kubernetes.io-created-for-pvc-name:persistent-storage-statefulset-azuredisk-0 kubernetes.io-created-for-pvc-namespace:default]) successfully" "csi.storage.k8s.io/pv/name"="pvc-feff1987-6fdb-4ff7-877a-8a3a988036e7" "disk.csi.azure.com/request-id"="5e8b543c-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/volume-name"="pvc-feff1987-6fdb-4ff7-877a-8a3a988036e7" I0131 22:55:36.403908 1 cloudprovisioner.go:142] "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-feff1987-6fdb-4ff7-877a-8a3a988036e7" "disk.csi.azure.com/request-id"="5e8b543c-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-feff1987-6fdb-4ff7-877a-8a3a988036e7" "latency"=2332899251 I0131 22:55:36.413407 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0131 22:55:36.413471 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).CreateVolume" "csi.storage.k8s.io/pv/name"="pvc-feff1987-6fdb-4ff7-877a-8a3a988036e7" "disk.csi.azure.com/request-id"="5e8b543c-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-feff1987-6fdb-4ff7-877a-8a3a988036e7" "latency"=2350998844 I0131 22:55:36.413499 1 crdprovisioner.go:228] "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-feff1987-6fdb-4ff7-877a-8a3a988036e7" "disk.csi.azure.com/request-id"="5e8b543c-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-feff1987-6fdb-4ff7-877a-8a3a988036e7" "latency"=2360306594 I0131 22:55:36.413544 1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=2.360357194 request="azuredisk_csi_driver_controller_create_volume" resource_group="kubetest-n9iqgmbq" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="test.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-n9iqgmbq/providers/Microsoft.Compute/disks/pvc-feff1987-6fdb-4ff7-877a-8a3a988036e7" result_code="succeeded" I0131 22:55:36.413561 1 utils.go:85] GRPC response: {"volume":{"accessible_topology":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"capacity_bytes":10737418240,"content_source":{"Type":{"Volume":{}}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-feff1987-6fdb-4ff7-877a-8a3a988036e7","csi.storage.k8s.io/pvc/name":"persistent-storage-statefulset-azuredisk-0","csi.storage.k8s.io/pvc/namespace":"default","requestedsizegib":"10","skuName":"StandardSSD_LRS"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-n9iqgmbq/providers/Microsoft.Compute/disks/pvc-feff1987-6fdb-4ff7-877a-8a3a988036e7"}} I0131 22:55:36.415052 1 azure_disk_utils.go:1103] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "csi.storage.k8s.io/pv/name"="pvc-feff1987-6fdb-4ff7-877a-8a3a988036e7" "disk.csi.azure.com/request-id"="5e8b543c-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-feff1987-6fdb-4ff7-877a-8a3a988036e7" "latency"=11078858 "number_of_net_retry"=0 "number_of_retry"=0 I0131 22:55:36.415100 1 azvolume.go:165] "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-feff1987-6fdb-4ff7-877a-8a3a988036e7" "disk.csi.azure.com/request-id"="5e8b543c-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "disk.csi.azure.com/volume-name"="pvc-feff1987-6fdb-4ff7-877a-8a3a988036e7" "latency"=2344121409 I0131 22:55:36.415130 1 workflow.go:155] "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-feff1987-6fdb-4ff7-877a-8a3a988036e7" "disk.csi.azure.com/request-id"="5e8b543c-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate" "disk.csi.azure.com/volume-name"="pvc-feff1987-6fdb-4ff7-877a-8a3a988036e7" "latency"=2351294146 I0131 22:55:37.157519 1 utils.go:78] GRPC call: /csi.v1.Controller/ControllerPublishVolume I0131 22:55:37.157543 1 utils.go:79] GRPC request: {"node_id":"k8s-agentpool1-24870519-vmss000001","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-feff1987-6fdb-4ff7-877a-8a3a988036e7","csi.storage.k8s.io/pvc/name":"persistent-storage-statefulset-azuredisk-0","csi.storage.k8s.io/pvc/namespace":"default","requestedsizegib":"10","skuName":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1675205701216-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-n9iqgmbq/providers/Microsoft.Compute/disks/pvc-feff1987-6fdb-4ff7-877a-8a3a988036e7"} I0131 22:55:37.157809 1 azure_disk_utils.go:1068] "msg"="Label selector is: disk.csi.azure.com/node-name=k8s-agentpool1-24870519-vmss000001,disk.csi.azure.com/volume-name!=pvc-feff1987-6fdb-4ff7-877a-8a3a988036e7." "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000001" "disk.csi.azure.com/request-id"="60650a6b-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-feff1987-6fdb-4ff7-877a-8a3a988036e7" I0131 22:55:37.163571 1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-feff1987-6fdb-4ff7-877a-8a3a988036e7-k8s-agentpool1-24870519-vmss000001-attachment) I0131 22:55:37.164266 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0131 22:55:37.170049 1 azure_disk_utils.go:1103] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000001" "disk.csi.azure.com/request-id"="60650a6b-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-feff1987-6fdb-4ff7-877a-8a3a988036e7" "latency"=5579629 "number_of_net_retry"=0 "number_of_retry"=0 I0131 22:55:37.170077 1 attach_detach.go:192] "msg"="Attaching volume" "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000001" "disk.csi.azure.com/request-id"="60650a6b-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-feff1987-6fdb-4ff7-877a-8a3a988036e7" I0131 22:55:37.171119 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0131 22:55:37.194185 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1222 I0131 22:55:37.194398 1 cloudprovisioner.go:405] "msg"="GetDiskLun returned: -1. Initiating attaching volume \"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-n9iqgmbq/providers/Microsoft.Compute/disks/pvc-feff1987-6fdb-4ff7-877a-8a3a988036e7\" to node \"k8s-agentpool1-24870519-vmss000001\"." "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000001" "disk.csi.azure.com/request-id"="60650a6b-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-feff1987-6fdb-4ff7-877a-8a3a988036e7" I0131 22:55:37.194444 1 cloudprovisioner.go:424] "msg"="Trying to attach volume \"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-n9iqgmbq/providers/Microsoft.Compute/disks/pvc-feff1987-6fdb-4ff7-877a-8a3a988036e7\" to node \"k8s-agentpool1-24870519-vmss000001\"." "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000001" "disk.csi.azure.com/request-id"="60650a6b-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-feff1987-6fdb-4ff7-877a-8a3a988036e7" I0131 22:55:37.197667 1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller I0131 22:55:38.195076 1 batch.go:224] "cloud-provider-azure: Delayed processing of batch due to start delay" type="batch" operation="attach_disk" key="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e|kubetest-n9iqgmbq|k8s-agentpool1-24870519-vmss000001" delay="1s" I0131 22:55:38.195183 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-n9iqgmbq): vm(k8s-agentpool1-24870519-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-n9iqgmbq/providers/Microsoft.Compute/disks/pvc-feff1987-6fdb-4ff7-877a-8a3a988036e7:AttachDiskOptions{diskName: "pvc-feff1987-6fdb-4ff7-877a-8a3a988036e7", lun: 0}]) I0131 22:55:38.206592 1 azure_disk_utils.go:1103] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3.4" "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000001" "disk.csi.azure.com/request-id"="60650a6b-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-feff1987-6fdb-4ff7-877a-8a3a988036e7" "latency"=11233058 "number_of_net_retry"=0 "number_of_retry"=0 I0131 22:55:38.206638 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0131 22:55:38.206682 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000001" "disk.csi.azure.com/request-id"="60650a6b-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-feff1987-6fdb-4ff7-877a-8a3a988036e7" "latency"=1043052733 I0131 22:55:38.206743 1 crdprovisioner.go:727] "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000001" "disk.csi.azure.com/request-id"="60650a6b-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLun" "disk.csi.azure.com/volume-name"="pvc-feff1987-6fdb-4ff7-877a-8a3a988036e7" "latency"=1043159233 I0131 22:55:38.206781 1 crdprovisioner.go:486] "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000001" "disk.csi.azure.com/request-id"="60650a6b-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).PublishVolume" "disk.csi.azure.com/volume-name"="pvc-feff1987-6fdb-4ff7-877a-8a3a988036e7" "latency"=1049067264 I0131 22:55:38.206831 1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=1.049148565 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-n9iqgmbq" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="test.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-n9iqgmbq/providers/Microsoft.Compute/disks/pvc-feff1987-6fdb-4ff7-877a-8a3a988036e7" node="k8s-agentpool1-24870519-vmss000001" result_code="succeeded" I0131 22:55:38.206841 1 utils.go:85] GRPC response: {"publish_context":{"LUN":"0"}} I0131 22:55:38.206970 1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-feff1987-6fdb-4ff7-877a-8a3a988036e7-k8s-agentpool1-24870519-vmss000001-attachment) I0131 22:55:38.427834 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-n9iqgmbq): vm(k8s-agentpool1-24870519-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-n9iqgmbq/providers/Microsoft.Compute/disks/pvc-feff1987-6fdb-4ff7-877a-8a3a988036e7:AttachDiskOptions{diskName: "pvc-feff1987-6fdb-4ff7-877a-8a3a988036e7", lun: 0}], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I0131 22:55:38.427874 1 azure_controller_common.go:344] azuredisk - trying to attach disks to node k8s-agentpool1-24870519-vmss000001: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-n9iqgmbq/providers/Microsoft.Compute/disks/pvc-feff1987-6fdb-4ff7-877a-8a3a988036e7:AttachDiskOptions{diskName: "pvc-feff1987-6fdb-4ff7-877a-8a3a988036e7", lun: 0}] I0131 22:55:39.208343 1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller I0131 22:55:41.229220 1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller I0131 22:55:43.241654 1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller I0131 22:55:45.256673 1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller I0131 22:55:47.271168 1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller ... skipping 4 lines ... I0131 22:55:48.534572 1 workflow.go:155] "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000001" "disk.csi.azure.com/request-id"="60650a6b-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).PublishVolume" "disk.csi.azure.com/volume-name"="pvc-feff1987-6fdb-4ff7-877a-8a3a988036e7" "latency"=11364435445 I0131 22:55:48.534687 1 azure_disk_utils.go:1165] "msg"="Skip update. No update needed." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).updateVolumeAttachmentWithResult" "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000001" "disk.csi.azure.com/request-id"="60650a6b-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-feff1987-6fdb-4ff7-877a-8a3a988036e7" I0131 22:55:48.534750 1 azure_disk_utils.go:1103] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).updateVolumeAttachmentWithResult" "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000001" "disk.csi.azure.com/request-id"="60650a6b-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-feff1987-6fdb-4ff7-877a-8a3a988036e7" "latency"=148000 "number_of_net_retry"=0 "number_of_retry"=0 I0131 22:55:48.544049 1 azure_disk_utils.go:1103] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3.4" "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000001" "disk.csi.azure.com/request-id"="60650a6b-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-feff1987-6fdb-4ff7-877a-8a3a988036e7" "latency"=9258748 "number_of_net_retry"=0 "number_of_retry"=0 I0131 22:55:48.544096 1 attach_detach.go:198] "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000001" "disk.csi.azure.com/request-id"="60650a6b-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3" "disk.csi.azure.com/volume-name"="pvc-feff1987-6fdb-4ff7-877a-8a3a988036e7" "latency"=11373978895 I0131 22:55:48.544123 1 workflow.go:155] "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000001" "disk.csi.azure.com/request-id"="60650a6b-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach" "disk.csi.azure.com/volume-name"="pvc-feff1987-6fdb-4ff7-877a-8a3a988036e7" "latency"=11379688624 I0131 22:55:48.544176 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0131 22:55:48.544217 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000001" "disk.csi.azure.com/request-id"="610523b4-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-feff1987-6fdb-4ff7-877a-8a3a988036e7" "latency"=10337182095 I0131 22:55:48.544245 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).ControllerPublishVolume.func2" "disk.csi.azure.com/request-id"="610523b4-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=10337307096 I0131 22:55:48.544269 1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=11.386588961 request="azuredisk_csi_driver_attach_volume_latency" resource_group="kubetest-n9iqgmbq" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="test.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-n9iqgmbq/providers/Microsoft.Compute/disks/pvc-feff1987-6fdb-4ff7-877a-8a3a988036e7" node="k8s-agentpool1-24870519-vmss000001" result_code="succeeded" I0131 22:55:48.544290 1 controllerserver_v2.go:264] "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="610523b4-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).ControllerPublishVolume.func2" "latency"=10337364096 I0131 22:55:49.280550 1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller I0131 22:55:51.295062 1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller ... skipping 21 lines ... I0131 22:55:57.206224 1 shared_state.go:489] "msg"="Storing pod statefulset-azuredisk-nonroot-0 and claim [] to podToClaimsMap map." I0131 22:55:57.218947 1 utils.go:78] GRPC call: /csi.v1.Controller/CreateVolume I0131 22:55:57.218963 1 utils.go:79] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"requisite":[{"segments":{"topology.test.csi.azure.com/zone":""}}]},"capacity_range":{"required_bytes":10737418240},"name":"pvc-3d318377-2dd2-4c2a-9669-be7055655892","parameters":{"csi.storage.k8s.io/pv/name":"pvc-3d318377-2dd2-4c2a-9669-be7055655892","csi.storage.k8s.io/pvc/name":"persistent-storage-statefulset-azuredisk-nonroot-0","csi.storage.k8s.io/pvc/namespace":"default","skuName":"StandardSSD_LRS"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]} I0131 22:55:57.219140 1 crdprovisioner.go:308] "msg"="Creating AzVolume CRI" "csi.storage.k8s.io/pv/name"="pvc-3d318377-2dd2-4c2a-9669-be7055655892" "disk.csi.azure.com/request-id"="6c5a2b27-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/volume-name"="pvc-3d318377-2dd2-4c2a-9669-be7055655892" I0131 22:55:57.227333 1 crdprovisioner.go:316] "msg"="Successfully created AzVolume CRI" "csi.storage.k8s.io/pv/name"="pvc-3d318377-2dd2-4c2a-9669-be7055655892" "disk.csi.azure.com/request-id"="6c5a2b27-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/volume-name"="pvc-3d318377-2dd2-4c2a-9669-be7055655892" I0131 22:55:57.227354 1 conditionwatcher.go:99] Adding a condition function for azvolume (pvc-3d318377-2dd2-4c2a-9669-be7055655892) I0131 22:55:57.233316 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0131 22:55:57.233646 1 azure_disk_utils.go:1103] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate" "csi.storage.k8s.io/pv/name"="pvc-3d318377-2dd2-4c2a-9669-be7055655892" "disk.csi.azure.com/request-id"="6c5a2b27-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-3d318377-2dd2-4c2a-9669-be7055655892" "latency"=7845441 "number_of_net_retry"=0 "number_of_retry"=0 I0131 22:55:57.233671 1 azvolume.go:158] "msg"="Creating Volume..." "csi.storage.k8s.io/pv/name"="pvc-3d318377-2dd2-4c2a-9669-be7055655892" "disk.csi.azure.com/request-id"="6c5a2b27-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/volume-name"="pvc-3d318377-2dd2-4c2a-9669-be7055655892" I0131 22:55:57.260346 1 util.go:124] Send.sendRequest got response with ContentLength 253, StatusCode 404 and responseBody length 253 I0131 22:55:57.260374 1 azure_diskclient.go:139] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-n9iqgmbq/providers/Microsoft.Compute/disks/pvc-3d318377-2dd2-4c2a-9669-be7055655892, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-3d318377-2dd2-4c2a-9669-be7055655892' under resource group 'kubetest-n9iqgmbq' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}} I0131 22:55:57.260424 1 cloudprovisioner.go:246] begin to create disk(pvc-3d318377-2dd2-4c2a-9669-be7055655892) account type(StandardSSD_LRS) rg(kubetest-n9iqgmbq) location() size(10) selectedAvailabilityZone() maxShares(0) I0131 22:55:57.318274 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 6662 I0131 22:55:57.318511 1 azure_managedDiskController.go:94] azureDisk - creating new managed Name:pvc-3d318377-2dd2-4c2a-9669-be7055655892 StorageAccountType:StandardSSD_LRS Size:10 I0131 22:55:57.326995 1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller I0131 22:55:59.337634 1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller I0131 22:55:59.642347 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1229 I0131 22:55:59.642536 1 azure_managedDiskController.go:280] azureDisk - created new MD Name:pvc-3d318377-2dd2-4c2a-9669-be7055655892 StorageAccountType:StandardSSD_LRS Size:10 I0131 22:55:59.642604 1 cloudprovisioner.go:320] "msg"="create disk(pvc-3d318377-2dd2-4c2a-9669-be7055655892) account type(StandardSSD_LRS) rg(kubetest-n9iqgmbq) location() size(10) tags(map[kubernetes.io-created-for-pv-name:pvc-3d318377-2dd2-4c2a-9669-be7055655892 kubernetes.io-created-for-pvc-name:persistent-storage-statefulset-azuredisk-nonroot-0 kubernetes.io-created-for-pvc-namespace:default]) successfully" "csi.storage.k8s.io/pv/name"="pvc-3d318377-2dd2-4c2a-9669-be7055655892" "disk.csi.azure.com/request-id"="6c5a2b27-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/volume-name"="pvc-3d318377-2dd2-4c2a-9669-be7055655892" I0131 22:55:59.642631 1 cloudprovisioner.go:142] "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-3d318377-2dd2-4c2a-9669-be7055655892" "disk.csi.azure.com/request-id"="6c5a2b27-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-3d318377-2dd2-4c2a-9669-be7055655892" "latency"=2408902252 I0131 22:55:59.653734 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0131 22:55:59.653809 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).CreateVolume" "csi.storage.k8s.io/pv/name"="pvc-3d318377-2dd2-4c2a-9669-be7055655892" "disk.csi.azure.com/request-id"="6c5a2b27-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-3d318377-2dd2-4c2a-9669-be7055655892" "latency"=2426391343 I0131 22:55:59.653844 1 crdprovisioner.go:228] "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-3d318377-2dd2-4c2a-9669-be7055655892" "disk.csi.azure.com/request-id"="6c5a2b27-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-3d318377-2dd2-4c2a-9669-be7055655892" "latency"=2434725487 I0131 22:55:59.653876 1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=2.434771187 request="azuredisk_csi_driver_controller_create_volume" resource_group="kubetest-n9iqgmbq" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="test.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-n9iqgmbq/providers/Microsoft.Compute/disks/pvc-3d318377-2dd2-4c2a-9669-be7055655892" result_code="succeeded" I0131 22:55:59.653891 1 utils.go:85] GRPC response: {"volume":{"accessible_topology":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"capacity_bytes":10737418240,"content_source":{"Type":{"Volume":{}}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-3d318377-2dd2-4c2a-9669-be7055655892","csi.storage.k8s.io/pvc/name":"persistent-storage-statefulset-azuredisk-nonroot-0","csi.storage.k8s.io/pvc/namespace":"default","requestedsizegib":"10","skuName":"StandardSSD_LRS"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-n9iqgmbq/providers/Microsoft.Compute/disks/pvc-3d318377-2dd2-4c2a-9669-be7055655892"}} I0131 22:55:59.654559 1 azure_disk_utils.go:1103] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "csi.storage.k8s.io/pv/name"="pvc-3d318377-2dd2-4c2a-9669-be7055655892" "disk.csi.azure.com/request-id"="6c5a2b27-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-3d318377-2dd2-4c2a-9669-be7055655892" "latency"=11871162 "number_of_net_retry"=0 "number_of_retry"=0 I0131 22:55:59.654605 1 azvolume.go:165] "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-3d318377-2dd2-4c2a-9669-be7055655892" "disk.csi.azure.com/request-id"="6c5a2b27-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "disk.csi.azure.com/volume-name"="pvc-3d318377-2dd2-4c2a-9669-be7055655892" "latency"=2420892715 I0131 22:55:59.654633 1 workflow.go:155] "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-3d318377-2dd2-4c2a-9669-be7055655892" "disk.csi.azure.com/request-id"="6c5a2b27-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate" "disk.csi.azure.com/volume-name"="pvc-3d318377-2dd2-4c2a-9669-be7055655892" "latency"=2428847356 I0131 22:56:00.315902 1 utils.go:78] GRPC call: /csi.v1.Controller/ControllerPublishVolume I0131 22:56:00.315925 1 utils.go:79] GRPC request: {"node_id":"k8s-agentpool1-24870519-vmss000000","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-3d318377-2dd2-4c2a-9669-be7055655892","csi.storage.k8s.io/pvc/name":"persistent-storage-statefulset-azuredisk-nonroot-0","csi.storage.k8s.io/pvc/namespace":"default","requestedsizegib":"10","skuName":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1675205701216-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-n9iqgmbq/providers/Microsoft.Compute/disks/pvc-3d318377-2dd2-4c2a-9669-be7055655892"} I0131 22:56:00.316176 1 azure_disk_utils.go:1068] "msg"="Label selector is: disk.csi.azure.com/node-name=k8s-agentpool1-24870519-vmss000000,disk.csi.azure.com/volume-name!=pvc-3d318377-2dd2-4c2a-9669-be7055655892." "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000000" "disk.csi.azure.com/request-id"="6e32b9f3-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-3d318377-2dd2-4c2a-9669-be7055655892" I0131 22:56:00.323798 1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-3d318377-2dd2-4c2a-9669-be7055655892-k8s-agentpool1-24870519-vmss000000-attachment) I0131 22:56:00.330604 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0131 22:56:00.330810 1 azure_disk_utils.go:1103] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000000" "disk.csi.azure.com/request-id"="6e32b9f3-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-3d318377-2dd2-4c2a-9669-be7055655892" "latency"=8649645 "number_of_net_retry"=0 "number_of_retry"=0 I0131 22:56:00.330837 1 attach_detach.go:192] "msg"="Attaching volume" "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000000" "disk.csi.azure.com/request-id"="6e32b9f3-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-3d318377-2dd2-4c2a-9669-be7055655892" I0131 22:56:00.373506 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1229 I0131 22:56:00.373736 1 cloudprovisioner.go:405] "msg"="GetDiskLun returned: -1. Initiating attaching volume \"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-n9iqgmbq/providers/Microsoft.Compute/disks/pvc-3d318377-2dd2-4c2a-9669-be7055655892\" to node \"k8s-agentpool1-24870519-vmss000000\"." "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000000" "disk.csi.azure.com/request-id"="6e32b9f3-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-3d318377-2dd2-4c2a-9669-be7055655892" I0131 22:56:00.373765 1 cloudprovisioner.go:424] "msg"="Trying to attach volume \"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-n9iqgmbq/providers/Microsoft.Compute/disks/pvc-3d318377-2dd2-4c2a-9669-be7055655892\" to node \"k8s-agentpool1-24870519-vmss000000\"." "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000000" "disk.csi.azure.com/request-id"="6e32b9f3-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-3d318377-2dd2-4c2a-9669-be7055655892" I0131 22:56:00.824314 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0131 22:56:00.830527 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0131 22:56:00.830561 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0131 22:56:00.830628 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0131 22:56:00.833958 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0131 22:56:01.351933 1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller I0131 22:56:01.373944 1 batch.go:224] "cloud-provider-azure: Delayed processing of batch due to start delay" type="batch" operation="attach_disk" key="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e|kubetest-n9iqgmbq|k8s-agentpool1-24870519-vmss000000" delay="1s" I0131 22:56:01.374055 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-n9iqgmbq): vm(k8s-agentpool1-24870519-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-n9iqgmbq/providers/Microsoft.Compute/disks/pvc-3d318377-2dd2-4c2a-9669-be7055655892:AttachDiskOptions{diskName: "pvc-3d318377-2dd2-4c2a-9669-be7055655892", lun: 0}]) I0131 22:56:01.382675 1 azure_disk_utils.go:1103] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3.4" "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000000" "disk.csi.azure.com/request-id"="6e32b9f3-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-3d318377-2dd2-4c2a-9669-be7055655892" "latency"=8405144 "number_of_net_retry"=0 "number_of_retry"=0 I0131 22:56:01.384144 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0131 22:56:01.384191 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000000" "disk.csi.azure.com/request-id"="6e32b9f3-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-3d318377-2dd2-4c2a-9669-be7055655892" "latency"=1060340425 I0131 22:56:01.384228 1 crdprovisioner.go:727] "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000000" "disk.csi.azure.com/request-id"="6e32b9f3-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLun" "disk.csi.azure.com/volume-name"="pvc-3d318377-2dd2-4c2a-9669-be7055655892" "latency"=1060436325 I0131 22:56:01.384261 1 crdprovisioner.go:486] "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000000" "disk.csi.azure.com/request-id"="6e32b9f3-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).PublishVolume" "disk.csi.azure.com/volume-name"="pvc-3d318377-2dd2-4c2a-9669-be7055655892" "latency"=1068190366 I0131 22:56:01.384297 1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=1.068254967 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-n9iqgmbq" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="test.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-n9iqgmbq/providers/Microsoft.Compute/disks/pvc-3d318377-2dd2-4c2a-9669-be7055655892" node="k8s-agentpool1-24870519-vmss000000" result_code="succeeded" I0131 22:56:01.384306 1 utils.go:85] GRPC response: {"publish_context":{"LUN":"0"}} I0131 22:56:01.384398 1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-3d318377-2dd2-4c2a-9669-be7055655892-k8s-agentpool1-24870519-vmss000000-attachment) I0131 22:56:01.551510 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-n9iqgmbq): vm(k8s-agentpool1-24870519-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-n9iqgmbq/providers/Microsoft.Compute/disks/pvc-3d318377-2dd2-4c2a-9669-be7055655892:AttachDiskOptions{diskName: "pvc-3d318377-2dd2-4c2a-9669-be7055655892", lun: 0}], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I0131 22:56:01.551550 1 azure_controller_common.go:344] azuredisk - trying to attach disks to node k8s-agentpool1-24870519-vmss000000: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-n9iqgmbq/providers/Microsoft.Compute/disks/pvc-3d318377-2dd2-4c2a-9669-be7055655892:AttachDiskOptions{diskName: "pvc-3d318377-2dd2-4c2a-9669-be7055655892", lun: 0}] I0131 22:56:03.364146 1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller I0131 22:56:05.386894 1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller I0131 22:56:07.397026 1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller I0131 22:56:09.411144 1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller I0131 22:56:11.419736 1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller I0131 22:56:11.663834 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-n9iqgmbq, k8s-agentpool1-24870519-vmss, k8s-agentpool1-24870519-vmss000000) successfully I0131 22:56:11.663871 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool1-24870519-vmss, kubetest-n9iqgmbq, k8s-agentpool1-24870519-vmss000000) for cacheKey(kubetest-n9iqgmbq/k8s-agentpool1-24870519-vmss) updated successfully I0131 22:56:11.663933 1 cloudprovisioner.go:448] "msg"="attach operation successful: volume \"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-n9iqgmbq/providers/Microsoft.Compute/disks/pvc-3d318377-2dd2-4c2a-9669-be7055655892\" attached to node \"k8s-agentpool1-24870519-vmss000000\"." "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000000" "disk.csi.azure.com/request-id"="6e32b9f3-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-3d318377-2dd2-4c2a-9669-be7055655892" I0131 22:56:11.663976 1 cloudprovisioner.go:440] "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000000" "disk.csi.azure.com/request-id"="6e32b9f3-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).PublishVolume.func3" "disk.csi.azure.com/volume-name"="pvc-3d318377-2dd2-4c2a-9669-be7055655892" "latency"=11290155601 I0131 22:56:11.664019 1 workflow.go:155] "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000000" "disk.csi.azure.com/request-id"="6e32b9f3-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).PublishVolume" "disk.csi.azure.com/volume-name"="pvc-3d318377-2dd2-4c2a-9669-be7055655892" "latency"=11333118325 I0131 22:56:11.664138 1 azure_disk_utils.go:1165] "msg"="Skip update. No update needed." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).updateVolumeAttachmentWithResult" "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000000" "disk.csi.azure.com/request-id"="6e32b9f3-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-3d318377-2dd2-4c2a-9669-be7055655892" I0131 22:56:11.664245 1 azure_disk_utils.go:1103] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).updateVolumeAttachmentWithResult" "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000000" "disk.csi.azure.com/request-id"="6e32b9f3-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-3d318377-2dd2-4c2a-9669-be7055655892" "latency"=134601 "number_of_net_retry"=0 "number_of_retry"=0 I0131 22:56:11.676487 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0131 22:56:11.676548 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000000" "disk.csi.azure.com/request-id"="6ed5bbf4-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-3d318377-2dd2-4c2a-9669-be7055655892" "latency"=10292080000 I0131 22:56:11.676593 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).ControllerPublishVolume.func2" "disk.csi.azure.com/request-id"="6ed5bbf4-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=10292221601 I0131 22:56:11.676631 1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=11.360583068 request="azuredisk_csi_driver_attach_volume_latency" resource_group="kubetest-n9iqgmbq" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="test.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-n9iqgmbq/providers/Microsoft.Compute/disks/pvc-3d318377-2dd2-4c2a-9669-be7055655892" node="k8s-agentpool1-24870519-vmss000000" result_code="succeeded" I0131 22:56:11.676659 1 controllerserver_v2.go:264] "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="6ed5bbf4-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).ControllerPublishVolume.func2" "latency"=10292300902 I0131 22:56:11.678965 1 azure_disk_utils.go:1103] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3.4" "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000000" "disk.csi.azure.com/request-id"="6e32b9f3-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-3d318377-2dd2-4c2a-9669-be7055655892" "latency"=14674377 "number_of_net_retry"=0 "number_of_retry"=0 I0131 22:56:11.678998 1 attach_detach.go:198] "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000000" "disk.csi.azure.com/request-id"="6e32b9f3-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3" "disk.csi.azure.com/volume-name"="pvc-3d318377-2dd2-4c2a-9669-be7055655892" "latency"=11348124703 ... skipping 27 lines ... I0131 22:56:22.698384 1 utils.go:78] GRPC call: /csi.v1.Controller/CreateVolume I0131 22:56:22.698411 1 utils.go:79] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"requisite":[{"segments":{"topology.test.csi.azure.com/zone":""}}]},"capacity_range":{"required_bytes":10737418240},"name":"pvc-1d7561cc-4a1b-4d53-9658-4ee8ef715c2c","parameters":{"csi.storage.k8s.io/pv/name":"pvc-1d7561cc-4a1b-4d53-9658-4ee8ef715c2c","csi.storage.k8s.io/pvc/name":"nginx-azuredisk-ephemeral-azuredisk01","csi.storage.k8s.io/pvc/namespace":"default","skuName":"StandardSSD_LRS"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]} I0131 22:56:22.698603 1 crdprovisioner.go:308] "msg"="Creating AzVolume CRI" "csi.storage.k8s.io/pv/name"="pvc-1d7561cc-4a1b-4d53-9658-4ee8ef715c2c" "disk.csi.azure.com/request-id"="7b8a06c0-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/volume-name"="pvc-1d7561cc-4a1b-4d53-9658-4ee8ef715c2c" I0131 22:56:22.712407 1 crdprovisioner.go:316] "msg"="Successfully created AzVolume CRI" "csi.storage.k8s.io/pv/name"="pvc-1d7561cc-4a1b-4d53-9658-4ee8ef715c2c" "disk.csi.azure.com/request-id"="7b8a06c0-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/volume-name"="pvc-1d7561cc-4a1b-4d53-9658-4ee8ef715c2c" I0131 22:56:22.712429 1 conditionwatcher.go:99] Adding a condition function for azvolume (pvc-1d7561cc-4a1b-4d53-9658-4ee8ef715c2c) I0131 22:56:22.719166 1 azure_disk_utils.go:1103] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate" "csi.storage.k8s.io/pv/name"="pvc-1d7561cc-4a1b-4d53-9658-4ee8ef715c2c" "disk.csi.azure.com/request-id"="7b8a06c0-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-1d7561cc-4a1b-4d53-9658-4ee8ef715c2c" "latency"=7005637 "number_of_net_retry"=0 "number_of_retry"=0 I0131 22:56:22.719181 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0131 22:56:22.719193 1 azvolume.go:158] "msg"="Creating Volume..." "csi.storage.k8s.io/pv/name"="pvc-1d7561cc-4a1b-4d53-9658-4ee8ef715c2c" "disk.csi.azure.com/request-id"="7b8a06c0-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/volume-name"="pvc-1d7561cc-4a1b-4d53-9658-4ee8ef715c2c" I0131 22:56:22.738696 1 util.go:124] Send.sendRequest got response with ContentLength 253, StatusCode 404 and responseBody length 253 I0131 22:56:22.738718 1 azure_diskclient.go:139] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-n9iqgmbq/providers/Microsoft.Compute/disks/pvc-1d7561cc-4a1b-4d53-9658-4ee8ef715c2c, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-1d7561cc-4a1b-4d53-9658-4ee8ef715c2c' under resource group 'kubetest-n9iqgmbq' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}} I0131 22:56:22.738764 1 cloudprovisioner.go:246] begin to create disk(pvc-1d7561cc-4a1b-4d53-9658-4ee8ef715c2c) account type(StandardSSD_LRS) rg(kubetest-n9iqgmbq) location() size(10) selectedAvailabilityZone() maxShares(0) I0131 22:56:22.779422 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 8264 I0131 22:56:22.779698 1 azure_managedDiskController.go:94] azureDisk - creating new managed Name:pvc-1d7561cc-4a1b-4d53-9658-4ee8ef715c2c StorageAccountType:StandardSSD_LRS Size:10 I0131 22:56:23.486808 1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller I0131 22:56:25.063770 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1217 I0131 22:56:25.063940 1 azure_managedDiskController.go:280] azureDisk - created new MD Name:pvc-1d7561cc-4a1b-4d53-9658-4ee8ef715c2c StorageAccountType:StandardSSD_LRS Size:10 I0131 22:56:25.064000 1 cloudprovisioner.go:320] "msg"="create disk(pvc-1d7561cc-4a1b-4d53-9658-4ee8ef715c2c) account type(StandardSSD_LRS) rg(kubetest-n9iqgmbq) location() size(10) tags(map[kubernetes.io-created-for-pv-name:pvc-1d7561cc-4a1b-4d53-9658-4ee8ef715c2c kubernetes.io-created-for-pvc-name:nginx-azuredisk-ephemeral-azuredisk01 kubernetes.io-created-for-pvc-namespace:default]) successfully" "csi.storage.k8s.io/pv/name"="pvc-1d7561cc-4a1b-4d53-9658-4ee8ef715c2c" "disk.csi.azure.com/request-id"="7b8a06c0-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/volume-name"="pvc-1d7561cc-4a1b-4d53-9658-4ee8ef715c2c" I0131 22:56:25.064038 1 cloudprovisioner.go:142] "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-1d7561cc-4a1b-4d53-9658-4ee8ef715c2c" "disk.csi.azure.com/request-id"="7b8a06c0-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-1d7561cc-4a1b-4d53-9658-4ee8ef715c2c" "latency"=2344787734 I0131 22:56:25.076335 1 azure_disk_utils.go:1103] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "csi.storage.k8s.io/pv/name"="pvc-1d7561cc-4a1b-4d53-9658-4ee8ef715c2c" "disk.csi.azure.com/request-id"="7b8a06c0-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-1d7561cc-4a1b-4d53-9658-4ee8ef715c2c" "latency"=12240464 "number_of_net_retry"=0 "number_of_retry"=0 I0131 22:56:25.076373 1 azvolume.go:165] "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-1d7561cc-4a1b-4d53-9658-4ee8ef715c2c" "disk.csi.azure.com/request-id"="7b8a06c0-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "disk.csi.azure.com/volume-name"="pvc-1d7561cc-4a1b-4d53-9658-4ee8ef715c2c" "latency"=2357151498 I0131 22:56:25.076411 1 workflow.go:155] "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-1d7561cc-4a1b-4d53-9658-4ee8ef715c2c" "disk.csi.azure.com/request-id"="7b8a06c0-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate" "disk.csi.azure.com/volume-name"="pvc-1d7561cc-4a1b-4d53-9658-4ee8ef715c2c" "latency"=2364254435 I0131 22:56:25.077151 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0131 22:56:25.077206 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).CreateVolume" "csi.storage.k8s.io/pv/name"="pvc-1d7561cc-4a1b-4d53-9658-4ee8ef715c2c" "disk.csi.azure.com/request-id"="7b8a06c0-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-1d7561cc-4a1b-4d53-9658-4ee8ef715c2c" "latency"=2364722637 I0131 22:56:25.077245 1 crdprovisioner.go:228] "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-1d7561cc-4a1b-4d53-9658-4ee8ef715c2c" "disk.csi.azure.com/request-id"="7b8a06c0-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-1d7561cc-4a1b-4d53-9658-4ee8ef715c2c" "latency"=2378659410 I0131 22:56:25.077279 1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=2.37870241 request="azuredisk_csi_driver_controller_create_volume" resource_group="kubetest-n9iqgmbq" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="test.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-n9iqgmbq/providers/Microsoft.Compute/disks/pvc-1d7561cc-4a1b-4d53-9658-4ee8ef715c2c" result_code="succeeded" I0131 22:56:25.077298 1 utils.go:85] GRPC response: {"volume":{"accessible_topology":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"capacity_bytes":10737418240,"content_source":{"Type":{"Volume":{}}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-1d7561cc-4a1b-4d53-9658-4ee8ef715c2c","csi.storage.k8s.io/pvc/name":"nginx-azuredisk-ephemeral-azuredisk01","csi.storage.k8s.io/pvc/namespace":"default","requestedsizegib":"10","skuName":"StandardSSD_LRS"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-n9iqgmbq/providers/Microsoft.Compute/disks/pvc-1d7561cc-4a1b-4d53-9658-4ee8ef715c2c"}} I0131 22:56:25.499269 1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller I0131 22:56:25.786982 1 utils.go:78] GRPC call: /csi.v1.Controller/ControllerPublishVolume I0131 22:56:25.787001 1 utils.go:79] GRPC request: {"node_id":"k8s-agentpool1-24870519-vmss000002","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-1d7561cc-4a1b-4d53-9658-4ee8ef715c2c","csi.storage.k8s.io/pvc/name":"nginx-azuredisk-ephemeral-azuredisk01","csi.storage.k8s.io/pvc/namespace":"default","requestedsizegib":"10","skuName":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1675205701216-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-n9iqgmbq/providers/Microsoft.Compute/disks/pvc-1d7561cc-4a1b-4d53-9658-4ee8ef715c2c"} I0131 22:56:25.787262 1 azure_disk_utils.go:1068] "msg"="Label selector is: disk.csi.azure.com/node-name=k8s-agentpool1-24870519-vmss000002,disk.csi.azure.com/volume-name!=pvc-1d7561cc-4a1b-4d53-9658-4ee8ef715c2c." "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000002" "disk.csi.azure.com/request-id"="7d614e3d-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-1d7561cc-4a1b-4d53-9658-4ee8ef715c2c" I0131 22:56:25.793982 1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-1d7561cc-4a1b-4d53-9658-4ee8ef715c2c-k8s-agentpool1-24870519-vmss000002-attachment) I0131 22:56:25.799853 1 azure_disk_utils.go:1103] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000002" "disk.csi.azure.com/request-id"="7d614e3d-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-1d7561cc-4a1b-4d53-9658-4ee8ef715c2c" "latency"=6710735 "number_of_net_retry"=0 "number_of_retry"=0 I0131 22:56:25.799881 1 attach_detach.go:192] "msg"="Attaching volume" "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000002" "disk.csi.azure.com/request-id"="7d614e3d-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-1d7561cc-4a1b-4d53-9658-4ee8ef715c2c" I0131 22:56:25.800959 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0131 22:56:25.823785 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1217 I0131 22:56:25.824003 1 cloudprovisioner.go:405] "msg"="GetDiskLun returned: -1. Initiating attaching volume \"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-n9iqgmbq/providers/Microsoft.Compute/disks/pvc-1d7561cc-4a1b-4d53-9658-4ee8ef715c2c\" to node \"k8s-agentpool1-24870519-vmss000002\"." "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000002" "disk.csi.azure.com/request-id"="7d614e3d-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-1d7561cc-4a1b-4d53-9658-4ee8ef715c2c" I0131 22:56:25.824046 1 cloudprovisioner.go:424] "msg"="Trying to attach volume \"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-n9iqgmbq/providers/Microsoft.Compute/disks/pvc-1d7561cc-4a1b-4d53-9658-4ee8ef715c2c\" to node \"k8s-agentpool1-24870519-vmss000002\"." "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000002" "disk.csi.azure.com/request-id"="7d614e3d-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-1d7561cc-4a1b-4d53-9658-4ee8ef715c2c" I0131 22:56:26.824429 1 batch.go:224] "cloud-provider-azure: Delayed processing of batch due to start delay" type="batch" operation="attach_disk" key="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e|kubetest-n9iqgmbq|k8s-agentpool1-24870519-vmss000002" delay="1s" I0131 22:56:26.824546 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-n9iqgmbq): vm(k8s-agentpool1-24870519-vmss000002) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-n9iqgmbq/providers/Microsoft.Compute/disks/pvc-1d7561cc-4a1b-4d53-9658-4ee8ef715c2c:AttachDiskOptions{diskName: "pvc-1d7561cc-4a1b-4d53-9658-4ee8ef715c2c", lun: 1}]) I0131 22:56:26.834790 1 azure_disk_utils.go:1103] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3.4" "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000002" "disk.csi.azure.com/request-id"="7d614e3d-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-1d7561cc-4a1b-4d53-9658-4ee8ef715c2c" "latency"=10077953 "number_of_net_retry"=0 "number_of_retry"=0 I0131 22:56:26.835594 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0131 22:56:26.835652 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000002" "disk.csi.azure.com/request-id"="7d614e3d-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-1d7561cc-4a1b-4d53-9658-4ee8ef715c2c" "latency"=1041612634 I0131 22:56:26.835684 1 crdprovisioner.go:727] "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000002" "disk.csi.azure.com/request-id"="7d614e3d-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLun" "disk.csi.azure.com/volume-name"="pvc-1d7561cc-4a1b-4d53-9658-4ee8ef715c2c" "latency"=1041714935 I0131 22:56:26.835706 1 crdprovisioner.go:486] "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000002" "disk.csi.azure.com/request-id"="7d614e3d-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).PublishVolume" "disk.csi.azure.com/volume-name"="pvc-1d7561cc-4a1b-4d53-9658-4ee8ef715c2c" "latency"=1048551770 I0131 22:56:26.835750 1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=1.048613571 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-n9iqgmbq" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="test.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-n9iqgmbq/providers/Microsoft.Compute/disks/pvc-1d7561cc-4a1b-4d53-9658-4ee8ef715c2c" node="k8s-agentpool1-24870519-vmss000002" result_code="succeeded" I0131 22:56:26.835765 1 utils.go:85] GRPC response: {"publish_context":{"LUN":"1"}} I0131 22:56:26.835896 1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-1d7561cc-4a1b-4d53-9658-4ee8ef715c2c-k8s-agentpool1-24870519-vmss000002-attachment) I0131 22:56:26.999201 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-n9iqgmbq): vm(k8s-agentpool1-24870519-vmss000002) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-n9iqgmbq/providers/Microsoft.Compute/disks/pvc-1d7561cc-4a1b-4d53-9658-4ee8ef715c2c:AttachDiskOptions{diskName: "pvc-1d7561cc-4a1b-4d53-9658-4ee8ef715c2c", lun: 1}], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I0131 22:56:26.999242 1 azure_controller_common.go:344] azuredisk - trying to attach disks to node k8s-agentpool1-24870519-vmss000002: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-n9iqgmbq/providers/Microsoft.Compute/disks/pvc-1d7561cc-4a1b-4d53-9658-4ee8ef715c2c:AttachDiskOptions{diskName: "pvc-1d7561cc-4a1b-4d53-9658-4ee8ef715c2c", lun: 1}] I0131 22:56:27.510945 1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller I0131 22:56:29.525037 1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller I0131 22:56:30.824551 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0131 22:56:30.830743 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0131 22:56:30.830780 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0131 22:56:30.830851 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0131 22:56:30.835123 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0131 22:56:31.532941 1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller I0131 22:56:33.545729 1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller I0131 22:56:35.560111 1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller I0131 22:56:37.100131 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-n9iqgmbq, k8s-agentpool1-24870519-vmss, k8s-agentpool1-24870519-vmss000002) successfully I0131 22:56:37.100167 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool1-24870519-vmss, kubetest-n9iqgmbq, k8s-agentpool1-24870519-vmss000002) for cacheKey(kubetest-n9iqgmbq/k8s-agentpool1-24870519-vmss) updated successfully I0131 22:56:37.100248 1 cloudprovisioner.go:448] "msg"="attach operation successful: volume \"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-n9iqgmbq/providers/Microsoft.Compute/disks/pvc-1d7561cc-4a1b-4d53-9658-4ee8ef715c2c\" attached to node \"k8s-agentpool1-24870519-vmss000002\"." "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000002" "disk.csi.azure.com/request-id"="7d614e3d-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-1d7561cc-4a1b-4d53-9658-4ee8ef715c2c" I0131 22:56:37.100310 1 cloudprovisioner.go:440] "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000002" "disk.csi.azure.com/request-id"="7d614e3d-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).PublishVolume.func3" "disk.csi.azure.com/volume-name"="pvc-1d7561cc-4a1b-4d53-9658-4ee8ef715c2c" "latency"=11276181703 I0131 22:56:37.100347 1 workflow.go:155] "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000002" "disk.csi.azure.com/request-id"="7d614e3d-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).PublishVolume" "disk.csi.azure.com/volume-name"="pvc-1d7561cc-4a1b-4d53-9658-4ee8ef715c2c" "latency"=11300407929 I0131 22:56:37.100486 1 azure_disk_utils.go:1165] "msg"="Skip update. No update needed." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).updateVolumeAttachmentWithResult" "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000002" "disk.csi.azure.com/request-id"="7d614e3d-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-1d7561cc-4a1b-4d53-9658-4ee8ef715c2c" I0131 22:56:37.100538 1 azure_disk_utils.go:1103] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).updateVolumeAttachmentWithResult" "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000002" "disk.csi.azure.com/request-id"="7d614e3d-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-1d7561cc-4a1b-4d53-9658-4ee8ef715c2c" "latency"=126800 "number_of_net_retry"=0 "number_of_retry"=0 I0131 22:56:37.114124 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0131 22:56:37.114194 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000002" "disk.csi.azure.com/request-id"="7e0152d3-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-1d7561cc-4a1b-4d53-9658-4ee8ef715c2c" "latency"=10278214796 I0131 22:56:37.114231 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).ControllerPublishVolume.func2" "disk.csi.azure.com/request-id"="7e0152d3-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=10278361597 I0131 22:56:37.114272 1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=11.327133869 request="azuredisk_csi_driver_attach_volume_latency" resource_group="kubetest-n9iqgmbq" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="test.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-n9iqgmbq/providers/Microsoft.Compute/disks/pvc-1d7561cc-4a1b-4d53-9658-4ee8ef715c2c" node="k8s-agentpool1-24870519-vmss000002" result_code="succeeded" I0131 22:56:37.114303 1 controllerserver_v2.go:264] "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="7e0152d3-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).ControllerPublishVolume.func2" "latency"=10278447397 I0131 22:56:37.115852 1 azure_disk_utils.go:1103] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3.4" "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000002" "disk.csi.azure.com/request-id"="7d614e3d-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-1d7561cc-4a1b-4d53-9658-4ee8ef715c2c" "latency"=15269079 "number_of_net_retry"=0 "number_of_retry"=0 I0131 22:56:37.115894 1 attach_detach.go:198] "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000002" "disk.csi.azure.com/request-id"="7d614e3d-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3" "disk.csi.azure.com/volume-name"="pvc-1d7561cc-4a1b-4d53-9658-4ee8ef715c2c" "latency"=11315972210 ... skipping 26 lines ... I0131 22:56:40.377080 1 shared_state.go:489] "msg"="Storing pod daemonset-azuredisk-ephemeral-phdkd and claim [] to podToClaimsMap map." I0131 22:56:40.402455 1 utils.go:78] GRPC call: /csi.v1.Controller/CreateVolume I0131 22:56:40.402474 1 utils.go:79] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"requisite":[{"segments":{"topology.test.csi.azure.com/zone":""}}]},"capacity_range":{"required_bytes":10737418240},"name":"pvc-dec955da-85f4-45eb-8dd0-53173daa5046","parameters":{"csi.storage.k8s.io/pv/name":"pvc-dec955da-85f4-45eb-8dd0-53173daa5046","csi.storage.k8s.io/pvc/name":"daemonset-azuredisk-ephemeral-phdkd-azuredisk","csi.storage.k8s.io/pvc/namespace":"default","skuName":"StandardSSD_LRS"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]} I0131 22:56:40.402676 1 crdprovisioner.go:308] "msg"="Creating AzVolume CRI" "csi.storage.k8s.io/pv/name"="pvc-dec955da-85f4-45eb-8dd0-53173daa5046" "disk.csi.azure.com/request-id"="86177420-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/volume-name"="pvc-dec955da-85f4-45eb-8dd0-53173daa5046" I0131 22:56:40.409397 1 crdprovisioner.go:316] "msg"="Successfully created AzVolume CRI" "csi.storage.k8s.io/pv/name"="pvc-dec955da-85f4-45eb-8dd0-53173daa5046" "disk.csi.azure.com/request-id"="86177420-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/volume-name"="pvc-dec955da-85f4-45eb-8dd0-53173daa5046" I0131 22:56:40.409417 1 conditionwatcher.go:99] Adding a condition function for azvolume (pvc-dec955da-85f4-45eb-8dd0-53173daa5046) I0131 22:56:40.410184 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0131 22:56:40.416634 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0131 22:56:40.417759 1 azure_disk_utils.go:1103] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate" "csi.storage.k8s.io/pv/name"="pvc-dec955da-85f4-45eb-8dd0-53173daa5046" "disk.csi.azure.com/request-id"="86177420-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-dec955da-85f4-45eb-8dd0-53173daa5046" "latency"=7343139 "number_of_net_retry"=0 "number_of_retry"=0 I0131 22:56:40.417788 1 azvolume.go:158] "msg"="Creating Volume..." "csi.storage.k8s.io/pv/name"="pvc-dec955da-85f4-45eb-8dd0-53173daa5046" "disk.csi.azure.com/request-id"="86177420-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/volume-name"="pvc-dec955da-85f4-45eb-8dd0-53173daa5046" I0131 22:56:40.436202 1 util.go:124] Send.sendRequest got response with ContentLength 253, StatusCode 404 and responseBody length 253 I0131 22:56:40.436257 1 azure_diskclient.go:139] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-n9iqgmbq/providers/Microsoft.Compute/disks/pvc-dec955da-85f4-45eb-8dd0-53173daa5046, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-dec955da-85f4-45eb-8dd0-53173daa5046' under resource group 'kubetest-n9iqgmbq' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}} I0131 22:56:40.436371 1 cloudprovisioner.go:246] begin to create disk(pvc-dec955da-85f4-45eb-8dd0-53173daa5046) account type(StandardSSD_LRS) rg(kubetest-n9iqgmbq) location() size(10) selectedAvailabilityZone() maxShares(0) I0131 22:56:40.477475 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 9854 I0131 22:56:40.477749 1 azure_managedDiskController.go:94] azureDisk - creating new managed Name:pvc-dec955da-85f4-45eb-8dd0-53173daa5046 StorageAccountType:StandardSSD_LRS Size:10 I0131 22:56:41.587332 1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller I0131 22:56:41.737933 1 utils.go:78] GRPC call: /csi.v1.Controller/CreateVolume I0131 22:56:41.737954 1 utils.go:79] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"requisite":[{"segments":{"topology.test.csi.azure.com/zone":""}}]},"capacity_range":{"required_bytes":10737418240},"name":"pvc-132f85ce-a89d-44fa-b5b7-a3c93612a97e","parameters":{"csi.storage.k8s.io/pv/name":"pvc-132f85ce-a89d-44fa-b5b7-a3c93612a97e","csi.storage.k8s.io/pvc/name":"daemonset-azuredisk-ephemeral-tztwc-azuredisk","csi.storage.k8s.io/pvc/namespace":"default","skuName":"StandardSSD_LRS"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]} ... skipping 2 lines ... I0131 22:56:41.745368 1 utils.go:79] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"requisite":[{"segments":{"topology.test.csi.azure.com/zone":""}}]},"capacity_range":{"required_bytes":10737418240},"name":"pvc-8978ac4e-7409-46bd-9fa7-5c789f516a04","parameters":{"csi.storage.k8s.io/pv/name":"pvc-8978ac4e-7409-46bd-9fa7-5c789f516a04","csi.storage.k8s.io/pvc/name":"daemonset-azuredisk-ephemeral-qzznj-azuredisk","csi.storage.k8s.io/pvc/namespace":"default","skuName":"StandardSSD_LRS"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]} I0131 22:56:41.745557 1 crdprovisioner.go:308] "msg"="Creating AzVolume CRI" "csi.storage.k8s.io/pv/name"="pvc-8978ac4e-7409-46bd-9fa7-5c789f516a04" "disk.csi.azure.com/request-id"="86e45c3d-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/volume-name"="pvc-8978ac4e-7409-46bd-9fa7-5c789f516a04" I0131 22:56:41.756670 1 crdprovisioner.go:316] "msg"="Successfully created AzVolume CRI" "csi.storage.k8s.io/pv/name"="pvc-8978ac4e-7409-46bd-9fa7-5c789f516a04" "disk.csi.azure.com/request-id"="86e45c3d-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/volume-name"="pvc-8978ac4e-7409-46bd-9fa7-5c789f516a04" I0131 22:56:41.756690 1 conditionwatcher.go:99] Adding a condition function for azvolume (pvc-8978ac4e-7409-46bd-9fa7-5c789f516a04) I0131 22:56:41.759302 1 crdprovisioner.go:316] "msg"="Successfully created AzVolume CRI" "csi.storage.k8s.io/pv/name"="pvc-132f85ce-a89d-44fa-b5b7-a3c93612a97e" "disk.csi.azure.com/request-id"="86e33a95-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/volume-name"="pvc-132f85ce-a89d-44fa-b5b7-a3c93612a97e" I0131 22:56:41.759356 1 conditionwatcher.go:99] Adding a condition function for azvolume (pvc-132f85ce-a89d-44fa-b5b7-a3c93612a97e) I0131 22:56:41.761180 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0131 22:56:41.761290 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0131 22:56:41.770694 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0131 22:56:41.772233 1 azure_disk_utils.go:1103] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate" "csi.storage.k8s.io/pv/name"="pvc-8978ac4e-7409-46bd-9fa7-5c789f516a04" "disk.csi.azure.com/request-id"="86e45c3d-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-8978ac4e-7409-46bd-9fa7-5c789f516a04" "latency"=10812456 "number_of_net_retry"=0 "number_of_retry"=0 I0131 22:56:41.772312 1 azvolume.go:158] "msg"="Creating Volume..." "csi.storage.k8s.io/pv/name"="pvc-8978ac4e-7409-46bd-9fa7-5c789f516a04" "disk.csi.azure.com/request-id"="86e45c3d-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/volume-name"="pvc-8978ac4e-7409-46bd-9fa7-5c789f516a04" I0131 22:56:41.783479 1 azure_disk_utils.go:1103] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate" "csi.storage.k8s.io/pv/name"="pvc-132f85ce-a89d-44fa-b5b7-a3c93612a97e" "disk.csi.azure.com/request-id"="86e33a95-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-132f85ce-a89d-44fa-b5b7-a3c93612a97e" "latency"=22653118 "number_of_net_retry"=0 "number_of_retry"=0 I0131 22:56:41.783507 1 azvolume.go:158] "msg"="Creating Volume..." "csi.storage.k8s.io/pv/name"="pvc-132f85ce-a89d-44fa-b5b7-a3c93612a97e" "disk.csi.azure.com/request-id"="86e33a95-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/volume-name"="pvc-132f85ce-a89d-44fa-b5b7-a3c93612a97e" I0131 22:56:41.783661 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0131 22:56:41.790654 1 util.go:124] Send.sendRequest got response with ContentLength 253, StatusCode 404 and responseBody length 253 I0131 22:56:41.790671 1 azure_diskclient.go:139] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-n9iqgmbq/providers/Microsoft.Compute/disks/pvc-8978ac4e-7409-46bd-9fa7-5c789f516a04, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-8978ac4e-7409-46bd-9fa7-5c789f516a04' under resource group 'kubetest-n9iqgmbq' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}} I0131 22:56:41.790716 1 cloudprovisioner.go:246] begin to create disk(pvc-8978ac4e-7409-46bd-9fa7-5c789f516a04) account type(StandardSSD_LRS) rg(kubetest-n9iqgmbq) location() size(10) selectedAvailabilityZone() maxShares(0) I0131 22:56:41.808291 1 util.go:124] Send.sendRequest got response with ContentLength 253, StatusCode 404 and responseBody length 253 I0131 22:56:41.808310 1 azure_diskclient.go:139] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-n9iqgmbq/providers/Microsoft.Compute/disks/pvc-132f85ce-a89d-44fa-b5b7-a3c93612a97e, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-132f85ce-a89d-44fa-b5b7-a3c93612a97e' under resource group 'kubetest-n9iqgmbq' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}} I0131 22:56:41.808358 1 cloudprovisioner.go:246] begin to create disk(pvc-132f85ce-a89d-44fa-b5b7-a3c93612a97e) account type(StandardSSD_LRS) rg(kubetest-n9iqgmbq) location() size(10) selectedAvailabilityZone() maxShares(0) I0131 22:56:41.866073 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 11218 I0131 22:56:41.866402 1 azure_managedDiskController.go:94] azureDisk - creating new managed Name:pvc-132f85ce-a89d-44fa-b5b7-a3c93612a97e StorageAccountType:StandardSSD_LRS Size:10 I0131 22:56:41.992342 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 11218 I0131 22:56:41.992663 1 azure_managedDiskController.go:94] azureDisk - creating new managed Name:pvc-8978ac4e-7409-46bd-9fa7-5c789f516a04 StorageAccountType:StandardSSD_LRS Size:10 I0131 22:56:42.779767 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1225 I0131 22:56:42.780008 1 azure_managedDiskController.go:280] azureDisk - created new MD Name:pvc-dec955da-85f4-45eb-8dd0-53173daa5046 StorageAccountType:StandardSSD_LRS Size:10 I0131 22:56:42.780075 1 cloudprovisioner.go:320] "msg"="create disk(pvc-dec955da-85f4-45eb-8dd0-53173daa5046) account type(StandardSSD_LRS) rg(kubetest-n9iqgmbq) location() size(10) tags(map[kubernetes.io-created-for-pv-name:pvc-dec955da-85f4-45eb-8dd0-53173daa5046 kubernetes.io-created-for-pvc-name:daemonset-azuredisk-ephemeral-phdkd-azuredisk kubernetes.io-created-for-pvc-namespace:default]) successfully" "csi.storage.k8s.io/pv/name"="pvc-dec955da-85f4-45eb-8dd0-53173daa5046" "disk.csi.azure.com/request-id"="86177420-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/volume-name"="pvc-dec955da-85f4-45eb-8dd0-53173daa5046" I0131 22:56:42.780116 1 cloudprovisioner.go:142] "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-dec955da-85f4-45eb-8dd0-53173daa5046" "disk.csi.azure.com/request-id"="86177420-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-dec955da-85f4-45eb-8dd0-53173daa5046" "latency"=2362258286 I0131 22:56:42.790314 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0131 22:56:42.790364 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).CreateVolume" "csi.storage.k8s.io/pv/name"="pvc-dec955da-85f4-45eb-8dd0-53173daa5046" "disk.csi.azure.com/request-id"="86177420-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-dec955da-85f4-45eb-8dd0-53173daa5046" "latency"=2380894183 I0131 22:56:42.790386 1 crdprovisioner.go:228] "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-dec955da-85f4-45eb-8dd0-53173daa5046" "disk.csi.azure.com/request-id"="86177420-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-dec955da-85f4-45eb-8dd0-53173daa5046" "latency"=2387734918 I0131 22:56:42.790426 1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=2.387779018 request="azuredisk_csi_driver_controller_create_volume" resource_group="kubetest-n9iqgmbq" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="test.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-n9iqgmbq/providers/Microsoft.Compute/disks/pvc-dec955da-85f4-45eb-8dd0-53173daa5046" result_code="succeeded" I0131 22:56:42.790444 1 utils.go:85] GRPC response: {"volume":{"accessible_topology":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"capacity_bytes":10737418240,"content_source":{"Type":{"Volume":{}}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-dec955da-85f4-45eb-8dd0-53173daa5046","csi.storage.k8s.io/pvc/name":"daemonset-azuredisk-ephemeral-phdkd-azuredisk","csi.storage.k8s.io/pvc/namespace":"default","requestedsizegib":"10","skuName":"StandardSSD_LRS"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-n9iqgmbq/providers/Microsoft.Compute/disks/pvc-dec955da-85f4-45eb-8dd0-53173daa5046"}} I0131 22:56:42.791646 1 azure_disk_utils.go:1103] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "csi.storage.k8s.io/pv/name"="pvc-dec955da-85f4-45eb-8dd0-53173daa5046" "disk.csi.azure.com/request-id"="86177420-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-dec955da-85f4-45eb-8dd0-53173daa5046" "latency"=11466260 "number_of_net_retry"=0 "number_of_retry"=0 I0131 22:56:42.791684 1 azvolume.go:165] "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-dec955da-85f4-45eb-8dd0-53173daa5046" "disk.csi.azure.com/request-id"="86177420-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "disk.csi.azure.com/volume-name"="pvc-dec955da-85f4-45eb-8dd0-53173daa5046" "latency"=2373855846 I0131 22:56:42.791705 1 workflow.go:155] "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-dec955da-85f4-45eb-8dd0-53173daa5046" "disk.csi.azure.com/request-id"="86177420-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate" "disk.csi.azure.com/volume-name"="pvc-dec955da-85f4-45eb-8dd0-53173daa5046" "latency"=2381307685 I0131 22:56:43.508638 1 utils.go:78] GRPC call: /csi.v1.Controller/ControllerPublishVolume I0131 22:56:43.508664 1 utils.go:79] GRPC request: {"node_id":"k8s-agentpool1-24870519-vmss000000","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-dec955da-85f4-45eb-8dd0-53173daa5046","csi.storage.k8s.io/pvc/name":"daemonset-azuredisk-ephemeral-phdkd-azuredisk","csi.storage.k8s.io/pvc/namespace":"default","requestedsizegib":"10","skuName":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1675205701216-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-n9iqgmbq/providers/Microsoft.Compute/disks/pvc-dec955da-85f4-45eb-8dd0-53173daa5046"} I0131 22:56:43.508903 1 azure_disk_utils.go:1068] "msg"="Label selector is: disk.csi.azure.com/node-name=k8s-agentpool1-24870519-vmss000000,disk.csi.azure.com/volume-name!=pvc-dec955da-85f4-45eb-8dd0-53173daa5046." "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000000" "disk.csi.azure.com/request-id"="87f16a39-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-dec955da-85f4-45eb-8dd0-53173daa5046" I0131 22:56:43.516395 1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-dec955da-85f4-45eb-8dd0-53173daa5046-k8s-agentpool1-24870519-vmss000000-attachment) I0131 22:56:43.521106 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0131 22:56:43.521287 1 azure_disk_utils.go:1103] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000000" "disk.csi.azure.com/request-id"="87f16a39-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-dec955da-85f4-45eb-8dd0-53173daa5046" "latency"=6750434 "number_of_net_retry"=0 "number_of_retry"=0 I0131 22:56:43.521320 1 attach_detach.go:192] "msg"="Attaching volume" "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000000" "disk.csi.azure.com/request-id"="87f16a39-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-dec955da-85f4-45eb-8dd0-53173daa5046" I0131 22:56:43.543064 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1225 I0131 22:56:43.543285 1 cloudprovisioner.go:405] "msg"="GetDiskLun returned: -1. Initiating attaching volume \"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-n9iqgmbq/providers/Microsoft.Compute/disks/pvc-dec955da-85f4-45eb-8dd0-53173daa5046\" to node \"k8s-agentpool1-24870519-vmss000000\"." "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000000" "disk.csi.azure.com/request-id"="87f16a39-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-dec955da-85f4-45eb-8dd0-53173daa5046" I0131 22:56:43.543319 1 cloudprovisioner.go:424] "msg"="Trying to attach volume \"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-n9iqgmbq/providers/Microsoft.Compute/disks/pvc-dec955da-85f4-45eb-8dd0-53173daa5046\" to node \"k8s-agentpool1-24870519-vmss000000\"." "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000000" "disk.csi.azure.com/request-id"="87f16a39-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-dec955da-85f4-45eb-8dd0-53173daa5046" I0131 22:56:43.598815 1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller I0131 22:56:44.148441 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1225 I0131 22:56:44.148700 1 azure_managedDiskController.go:280] azureDisk - created new MD Name:pvc-132f85ce-a89d-44fa-b5b7-a3c93612a97e StorageAccountType:StandardSSD_LRS Size:10 I0131 22:56:44.148767 1 cloudprovisioner.go:320] "msg"="create disk(pvc-132f85ce-a89d-44fa-b5b7-a3c93612a97e) account type(StandardSSD_LRS) rg(kubetest-n9iqgmbq) location() size(10) tags(map[kubernetes.io-created-for-pv-name:pvc-132f85ce-a89d-44fa-b5b7-a3c93612a97e kubernetes.io-created-for-pvc-name:daemonset-azuredisk-ephemeral-tztwc-azuredisk kubernetes.io-created-for-pvc-namespace:default]) successfully" "csi.storage.k8s.io/pv/name"="pvc-132f85ce-a89d-44fa-b5b7-a3c93612a97e" "disk.csi.azure.com/request-id"="86e33a95-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/volume-name"="pvc-132f85ce-a89d-44fa-b5b7-a3c93612a97e" I0131 22:56:44.148814 1 cloudprovisioner.go:142] "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-132f85ce-a89d-44fa-b5b7-a3c93612a97e" "disk.csi.azure.com/request-id"="86e33a95-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-132f85ce-a89d-44fa-b5b7-a3c93612a97e" "latency"=2365223742 I0131 22:56:44.159733 1 azure_disk_utils.go:1103] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "csi.storage.k8s.io/pv/name"="pvc-132f85ce-a89d-44fa-b5b7-a3c93612a97e" "disk.csi.azure.com/request-id"="86e33a95-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-132f85ce-a89d-44fa-b5b7-a3c93612a97e" "latency"=10859356 "number_of_net_retry"=0 "number_of_retry"=0 I0131 22:56:44.159788 1 azvolume.go:165] "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-132f85ce-a89d-44fa-b5b7-a3c93612a97e" "disk.csi.azure.com/request-id"="86e33a95-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "disk.csi.azure.com/volume-name"="pvc-132f85ce-a89d-44fa-b5b7-a3c93612a97e" "latency"=2376227399 I0131 22:56:44.159815 1 workflow.go:155] "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-132f85ce-a89d-44fa-b5b7-a3c93612a97e" "disk.csi.azure.com/request-id"="86e33a95-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate" "disk.csi.azure.com/volume-name"="pvc-132f85ce-a89d-44fa-b5b7-a3c93612a97e" "latency"=2398999518 I0131 22:56:44.159993 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0131 22:56:44.160048 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).CreateVolume" "csi.storage.k8s.io/pv/name"="pvc-132f85ce-a89d-44fa-b5b7-a3c93612a97e" "disk.csi.azure.com/request-id"="86e33a95-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-132f85ce-a89d-44fa-b5b7-a3c93612a97e" "latency"=2400601426 I0131 22:56:44.160071 1 crdprovisioner.go:228] "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-132f85ce-a89d-44fa-b5b7-a3c93612a97e" "disk.csi.azure.com/request-id"="86e33a95-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-132f85ce-a89d-44fa-b5b7-a3c93612a97e" "latency"=2421958837 I0131 22:56:44.160099 1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=2.421994737 request="azuredisk_csi_driver_controller_create_volume" resource_group="kubetest-n9iqgmbq" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="test.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-n9iqgmbq/providers/Microsoft.Compute/disks/pvc-132f85ce-a89d-44fa-b5b7-a3c93612a97e" result_code="succeeded" I0131 22:56:44.160109 1 utils.go:85] GRPC response: {"volume":{"accessible_topology":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"capacity_bytes":10737418240,"content_source":{"Type":{"Volume":{}}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-132f85ce-a89d-44fa-b5b7-a3c93612a97e","csi.storage.k8s.io/pvc/name":"daemonset-azuredisk-ephemeral-tztwc-azuredisk","csi.storage.k8s.io/pvc/namespace":"default","requestedsizegib":"10","skuName":"StandardSSD_LRS"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-n9iqgmbq/providers/Microsoft.Compute/disks/pvc-132f85ce-a89d-44fa-b5b7-a3c93612a97e"}} I0131 22:56:44.397451 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1225 I0131 22:56:44.397654 1 azure_managedDiskController.go:280] azureDisk - created new MD Name:pvc-8978ac4e-7409-46bd-9fa7-5c789f516a04 StorageAccountType:StandardSSD_LRS Size:10 I0131 22:56:44.397722 1 cloudprovisioner.go:320] "msg"="create disk(pvc-8978ac4e-7409-46bd-9fa7-5c789f516a04) account type(StandardSSD_LRS) rg(kubetest-n9iqgmbq) location() size(10) tags(map[kubernetes.io-created-for-pv-name:pvc-8978ac4e-7409-46bd-9fa7-5c789f516a04 kubernetes.io-created-for-pvc-name:daemonset-azuredisk-ephemeral-qzznj-azuredisk kubernetes.io-created-for-pvc-namespace:default]) successfully" "csi.storage.k8s.io/pv/name"="pvc-8978ac4e-7409-46bd-9fa7-5c789f516a04" "disk.csi.azure.com/request-id"="86e45c3d-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/volume-name"="pvc-8978ac4e-7409-46bd-9fa7-5c789f516a04" I0131 22:56:44.397756 1 cloudprovisioner.go:142] "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-8978ac4e-7409-46bd-9fa7-5c789f516a04" "disk.csi.azure.com/request-id"="86e45c3d-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-8978ac4e-7409-46bd-9fa7-5c789f516a04" "latency"=2625183486 I0131 22:56:44.406383 1 azure_disk_utils.go:1103] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "csi.storage.k8s.io/pv/name"="pvc-8978ac4e-7409-46bd-9fa7-5c789f516a04" "disk.csi.azure.com/request-id"="86e45c3d-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-8978ac4e-7409-46bd-9fa7-5c789f516a04" "latency"=8563644 "number_of_net_retry"=0 "number_of_retry"=0 I0131 22:56:44.406436 1 azvolume.go:165] "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-8978ac4e-7409-46bd-9fa7-5c789f516a04" "disk.csi.azure.com/request-id"="86e45c3d-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "disk.csi.azure.com/volume-name"="pvc-8978ac4e-7409-46bd-9fa7-5c789f516a04" "latency"=2634029932 I0131 22:56:44.406464 1 workflow.go:155] "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-8978ac4e-7409-46bd-9fa7-5c789f516a04" "disk.csi.azure.com/request-id"="86e45c3d-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate" "disk.csi.azure.com/volume-name"="pvc-8978ac4e-7409-46bd-9fa7-5c789f516a04" "latency"=2645089489 I0131 22:56:44.406579 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0131 22:56:44.406635 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).CreateVolume" "csi.storage.k8s.io/pv/name"="pvc-8978ac4e-7409-46bd-9fa7-5c789f516a04" "disk.csi.azure.com/request-id"="86e45c3d-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-8978ac4e-7409-46bd-9fa7-5c789f516a04" "latency"=2649892114 I0131 22:56:44.406660 1 crdprovisioner.go:228] "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-8978ac4e-7409-46bd-9fa7-5c789f516a04" "disk.csi.azure.com/request-id"="86e45c3d-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-8978ac4e-7409-46bd-9fa7-5c789f516a04" "latency"=2661129272 I0131 22:56:44.406697 1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=2.661166873 request="azuredisk_csi_driver_controller_create_volume" resource_group="kubetest-n9iqgmbq" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="test.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-n9iqgmbq/providers/Microsoft.Compute/disks/pvc-8978ac4e-7409-46bd-9fa7-5c789f516a04" result_code="succeeded" I0131 22:56:44.406708 1 utils.go:85] GRPC response: {"volume":{"accessible_topology":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"capacity_bytes":10737418240,"content_source":{"Type":{"Volume":{}}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-8978ac4e-7409-46bd-9fa7-5c789f516a04","csi.storage.k8s.io/pvc/name":"daemonset-azuredisk-ephemeral-qzznj-azuredisk","csi.storage.k8s.io/pvc/namespace":"default","requestedsizegib":"10","skuName":"StandardSSD_LRS"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-n9iqgmbq/providers/Microsoft.Compute/disks/pvc-8978ac4e-7409-46bd-9fa7-5c789f516a04"}} I0131 22:56:44.543555 1 batch.go:224] "cloud-provider-azure: Delayed processing of batch due to start delay" type="batch" operation="attach_disk" key="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e|kubetest-n9iqgmbq|k8s-agentpool1-24870519-vmss000000" delay="1s" I0131 22:56:44.543667 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-n9iqgmbq): vm(k8s-agentpool1-24870519-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-n9iqgmbq/providers/Microsoft.Compute/disks/pvc-dec955da-85f4-45eb-8dd0-53173daa5046:AttachDiskOptions{diskName: "pvc-dec955da-85f4-45eb-8dd0-53173daa5046", lun: 1}]) I0131 22:56:44.553135 1 azure_disk_utils.go:1103] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3.4" "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000000" "disk.csi.azure.com/request-id"="87f16a39-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-dec955da-85f4-45eb-8dd0-53173daa5046" "latency"=9332048 "number_of_net_retry"=0 "number_of_retry"=0 I0131 22:56:44.553172 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0131 22:56:44.553223 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000000" "disk.csi.azure.com/request-id"="87f16a39-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-dec955da-85f4-45eb-8dd0-53173daa5046" "latency"=1036768656 I0131 22:56:44.553264 1 crdprovisioner.go:727] "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000000" "disk.csi.azure.com/request-id"="87f16a39-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLun" "disk.csi.azure.com/volume-name"="pvc-dec955da-85f4-45eb-8dd0-53173daa5046" "latency"=1036934556 I0131 22:56:44.553328 1 crdprovisioner.go:486] "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000000" "disk.csi.azure.com/request-id"="87f16a39-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).PublishVolume" "disk.csi.azure.com/volume-name"="pvc-dec955da-85f4-45eb-8dd0-53173daa5046" "latency"=1044489995 I0131 22:56:44.553373 1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=1.044579396 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-n9iqgmbq" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="test.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-n9iqgmbq/providers/Microsoft.Compute/disks/pvc-dec955da-85f4-45eb-8dd0-53173daa5046" node="k8s-agentpool1-24870519-vmss000000" result_code="succeeded" I0131 22:56:44.553391 1 utils.go:85] GRPC response: {"publish_context":{"LUN":"1"}} I0131 22:56:44.553505 1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-dec955da-85f4-45eb-8dd0-53173daa5046-k8s-agentpool1-24870519-vmss000000-attachment) ... skipping 2 lines ... I0131 22:56:44.842395 1 azure_disk_utils.go:1068] "msg"="Label selector is: disk.csi.azure.com/node-name=k8s-agentpool1-24870519-vmss000001,disk.csi.azure.com/volume-name!=pvc-8978ac4e-7409-46bd-9fa7-5c789f516a04." "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000001" "disk.csi.azure.com/request-id"="88bce373-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-8978ac4e-7409-46bd-9fa7-5c789f516a04" I0131 22:56:44.846332 1 utils.go:78] GRPC call: /csi.v1.Controller/ControllerPublishVolume I0131 22:56:44.846352 1 utils.go:79] GRPC request: {"node_id":"k8s-agentpool1-24870519-vmss000002","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-132f85ce-a89d-44fa-b5b7-a3c93612a97e","csi.storage.k8s.io/pvc/name":"daemonset-azuredisk-ephemeral-tztwc-azuredisk","csi.storage.k8s.io/pvc/namespace":"default","requestedsizegib":"10","skuName":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1675205701216-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-n9iqgmbq/providers/Microsoft.Compute/disks/pvc-132f85ce-a89d-44fa-b5b7-a3c93612a97e"} I0131 22:56:44.846596 1 azure_disk_utils.go:1068] "msg"="Label selector is: disk.csi.azure.com/node-name=k8s-agentpool1-24870519-vmss000002,disk.csi.azure.com/volume-name!=pvc-132f85ce-a89d-44fa-b5b7-a3c93612a97e." "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000002" "disk.csi.azure.com/request-id"="88bd8724-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-132f85ce-a89d-44fa-b5b7-a3c93612a97e" I0131 22:56:44.849362 1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-8978ac4e-7409-46bd-9fa7-5c789f516a04-k8s-agentpool1-24870519-vmss000001-attachment) I0131 22:56:44.855004 1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-132f85ce-a89d-44fa-b5b7-a3c93612a97e-k8s-agentpool1-24870519-vmss000002-attachment) I0131 22:56:44.855049 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0131 22:56:44.861998 1 azure_disk_utils.go:1103] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000002" "disk.csi.azure.com/request-id"="88bd8724-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-132f85ce-a89d-44fa-b5b7-a3c93612a97e" "latency"=6818036 "number_of_net_retry"=0 "number_of_retry"=0 I0131 22:56:44.862043 1 attach_detach.go:192] "msg"="Attaching volume" "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000002" "disk.csi.azure.com/request-id"="88bd8724-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-132f85ce-a89d-44fa-b5b7-a3c93612a97e" I0131 22:56:44.862049 1 azure_disk_utils.go:1103] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000001" "disk.csi.azure.com/request-id"="88bce373-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-8978ac4e-7409-46bd-9fa7-5c789f516a04" "latency"=13456169 "number_of_net_retry"=0 "number_of_retry"=0 I0131 22:56:44.862075 1 attach_detach.go:192] "msg"="Attaching volume" "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000001" "disk.csi.azure.com/request-id"="88bce373-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-8978ac4e-7409-46bd-9fa7-5c789f516a04" I0131 22:56:44.864495 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0131 22:56:44.864692 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0131 22:56:44.896561 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1225 I0131 22:56:44.896796 1 cloudprovisioner.go:405] "msg"="GetDiskLun returned: -1. Initiating attaching volume \"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-n9iqgmbq/providers/Microsoft.Compute/disks/pvc-8978ac4e-7409-46bd-9fa7-5c789f516a04\" to node \"k8s-agentpool1-24870519-vmss000001\"." "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000001" "disk.csi.azure.com/request-id"="88bce373-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-8978ac4e-7409-46bd-9fa7-5c789f516a04" I0131 22:56:44.896841 1 cloudprovisioner.go:424] "msg"="Trying to attach volume \"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-n9iqgmbq/providers/Microsoft.Compute/disks/pvc-8978ac4e-7409-46bd-9fa7-5c789f516a04\" to node \"k8s-agentpool1-24870519-vmss000001\"." "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000001" "disk.csi.azure.com/request-id"="88bce373-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-8978ac4e-7409-46bd-9fa7-5c789f516a04" I0131 22:56:44.916242 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1225 I0131 22:56:44.916419 1 cloudprovisioner.go:405] "msg"="GetDiskLun returned: -1. Initiating attaching volume \"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-n9iqgmbq/providers/Microsoft.Compute/disks/pvc-132f85ce-a89d-44fa-b5b7-a3c93612a97e\" to node \"k8s-agentpool1-24870519-vmss000002\"." "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000002" "disk.csi.azure.com/request-id"="88bd8724-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-132f85ce-a89d-44fa-b5b7-a3c93612a97e" I0131 22:56:44.916447 1 cloudprovisioner.go:424] "msg"="Trying to attach volume \"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-n9iqgmbq/providers/Microsoft.Compute/disks/pvc-132f85ce-a89d-44fa-b5b7-a3c93612a97e\" to node \"k8s-agentpool1-24870519-vmss000002\"." "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000002" "disk.csi.azure.com/request-id"="88bd8724-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-132f85ce-a89d-44fa-b5b7-a3c93612a97e" I0131 22:56:45.320508 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-n9iqgmbq): vm(k8s-agentpool1-24870519-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-n9iqgmbq/providers/Microsoft.Compute/disks/pvc-dec955da-85f4-45eb-8dd0-53173daa5046:AttachDiskOptions{diskName: "pvc-dec955da-85f4-45eb-8dd0-53173daa5046", lun: 1}], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I0131 22:56:45.320550 1 azure_controller_common.go:344] azuredisk - trying to attach disks to node k8s-agentpool1-24870519-vmss000000: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-n9iqgmbq/providers/Microsoft.Compute/disks/pvc-dec955da-85f4-45eb-8dd0-53173daa5046:AttachDiskOptions{diskName: "pvc-dec955da-85f4-45eb-8dd0-53173daa5046", lun: 1}] I0131 22:56:45.610822 1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller I0131 22:56:45.897316 1 batch.go:224] "cloud-provider-azure: Delayed processing of batch due to start delay" type="batch" operation="attach_disk" key="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e|kubetest-n9iqgmbq|k8s-agentpool1-24870519-vmss000001" delay="1s" I0131 22:56:45.897464 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-n9iqgmbq): vm(k8s-agentpool1-24870519-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-n9iqgmbq/providers/Microsoft.Compute/disks/pvc-8978ac4e-7409-46bd-9fa7-5c789f516a04:AttachDiskOptions{diskName: "pvc-8978ac4e-7409-46bd-9fa7-5c789f516a04", lun: 1}]) I0131 22:56:45.911541 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0131 22:56:45.911770 1 azure_disk_utils.go:1103] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3.4" "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000001" "disk.csi.azure.com/request-id"="88bce373-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-8978ac4e-7409-46bd-9fa7-5c789f516a04" "latency"=14139273 "number_of_net_retry"=0 "number_of_retry"=0 I0131 22:56:45.911822 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000001" "disk.csi.azure.com/request-id"="88bce373-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-8978ac4e-7409-46bd-9fa7-5c789f516a04" "latency"=1062388688 I0131 22:56:45.911851 1 crdprovisioner.go:727] "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000001" "disk.csi.azure.com/request-id"="88bce373-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLun" "disk.csi.azure.com/volume-name"="pvc-8978ac4e-7409-46bd-9fa7-5c789f516a04" "latency"=1062505589 I0131 22:56:45.911884 1 crdprovisioner.go:486] "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000001" "disk.csi.azure.com/request-id"="88bce373-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).PublishVolume" "disk.csi.azure.com/volume-name"="pvc-8978ac4e-7409-46bd-9fa7-5c789f516a04" "latency"=1069591825 I0131 22:56:45.911918 1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=1.069650625 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-n9iqgmbq" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="test.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-n9iqgmbq/providers/Microsoft.Compute/disks/pvc-8978ac4e-7409-46bd-9fa7-5c789f516a04" node="k8s-agentpool1-24870519-vmss000001" result_code="succeeded" I0131 22:56:45.911928 1 utils.go:85] GRPC response: {"publish_context":{"LUN":"1"}} I0131 22:56:45.912078 1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-8978ac4e-7409-46bd-9fa7-5c789f516a04-k8s-agentpool1-24870519-vmss000001-attachment) I0131 22:56:45.917609 1 batch.go:224] "cloud-provider-azure: Delayed processing of batch due to start delay" type="batch" operation="attach_disk" key="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e|kubetest-n9iqgmbq|k8s-agentpool1-24870519-vmss000002" delay="1s" I0131 22:56:45.917682 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-n9iqgmbq): vm(k8s-agentpool1-24870519-vmss000002) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-n9iqgmbq/providers/Microsoft.Compute/disks/pvc-132f85ce-a89d-44fa-b5b7-a3c93612a97e:AttachDiskOptions{diskName: "pvc-132f85ce-a89d-44fa-b5b7-a3c93612a97e", lun: 2}]) I0131 22:56:45.926563 1 azure_disk_utils.go:1103] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3.4" "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000002" "disk.csi.azure.com/request-id"="88bd8724-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-132f85ce-a89d-44fa-b5b7-a3c93612a97e" "latency"=8794846 "number_of_net_retry"=0 "number_of_retry"=0 I0131 22:56:45.928006 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0131 22:56:45.928061 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000002" "disk.csi.azure.com/request-id"="88bd8724-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-132f85ce-a89d-44fa-b5b7-a3c93612a97e" "latency"=1072994242 I0131 22:56:45.928089 1 crdprovisioner.go:727] "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000002" "disk.csi.azure.com/request-id"="88bd8724-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLun" "disk.csi.azure.com/volume-name"="pvc-132f85ce-a89d-44fa-b5b7-a3c93612a97e" "latency"=1073090843 I0131 22:56:45.928116 1 crdprovisioner.go:486] "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000002" "disk.csi.azure.com/request-id"="88bd8724-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).PublishVolume" "disk.csi.azure.com/volume-name"="pvc-132f85ce-a89d-44fa-b5b7-a3c93612a97e" "latency"=1081633087 I0131 22:56:45.928161 1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=1.081703987 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-n9iqgmbq" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="test.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-n9iqgmbq/providers/Microsoft.Compute/disks/pvc-132f85ce-a89d-44fa-b5b7-a3c93612a97e" node="k8s-agentpool1-24870519-vmss000002" result_code="succeeded" I0131 22:56:45.928176 1 utils.go:85] GRPC response: {"publish_context":{"LUN":"2"}} I0131 22:56:45.928277 1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-132f85ce-a89d-44fa-b5b7-a3c93612a97e-k8s-agentpool1-24870519-vmss000002-attachment) I0131 22:56:46.057532 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-n9iqgmbq): vm(k8s-agentpool1-24870519-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-n9iqgmbq/providers/Microsoft.Compute/disks/pvc-8978ac4e-7409-46bd-9fa7-5c789f516a04:AttachDiskOptions{diskName: "pvc-8978ac4e-7409-46bd-9fa7-5c789f516a04", lun: 1}], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I0131 22:56:46.057621 1 azure_controller_common.go:344] azuredisk - trying to attach disks to node k8s-agentpool1-24870519-vmss000001: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-n9iqgmbq/providers/Microsoft.Compute/disks/pvc-8978ac4e-7409-46bd-9fa7-5c789f516a04:AttachDiskOptions{diskName: "pvc-8978ac4e-7409-46bd-9fa7-5c789f516a04", lun: 1}] I0131 22:56:47.372925 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-n9iqgmbq): vm(k8s-agentpool1-24870519-vmss000002) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-n9iqgmbq/providers/Microsoft.Compute/disks/pvc-132f85ce-a89d-44fa-b5b7-a3c93612a97e:AttachDiskOptions{diskName: "pvc-132f85ce-a89d-44fa-b5b7-a3c93612a97e", lun: 2}], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I0131 22:56:47.372968 1 azure_controller_common.go:344] azuredisk - trying to attach disks to node k8s-agentpool1-24870519-vmss000002: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-n9iqgmbq/providers/Microsoft.Compute/disks/pvc-132f85ce-a89d-44fa-b5b7-a3c93612a97e:AttachDiskOptions{diskName: "pvc-132f85ce-a89d-44fa-b5b7-a3c93612a97e", lun: 2}] I0131 22:56:47.620465 1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller I0131 22:56:49.632497 1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller I0131 22:56:51.641201 1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller I0131 22:56:53.654564 1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller I0131 22:56:55.403035 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-n9iqgmbq, k8s-agentpool1-24870519-vmss, k8s-agentpool1-24870519-vmss000000) successfully ... skipping 3 lines ... I0131 22:56:55.403189 1 workflow.go:155] "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000000" "disk.csi.azure.com/request-id"="87f16a39-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).PublishVolume" "disk.csi.azure.com/volume-name"="pvc-dec955da-85f4-45eb-8dd0-53173daa5046" "latency"=11881814903 I0131 22:56:55.403338 1 azure_disk_utils.go:1165] "msg"="Skip update. No update needed." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).updateVolumeAttachmentWithResult" "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000000" "disk.csi.azure.com/request-id"="87f16a39-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-dec955da-85f4-45eb-8dd0-53173daa5046" I0131 22:56:55.403398 1 azure_disk_utils.go:1103] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).updateVolumeAttachmentWithResult" "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000000" "disk.csi.azure.com/request-id"="87f16a39-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-dec955da-85f4-45eb-8dd0-53173daa5046" "latency"=168501 "number_of_net_retry"=0 "number_of_retry"=0 I0131 22:56:55.417439 1 azure_disk_utils.go:1103] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3.4" "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000000" "disk.csi.azure.com/request-id"="87f16a39-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-dec955da-85f4-45eb-8dd0-53173daa5046" "latency"=13990873 "number_of_net_retry"=0 "number_of_retry"=0 I0131 22:56:55.417474 1 attach_detach.go:198] "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000000" "disk.csi.azure.com/request-id"="87f16a39-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3" "disk.csi.azure.com/volume-name"="pvc-dec955da-85f4-45eb-8dd0-53173daa5046" "latency"=11896116277 I0131 22:56:55.417496 1 workflow.go:155] "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000000" "disk.csi.azure.com/request-id"="87f16a39-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach" "disk.csi.azure.com/volume-name"="pvc-dec955da-85f4-45eb-8dd0-53173daa5046" "latency"=11903009913 I0131 22:56:55.418383 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0131 22:56:55.418446 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000000" "disk.csi.azure.com/request-id"="8890d116-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-dec955da-85f4-45eb-8dd0-53173daa5046" "latency"=10864865950 I0131 22:56:55.418469 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).ControllerPublishVolume.func2" "disk.csi.azure.com/request-id"="8890d116-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=10864998952 I0131 22:56:55.418500 1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=11.909711848 request="azuredisk_csi_driver_attach_volume_latency" resource_group="kubetest-n9iqgmbq" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="test.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-n9iqgmbq/providers/Microsoft.Compute/disks/pvc-dec955da-85f4-45eb-8dd0-53173daa5046" node="k8s-agentpool1-24870519-vmss000000" result_code="succeeded" I0131 22:56:55.418522 1 controllerserver_v2.go:264] "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="8890d116-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).ControllerPublishVolume.func2" "latency"=10865061352 I0131 22:56:55.668450 1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller I0131 22:56:56.138538 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-n9iqgmbq, k8s-agentpool1-24870519-vmss, k8s-agentpool1-24870519-vmss000001) successfully ... skipping 3 lines ... I0131 22:56:56.138700 1 workflow.go:155] "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000001" "disk.csi.azure.com/request-id"="88bce373-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).PublishVolume" "disk.csi.azure.com/volume-name"="pvc-8978ac4e-7409-46bd-9fa7-5c789f516a04" "latency"=11276578696 I0131 22:56:56.138832 1 azure_disk_utils.go:1165] "msg"="Skip update. No update needed." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).updateVolumeAttachmentWithResult" "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000001" "disk.csi.azure.com/request-id"="88bce373-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-8978ac4e-7409-46bd-9fa7-5c789f516a04" I0131 22:56:56.138893 1 azure_disk_utils.go:1103] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).updateVolumeAttachmentWithResult" "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000001" "disk.csi.azure.com/request-id"="88bce373-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-8978ac4e-7409-46bd-9fa7-5c789f516a04" "latency"=125900 "number_of_net_retry"=0 "number_of_retry"=0 I0131 22:56:56.146580 1 azure_disk_utils.go:1103] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3.4" "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000001" "disk.csi.azure.com/request-id"="88bce373-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-8978ac4e-7409-46bd-9fa7-5c789f516a04" "latency"=7613140 "number_of_net_retry"=0 "number_of_retry"=0 I0131 22:56:56.146705 1 attach_detach.go:198] "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000001" "disk.csi.azure.com/request-id"="88bce373-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3" "disk.csi.azure.com/volume-name"="pvc-8978ac4e-7409-46bd-9fa7-5c789f516a04" "latency"=11284521837 I0131 22:56:56.146745 1 workflow.go:155] "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000001" "disk.csi.azure.com/request-id"="88bce373-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach" "disk.csi.azure.com/volume-name"="pvc-8978ac4e-7409-46bd-9fa7-5c789f516a04" "latency"=11298180708 I0131 22:56:56.146979 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0131 22:56:56.147035 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000001" "disk.csi.azure.com/request-id"="89601ebd-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-8978ac4e-7409-46bd-9fa7-5c789f516a04" "latency"=10234886815 I0131 22:56:56.147071 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).ControllerPublishVolume.func2" "disk.csi.azure.com/request-id"="89601ebd-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=10235010716 I0131 22:56:56.147105 1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=11.304833642 request="azuredisk_csi_driver_attach_volume_latency" resource_group="kubetest-n9iqgmbq" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="test.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-n9iqgmbq/providers/Microsoft.Compute/disks/pvc-8978ac4e-7409-46bd-9fa7-5c789f516a04" node="k8s-agentpool1-24870519-vmss000001" result_code="succeeded" I0131 22:56:56.147127 1 controllerserver_v2.go:264] "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="89601ebd-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).ControllerPublishVolume.func2" "latency"=10235087516 I0131 22:56:57.474950 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-n9iqgmbq, k8s-agentpool1-24870519-vmss, k8s-agentpool1-24870519-vmss000002) successfully I0131 22:56:57.474981 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool1-24870519-vmss, kubetest-n9iqgmbq, k8s-agentpool1-24870519-vmss000002) for cacheKey(kubetest-n9iqgmbq/k8s-agentpool1-24870519-vmss) updated successfully ... skipping 2 lines ... I0131 22:56:57.475103 1 workflow.go:155] "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000002" "disk.csi.azure.com/request-id"="88bd8724-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).PublishVolume" "disk.csi.azure.com/volume-name"="pvc-132f85ce-a89d-44fa-b5b7-a3c93612a97e" "latency"=12612997334 I0131 22:56:57.475225 1 azure_disk_utils.go:1165] "msg"="Skip update. No update needed." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).updateVolumeAttachmentWithResult" "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000002" "disk.csi.azure.com/request-id"="88bd8724-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-132f85ce-a89d-44fa-b5b7-a3c93612a97e" I0131 22:56:57.475282 1 azure_disk_utils.go:1103] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).updateVolumeAttachmentWithResult" "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000002" "disk.csi.azure.com/request-id"="88bd8724-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-132f85ce-a89d-44fa-b5b7-a3c93612a97e" "latency"=120900 "number_of_net_retry"=0 "number_of_retry"=0 I0131 22:56:57.485702 1 azure_disk_utils.go:1103] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3.4" "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000002" "disk.csi.azure.com/request-id"="88bd8724-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-132f85ce-a89d-44fa-b5b7-a3c93612a97e" "latency"=10373353 "number_of_net_retry"=0 "number_of_retry"=0 I0131 22:56:57.485741 1 attach_detach.go:198] "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000002" "disk.csi.azure.com/request-id"="88bd8724-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3" "disk.csi.azure.com/volume-name"="pvc-132f85ce-a89d-44fa-b5b7-a3c93612a97e" "latency"=12623650989 I0131 22:56:57.485760 1 workflow.go:155] "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000002" "disk.csi.azure.com/request-id"="88bd8724-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach" "disk.csi.azure.com/volume-name"="pvc-132f85ce-a89d-44fa-b5b7-a3c93612a97e" "latency"=12630635125 I0131 22:56:57.486477 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0131 22:56:57.486528 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000002" "disk.csi.azure.com/request-id"="89629735-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-132f85ce-a89d-44fa-b5b7-a3c93612a97e" "latency"=11558178785 I0131 22:56:57.486555 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).ControllerPublishVolume.func2" "disk.csi.azure.com/request-id"="89629735-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=11558307086 I0131 22:56:57.486609 1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=12.640141973 request="azuredisk_csi_driver_attach_volume_latency" resource_group="kubetest-n9iqgmbq" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="test.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-n9iqgmbq/providers/Microsoft.Compute/disks/pvc-132f85ce-a89d-44fa-b5b7-a3c93612a97e" node="k8s-agentpool1-24870519-vmss000002" result_code="succeeded" I0131 22:56:57.486635 1 controllerserver_v2.go:264] "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="89629735-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).ControllerPublishVolume.func2" "latency"=11558401486 I0131 22:56:57.677734 1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller I0131 22:56:58.055795 1 shared_state.go:419] "msg"="Adding pod daemonset-azuredisk-ephemeral-phdkd to shared map with keyName default/daemonset-azuredisk-ephemeral-phdkd." ... skipping 62 lines ... I0131 22:57:41.464397 1 utils.go:78] GRPC call: /csi.v1.Controller/ControllerUnpublishVolume I0131 22:57:41.464422 1 utils.go:79] GRPC request: {"node_id":"k8s-agentpool1-24870519-vmss000002","volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-n9iqgmbq/providers/Microsoft.Compute/disks/pvc-ca962c87-9010-49bd-aee3-69e8c86cf241"} I0131 22:57:41.464871 1 crdprovisioner.go:927] "msg"="Requesting AzVolumeAttachment (pvc-ca962c87-9010-49bd-aee3-69e8c86cf241-k8s-agentpool1-24870519-vmss000002-attachment) detachment" "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000002" "disk.csi.azure.com/request-id"="5321ebaf-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-ca962c87-9010-49bd-aee3-69e8c86cf241" I0131 22:57:41.471773 1 replica.go:160] "msg"="Garbage collection of AzVolumeAttachments for AzVolume (pvc-ca962c87-9010-49bd-aee3-69e8c86cf241) scheduled in 5m0s." "disk.csi.azure.com/request-id"="aa7ddc0e-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/volume-name"="pvc-ca962c87-9010-49bd-aee3-69e8c86cf241" I0131 22:57:41.472327 1 azure_disk_utils.go:1103] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).detachVolume" "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000002" "disk.csi.azure.com/request-id"="5321ebaf-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-ca962c87-9010-49bd-aee3-69e8c86cf241" "latency"=7417039 "number_of_net_retry"=0 "number_of_retry"=0 I0131 22:57:41.472396 1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-ca962c87-9010-49bd-aee3-69e8c86cf241-k8s-agentpool1-24870519-vmss000002-attachment) I0131 22:57:41.489093 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0131 22:57:41.489113 1 azure_disk_utils.go:1103] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach" "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000002" "disk.csi.azure.com/request-id"="5321ebaf-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-ca962c87-9010-49bd-aee3-69e8c86cf241" "latency"=17283390 "number_of_net_retry"=0 "number_of_retry"=0 I0131 22:57:41.489146 1 attach_detach.go:340] "msg"="Detaching volume" "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000002" "disk.csi.azure.com/request-id"="5321ebaf-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-ca962c87-9010-49bd-aee3-69e8c86cf241" I0131 22:57:41.489279 1 cloudprovisioner.go:481] "msg"="Trying to detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-n9iqgmbq/providers/Microsoft.Compute/disks/pvc-ca962c87-9010-49bd-aee3-69e8c86cf241 from node k8s-agentpool1-24870519-vmss000002" "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000002" "disk.csi.azure.com/request-id"="5321ebaf-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-ca962c87-9010-49bd-aee3-69e8c86cf241" I0131 22:57:41.963540 1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller I0131 22:57:42.490326 1 batch.go:224] "cloud-provider-azure: Delayed processing of batch due to start delay" type="batch" operation="detach_disk" key="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e|kubetest-n9iqgmbq|k8s-agentpool1-24870519-vmss000002" delay="1s" I0131 22:57:42.490389 1 azure_controller_common.go:461] azuredisk - trying to detach disks from node k8s-agentpool1-24870519-vmss000002: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-n9iqgmbq/providers/Microsoft.Compute/disks/pvc-ca962c87-9010-49bd-aee3-69e8c86cf241:pvc-ca962c87-9010-49bd-aee3-69e8c86cf241] ... skipping 8 lines ... I0131 22:57:56.053549 1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller I0131 22:57:58.063287 1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller I0131 22:58:00.075759 1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller I0131 22:58:00.826611 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0131 22:58:00.831802 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0131 22:58:00.831883 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0131 22:58:00.831896 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0131 22:58:00.837326 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0131 22:58:02.085203 1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller I0131 22:58:02.738603 1 azure_controller_vmss.go:252] azureDisk - update(kubetest-n9iqgmbq): vm(k8s-agentpool1-24870519-vmss000002) - detach disk(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-n9iqgmbq/providers/Microsoft.Compute/disks/pvc-ca962c87-9010-49bd-aee3-69e8c86cf241:pvc-ca962c87-9010-49bd-aee3-69e8c86cf241]) returned with <nil> I0131 22:58:02.738651 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-n9iqgmbq, k8s-agentpool1-24870519-vmss, k8s-agentpool1-24870519-vmss000002) successfully I0131 22:58:02.738669 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool1-24870519-vmss, kubetest-n9iqgmbq, k8s-agentpool1-24870519-vmss000002) for cacheKey(kubetest-n9iqgmbq/k8s-agentpool1-24870519-vmss) updated successfully I0131 22:58:02.738678 1 azure_controller_common.go:473] azuredisk - successfully detached disks from node k8s-agentpool1-24870519-vmss000002: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-n9iqgmbq/providers/Microsoft.Compute/disks/pvc-ca962c87-9010-49bd-aee3-69e8c86cf241:pvc-ca962c87-9010-49bd-aee3-69e8c86cf241] I0131 22:58:02.738708 1 azure_controller_common.go:434] azureDisk - detach disk(pvc-ca962c87-9010-49bd-aee3-69e8c86cf241, /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-n9iqgmbq/providers/Microsoft.Compute/disks/pvc-ca962c87-9010-49bd-aee3-69e8c86cf241) succeeded I0131 22:58:02.738750 1 cloudprovisioner.go:491] "msg"="detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-n9iqgmbq/providers/Microsoft.Compute/disks/pvc-ca962c87-9010-49bd-aee3-69e8c86cf241 from node k8s-agentpool1-24870519-vmss000002 successfully" "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000002" "disk.csi.azure.com/request-id"="5321ebaf-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-ca962c87-9010-49bd-aee3-69e8c86cf241" I0131 22:58:02.738783 1 cloudprovisioner.go:471] "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000002" "disk.csi.azure.com/request-id"="5321ebaf-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).UnpublishVolume" "disk.csi.azure.com/volume-name"="pvc-ca962c87-9010-49bd-aee3-69e8c86cf241" "latency"=21249541274 I0131 22:58:02.753133 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0131 22:58:02.754016 1 attach_detach.go:346] "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000002" "disk.csi.azure.com/request-id"="5321ebaf-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach.func3" "disk.csi.azure.com/volume-name"="pvc-ca962c87-9010-49bd-aee3-69e8c86cf241" "latency"=21264815054 I0131 22:58:02.754048 1 workflow.go:155] "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000002" "disk.csi.azure.com/request-id"="5321ebaf-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach" "disk.csi.azure.com/volume-name"="pvc-ca962c87-9010-49bd-aee3-69e8c86cf241" "latency"=21282247445 I0131 22:58:02.762797 1 azure_disk_utils.go:1103] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).removeFinalizer" "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000002" "disk.csi.azure.com/request-id"="5321ebaf-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-ca962c87-9010-49bd-aee3-69e8c86cf241" "latency"=8605445 "number_of_net_retry"=0 "number_of_retry"=0 I0131 22:58:02.762832 1 attach_detach.go:379] "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000002" "disk.csi.azure.com/request-id"="5321ebaf-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).removeFinalizer" "disk.csi.azure.com/volume-name"="pvc-ca962c87-9010-49bd-aee3-69e8c86cf241" "latency"=8689545 I0131 22:58:02.763574 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0131 22:58:02.763631 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForDetach" "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000002" "disk.csi.azure.com/request-id"="5321ebaf-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-ca962c87-9010-49bd-aee3-69e8c86cf241" "latency"=21291177392 I0131 22:58:02.763668 1 crdprovisioner.go:938] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).detachVolume" "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000002" "disk.csi.azure.com/request-id"="5321ebaf-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForDetach" "disk.csi.azure.com/volume-name"="pvc-ca962c87-9010-49bd-aee3-69e8c86cf241" "latency"=21291299293 I0131 22:58:02.763689 1 crdprovisioner.go:836] "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000002" "disk.csi.azure.com/request-id"="aa7ccd51-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).UnpublishVolume" "disk.csi.azure.com/volume-name"="pvc-ca962c87-9010-49bd-aee3-69e8c86cf241" "latency"=21298863832 I0131 22:58:02.763726 1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=21.298965432 request="azuredisk_csi_driver_controller_unpublish_volume" resource_group="kubetest-n9iqgmbq" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="test.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-n9iqgmbq/providers/Microsoft.Compute/disks/pvc-ca962c87-9010-49bd-aee3-69e8c86cf241" node="k8s-agentpool1-24870519-vmss000002" result_code="succeeded" I0131 22:58:02.763738 1 utils.go:85] GRPC response: {} I0131 22:58:04.112092 1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller I0131 22:58:06.126600 1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller I0131 22:58:06.709471 1 utils.go:78] GRPC call: /csi.v1.Controller/DeleteVolume I0131 22:58:06.709494 1 utils.go:79] GRPC request: {"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-n9iqgmbq/providers/Microsoft.Compute/disks/pvc-ca962c87-9010-49bd-aee3-69e8c86cf241"} I0131 22:58:06.709611 1 controllerserver_v2.go:201] deleting disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-n9iqgmbq/providers/Microsoft.Compute/disks/pvc-ca962c87-9010-49bd-aee3-69e8c86cf241) I0131 22:58:06.718853 1 azure_disk_utils.go:1103] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).DeleteVolume" "csi.storage.k8s.io/pv/name"="pvc-ca962c87-9010-49bd-aee3-69e8c86cf241" "disk.csi.azure.com/request-id"="b988dcd4-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-ca962c87-9010-49bd-aee3-69e8c86cf241" "latency"=9128948 "number_of_net_retry"=0 "number_of_retry"=0 I0131 22:58:06.718876 1 conditionwatcher.go:99] Adding a condition function for azvolume (pvc-ca962c87-9010-49bd-aee3-69e8c86cf241) I0131 22:58:06.719287 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0131 22:58:06.730468 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0131 22:58:06.742601 1 azure_disk_utils.go:1103] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerDelete" "csi.storage.k8s.io/pv/name"="pvc-ca962c87-9010-49bd-aee3-69e8c86cf241" "disk.csi.azure.com/request-id"="514fda4d-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-ca962c87-9010-49bd-aee3-69e8c86cf241" "latency"=12080463 "number_of_net_retry"=0 "number_of_retry"=0 I0131 22:58:06.742625 1 azvolume.go:246] "msg"="Deleting Volume..." "csi.storage.k8s.io/pv/name"="pvc-ca962c87-9010-49bd-aee3-69e8c86cf241" "disk.csi.azure.com/request-id"="514fda4d-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/volume-name"="pvc-ca962c87-9010-49bd-aee3-69e8c86cf241" I0131 22:58:06.742667 1 shared_state.go:1037] "msg"="AzVolumeAttachment clean up requested by azvolume-controller for AzVolume (pvc-ca962c87-9010-49bd-aee3-69e8c86cf241)" "csi.storage.k8s.io/pv/name"="pvc-ca962c87-9010-49bd-aee3-69e8c86cf241" "disk.csi.azure.com/request-id"="514fda4d-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/volume-name"="pvc-ca962c87-9010-49bd-aee3-69e8c86cf241" I0131 22:58:06.742680 1 azure_disk_utils.go:1039] "msg"="Getting AzVolumeAttachment list for volume (pvc-ca962c87-9010-49bd-aee3-69e8c86cf241)" "csi.storage.k8s.io/pv/name"="pvc-ca962c87-9010-49bd-aee3-69e8c86cf241" "disk.csi.azure.com/request-id"="514fda4d-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/volume-name"="pvc-ca962c87-9010-49bd-aee3-69e8c86cf241" I0131 22:58:06.742711 1 azure_disk_utils.go:1068] "msg"="Label selector is: disk.csi.azure.com/volume-name=pvc-ca962c87-9010-49bd-aee3-69e8c86cf241." "csi.storage.k8s.io/pv/name"="pvc-ca962c87-9010-49bd-aee3-69e8c86cf241" "disk.csi.azure.com/request-id"="514fda4d-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/volume-name"="pvc-ca962c87-9010-49bd-aee3-69e8c86cf241" I0131 22:58:06.742757 1 shared_state.go:1035] "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-ca962c87-9010-49bd-aee3-69e8c86cf241" "disk.csi.azure.com/request-id"="514fda4d-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*SharedState).cleanUpAzVolumeAttachmentByVolume" "disk.csi.azure.com/volume-name"="pvc-ca962c87-9010-49bd-aee3-69e8c86cf241" "latency"=80600 I0131 22:58:06.744078 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0131 22:58:06.770354 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1192 I0131 22:58:07.827424 1 pod.go:65] AzDiskControllerManager "msg"="No need to reconcile. Pod was deleted." "controller"="pod" "disk.csi/azure.com/pod-name"="statefulset-azuredisk-0" "namespace"="azure-disk-csi" "partition"="csi-azuredisk-controller" I0131 22:58:08.136841 1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller I0131 22:58:08.501049 1 pod.go:65] AzDiskControllerManager "msg"="No need to reconcile. Pod was deleted." "controller"="pod" "disk.csi/azure.com/pod-name"="statefulset-azuredisk-nonroot-0" "namespace"="azure-disk-csi" "partition"="csi-azuredisk-controller" I0131 22:58:08.930326 1 pod.go:65] AzDiskControllerManager "msg"="No need to reconcile. Pod was deleted." "controller"="pod" "disk.csi/azure.com/pod-name"="nginx-azuredisk-ephemeral" "namespace"="azure-disk-csi" "partition"="csi-azuredisk-controller" I0131 22:58:08.968364 1 shared_state.go:1037] "msg"="AzVolumeAttachment clean up requested by pv-controller for AzVolume (pvc-1d7561cc-4a1b-4d53-9658-4ee8ef715c2c)" "disk.csi.azure.com/request-id"="bae17fbe-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/volume-name"="pvc-1d7561cc-4a1b-4d53-9658-4ee8ef715c2c" ... skipping 9 lines ... I0131 22:58:11.585758 1 replica.go:160] "msg"="Garbage collection of AzVolumeAttachments for AzVolume (pvc-3d318377-2dd2-4c2a-9669-be7055655892) scheduled in 5m0s." "disk.csi.azure.com/request-id"="bc70e36d-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/volume-name"="pvc-3d318377-2dd2-4c2a-9669-be7055655892" I0131 22:58:11.586484 1 azure_disk_utils.go:1103] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).detachVolume" "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000000" "disk.csi.azure.com/request-id"="6e32b9f3-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-3d318377-2dd2-4c2a-9669-be7055655892" "latency"=7384738 "number_of_net_retry"=0 "number_of_retry"=0 I0131 22:58:11.586532 1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-3d318377-2dd2-4c2a-9669-be7055655892-k8s-agentpool1-24870519-vmss000000-attachment) I0131 22:58:11.592045 1 azure_disk_utils.go:1103] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach" "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000000" "disk.csi.azure.com/request-id"="6e32b9f3-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-3d318377-2dd2-4c2a-9669-be7055655892" "latency"=6267232 "number_of_net_retry"=0 "number_of_retry"=0 I0131 22:58:11.592075 1 attach_detach.go:340] "msg"="Detaching volume" "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000000" "disk.csi.azure.com/request-id"="6e32b9f3-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-3d318377-2dd2-4c2a-9669-be7055655892" I0131 22:58:11.592141 1 cloudprovisioner.go:481] "msg"="Trying to detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-n9iqgmbq/providers/Microsoft.Compute/disks/pvc-3d318377-2dd2-4c2a-9669-be7055655892 from node k8s-agentpool1-24870519-vmss000000" "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000000" "disk.csi.azure.com/request-id"="6e32b9f3-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-3d318377-2dd2-4c2a-9669-be7055655892" I0131 22:58:11.592590 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0131 22:58:11.970297 1 azure_managedDiskController.go:317] azureDisk - deleted a managed disk: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-n9iqgmbq/providers/Microsoft.Compute/disks/pvc-ca962c87-9010-49bd-aee3-69e8c86cf241 I0131 22:58:11.970360 1 cloudprovisioner.go:337] "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-ca962c87-9010-49bd-aee3-69e8c86cf241" "disk.csi.azure.com/request-id"="514fda4d-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).DeleteVolume" "disk.csi.azure.com/volume-name"="pvc-ca962c87-9010-49bd-aee3-69e8c86cf241" "latency"=5227547761 I0131 22:58:11.987870 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0131 22:58:11.987950 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).DeleteVolume" "csi.storage.k8s.io/pv/name"="pvc-ca962c87-9010-49bd-aee3-69e8c86cf241" "disk.csi.azure.com/request-id"="b988dcd4-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-ca962c87-9010-49bd-aee3-69e8c86cf241" "latency"=5249416873 I0131 22:58:11.988002 1 crdprovisioner.go:375] "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-ca962c87-9010-49bd-aee3-69e8c86cf241" "disk.csi.azure.com/request-id"="b988dcd4-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).DeleteVolume" "disk.csi.azure.com/volume-name"="pvc-ca962c87-9010-49bd-aee3-69e8c86cf241" "latency"=5278300324 I0131 22:58:11.988059 1 controllerserver_v2.go:203] delete disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-n9iqgmbq/providers/Microsoft.Compute/disks/pvc-ca962c87-9010-49bd-aee3-69e8c86cf241) returned with <nil> I0131 22:58:11.988105 1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=5.278458524 request="azuredisk_csi_driver_controller_delete_volume" resource_group="kubetest-n9iqgmbq" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="test.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-n9iqgmbq/providers/Microsoft.Compute/disks/pvc-ca962c87-9010-49bd-aee3-69e8c86cf241" result_code="succeeded" I0131 22:58:11.988124 1 utils.go:85] GRPC response: {} I0131 22:58:11.989338 1 azure_disk_utils.go:1103] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerDelete.func4" "csi.storage.k8s.io/pv/name"="pvc-ca962c87-9010-49bd-aee3-69e8c86cf241" "disk.csi.azure.com/request-id"="514fda4d-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-ca962c87-9010-49bd-aee3-69e8c86cf241" "latency"=18931296 "number_of_net_retry"=0 "number_of_retry"=0 ... skipping 2 lines ... I0131 22:58:12.008728 1 utils.go:78] GRPC call: /csi.v1.Controller/ControllerUnpublishVolume I0131 22:58:12.008751 1 utils.go:79] GRPC request: {"node_id":"k8s-agentpool1-24870519-vmss000002","volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-n9iqgmbq/providers/Microsoft.Compute/disks/pvc-1d7561cc-4a1b-4d53-9658-4ee8ef715c2c"} I0131 22:58:12.008888 1 crdprovisioner.go:927] "msg"="Requesting AzVolumeAttachment (pvc-1d7561cc-4a1b-4d53-9658-4ee8ef715c2c-k8s-agentpool1-24870519-vmss000002-attachment) detachment" "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000002" "disk.csi.azure.com/request-id"="7d614e3d-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-1d7561cc-4a1b-4d53-9658-4ee8ef715c2c" I0131 22:58:12.018727 1 azure_disk_utils.go:1103] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).detachVolume" "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000002" "disk.csi.azure.com/request-id"="7d614e3d-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-1d7561cc-4a1b-4d53-9658-4ee8ef715c2c" "latency"=9779650 "number_of_net_retry"=0 "number_of_retry"=0 I0131 22:58:12.018775 1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-1d7561cc-4a1b-4d53-9658-4ee8ef715c2c-k8s-agentpool1-24870519-vmss000002-attachment) I0131 22:58:12.018998 1 replica.go:160] "msg"="Garbage collection of AzVolumeAttachments for AzVolume (pvc-1d7561cc-4a1b-4d53-9658-4ee8ef715c2c) scheduled in 5m0s." "disk.csi.azure.com/request-id"="bcb2fef4-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/volume-name"="pvc-1d7561cc-4a1b-4d53-9658-4ee8ef715c2c" I0131 22:58:12.027680 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0131 22:58:12.032779 1 azure_disk_utils.go:1103] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach" "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000002" "disk.csi.azure.com/request-id"="7d614e3d-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-1d7561cc-4a1b-4d53-9658-4ee8ef715c2c" "latency"=13722370 "number_of_net_retry"=0 "number_of_retry"=0 I0131 22:58:12.032809 1 attach_detach.go:340] "msg"="Detaching volume" "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000002" "disk.csi.azure.com/request-id"="7d614e3d-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-1d7561cc-4a1b-4d53-9658-4ee8ef715c2c" I0131 22:58:12.032874 1 cloudprovisioner.go:481] "msg"="Trying to detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-n9iqgmbq/providers/Microsoft.Compute/disks/pvc-1d7561cc-4a1b-4d53-9658-4ee8ef715c2c from node k8s-agentpool1-24870519-vmss000002" "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000002" "disk.csi.azure.com/request-id"="7d614e3d-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-1d7561cc-4a1b-4d53-9658-4ee8ef715c2c" I0131 22:58:12.038796 1 shared_state.go:574] "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="bcb6045c-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*SharedState).deletePV" "latency"=11000 I0131 22:58:12.158895 1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller I0131 22:58:12.592895 1 batch.go:224] "cloud-provider-azure: Delayed processing of batch due to start delay" type="batch" operation="detach_disk" key="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e|kubetest-n9iqgmbq|k8s-agentpool1-24870519-vmss000000" delay="1s" ... skipping 7 lines ... I0131 22:58:13.628136 1 utils.go:78] GRPC call: /csi.v1.Controller/ControllerUnpublishVolume I0131 22:58:13.628156 1 utils.go:79] GRPC request: {"node_id":"k8s-agentpool1-24870519-vmss000001","volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-n9iqgmbq/providers/Microsoft.Compute/disks/pvc-feff1987-6fdb-4ff7-877a-8a3a988036e7"} I0131 22:58:13.628340 1 crdprovisioner.go:927] "msg"="Requesting AzVolumeAttachment (pvc-feff1987-6fdb-4ff7-877a-8a3a988036e7-k8s-agentpool1-24870519-vmss000001-attachment) detachment" "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000001" "disk.csi.azure.com/request-id"="60650a6b-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-feff1987-6fdb-4ff7-877a-8a3a988036e7" I0131 22:58:13.635996 1 replica.go:160] "msg"="Garbage collection of AzVolumeAttachments for AzVolume (pvc-feff1987-6fdb-4ff7-877a-8a3a988036e7) scheduled in 5m0s." "disk.csi.azure.com/request-id"="bda9bab5-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/volume-name"="pvc-feff1987-6fdb-4ff7-877a-8a3a988036e7" I0131 22:58:13.636594 1 azure_disk_utils.go:1103] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).detachVolume" "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000001" "disk.csi.azure.com/request-id"="60650a6b-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-feff1987-6fdb-4ff7-877a-8a3a988036e7" "latency"=8163442 "number_of_net_retry"=0 "number_of_retry"=0 I0131 22:58:13.636644 1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-feff1987-6fdb-4ff7-877a-8a3a988036e7-k8s-agentpool1-24870519-vmss000001-attachment) I0131 22:58:13.643205 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0131 22:58:13.643479 1 azure_disk_utils.go:1103] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach" "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000001" "disk.csi.azure.com/request-id"="60650a6b-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-feff1987-6fdb-4ff7-877a-8a3a988036e7" "latency"=7464738 "number_of_net_retry"=0 "number_of_retry"=0 I0131 22:58:13.643509 1 attach_detach.go:340] "msg"="Detaching volume" "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000001" "disk.csi.azure.com/request-id"="60650a6b-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-feff1987-6fdb-4ff7-877a-8a3a988036e7" I0131 22:58:13.643606 1 cloudprovisioner.go:481] "msg"="Trying to detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-n9iqgmbq/providers/Microsoft.Compute/disks/pvc-feff1987-6fdb-4ff7-877a-8a3a988036e7 from node k8s-agentpool1-24870519-vmss000001" "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000001" "disk.csi.azure.com/request-id"="60650a6b-a1ba-11ed-8e8f-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-feff1987-6fdb-4ff7-877a-8a3a988036e7" print out csi-test-node logs ... ====================================================================================== dumping logs for kube-system/csi-test-node-7mn2r/test ... skipping 34 lines ... I0131 22:54:55.249254 1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumes?allowWatchBookmarks=true&resourceVersion=2763&timeout=8m58s&timeoutSeconds=538&watch=true 200 OK in 18 milliseconds I0131 22:54:55.249291 1 round_trippers.go:553] GET https://10.0.0.1:443/api/v1/nodes?allowWatchBookmarks=true&resourceVersion=2755&timeout=6m24s&timeoutSeconds=384&watch=true 200 OK in 3 milliseconds I0131 22:54:55.292450 1 shared_informer.go:303] caches populated I0131 22:54:55.292485 1 azuredisk_v2.go:225] driver userAgent: test.csi.azure.com/latest-v2-930851ca0d19682b1fa9f893a9e8fa4374a3f5f9 e2e-test I0131 22:54:55.292506 1 azure_disk_utils.go:564] reading cloud config from secret kube-system/azure-cloud-provider I0131 22:54:55.295969 1 round_trippers.go:553] GET https://10.0.0.1:443/api/v1/namespaces/kube-system/secrets/azure-cloud-provider 404 Not Found in 3 milliseconds I0131 22:54:55.296181 1 azure_disk_utils.go:571] InitializeCloudFromSecret: failed to get cloud config from secret kube-system/azure-cloud-provider: failed to get secret kube-system/azure-cloud-provider: secrets "azure-cloud-provider" not found I0131 22:54:55.296197 1 azure_disk_utils.go:576] could not read cloud config from secret kube-system/azure-cloud-provider I0131 22:54:55.296203 1 azure_disk_utils.go:586] use default AZURE_CREDENTIAL_FILE env var: /etc/kubernetes/azure.json I0131 22:54:55.296227 1 azure_disk_utils.go:594] read cloud config from file: /etc/kubernetes/azure.json successfully I0131 22:54:55.296861 1 azure_auth.go:253] Using AzurePublicCloud environment I0131 22:54:55.296880 1 azure_auth.go:104] azure: using managed identity extension to retrieve access token I0131 22:54:55.296886 1 azure_auth.go:110] azure: using User Assigned MSI ID to retrieve access token ... skipping 90 lines ... I0131 22:55:39.669940 1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume I0131 22:55:39.669956 1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-feff1987-6fdb-4ff7-877a-8a3a988036e7/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-feff1987-6fdb-4ff7-877a-8a3a988036e7","csi.storage.k8s.io/pvc/name":"persistent-storage-statefulset-azuredisk-0","csi.storage.k8s.io/pvc/namespace":"default","requestedsizegib":"10","skuName":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1675205701216-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-n9iqgmbq/providers/Microsoft.Compute/disks/pvc-feff1987-6fdb-4ff7-877a-8a3a988036e7"} I0131 22:55:39.670226 1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-feff1987-6fdb-4ff7-877a-8a3a988036e7-k8s-agentpool1-24870519-vmss000001-attachment) I0131 22:55:42.237581 1 azuredisk_v2.go:578] "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-31T22:55:42Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000001" "disk.csi.azure.com/request-id"="636c2840-a1ba-11ed-b061-6045bdf48aeb" I0131 22:55:42.248879 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-24870519-vmss000001/status 200 OK in 11 milliseconds I0131 22:55:42.249299 1 azuredisk_v2.go:565] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000001" "disk.csi.azure.com/request-id"="636c2840-a1ba-11ed-b061-6045bdf48aeb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=11746030 I0131 22:55:48.542181 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0131 22:55:48.542276 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000001" "disk.csi.azure.com/request-id"="61e46ad6-a1ba-11ed-b061-6045bdf48aeb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-feff1987-6fdb-4ff7-877a-8a3a988036e7" "latency"=8871967165 I0131 22:55:48.542302 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="61e46ad6-a1ba-11ed-b061-6045bdf48aeb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=8872097067 I0131 22:55:50.283028 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/ I0131 22:55:50.283075 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType StandardSSD_LRS I0131 22:55:50.283105 1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-feff1987-6fdb-4ff7-877a-8a3a988036e7/globalmount with mount options([]) I0131 22:55:50.283116 1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0]) ... skipping 64 lines ... I0131 22:56:51.445270 1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]} I0131 22:56:51.446043 1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume I0131 22:56:51.446057 1 utils.go:79] GRPC request: {"publish_context":{"LUN":"1"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8978ac4e-7409-46bd-9fa7-5c789f516a04/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-8978ac4e-7409-46bd-9fa7-5c789f516a04","csi.storage.k8s.io/pvc/name":"daemonset-azuredisk-ephemeral-qzznj-azuredisk","csi.storage.k8s.io/pvc/namespace":"default","requestedsizegib":"10","skuName":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1675205701216-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-n9iqgmbq/providers/Microsoft.Compute/disks/pvc-8978ac4e-7409-46bd-9fa7-5c789f516a04"} I0131 22:56:51.446218 1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-8978ac4e-7409-46bd-9fa7-5c789f516a04-k8s-agentpool1-24870519-vmss000001-attachment) I0131 22:56:55.231492 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0131 22:56:55.231510 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0131 22:56:55.231838 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0131 22:56:55.234814 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0131 22:56:55.249450 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0131 22:56:55.325049 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0131 22:56:56.144295 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0131 22:56:56.144365 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000001" "disk.csi.azure.com/request-id"="8cac909e-a1ba-11ed-b061-6045bdf48aeb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-8978ac4e-7409-46bd-9fa7-5c789f516a04" "latency"=4698081027 I0131 22:56:56.144394 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="8cac909e-a1ba-11ed-b061-6045bdf48aeb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=4698194229 I0131 22:56:56.766122 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdd under /dev/disk/azure/scsi1/ I0131 22:56:56.766176 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType StandardSSD_LRS I0131 22:56:56.766208 1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun1 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8978ac4e-7409-46bd-9fa7-5c789f516a04/globalmount with mount options([]) I0131 22:56:56.766218 1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun1" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun1]) ... skipping 116 lines ... I0131 22:55:02.017315 1 round_trippers.go:553] GET https://10.0.0.1:443/apis/apiextensions.k8s.io/v1/customresourcedefinitions?allowWatchBookmarks=true&resourceVersion=2567&timeout=8m24s&timeoutSeconds=504&watch=true 200 OK in 1 milliseconds I0131 22:55:02.017924 1 round_trippers.go:553] GET https://10.0.0.1:443/api/v1/nodes?allowWatchBookmarks=true&resourceVersion=2814&timeout=5m55s&timeoutSeconds=355&watch=true 200 OK in 1 milliseconds I0131 22:55:02.059164 1 shared_informer.go:303] caches populated I0131 22:55:02.059199 1 azuredisk_v2.go:225] driver userAgent: test.csi.azure.com/latest-v2-930851ca0d19682b1fa9f893a9e8fa4374a3f5f9 e2e-test I0131 22:55:02.059211 1 azure_disk_utils.go:564] reading cloud config from secret kube-system/azure-cloud-provider I0131 22:55:02.062487 1 round_trippers.go:553] GET https://10.0.0.1:443/api/v1/namespaces/kube-system/secrets/azure-cloud-provider 404 Not Found in 3 milliseconds I0131 22:55:02.062790 1 azure_disk_utils.go:571] InitializeCloudFromSecret: failed to get cloud config from secret kube-system/azure-cloud-provider: failed to get secret kube-system/azure-cloud-provider: secrets "azure-cloud-provider" not found I0131 22:55:02.062807 1 azure_disk_utils.go:576] could not read cloud config from secret kube-system/azure-cloud-provider I0131 22:55:02.062813 1 azure_disk_utils.go:586] use default AZURE_CREDENTIAL_FILE env var: /etc/kubernetes/azure.json I0131 22:55:02.062842 1 azure_disk_utils.go:594] read cloud config from file: /etc/kubernetes/azure.json successfully I0131 22:55:02.063555 1 azure_auth.go:253] Using AzurePublicCloud environment I0131 22:55:02.063573 1 azure_auth.go:104] azure: using managed identity extension to retrieve access token I0131 22:55:02.063579 1 azure_auth.go:110] azure: using User Assigned MSI ID to retrieve access token ... skipping 98 lines ... I0131 22:56:09.438592 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities I0131 22:56:09.438608 1 utils.go:79] GRPC request: {} I0131 22:56:09.438642 1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]} I0131 22:56:09.439677 1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume I0131 22:56:09.439693 1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-3d318377-2dd2-4c2a-9669-be7055655892/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-3d318377-2dd2-4c2a-9669-be7055655892","csi.storage.k8s.io/pvc/name":"persistent-storage-statefulset-azuredisk-nonroot-0","csi.storage.k8s.io/pvc/namespace":"default","requestedsizegib":"10","skuName":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1675205701216-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-n9iqgmbq/providers/Microsoft.Compute/disks/pvc-3d318377-2dd2-4c2a-9669-be7055655892"} I0131 22:56:09.439918 1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-3d318377-2dd2-4c2a-9669-be7055655892-k8s-agentpool1-24870519-vmss000000-attachment) I0131 22:56:11.676889 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0131 22:56:11.676988 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000000" "disk.csi.azure.com/request-id"="73a2e984-a1ba-11ed-b9d6-6045bdf481f1" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-3d318377-2dd2-4c2a-9669-be7055655892" "latency"=2236985720 I0131 22:56:11.677020 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="73a2e984-a1ba-11ed-b9d6-6045bdf481f1" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=2237114922 I0131 22:56:13.376138 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/ I0131 22:56:13.376187 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType StandardSSD_LRS I0131 22:56:13.376216 1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-3d318377-2dd2-4c2a-9669-be7055655892/globalmount with mount options([]) I0131 22:56:13.376227 1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0]) ... skipping 54 lines ... I0131 22:56:50.210555 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities I0131 22:56:50.210572 1 utils.go:79] GRPC request: {} I0131 22:56:50.210600 1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]} I0131 22:56:50.211474 1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume I0131 22:56:50.211491 1 utils.go:79] GRPC request: {"publish_context":{"LUN":"1"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-dec955da-85f4-45eb-8dd0-53173daa5046/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-dec955da-85f4-45eb-8dd0-53173daa5046","csi.storage.k8s.io/pvc/name":"daemonset-azuredisk-ephemeral-phdkd-azuredisk","csi.storage.k8s.io/pvc/namespace":"default","requestedsizegib":"10","skuName":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1675205701216-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-n9iqgmbq/providers/Microsoft.Compute/disks/pvc-dec955da-85f4-45eb-8dd0-53173daa5046"} I0131 22:56:50.211649 1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-dec955da-85f4-45eb-8dd0-53173daa5046-k8s-agentpool1-24870519-vmss000000-attachment) I0131 22:56:55.418879 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0131 22:56:55.418947 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000000" "disk.csi.azure.com/request-id"="8bf02f72-a1ba-11ed-b9d6-6045bdf481f1" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-dec955da-85f4-45eb-8dd0-53173daa5046" "latency"=5207241212 I0131 22:56:55.418971 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="8bf02f72-a1ba-11ed-b9d6-6045bdf481f1" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=5207338513 I0131 22:56:56.014403 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdd under /dev/disk/azure/scsi1/ I0131 22:56:56.014449 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType StandardSSD_LRS I0131 22:56:56.014481 1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun1 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-dec955da-85f4-45eb-8dd0-53173daa5046/globalmount with mount options([]) I0131 22:56:56.014499 1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun1" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun1]) ... skipping 124 lines ... I0131 22:54:55.772393 1 round_trippers.go:553] GET https://10.0.0.1:443/api/v1/nodes?allowWatchBookmarks=true&resourceVersion=2785&timeout=9m43s&timeoutSeconds=583&watch=true 200 OK in 3 milliseconds I0131 22:54:55.772516 1 round_trippers.go:553] GET https://10.0.0.1:443/apis/apiextensions.k8s.io/v1/customresourcedefinitions?allowWatchBookmarks=true&resourceVersion=2567&timeout=8m27s&timeoutSeconds=507&watch=true 200 OK in 3 milliseconds I0131 22:54:55.820694 1 shared_informer.go:303] caches populated I0131 22:54:55.820734 1 azuredisk_v2.go:225] driver userAgent: test.csi.azure.com/latest-v2-930851ca0d19682b1fa9f893a9e8fa4374a3f5f9 e2e-test I0131 22:54:55.820758 1 azure_disk_utils.go:564] reading cloud config from secret kube-system/azure-cloud-provider I0131 22:54:55.824309 1 round_trippers.go:553] GET https://10.0.0.1:443/api/v1/namespaces/kube-system/secrets/azure-cloud-provider 404 Not Found in 3 milliseconds I0131 22:54:55.824543 1 azure_disk_utils.go:571] InitializeCloudFromSecret: failed to get cloud config from secret kube-system/azure-cloud-provider: failed to get secret kube-system/azure-cloud-provider: secrets "azure-cloud-provider" not found I0131 22:54:55.824558 1 azure_disk_utils.go:576] could not read cloud config from secret kube-system/azure-cloud-provider I0131 22:54:55.824563 1 azure_disk_utils.go:586] use default AZURE_CREDENTIAL_FILE env var: /etc/kubernetes/azure.json I0131 22:54:55.824585 1 azure_disk_utils.go:594] read cloud config from file: /etc/kubernetes/azure.json successfully I0131 22:54:55.825295 1 azure_auth.go:253] Using AzurePublicCloud environment I0131 22:54:55.825316 1 azure_auth.go:104] azure: using managed identity extension to retrieve access token I0131 22:54:55.825322 1 azure_auth.go:110] azure: using User Assigned MSI ID to retrieve access token ... skipping 80 lines ... I0131 22:55:18.461480 1 utils.go:79] GRPC request: {} I0131 22:55:18.461513 1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]} I0131 22:55:18.462650 1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume I0131 22:55:18.462668 1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-ca962c87-9010-49bd-aee3-69e8c86cf241/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-ca962c87-9010-49bd-aee3-69e8c86cf241","csi.storage.k8s.io/pvc/name":"pvc-azuredisk","csi.storage.k8s.io/pvc/namespace":"default","requestedsizegib":"10","skuName":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1675205701216-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-n9iqgmbq/providers/Microsoft.Compute/disks/pvc-ca962c87-9010-49bd-aee3-69e8c86cf241"} I0131 22:55:18.462985 1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-ca962c87-9010-49bd-aee3-69e8c86cf241-k8s-agentpool1-24870519-vmss000002-attachment) I0131 22:55:25.756970 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0131 22:55:25.757041 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0131 22:55:25.757055 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0131 22:55:25.757113 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0131 22:55:25.769293 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0131 22:55:25.848692 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0131 22:55:27.303640 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0131 22:55:27.303709 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000002" "disk.csi.azure.com/request-id"="55407275-a1ba-11ed-bf53-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-ca962c87-9010-49bd-aee3-69e8c86cf241" "latency"=8840644830 I0131 22:55:27.303743 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="55407275-a1ba-11ed-bf53-6045bdf48a25" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=8840787630 I0131 22:55:29.132388 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/ I0131 22:55:29.132439 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType StandardSSD_LRS I0131 22:55:29.132471 1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-ca962c87-9010-49bd-aee3-69e8c86cf241/globalmount with mount options([]) I0131 22:55:29.132480 1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0]) ... skipping 65 lines ... I0131 22:56:32.474312 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetVolumeStats I0131 22:56:32.474330 1 utils.go:79] GRPC request: {"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-n9iqgmbq/providers/Microsoft.Compute/disks/pvc-ca962c87-9010-49bd-aee3-69e8c86cf241","volume_path":"/var/lib/kubelet/pods/f7c2132a-4d08-42ad-b5e4-3d7adf140a60/volumes/kubernetes.io~csi/pvc-ca962c87-9010-49bd-aee3-69e8c86cf241/mount"} I0131 22:56:32.474441 1 utils.go:85] GRPC response: {"usage":[{"available":10447216640,"total":10464022528,"unit":1,"used":28672},{"available":655348,"total":655360,"unit":2,"used":12}]} I0131 22:56:34.628550 1 azuredisk_v2.go:578] "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-01-31T22:56:34Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000002" "disk.csi.azure.com/request-id"="82a663a8-a1ba-11ed-bf53-6045bdf48a25" I0131 22:56:34.649552 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-24870519-vmss000002/status 200 OK in 20 milliseconds I0131 22:56:34.650745 1 azuredisk_v2.go:565] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000002" "disk.csi.azure.com/request-id"="82a663a8-a1ba-11ed-bf53-6045bdf48a25" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=22223715 I0131 22:56:37.113702 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0131 22:56:37.113765 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000002" "disk.csi.azure.com/request-id"="7ff9cd1e-a1ba-11ed-bf53-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-1d7561cc-4a1b-4d53-9658-4ee8ef715c2c" "latency"=6971693347 I0131 22:56:37.113794 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="7ff9cd1e-a1ba-11ed-bf53-6045bdf48a25" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=6971797648 I0131 22:56:37.712987 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdd under /dev/disk/azure/scsi1/ I0131 22:56:37.713033 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType StandardSSD_LRS I0131 22:56:37.713064 1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun1 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-1d7561cc-4a1b-4d53-9658-4ee8ef715c2c/globalmount with mount options([]) I0131 22:56:37.713078 1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun1" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun1]) ... skipping 40 lines ... I0131 22:56:50.576281 1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume I0131 22:56:50.576296 1 utils.go:79] GRPC request: {"publish_context":{"LUN":"2"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-132f85ce-a89d-44fa-b5b7-a3c93612a97e/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-132f85ce-a89d-44fa-b5b7-a3c93612a97e","csi.storage.k8s.io/pvc/name":"daemonset-azuredisk-ephemeral-tztwc-azuredisk","csi.storage.k8s.io/pvc/namespace":"default","requestedsizegib":"10","skuName":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1675205701216-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-n9iqgmbq/providers/Microsoft.Compute/disks/pvc-132f85ce-a89d-44fa-b5b7-a3c93612a97e"} I0131 22:56:50.576505 1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-132f85ce-a89d-44fa-b5b7-a3c93612a97e-k8s-agentpool1-24870519-vmss000002-attachment) I0131 22:56:55.758621 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0131 22:56:55.758627 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0131 22:56:55.758634 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0131 22:56:55.758860 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0131 22:56:55.770867 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0131 22:56:55.849983 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0131 22:56:57.486076 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0131 22:56:57.486154 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-24870519-vmss000002" "disk.csi.azure.com/request-id"="8c27db3a-a1ba-11ed-bf53-6045bdf48a25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-132f85ce-a89d-44fa-b5b7-a3c93612a97e" "latency"=6909562168 I0131 22:56:57.486188 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="8c27db3a-a1ba-11ed-bf53-6045bdf48a25" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=6909703268 I0131 22:56:58.129827 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun2 by sde under /dev/disk/azure/scsi1/ I0131 22:56:58.129894 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun2. perfProfile none accountType StandardSSD_LRS I0131 22:56:58.129928 1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun2 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-132f85ce-a89d-44fa-b5b7-a3c93612a97e/globalmount with mount options([]) I0131 22:56:58.129944 1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun2" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun2]) ... skipping 141 lines ... I0131 22:54:54.690499 1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumeattachments?allowWatchBookmarks=true&resourceVersion=2764&timeout=9m4s&timeoutSeconds=544&watch=true 200 OK in 0 milliseconds I0131 22:54:54.695936 1 round_trippers.go:553] GET https://10.0.0.1:443/api/v1/nodes?allowWatchBookmarks=true&resourceVersion=2755&timeout=6m39s&timeoutSeconds=399&watch=true 200 OK in 4 milliseconds I0131 22:54:54.729292 1 shared_informer.go:303] caches populated I0131 22:54:54.729329 1 azuredisk_v2.go:225] driver userAgent: test.csi.azure.com/latest-v2-930851ca0d19682b1fa9f893a9e8fa4374a3f5f9 e2e-test I0131 22:54:54.729360 1 azure_disk_utils.go:564] reading cloud config from secret kube-system/azure-cloud-provider I0131 22:54:54.731905 1 round_trippers.go:553] GET https://10.0.0.1:443/api/v1/namespaces/kube-system/secrets/azure-cloud-provider 404 Not Found in 2 milliseconds I0131 22:54:54.732262 1 azure_disk_utils.go:571] InitializeCloudFromSecret: failed to get cloud config from secret kube-system/azure-cloud-provider: failed to get secret kube-system/azure-cloud-provider: secrets "azure-cloud-provider" not found I0131 22:54:54.732283 1 azure_disk_utils.go:576] could not read cloud config from secret kube-system/azure-cloud-provider I0131 22:54:54.732292 1 azure_disk_utils.go:586] use default AZURE_CREDENTIAL_FILE env var: /etc/kubernetes/azure.json I0131 22:54:54.732330 1 azure_disk_utils.go:594] read cloud config from file: /etc/kubernetes/azure.json successfully I0131 22:54:54.733177 1 azure_auth.go:253] Using AzurePublicCloud environment I0131 22:54:54.733198 1 azure_auth.go:104] azure: using managed identity extension to retrieve access token I0131 22:54:54.733205 1 azure_auth.go:110] azure: using User Assigned MSI ID to retrieve access token ... skipping 484 lines ... # HELP go_gc_heap_objects_objects Number of objects, live or unswept, occupying heap memory. # TYPE go_gc_heap_objects_objects gauge go_gc_heap_objects_objects 57106 # HELP go_gc_heap_tiny_allocs_objects_total Count of small allocations that are packed together into blocks. These allocations are counted separately from other allocations because each individual allocation is not tracked by the runtime, only their block. Each block is already accounted for in allocs-by-size and frees-by-size. # TYPE go_gc_heap_tiny_allocs_objects_total counter go_gc_heap_tiny_allocs_objects_total 52070 # HELP go_gc_limiter_last_enabled_gc_cycle GC cycle the last time the GC CPU limiter was enabled. This metric is useful for diagnosing the root cause of an out-of-memory error, because the limiter trades memory for CPU time when the GC's CPU time gets too high. This is most likely to occur with use of SetMemoryLimit. The first GC cycle is cycle 1, so a value of 0 indicates that it was never enabled. # TYPE go_gc_limiter_last_enabled_gc_cycle gauge go_gc_limiter_last_enabled_gc_cycle 0 # HELP go_gc_pauses_seconds Distribution individual GC-related stop-the-world pause latencies. # TYPE go_gc_pauses_seconds histogram go_gc_pauses_seconds_bucket{le="9.999999999999999e-10"} 0 go_gc_pauses_seconds_bucket{le="9.999999999999999e-09"} 0 ... skipping 175 lines ... # HELP process_virtual_memory_max_bytes Maximum amount of virtual memory available in bytes. # TYPE process_virtual_memory_max_bytes gauge process_virtual_memory_max_bytes 1.8446744073709552e+19 # HELP registered_metric_total [ALPHA] The count of registered metrics broken by stability level and deprecation version. # TYPE registered_metric_total counter registered_metric_total{deprecated_version="",stability_level="ALPHA"} 16 make: *** [Makefile:341: e2e-test] Error 127 2023/01/31 22:58:31 process.go:155: Step 'make e2e-test' finished in 16m54.961470144s 2023/01/31 22:58:31 aksengine_helpers.go:425: downloading /root/tmp981692480/log-dump.sh from https://raw.githubusercontent.com/kubernetes-sigs/cloud-provider-azure/master/hack/log-dump/log-dump.sh 2023/01/31 22:58:31 util.go:70: curl https://raw.githubusercontent.com/kubernetes-sigs/cloud-provider-azure/master/hack/log-dump/log-dump.sh 2023/01/31 22:58:31 process.go:153: Running: chmod +x /root/tmp981692480/log-dump.sh 2023/01/31 22:58:31 process.go:155: Step 'chmod +x /root/tmp981692480/log-dump.sh' finished in 3.804678ms 2023/01/31 22:58:31 aksengine_helpers.go:425: downloading /root/tmp981692480/log-dump-daemonset.yaml from https://raw.githubusercontent.com/kubernetes-sigs/cloud-provider-azure/master/hack/log-dump/log-dump-daemonset.yaml ... skipping 75 lines ... ssh key file /root/.ssh/id_rsa does not exist. Exiting. 2023/01/31 22:59:38 process.go:155: Step 'bash -c /root/tmp981692480/win-ci-logs-collector.sh kubetest-n9iqgmbq.westeurope.cloudapp.azure.com /root/tmp981692480 /root/.ssh/id_rsa' finished in 6.783194ms 2023/01/31 22:59:38 aksengine.go:1141: Deleting resource group: kubetest-n9iqgmbq. 2023/01/31 23:06:48 process.go:96: Saved XML output to /logs/artifacts/junit_runner.xml. 2023/01/31 23:06:48 process.go:153: Running: bash -c . hack/lib/version.sh && KUBE_ROOT=. kube::version::get_version_vars && echo "${KUBE_GIT_VERSION-}" 2023/01/31 23:06:48 process.go:155: Step 'bash -c . hack/lib/version.sh && KUBE_ROOT=. kube::version::get_version_vars && echo "${KUBE_GIT_VERSION-}"' finished in 251.338278ms 2023/01/31 23:06:48 main.go:328: Something went wrong: encountered 1 errors: [error during make e2e-test: exit status 2] + EXIT_VALUE=1 + set +o xtrace Cleaning up after docker in docker. ================================================================================ Cleaning up after docker 13fcc2be6d37 ... skipping 4 lines ...