kubernetes-sigs/azuredisk-csi-driver PR #1676 pull-azuredisk-csi-driver-e2e-multi-az #1608655629700829184
Recent runs || View in Spyglass
Recent runs || View in Spyglass
| PR | andyzhangx: fix: set ForceAttemptHTTP2 as false to increase ARM throttling limit |
| Result | FAILURE |
| Tests | 1 failed / 13 succeeded |
| Started | |
| Elapsed | 1h21m |
| Revision | d9d8b72c6cc8290accd670bd9068d9074c59dddd |
| Refs |
1676 |
| job-version | v1.27.0-alpha.0.662+00aae4c10c204d |
| kubetest-version | v20221223-736a4da5ba |
| revision | v1.27.0-alpha.0.662+00aae4c10c204d |
error during make e2e-test: exit status 2
from junit_runner.xml
Filter through log files | View test history on testgrid
kubetest Check APIReachability
kubetest Deferred TearDown
kubetest DumpClusterLogs
kubetest GetDeployer
kubetest IsUp
kubetest Prepare
kubetest TearDown
kubetest TearDown Previous
kubetest Timeout
kubetest Up
kubetest kubectl version
kubetest list nodes
kubetest test setup
... skipping 107 lines ... 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 11345 100 11345 0 0 187k 0 --:--:-- --:--:-- --:--:-- 191k Downloading https://get.helm.sh/helm-v3.10.3-linux-amd64.tar.gz Verifying checksum... Done. Preparing to install helm into /usr/local/bin helm installed into /usr/local/bin/helm docker pull k8sprow.azurecr.io/azuredisk-csi:v1.26.0-4cfb4ee1bf05ca9ba058389347ac5904b562074b || make container-all push-manifest Error response from daemon: manifest for k8sprow.azurecr.io/azuredisk-csi:v1.26.0-4cfb4ee1bf05ca9ba058389347ac5904b562074b not found: manifest unknown: manifest tagged by "v1.26.0-4cfb4ee1bf05ca9ba058389347ac5904b562074b" is not found make[2]: Entering directory '/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver' CGO_ENABLED=0 GOOS=windows go build -a -ldflags "-X sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.driverVersion=v1.26.0-4cfb4ee1bf05ca9ba058389347ac5904b562074b -X sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.gitCommit=4cfb4ee1bf05ca9ba058389347ac5904b562074b -X sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.buildDate=2022-12-30T02:55:39Z -extldflags "-static"" -mod vendor -o _output/amd64/azurediskplugin.exe ./pkg/azurediskplugin docker buildx rm container-builder || true ERROR: no builder "container-builder" found docker buildx create --use --name=container-builder container-builder # enable qemu for arm64 build # https://github.com/docker/buildx/issues/464#issuecomment-741507760 docker run --privileged --rm tonistiigi/binfmt --uninstall qemu-aarch64 Unable to find image 'tonistiigi/binfmt:latest' locally ... skipping 1760 lines ... type: string type: object oneOf: - required: ["persistentVolumeClaimName"] - required: ["volumeSnapshotContentName"] volumeSnapshotClassName: description: 'VolumeSnapshotClassName is the name of the VolumeSnapshotClass requested by the VolumeSnapshot. VolumeSnapshotClassName may be left nil to indicate that the default SnapshotClass should be used. A given cluster may have multiple default Volume SnapshotClasses: one default per CSI Driver. If a VolumeSnapshot does not specify a SnapshotClass, VolumeSnapshotSource will be checked to figure out what the associated CSI Driver is, and the default VolumeSnapshotClass associated with that CSI Driver will be used. If more than one VolumeSnapshotClass exist for a given CSI Driver and more than one have been marked as default, CreateSnapshot will fail and generate an event. Empty string is not allowed for this field.' type: string required: - source type: object status: description: status represents the current information of a snapshot. Consumers must verify binding between VolumeSnapshot and VolumeSnapshotContent objects is successful (by validating that both VolumeSnapshot and VolumeSnapshotContent point at each other) before using this object. ... skipping 2 lines ... description: 'boundVolumeSnapshotContentName is the name of the VolumeSnapshotContent object to which this VolumeSnapshot object intends to bind to. If not specified, it indicates that the VolumeSnapshot object has not been successfully bound to a VolumeSnapshotContent object yet. NOTE: To avoid possible security issues, consumers must verify binding between VolumeSnapshot and VolumeSnapshotContent objects is successful (by validating that both VolumeSnapshot and VolumeSnapshotContent point at each other) before using this object.' type: string creationTime: description: creationTime is the timestamp when the point-in-time snapshot is taken by the underlying storage system. In dynamic snapshot creation case, this field will be filled in by the snapshot controller with the "creation_time" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "creation_time" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. If not specified, it may indicate that the creation time of the snapshot is unknown. format: date-time type: string error: description: error is the last observed error during snapshot creation, if any. This field could be helpful to upper level controllers(i.e., application controller) to decide whether they should continue on waiting for the snapshot to be created based on the type of error reported. The snapshot controller will keep retrying when an error occurrs during the snapshot creation. Upon success, this error field will be cleared. properties: message: description: 'message is a string detailing the encountered error during snapshot creation if specified. NOTE: message may be logged, and it should not contain sensitive information.' type: string time: description: time is the timestamp when the error was encountered. format: date-time type: string type: object readyToUse: description: readyToUse indicates if the snapshot is ready to be used to restore a volume. In dynamic snapshot creation case, this field will be filled in by the snapshot controller with the "ready_to_use" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "ready_to_use" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it, otherwise, this field will be set to "True". If not specified, it means the readiness of a snapshot is unknown. type: boolean restoreSize: type: string description: restoreSize represents the minimum size of volume required to create a volume from this snapshot. In dynamic snapshot creation case, this field will be filled in by the snapshot controller with the "size_bytes" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "size_bytes" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. When restoring a volume from this snapshot, the size of the volume MUST NOT be smaller than the restoreSize if it is specified, otherwise the restoration will fail. If not specified, it indicates that the size is unknown. pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ x-kubernetes-int-or-string: true type: object required: - spec type: object ... skipping 60 lines ... type: string volumeSnapshotContentName: description: volumeSnapshotContentName specifies the name of a pre-existing VolumeSnapshotContent object representing an existing volume snapshot. This field should be set if the snapshot already exists and only needs a representation in Kubernetes. This field is immutable. type: string type: object volumeSnapshotClassName: description: 'VolumeSnapshotClassName is the name of the VolumeSnapshotClass requested by the VolumeSnapshot. VolumeSnapshotClassName may be left nil to indicate that the default SnapshotClass should be used. A given cluster may have multiple default Volume SnapshotClasses: one default per CSI Driver. If a VolumeSnapshot does not specify a SnapshotClass, VolumeSnapshotSource will be checked to figure out what the associated CSI Driver is, and the default VolumeSnapshotClass associated with that CSI Driver will be used. If more than one VolumeSnapshotClass exist for a given CSI Driver and more than one have been marked as default, CreateSnapshot will fail and generate an event. Empty string is not allowed for this field.' type: string required: - source type: object status: description: status represents the current information of a snapshot. Consumers must verify binding between VolumeSnapshot and VolumeSnapshotContent objects is successful (by validating that both VolumeSnapshot and VolumeSnapshotContent point at each other) before using this object. ... skipping 2 lines ... description: 'boundVolumeSnapshotContentName is the name of the VolumeSnapshotContent object to which this VolumeSnapshot object intends to bind to. If not specified, it indicates that the VolumeSnapshot object has not been successfully bound to a VolumeSnapshotContent object yet. NOTE: To avoid possible security issues, consumers must verify binding between VolumeSnapshot and VolumeSnapshotContent objects is successful (by validating that both VolumeSnapshot and VolumeSnapshotContent point at each other) before using this object.' type: string creationTime: description: creationTime is the timestamp when the point-in-time snapshot is taken by the underlying storage system. In dynamic snapshot creation case, this field will be filled in by the snapshot controller with the "creation_time" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "creation_time" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. If not specified, it may indicate that the creation time of the snapshot is unknown. format: date-time type: string error: description: error is the last observed error during snapshot creation, if any. This field could be helpful to upper level controllers(i.e., application controller) to decide whether they should continue on waiting for the snapshot to be created based on the type of error reported. The snapshot controller will keep retrying when an error occurrs during the snapshot creation. Upon success, this error field will be cleared. properties: message: description: 'message is a string detailing the encountered error during snapshot creation if specified. NOTE: message may be logged, and it should not contain sensitive information.' type: string time: description: time is the timestamp when the error was encountered. format: date-time type: string type: object readyToUse: description: readyToUse indicates if the snapshot is ready to be used to restore a volume. In dynamic snapshot creation case, this field will be filled in by the snapshot controller with the "ready_to_use" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "ready_to_use" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it, otherwise, this field will be set to "True". If not specified, it means the readiness of a snapshot is unknown. type: boolean restoreSize: type: string description: restoreSize represents the minimum size of volume required to create a volume from this snapshot. In dynamic snapshot creation case, this field will be filled in by the snapshot controller with the "size_bytes" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "size_bytes" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. When restoring a volume from this snapshot, the size of the volume MUST NOT be smaller than the restoreSize if it is specified, otherwise the restoration will fail. If not specified, it indicates that the size is unknown. pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ x-kubernetes-int-or-string: true type: object required: - spec type: object ... skipping 254 lines ... description: status represents the current information of a snapshot. properties: creationTime: description: creationTime is the timestamp when the point-in-time snapshot is taken by the underlying storage system. In dynamic snapshot creation case, this field will be filled in by the CSI snapshotter sidecar with the "creation_time" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "creation_time" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. If not specified, it indicates the creation time is unknown. The format of this field is a Unix nanoseconds time encoded as an int64. On Unix, the command `date +%s%N` returns the current time in nanoseconds since 1970-01-01 00:00:00 UTC. format: int64 type: integer error: description: error is the last observed error during snapshot creation, if any. Upon success after retry, this error field will be cleared. properties: message: description: 'message is a string detailing the encountered error during snapshot creation if specified. NOTE: message may be logged, and it should not contain sensitive information.' type: string time: description: time is the timestamp when the error was encountered. format: date-time type: string type: object readyToUse: description: readyToUse indicates if a snapshot is ready to be used to restore a volume. In dynamic snapshot creation case, this field will be filled in by the CSI snapshotter sidecar with the "ready_to_use" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "ready_to_use" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it, otherwise, this field will be set to "True". If not specified, it means the readiness of a snapshot is unknown. type: boolean restoreSize: description: restoreSize represents the complete size of the snapshot in bytes. In dynamic snapshot creation case, this field will be filled in by the CSI snapshotter sidecar with the "size_bytes" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "size_bytes" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. When restoring a volume from this snapshot, the size of the volume MUST NOT be smaller than the restoreSize if it is specified, otherwise the restoration will fail. If not specified, it indicates that the size is unknown. format: int64 minimum: 0 type: integer snapshotHandle: description: snapshotHandle is the CSI "snapshot_id" of a snapshot on the underlying storage system. If not specified, it indicates that dynamic snapshot creation has either failed or it is still in progress. type: string type: object required: - spec type: object served: true ... skipping 108 lines ... description: status represents the current information of a snapshot. properties: creationTime: description: creationTime is the timestamp when the point-in-time snapshot is taken by the underlying storage system. In dynamic snapshot creation case, this field will be filled in by the CSI snapshotter sidecar with the "creation_time" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "creation_time" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. If not specified, it indicates the creation time is unknown. The format of this field is a Unix nanoseconds time encoded as an int64. On Unix, the command `date +%s%N` returns the current time in nanoseconds since 1970-01-01 00:00:00 UTC. format: int64 type: integer error: description: error is the last observed error during snapshot creation, if any. Upon success after retry, this error field will be cleared. properties: message: description: 'message is a string detailing the encountered error during snapshot creation if specified. NOTE: message may be logged, and it should not contain sensitive information.' type: string time: description: time is the timestamp when the error was encountered. format: date-time type: string type: object readyToUse: description: readyToUse indicates if a snapshot is ready to be used to restore a volume. In dynamic snapshot creation case, this field will be filled in by the CSI snapshotter sidecar with the "ready_to_use" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "ready_to_use" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it, otherwise, this field will be set to "True". If not specified, it means the readiness of a snapshot is unknown. type: boolean restoreSize: description: restoreSize represents the complete size of the snapshot in bytes. In dynamic snapshot creation case, this field will be filled in by the CSI snapshotter sidecar with the "size_bytes" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "size_bytes" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. When restoring a volume from this snapshot, the size of the volume MUST NOT be smaller than the restoreSize if it is specified, otherwise the restoration will fail. If not specified, it indicates that the size is unknown. format: int64 minimum: 0 type: integer snapshotHandle: description: snapshotHandle is the CSI "snapshot_id" of a snapshot on the underlying storage system. If not specified, it indicates that dynamic snapshot creation has either failed or it is still in progress. type: string type: object required: - spec type: object served: true ... skipping 865 lines ... image: "mcr.microsoft.com/oss/kubernetes-csi/csi-resizer:v1.6.0" args: - "-csi-address=$(ADDRESS)" - "-v=2" - "-leader-election" - "--leader-election-namespace=kube-system" - '-handle-volume-inuse-error=false' - '-feature-gates=RecoverVolumeExpansionFailure=true' - "-timeout=240s" env: - name: ADDRESS value: /csi/csi.sock volumeMounts: ... skipping 216 lines ... [1mSTEP:[0m setting up the StorageClass [38;5;243m12/30/22 03:08:45.654[0m [1mSTEP:[0m creating a StorageClass [38;5;243m12/30/22 03:08:45.655[0m [1mSTEP:[0m setting up the PVC and PV [38;5;243m12/30/22 03:08:45.714[0m [1mSTEP:[0m creating a PVC [38;5;243m12/30/22 03:08:45.714[0m [1mSTEP:[0m setting up the pod [38;5;243m12/30/22 03:08:45.775[0m [1mSTEP:[0m deploying the pod [38;5;243m12/30/22 03:08:45.775[0m [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m12/30/22 03:08:45.835[0m Dec 30 03:08:45.835: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-zrvx8" in namespace "azuredisk-8081" to be "Succeeded or Failed" Dec 30 03:08:45.893: INFO: Pod "azuredisk-volume-tester-zrvx8": Phase="Pending", Reason="", readiness=false. Elapsed: 58.359599ms Dec 30 03:08:47.951: INFO: Pod "azuredisk-volume-tester-zrvx8": Phase="Pending", Reason="", readiness=false. Elapsed: 2.116221845s Dec 30 03:08:49.952: INFO: Pod "azuredisk-volume-tester-zrvx8": Phase="Pending", Reason="", readiness=false. Elapsed: 4.116723199s Dec 30 03:08:51.953: INFO: Pod "azuredisk-volume-tester-zrvx8": Phase="Pending", Reason="", readiness=false. Elapsed: 6.117844837s Dec 30 03:08:53.964: INFO: Pod "azuredisk-volume-tester-zrvx8": Phase="Pending", Reason="", readiness=false. Elapsed: 8.129288929s Dec 30 03:08:55.958: INFO: Pod "azuredisk-volume-tester-zrvx8": Phase="Pending", Reason="", readiness=false. Elapsed: 10.123469088s ... skipping 6 lines ... Dec 30 03:09:09.952: INFO: Pod "azuredisk-volume-tester-zrvx8": Phase="Pending", Reason="", readiness=false. Elapsed: 24.117420843s Dec 30 03:09:11.952: INFO: Pod "azuredisk-volume-tester-zrvx8": Phase="Pending", Reason="", readiness=false. Elapsed: 26.116917018s Dec 30 03:09:13.951: INFO: Pod "azuredisk-volume-tester-zrvx8": Phase="Pending", Reason="", readiness=false. Elapsed: 28.116430278s Dec 30 03:09:15.954: INFO: Pod "azuredisk-volume-tester-zrvx8": Phase="Pending", Reason="", readiness=false. Elapsed: 30.118697556s Dec 30 03:09:17.952: INFO: Pod "azuredisk-volume-tester-zrvx8": Phase="Succeeded", Reason="", readiness=false. Elapsed: 32.117375384s [1mSTEP:[0m Saw pod success [38;5;243m12/30/22 03:09:17.953[0m Dec 30 03:09:17.954: INFO: Pod "azuredisk-volume-tester-zrvx8" satisfied condition "Succeeded or Failed" Dec 30 03:09:17.955: INFO: deleting Pod "azuredisk-8081"/"azuredisk-volume-tester-zrvx8" Dec 30 03:09:18.060: INFO: Pod azuredisk-volume-tester-zrvx8 has the following logs: hello world [1mSTEP:[0m Deleting pod azuredisk-volume-tester-zrvx8 in namespace azuredisk-8081 [38;5;243m12/30/22 03:09:18.06[0m [1mSTEP:[0m validating provisioned PV [38;5;243m12/30/22 03:09:18.182[0m [1mSTEP:[0m checking the PV [38;5;243m12/30/22 03:09:18.241[0m ... skipping 44 lines ... [1mSTEP:[0m setting up the StorageClass [38;5;243m12/30/22 03:08:45.654[0m [1mSTEP:[0m creating a StorageClass [38;5;243m12/30/22 03:08:45.655[0m [1mSTEP:[0m setting up the PVC and PV [38;5;243m12/30/22 03:08:45.714[0m [1mSTEP:[0m creating a PVC [38;5;243m12/30/22 03:08:45.714[0m [1mSTEP:[0m setting up the pod [38;5;243m12/30/22 03:08:45.775[0m [1mSTEP:[0m deploying the pod [38;5;243m12/30/22 03:08:45.775[0m [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m12/30/22 03:08:45.835[0m Dec 30 03:08:45.835: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-zrvx8" in namespace "azuredisk-8081" to be "Succeeded or Failed" Dec 30 03:08:45.893: INFO: Pod "azuredisk-volume-tester-zrvx8": Phase="Pending", Reason="", readiness=false. Elapsed: 58.359599ms Dec 30 03:08:47.951: INFO: Pod "azuredisk-volume-tester-zrvx8": Phase="Pending", Reason="", readiness=false. Elapsed: 2.116221845s Dec 30 03:08:49.952: INFO: Pod "azuredisk-volume-tester-zrvx8": Phase="Pending", Reason="", readiness=false. Elapsed: 4.116723199s Dec 30 03:08:51.953: INFO: Pod "azuredisk-volume-tester-zrvx8": Phase="Pending", Reason="", readiness=false. Elapsed: 6.117844837s Dec 30 03:08:53.964: INFO: Pod "azuredisk-volume-tester-zrvx8": Phase="Pending", Reason="", readiness=false. Elapsed: 8.129288929s Dec 30 03:08:55.958: INFO: Pod "azuredisk-volume-tester-zrvx8": Phase="Pending", Reason="", readiness=false. Elapsed: 10.123469088s ... skipping 6 lines ... Dec 30 03:09:09.952: INFO: Pod "azuredisk-volume-tester-zrvx8": Phase="Pending", Reason="", readiness=false. Elapsed: 24.117420843s Dec 30 03:09:11.952: INFO: Pod "azuredisk-volume-tester-zrvx8": Phase="Pending", Reason="", readiness=false. Elapsed: 26.116917018s Dec 30 03:09:13.951: INFO: Pod "azuredisk-volume-tester-zrvx8": Phase="Pending", Reason="", readiness=false. Elapsed: 28.116430278s Dec 30 03:09:15.954: INFO: Pod "azuredisk-volume-tester-zrvx8": Phase="Pending", Reason="", readiness=false. Elapsed: 30.118697556s Dec 30 03:09:17.952: INFO: Pod "azuredisk-volume-tester-zrvx8": Phase="Succeeded", Reason="", readiness=false. Elapsed: 32.117375384s [1mSTEP:[0m Saw pod success [38;5;243m12/30/22 03:09:17.953[0m Dec 30 03:09:17.954: INFO: Pod "azuredisk-volume-tester-zrvx8" satisfied condition "Succeeded or Failed" Dec 30 03:09:17.955: INFO: deleting Pod "azuredisk-8081"/"azuredisk-volume-tester-zrvx8" Dec 30 03:09:18.060: INFO: Pod azuredisk-volume-tester-zrvx8 has the following logs: hello world [1mSTEP:[0m Deleting pod azuredisk-volume-tester-zrvx8 in namespace azuredisk-8081 [38;5;243m12/30/22 03:09:18.06[0m [1mSTEP:[0m validating provisioned PV [38;5;243m12/30/22 03:09:18.182[0m [1mSTEP:[0m checking the PV [38;5;243m12/30/22 03:09:18.241[0m ... skipping 39 lines ... Dec 30 03:10:02.247: INFO: PersistentVolumeClaim pvc-txd99 found but phase is Pending instead of Bound. Dec 30 03:10:04.305: INFO: PersistentVolumeClaim pvc-txd99 found and phase=Bound (4.17412715s) [1mSTEP:[0m checking the PVC [38;5;243m12/30/22 03:10:04.305[0m [1mSTEP:[0m validating provisioned PV [38;5;243m12/30/22 03:10:04.363[0m [1mSTEP:[0m checking the PV [38;5;243m12/30/22 03:10:04.422[0m [1mSTEP:[0m deploying the pod [38;5;243m12/30/22 03:10:04.422[0m [1mSTEP:[0m checking that the pods command exits with no error [38;5;243m12/30/22 03:10:04.481[0m Dec 30 03:10:04.482: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-ctnt7" in namespace "azuredisk-2540" to be "Succeeded or Failed" Dec 30 03:10:04.539: INFO: Pod "azuredisk-volume-tester-ctnt7": Phase="Pending", Reason="", readiness=false. Elapsed: 57.67705ms Dec 30 03:10:06.598: INFO: Pod "azuredisk-volume-tester-ctnt7": Phase="Pending", Reason="", readiness=false. Elapsed: 2.116521994s Dec 30 03:10:08.600: INFO: Pod "azuredisk-volume-tester-ctnt7": Phase="Pending", Reason="", readiness=false. Elapsed: 4.118227554s Dec 30 03:10:10.599: INFO: Pod "azuredisk-volume-tester-ctnt7": Phase="Pending", Reason="", readiness=false. Elapsed: 6.116889529s Dec 30 03:10:12.599: INFO: Pod "azuredisk-volume-tester-ctnt7": Phase="Pending", Reason="", readiness=false. Elapsed: 8.116891997s Dec 30 03:10:14.600: INFO: Pod "azuredisk-volume-tester-ctnt7": Phase="Pending", Reason="", readiness=false. Elapsed: 10.117918046s Dec 30 03:10:16.598: INFO: Pod "azuredisk-volume-tester-ctnt7": Phase="Pending", Reason="", readiness=false. Elapsed: 12.115875034s Dec 30 03:10:18.598: INFO: Pod "azuredisk-volume-tester-ctnt7": Phase="Pending", Reason="", readiness=false. Elapsed: 14.116663882s Dec 30 03:10:20.598: INFO: Pod "azuredisk-volume-tester-ctnt7": Phase="Pending", Reason="", readiness=false. Elapsed: 16.116497232s Dec 30 03:10:22.600: INFO: Pod "azuredisk-volume-tester-ctnt7": Phase="Pending", Reason="", readiness=false. Elapsed: 18.118797162s Dec 30 03:10:24.600: INFO: Pod "azuredisk-volume-tester-ctnt7": Phase="Pending", Reason="", readiness=false. Elapsed: 20.118708747s Dec 30 03:10:26.599: INFO: Pod "azuredisk-volume-tester-ctnt7": Phase="Succeeded", Reason="", readiness=false. Elapsed: 22.117615907s [1mSTEP:[0m Saw pod success [38;5;243m12/30/22 03:10:26.599[0m Dec 30 03:10:26.599: INFO: Pod "azuredisk-volume-tester-ctnt7" satisfied condition "Succeeded or Failed" Dec 30 03:10:26.600: INFO: deleting Pod "azuredisk-2540"/"azuredisk-volume-tester-ctnt7" Dec 30 03:10:26.688: INFO: Pod azuredisk-volume-tester-ctnt7 has the following logs: hello world [1mSTEP:[0m Deleting pod azuredisk-volume-tester-ctnt7 in namespace azuredisk-2540 [38;5;243m12/30/22 03:10:26.688[0m Dec 30 03:10:26.754: INFO: deleting PVC "azuredisk-2540"/"pvc-txd99" Dec 30 03:10:26.754: INFO: Deleting PersistentVolumeClaim "pvc-txd99" ... skipping 38 lines ... Dec 30 03:10:02.247: INFO: PersistentVolumeClaim pvc-txd99 found but phase is Pending instead of Bound. Dec 30 03:10:04.305: INFO: PersistentVolumeClaim pvc-txd99 found and phase=Bound (4.17412715s) [1mSTEP:[0m checking the PVC [38;5;243m12/30/22 03:10:04.305[0m [1mSTEP:[0m validating provisioned PV [38;5;243m12/30/22 03:10:04.363[0m [1mSTEP:[0m checking the PV [38;5;243m12/30/22 03:10:04.422[0m [1mSTEP:[0m deploying the pod [38;5;243m12/30/22 03:10:04.422[0m [1mSTEP:[0m checking that the pods command exits with no error [38;5;243m12/30/22 03:10:04.481[0m Dec 30 03:10:04.482: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-ctnt7" in namespace "azuredisk-2540" to be "Succeeded or Failed" Dec 30 03:10:04.539: INFO: Pod "azuredisk-volume-tester-ctnt7": Phase="Pending", Reason="", readiness=false. Elapsed: 57.67705ms Dec 30 03:10:06.598: INFO: Pod "azuredisk-volume-tester-ctnt7": Phase="Pending", Reason="", readiness=false. Elapsed: 2.116521994s Dec 30 03:10:08.600: INFO: Pod "azuredisk-volume-tester-ctnt7": Phase="Pending", Reason="", readiness=false. Elapsed: 4.118227554s Dec 30 03:10:10.599: INFO: Pod "azuredisk-volume-tester-ctnt7": Phase="Pending", Reason="", readiness=false. Elapsed: 6.116889529s Dec 30 03:10:12.599: INFO: Pod "azuredisk-volume-tester-ctnt7": Phase="Pending", Reason="", readiness=false. Elapsed: 8.116891997s Dec 30 03:10:14.600: INFO: Pod "azuredisk-volume-tester-ctnt7": Phase="Pending", Reason="", readiness=false. Elapsed: 10.117918046s Dec 30 03:10:16.598: INFO: Pod "azuredisk-volume-tester-ctnt7": Phase="Pending", Reason="", readiness=false. Elapsed: 12.115875034s Dec 30 03:10:18.598: INFO: Pod "azuredisk-volume-tester-ctnt7": Phase="Pending", Reason="", readiness=false. Elapsed: 14.116663882s Dec 30 03:10:20.598: INFO: Pod "azuredisk-volume-tester-ctnt7": Phase="Pending", Reason="", readiness=false. Elapsed: 16.116497232s Dec 30 03:10:22.600: INFO: Pod "azuredisk-volume-tester-ctnt7": Phase="Pending", Reason="", readiness=false. Elapsed: 18.118797162s Dec 30 03:10:24.600: INFO: Pod "azuredisk-volume-tester-ctnt7": Phase="Pending", Reason="", readiness=false. Elapsed: 20.118708747s Dec 30 03:10:26.599: INFO: Pod "azuredisk-volume-tester-ctnt7": Phase="Succeeded", Reason="", readiness=false. Elapsed: 22.117615907s [1mSTEP:[0m Saw pod success [38;5;243m12/30/22 03:10:26.599[0m Dec 30 03:10:26.599: INFO: Pod "azuredisk-volume-tester-ctnt7" satisfied condition "Succeeded or Failed" Dec 30 03:10:26.600: INFO: deleting Pod "azuredisk-2540"/"azuredisk-volume-tester-ctnt7" Dec 30 03:10:26.688: INFO: Pod azuredisk-volume-tester-ctnt7 has the following logs: hello world [1mSTEP:[0m Deleting pod azuredisk-volume-tester-ctnt7 in namespace azuredisk-2540 [38;5;243m12/30/22 03:10:26.688[0m Dec 30 03:10:26.754: INFO: deleting PVC "azuredisk-2540"/"pvc-txd99" Dec 30 03:10:26.754: INFO: Deleting PersistentVolumeClaim "pvc-txd99" ... skipping 30 lines ... [1mSTEP:[0m setting up the StorageClass [38;5;243m12/30/22 03:11:08.624[0m [1mSTEP:[0m creating a StorageClass [38;5;243m12/30/22 03:11:08.624[0m [1mSTEP:[0m setting up the PVC and PV [38;5;243m12/30/22 03:11:08.685[0m [1mSTEP:[0m creating a PVC [38;5;243m12/30/22 03:11:08.686[0m [1mSTEP:[0m setting up the pod [38;5;243m12/30/22 03:11:08.749[0m [1mSTEP:[0m deploying the pod [38;5;243m12/30/22 03:11:08.749[0m [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m12/30/22 03:11:08.809[0m Dec 30 03:11:08.810: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-wlskg" in namespace "azuredisk-4728" to be "Succeeded or Failed" Dec 30 03:11:08.869: INFO: Pod "azuredisk-volume-tester-wlskg": Phase="Pending", Reason="", readiness=false. Elapsed: 59.594618ms Dec 30 03:11:10.929: INFO: Pod "azuredisk-volume-tester-wlskg": Phase="Pending", Reason="", readiness=false. Elapsed: 2.119437562s Dec 30 03:11:12.930: INFO: Pod "azuredisk-volume-tester-wlskg": Phase="Pending", Reason="", readiness=false. Elapsed: 4.120211716s Dec 30 03:11:14.929: INFO: Pod "azuredisk-volume-tester-wlskg": Phase="Pending", Reason="", readiness=false. Elapsed: 6.11908193s Dec 30 03:11:16.929: INFO: Pod "azuredisk-volume-tester-wlskg": Phase="Pending", Reason="", readiness=false. Elapsed: 8.11916048s Dec 30 03:11:18.929: INFO: Pod "azuredisk-volume-tester-wlskg": Phase="Pending", Reason="", readiness=false. Elapsed: 10.119518241s ... skipping 12 lines ... Dec 30 03:11:44.928: INFO: Pod "azuredisk-volume-tester-wlskg": Phase="Pending", Reason="", readiness=false. Elapsed: 36.118697328s Dec 30 03:11:46.929: INFO: Pod "azuredisk-volume-tester-wlskg": Phase="Pending", Reason="", readiness=false. Elapsed: 38.119416158s Dec 30 03:11:48.929: INFO: Pod "azuredisk-volume-tester-wlskg": Phase="Pending", Reason="", readiness=false. Elapsed: 40.119555114s Dec 30 03:11:50.928: INFO: Pod "azuredisk-volume-tester-wlskg": Phase="Pending", Reason="", readiness=false. Elapsed: 42.118646937s Dec 30 03:11:52.928: INFO: Pod "azuredisk-volume-tester-wlskg": Phase="Succeeded", Reason="", readiness=false. Elapsed: 44.118038152s [1mSTEP:[0m Saw pod success [38;5;243m12/30/22 03:11:52.928[0m Dec 30 03:11:52.928: INFO: Pod "azuredisk-volume-tester-wlskg" satisfied condition "Succeeded or Failed" Dec 30 03:11:52.928: INFO: deleting Pod "azuredisk-4728"/"azuredisk-volume-tester-wlskg" Dec 30 03:11:53.018: INFO: Pod azuredisk-volume-tester-wlskg has the following logs: hello world [1mSTEP:[0m Deleting pod azuredisk-volume-tester-wlskg in namespace azuredisk-4728 [38;5;243m12/30/22 03:11:53.018[0m [1mSTEP:[0m validating provisioned PV [38;5;243m12/30/22 03:11:53.142[0m [1mSTEP:[0m checking the PV [38;5;243m12/30/22 03:11:53.2[0m ... skipping 33 lines ... [1mSTEP:[0m setting up the StorageClass [38;5;243m12/30/22 03:11:08.624[0m [1mSTEP:[0m creating a StorageClass [38;5;243m12/30/22 03:11:08.624[0m [1mSTEP:[0m setting up the PVC and PV [38;5;243m12/30/22 03:11:08.685[0m [1mSTEP:[0m creating a PVC [38;5;243m12/30/22 03:11:08.686[0m [1mSTEP:[0m setting up the pod [38;5;243m12/30/22 03:11:08.749[0m [1mSTEP:[0m deploying the pod [38;5;243m12/30/22 03:11:08.749[0m [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m12/30/22 03:11:08.809[0m Dec 30 03:11:08.810: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-wlskg" in namespace "azuredisk-4728" to be "Succeeded or Failed" Dec 30 03:11:08.869: INFO: Pod "azuredisk-volume-tester-wlskg": Phase="Pending", Reason="", readiness=false. Elapsed: 59.594618ms Dec 30 03:11:10.929: INFO: Pod "azuredisk-volume-tester-wlskg": Phase="Pending", Reason="", readiness=false. Elapsed: 2.119437562s Dec 30 03:11:12.930: INFO: Pod "azuredisk-volume-tester-wlskg": Phase="Pending", Reason="", readiness=false. Elapsed: 4.120211716s Dec 30 03:11:14.929: INFO: Pod "azuredisk-volume-tester-wlskg": Phase="Pending", Reason="", readiness=false. Elapsed: 6.11908193s Dec 30 03:11:16.929: INFO: Pod "azuredisk-volume-tester-wlskg": Phase="Pending", Reason="", readiness=false. Elapsed: 8.11916048s Dec 30 03:11:18.929: INFO: Pod "azuredisk-volume-tester-wlskg": Phase="Pending", Reason="", readiness=false. Elapsed: 10.119518241s ... skipping 12 lines ... Dec 30 03:11:44.928: INFO: Pod "azuredisk-volume-tester-wlskg": Phase="Pending", Reason="", readiness=false. Elapsed: 36.118697328s Dec 30 03:11:46.929: INFO: Pod "azuredisk-volume-tester-wlskg": Phase="Pending", Reason="", readiness=false. Elapsed: 38.119416158s Dec 30 03:11:48.929: INFO: Pod "azuredisk-volume-tester-wlskg": Phase="Pending", Reason="", readiness=false. Elapsed: 40.119555114s Dec 30 03:11:50.928: INFO: Pod "azuredisk-volume-tester-wlskg": Phase="Pending", Reason="", readiness=false. Elapsed: 42.118646937s Dec 30 03:11:52.928: INFO: Pod "azuredisk-volume-tester-wlskg": Phase="Succeeded", Reason="", readiness=false. Elapsed: 44.118038152s [1mSTEP:[0m Saw pod success [38;5;243m12/30/22 03:11:52.928[0m Dec 30 03:11:52.928: INFO: Pod "azuredisk-volume-tester-wlskg" satisfied condition "Succeeded or Failed" Dec 30 03:11:52.928: INFO: deleting Pod "azuredisk-4728"/"azuredisk-volume-tester-wlskg" Dec 30 03:11:53.018: INFO: Pod azuredisk-volume-tester-wlskg has the following logs: hello world [1mSTEP:[0m Deleting pod azuredisk-volume-tester-wlskg in namespace azuredisk-4728 [38;5;243m12/30/22 03:11:53.018[0m [1mSTEP:[0m validating provisioned PV [38;5;243m12/30/22 03:11:53.142[0m [1mSTEP:[0m checking the PV [38;5;243m12/30/22 03:11:53.2[0m ... skipping 34 lines ... [1mSTEP:[0m setting up the PVC and PV [38;5;243m12/30/22 03:12:35.133[0m [1mSTEP:[0m creating a PVC [38;5;243m12/30/22 03:12:35.133[0m [1mSTEP:[0m setting up the pod [38;5;243m12/30/22 03:12:35.193[0m [1mSTEP:[0m deploying the pod [38;5;243m12/30/22 03:12:35.194[0m [1mSTEP:[0m checking that the pod has 'FailedMount' event [38;5;243m12/30/22 03:12:35.253[0m Dec 30 03:12:57.370: INFO: deleting Pod "azuredisk-5466"/"azuredisk-volume-tester-rlkpm" Dec 30 03:12:57.432: INFO: Error getting logs for pod azuredisk-volume-tester-rlkpm: the server rejected our request for an unknown reason (get pods azuredisk-volume-tester-rlkpm) [1mSTEP:[0m Deleting pod azuredisk-volume-tester-rlkpm in namespace azuredisk-5466 [38;5;243m12/30/22 03:12:57.432[0m [1mSTEP:[0m validating provisioned PV [38;5;243m12/30/22 03:12:57.549[0m [1mSTEP:[0m checking the PV [38;5;243m12/30/22 03:12:57.608[0m Dec 30 03:12:57.608: INFO: deleting PVC "azuredisk-5466"/"pvc-6862x" Dec 30 03:12:57.608: INFO: Deleting PersistentVolumeClaim "pvc-6862x" [1mSTEP:[0m waiting for claim's PV "pvc-6eed3ff8-9ce1-43fb-a6c8-1a50bff430f5" to be deleted [38;5;243m12/30/22 03:12:57.668[0m ... skipping 32 lines ... [1mSTEP:[0m setting up the PVC and PV [38;5;243m12/30/22 03:12:35.133[0m [1mSTEP:[0m creating a PVC [38;5;243m12/30/22 03:12:35.133[0m [1mSTEP:[0m setting up the pod [38;5;243m12/30/22 03:12:35.193[0m [1mSTEP:[0m deploying the pod [38;5;243m12/30/22 03:12:35.194[0m [1mSTEP:[0m checking that the pod has 'FailedMount' event [38;5;243m12/30/22 03:12:35.253[0m Dec 30 03:12:57.370: INFO: deleting Pod "azuredisk-5466"/"azuredisk-volume-tester-rlkpm" Dec 30 03:12:57.432: INFO: Error getting logs for pod azuredisk-volume-tester-rlkpm: the server rejected our request for an unknown reason (get pods azuredisk-volume-tester-rlkpm) [1mSTEP:[0m Deleting pod azuredisk-volume-tester-rlkpm in namespace azuredisk-5466 [38;5;243m12/30/22 03:12:57.432[0m [1mSTEP:[0m validating provisioned PV [38;5;243m12/30/22 03:12:57.549[0m [1mSTEP:[0m checking the PV [38;5;243m12/30/22 03:12:57.608[0m Dec 30 03:12:57.608: INFO: deleting PVC "azuredisk-5466"/"pvc-6862x" Dec 30 03:12:57.608: INFO: Deleting PersistentVolumeClaim "pvc-6862x" [1mSTEP:[0m waiting for claim's PV "pvc-6eed3ff8-9ce1-43fb-a6c8-1a50bff430f5" to be deleted [38;5;243m12/30/22 03:12:57.668[0m ... skipping 29 lines ... [1mSTEP:[0m setting up the StorageClass [38;5;243m12/30/22 03:13:39.391[0m [1mSTEP:[0m creating a StorageClass [38;5;243m12/30/22 03:13:39.391[0m [1mSTEP:[0m setting up the PVC and PV [38;5;243m12/30/22 03:13:39.45[0m [1mSTEP:[0m creating a PVC [38;5;243m12/30/22 03:13:39.45[0m [1mSTEP:[0m setting up the pod [38;5;243m12/30/22 03:13:39.52[0m [1mSTEP:[0m deploying the pod [38;5;243m12/30/22 03:13:39.52[0m [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m12/30/22 03:13:39.583[0m Dec 30 03:13:39.583: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-s4wg7" in namespace "azuredisk-2790" to be "Succeeded or Failed" Dec 30 03:13:39.641: INFO: Pod "azuredisk-volume-tester-s4wg7": Phase="Pending", Reason="", readiness=false. Elapsed: 57.647504ms Dec 30 03:13:41.701: INFO: Pod "azuredisk-volume-tester-s4wg7": Phase="Pending", Reason="", readiness=false. Elapsed: 2.118035219s Dec 30 03:13:43.699: INFO: Pod "azuredisk-volume-tester-s4wg7": Phase="Pending", Reason="", readiness=false. Elapsed: 4.11598014s Dec 30 03:13:45.699: INFO: Pod "azuredisk-volume-tester-s4wg7": Phase="Pending", Reason="", readiness=false. Elapsed: 6.115811939s Dec 30 03:13:47.699: INFO: Pod "azuredisk-volume-tester-s4wg7": Phase="Pending", Reason="", readiness=false. Elapsed: 8.115595595s Dec 30 03:13:49.699: INFO: Pod "azuredisk-volume-tester-s4wg7": Phase="Pending", Reason="", readiness=false. Elapsed: 10.116219789s ... skipping 3 lines ... Dec 30 03:13:57.701: INFO: Pod "azuredisk-volume-tester-s4wg7": Phase="Pending", Reason="", readiness=false. Elapsed: 18.117344815s Dec 30 03:13:59.703: INFO: Pod "azuredisk-volume-tester-s4wg7": Phase="Pending", Reason="", readiness=false. Elapsed: 20.120239926s Dec 30 03:14:01.699: INFO: Pod "azuredisk-volume-tester-s4wg7": Phase="Pending", Reason="", readiness=false. Elapsed: 22.11622269s Dec 30 03:14:03.701: INFO: Pod "azuredisk-volume-tester-s4wg7": Phase="Pending", Reason="", readiness=false. Elapsed: 24.118231655s Dec 30 03:14:05.700: INFO: Pod "azuredisk-volume-tester-s4wg7": Phase="Succeeded", Reason="", readiness=false. Elapsed: 26.116540277s [1mSTEP:[0m Saw pod success [38;5;243m12/30/22 03:14:05.7[0m Dec 30 03:14:05.700: INFO: Pod "azuredisk-volume-tester-s4wg7" satisfied condition "Succeeded or Failed" Dec 30 03:14:05.700: INFO: deleting Pod "azuredisk-2790"/"azuredisk-volume-tester-s4wg7" Dec 30 03:14:05.764: INFO: Pod azuredisk-volume-tester-s4wg7 has the following logs: e2e-test [1mSTEP:[0m Deleting pod azuredisk-volume-tester-s4wg7 in namespace azuredisk-2790 [38;5;243m12/30/22 03:14:05.764[0m [1mSTEP:[0m validating provisioned PV [38;5;243m12/30/22 03:14:05.884[0m [1mSTEP:[0m checking the PV [38;5;243m12/30/22 03:14:05.942[0m ... skipping 33 lines ... [1mSTEP:[0m setting up the StorageClass [38;5;243m12/30/22 03:13:39.391[0m [1mSTEP:[0m creating a StorageClass [38;5;243m12/30/22 03:13:39.391[0m [1mSTEP:[0m setting up the PVC and PV [38;5;243m12/30/22 03:13:39.45[0m [1mSTEP:[0m creating a PVC [38;5;243m12/30/22 03:13:39.45[0m [1mSTEP:[0m setting up the pod [38;5;243m12/30/22 03:13:39.52[0m [1mSTEP:[0m deploying the pod [38;5;243m12/30/22 03:13:39.52[0m [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m12/30/22 03:13:39.583[0m Dec 30 03:13:39.583: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-s4wg7" in namespace "azuredisk-2790" to be "Succeeded or Failed" Dec 30 03:13:39.641: INFO: Pod "azuredisk-volume-tester-s4wg7": Phase="Pending", Reason="", readiness=false. Elapsed: 57.647504ms Dec 30 03:13:41.701: INFO: Pod "azuredisk-volume-tester-s4wg7": Phase="Pending", Reason="", readiness=false. Elapsed: 2.118035219s Dec 30 03:13:43.699: INFO: Pod "azuredisk-volume-tester-s4wg7": Phase="Pending", Reason="", readiness=false. Elapsed: 4.11598014s Dec 30 03:13:45.699: INFO: Pod "azuredisk-volume-tester-s4wg7": Phase="Pending", Reason="", readiness=false. Elapsed: 6.115811939s Dec 30 03:13:47.699: INFO: Pod "azuredisk-volume-tester-s4wg7": Phase="Pending", Reason="", readiness=false. Elapsed: 8.115595595s Dec 30 03:13:49.699: INFO: Pod "azuredisk-volume-tester-s4wg7": Phase="Pending", Reason="", readiness=false. Elapsed: 10.116219789s ... skipping 3 lines ... Dec 30 03:13:57.701: INFO: Pod "azuredisk-volume-tester-s4wg7": Phase="Pending", Reason="", readiness=false. Elapsed: 18.117344815s Dec 30 03:13:59.703: INFO: Pod "azuredisk-volume-tester-s4wg7": Phase="Pending", Reason="", readiness=false. Elapsed: 20.120239926s Dec 30 03:14:01.699: INFO: Pod "azuredisk-volume-tester-s4wg7": Phase="Pending", Reason="", readiness=false. Elapsed: 22.11622269s Dec 30 03:14:03.701: INFO: Pod "azuredisk-volume-tester-s4wg7": Phase="Pending", Reason="", readiness=false. Elapsed: 24.118231655s Dec 30 03:14:05.700: INFO: Pod "azuredisk-volume-tester-s4wg7": Phase="Succeeded", Reason="", readiness=false. Elapsed: 26.116540277s [1mSTEP:[0m Saw pod success [38;5;243m12/30/22 03:14:05.7[0m Dec 30 03:14:05.700: INFO: Pod "azuredisk-volume-tester-s4wg7" satisfied condition "Succeeded or Failed" Dec 30 03:14:05.700: INFO: deleting Pod "azuredisk-2790"/"azuredisk-volume-tester-s4wg7" Dec 30 03:14:05.764: INFO: Pod azuredisk-volume-tester-s4wg7 has the following logs: e2e-test [1mSTEP:[0m Deleting pod azuredisk-volume-tester-s4wg7 in namespace azuredisk-2790 [38;5;243m12/30/22 03:14:05.764[0m [1mSTEP:[0m validating provisioned PV [38;5;243m12/30/22 03:14:05.884[0m [1mSTEP:[0m checking the PV [38;5;243m12/30/22 03:14:05.942[0m ... skipping 37 lines ... [1mSTEP:[0m creating volume in external rg azuredisk-csi-driver-test-1da21cf0-87f0-11ed-a90a-762e3f62171e [38;5;243m12/30/22 03:14:49.564[0m [1mSTEP:[0m setting up the StorageClass [38;5;243m12/30/22 03:14:49.564[0m [1mSTEP:[0m creating a StorageClass [38;5;243m12/30/22 03:14:49.564[0m [1mSTEP:[0m setting up the PVC and PV [38;5;243m12/30/22 03:14:49.623[0m [1mSTEP:[0m creating a PVC [38;5;243m12/30/22 03:14:49.623[0m [1mSTEP:[0m deploying the pod [38;5;243m12/30/22 03:14:49.685[0m [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m12/30/22 03:14:49.744[0m Dec 30 03:14:49.745: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-m6567" in namespace "azuredisk-5356" to be "Succeeded or Failed" Dec 30 03:14:49.803: INFO: Pod "azuredisk-volume-tester-m6567": Phase="Pending", Reason="", readiness=false. Elapsed: 57.609573ms Dec 30 03:14:51.862: INFO: Pod "azuredisk-volume-tester-m6567": Phase="Pending", Reason="", readiness=false. Elapsed: 2.117312513s Dec 30 03:14:53.862: INFO: Pod "azuredisk-volume-tester-m6567": Phase="Pending", Reason="", readiness=false. Elapsed: 4.117154861s Dec 30 03:14:55.864: INFO: Pod "azuredisk-volume-tester-m6567": Phase="Pending", Reason="", readiness=false. Elapsed: 6.118661877s Dec 30 03:14:57.864: INFO: Pod "azuredisk-volume-tester-m6567": Phase="Pending", Reason="", readiness=false. Elapsed: 8.118685766s Dec 30 03:14:59.862: INFO: Pod "azuredisk-volume-tester-m6567": Phase="Pending", Reason="", readiness=false. Elapsed: 10.117257922s ... skipping 2 lines ... Dec 30 03:15:05.862: INFO: Pod "azuredisk-volume-tester-m6567": Phase="Pending", Reason="", readiness=false. Elapsed: 16.116705512s Dec 30 03:15:07.862: INFO: Pod "azuredisk-volume-tester-m6567": Phase="Pending", Reason="", readiness=false. Elapsed: 18.116586512s Dec 30 03:15:09.864: INFO: Pod "azuredisk-volume-tester-m6567": Phase="Pending", Reason="", readiness=false. Elapsed: 20.119464327s Dec 30 03:15:11.867: INFO: Pod "azuredisk-volume-tester-m6567": Phase="Pending", Reason="", readiness=false. Elapsed: 22.1223793s Dec 30 03:15:13.862: INFO: Pod "azuredisk-volume-tester-m6567": Phase="Succeeded", Reason="", readiness=false. Elapsed: 24.116793191s [1mSTEP:[0m Saw pod success [38;5;243m12/30/22 03:15:13.862[0m Dec 30 03:15:13.862: INFO: Pod "azuredisk-volume-tester-m6567" satisfied condition "Succeeded or Failed" Dec 30 03:15:13.862: INFO: deleting Pod "azuredisk-5356"/"azuredisk-volume-tester-m6567" Dec 30 03:15:13.928: INFO: Pod azuredisk-volume-tester-m6567 has the following logs: hello world [1mSTEP:[0m Deleting pod azuredisk-volume-tester-m6567 in namespace azuredisk-5356 [38;5;243m12/30/22 03:15:13.928[0m [1mSTEP:[0m validating provisioned PV [38;5;243m12/30/22 03:15:14.049[0m [1mSTEP:[0m checking the PV [38;5;243m12/30/22 03:15:14.106[0m ... skipping 37 lines ... [1mSTEP:[0m creating volume in external rg azuredisk-csi-driver-test-1da21cf0-87f0-11ed-a90a-762e3f62171e [38;5;243m12/30/22 03:14:49.564[0m [1mSTEP:[0m setting up the StorageClass [38;5;243m12/30/22 03:14:49.564[0m [1mSTEP:[0m creating a StorageClass [38;5;243m12/30/22 03:14:49.564[0m [1mSTEP:[0m setting up the PVC and PV [38;5;243m12/30/22 03:14:49.623[0m [1mSTEP:[0m creating a PVC [38;5;243m12/30/22 03:14:49.623[0m [1mSTEP:[0m deploying the pod [38;5;243m12/30/22 03:14:49.685[0m [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m12/30/22 03:14:49.744[0m Dec 30 03:14:49.745: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-m6567" in namespace "azuredisk-5356" to be "Succeeded or Failed" Dec 30 03:14:49.803: INFO: Pod "azuredisk-volume-tester-m6567": Phase="Pending", Reason="", readiness=false. Elapsed: 57.609573ms Dec 30 03:14:51.862: INFO: Pod "azuredisk-volume-tester-m6567": Phase="Pending", Reason="", readiness=false. Elapsed: 2.117312513s Dec 30 03:14:53.862: INFO: Pod "azuredisk-volume-tester-m6567": Phase="Pending", Reason="", readiness=false. Elapsed: 4.117154861s Dec 30 03:14:55.864: INFO: Pod "azuredisk-volume-tester-m6567": Phase="Pending", Reason="", readiness=false. Elapsed: 6.118661877s Dec 30 03:14:57.864: INFO: Pod "azuredisk-volume-tester-m6567": Phase="Pending", Reason="", readiness=false. Elapsed: 8.118685766s Dec 30 03:14:59.862: INFO: Pod "azuredisk-volume-tester-m6567": Phase="Pending", Reason="", readiness=false. Elapsed: 10.117257922s ... skipping 2 lines ... Dec 30 03:15:05.862: INFO: Pod "azuredisk-volume-tester-m6567": Phase="Pending", Reason="", readiness=false. Elapsed: 16.116705512s Dec 30 03:15:07.862: INFO: Pod "azuredisk-volume-tester-m6567": Phase="Pending", Reason="", readiness=false. Elapsed: 18.116586512s Dec 30 03:15:09.864: INFO: Pod "azuredisk-volume-tester-m6567": Phase="Pending", Reason="", readiness=false. Elapsed: 20.119464327s Dec 30 03:15:11.867: INFO: Pod "azuredisk-volume-tester-m6567": Phase="Pending", Reason="", readiness=false. Elapsed: 22.1223793s Dec 30 03:15:13.862: INFO: Pod "azuredisk-volume-tester-m6567": Phase="Succeeded", Reason="", readiness=false. Elapsed: 24.116793191s [1mSTEP:[0m Saw pod success [38;5;243m12/30/22 03:15:13.862[0m Dec 30 03:15:13.862: INFO: Pod "azuredisk-volume-tester-m6567" satisfied condition "Succeeded or Failed" Dec 30 03:15:13.862: INFO: deleting Pod "azuredisk-5356"/"azuredisk-volume-tester-m6567" Dec 30 03:15:13.928: INFO: Pod azuredisk-volume-tester-m6567 has the following logs: hello world [1mSTEP:[0m Deleting pod azuredisk-volume-tester-m6567 in namespace azuredisk-5356 [38;5;243m12/30/22 03:15:13.928[0m [1mSTEP:[0m validating provisioned PV [38;5;243m12/30/22 03:15:14.049[0m [1mSTEP:[0m checking the PV [38;5;243m12/30/22 03:15:14.106[0m ... skipping 44 lines ... [1mSTEP:[0m creating volume in external rg azuredisk-csi-driver-test-504d9f85-87f0-11ed-a90a-762e3f62171e [38;5;243m12/30/22 03:16:13.178[0m [1mSTEP:[0m setting up the StorageClass [38;5;243m12/30/22 03:16:13.178[0m [1mSTEP:[0m creating a StorageClass [38;5;243m12/30/22 03:16:13.178[0m [1mSTEP:[0m setting up the PVC and PV [38;5;243m12/30/22 03:16:13.238[0m [1mSTEP:[0m creating a PVC [38;5;243m12/30/22 03:16:13.238[0m [1mSTEP:[0m deploying the pod [38;5;243m12/30/22 03:16:13.298[0m [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m12/30/22 03:16:13.357[0m Dec 30 03:16:13.357: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-kjx5j" in namespace "azuredisk-5194" to be "Succeeded or Failed" Dec 30 03:16:13.414: INFO: Pod "azuredisk-volume-tester-kjx5j": Phase="Pending", Reason="", readiness=false. Elapsed: 56.869285ms Dec 30 03:16:15.473: INFO: Pod "azuredisk-volume-tester-kjx5j": Phase="Pending", Reason="", readiness=false. Elapsed: 2.116347939s Dec 30 03:16:17.471: INFO: Pod "azuredisk-volume-tester-kjx5j": Phase="Pending", Reason="", readiness=false. Elapsed: 4.114341192s Dec 30 03:16:19.474: INFO: Pod "azuredisk-volume-tester-kjx5j": Phase="Pending", Reason="", readiness=false. Elapsed: 6.116758603s Dec 30 03:16:21.473: INFO: Pod "azuredisk-volume-tester-kjx5j": Phase="Pending", Reason="", readiness=false. Elapsed: 8.116301334s Dec 30 03:16:23.472: INFO: Pod "azuredisk-volume-tester-kjx5j": Phase="Pending", Reason="", readiness=false. Elapsed: 10.114882194s ... skipping 9 lines ... Dec 30 03:16:43.474: INFO: Pod "azuredisk-volume-tester-kjx5j": Phase="Pending", Reason="", readiness=false. Elapsed: 30.117461037s Dec 30 03:16:45.472: INFO: Pod "azuredisk-volume-tester-kjx5j": Phase="Pending", Reason="", readiness=false. Elapsed: 32.114932005s Dec 30 03:16:47.474: INFO: Pod "azuredisk-volume-tester-kjx5j": Phase="Pending", Reason="", readiness=false. Elapsed: 34.116828751s Dec 30 03:16:49.473: INFO: Pod "azuredisk-volume-tester-kjx5j": Phase="Pending", Reason="", readiness=false. Elapsed: 36.11629941s Dec 30 03:16:51.480: INFO: Pod "azuredisk-volume-tester-kjx5j": Phase="Succeeded", Reason="", readiness=false. Elapsed: 38.123426298s [1mSTEP:[0m Saw pod success [38;5;243m12/30/22 03:16:51.481[0m Dec 30 03:16:51.481: INFO: Pod "azuredisk-volume-tester-kjx5j" satisfied condition "Succeeded or Failed" Dec 30 03:16:51.481: INFO: deleting Pod "azuredisk-5194"/"azuredisk-volume-tester-kjx5j" Dec 30 03:16:51.575: INFO: Pod azuredisk-volume-tester-kjx5j has the following logs: hello world hello world [1mSTEP:[0m Deleting pod azuredisk-volume-tester-kjx5j in namespace azuredisk-5194 [38;5;243m12/30/22 03:16:51.575[0m [1mSTEP:[0m validating provisioned PV [38;5;243m12/30/22 03:16:51.695[0m ... skipping 57 lines ... [1mSTEP:[0m creating volume in external rg azuredisk-csi-driver-test-504d9f85-87f0-11ed-a90a-762e3f62171e [38;5;243m12/30/22 03:16:13.178[0m [1mSTEP:[0m setting up the StorageClass [38;5;243m12/30/22 03:16:13.178[0m [1mSTEP:[0m creating a StorageClass [38;5;243m12/30/22 03:16:13.178[0m [1mSTEP:[0m setting up the PVC and PV [38;5;243m12/30/22 03:16:13.238[0m [1mSTEP:[0m creating a PVC [38;5;243m12/30/22 03:16:13.238[0m [1mSTEP:[0m deploying the pod [38;5;243m12/30/22 03:16:13.298[0m [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m12/30/22 03:16:13.357[0m Dec 30 03:16:13.357: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-kjx5j" in namespace "azuredisk-5194" to be "Succeeded or Failed" Dec 30 03:16:13.414: INFO: Pod "azuredisk-volume-tester-kjx5j": Phase="Pending", Reason="", readiness=false. Elapsed: 56.869285ms Dec 30 03:16:15.473: INFO: Pod "azuredisk-volume-tester-kjx5j": Phase="Pending", Reason="", readiness=false. Elapsed: 2.116347939s Dec 30 03:16:17.471: INFO: Pod "azuredisk-volume-tester-kjx5j": Phase="Pending", Reason="", readiness=false. Elapsed: 4.114341192s Dec 30 03:16:19.474: INFO: Pod "azuredisk-volume-tester-kjx5j": Phase="Pending", Reason="", readiness=false. Elapsed: 6.116758603s Dec 30 03:16:21.473: INFO: Pod "azuredisk-volume-tester-kjx5j": Phase="Pending", Reason="", readiness=false. Elapsed: 8.116301334s Dec 30 03:16:23.472: INFO: Pod "azuredisk-volume-tester-kjx5j": Phase="Pending", Reason="", readiness=false. Elapsed: 10.114882194s ... skipping 9 lines ... Dec 30 03:16:43.474: INFO: Pod "azuredisk-volume-tester-kjx5j": Phase="Pending", Reason="", readiness=false. Elapsed: 30.117461037s Dec 30 03:16:45.472: INFO: Pod "azuredisk-volume-tester-kjx5j": Phase="Pending", Reason="", readiness=false. Elapsed: 32.114932005s Dec 30 03:16:47.474: INFO: Pod "azuredisk-volume-tester-kjx5j": Phase="Pending", Reason="", readiness=false. Elapsed: 34.116828751s Dec 30 03:16:49.473: INFO: Pod "azuredisk-volume-tester-kjx5j": Phase="Pending", Reason="", readiness=false. Elapsed: 36.11629941s Dec 30 03:16:51.480: INFO: Pod "azuredisk-volume-tester-kjx5j": Phase="Succeeded", Reason="", readiness=false. Elapsed: 38.123426298s [1mSTEP:[0m Saw pod success [38;5;243m12/30/22 03:16:51.481[0m Dec 30 03:16:51.481: INFO: Pod "azuredisk-volume-tester-kjx5j" satisfied condition "Succeeded or Failed" Dec 30 03:16:51.481: INFO: deleting Pod "azuredisk-5194"/"azuredisk-volume-tester-kjx5j" Dec 30 03:16:51.575: INFO: Pod azuredisk-volume-tester-kjx5j has the following logs: hello world hello world [1mSTEP:[0m Deleting pod azuredisk-volume-tester-kjx5j in namespace azuredisk-5194 [38;5;243m12/30/22 03:16:51.575[0m [1mSTEP:[0m validating provisioned PV [38;5;243m12/30/22 03:16:51.695[0m ... skipping 47 lines ... [1mSTEP:[0m setting up the StorageClass [38;5;243m12/30/22 03:18:15.862[0m [1mSTEP:[0m creating a StorageClass [38;5;243m12/30/22 03:18:15.862[0m [1mSTEP:[0m setting up the PVC and PV [38;5;243m12/30/22 03:18:15.921[0m [1mSTEP:[0m creating a PVC [38;5;243m12/30/22 03:18:15.921[0m [1mSTEP:[0m setting up the pod [38;5;243m12/30/22 03:18:15.982[0m [1mSTEP:[0m deploying the pod [38;5;243m12/30/22 03:18:15.982[0m [1mSTEP:[0m checking that the pod's command exits with an error [38;5;243m12/30/22 03:18:16.04[0m Dec 30 03:18:16.040: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-xh7h8" in namespace "azuredisk-1353" to be "Error status code" Dec 30 03:18:16.098: INFO: Pod "azuredisk-volume-tester-xh7h8": Phase="Pending", Reason="", readiness=false. Elapsed: 57.707376ms Dec 30 03:18:18.157: INFO: Pod "azuredisk-volume-tester-xh7h8": Phase="Pending", Reason="", readiness=false. Elapsed: 2.116556936s Dec 30 03:18:20.160: INFO: Pod "azuredisk-volume-tester-xh7h8": Phase="Pending", Reason="", readiness=false. Elapsed: 4.119528467s Dec 30 03:18:22.165: INFO: Pod "azuredisk-volume-tester-xh7h8": Phase="Pending", Reason="", readiness=false. Elapsed: 6.125245887s Dec 30 03:18:24.157: INFO: Pod "azuredisk-volume-tester-xh7h8": Phase="Pending", Reason="", readiness=false. Elapsed: 8.117254274s Dec 30 03:18:26.156: INFO: Pod "azuredisk-volume-tester-xh7h8": Phase="Pending", Reason="", readiness=false. Elapsed: 10.1159858s ... skipping 24 lines ... Dec 30 03:19:16.157: INFO: Pod "azuredisk-volume-tester-xh7h8": Phase="Pending", Reason="", readiness=false. Elapsed: 1m0.117154163s Dec 30 03:19:18.157: INFO: Pod "azuredisk-volume-tester-xh7h8": Phase="Pending", Reason="", readiness=false. Elapsed: 1m2.116892724s Dec 30 03:19:20.155: INFO: Pod "azuredisk-volume-tester-xh7h8": Phase="Pending", Reason="", readiness=false. Elapsed: 1m4.114937549s Dec 30 03:19:22.159: INFO: Pod "azuredisk-volume-tester-xh7h8": Phase="Pending", Reason="", readiness=false. Elapsed: 1m6.119085182s Dec 30 03:19:24.157: INFO: Pod "azuredisk-volume-tester-xh7h8": Phase="Pending", Reason="", readiness=false. Elapsed: 1m8.116776484s Dec 30 03:19:26.157: INFO: Pod "azuredisk-volume-tester-xh7h8": Phase="Pending", Reason="", readiness=false. Elapsed: 1m10.117028808s Dec 30 03:19:28.156: INFO: Pod "azuredisk-volume-tester-xh7h8": Phase="Failed", Reason="", readiness=false. Elapsed: 1m12.116134746s [1mSTEP:[0m Saw pod failure [38;5;243m12/30/22 03:19:28.156[0m Dec 30 03:19:28.156: INFO: Pod "azuredisk-volume-tester-xh7h8" satisfied condition "Error status code" [1mSTEP:[0m checking that pod logs contain expected message [38;5;243m12/30/22 03:19:28.156[0m Dec 30 03:19:28.244: INFO: deleting Pod "azuredisk-1353"/"azuredisk-volume-tester-xh7h8" Dec 30 03:19:28.305: INFO: Pod azuredisk-volume-tester-xh7h8 has the following logs: touch: /mnt/test-1/data: Read-only file system [1mSTEP:[0m Deleting pod azuredisk-volume-tester-xh7h8 in namespace azuredisk-1353 [38;5;243m12/30/22 03:19:28.305[0m [1mSTEP:[0m validating provisioned PV [38;5;243m12/30/22 03:19:28.427[0m ... skipping 34 lines ... [1mSTEP:[0m setting up the StorageClass [38;5;243m12/30/22 03:18:15.862[0m [1mSTEP:[0m creating a StorageClass [38;5;243m12/30/22 03:18:15.862[0m [1mSTEP:[0m setting up the PVC and PV [38;5;243m12/30/22 03:18:15.921[0m [1mSTEP:[0m creating a PVC [38;5;243m12/30/22 03:18:15.921[0m [1mSTEP:[0m setting up the pod [38;5;243m12/30/22 03:18:15.982[0m [1mSTEP:[0m deploying the pod [38;5;243m12/30/22 03:18:15.982[0m [1mSTEP:[0m checking that the pod's command exits with an error [38;5;243m12/30/22 03:18:16.04[0m Dec 30 03:18:16.040: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-xh7h8" in namespace "azuredisk-1353" to be "Error status code" Dec 30 03:18:16.098: INFO: Pod "azuredisk-volume-tester-xh7h8": Phase="Pending", Reason="", readiness=false. Elapsed: 57.707376ms Dec 30 03:18:18.157: INFO: Pod "azuredisk-volume-tester-xh7h8": Phase="Pending", Reason="", readiness=false. Elapsed: 2.116556936s Dec 30 03:18:20.160: INFO: Pod "azuredisk-volume-tester-xh7h8": Phase="Pending", Reason="", readiness=false. Elapsed: 4.119528467s Dec 30 03:18:22.165: INFO: Pod "azuredisk-volume-tester-xh7h8": Phase="Pending", Reason="", readiness=false. Elapsed: 6.125245887s Dec 30 03:18:24.157: INFO: Pod "azuredisk-volume-tester-xh7h8": Phase="Pending", Reason="", readiness=false. Elapsed: 8.117254274s Dec 30 03:18:26.156: INFO: Pod "azuredisk-volume-tester-xh7h8": Phase="Pending", Reason="", readiness=false. Elapsed: 10.1159858s ... skipping 24 lines ... Dec 30 03:19:16.157: INFO: Pod "azuredisk-volume-tester-xh7h8": Phase="Pending", Reason="", readiness=false. Elapsed: 1m0.117154163s Dec 30 03:19:18.157: INFO: Pod "azuredisk-volume-tester-xh7h8": Phase="Pending", Reason="", readiness=false. Elapsed: 1m2.116892724s Dec 30 03:19:20.155: INFO: Pod "azuredisk-volume-tester-xh7h8": Phase="Pending", Reason="", readiness=false. Elapsed: 1m4.114937549s Dec 30 03:19:22.159: INFO: Pod "azuredisk-volume-tester-xh7h8": Phase="Pending", Reason="", readiness=false. Elapsed: 1m6.119085182s Dec 30 03:19:24.157: INFO: Pod "azuredisk-volume-tester-xh7h8": Phase="Pending", Reason="", readiness=false. Elapsed: 1m8.116776484s Dec 30 03:19:26.157: INFO: Pod "azuredisk-volume-tester-xh7h8": Phase="Pending", Reason="", readiness=false. Elapsed: 1m10.117028808s Dec 30 03:19:28.156: INFO: Pod "azuredisk-volume-tester-xh7h8": Phase="Failed", Reason="", readiness=false. Elapsed: 1m12.116134746s [1mSTEP:[0m Saw pod failure [38;5;243m12/30/22 03:19:28.156[0m Dec 30 03:19:28.156: INFO: Pod "azuredisk-volume-tester-xh7h8" satisfied condition "Error status code" [1mSTEP:[0m checking that pod logs contain expected message [38;5;243m12/30/22 03:19:28.156[0m Dec 30 03:19:28.244: INFO: deleting Pod "azuredisk-1353"/"azuredisk-volume-tester-xh7h8" Dec 30 03:19:28.305: INFO: Pod azuredisk-volume-tester-xh7h8 has the following logs: touch: /mnt/test-1/data: Read-only file system [1mSTEP:[0m Deleting pod azuredisk-volume-tester-xh7h8 in namespace azuredisk-1353 [38;5;243m12/30/22 03:19:28.305[0m [1mSTEP:[0m validating provisioned PV [38;5;243m12/30/22 03:19:28.427[0m ... skipping 661 lines ... [1mSTEP:[0m setting up the StorageClass [38;5;243m12/30/22 03:27:41.87[0m [1mSTEP:[0m creating a StorageClass [38;5;243m12/30/22 03:27:41.87[0m [1mSTEP:[0m setting up the PVC and PV [38;5;243m12/30/22 03:27:41.931[0m [1mSTEP:[0m creating a PVC [38;5;243m12/30/22 03:27:41.931[0m [1mSTEP:[0m setting up the pod [38;5;243m12/30/22 03:27:41.992[0m [1mSTEP:[0m deploying the pod [38;5;243m12/30/22 03:27:41.993[0m [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m12/30/22 03:27:42.051[0m Dec 30 03:27:42.051: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-t7xdp" in namespace "azuredisk-59" to be "Succeeded or Failed" Dec 30 03:27:42.109: INFO: Pod "azuredisk-volume-tester-t7xdp": Phase="Pending", Reason="", readiness=false. Elapsed: 57.162302ms Dec 30 03:27:44.168: INFO: Pod "azuredisk-volume-tester-t7xdp": Phase="Pending", Reason="", readiness=false. Elapsed: 2.117081507s Dec 30 03:27:46.170: INFO: Pod "azuredisk-volume-tester-t7xdp": Phase="Pending", Reason="", readiness=false. Elapsed: 4.118442845s Dec 30 03:27:48.167: INFO: Pod "azuredisk-volume-tester-t7xdp": Phase="Pending", Reason="", readiness=false. Elapsed: 6.115663183s Dec 30 03:27:50.167: INFO: Pod "azuredisk-volume-tester-t7xdp": Phase="Pending", Reason="", readiness=false. Elapsed: 8.1157916s Dec 30 03:27:52.167: INFO: Pod "azuredisk-volume-tester-t7xdp": Phase="Pending", Reason="", readiness=false. Elapsed: 10.11595661s ... skipping 10 lines ... Dec 30 03:28:14.169: INFO: Pod "azuredisk-volume-tester-t7xdp": Phase="Pending", Reason="", readiness=false. Elapsed: 32.117974853s Dec 30 03:28:16.168: INFO: Pod "azuredisk-volume-tester-t7xdp": Phase="Pending", Reason="", readiness=false. Elapsed: 34.116306074s Dec 30 03:28:18.166: INFO: Pod "azuredisk-volume-tester-t7xdp": Phase="Pending", Reason="", readiness=false. Elapsed: 36.115065301s Dec 30 03:28:20.170: INFO: Pod "azuredisk-volume-tester-t7xdp": Phase="Pending", Reason="", readiness=false. Elapsed: 38.118355295s Dec 30 03:28:22.168: INFO: Pod "azuredisk-volume-tester-t7xdp": Phase="Succeeded", Reason="", readiness=false. Elapsed: 40.116727283s [1mSTEP:[0m Saw pod success [38;5;243m12/30/22 03:28:22.168[0m Dec 30 03:28:22.169: INFO: Pod "azuredisk-volume-tester-t7xdp" satisfied condition "Succeeded or Failed" [1mSTEP:[0m sleep 5s and then clone volume [38;5;243m12/30/22 03:28:22.169[0m [1mSTEP:[0m cloning existing volume [38;5;243m12/30/22 03:28:27.17[0m [1mSTEP:[0m setting up the PVC and PV [38;5;243m12/30/22 03:28:27.284[0m [1mSTEP:[0m creating a PVC [38;5;243m12/30/22 03:28:27.285[0m [1mSTEP:[0m setting up the pod [38;5;243m12/30/22 03:28:27.346[0m [1mSTEP:[0m deploying a second pod with cloned volume [38;5;243m12/30/22 03:28:27.347[0m [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m12/30/22 03:28:27.406[0m Dec 30 03:28:27.406: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-ljlct" in namespace "azuredisk-59" to be "Succeeded or Failed" Dec 30 03:28:27.464: INFO: Pod "azuredisk-volume-tester-ljlct": Phase="Pending", Reason="", readiness=false. Elapsed: 58.465849ms Dec 30 03:28:29.524: INFO: Pod "azuredisk-volume-tester-ljlct": Phase="Pending", Reason="", readiness=false. Elapsed: 2.118219066s Dec 30 03:28:31.523: INFO: Pod "azuredisk-volume-tester-ljlct": Phase="Pending", Reason="", readiness=false. Elapsed: 4.116958577s Dec 30 03:28:33.524: INFO: Pod "azuredisk-volume-tester-ljlct": Phase="Pending", Reason="", readiness=false. Elapsed: 6.117915407s Dec 30 03:28:35.523: INFO: Pod "azuredisk-volume-tester-ljlct": Phase="Pending", Reason="", readiness=false. Elapsed: 8.117088053s Dec 30 03:28:37.523: INFO: Pod "azuredisk-volume-tester-ljlct": Phase="Pending", Reason="", readiness=false. Elapsed: 10.117108027s ... skipping 10 lines ... Dec 30 03:28:59.523: INFO: Pod "azuredisk-volume-tester-ljlct": Phase="Pending", Reason="", readiness=false. Elapsed: 32.117379066s Dec 30 03:29:01.523: INFO: Pod "azuredisk-volume-tester-ljlct": Phase="Pending", Reason="", readiness=false. Elapsed: 34.116919783s Dec 30 03:29:03.522: INFO: Pod "azuredisk-volume-tester-ljlct": Phase="Pending", Reason="", readiness=false. Elapsed: 36.116369759s Dec 30 03:29:05.522: INFO: Pod "azuredisk-volume-tester-ljlct": Phase="Pending", Reason="", readiness=false. Elapsed: 38.116150095s Dec 30 03:29:07.522: INFO: Pod "azuredisk-volume-tester-ljlct": Phase="Succeeded", Reason="", readiness=false. Elapsed: 40.115907605s [1mSTEP:[0m Saw pod success [38;5;243m12/30/22 03:29:07.522[0m Dec 30 03:29:07.522: INFO: Pod "azuredisk-volume-tester-ljlct" satisfied condition "Succeeded or Failed" Dec 30 03:29:07.522: INFO: deleting Pod "azuredisk-59"/"azuredisk-volume-tester-ljlct" Dec 30 03:29:07.619: INFO: Pod azuredisk-volume-tester-ljlct has the following logs: hello world [1mSTEP:[0m Deleting pod azuredisk-volume-tester-ljlct in namespace azuredisk-59 [38;5;243m12/30/22 03:29:07.619[0m [1mSTEP:[0m validating provisioned PV [38;5;243m12/30/22 03:29:07.741[0m [1mSTEP:[0m checking the PV [38;5;243m12/30/22 03:29:07.799[0m ... skipping 47 lines ... [1mSTEP:[0m setting up the StorageClass [38;5;243m12/30/22 03:27:41.87[0m [1mSTEP:[0m creating a StorageClass [38;5;243m12/30/22 03:27:41.87[0m [1mSTEP:[0m setting up the PVC and PV [38;5;243m12/30/22 03:27:41.931[0m [1mSTEP:[0m creating a PVC [38;5;243m12/30/22 03:27:41.931[0m [1mSTEP:[0m setting up the pod [38;5;243m12/30/22 03:27:41.992[0m [1mSTEP:[0m deploying the pod [38;5;243m12/30/22 03:27:41.993[0m [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m12/30/22 03:27:42.051[0m Dec 30 03:27:42.051: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-t7xdp" in namespace "azuredisk-59" to be "Succeeded or Failed" Dec 30 03:27:42.109: INFO: Pod "azuredisk-volume-tester-t7xdp": Phase="Pending", Reason="", readiness=false. Elapsed: 57.162302ms Dec 30 03:27:44.168: INFO: Pod "azuredisk-volume-tester-t7xdp": Phase="Pending", Reason="", readiness=false. Elapsed: 2.117081507s Dec 30 03:27:46.170: INFO: Pod "azuredisk-volume-tester-t7xdp": Phase="Pending", Reason="", readiness=false. Elapsed: 4.118442845s Dec 30 03:27:48.167: INFO: Pod "azuredisk-volume-tester-t7xdp": Phase="Pending", Reason="", readiness=false. Elapsed: 6.115663183s Dec 30 03:27:50.167: INFO: Pod "azuredisk-volume-tester-t7xdp": Phase="Pending", Reason="", readiness=false. Elapsed: 8.1157916s Dec 30 03:27:52.167: INFO: Pod "azuredisk-volume-tester-t7xdp": Phase="Pending", Reason="", readiness=false. Elapsed: 10.11595661s ... skipping 10 lines ... Dec 30 03:28:14.169: INFO: Pod "azuredisk-volume-tester-t7xdp": Phase="Pending", Reason="", readiness=false. Elapsed: 32.117974853s Dec 30 03:28:16.168: INFO: Pod "azuredisk-volume-tester-t7xdp": Phase="Pending", Reason="", readiness=false. Elapsed: 34.116306074s Dec 30 03:28:18.166: INFO: Pod "azuredisk-volume-tester-t7xdp": Phase="Pending", Reason="", readiness=false. Elapsed: 36.115065301s Dec 30 03:28:20.170: INFO: Pod "azuredisk-volume-tester-t7xdp": Phase="Pending", Reason="", readiness=false. Elapsed: 38.118355295s Dec 30 03:28:22.168: INFO: Pod "azuredisk-volume-tester-t7xdp": Phase="Succeeded", Reason="", readiness=false. Elapsed: 40.116727283s [1mSTEP:[0m Saw pod success [38;5;243m12/30/22 03:28:22.168[0m Dec 30 03:28:22.169: INFO: Pod "azuredisk-volume-tester-t7xdp" satisfied condition "Succeeded or Failed" [1mSTEP:[0m sleep 5s and then clone volume [38;5;243m12/30/22 03:28:22.169[0m [1mSTEP:[0m cloning existing volume [38;5;243m12/30/22 03:28:27.17[0m [1mSTEP:[0m setting up the PVC and PV [38;5;243m12/30/22 03:28:27.284[0m [1mSTEP:[0m creating a PVC [38;5;243m12/30/22 03:28:27.285[0m [1mSTEP:[0m setting up the pod [38;5;243m12/30/22 03:28:27.346[0m [1mSTEP:[0m deploying a second pod with cloned volume [38;5;243m12/30/22 03:28:27.347[0m [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m12/30/22 03:28:27.406[0m Dec 30 03:28:27.406: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-ljlct" in namespace "azuredisk-59" to be "Succeeded or Failed" Dec 30 03:28:27.464: INFO: Pod "azuredisk-volume-tester-ljlct": Phase="Pending", Reason="", readiness=false. Elapsed: 58.465849ms Dec 30 03:28:29.524: INFO: Pod "azuredisk-volume-tester-ljlct": Phase="Pending", Reason="", readiness=false. Elapsed: 2.118219066s Dec 30 03:28:31.523: INFO: Pod "azuredisk-volume-tester-ljlct": Phase="Pending", Reason="", readiness=false. Elapsed: 4.116958577s Dec 30 03:28:33.524: INFO: Pod "azuredisk-volume-tester-ljlct": Phase="Pending", Reason="", readiness=false. Elapsed: 6.117915407s Dec 30 03:28:35.523: INFO: Pod "azuredisk-volume-tester-ljlct": Phase="Pending", Reason="", readiness=false. Elapsed: 8.117088053s Dec 30 03:28:37.523: INFO: Pod "azuredisk-volume-tester-ljlct": Phase="Pending", Reason="", readiness=false. Elapsed: 10.117108027s ... skipping 10 lines ... Dec 30 03:28:59.523: INFO: Pod "azuredisk-volume-tester-ljlct": Phase="Pending", Reason="", readiness=false. Elapsed: 32.117379066s Dec 30 03:29:01.523: INFO: Pod "azuredisk-volume-tester-ljlct": Phase="Pending", Reason="", readiness=false. Elapsed: 34.116919783s Dec 30 03:29:03.522: INFO: Pod "azuredisk-volume-tester-ljlct": Phase="Pending", Reason="", readiness=false. Elapsed: 36.116369759s Dec 30 03:29:05.522: INFO: Pod "azuredisk-volume-tester-ljlct": Phase="Pending", Reason="", readiness=false. Elapsed: 38.116150095s Dec 30 03:29:07.522: INFO: Pod "azuredisk-volume-tester-ljlct": Phase="Succeeded", Reason="", readiness=false. Elapsed: 40.115907605s [1mSTEP:[0m Saw pod success [38;5;243m12/30/22 03:29:07.522[0m Dec 30 03:29:07.522: INFO: Pod "azuredisk-volume-tester-ljlct" satisfied condition "Succeeded or Failed" Dec 30 03:29:07.522: INFO: deleting Pod "azuredisk-59"/"azuredisk-volume-tester-ljlct" Dec 30 03:29:07.619: INFO: Pod azuredisk-volume-tester-ljlct has the following logs: hello world [1mSTEP:[0m Deleting pod azuredisk-volume-tester-ljlct in namespace azuredisk-59 [38;5;243m12/30/22 03:29:07.619[0m [1mSTEP:[0m validating provisioned PV [38;5;243m12/30/22 03:29:07.741[0m [1mSTEP:[0m checking the PV [38;5;243m12/30/22 03:29:07.799[0m ... skipping 46 lines ... [1mSTEP:[0m setting up the StorageClass [38;5;243m12/30/22 03:29:59.984[0m [1mSTEP:[0m creating a StorageClass [38;5;243m12/30/22 03:29:59.984[0m [1mSTEP:[0m setting up the PVC and PV [38;5;243m12/30/22 03:30:00.044[0m [1mSTEP:[0m creating a PVC [38;5;243m12/30/22 03:30:00.044[0m [1mSTEP:[0m setting up the pod [38;5;243m12/30/22 03:30:00.104[0m [1mSTEP:[0m deploying the pod [38;5;243m12/30/22 03:30:00.105[0m [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m12/30/22 03:30:00.163[0m Dec 30 03:30:00.163: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-4gnkx" in namespace "azuredisk-2546" to be "Succeeded or Failed" Dec 30 03:30:00.220: INFO: Pod "azuredisk-volume-tester-4gnkx": Phase="Pending", Reason="", readiness=false. Elapsed: 56.600399ms Dec 30 03:30:02.278: INFO: Pod "azuredisk-volume-tester-4gnkx": Phase="Pending", Reason="", readiness=false. Elapsed: 2.114857486s Dec 30 03:30:04.278: INFO: Pod "azuredisk-volume-tester-4gnkx": Phase="Pending", Reason="", readiness=false. Elapsed: 4.115290389s Dec 30 03:30:06.279: INFO: Pod "azuredisk-volume-tester-4gnkx": Phase="Pending", Reason="", readiness=false. Elapsed: 6.115914411s Dec 30 03:30:08.278: INFO: Pod "azuredisk-volume-tester-4gnkx": Phase="Pending", Reason="", readiness=false. Elapsed: 8.115442842s Dec 30 03:30:10.280: INFO: Pod "azuredisk-volume-tester-4gnkx": Phase="Pending", Reason="", readiness=false. Elapsed: 10.116601254s ... skipping 2 lines ... Dec 30 03:30:16.279: INFO: Pod "azuredisk-volume-tester-4gnkx": Phase="Pending", Reason="", readiness=false. Elapsed: 16.115721688s Dec 30 03:30:18.279: INFO: Pod "azuredisk-volume-tester-4gnkx": Phase="Pending", Reason="", readiness=false. Elapsed: 18.116327847s Dec 30 03:30:20.279: INFO: Pod "azuredisk-volume-tester-4gnkx": Phase="Pending", Reason="", readiness=false. Elapsed: 20.115873326s Dec 30 03:30:22.277: INFO: Pod "azuredisk-volume-tester-4gnkx": Phase="Pending", Reason="", readiness=false. Elapsed: 22.113895597s Dec 30 03:30:24.278: INFO: Pod "azuredisk-volume-tester-4gnkx": Phase="Succeeded", Reason="", readiness=false. Elapsed: 24.115003326s [1mSTEP:[0m Saw pod success [38;5;243m12/30/22 03:30:24.278[0m Dec 30 03:30:24.278: INFO: Pod "azuredisk-volume-tester-4gnkx" satisfied condition "Succeeded or Failed" [1mSTEP:[0m sleep 5s and then clone volume [38;5;243m12/30/22 03:30:24.278[0m [1mSTEP:[0m cloning existing volume [38;5;243m12/30/22 03:30:29.28[0m [1mSTEP:[0m setting up the PVC and PV [38;5;243m12/30/22 03:30:29.4[0m [1mSTEP:[0m creating a PVC [38;5;243m12/30/22 03:30:29.4[0m [1mSTEP:[0m setting up the pod [38;5;243m12/30/22 03:30:29.461[0m [1mSTEP:[0m deploying a second pod with cloned volume [38;5;243m12/30/22 03:30:29.462[0m [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m12/30/22 03:30:29.521[0m Dec 30 03:30:29.522: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-rl8wv" in namespace "azuredisk-2546" to be "Succeeded or Failed" Dec 30 03:30:29.579: INFO: Pod "azuredisk-volume-tester-rl8wv": Phase="Pending", Reason="", readiness=false. Elapsed: 57.252899ms Dec 30 03:30:31.636: INFO: Pod "azuredisk-volume-tester-rl8wv": Phase="Pending", Reason="", readiness=false. Elapsed: 2.114762317s Dec 30 03:30:33.637: INFO: Pod "azuredisk-volume-tester-rl8wv": Phase="Pending", Reason="", readiness=false. Elapsed: 4.115289101s Dec 30 03:30:35.639: INFO: Pod "azuredisk-volume-tester-rl8wv": Phase="Pending", Reason="", readiness=false. Elapsed: 6.117323483s Dec 30 03:30:37.639: INFO: Pod "azuredisk-volume-tester-rl8wv": Phase="Pending", Reason="", readiness=false. Elapsed: 8.116917512s Dec 30 03:30:39.636: INFO: Pod "azuredisk-volume-tester-rl8wv": Phase="Pending", Reason="", readiness=false. Elapsed: 10.114619894s ... skipping 10 lines ... Dec 30 03:31:01.638: INFO: Pod "azuredisk-volume-tester-rl8wv": Phase="Pending", Reason="", readiness=false. Elapsed: 32.116371794s Dec 30 03:31:03.639: INFO: Pod "azuredisk-volume-tester-rl8wv": Phase="Pending", Reason="", readiness=false. Elapsed: 34.117451402s Dec 30 03:31:05.639: INFO: Pod "azuredisk-volume-tester-rl8wv": Phase="Pending", Reason="", readiness=false. Elapsed: 36.117263365s Dec 30 03:31:07.638: INFO: Pod "azuredisk-volume-tester-rl8wv": Phase="Pending", Reason="", readiness=false. Elapsed: 38.11662489s Dec 30 03:31:09.637: INFO: Pod "azuredisk-volume-tester-rl8wv": Phase="Succeeded", Reason="", readiness=false. Elapsed: 40.115532508s [1mSTEP:[0m Saw pod success [38;5;243m12/30/22 03:31:09.637[0m Dec 30 03:31:09.637: INFO: Pod "azuredisk-volume-tester-rl8wv" satisfied condition "Succeeded or Failed" Dec 30 03:31:09.637: INFO: deleting Pod "azuredisk-2546"/"azuredisk-volume-tester-rl8wv" Dec 30 03:31:09.699: INFO: Pod azuredisk-volume-tester-rl8wv has the following logs: 20.0G [1mSTEP:[0m Deleting pod azuredisk-volume-tester-rl8wv in namespace azuredisk-2546 [38;5;243m12/30/22 03:31:09.699[0m [1mSTEP:[0m validating provisioned PV [38;5;243m12/30/22 03:31:09.817[0m [1mSTEP:[0m checking the PV [38;5;243m12/30/22 03:31:09.873[0m ... skipping 47 lines ... [1mSTEP:[0m setting up the StorageClass [38;5;243m12/30/22 03:29:59.984[0m [1mSTEP:[0m creating a StorageClass [38;5;243m12/30/22 03:29:59.984[0m [1mSTEP:[0m setting up the PVC and PV [38;5;243m12/30/22 03:30:00.044[0m [1mSTEP:[0m creating a PVC [38;5;243m12/30/22 03:30:00.044[0m [1mSTEP:[0m setting up the pod [38;5;243m12/30/22 03:30:00.104[0m [1mSTEP:[0m deploying the pod [38;5;243m12/30/22 03:30:00.105[0m [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m12/30/22 03:30:00.163[0m Dec 30 03:30:00.163: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-4gnkx" in namespace "azuredisk-2546" to be "Succeeded or Failed" Dec 30 03:30:00.220: INFO: Pod "azuredisk-volume-tester-4gnkx": Phase="Pending", Reason="", readiness=false. Elapsed: 56.600399ms Dec 30 03:30:02.278: INFO: Pod "azuredisk-volume-tester-4gnkx": Phase="Pending", Reason="", readiness=false. Elapsed: 2.114857486s Dec 30 03:30:04.278: INFO: Pod "azuredisk-volume-tester-4gnkx": Phase="Pending", Reason="", readiness=false. Elapsed: 4.115290389s Dec 30 03:30:06.279: INFO: Pod "azuredisk-volume-tester-4gnkx": Phase="Pending", Reason="", readiness=false. Elapsed: 6.115914411s Dec 30 03:30:08.278: INFO: Pod "azuredisk-volume-tester-4gnkx": Phase="Pending", Reason="", readiness=false. Elapsed: 8.115442842s Dec 30 03:30:10.280: INFO: Pod "azuredisk-volume-tester-4gnkx": Phase="Pending", Reason="", readiness=false. Elapsed: 10.116601254s ... skipping 2 lines ... Dec 30 03:30:16.279: INFO: Pod "azuredisk-volume-tester-4gnkx": Phase="Pending", Reason="", readiness=false. Elapsed: 16.115721688s Dec 30 03:30:18.279: INFO: Pod "azuredisk-volume-tester-4gnkx": Phase="Pending", Reason="", readiness=false. Elapsed: 18.116327847s Dec 30 03:30:20.279: INFO: Pod "azuredisk-volume-tester-4gnkx": Phase="Pending", Reason="", readiness=false. Elapsed: 20.115873326s Dec 30 03:30:22.277: INFO: Pod "azuredisk-volume-tester-4gnkx": Phase="Pending", Reason="", readiness=false. Elapsed: 22.113895597s Dec 30 03:30:24.278: INFO: Pod "azuredisk-volume-tester-4gnkx": Phase="Succeeded", Reason="", readiness=false. Elapsed: 24.115003326s [1mSTEP:[0m Saw pod success [38;5;243m12/30/22 03:30:24.278[0m Dec 30 03:30:24.278: INFO: Pod "azuredisk-volume-tester-4gnkx" satisfied condition "Succeeded or Failed" [1mSTEP:[0m sleep 5s and then clone volume [38;5;243m12/30/22 03:30:24.278[0m [1mSTEP:[0m cloning existing volume [38;5;243m12/30/22 03:30:29.28[0m [1mSTEP:[0m setting up the PVC and PV [38;5;243m12/30/22 03:30:29.4[0m [1mSTEP:[0m creating a PVC [38;5;243m12/30/22 03:30:29.4[0m [1mSTEP:[0m setting up the pod [38;5;243m12/30/22 03:30:29.461[0m [1mSTEP:[0m deploying a second pod with cloned volume [38;5;243m12/30/22 03:30:29.462[0m [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m12/30/22 03:30:29.521[0m Dec 30 03:30:29.522: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-rl8wv" in namespace "azuredisk-2546" to be "Succeeded or Failed" Dec 30 03:30:29.579: INFO: Pod "azuredisk-volume-tester-rl8wv": Phase="Pending", Reason="", readiness=false. Elapsed: 57.252899ms Dec 30 03:30:31.636: INFO: Pod "azuredisk-volume-tester-rl8wv": Phase="Pending", Reason="", readiness=false. Elapsed: 2.114762317s Dec 30 03:30:33.637: INFO: Pod "azuredisk-volume-tester-rl8wv": Phase="Pending", Reason="", readiness=false. Elapsed: 4.115289101s Dec 30 03:30:35.639: INFO: Pod "azuredisk-volume-tester-rl8wv": Phase="Pending", Reason="", readiness=false. Elapsed: 6.117323483s Dec 30 03:30:37.639: INFO: Pod "azuredisk-volume-tester-rl8wv": Phase="Pending", Reason="", readiness=false. Elapsed: 8.116917512s Dec 30 03:30:39.636: INFO: Pod "azuredisk-volume-tester-rl8wv": Phase="Pending", Reason="", readiness=false. Elapsed: 10.114619894s ... skipping 10 lines ... Dec 30 03:31:01.638: INFO: Pod "azuredisk-volume-tester-rl8wv": Phase="Pending", Reason="", readiness=false. Elapsed: 32.116371794s Dec 30 03:31:03.639: INFO: Pod "azuredisk-volume-tester-rl8wv": Phase="Pending", Reason="", readiness=false. Elapsed: 34.117451402s Dec 30 03:31:05.639: INFO: Pod "azuredisk-volume-tester-rl8wv": Phase="Pending", Reason="", readiness=false. Elapsed: 36.117263365s Dec 30 03:31:07.638: INFO: Pod "azuredisk-volume-tester-rl8wv": Phase="Pending", Reason="", readiness=false. Elapsed: 38.11662489s Dec 30 03:31:09.637: INFO: Pod "azuredisk-volume-tester-rl8wv": Phase="Succeeded", Reason="", readiness=false. Elapsed: 40.115532508s [1mSTEP:[0m Saw pod success [38;5;243m12/30/22 03:31:09.637[0m Dec 30 03:31:09.637: INFO: Pod "azuredisk-volume-tester-rl8wv" satisfied condition "Succeeded or Failed" Dec 30 03:31:09.637: INFO: deleting Pod "azuredisk-2546"/"azuredisk-volume-tester-rl8wv" Dec 30 03:31:09.699: INFO: Pod azuredisk-volume-tester-rl8wv has the following logs: 20.0G [1mSTEP:[0m Deleting pod azuredisk-volume-tester-rl8wv in namespace azuredisk-2546 [38;5;243m12/30/22 03:31:09.699[0m [1mSTEP:[0m validating provisioned PV [38;5;243m12/30/22 03:31:09.817[0m [1mSTEP:[0m checking the PV [38;5;243m12/30/22 03:31:09.873[0m ... skipping 56 lines ... [1mSTEP:[0m setting up the StorageClass [38;5;243m12/30/22 03:32:02.337[0m [1mSTEP:[0m creating a StorageClass [38;5;243m12/30/22 03:32:02.337[0m [1mSTEP:[0m setting up the PVC and PV [38;5;243m12/30/22 03:32:02.395[0m [1mSTEP:[0m creating a PVC [38;5;243m12/30/22 03:32:02.395[0m [1mSTEP:[0m setting up the pod [38;5;243m12/30/22 03:32:02.454[0m [1mSTEP:[0m deploying the pod [38;5;243m12/30/22 03:32:02.454[0m [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m12/30/22 03:32:02.512[0m Dec 30 03:32:02.512: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-d89kd" in namespace "azuredisk-1598" to be "Succeeded or Failed" Dec 30 03:32:02.571: INFO: Pod "azuredisk-volume-tester-d89kd": Phase="Pending", Reason="", readiness=false. Elapsed: 58.973587ms Dec 30 03:32:04.630: INFO: Pod "azuredisk-volume-tester-d89kd": Phase="Pending", Reason="", readiness=false. Elapsed: 2.117719885s Dec 30 03:32:06.630: INFO: Pod "azuredisk-volume-tester-d89kd": Phase="Pending", Reason="", readiness=false. Elapsed: 4.117432434s Dec 30 03:32:08.631: INFO: Pod "azuredisk-volume-tester-d89kd": Phase="Pending", Reason="", readiness=false. Elapsed: 6.118094924s Dec 30 03:32:10.630: INFO: Pod "azuredisk-volume-tester-d89kd": Phase="Pending", Reason="", readiness=false. Elapsed: 8.117594745s Dec 30 03:32:12.630: INFO: Pod "azuredisk-volume-tester-d89kd": Phase="Pending", Reason="", readiness=false. Elapsed: 10.117763151s ... skipping 9 lines ... Dec 30 03:32:32.631: INFO: Pod "azuredisk-volume-tester-d89kd": Phase="Pending", Reason="", readiness=false. Elapsed: 30.118382093s Dec 30 03:32:34.630: INFO: Pod "azuredisk-volume-tester-d89kd": Phase="Pending", Reason="", readiness=false. Elapsed: 32.117298747s Dec 30 03:32:36.633: INFO: Pod "azuredisk-volume-tester-d89kd": Phase="Pending", Reason="", readiness=false. Elapsed: 34.120663641s Dec 30 03:32:38.630: INFO: Pod "azuredisk-volume-tester-d89kd": Phase="Pending", Reason="", readiness=false. Elapsed: 36.117913684s Dec 30 03:32:40.630: INFO: Pod "azuredisk-volume-tester-d89kd": Phase="Succeeded", Reason="", readiness=false. Elapsed: 38.117759353s [1mSTEP:[0m Saw pod success [38;5;243m12/30/22 03:32:40.63[0m Dec 30 03:32:40.631: INFO: Pod "azuredisk-volume-tester-d89kd" satisfied condition "Succeeded or Failed" Dec 30 03:32:40.631: INFO: deleting Pod "azuredisk-1598"/"azuredisk-volume-tester-d89kd" Dec 30 03:32:40.691: INFO: Pod azuredisk-volume-tester-d89kd has the following logs: hello world hello world hello world [1mSTEP:[0m Deleting pod azuredisk-volume-tester-d89kd in namespace azuredisk-1598 [38;5;243m12/30/22 03:32:40.692[0m ... skipping 75 lines ... [1mSTEP:[0m setting up the StorageClass [38;5;243m12/30/22 03:32:02.337[0m [1mSTEP:[0m creating a StorageClass [38;5;243m12/30/22 03:32:02.337[0m [1mSTEP:[0m setting up the PVC and PV [38;5;243m12/30/22 03:32:02.395[0m [1mSTEP:[0m creating a PVC [38;5;243m12/30/22 03:32:02.395[0m [1mSTEP:[0m setting up the pod [38;5;243m12/30/22 03:32:02.454[0m [1mSTEP:[0m deploying the pod [38;5;243m12/30/22 03:32:02.454[0m [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m12/30/22 03:32:02.512[0m Dec 30 03:32:02.512: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-d89kd" in namespace "azuredisk-1598" to be "Succeeded or Failed" Dec 30 03:32:02.571: INFO: Pod "azuredisk-volume-tester-d89kd": Phase="Pending", Reason="", readiness=false. Elapsed: 58.973587ms Dec 30 03:32:04.630: INFO: Pod "azuredisk-volume-tester-d89kd": Phase="Pending", Reason="", readiness=false. Elapsed: 2.117719885s Dec 30 03:32:06.630: INFO: Pod "azuredisk-volume-tester-d89kd": Phase="Pending", Reason="", readiness=false. Elapsed: 4.117432434s Dec 30 03:32:08.631: INFO: Pod "azuredisk-volume-tester-d89kd": Phase="Pending", Reason="", readiness=false. Elapsed: 6.118094924s Dec 30 03:32:10.630: INFO: Pod "azuredisk-volume-tester-d89kd": Phase="Pending", Reason="", readiness=false. Elapsed: 8.117594745s Dec 30 03:32:12.630: INFO: Pod "azuredisk-volume-tester-d89kd": Phase="Pending", Reason="", readiness=false. Elapsed: 10.117763151s ... skipping 9 lines ... Dec 30 03:32:32.631: INFO: Pod "azuredisk-volume-tester-d89kd": Phase="Pending", Reason="", readiness=false. Elapsed: 30.118382093s Dec 30 03:32:34.630: INFO: Pod "azuredisk-volume-tester-d89kd": Phase="Pending", Reason="", readiness=false. Elapsed: 32.117298747s Dec 30 03:32:36.633: INFO: Pod "azuredisk-volume-tester-d89kd": Phase="Pending", Reason="", readiness=false. Elapsed: 34.120663641s Dec 30 03:32:38.630: INFO: Pod "azuredisk-volume-tester-d89kd": Phase="Pending", Reason="", readiness=false. Elapsed: 36.117913684s Dec 30 03:32:40.630: INFO: Pod "azuredisk-volume-tester-d89kd": Phase="Succeeded", Reason="", readiness=false. Elapsed: 38.117759353s [1mSTEP:[0m Saw pod success [38;5;243m12/30/22 03:32:40.63[0m Dec 30 03:32:40.631: INFO: Pod "azuredisk-volume-tester-d89kd" satisfied condition "Succeeded or Failed" Dec 30 03:32:40.631: INFO: deleting Pod "azuredisk-1598"/"azuredisk-volume-tester-d89kd" Dec 30 03:32:40.691: INFO: Pod azuredisk-volume-tester-d89kd has the following logs: hello world hello world hello world [1mSTEP:[0m Deleting pod azuredisk-volume-tester-d89kd in namespace azuredisk-1598 [38;5;243m12/30/22 03:32:40.692[0m ... skipping 69 lines ... [1mSTEP:[0m setting up the StorageClass [38;5;243m12/30/22 03:34:13.987[0m [1mSTEP:[0m creating a StorageClass [38;5;243m12/30/22 03:34:13.987[0m [1mSTEP:[0m setting up the PVC and PV [38;5;243m12/30/22 03:34:14.045[0m [1mSTEP:[0m creating a PVC [38;5;243m12/30/22 03:34:14.045[0m [1mSTEP:[0m setting up the pod [38;5;243m12/30/22 03:34:14.105[0m [1mSTEP:[0m deploying the pod [38;5;243m12/30/22 03:34:14.105[0m [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m12/30/22 03:34:14.164[0m Dec 30 03:34:14.164: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-s4vs7" in namespace "azuredisk-3410" to be "Succeeded or Failed" Dec 30 03:34:14.222: INFO: Pod "azuredisk-volume-tester-s4vs7": Phase="Pending", Reason="", readiness=false. Elapsed: 57.370751ms Dec 30 03:34:16.280: INFO: Pod "azuredisk-volume-tester-s4vs7": Phase="Pending", Reason="", readiness=false. Elapsed: 2.115757522s Dec 30 03:34:18.280: INFO: Pod "azuredisk-volume-tester-s4vs7": Phase="Pending", Reason="", readiness=false. Elapsed: 4.115933057s Dec 30 03:34:20.280: INFO: Pod "azuredisk-volume-tester-s4vs7": Phase="Pending", Reason="", readiness=false. Elapsed: 6.115505326s Dec 30 03:34:22.282: INFO: Pod "azuredisk-volume-tester-s4vs7": Phase="Pending", Reason="", readiness=false. Elapsed: 8.117906386s Dec 30 03:34:24.283: INFO: Pod "azuredisk-volume-tester-s4vs7": Phase="Pending", Reason="", readiness=false. Elapsed: 10.119048126s ... skipping 10 lines ... Dec 30 03:34:46.280: INFO: Pod "azuredisk-volume-tester-s4vs7": Phase="Pending", Reason="", readiness=false. Elapsed: 32.115413396s Dec 30 03:34:48.280: INFO: Pod "azuredisk-volume-tester-s4vs7": Phase="Pending", Reason="", readiness=false. Elapsed: 34.115824124s Dec 30 03:34:50.280: INFO: Pod "azuredisk-volume-tester-s4vs7": Phase="Pending", Reason="", readiness=false. Elapsed: 36.115879509s Dec 30 03:34:52.281: INFO: Pod "azuredisk-volume-tester-s4vs7": Phase="Pending", Reason="", readiness=false. Elapsed: 38.116707971s Dec 30 03:34:54.282: INFO: Pod "azuredisk-volume-tester-s4vs7": Phase="Succeeded", Reason="", readiness=false. Elapsed: 40.117399379s [1mSTEP:[0m Saw pod success [38;5;243m12/30/22 03:34:54.282[0m Dec 30 03:34:54.282: INFO: Pod "azuredisk-volume-tester-s4vs7" satisfied condition "Succeeded or Failed" Dec 30 03:34:54.282: INFO: deleting Pod "azuredisk-3410"/"azuredisk-volume-tester-s4vs7" Dec 30 03:34:54.378: INFO: Pod azuredisk-volume-tester-s4vs7 has the following logs: 100+0 records in 100+0 records out 104857600 bytes (100.0MB) copied, 0.069770 seconds, 1.4GB/s hello world ... skipping 59 lines ... [1mSTEP:[0m setting up the StorageClass [38;5;243m12/30/22 03:34:13.987[0m [1mSTEP:[0m creating a StorageClass [38;5;243m12/30/22 03:34:13.987[0m [1mSTEP:[0m setting up the PVC and PV [38;5;243m12/30/22 03:34:14.045[0m [1mSTEP:[0m creating a PVC [38;5;243m12/30/22 03:34:14.045[0m [1mSTEP:[0m setting up the pod [38;5;243m12/30/22 03:34:14.105[0m [1mSTEP:[0m deploying the pod [38;5;243m12/30/22 03:34:14.105[0m [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m12/30/22 03:34:14.164[0m Dec 30 03:34:14.164: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-s4vs7" in namespace "azuredisk-3410" to be "Succeeded or Failed" Dec 30 03:34:14.222: INFO: Pod "azuredisk-volume-tester-s4vs7": Phase="Pending", Reason="", readiness=false. Elapsed: 57.370751ms Dec 30 03:34:16.280: INFO: Pod "azuredisk-volume-tester-s4vs7": Phase="Pending", Reason="", readiness=false. Elapsed: 2.115757522s Dec 30 03:34:18.280: INFO: Pod "azuredisk-volume-tester-s4vs7": Phase="Pending", Reason="", readiness=false. Elapsed: 4.115933057s Dec 30 03:34:20.280: INFO: Pod "azuredisk-volume-tester-s4vs7": Phase="Pending", Reason="", readiness=false. Elapsed: 6.115505326s Dec 30 03:34:22.282: INFO: Pod "azuredisk-volume-tester-s4vs7": Phase="Pending", Reason="", readiness=false. Elapsed: 8.117906386s Dec 30 03:34:24.283: INFO: Pod "azuredisk-volume-tester-s4vs7": Phase="Pending", Reason="", readiness=false. Elapsed: 10.119048126s ... skipping 10 lines ... Dec 30 03:34:46.280: INFO: Pod "azuredisk-volume-tester-s4vs7": Phase="Pending", Reason="", readiness=false. Elapsed: 32.115413396s Dec 30 03:34:48.280: INFO: Pod "azuredisk-volume-tester-s4vs7": Phase="Pending", Reason="", readiness=false. Elapsed: 34.115824124s Dec 30 03:34:50.280: INFO: Pod "azuredisk-volume-tester-s4vs7": Phase="Pending", Reason="", readiness=false. Elapsed: 36.115879509s Dec 30 03:34:52.281: INFO: Pod "azuredisk-volume-tester-s4vs7": Phase="Pending", Reason="", readiness=false. Elapsed: 38.116707971s Dec 30 03:34:54.282: INFO: Pod "azuredisk-volume-tester-s4vs7": Phase="Succeeded", Reason="", readiness=false. Elapsed: 40.117399379s [1mSTEP:[0m Saw pod success [38;5;243m12/30/22 03:34:54.282[0m Dec 30 03:34:54.282: INFO: Pod "azuredisk-volume-tester-s4vs7" satisfied condition "Succeeded or Failed" Dec 30 03:34:54.282: INFO: deleting Pod "azuredisk-3410"/"azuredisk-volume-tester-s4vs7" Dec 30 03:34:54.378: INFO: Pod azuredisk-volume-tester-s4vs7 has the following logs: 100+0 records in 100+0 records out 104857600 bytes (100.0MB) copied, 0.069770 seconds, 1.4GB/s hello world ... skipping 52 lines ... Dec 30 03:36:17.045: INFO: >>> kubeConfig: /root/tmp1650667201/kubeconfig/kubeconfig.westus2.json [1mSTEP:[0m setting up the StorageClass [38;5;243m12/30/22 03:36:17.046[0m [1mSTEP:[0m creating a StorageClass [38;5;243m12/30/22 03:36:17.046[0m [1mSTEP:[0m setting up the PVC and PV [38;5;243m12/30/22 03:36:17.105[0m [1mSTEP:[0m creating a PVC [38;5;243m12/30/22 03:36:17.105[0m [1mSTEP:[0m deploying the pod [38;5;243m12/30/22 03:36:17.165[0m [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m12/30/22 03:36:17.224[0m Dec 30 03:36:17.224: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-s4z6w" in namespace "azuredisk-8582" to be "Succeeded or Failed" Dec 30 03:36:17.281: INFO: Pod "azuredisk-volume-tester-s4z6w": Phase="Pending", Reason="", readiness=false. Elapsed: 56.605477ms Dec 30 03:36:19.339: INFO: Pod "azuredisk-volume-tester-s4z6w": Phase="Pending", Reason="", readiness=false. Elapsed: 2.114957238s Dec 30 03:36:21.340: INFO: Pod "azuredisk-volume-tester-s4z6w": Phase="Pending", Reason="", readiness=false. Elapsed: 4.116194839s Dec 30 03:36:23.339: INFO: Pod "azuredisk-volume-tester-s4z6w": Phase="Pending", Reason="", readiness=false. Elapsed: 6.114715311s Dec 30 03:36:25.339: INFO: Pod "azuredisk-volume-tester-s4z6w": Phase="Pending", Reason="", readiness=false. Elapsed: 8.115289613s Dec 30 03:36:27.339: INFO: Pod "azuredisk-volume-tester-s4z6w": Phase="Pending", Reason="", readiness=false. Elapsed: 10.115032131s ... skipping 2 lines ... Dec 30 03:36:33.338: INFO: Pod "azuredisk-volume-tester-s4z6w": Phase="Pending", Reason="", readiness=false. Elapsed: 16.114533126s Dec 30 03:36:35.340: INFO: Pod "azuredisk-volume-tester-s4z6w": Phase="Pending", Reason="", readiness=false. Elapsed: 18.115965498s Dec 30 03:36:37.338: INFO: Pod "azuredisk-volume-tester-s4z6w": Phase="Pending", Reason="", readiness=false. Elapsed: 20.114456013s Dec 30 03:36:39.339: INFO: Pod "azuredisk-volume-tester-s4z6w": Phase="Pending", Reason="", readiness=false. Elapsed: 22.11460048s Dec 30 03:36:41.340: INFO: Pod "azuredisk-volume-tester-s4z6w": Phase="Succeeded", Reason="", readiness=false. Elapsed: 24.115796482s [1mSTEP:[0m Saw pod success [38;5;243m12/30/22 03:36:41.34[0m Dec 30 03:36:41.340: INFO: Pod "azuredisk-volume-tester-s4z6w" satisfied condition "Succeeded or Failed" [1mSTEP:[0m Checking Prow test resource group [38;5;243m12/30/22 03:36:41.34[0m 2022/12/30 03:36:41 Running in Prow, converting AZURE_CREDENTIALS to AZURE_CREDENTIAL_FILE 2022/12/30 03:36:41 Reading credentials file /etc/azure-cred/credentials [1mSTEP:[0m Prow test resource group: kubetest-du8j4fcj [38;5;243m12/30/22 03:36:41.341[0m [1mSTEP:[0m Creating external resource group: azuredisk-csi-driver-test-2c998203-87f3-11ed-a90a-762e3f62171e [38;5;243m12/30/22 03:36:41.341[0m [1mSTEP:[0m creating volume snapshot class with external rg azuredisk-csi-driver-test-2c998203-87f3-11ed-a90a-762e3f62171e [38;5;243m12/30/22 03:36:43.084[0m ... skipping 5 lines ... [1mSTEP:[0m setting up the StorageClass [38;5;243m12/30/22 03:36:58.274[0m [1mSTEP:[0m creating a StorageClass [38;5;243m12/30/22 03:36:58.275[0m [1mSTEP:[0m setting up the PVC and PV [38;5;243m12/30/22 03:36:58.334[0m [1mSTEP:[0m creating a PVC [38;5;243m12/30/22 03:36:58.335[0m [1mSTEP:[0m setting up the pod [38;5;243m12/30/22 03:36:58.395[0m [1mSTEP:[0m deploying a pod with a volume restored from the snapshot [38;5;243m12/30/22 03:36:58.395[0m [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m12/30/22 03:36:58.452[0m Dec 30 03:36:58.453: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-whf6n" in namespace "azuredisk-8582" to be "Succeeded or Failed" Dec 30 03:36:58.511: INFO: Pod "azuredisk-volume-tester-whf6n": Phase="Pending", Reason="", readiness=false. Elapsed: 58.15607ms Dec 30 03:37:00.570: INFO: Pod "azuredisk-volume-tester-whf6n": Phase="Pending", Reason="", readiness=false. Elapsed: 2.117300356s Dec 30 03:37:02.569: INFO: Pod "azuredisk-volume-tester-whf6n": Phase="Pending", Reason="", readiness=false. Elapsed: 4.116827381s Dec 30 03:37:04.571: INFO: Pod "azuredisk-volume-tester-whf6n": Phase="Pending", Reason="", readiness=false. Elapsed: 6.117915689s Dec 30 03:37:06.568: INFO: Pod "azuredisk-volume-tester-whf6n": Phase="Pending", Reason="", readiness=false. Elapsed: 8.115754872s Dec 30 03:37:08.572: INFO: Pod "azuredisk-volume-tester-whf6n": Phase="Pending", Reason="", readiness=false. Elapsed: 10.119535322s ... skipping 2 lines ... Dec 30 03:37:14.569: INFO: Pod "azuredisk-volume-tester-whf6n": Phase="Pending", Reason="", readiness=false. Elapsed: 16.11600722s Dec 30 03:37:16.569: INFO: Pod "azuredisk-volume-tester-whf6n": Phase="Pending", Reason="", readiness=false. Elapsed: 18.116229577s Dec 30 03:37:18.575: INFO: Pod "azuredisk-volume-tester-whf6n": Phase="Pending", Reason="", readiness=false. Elapsed: 20.121976877s Dec 30 03:37:20.571: INFO: Pod "azuredisk-volume-tester-whf6n": Phase="Pending", Reason="", readiness=false. Elapsed: 22.11802779s Dec 30 03:37:22.571: INFO: Pod "azuredisk-volume-tester-whf6n": Phase="Succeeded", Reason="", readiness=false. Elapsed: 24.118358906s [1mSTEP:[0m Saw pod success [38;5;243m12/30/22 03:37:22.571[0m Dec 30 03:37:22.571: INFO: Pod "azuredisk-volume-tester-whf6n" satisfied condition "Succeeded or Failed" Dec 30 03:37:22.571: INFO: deleting Pod "azuredisk-8582"/"azuredisk-volume-tester-whf6n" Dec 30 03:37:22.660: INFO: Pod azuredisk-volume-tester-whf6n has the following logs: hello world [1mSTEP:[0m Deleting pod azuredisk-volume-tester-whf6n in namespace azuredisk-8582 [38;5;243m12/30/22 03:37:22.66[0m [1mSTEP:[0m validating provisioned PV [38;5;243m12/30/22 03:37:22.782[0m [1mSTEP:[0m checking the PV [38;5;243m12/30/22 03:37:22.844[0m ... skipping 50 lines ... Dec 30 03:36:17.045: INFO: >>> kubeConfig: /root/tmp1650667201/kubeconfig/kubeconfig.westus2.json [1mSTEP:[0m setting up the StorageClass [38;5;243m12/30/22 03:36:17.046[0m [1mSTEP:[0m creating a StorageClass [38;5;243m12/30/22 03:36:17.046[0m [1mSTEP:[0m setting up the PVC and PV [38;5;243m12/30/22 03:36:17.105[0m [1mSTEP:[0m creating a PVC [38;5;243m12/30/22 03:36:17.105[0m [1mSTEP:[0m deploying the pod [38;5;243m12/30/22 03:36:17.165[0m [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m12/30/22 03:36:17.224[0m Dec 30 03:36:17.224: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-s4z6w" in namespace "azuredisk-8582" to be "Succeeded or Failed" Dec 30 03:36:17.281: INFO: Pod "azuredisk-volume-tester-s4z6w": Phase="Pending", Reason="", readiness=false. Elapsed: 56.605477ms Dec 30 03:36:19.339: INFO: Pod "azuredisk-volume-tester-s4z6w": Phase="Pending", Reason="", readiness=false. Elapsed: 2.114957238s Dec 30 03:36:21.340: INFO: Pod "azuredisk-volume-tester-s4z6w": Phase="Pending", Reason="", readiness=false. Elapsed: 4.116194839s Dec 30 03:36:23.339: INFO: Pod "azuredisk-volume-tester-s4z6w": Phase="Pending", Reason="", readiness=false. Elapsed: 6.114715311s Dec 30 03:36:25.339: INFO: Pod "azuredisk-volume-tester-s4z6w": Phase="Pending", Reason="", readiness=false. Elapsed: 8.115289613s Dec 30 03:36:27.339: INFO: Pod "azuredisk-volume-tester-s4z6w": Phase="Pending", Reason="", readiness=false. Elapsed: 10.115032131s ... skipping 2 lines ... Dec 30 03:36:33.338: INFO: Pod "azuredisk-volume-tester-s4z6w": Phase="Pending", Reason="", readiness=false. Elapsed: 16.114533126s Dec 30 03:36:35.340: INFO: Pod "azuredisk-volume-tester-s4z6w": Phase="Pending", Reason="", readiness=false. Elapsed: 18.115965498s Dec 30 03:36:37.338: INFO: Pod "azuredisk-volume-tester-s4z6w": Phase="Pending", Reason="", readiness=false. Elapsed: 20.114456013s Dec 30 03:36:39.339: INFO: Pod "azuredisk-volume-tester-s4z6w": Phase="Pending", Reason="", readiness=false. Elapsed: 22.11460048s Dec 30 03:36:41.340: INFO: Pod "azuredisk-volume-tester-s4z6w": Phase="Succeeded", Reason="", readiness=false. Elapsed: 24.115796482s [1mSTEP:[0m Saw pod success [38;5;243m12/30/22 03:36:41.34[0m Dec 30 03:36:41.340: INFO: Pod "azuredisk-volume-tester-s4z6w" satisfied condition "Succeeded or Failed" [1mSTEP:[0m Checking Prow test resource group [38;5;243m12/30/22 03:36:41.34[0m [1mSTEP:[0m Prow test resource group: kubetest-du8j4fcj [38;5;243m12/30/22 03:36:41.341[0m [1mSTEP:[0m Creating external resource group: azuredisk-csi-driver-test-2c998203-87f3-11ed-a90a-762e3f62171e [38;5;243m12/30/22 03:36:41.341[0m [1mSTEP:[0m creating volume snapshot class with external rg azuredisk-csi-driver-test-2c998203-87f3-11ed-a90a-762e3f62171e [38;5;243m12/30/22 03:36:43.084[0m [1mSTEP:[0m setting up the VolumeSnapshotClass [38;5;243m12/30/22 03:36:43.085[0m [1mSTEP:[0m creating a VolumeSnapshotClass [38;5;243m12/30/22 03:36:43.085[0m ... skipping 3 lines ... [1mSTEP:[0m setting up the StorageClass [38;5;243m12/30/22 03:36:58.274[0m [1mSTEP:[0m creating a StorageClass [38;5;243m12/30/22 03:36:58.275[0m [1mSTEP:[0m setting up the PVC and PV [38;5;243m12/30/22 03:36:58.334[0m [1mSTEP:[0m creating a PVC [38;5;243m12/30/22 03:36:58.335[0m [1mSTEP:[0m setting up the pod [38;5;243m12/30/22 03:36:58.395[0m [1mSTEP:[0m deploying a pod with a volume restored from the snapshot [38;5;243m12/30/22 03:36:58.395[0m [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m12/30/22 03:36:58.452[0m Dec 30 03:36:58.453: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-whf6n" in namespace "azuredisk-8582" to be "Succeeded or Failed" Dec 30 03:36:58.511: INFO: Pod "azuredisk-volume-tester-whf6n": Phase="Pending", Reason="", readiness=false. Elapsed: 58.15607ms Dec 30 03:37:00.570: INFO: Pod "azuredisk-volume-tester-whf6n": Phase="Pending", Reason="", readiness=false. Elapsed: 2.117300356s Dec 30 03:37:02.569: INFO: Pod "azuredisk-volume-tester-whf6n": Phase="Pending", Reason="", readiness=false. Elapsed: 4.116827381s Dec 30 03:37:04.571: INFO: Pod "azuredisk-volume-tester-whf6n": Phase="Pending", Reason="", readiness=false. Elapsed: 6.117915689s Dec 30 03:37:06.568: INFO: Pod "azuredisk-volume-tester-whf6n": Phase="Pending", Reason="", readiness=false. Elapsed: 8.115754872s Dec 30 03:37:08.572: INFO: Pod "azuredisk-volume-tester-whf6n": Phase="Pending", Reason="", readiness=false. Elapsed: 10.119535322s ... skipping 2 lines ... Dec 30 03:37:14.569: INFO: Pod "azuredisk-volume-tester-whf6n": Phase="Pending", Reason="", readiness=false. Elapsed: 16.11600722s Dec 30 03:37:16.569: INFO: Pod "azuredisk-volume-tester-whf6n": Phase="Pending", Reason="", readiness=false. Elapsed: 18.116229577s Dec 30 03:37:18.575: INFO: Pod "azuredisk-volume-tester-whf6n": Phase="Pending", Reason="", readiness=false. Elapsed: 20.121976877s Dec 30 03:37:20.571: INFO: Pod "azuredisk-volume-tester-whf6n": Phase="Pending", Reason="", readiness=false. Elapsed: 22.11802779s Dec 30 03:37:22.571: INFO: Pod "azuredisk-volume-tester-whf6n": Phase="Succeeded", Reason="", readiness=false. Elapsed: 24.118358906s [1mSTEP:[0m Saw pod success [38;5;243m12/30/22 03:37:22.571[0m Dec 30 03:37:22.571: INFO: Pod "azuredisk-volume-tester-whf6n" satisfied condition "Succeeded or Failed" Dec 30 03:37:22.571: INFO: deleting Pod "azuredisk-8582"/"azuredisk-volume-tester-whf6n" Dec 30 03:37:22.660: INFO: Pod azuredisk-volume-tester-whf6n has the following logs: hello world [1mSTEP:[0m Deleting pod azuredisk-volume-tester-whf6n in namespace azuredisk-8582 [38;5;243m12/30/22 03:37:22.66[0m [1mSTEP:[0m validating provisioned PV [38;5;243m12/30/22 03:37:22.782[0m [1mSTEP:[0m checking the PV [38;5;243m12/30/22 03:37:22.844[0m ... skipping 49 lines ... Dec 30 03:39:31.690: INFO: >>> kubeConfig: /root/tmp1650667201/kubeconfig/kubeconfig.westus2.json [1mSTEP:[0m setting up the StorageClass [38;5;243m12/30/22 03:39:31.692[0m [1mSTEP:[0m creating a StorageClass [38;5;243m12/30/22 03:39:31.692[0m [1mSTEP:[0m setting up the PVC and PV [38;5;243m12/30/22 03:39:31.75[0m [1mSTEP:[0m creating a PVC [38;5;243m12/30/22 03:39:31.75[0m [1mSTEP:[0m deploying the pod [38;5;243m12/30/22 03:39:31.81[0m [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m12/30/22 03:39:31.868[0m Dec 30 03:39:31.868: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-996mg" in namespace "azuredisk-7726" to be "Succeeded or Failed" Dec 30 03:39:31.926: INFO: Pod "azuredisk-volume-tester-996mg": Phase="Pending", Reason="", readiness=false. Elapsed: 57.870473ms Dec 30 03:39:33.985: INFO: Pod "azuredisk-volume-tester-996mg": Phase="Pending", Reason="", readiness=false. Elapsed: 2.11624104s Dec 30 03:39:35.985: INFO: Pod "azuredisk-volume-tester-996mg": Phase="Pending", Reason="", readiness=false. Elapsed: 4.11639493s Dec 30 03:39:37.985: INFO: Pod "azuredisk-volume-tester-996mg": Phase="Pending", Reason="", readiness=false. Elapsed: 6.116833353s Dec 30 03:39:39.991: INFO: Pod "azuredisk-volume-tester-996mg": Phase="Pending", Reason="", readiness=false. Elapsed: 8.122815199s Dec 30 03:39:41.987: INFO: Pod "azuredisk-volume-tester-996mg": Phase="Pending", Reason="", readiness=false. Elapsed: 10.118890869s ... skipping 2 lines ... Dec 30 03:39:47.986: INFO: Pod "azuredisk-volume-tester-996mg": Phase="Pending", Reason="", readiness=false. Elapsed: 16.117517426s Dec 30 03:39:49.985: INFO: Pod "azuredisk-volume-tester-996mg": Phase="Pending", Reason="", readiness=false. Elapsed: 18.116228905s Dec 30 03:39:51.985: INFO: Pod "azuredisk-volume-tester-996mg": Phase="Pending", Reason="", readiness=false. Elapsed: 20.117168936s Dec 30 03:39:53.985: INFO: Pod "azuredisk-volume-tester-996mg": Phase="Pending", Reason="", readiness=false. Elapsed: 22.117037918s Dec 30 03:39:55.987: INFO: Pod "azuredisk-volume-tester-996mg": Phase="Succeeded", Reason="", readiness=false. Elapsed: 24.118260724s [1mSTEP:[0m Saw pod success [38;5;243m12/30/22 03:39:55.987[0m Dec 30 03:39:55.987: INFO: Pod "azuredisk-volume-tester-996mg" satisfied condition "Succeeded or Failed" [1mSTEP:[0m Checking Prow test resource group [38;5;243m12/30/22 03:39:55.987[0m 2022/12/30 03:39:55 Running in Prow, converting AZURE_CREDENTIALS to AZURE_CREDENTIAL_FILE 2022/12/30 03:39:55 Reading credentials file /etc/azure-cred/credentials [1mSTEP:[0m Prow test resource group: kubetest-du8j4fcj [38;5;243m12/30/22 03:39:55.987[0m [1mSTEP:[0m Creating external resource group: azuredisk-csi-driver-test-a09e3f30-87f3-11ed-a90a-762e3f62171e [38;5;243m12/30/22 03:39:55.988[0m [1mSTEP:[0m creating volume snapshot class with external rg azuredisk-csi-driver-test-a09e3f30-87f3-11ed-a90a-762e3f62171e [38;5;243m12/30/22 03:39:56.971[0m ... skipping 12 lines ... [1mSTEP:[0m creating a StorageClass [38;5;243m12/30/22 03:40:14.327[0m [1mSTEP:[0m setting up the PVC and PV [38;5;243m12/30/22 03:40:14.384[0m [1mSTEP:[0m creating a PVC [38;5;243m12/30/22 03:40:14.384[0m [1mSTEP:[0m setting up the pod [38;5;243m12/30/22 03:40:14.453[0m [1mSTEP:[0m Set pod anti-affinity to make sure two pods are scheduled on different nodes [38;5;243m12/30/22 03:40:14.453[0m [1mSTEP:[0m deploying a pod with a volume restored from the snapshot [38;5;243m12/30/22 03:40:14.453[0m [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m12/30/22 03:40:14.512[0m Dec 30 03:40:14.512: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-hfgmq" in namespace "azuredisk-7726" to be "Succeeded or Failed" Dec 30 03:40:14.569: INFO: Pod "azuredisk-volume-tester-hfgmq": Phase="Pending", Reason="", readiness=false. Elapsed: 57.146472ms Dec 30 03:40:16.627: INFO: Pod "azuredisk-volume-tester-hfgmq": Phase="Pending", Reason="", readiness=false. Elapsed: 2.115231906s Dec 30 03:40:18.629: INFO: Pod "azuredisk-volume-tester-hfgmq": Phase="Pending", Reason="", readiness=false. Elapsed: 4.116535144s Dec 30 03:40:20.629: INFO: Pod "azuredisk-volume-tester-hfgmq": Phase="Pending", Reason="", readiness=false. Elapsed: 6.11651577s Dec 30 03:40:22.629: INFO: Pod "azuredisk-volume-tester-hfgmq": Phase="Pending", Reason="", readiness=false. Elapsed: 8.116893293s Dec 30 03:40:24.628: INFO: Pod "azuredisk-volume-tester-hfgmq": Phase="Pending", Reason="", readiness=false. Elapsed: 10.116391548s Dec 30 03:40:26.627: INFO: Pod "azuredisk-volume-tester-hfgmq": Phase="Pending", Reason="", readiness=false. Elapsed: 12.114957625s Dec 30 03:40:28.629: INFO: Pod "azuredisk-volume-tester-hfgmq": Phase="Pending", Reason="", readiness=false. Elapsed: 14.116825584s Dec 30 03:40:30.629: INFO: Pod "azuredisk-volume-tester-hfgmq": Phase="Pending", Reason="", readiness=false. Elapsed: 16.116734717s Dec 30 03:40:32.628: INFO: Pod "azuredisk-volume-tester-hfgmq": Phase="Pending", Reason="", readiness=false. Elapsed: 18.116430397s Dec 30 03:40:34.628: INFO: Pod "azuredisk-volume-tester-hfgmq": Phase="Pending", Reason="", readiness=false. Elapsed: 20.115527072s Dec 30 03:40:36.628: INFO: Pod "azuredisk-volume-tester-hfgmq": Phase="Pending", Reason="", readiness=false. Elapsed: 22.11584186s Dec 30 03:40:38.629: INFO: Pod "azuredisk-volume-tester-hfgmq": Phase="Failed", Reason="", readiness=false. Elapsed: 24.117040367s Dec 30 03:40:38.629: INFO: Unexpected error: <*fmt.wrapError | 0xc000ba2280>: { msg: "error while waiting for pod azuredisk-7726/azuredisk-volume-tester-hfgmq to be Succeeded or Failed: pod \"azuredisk-volume-tester-hfgmq\" failed with status: {Phase:Failed Conditions:[{Type:Initialized Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2022-12-30 03:40:17 +0000 UTC Reason: Message:} {Type:Ready Status:False LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2022-12-30 03:40:17 +0000 UTC Reason:ContainersNotReady Message:containers with unready status: [volume-tester]} {Type:ContainersReady Status:False LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2022-12-30 03:40:17 +0000 UTC Reason:ContainersNotReady Message:containers with unready status: [volume-tester]} {Type:PodScheduled Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2022-12-30 03:40:17 +0000 UTC Reason: Message:}] Message: Reason: NominatedNodeName: HostIP:10.248.0.32 PodIP:10.248.0.49 PodIPs:[{IP:10.248.0.49}] StartTime:2022-12-30 03:40:17 +0000 UTC InitContainerStatuses:[] ContainerStatuses:[{Name:volume-tester State:{Waiting:nil Running:nil Terminated:&ContainerStateTerminated{ExitCode:2,Signal:0,Reason:Error,Message:,StartedAt:2022-12-30 03:40:36 +0000 UTC,FinishedAt:2022-12-30 03:40:36 +0000 UTC,ContainerID:containerd://9e3ab7be3d104781dcbfa70bbf9f37893bb5194f9db50a8432ab3c3cfb1d933d,}} LastTerminationState:{Waiting:nil Running:nil Terminated:nil} Ready:false RestartCount:0 Image:registry.k8s.io/e2e-test-images/busybox:1.29-4 ImageID:registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 ContainerID:containerd://9e3ab7be3d104781dcbfa70bbf9f37893bb5194f9db50a8432ab3c3cfb1d933d Started:0xc000ae4a10}] QOSClass:BestEffort EphemeralContainerStatuses:[]}", err: <*errors.errorString | 0xc000212830>{ s: "pod \"azuredisk-volume-tester-hfgmq\" failed with status: {Phase:Failed Conditions:[{Type:Initialized Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2022-12-30 03:40:17 +0000 UTC Reason: Message:} {Type:Ready Status:False LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2022-12-30 03:40:17 +0000 UTC Reason:ContainersNotReady Message:containers with unready status: [volume-tester]} {Type:ContainersReady Status:False LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2022-12-30 03:40:17 +0000 UTC Reason:ContainersNotReady Message:containers with unready status: [volume-tester]} {Type:PodScheduled Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2022-12-30 03:40:17 +0000 UTC Reason: Message:}] Message: Reason: NominatedNodeName: HostIP:10.248.0.32 PodIP:10.248.0.49 PodIPs:[{IP:10.248.0.49}] StartTime:2022-12-30 03:40:17 +0000 UTC InitContainerStatuses:[] ContainerStatuses:[{Name:volume-tester State:{Waiting:nil Running:nil Terminated:&ContainerStateTerminated{ExitCode:2,Signal:0,Reason:Error,Message:,StartedAt:2022-12-30 03:40:36 +0000 UTC,FinishedAt:2022-12-30 03:40:36 +0000 UTC,ContainerID:containerd://9e3ab7be3d104781dcbfa70bbf9f37893bb5194f9db50a8432ab3c3cfb1d933d,}} LastTerminationState:{Waiting:nil Running:nil Terminated:nil} Ready:false RestartCount:0 Image:registry.k8s.io/e2e-test-images/busybox:1.29-4 ImageID:registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 ContainerID:containerd://9e3ab7be3d104781dcbfa70bbf9f37893bb5194f9db50a8432ab3c3cfb1d933d Started:0xc000ae4a10}] QOSClass:BestEffort EphemeralContainerStatuses:[]}", }, } Dec 30 03:40:38.629: FAIL: error while waiting for pod azuredisk-7726/azuredisk-volume-tester-hfgmq to be Succeeded or Failed: pod "azuredisk-volume-tester-hfgmq" failed with status: {Phase:Failed Conditions:[{Type:Initialized Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2022-12-30 03:40:17 +0000 UTC Reason: Message:} {Type:Ready Status:False LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2022-12-30 03:40:17 +0000 UTC Reason:ContainersNotReady Message:containers with unready status: [volume-tester]} {Type:ContainersReady Status:False LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2022-12-30 03:40:17 +0000 UTC Reason:ContainersNotReady Message:containers with unready status: [volume-tester]} {Type:PodScheduled Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2022-12-30 03:40:17 +0000 UTC Reason: Message:}] Message: Reason: NominatedNodeName: HostIP:10.248.0.32 PodIP:10.248.0.49 PodIPs:[{IP:10.248.0.49}] StartTime:2022-12-30 03:40:17 +0000 UTC InitContainerStatuses:[] ContainerStatuses:[{Name:volume-tester State:{Waiting:nil Running:nil Terminated:&ContainerStateTerminated{ExitCode:2,Signal:0,Reason:Error,Message:,StartedAt:2022-12-30 03:40:36 +0000 UTC,FinishedAt:2022-12-30 03:40:36 +0000 UTC,ContainerID:containerd://9e3ab7be3d104781dcbfa70bbf9f37893bb5194f9db50a8432ab3c3cfb1d933d,}} LastTerminationState:{Waiting:nil Running:nil Terminated:nil} Ready:false RestartCount:0 Image:registry.k8s.io/e2e-test-images/busybox:1.29-4 ImageID:registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 ContainerID:containerd://9e3ab7be3d104781dcbfa70bbf9f37893bb5194f9db50a8432ab3c3cfb1d933d Started:0xc000ae4a10}] QOSClass:BestEffort EphemeralContainerStatuses:[]} Full Stack Trace sigs.k8s.io/azuredisk-csi-driver/test/e2e/testsuites.(*TestPod).WaitForSuccess(0x2250430?) /home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/testsuites/testsuites.go:823 +0x5d sigs.k8s.io/azuredisk-csi-driver/test/e2e/testsuites.(*DynamicallyProvisionedVolumeSnapshotTest).Run(0xc000c51d78, {0x270a540, 0xc000d9d040}, {0x26f5740, 0xc0001b3360}, 0xc00078f340?) /home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/testsuites/dynamically_provisioned_volume_snapshot_tester.go:142 +0x1358 ... skipping 42 lines ... Dec 30 03:42:46.741: INFO: Waiting up to 5m0s for PersistentVolumeClaim azuredisk-7726 to be removed Dec 30 03:42:46.797: INFO: Claim "azuredisk-7726" in namespace "pvc-d2cnz" doesn't exist in the system Dec 30 03:42:46.797: INFO: deleting StorageClass azuredisk-7726-disk.csi.azure.com-dynamic-sc-9zvvn [1mSTEP:[0m dump namespace information after failure [38;5;243m12/30/22 03:42:46.858[0m [1mSTEP:[0m Destroying namespace "azuredisk-7726" for this suite. [38;5;243m12/30/22 03:42:46.858[0m [38;5;243m------------------------------[0m [38;5;9m• [FAILED] [196.141 seconds][0m Dynamic Provisioning [38;5;243m/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning_test.go:41[0m [multi-az] [38;5;243m/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning_test.go:48[0m [38;5;9m[1m[It] should create a pod, write to its pv, take a volume snapshot, overwrite data in original pv, create another pod from the snapshot, and read unaltered original data from original pv[disk.csi.azure.com][0m [38;5;243m/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning_test.go:747[0m ... skipping 7 lines ... Dec 30 03:39:31.690: INFO: >>> kubeConfig: /root/tmp1650667201/kubeconfig/kubeconfig.westus2.json [1mSTEP:[0m setting up the StorageClass [38;5;243m12/30/22 03:39:31.692[0m [1mSTEP:[0m creating a StorageClass [38;5;243m12/30/22 03:39:31.692[0m [1mSTEP:[0m setting up the PVC and PV [38;5;243m12/30/22 03:39:31.75[0m [1mSTEP:[0m creating a PVC [38;5;243m12/30/22 03:39:31.75[0m [1mSTEP:[0m deploying the pod [38;5;243m12/30/22 03:39:31.81[0m [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m12/30/22 03:39:31.868[0m Dec 30 03:39:31.868: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-996mg" in namespace "azuredisk-7726" to be "Succeeded or Failed" Dec 30 03:39:31.926: INFO: Pod "azuredisk-volume-tester-996mg": Phase="Pending", Reason="", readiness=false. Elapsed: 57.870473ms Dec 30 03:39:33.985: INFO: Pod "azuredisk-volume-tester-996mg": Phase="Pending", Reason="", readiness=false. Elapsed: 2.11624104s Dec 30 03:39:35.985: INFO: Pod "azuredisk-volume-tester-996mg": Phase="Pending", Reason="", readiness=false. Elapsed: 4.11639493s Dec 30 03:39:37.985: INFO: Pod "azuredisk-volume-tester-996mg": Phase="Pending", Reason="", readiness=false. Elapsed: 6.116833353s Dec 30 03:39:39.991: INFO: Pod "azuredisk-volume-tester-996mg": Phase="Pending", Reason="", readiness=false. Elapsed: 8.122815199s Dec 30 03:39:41.987: INFO: Pod "azuredisk-volume-tester-996mg": Phase="Pending", Reason="", readiness=false. Elapsed: 10.118890869s ... skipping 2 lines ... Dec 30 03:39:47.986: INFO: Pod "azuredisk-volume-tester-996mg": Phase="Pending", Reason="", readiness=false. Elapsed: 16.117517426s Dec 30 03:39:49.985: INFO: Pod "azuredisk-volume-tester-996mg": Phase="Pending", Reason="", readiness=false. Elapsed: 18.116228905s Dec 30 03:39:51.985: INFO: Pod "azuredisk-volume-tester-996mg": Phase="Pending", Reason="", readiness=false. Elapsed: 20.117168936s Dec 30 03:39:53.985: INFO: Pod "azuredisk-volume-tester-996mg": Phase="Pending", Reason="", readiness=false. Elapsed: 22.117037918s Dec 30 03:39:55.987: INFO: Pod "azuredisk-volume-tester-996mg": Phase="Succeeded", Reason="", readiness=false. Elapsed: 24.118260724s [1mSTEP:[0m Saw pod success [38;5;243m12/30/22 03:39:55.987[0m Dec 30 03:39:55.987: INFO: Pod "azuredisk-volume-tester-996mg" satisfied condition "Succeeded or Failed" [1mSTEP:[0m Checking Prow test resource group [38;5;243m12/30/22 03:39:55.987[0m [1mSTEP:[0m Prow test resource group: kubetest-du8j4fcj [38;5;243m12/30/22 03:39:55.987[0m [1mSTEP:[0m Creating external resource group: azuredisk-csi-driver-test-a09e3f30-87f3-11ed-a90a-762e3f62171e [38;5;243m12/30/22 03:39:55.988[0m [1mSTEP:[0m creating volume snapshot class with external rg azuredisk-csi-driver-test-a09e3f30-87f3-11ed-a90a-762e3f62171e [38;5;243m12/30/22 03:39:56.971[0m [1mSTEP:[0m setting up the VolumeSnapshotClass [38;5;243m12/30/22 03:39:56.971[0m [1mSTEP:[0m creating a VolumeSnapshotClass [38;5;243m12/30/22 03:39:56.971[0m ... skipping 10 lines ... [1mSTEP:[0m creating a StorageClass [38;5;243m12/30/22 03:40:14.327[0m [1mSTEP:[0m setting up the PVC and PV [38;5;243m12/30/22 03:40:14.384[0m [1mSTEP:[0m creating a PVC [38;5;243m12/30/22 03:40:14.384[0m [1mSTEP:[0m setting up the pod [38;5;243m12/30/22 03:40:14.453[0m [1mSTEP:[0m Set pod anti-affinity to make sure two pods are scheduled on different nodes [38;5;243m12/30/22 03:40:14.453[0m [1mSTEP:[0m deploying a pod with a volume restored from the snapshot [38;5;243m12/30/22 03:40:14.453[0m [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m12/30/22 03:40:14.512[0m Dec 30 03:40:14.512: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-hfgmq" in namespace "azuredisk-7726" to be "Succeeded or Failed" Dec 30 03:40:14.569: INFO: Pod "azuredisk-volume-tester-hfgmq": Phase="Pending", Reason="", readiness=false. Elapsed: 57.146472ms Dec 30 03:40:16.627: INFO: Pod "azuredisk-volume-tester-hfgmq": Phase="Pending", Reason="", readiness=false. Elapsed: 2.115231906s Dec 30 03:40:18.629: INFO: Pod "azuredisk-volume-tester-hfgmq": Phase="Pending", Reason="", readiness=false. Elapsed: 4.116535144s Dec 30 03:40:20.629: INFO: Pod "azuredisk-volume-tester-hfgmq": Phase="Pending", Reason="", readiness=false. Elapsed: 6.11651577s Dec 30 03:40:22.629: INFO: Pod "azuredisk-volume-tester-hfgmq": Phase="Pending", Reason="", readiness=false. Elapsed: 8.116893293s Dec 30 03:40:24.628: INFO: Pod "azuredisk-volume-tester-hfgmq": Phase="Pending", Reason="", readiness=false. Elapsed: 10.116391548s Dec 30 03:40:26.627: INFO: Pod "azuredisk-volume-tester-hfgmq": Phase="Pending", Reason="", readiness=false. Elapsed: 12.114957625s Dec 30 03:40:28.629: INFO: Pod "azuredisk-volume-tester-hfgmq": Phase="Pending", Reason="", readiness=false. Elapsed: 14.116825584s Dec 30 03:40:30.629: INFO: Pod "azuredisk-volume-tester-hfgmq": Phase="Pending", Reason="", readiness=false. Elapsed: 16.116734717s Dec 30 03:40:32.628: INFO: Pod "azuredisk-volume-tester-hfgmq": Phase="Pending", Reason="", readiness=false. Elapsed: 18.116430397s Dec 30 03:40:34.628: INFO: Pod "azuredisk-volume-tester-hfgmq": Phase="Pending", Reason="", readiness=false. Elapsed: 20.115527072s Dec 30 03:40:36.628: INFO: Pod "azuredisk-volume-tester-hfgmq": Phase="Pending", Reason="", readiness=false. Elapsed: 22.11584186s Dec 30 03:40:38.629: INFO: Pod "azuredisk-volume-tester-hfgmq": Phase="Failed", Reason="", readiness=false. Elapsed: 24.117040367s Dec 30 03:40:38.629: INFO: Unexpected error: <*fmt.wrapError | 0xc000ba2280>: { msg: "error while waiting for pod azuredisk-7726/azuredisk-volume-tester-hfgmq to be Succeeded or Failed: pod \"azuredisk-volume-tester-hfgmq\" failed with status: {Phase:Failed Conditions:[{Type:Initialized Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2022-12-30 03:40:17 +0000 UTC Reason: Message:} {Type:Ready Status:False LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2022-12-30 03:40:17 +0000 UTC Reason:ContainersNotReady Message:containers with unready status: [volume-tester]} {Type:ContainersReady Status:False LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2022-12-30 03:40:17 +0000 UTC Reason:ContainersNotReady Message:containers with unready status: [volume-tester]} {Type:PodScheduled Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2022-12-30 03:40:17 +0000 UTC Reason: Message:}] Message: Reason: NominatedNodeName: HostIP:10.248.0.32 PodIP:10.248.0.49 PodIPs:[{IP:10.248.0.49}] StartTime:2022-12-30 03:40:17 +0000 UTC InitContainerStatuses:[] ContainerStatuses:[{Name:volume-tester State:{Waiting:nil Running:nil Terminated:&ContainerStateTerminated{ExitCode:2,Signal:0,Reason:Error,Message:,StartedAt:2022-12-30 03:40:36 +0000 UTC,FinishedAt:2022-12-30 03:40:36 +0000 UTC,ContainerID:containerd://9e3ab7be3d104781dcbfa70bbf9f37893bb5194f9db50a8432ab3c3cfb1d933d,}} LastTerminationState:{Waiting:nil Running:nil Terminated:nil} Ready:false RestartCount:0 Image:registry.k8s.io/e2e-test-images/busybox:1.29-4 ImageID:registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 ContainerID:containerd://9e3ab7be3d104781dcbfa70bbf9f37893bb5194f9db50a8432ab3c3cfb1d933d Started:0xc000ae4a10}] QOSClass:BestEffort EphemeralContainerStatuses:[]}", err: <*errors.errorString | 0xc000212830>{ s: "pod \"azuredisk-volume-tester-hfgmq\" failed with status: {Phase:Failed Conditions:[{Type:Initialized Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2022-12-30 03:40:17 +0000 UTC Reason: Message:} {Type:Ready Status:False LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2022-12-30 03:40:17 +0000 UTC Reason:ContainersNotReady Message:containers with unready status: [volume-tester]} {Type:ContainersReady Status:False LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2022-12-30 03:40:17 +0000 UTC Reason:ContainersNotReady Message:containers with unready status: [volume-tester]} {Type:PodScheduled Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2022-12-30 03:40:17 +0000 UTC Reason: Message:}] Message: Reason: NominatedNodeName: HostIP:10.248.0.32 PodIP:10.248.0.49 PodIPs:[{IP:10.248.0.49}] StartTime:2022-12-30 03:40:17 +0000 UTC InitContainerStatuses:[] ContainerStatuses:[{Name:volume-tester State:{Waiting:nil Running:nil Terminated:&ContainerStateTerminated{ExitCode:2,Signal:0,Reason:Error,Message:,StartedAt:2022-12-30 03:40:36 +0000 UTC,FinishedAt:2022-12-30 03:40:36 +0000 UTC,ContainerID:containerd://9e3ab7be3d104781dcbfa70bbf9f37893bb5194f9db50a8432ab3c3cfb1d933d,}} LastTerminationState:{Waiting:nil Running:nil Terminated:nil} Ready:false RestartCount:0 Image:registry.k8s.io/e2e-test-images/busybox:1.29-4 ImageID:registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 ContainerID:containerd://9e3ab7be3d104781dcbfa70bbf9f37893bb5194f9db50a8432ab3c3cfb1d933d Started:0xc000ae4a10}] QOSClass:BestEffort EphemeralContainerStatuses:[]}", }, } Dec 30 03:40:38.629: FAIL: error while waiting for pod azuredisk-7726/azuredisk-volume-tester-hfgmq to be Succeeded or Failed: pod "azuredisk-volume-tester-hfgmq" failed with status: {Phase:Failed Conditions:[{Type:Initialized Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2022-12-30 03:40:17 +0000 UTC Reason: Message:} {Type:Ready Status:False LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2022-12-30 03:40:17 +0000 UTC Reason:ContainersNotReady Message:containers with unready status: [volume-tester]} {Type:ContainersReady Status:False LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2022-12-30 03:40:17 +0000 UTC Reason:ContainersNotReady Message:containers with unready status: [volume-tester]} {Type:PodScheduled Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2022-12-30 03:40:17 +0000 UTC Reason: Message:}] Message: Reason: NominatedNodeName: HostIP:10.248.0.32 PodIP:10.248.0.49 PodIPs:[{IP:10.248.0.49}] StartTime:2022-12-30 03:40:17 +0000 UTC InitContainerStatuses:[] ContainerStatuses:[{Name:volume-tester State:{Waiting:nil Running:nil Terminated:&ContainerStateTerminated{ExitCode:2,Signal:0,Reason:Error,Message:,StartedAt:2022-12-30 03:40:36 +0000 UTC,FinishedAt:2022-12-30 03:40:36 +0000 UTC,ContainerID:containerd://9e3ab7be3d104781dcbfa70bbf9f37893bb5194f9db50a8432ab3c3cfb1d933d,}} LastTerminationState:{Waiting:nil Running:nil Terminated:nil} Ready:false RestartCount:0 Image:registry.k8s.io/e2e-test-images/busybox:1.29-4 ImageID:registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 ContainerID:containerd://9e3ab7be3d104781dcbfa70bbf9f37893bb5194f9db50a8432ab3c3cfb1d933d Started:0xc000ae4a10}] QOSClass:BestEffort EphemeralContainerStatuses:[]} Full Stack Trace sigs.k8s.io/azuredisk-csi-driver/test/e2e/testsuites.(*TestPod).WaitForSuccess(0x2250430?) /home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/testsuites/testsuites.go:823 +0x5d sigs.k8s.io/azuredisk-csi-driver/test/e2e/testsuites.(*DynamicallyProvisionedVolumeSnapshotTest).Run(0xc000c51d78, {0x270a540, 0xc000d9d040}, {0x26f5740, 0xc0001b3360}, 0xc00078f340?) /home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/testsuites/dynamically_provisioned_volume_snapshot_tester.go:142 +0x1358 ... skipping 43 lines ... Dec 30 03:42:46.797: INFO: Claim "azuredisk-7726" in namespace "pvc-d2cnz" doesn't exist in the system Dec 30 03:42:46.797: INFO: deleting StorageClass azuredisk-7726-disk.csi.azure.com-dynamic-sc-9zvvn [1mSTEP:[0m dump namespace information after failure [38;5;243m12/30/22 03:42:46.858[0m [1mSTEP:[0m Destroying namespace "azuredisk-7726" for this suite. [38;5;243m12/30/22 03:42:46.858[0m [38;5;243m<< End Captured GinkgoWriter Output[0m [38;5;9mDec 30 03:40:38.629: error while waiting for pod azuredisk-7726/azuredisk-volume-tester-hfgmq to be Succeeded or Failed: pod "azuredisk-volume-tester-hfgmq" failed with status: {Phase:Failed Conditions:[{Type:Initialized Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2022-12-30 03:40:17 +0000 UTC Reason: Message:} {Type:Ready Status:False LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2022-12-30 03:40:17 +0000 UTC Reason:ContainersNotReady Message:containers with unready status: [volume-tester]} {Type:ContainersReady Status:False LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2022-12-30 03:40:17 +0000 UTC Reason:ContainersNotReady Message:containers with unready status: [volume-tester]} {Type:PodScheduled Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2022-12-30 03:40:17 +0000 UTC Reason: Message:}] Message: Reason: NominatedNodeName: HostIP:10.248.0.32 PodIP:10.248.0.49 PodIPs:[{IP:10.248.0.49}] StartTime:2022-12-30 03:40:17 +0000 UTC InitContainerStatuses:[] ContainerStatuses:[{Name:volume-tester State:{Waiting:nil Running:nil Terminated:&ContainerStateTerminated{ExitCode:2,Signal:0,Reason:Error,Message:,StartedAt:2022-12-30 03:40:36 +0000 UTC,FinishedAt:2022-12-30 03:40:36 +0000 UTC,ContainerID:containerd://9e3ab7be3d104781dcbfa70bbf9f37893bb5194f9db50a8432ab3c3cfb1d933d,}} LastTerminationState:{Waiting:nil Running:nil Terminated:nil} Ready:false RestartCount:0 Image:registry.k8s.io/e2e-test-images/busybox:1.29-4 ImageID:registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 ContainerID:containerd://9e3ab7be3d104781dcbfa70bbf9f37893bb5194f9db50a8432ab3c3cfb1d933d Started:0xc000ae4a10}] QOSClass:BestEffort EphemeralContainerStatuses:[]}[0m [38;5;9mIn [1m[It][0m[38;5;9m at: [1m/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/testsuites/testsuites.go:823[0m [1mThere were additional failures detected after the initial failure:[0m [38;5;13m[PANICKED][0m [38;5;13mTest Panicked[0m [38;5;13mIn [1m[DeferCleanup (Each)][0m[38;5;13m at: [1m/usr/local/go/src/runtime/panic.go:260[0m [38;5;13mruntime error: invalid memory address or nil pointer dereference[0m [38;5;13mFull Stack Trace[0m k8s.io/kubernetes/test/e2e/framework.(*Framework).dumpNamespaceInfo.func1() /home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:274 +0x5c k8s.io/kubernetes/test/e2e/framework.(*Framework).dumpNamespaceInfo(0xc00012c3c0) /home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:271 +0x179 ... skipping 25 lines ... [1mSTEP:[0m creating a PVC [38;5;243m12/30/22 03:42:48.003[0m [1mSTEP:[0m setting up the StorageClass [38;5;243m12/30/22 03:42:48.066[0m [1mSTEP:[0m creating a StorageClass [38;5;243m12/30/22 03:42:48.066[0m [1mSTEP:[0m setting up the PVC and PV [38;5;243m12/30/22 03:42:48.123[0m [1mSTEP:[0m creating a PVC [38;5;243m12/30/22 03:42:48.123[0m [1mSTEP:[0m deploying the pod [38;5;243m12/30/22 03:42:48.181[0m [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m12/30/22 03:42:48.24[0m Dec 30 03:42:48.240: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-k2cs8" in namespace "azuredisk-3086" to be "Succeeded or Failed" Dec 30 03:42:48.298: INFO: Pod "azuredisk-volume-tester-k2cs8": Phase="Pending", Reason="", readiness=false. Elapsed: 57.288038ms Dec 30 03:42:50.357: INFO: Pod "azuredisk-volume-tester-k2cs8": Phase="Pending", Reason="", readiness=false. Elapsed: 2.116871862s Dec 30 03:42:52.356: INFO: Pod "azuredisk-volume-tester-k2cs8": Phase="Pending", Reason="", readiness=false. Elapsed: 4.115794093s Dec 30 03:42:54.359: INFO: Pod "azuredisk-volume-tester-k2cs8": Phase="Pending", Reason="", readiness=false. Elapsed: 6.118523447s Dec 30 03:42:56.356: INFO: Pod "azuredisk-volume-tester-k2cs8": Phase="Pending", Reason="", readiness=false. Elapsed: 8.115586779s Dec 30 03:42:58.356: INFO: Pod "azuredisk-volume-tester-k2cs8": Phase="Pending", Reason="", readiness=false. Elapsed: 10.115886856s ... skipping 10 lines ... Dec 30 03:43:20.357: INFO: Pod "azuredisk-volume-tester-k2cs8": Phase="Pending", Reason="", readiness=false. Elapsed: 32.116796348s Dec 30 03:43:22.357: INFO: Pod "azuredisk-volume-tester-k2cs8": Phase="Pending", Reason="", readiness=false. Elapsed: 34.116722719s Dec 30 03:43:24.356: INFO: Pod "azuredisk-volume-tester-k2cs8": Phase="Pending", Reason="", readiness=false. Elapsed: 36.115848577s Dec 30 03:43:26.356: INFO: Pod "azuredisk-volume-tester-k2cs8": Phase="Pending", Reason="", readiness=false. Elapsed: 38.115933745s Dec 30 03:43:28.357: INFO: Pod "azuredisk-volume-tester-k2cs8": Phase="Succeeded", Reason="", readiness=false. Elapsed: 40.116286929s [1mSTEP:[0m Saw pod success [38;5;243m12/30/22 03:43:28.357[0m Dec 30 03:43:28.357: INFO: Pod "azuredisk-volume-tester-k2cs8" satisfied condition "Succeeded or Failed" Dec 30 03:43:28.357: INFO: deleting Pod "azuredisk-3086"/"azuredisk-volume-tester-k2cs8" Dec 30 03:43:28.417: INFO: Pod azuredisk-volume-tester-k2cs8 has the following logs: hello world [1mSTEP:[0m Deleting pod azuredisk-volume-tester-k2cs8 in namespace azuredisk-3086 [38;5;243m12/30/22 03:43:28.417[0m [1mSTEP:[0m validating provisioned PV [38;5;243m12/30/22 03:43:28.539[0m [1mSTEP:[0m checking the PV [38;5;243m12/30/22 03:43:28.596[0m ... skipping 65 lines ... [1mSTEP:[0m creating a PVC [38;5;243m12/30/22 03:42:48.003[0m [1mSTEP:[0m setting up the StorageClass [38;5;243m12/30/22 03:42:48.066[0m [1mSTEP:[0m creating a StorageClass [38;5;243m12/30/22 03:42:48.066[0m [1mSTEP:[0m setting up the PVC and PV [38;5;243m12/30/22 03:42:48.123[0m [1mSTEP:[0m creating a PVC [38;5;243m12/30/22 03:42:48.123[0m [1mSTEP:[0m deploying the pod [38;5;243m12/30/22 03:42:48.181[0m [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m12/30/22 03:42:48.24[0m Dec 30 03:42:48.240: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-k2cs8" in namespace "azuredisk-3086" to be "Succeeded or Failed" Dec 30 03:42:48.298: INFO: Pod "azuredisk-volume-tester-k2cs8": Phase="Pending", Reason="", readiness=false. Elapsed: 57.288038ms Dec 30 03:42:50.357: INFO: Pod "azuredisk-volume-tester-k2cs8": Phase="Pending", Reason="", readiness=false. Elapsed: 2.116871862s Dec 30 03:42:52.356: INFO: Pod "azuredisk-volume-tester-k2cs8": Phase="Pending", Reason="", readiness=false. Elapsed: 4.115794093s Dec 30 03:42:54.359: INFO: Pod "azuredisk-volume-tester-k2cs8": Phase="Pending", Reason="", readiness=false. Elapsed: 6.118523447s Dec 30 03:42:56.356: INFO: Pod "azuredisk-volume-tester-k2cs8": Phase="Pending", Reason="", readiness=false. Elapsed: 8.115586779s Dec 30 03:42:58.356: INFO: Pod "azuredisk-volume-tester-k2cs8": Phase="Pending", Reason="", readiness=false. Elapsed: 10.115886856s ... skipping 10 lines ... Dec 30 03:43:20.357: INFO: Pod "azuredisk-volume-tester-k2cs8": Phase="Pending", Reason="", readiness=false. Elapsed: 32.116796348s Dec 30 03:43:22.357: INFO: Pod "azuredisk-volume-tester-k2cs8": Phase="Pending", Reason="", readiness=false. Elapsed: 34.116722719s Dec 30 03:43:24.356: INFO: Pod "azuredisk-volume-tester-k2cs8": Phase="Pending", Reason="", readiness=false. Elapsed: 36.115848577s Dec 30 03:43:26.356: INFO: Pod "azuredisk-volume-tester-k2cs8": Phase="Pending", Reason="", readiness=false. Elapsed: 38.115933745s Dec 30 03:43:28.357: INFO: Pod "azuredisk-volume-tester-k2cs8": Phase="Succeeded", Reason="", readiness=false. Elapsed: 40.116286929s [1mSTEP:[0m Saw pod success [38;5;243m12/30/22 03:43:28.357[0m Dec 30 03:43:28.357: INFO: Pod "azuredisk-volume-tester-k2cs8" satisfied condition "Succeeded or Failed" Dec 30 03:43:28.357: INFO: deleting Pod "azuredisk-3086"/"azuredisk-volume-tester-k2cs8" Dec 30 03:43:28.417: INFO: Pod azuredisk-volume-tester-k2cs8 has the following logs: hello world [1mSTEP:[0m Deleting pod azuredisk-volume-tester-k2cs8 in namespace azuredisk-3086 [38;5;243m12/30/22 03:43:28.417[0m [1mSTEP:[0m validating provisioned PV [38;5;243m12/30/22 03:43:28.539[0m [1mSTEP:[0m checking the PV [38;5;243m12/30/22 03:43:28.596[0m ... skipping 1091 lines ... [1mSTEP:[0m setting up the StorageClass [38;5;243m12/30/22 03:59:08.641[0m [1mSTEP:[0m creating a StorageClass [38;5;243m12/30/22 03:59:08.641[0m [1mSTEP:[0m setting up the PVC and PV [38;5;243m12/30/22 03:59:08.7[0m [1mSTEP:[0m creating a PVC [38;5;243m12/30/22 03:59:08.7[0m [1mSTEP:[0m setting up the pod [38;5;243m12/30/22 03:59:08.761[0m [1mSTEP:[0m deploying the pod [38;5;243m12/30/22 03:59:08.762[0m [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m12/30/22 03:59:08.824[0m Dec 30 03:59:08.824: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-bfpmx" in namespace "azuredisk-1092" to be "Succeeded or Failed" Dec 30 03:59:08.882: INFO: Pod "azuredisk-volume-tester-bfpmx": Phase="Pending", Reason="", readiness=false. Elapsed: 57.543083ms Dec 30 03:59:10.942: INFO: Pod "azuredisk-volume-tester-bfpmx": Phase="Pending", Reason="", readiness=false. Elapsed: 2.1178207s Dec 30 03:59:12.945: INFO: Pod "azuredisk-volume-tester-bfpmx": Phase="Pending", Reason="", readiness=false. Elapsed: 4.12112212s Dec 30 03:59:14.941: INFO: Pod "azuredisk-volume-tester-bfpmx": Phase="Pending", Reason="", readiness=false. Elapsed: 6.116521516s Dec 30 03:59:16.940: INFO: Pod "azuredisk-volume-tester-bfpmx": Phase="Pending", Reason="", readiness=false. Elapsed: 8.115791812s Dec 30 03:59:18.947: INFO: Pod "azuredisk-volume-tester-bfpmx": Phase="Pending", Reason="", readiness=false. Elapsed: 10.122724316s ... skipping 2 lines ... Dec 30 03:59:24.940: INFO: Pod "azuredisk-volume-tester-bfpmx": Phase="Pending", Reason="", readiness=false. Elapsed: 16.11571995s Dec 30 03:59:26.941: INFO: Pod "azuredisk-volume-tester-bfpmx": Phase="Pending", Reason="", readiness=false. Elapsed: 18.117117747s Dec 30 03:59:28.940: INFO: Pod "azuredisk-volume-tester-bfpmx": Phase="Pending", Reason="", readiness=false. Elapsed: 20.116151061s Dec 30 03:59:30.940: INFO: Pod "azuredisk-volume-tester-bfpmx": Phase="Pending", Reason="", readiness=false. Elapsed: 22.116245066s Dec 30 03:59:32.942: INFO: Pod "azuredisk-volume-tester-bfpmx": Phase="Succeeded", Reason="", readiness=false. Elapsed: 24.11836083s [1mSTEP:[0m Saw pod success [38;5;243m12/30/22 03:59:32.942[0m Dec 30 03:59:32.943: INFO: Pod "azuredisk-volume-tester-bfpmx" satisfied condition "Succeeded or Failed" Dec 30 03:59:32.943: INFO: deleting Pod "azuredisk-1092"/"azuredisk-volume-tester-bfpmx" Dec 30 03:59:33.038: INFO: Pod azuredisk-volume-tester-bfpmx has the following logs: hello world [1mSTEP:[0m Deleting pod azuredisk-volume-tester-bfpmx in namespace azuredisk-1092 [38;5;243m12/30/22 03:59:33.039[0m [1mSTEP:[0m validating provisioned PV [38;5;243m12/30/22 03:59:33.164[0m [1mSTEP:[0m checking the PV [38;5;243m12/30/22 03:59:33.221[0m ... skipping 33 lines ... [1mSTEP:[0m setting up the StorageClass [38;5;243m12/30/22 03:59:08.641[0m [1mSTEP:[0m creating a StorageClass [38;5;243m12/30/22 03:59:08.641[0m [1mSTEP:[0m setting up the PVC and PV [38;5;243m12/30/22 03:59:08.7[0m [1mSTEP:[0m creating a PVC [38;5;243m12/30/22 03:59:08.7[0m [1mSTEP:[0m setting up the pod [38;5;243m12/30/22 03:59:08.761[0m [1mSTEP:[0m deploying the pod [38;5;243m12/30/22 03:59:08.762[0m [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m12/30/22 03:59:08.824[0m Dec 30 03:59:08.824: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-bfpmx" in namespace "azuredisk-1092" to be "Succeeded or Failed" Dec 30 03:59:08.882: INFO: Pod "azuredisk-volume-tester-bfpmx": Phase="Pending", Reason="", readiness=false. Elapsed: 57.543083ms Dec 30 03:59:10.942: INFO: Pod "azuredisk-volume-tester-bfpmx": Phase="Pending", Reason="", readiness=false. Elapsed: 2.1178207s Dec 30 03:59:12.945: INFO: Pod "azuredisk-volume-tester-bfpmx": Phase="Pending", Reason="", readiness=false. Elapsed: 4.12112212s Dec 30 03:59:14.941: INFO: Pod "azuredisk-volume-tester-bfpmx": Phase="Pending", Reason="", readiness=false. Elapsed: 6.116521516s Dec 30 03:59:16.940: INFO: Pod "azuredisk-volume-tester-bfpmx": Phase="Pending", Reason="", readiness=false. Elapsed: 8.115791812s Dec 30 03:59:18.947: INFO: Pod "azuredisk-volume-tester-bfpmx": Phase="Pending", Reason="", readiness=false. Elapsed: 10.122724316s ... skipping 2 lines ... Dec 30 03:59:24.940: INFO: Pod "azuredisk-volume-tester-bfpmx": Phase="Pending", Reason="", readiness=false. Elapsed: 16.11571995s Dec 30 03:59:26.941: INFO: Pod "azuredisk-volume-tester-bfpmx": Phase="Pending", Reason="", readiness=false. Elapsed: 18.117117747s Dec 30 03:59:28.940: INFO: Pod "azuredisk-volume-tester-bfpmx": Phase="Pending", Reason="", readiness=false. Elapsed: 20.116151061s Dec 30 03:59:30.940: INFO: Pod "azuredisk-volume-tester-bfpmx": Phase="Pending", Reason="", readiness=false. Elapsed: 22.116245066s Dec 30 03:59:32.942: INFO: Pod "azuredisk-volume-tester-bfpmx": Phase="Succeeded", Reason="", readiness=false. Elapsed: 24.11836083s [1mSTEP:[0m Saw pod success [38;5;243m12/30/22 03:59:32.942[0m Dec 30 03:59:32.943: INFO: Pod "azuredisk-volume-tester-bfpmx" satisfied condition "Succeeded or Failed" Dec 30 03:59:32.943: INFO: deleting Pod "azuredisk-1092"/"azuredisk-volume-tester-bfpmx" Dec 30 03:59:33.038: INFO: Pod azuredisk-volume-tester-bfpmx has the following logs: hello world [1mSTEP:[0m Deleting pod azuredisk-volume-tester-bfpmx in namespace azuredisk-1092 [38;5;243m12/30/22 03:59:33.039[0m [1mSTEP:[0m validating provisioned PV [38;5;243m12/30/22 03:59:33.164[0m [1mSTEP:[0m checking the PV [38;5;243m12/30/22 03:59:33.221[0m ... skipping 95 lines ... Platform: linux/amd64 Topology Key: topology.disk.csi.azure.com/zone Streaming logs below: I1230 03:08:40.364612 1 azuredisk.go:175] driver userAgent: disk.csi.azure.com/v1.26.0-4cfb4ee1bf05ca9ba058389347ac5904b562074b e2e-test I1230 03:08:40.366520 1 azure_disk_utils.go:162] reading cloud config from secret kube-system/azure-cloud-provider I1230 03:08:40.393979 1 azure_disk_utils.go:169] InitializeCloudFromSecret: failed to get cloud config from secret kube-system/azure-cloud-provider: failed to get secret kube-system/azure-cloud-provider: secrets "azure-cloud-provider" not found I1230 03:08:40.394004 1 azure_disk_utils.go:174] could not read cloud config from secret kube-system/azure-cloud-provider I1230 03:08:40.394014 1 azure_disk_utils.go:184] use default AZURE_CREDENTIAL_FILE env var: /etc/kubernetes/azure.json I1230 03:08:40.394046 1 azure_disk_utils.go:192] read cloud config from file: /etc/kubernetes/azure.json successfully I1230 03:08:40.394776 1 azure_auth.go:245] Using AzurePublicCloud environment I1230 03:08:40.394826 1 azure_auth.go:130] azure: using client_id+client_secret to retrieve access token I1230 03:08:40.394866 1 azure.go:775] Azure cloudprovider using try backoff: retries=6, exponent=1.500000, duration=5, jitter=1.000000 ... skipping 25 lines ... I1230 03:08:40.395331 1 azure_blobclient.go:67] Azure BlobClient using API version: 2021-09-01 I1230 03:08:40.395356 1 azure_vmasclient.go:70] Azure AvailabilitySetsClient (read ops) using rate limit config: QPS=6, bucket=20 I1230 03:08:40.395373 1 azure_vmasclient.go:73] Azure AvailabilitySetsClient (write ops) using rate limit config: QPS=100, bucket=1000 I1230 03:08:40.395449 1 azure.go:1006] attach/detach disk operation rate limit QPS: 6.000000, Bucket: 10 I1230 03:08:40.395476 1 azuredisk.go:192] disable UseInstanceMetadata for controller I1230 03:08:40.395485 1 azuredisk.go:204] cloud: AzurePublicCloud, location: westus2, rg: kubetest-du8j4fcj, VMType: vmss, PrimaryScaleSetName: k8s-agentpool-17591246-vmss, PrimaryAvailabilitySetName: , DisableAvailabilitySetNodes: false I1230 03:08:40.398808 1 mount_linux.go:287] 'umount /tmp/kubelet-detect-safe-umount4249916794' failed with: exit status 32, output: umount: /tmp/kubelet-detect-safe-umount4249916794: must be superuser to unmount. I1230 03:08:40.398835 1 mount_linux.go:289] Detected umount with unsafe 'not mounted' behavior I1230 03:08:40.398933 1 driver.go:81] Enabling controller service capability: CREATE_DELETE_VOLUME I1230 03:08:40.398944 1 driver.go:81] Enabling controller service capability: PUBLISH_UNPUBLISH_VOLUME I1230 03:08:40.398951 1 driver.go:81] Enabling controller service capability: CREATE_DELETE_SNAPSHOT I1230 03:08:40.398958 1 driver.go:81] Enabling controller service capability: CLONE_VOLUME I1230 03:08:40.398964 1 driver.go:81] Enabling controller service capability: EXPAND_VOLUME ... skipping 57 lines ... I1230 03:08:49.226546 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 24988 I1230 03:08:49.310678 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 32358 I1230 03:08:49.314324 1 controllerserver.go:383] GetDiskLun returned: cannot find Lun for disk pvc-4be03758-bf6c-473a-8086-fad258f56014. Initiating attaching volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-4be03758-bf6c-473a-8086-fad258f56014 to node k8s-agentpool-17591246-vmss000000 (vmState Succeeded). I1230 03:08:49.314361 1 controllerserver.go:408] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-4be03758-bf6c-473a-8086-fad258f56014 to node k8s-agentpool-17591246-vmss000000 I1230 03:08:49.314435 1 azure_controller_common.go:255] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-4be03758-bf6c-473a-8086-fad258f56014 lun 0 to node k8s-agentpool-17591246-vmss000000, diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-du8j4fcj/providers/microsoft.compute/disks/pvc-4be03758-bf6c-473a-8086-fad258f56014:%!s(*provider.AttachDiskOptions=&{None pvc-4be03758-bf6c-473a-8086-fad258f56014 false 0})] I1230 03:08:49.314479 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-du8j4fcj): vm(k8s-agentpool-17591246-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-du8j4fcj/providers/microsoft.compute/disks/pvc-4be03758-bf6c-473a-8086-fad258f56014:%!s(*provider.AttachDiskOptions=&{None pvc-4be03758-bf6c-473a-8086-fad258f56014 false 0})]) I1230 03:08:50.350949 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-du8j4fcj): vm(k8s-agentpool-17591246-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-du8j4fcj/providers/microsoft.compute/disks/pvc-4be03758-bf6c-473a-8086-fad258f56014:%!s(*provider.AttachDiskOptions=&{None pvc-4be03758-bf6c-473a-8086-fad258f56014 false 0})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I1230 03:09:00.490722 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-du8j4fcj, k8s-agentpool-17591246-vmss, k8s-agentpool-17591246-vmss000000) successfully I1230 03:09:00.490765 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool-17591246-vmss, kubetest-du8j4fcj, k8s-agentpool-17591246-vmss000000) for cacheKey(kubetest-du8j4fcj/k8s-agentpool-17591246-vmss) updated successfully I1230 03:09:00.490783 1 controllerserver.go:413] Attach operation successful: volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-4be03758-bf6c-473a-8086-fad258f56014 attached to node k8s-agentpool-17591246-vmss000000. I1230 03:09:00.490793 1 controllerserver.go:433] attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-4be03758-bf6c-473a-8086-fad258f56014 to node k8s-agentpool-17591246-vmss000000 successfully I1230 03:09:00.490851 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=11.357159229 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-du8j4fcj" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-4be03758-bf6c-473a-8086-fad258f56014" node="k8s-agentpool-17591246-vmss000000" result_code="succeeded" I1230 03:09:00.490866 1 utils.go:84] GRPC response: {"publish_context":{"LUN":"0"}} ... skipping 22 lines ... I1230 03:10:04.504186 1 utils.go:78] GRPC request: {"node_id":"k8s-agentpool-17591246-vmss000001","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-41d31c68-d589-47ea-8d9e-2a7e36969f37","csi.storage.k8s.io/pvc/name":"pvc-txd99","csi.storage.k8s.io/pvc/namespace":"azuredisk-2540","enableAsyncAttach":"false","enableasyncattach":"false","networkAccessPolicy":"DenyAll","requestedsizegib":"10","skuName":"Standard_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1672369721109-8081-disk.csi.azure.com","userAgent":"azuredisk-e2e-test"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-41d31c68-d589-47ea-8d9e-2a7e36969f37"} I1230 03:10:04.527043 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1218 I1230 03:10:04.527424 1 controllerserver.go:383] GetDiskLun returned: cannot find Lun for disk pvc-41d31c68-d589-47ea-8d9e-2a7e36969f37. Initiating attaching volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-41d31c68-d589-47ea-8d9e-2a7e36969f37 to node k8s-agentpool-17591246-vmss000001 (vmState Succeeded). I1230 03:10:04.527455 1 controllerserver.go:408] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-41d31c68-d589-47ea-8d9e-2a7e36969f37 to node k8s-agentpool-17591246-vmss000001 I1230 03:10:04.527493 1 azure_controller_common.go:255] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-41d31c68-d589-47ea-8d9e-2a7e36969f37 lun 0 to node k8s-agentpool-17591246-vmss000001, diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-du8j4fcj/providers/microsoft.compute/disks/pvc-41d31c68-d589-47ea-8d9e-2a7e36969f37:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-41d31c68-d589-47ea-8d9e-2a7e36969f37 false 0})] I1230 03:10:04.527534 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-du8j4fcj): vm(k8s-agentpool-17591246-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-du8j4fcj/providers/microsoft.compute/disks/pvc-41d31c68-d589-47ea-8d9e-2a7e36969f37:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-41d31c68-d589-47ea-8d9e-2a7e36969f37 false 0})]) I1230 03:10:04.720453 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-du8j4fcj): vm(k8s-agentpool-17591246-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-du8j4fcj/providers/microsoft.compute/disks/pvc-41d31c68-d589-47ea-8d9e-2a7e36969f37:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-41d31c68-d589-47ea-8d9e-2a7e36969f37 false 0})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I1230 03:10:19.925532 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-du8j4fcj, k8s-agentpool-17591246-vmss, k8s-agentpool-17591246-vmss000001) successfully I1230 03:10:19.925615 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool-17591246-vmss, kubetest-du8j4fcj, k8s-agentpool-17591246-vmss000001) for cacheKey(kubetest-du8j4fcj/k8s-agentpool-17591246-vmss) updated successfully I1230 03:10:19.925637 1 controllerserver.go:413] Attach operation successful: volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-41d31c68-d589-47ea-8d9e-2a7e36969f37 attached to node k8s-agentpool-17591246-vmss000001. I1230 03:10:19.925654 1 controllerserver.go:433] attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-41d31c68-d589-47ea-8d9e-2a7e36969f37 to node k8s-agentpool-17591246-vmss000001 successfully I1230 03:10:19.925699 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=15.398274915 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-du8j4fcj" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-41d31c68-d589-47ea-8d9e-2a7e36969f37" node="k8s-agentpool-17591246-vmss000001" result_code="succeeded" I1230 03:10:19.925714 1 utils.go:84] GRPC response: {"publish_context":{"LUN":"0"}} ... skipping 27 lines ... I1230 03:11:11.897449 1 utils.go:78] GRPC request: {"node_id":"k8s-agentpool-17591246-vmss000000","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-ef878302-a013-4f6f-a1e8-297da8b3ad19","csi.storage.k8s.io/pvc/name":"pvc-qbmx9","csi.storage.k8s.io/pvc/namespace":"azuredisk-4728","enableAsyncAttach":"false","enableBursting":"true","enableasyncattach":"false","perfProfile":"Basic","requestedsizegib":"1024","skuName":"Premium_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1672369721109-8081-disk.csi.azure.com","userAgent":"azuredisk-e2e-test"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-ef878302-a013-4f6f-a1e8-297da8b3ad19"} I1230 03:11:11.925471 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1338 I1230 03:11:11.925859 1 controllerserver.go:383] GetDiskLun returned: cannot find Lun for disk pvc-ef878302-a013-4f6f-a1e8-297da8b3ad19. Initiating attaching volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-ef878302-a013-4f6f-a1e8-297da8b3ad19 to node k8s-agentpool-17591246-vmss000000 (vmState Succeeded). I1230 03:11:11.925896 1 controllerserver.go:408] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-ef878302-a013-4f6f-a1e8-297da8b3ad19 to node k8s-agentpool-17591246-vmss000000 I1230 03:11:11.925934 1 azure_controller_common.go:255] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-ef878302-a013-4f6f-a1e8-297da8b3ad19 lun 0 to node k8s-agentpool-17591246-vmss000000, diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-du8j4fcj/providers/microsoft.compute/disks/pvc-ef878302-a013-4f6f-a1e8-297da8b3ad19:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-ef878302-a013-4f6f-a1e8-297da8b3ad19 false 0})] I1230 03:11:11.925983 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-du8j4fcj): vm(k8s-agentpool-17591246-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-du8j4fcj/providers/microsoft.compute/disks/pvc-ef878302-a013-4f6f-a1e8-297da8b3ad19:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-ef878302-a013-4f6f-a1e8-297da8b3ad19 false 0})]) I1230 03:11:12.105548 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-du8j4fcj): vm(k8s-agentpool-17591246-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-du8j4fcj/providers/microsoft.compute/disks/pvc-ef878302-a013-4f6f-a1e8-297da8b3ad19:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-ef878302-a013-4f6f-a1e8-297da8b3ad19 false 0})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I1230 03:11:22.252910 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-du8j4fcj, k8s-agentpool-17591246-vmss, k8s-agentpool-17591246-vmss000000) successfully I1230 03:11:22.252950 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool-17591246-vmss, kubetest-du8j4fcj, k8s-agentpool-17591246-vmss000000) for cacheKey(kubetest-du8j4fcj/k8s-agentpool-17591246-vmss) updated successfully I1230 03:11:22.252972 1 controllerserver.go:413] Attach operation successful: volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-ef878302-a013-4f6f-a1e8-297da8b3ad19 attached to node k8s-agentpool-17591246-vmss000000. I1230 03:11:22.252987 1 controllerserver.go:433] attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-ef878302-a013-4f6f-a1e8-297da8b3ad19 to node k8s-agentpool-17591246-vmss000000 successfully I1230 03:11:22.253034 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=10.327174327 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-du8j4fcj" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-ef878302-a013-4f6f-a1e8-297da8b3ad19" node="k8s-agentpool-17591246-vmss000000" result_code="succeeded" I1230 03:11:22.253056 1 utils.go:84] GRPC response: {"publish_context":{"LUN":"0"}} ... skipping 14 lines ... I1230 03:12:38.344373 1 utils.go:78] GRPC request: {"node_id":"k8s-agentpool-17591246-vmss000000","volume_capability":{"AccessType":{"Mount":{"mount_flags":["invalid","mount","options"]}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-6eed3ff8-9ce1-43fb-a6c8-1a50bff430f5","csi.storage.k8s.io/pvc/name":"pvc-6862x","csi.storage.k8s.io/pvc/namespace":"azuredisk-5466","requestedsizegib":"10","skuName":"StandardSSD_ZRS","storage.kubernetes.io/csiProvisionerIdentity":"1672369721109-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-6eed3ff8-9ce1-43fb-a6c8-1a50bff430f5"} I1230 03:12:38.375693 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1192 I1230 03:12:38.376047 1 controllerserver.go:383] GetDiskLun returned: cannot find Lun for disk pvc-6eed3ff8-9ce1-43fb-a6c8-1a50bff430f5. Initiating attaching volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-6eed3ff8-9ce1-43fb-a6c8-1a50bff430f5 to node k8s-agentpool-17591246-vmss000000 (vmState Succeeded). I1230 03:12:38.376085 1 controllerserver.go:408] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-6eed3ff8-9ce1-43fb-a6c8-1a50bff430f5 to node k8s-agentpool-17591246-vmss000000 I1230 03:12:38.376125 1 azure_controller_common.go:255] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-6eed3ff8-9ce1-43fb-a6c8-1a50bff430f5 lun 0 to node k8s-agentpool-17591246-vmss000000, diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-du8j4fcj/providers/microsoft.compute/disks/pvc-6eed3ff8-9ce1-43fb-a6c8-1a50bff430f5:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-6eed3ff8-9ce1-43fb-a6c8-1a50bff430f5 false 0})] I1230 03:12:38.376169 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-du8j4fcj): vm(k8s-agentpool-17591246-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-du8j4fcj/providers/microsoft.compute/disks/pvc-6eed3ff8-9ce1-43fb-a6c8-1a50bff430f5:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-6eed3ff8-9ce1-43fb-a6c8-1a50bff430f5 false 0})]) I1230 03:12:38.510406 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-du8j4fcj): vm(k8s-agentpool-17591246-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-du8j4fcj/providers/microsoft.compute/disks/pvc-6eed3ff8-9ce1-43fb-a6c8-1a50bff430f5:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-6eed3ff8-9ce1-43fb-a6c8-1a50bff430f5 false 0})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I1230 03:12:48.643258 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-du8j4fcj, k8s-agentpool-17591246-vmss, k8s-agentpool-17591246-vmss000000) successfully I1230 03:12:48.643311 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool-17591246-vmss, kubetest-du8j4fcj, k8s-agentpool-17591246-vmss000000) for cacheKey(kubetest-du8j4fcj/k8s-agentpool-17591246-vmss) updated successfully I1230 03:12:48.643334 1 controllerserver.go:413] Attach operation successful: volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-6eed3ff8-9ce1-43fb-a6c8-1a50bff430f5 attached to node k8s-agentpool-17591246-vmss000000. I1230 03:12:48.643351 1 controllerserver.go:433] attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-6eed3ff8-9ce1-43fb-a6c8-1a50bff430f5 to node k8s-agentpool-17591246-vmss000000 successfully I1230 03:12:48.643396 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=10.267350822 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-du8j4fcj" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-6eed3ff8-9ce1-43fb-a6c8-1a50bff430f5" node="k8s-agentpool-17591246-vmss000000" result_code="succeeded" I1230 03:12:48.643422 1 utils.go:84] GRPC response: {"publish_context":{"LUN":"0"}} ... skipping 35 lines ... I1230 03:13:42.623291 1 utils.go:78] GRPC request: {"node_id":"k8s-agentpool-17591246-vmss000000","volume_capability":{"AccessType":{"Block":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-5e94aef0-afd5-46ef-b079-f4bad5b9b2e9","csi.storage.k8s.io/pvc/name":"pvc-czzmh","csi.storage.k8s.io/pvc/namespace":"azuredisk-2790","requestedsizegib":"10","skuName":"StandardSSD_ZRS","storage.kubernetes.io/csiProvisionerIdentity":"1672369721109-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-5e94aef0-afd5-46ef-b079-f4bad5b9b2e9"} I1230 03:13:42.653456 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1193 I1230 03:13:42.654007 1 controllerserver.go:383] GetDiskLun returned: cannot find Lun for disk pvc-5e94aef0-afd5-46ef-b079-f4bad5b9b2e9. Initiating attaching volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-5e94aef0-afd5-46ef-b079-f4bad5b9b2e9 to node k8s-agentpool-17591246-vmss000000 (vmState Succeeded). I1230 03:13:42.654048 1 controllerserver.go:408] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-5e94aef0-afd5-46ef-b079-f4bad5b9b2e9 to node k8s-agentpool-17591246-vmss000000 I1230 03:13:42.654197 1 azure_controller_common.go:255] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-5e94aef0-afd5-46ef-b079-f4bad5b9b2e9 lun 0 to node k8s-agentpool-17591246-vmss000000, diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-du8j4fcj/providers/microsoft.compute/disks/pvc-5e94aef0-afd5-46ef-b079-f4bad5b9b2e9:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-5e94aef0-afd5-46ef-b079-f4bad5b9b2e9 false 0})] I1230 03:13:42.654354 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-du8j4fcj): vm(k8s-agentpool-17591246-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-du8j4fcj/providers/microsoft.compute/disks/pvc-5e94aef0-afd5-46ef-b079-f4bad5b9b2e9:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-5e94aef0-afd5-46ef-b079-f4bad5b9b2e9 false 0})]) I1230 03:13:42.852792 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-du8j4fcj): vm(k8s-agentpool-17591246-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-du8j4fcj/providers/microsoft.compute/disks/pvc-5e94aef0-afd5-46ef-b079-f4bad5b9b2e9:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-5e94aef0-afd5-46ef-b079-f4bad5b9b2e9 false 0})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I1230 03:13:53.028561 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-du8j4fcj, k8s-agentpool-17591246-vmss, k8s-agentpool-17591246-vmss000000) successfully I1230 03:13:53.028629 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool-17591246-vmss, kubetest-du8j4fcj, k8s-agentpool-17591246-vmss000000) for cacheKey(kubetest-du8j4fcj/k8s-agentpool-17591246-vmss) updated successfully I1230 03:13:53.028646 1 controllerserver.go:413] Attach operation successful: volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-5e94aef0-afd5-46ef-b079-f4bad5b9b2e9 attached to node k8s-agentpool-17591246-vmss000000. I1230 03:13:53.028656 1 controllerserver.go:433] attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-5e94aef0-afd5-46ef-b079-f4bad5b9b2e9 to node k8s-agentpool-17591246-vmss000000 successfully I1230 03:13:53.028824 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=10.374691062 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-du8j4fcj" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-5e94aef0-afd5-46ef-b079-f4bad5b9b2e9" node="k8s-agentpool-17591246-vmss000000" result_code="succeeded" I1230 03:13:53.028850 1 utils.go:84] GRPC response: {"publish_context":{"LUN":"0"}} ... skipping 35 lines ... I1230 03:14:52.812254 1 utils.go:78] GRPC request: {"node_id":"k8s-agentpool-17591246-vmss000000","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-247ef8cf-cacf-43d8-9263-4145cc9afa17","csi.storage.k8s.io/pvc/name":"pvc-8hpm9","csi.storage.k8s.io/pvc/namespace":"azuredisk-5356","requestedsizegib":"10","resourceGroup":"azuredisk-csi-driver-test-1da21cf0-87f0-11ed-a90a-762e3f62171e","skuName":"StandardSSD_ZRS","storage.kubernetes.io/csiProvisionerIdentity":"1672369721109-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/azuredisk-csi-driver-test-1da21cf0-87f0-11ed-a90a-762e3f62171e/providers/Microsoft.Compute/disks/pvc-247ef8cf-cacf-43d8-9263-4145cc9afa17"} I1230 03:14:52.858677 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1238 I1230 03:14:52.859360 1 controllerserver.go:383] GetDiskLun returned: cannot find Lun for disk pvc-247ef8cf-cacf-43d8-9263-4145cc9afa17. Initiating attaching volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/azuredisk-csi-driver-test-1da21cf0-87f0-11ed-a90a-762e3f62171e/providers/Microsoft.Compute/disks/pvc-247ef8cf-cacf-43d8-9263-4145cc9afa17 to node k8s-agentpool-17591246-vmss000000 (vmState Succeeded). I1230 03:14:52.859395 1 controllerserver.go:408] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/azuredisk-csi-driver-test-1da21cf0-87f0-11ed-a90a-762e3f62171e/providers/Microsoft.Compute/disks/pvc-247ef8cf-cacf-43d8-9263-4145cc9afa17 to node k8s-agentpool-17591246-vmss000000 I1230 03:14:52.859499 1 azure_controller_common.go:255] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/azuredisk-csi-driver-test-1da21cf0-87f0-11ed-a90a-762e3f62171e/providers/Microsoft.Compute/disks/pvc-247ef8cf-cacf-43d8-9263-4145cc9afa17 lun 0 to node k8s-agentpool-17591246-vmss000000, diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/azuredisk-csi-driver-test-1da21cf0-87f0-11ed-a90a-762e3f62171e/providers/microsoft.compute/disks/pvc-247ef8cf-cacf-43d8-9263-4145cc9afa17:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-247ef8cf-cacf-43d8-9263-4145cc9afa17 false 0})] I1230 03:14:52.859579 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-du8j4fcj): vm(k8s-agentpool-17591246-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/azuredisk-csi-driver-test-1da21cf0-87f0-11ed-a90a-762e3f62171e/providers/microsoft.compute/disks/pvc-247ef8cf-cacf-43d8-9263-4145cc9afa17:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-247ef8cf-cacf-43d8-9263-4145cc9afa17 false 0})]) I1230 03:14:53.034198 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-du8j4fcj): vm(k8s-agentpool-17591246-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/azuredisk-csi-driver-test-1da21cf0-87f0-11ed-a90a-762e3f62171e/providers/microsoft.compute/disks/pvc-247ef8cf-cacf-43d8-9263-4145cc9afa17:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-247ef8cf-cacf-43d8-9263-4145cc9afa17 false 0})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I1230 03:15:08.322633 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-du8j4fcj, k8s-agentpool-17591246-vmss, k8s-agentpool-17591246-vmss000000) successfully I1230 03:15:08.322674 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool-17591246-vmss, kubetest-du8j4fcj, k8s-agentpool-17591246-vmss000000) for cacheKey(kubetest-du8j4fcj/k8s-agentpool-17591246-vmss) updated successfully I1230 03:15:08.322698 1 controllerserver.go:413] Attach operation successful: volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/azuredisk-csi-driver-test-1da21cf0-87f0-11ed-a90a-762e3f62171e/providers/Microsoft.Compute/disks/pvc-247ef8cf-cacf-43d8-9263-4145cc9afa17 attached to node k8s-agentpool-17591246-vmss000000. I1230 03:15:08.322715 1 controllerserver.go:433] attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/azuredisk-csi-driver-test-1da21cf0-87f0-11ed-a90a-762e3f62171e/providers/Microsoft.Compute/disks/pvc-247ef8cf-cacf-43d8-9263-4145cc9afa17 to node k8s-agentpool-17591246-vmss000000 successfully I1230 03:15:08.322765 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=15.463420325 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-du8j4fcj" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/azuredisk-csi-driver-test-1da21cf0-87f0-11ed-a90a-762e3f62171e/providers/Microsoft.Compute/disks/pvc-247ef8cf-cacf-43d8-9263-4145cc9afa17" node="k8s-agentpool-17591246-vmss000000" result_code="succeeded" I1230 03:15:08.322792 1 utils.go:84] GRPC response: {"publish_context":{"LUN":"0"}} ... skipping 40 lines ... I1230 03:16:16.416346 1 controllerserver.go:408] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/azuredisk-csi-driver-test-504d9f85-87f0-11ed-a90a-762e3f62171e/providers/Microsoft.Compute/disks/pvc-5431c1a6-4064-4758-ae85-661b19327e29 to node k8s-agentpool-17591246-vmss000000 I1230 03:16:16.416385 1 azure_controller_common.go:255] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/azuredisk-csi-driver-test-504d9f85-87f0-11ed-a90a-762e3f62171e/providers/Microsoft.Compute/disks/pvc-5431c1a6-4064-4758-ae85-661b19327e29 lun 0 to node k8s-agentpool-17591246-vmss000000, diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/azuredisk-csi-driver-test-504d9f85-87f0-11ed-a90a-762e3f62171e/providers/microsoft.compute/disks/pvc-5431c1a6-4064-4758-ae85-661b19327e29:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-5431c1a6-4064-4758-ae85-661b19327e29 false 0})] I1230 03:16:16.416458 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-du8j4fcj): vm(k8s-agentpool-17591246-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/azuredisk-csi-driver-test-504d9f85-87f0-11ed-a90a-762e3f62171e/providers/microsoft.compute/disks/pvc-5431c1a6-4064-4758-ae85-661b19327e29:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-5431c1a6-4064-4758-ae85-661b19327e29 false 0})]) I1230 03:16:16.441879 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1238 I1230 03:16:16.442189 1 controllerserver.go:383] GetDiskLun returned: cannot find Lun for disk pvc-eba8b0df-026f-4369-bbff-a2164bc6c0ea. Initiating attaching volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/azuredisk-csi-driver-test-4fd92cdb-87f0-11ed-a90a-762e3f62171e/providers/Microsoft.Compute/disks/pvc-eba8b0df-026f-4369-bbff-a2164bc6c0ea to node k8s-agentpool-17591246-vmss000000 (vmState Succeeded). I1230 03:16:16.442223 1 controllerserver.go:408] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/azuredisk-csi-driver-test-4fd92cdb-87f0-11ed-a90a-762e3f62171e/providers/Microsoft.Compute/disks/pvc-eba8b0df-026f-4369-bbff-a2164bc6c0ea to node k8s-agentpool-17591246-vmss000000 I1230 03:16:17.273411 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-du8j4fcj): vm(k8s-agentpool-17591246-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/azuredisk-csi-driver-test-504d9f85-87f0-11ed-a90a-762e3f62171e/providers/microsoft.compute/disks/pvc-5431c1a6-4064-4758-ae85-661b19327e29:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-5431c1a6-4064-4758-ae85-661b19327e29 false 0})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I1230 03:16:27.352934 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-du8j4fcj, k8s-agentpool-17591246-vmss, k8s-agentpool-17591246-vmss000000) successfully I1230 03:16:27.352985 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool-17591246-vmss, kubetest-du8j4fcj, k8s-agentpool-17591246-vmss000000) for cacheKey(kubetest-du8j4fcj/k8s-agentpool-17591246-vmss) updated successfully I1230 03:16:27.353024 1 controllerserver.go:413] Attach operation successful: volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/azuredisk-csi-driver-test-504d9f85-87f0-11ed-a90a-762e3f62171e/providers/Microsoft.Compute/disks/pvc-5431c1a6-4064-4758-ae85-661b19327e29 attached to node k8s-agentpool-17591246-vmss000000. I1230 03:16:27.353042 1 controllerserver.go:433] attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/azuredisk-csi-driver-test-504d9f85-87f0-11ed-a90a-762e3f62171e/providers/Microsoft.Compute/disks/pvc-5431c1a6-4064-4758-ae85-661b19327e29 to node k8s-agentpool-17591246-vmss000000 successfully I1230 03:16:27.353091 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=10.936775911 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-du8j4fcj" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/azuredisk-csi-driver-test-504d9f85-87f0-11ed-a90a-762e3f62171e/providers/Microsoft.Compute/disks/pvc-5431c1a6-4064-4758-ae85-661b19327e29" node="k8s-agentpool-17591246-vmss000000" result_code="succeeded" I1230 03:16:27.353101 1 azure_controller_common.go:255] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/azuredisk-csi-driver-test-4fd92cdb-87f0-11ed-a90a-762e3f62171e/providers/Microsoft.Compute/disks/pvc-eba8b0df-026f-4369-bbff-a2164bc6c0ea lun 1 to node k8s-agentpool-17591246-vmss000000, diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/azuredisk-csi-driver-test-4fd92cdb-87f0-11ed-a90a-762e3f62171e/providers/microsoft.compute/disks/pvc-eba8b0df-026f-4369-bbff-a2164bc6c0ea:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-eba8b0df-026f-4369-bbff-a2164bc6c0ea false 1})] ... skipping 4 lines ... I1230 03:16:27.397428 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1466 I1230 03:16:27.397746 1 azure_controller_common.go:516] azureDisk - find disk: lun 0 name pvc-5431c1a6-4064-4758-ae85-661b19327e29 uri /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/azuredisk-csi-driver-test-504d9f85-87f0-11ed-a90a-762e3f62171e/providers/Microsoft.Compute/disks/pvc-5431c1a6-4064-4758-ae85-661b19327e29 I1230 03:16:27.397831 1 controllerserver.go:383] GetDiskLun returned: <nil>. Initiating attaching volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/azuredisk-csi-driver-test-504d9f85-87f0-11ed-a90a-762e3f62171e/providers/Microsoft.Compute/disks/pvc-5431c1a6-4064-4758-ae85-661b19327e29 to node k8s-agentpool-17591246-vmss000000 (vmState Succeeded). I1230 03:16:27.397855 1 controllerserver.go:398] Attach operation is successful. volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/azuredisk-csi-driver-test-504d9f85-87f0-11ed-a90a-762e3f62171e/providers/Microsoft.Compute/disks/pvc-5431c1a6-4064-4758-ae85-661b19327e29 is already attached to node k8s-agentpool-17591246-vmss000000 at lun 0. I1230 03:16:27.397919 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=0.0001605 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-du8j4fcj" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/azuredisk-csi-driver-test-504d9f85-87f0-11ed-a90a-762e3f62171e/providers/Microsoft.Compute/disks/pvc-5431c1a6-4064-4758-ae85-661b19327e29" node="k8s-agentpool-17591246-vmss000000" result_code="succeeded" I1230 03:16:27.397939 1 utils.go:84] GRPC response: {"publish_context":{"LUN":"0"}} I1230 03:16:27.502714 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-du8j4fcj): vm(k8s-agentpool-17591246-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/azuredisk-csi-driver-test-4fd92cdb-87f0-11ed-a90a-762e3f62171e/providers/microsoft.compute/disks/pvc-eba8b0df-026f-4369-bbff-a2164bc6c0ea:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-eba8b0df-026f-4369-bbff-a2164bc6c0ea false 1})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I1230 03:16:42.696031 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-du8j4fcj, k8s-agentpool-17591246-vmss, k8s-agentpool-17591246-vmss000000) successfully I1230 03:16:42.696073 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool-17591246-vmss, kubetest-du8j4fcj, k8s-agentpool-17591246-vmss000000) for cacheKey(kubetest-du8j4fcj/k8s-agentpool-17591246-vmss) updated successfully I1230 03:16:42.696114 1 controllerserver.go:413] Attach operation successful: volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/azuredisk-csi-driver-test-4fd92cdb-87f0-11ed-a90a-762e3f62171e/providers/Microsoft.Compute/disks/pvc-eba8b0df-026f-4369-bbff-a2164bc6c0ea attached to node k8s-agentpool-17591246-vmss000000. I1230 03:16:42.696132 1 controllerserver.go:433] attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/azuredisk-csi-driver-test-4fd92cdb-87f0-11ed-a90a-762e3f62171e/providers/Microsoft.Compute/disks/pvc-eba8b0df-026f-4369-bbff-a2164bc6c0ea to node k8s-agentpool-17591246-vmss000000 successfully I1230 03:16:42.696182 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=26.253986991 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-du8j4fcj" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/azuredisk-csi-driver-test-4fd92cdb-87f0-11ed-a90a-762e3f62171e/providers/Microsoft.Compute/disks/pvc-eba8b0df-026f-4369-bbff-a2164bc6c0ea" node="k8s-agentpool-17591246-vmss000000" result_code="succeeded" I1230 03:16:42.696224 1 utils.go:84] GRPC response: {"publish_context":{"LUN":"1"}} ... skipping 40 lines ... I1230 03:18:19.061519 1 utils.go:78] GRPC request: {"node_id":"k8s-agentpool-17591246-vmss000000","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-37ba13d9-4ee7-4df5-bc08-da194c710cc5","csi.storage.k8s.io/pvc/name":"pvc-672hz","csi.storage.k8s.io/pvc/namespace":"azuredisk-1353","requestedsizegib":"10","skuName":"Premium_ZRS","storage.kubernetes.io/csiProvisionerIdentity":"1672369721109-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-37ba13d9-4ee7-4df5-bc08-da194c710cc5"} I1230 03:18:19.085466 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1207 I1230 03:18:19.085803 1 controllerserver.go:383] GetDiskLun returned: cannot find Lun for disk pvc-37ba13d9-4ee7-4df5-bc08-da194c710cc5. Initiating attaching volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-37ba13d9-4ee7-4df5-bc08-da194c710cc5 to node k8s-agentpool-17591246-vmss000000 (vmState Succeeded). I1230 03:18:19.085841 1 controllerserver.go:408] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-37ba13d9-4ee7-4df5-bc08-da194c710cc5 to node k8s-agentpool-17591246-vmss000000 I1230 03:18:19.085880 1 azure_controller_common.go:255] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-37ba13d9-4ee7-4df5-bc08-da194c710cc5 lun 0 to node k8s-agentpool-17591246-vmss000000, diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-du8j4fcj/providers/microsoft.compute/disks/pvc-37ba13d9-4ee7-4df5-bc08-da194c710cc5:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-37ba13d9-4ee7-4df5-bc08-da194c710cc5 false 0})] I1230 03:18:19.086479 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-du8j4fcj): vm(k8s-agentpool-17591246-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-du8j4fcj/providers/microsoft.compute/disks/pvc-37ba13d9-4ee7-4df5-bc08-da194c710cc5:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-37ba13d9-4ee7-4df5-bc08-da194c710cc5 false 0})]) I1230 03:18:19.210288 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-du8j4fcj): vm(k8s-agentpool-17591246-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-du8j4fcj/providers/microsoft.compute/disks/pvc-37ba13d9-4ee7-4df5-bc08-da194c710cc5:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-37ba13d9-4ee7-4df5-bc08-da194c710cc5 false 0})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I1230 03:18:59.473533 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 24988 I1230 03:18:59.477111 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-du8j4fcj, k8s-agentpool-17591246-vmss, k8s-agentpool-17591246-vmss000000) successfully I1230 03:18:59.477154 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool-17591246-vmss, kubetest-du8j4fcj, k8s-agentpool-17591246-vmss000000) for cacheKey(kubetest-du8j4fcj/k8s-agentpool-17591246-vmss) updated successfully I1230 03:18:59.477177 1 controllerserver.go:413] Attach operation successful: volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-37ba13d9-4ee7-4df5-bc08-da194c710cc5 attached to node k8s-agentpool-17591246-vmss000000. I1230 03:18:59.477199 1 controllerserver.go:433] attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-37ba13d9-4ee7-4df5-bc08-da194c710cc5 to node k8s-agentpool-17591246-vmss000000 successfully I1230 03:18:59.477269 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=40.391444917 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-du8j4fcj" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-37ba13d9-4ee7-4df5-bc08-da194c710cc5" node="k8s-agentpool-17591246-vmss000000" result_code="succeeded" ... skipping 36 lines ... I1230 03:20:13.350103 1 utils.go:78] GRPC request: {"node_id":"k8s-agentpool-17591246-vmss000000","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-5d1e1f14-13e3-4db3-be93-519da1378330","csi.storage.k8s.io/pvc/name":"pvc-x65zl","csi.storage.k8s.io/pvc/namespace":"azuredisk-2888","requestedsizegib":"10","skuName":"StandardSSD_ZRS","storage.kubernetes.io/csiProvisionerIdentity":"1672369721109-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-5d1e1f14-13e3-4db3-be93-519da1378330"} I1230 03:20:13.376775 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1193 I1230 03:20:13.377298 1 controllerserver.go:383] GetDiskLun returned: cannot find Lun for disk pvc-5d1e1f14-13e3-4db3-be93-519da1378330. Initiating attaching volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-5d1e1f14-13e3-4db3-be93-519da1378330 to node k8s-agentpool-17591246-vmss000000 (vmState Succeeded). I1230 03:20:13.377350 1 controllerserver.go:408] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-5d1e1f14-13e3-4db3-be93-519da1378330 to node k8s-agentpool-17591246-vmss000000 I1230 03:20:13.377384 1 azure_controller_common.go:255] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-5d1e1f14-13e3-4db3-be93-519da1378330 lun 0 to node k8s-agentpool-17591246-vmss000000, diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-du8j4fcj/providers/microsoft.compute/disks/pvc-5d1e1f14-13e3-4db3-be93-519da1378330:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-5d1e1f14-13e3-4db3-be93-519da1378330 false 0})] I1230 03:20:13.377441 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-du8j4fcj): vm(k8s-agentpool-17591246-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-du8j4fcj/providers/microsoft.compute/disks/pvc-5d1e1f14-13e3-4db3-be93-519da1378330:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-5d1e1f14-13e3-4db3-be93-519da1378330 false 0})]) I1230 03:20:13.507096 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-du8j4fcj): vm(k8s-agentpool-17591246-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-du8j4fcj/providers/microsoft.compute/disks/pvc-5d1e1f14-13e3-4db3-be93-519da1378330:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-5d1e1f14-13e3-4db3-be93-519da1378330 false 0})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I1230 03:20:23.592388 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-du8j4fcj, k8s-agentpool-17591246-vmss, k8s-agentpool-17591246-vmss000000) successfully I1230 03:20:23.592446 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool-17591246-vmss, kubetest-du8j4fcj, k8s-agentpool-17591246-vmss000000) for cacheKey(kubetest-du8j4fcj/k8s-agentpool-17591246-vmss) updated successfully I1230 03:20:23.592471 1 controllerserver.go:413] Attach operation successful: volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-5d1e1f14-13e3-4db3-be93-519da1378330 attached to node k8s-agentpool-17591246-vmss000000. I1230 03:20:23.592486 1 controllerserver.go:433] attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-5d1e1f14-13e3-4db3-be93-519da1378330 to node k8s-agentpool-17591246-vmss000000 successfully I1230 03:20:23.592557 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=10.215234895 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-du8j4fcj" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-5d1e1f14-13e3-4db3-be93-519da1378330" node="k8s-agentpool-17591246-vmss000000" result_code="succeeded" I1230 03:20:23.592576 1 utils.go:84] GRPC response: {"publish_context":{"LUN":"0"}} ... skipping 9 lines ... I1230 03:20:37.721696 1 utils.go:78] GRPC request: {"node_id":"k8s-agentpool-17591246-vmss000000","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-48e2ebd8-9276-46f7-a44d-00f6b700d228","csi.storage.k8s.io/pvc/name":"pvc-c7qp9","csi.storage.k8s.io/pvc/namespace":"azuredisk-2888","requestedsizegib":"10","skuName":"StandardSSD_ZRS","storage.kubernetes.io/csiProvisionerIdentity":"1672369721109-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-48e2ebd8-9276-46f7-a44d-00f6b700d228"} I1230 03:20:37.772529 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1193 I1230 03:20:37.772962 1 controllerserver.go:383] GetDiskLun returned: cannot find Lun for disk pvc-48e2ebd8-9276-46f7-a44d-00f6b700d228. Initiating attaching volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-48e2ebd8-9276-46f7-a44d-00f6b700d228 to node k8s-agentpool-17591246-vmss000000 (vmState Succeeded). I1230 03:20:37.773028 1 controllerserver.go:408] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-48e2ebd8-9276-46f7-a44d-00f6b700d228 to node k8s-agentpool-17591246-vmss000000 I1230 03:20:37.773085 1 azure_controller_common.go:255] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-48e2ebd8-9276-46f7-a44d-00f6b700d228 lun 1 to node k8s-agentpool-17591246-vmss000000, diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-du8j4fcj/providers/microsoft.compute/disks/pvc-48e2ebd8-9276-46f7-a44d-00f6b700d228:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-48e2ebd8-9276-46f7-a44d-00f6b700d228 false 1})] I1230 03:20:37.773209 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-du8j4fcj): vm(k8s-agentpool-17591246-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-du8j4fcj/providers/microsoft.compute/disks/pvc-48e2ebd8-9276-46f7-a44d-00f6b700d228:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-48e2ebd8-9276-46f7-a44d-00f6b700d228 false 1})]) I1230 03:20:37.898954 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-du8j4fcj): vm(k8s-agentpool-17591246-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-du8j4fcj/providers/microsoft.compute/disks/pvc-48e2ebd8-9276-46f7-a44d-00f6b700d228:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-48e2ebd8-9276-46f7-a44d-00f6b700d228 false 1})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I1230 03:20:53.009200 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-du8j4fcj, k8s-agentpool-17591246-vmss, k8s-agentpool-17591246-vmss000000) successfully I1230 03:20:53.009248 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool-17591246-vmss, kubetest-du8j4fcj, k8s-agentpool-17591246-vmss000000) for cacheKey(kubetest-du8j4fcj/k8s-agentpool-17591246-vmss) updated successfully I1230 03:20:53.009295 1 controllerserver.go:413] Attach operation successful: volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-48e2ebd8-9276-46f7-a44d-00f6b700d228 attached to node k8s-agentpool-17591246-vmss000000. I1230 03:20:53.009337 1 controllerserver.go:433] attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-48e2ebd8-9276-46f7-a44d-00f6b700d228 to node k8s-agentpool-17591246-vmss000000 successfully I1230 03:20:53.009456 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=15.236426216 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-du8j4fcj" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-48e2ebd8-9276-46f7-a44d-00f6b700d228" node="k8s-agentpool-17591246-vmss000000" result_code="succeeded" I1230 03:20:53.009479 1 utils.go:84] GRPC response: {"publish_context":{"LUN":"1"}} ... skipping 9 lines ... I1230 03:20:59.980285 1 utils.go:78] GRPC request: {"node_id":"k8s-agentpool-17591246-vmss000001","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-26cba7b7-1733-4254-9199-c4cf87fbc032","csi.storage.k8s.io/pvc/name":"pvc-r5gd6","csi.storage.k8s.io/pvc/namespace":"azuredisk-2888","requestedsizegib":"10","skuName":"StandardSSD_ZRS","storage.kubernetes.io/csiProvisionerIdentity":"1672369721109-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-26cba7b7-1733-4254-9199-c4cf87fbc032"} I1230 03:21:00.002274 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1193 I1230 03:21:00.002698 1 controllerserver.go:383] GetDiskLun returned: cannot find Lun for disk pvc-26cba7b7-1733-4254-9199-c4cf87fbc032. Initiating attaching volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-26cba7b7-1733-4254-9199-c4cf87fbc032 to node k8s-agentpool-17591246-vmss000001 (vmState Succeeded). I1230 03:21:00.002727 1 controllerserver.go:408] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-26cba7b7-1733-4254-9199-c4cf87fbc032 to node k8s-agentpool-17591246-vmss000001 I1230 03:21:00.002812 1 azure_controller_common.go:255] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-26cba7b7-1733-4254-9199-c4cf87fbc032 lun 0 to node k8s-agentpool-17591246-vmss000001, diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-du8j4fcj/providers/microsoft.compute/disks/pvc-26cba7b7-1733-4254-9199-c4cf87fbc032:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-26cba7b7-1733-4254-9199-c4cf87fbc032 false 0})] I1230 03:21:00.002850 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-du8j4fcj): vm(k8s-agentpool-17591246-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-du8j4fcj/providers/microsoft.compute/disks/pvc-26cba7b7-1733-4254-9199-c4cf87fbc032:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-26cba7b7-1733-4254-9199-c4cf87fbc032 false 0})]) I1230 03:21:00.136918 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-du8j4fcj): vm(k8s-agentpool-17591246-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-du8j4fcj/providers/microsoft.compute/disks/pvc-26cba7b7-1733-4254-9199-c4cf87fbc032:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-26cba7b7-1733-4254-9199-c4cf87fbc032 false 0})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I1230 03:21:10.290132 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-du8j4fcj, k8s-agentpool-17591246-vmss, k8s-agentpool-17591246-vmss000001) successfully I1230 03:21:10.290177 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool-17591246-vmss, kubetest-du8j4fcj, k8s-agentpool-17591246-vmss000001) for cacheKey(kubetest-du8j4fcj/k8s-agentpool-17591246-vmss) updated successfully I1230 03:21:10.290204 1 controllerserver.go:413] Attach operation successful: volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-26cba7b7-1733-4254-9199-c4cf87fbc032 attached to node k8s-agentpool-17591246-vmss000001. I1230 03:21:10.290222 1 controllerserver.go:433] attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-26cba7b7-1733-4254-9199-c4cf87fbc032 to node k8s-agentpool-17591246-vmss000001 successfully I1230 03:21:10.290512 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=10.287791771 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-du8j4fcj" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-26cba7b7-1733-4254-9199-c4cf87fbc032" node="k8s-agentpool-17591246-vmss000001" result_code="succeeded" I1230 03:21:10.290545 1 utils.go:84] GRPC response: {"publish_context":{"LUN":"0"}} ... skipping 42 lines ... I1230 03:24:54.276428 1 azure_vmss_cache.go:327] refresh the cache of NonVmssUniformNodesCache in rg map[kubetest-du8j4fcj:{}] I1230 03:24:54.299060 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 12 I1230 03:24:54.299144 1 controllerserver.go:383] GetDiskLun returned: cannot find Lun for disk pvc-7e1d2cf4-c636-4564-93d3-1a0201cb52ff. Initiating attaching volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-7e1d2cf4-c636-4564-93d3-1a0201cb52ff to node k8s-agentpool-17591246-vmss000000 (vmState Succeeded). I1230 03:24:54.299240 1 controllerserver.go:408] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-7e1d2cf4-c636-4564-93d3-1a0201cb52ff to node k8s-agentpool-17591246-vmss000000 I1230 03:24:54.299365 1 azure_controller_common.go:255] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-7e1d2cf4-c636-4564-93d3-1a0201cb52ff lun 0 to node k8s-agentpool-17591246-vmss000000, diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-du8j4fcj/providers/microsoft.compute/disks/pvc-7e1d2cf4-c636-4564-93d3-1a0201cb52ff:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-7e1d2cf4-c636-4564-93d3-1a0201cb52ff false 0})] I1230 03:24:54.299519 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-du8j4fcj): vm(k8s-agentpool-17591246-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-du8j4fcj/providers/microsoft.compute/disks/pvc-7e1d2cf4-c636-4564-93d3-1a0201cb52ff:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-7e1d2cf4-c636-4564-93d3-1a0201cb52ff false 0})]) I1230 03:24:54.476444 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-du8j4fcj): vm(k8s-agentpool-17591246-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-du8j4fcj/providers/microsoft.compute/disks/pvc-7e1d2cf4-c636-4564-93d3-1a0201cb52ff:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-7e1d2cf4-c636-4564-93d3-1a0201cb52ff false 0})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I1230 03:25:04.710082 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-du8j4fcj, k8s-agentpool-17591246-vmss, k8s-agentpool-17591246-vmss000000) successfully I1230 03:25:04.710139 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool-17591246-vmss, kubetest-du8j4fcj, k8s-agentpool-17591246-vmss000000) for cacheKey(kubetest-du8j4fcj/k8s-agentpool-17591246-vmss) updated successfully I1230 03:25:04.710175 1 controllerserver.go:413] Attach operation successful: volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-7e1d2cf4-c636-4564-93d3-1a0201cb52ff attached to node k8s-agentpool-17591246-vmss000000. I1230 03:25:04.710206 1 controllerserver.go:433] attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-7e1d2cf4-c636-4564-93d3-1a0201cb52ff to node k8s-agentpool-17591246-vmss000000 successfully I1230 03:25:04.710270 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=10.433803718 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-du8j4fcj" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-7e1d2cf4-c636-4564-93d3-1a0201cb52ff" node="k8s-agentpool-17591246-vmss000000" result_code="succeeded" I1230 03:25:04.710287 1 utils.go:84] GRPC response: {"publish_context":{"LUN":"0"}} ... skipping 27 lines ... I1230 03:27:45.081437 1 utils.go:78] GRPC request: {"node_id":"k8s-agentpool-17591246-vmss000000","volume_capability":{"AccessType":{"Mount":{"fs_type":"xfs"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-5de4b024-0bce-4c3e-a613-4716ff15d5ce","csi.storage.k8s.io/pvc/name":"pvc-7lrhk","csi.storage.k8s.io/pvc/namespace":"azuredisk-59","fsType":"xfs","requestedsizegib":"10","skuName":"Standard_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1672369721109-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-5de4b024-0bce-4c3e-a613-4716ff15d5ce"} I1230 03:27:45.145662 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1217 I1230 03:27:45.146095 1 controllerserver.go:383] GetDiskLun returned: cannot find Lun for disk pvc-5de4b024-0bce-4c3e-a613-4716ff15d5ce. Initiating attaching volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-5de4b024-0bce-4c3e-a613-4716ff15d5ce to node k8s-agentpool-17591246-vmss000000 (vmState Succeeded). I1230 03:27:45.146133 1 controllerserver.go:408] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-5de4b024-0bce-4c3e-a613-4716ff15d5ce to node k8s-agentpool-17591246-vmss000000 I1230 03:27:45.146175 1 azure_controller_common.go:255] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-5de4b024-0bce-4c3e-a613-4716ff15d5ce lun 0 to node k8s-agentpool-17591246-vmss000000, diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-du8j4fcj/providers/microsoft.compute/disks/pvc-5de4b024-0bce-4c3e-a613-4716ff15d5ce:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-5de4b024-0bce-4c3e-a613-4716ff15d5ce false 0})] I1230 03:27:45.146223 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-du8j4fcj): vm(k8s-agentpool-17591246-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-du8j4fcj/providers/microsoft.compute/disks/pvc-5de4b024-0bce-4c3e-a613-4716ff15d5ce:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-5de4b024-0bce-4c3e-a613-4716ff15d5ce false 0})]) I1230 03:27:45.811103 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-du8j4fcj): vm(k8s-agentpool-17591246-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-du8j4fcj/providers/microsoft.compute/disks/pvc-5de4b024-0bce-4c3e-a613-4716ff15d5ce:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-5de4b024-0bce-4c3e-a613-4716ff15d5ce false 0})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I1230 03:28:00.922580 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-du8j4fcj, k8s-agentpool-17591246-vmss, k8s-agentpool-17591246-vmss000000) successfully I1230 03:28:00.922623 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool-17591246-vmss, kubetest-du8j4fcj, k8s-agentpool-17591246-vmss000000) for cacheKey(kubetest-du8j4fcj/k8s-agentpool-17591246-vmss) updated successfully I1230 03:28:00.922646 1 controllerserver.go:413] Attach operation successful: volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-5de4b024-0bce-4c3e-a613-4716ff15d5ce attached to node k8s-agentpool-17591246-vmss000000. I1230 03:28:00.922661 1 controllerserver.go:433] attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-5de4b024-0bce-4c3e-a613-4716ff15d5ce to node k8s-agentpool-17591246-vmss000000 successfully I1230 03:28:00.922705 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=15.776616138 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-du8j4fcj" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-5de4b024-0bce-4c3e-a613-4716ff15d5ce" node="k8s-agentpool-17591246-vmss000000" result_code="succeeded" I1230 03:28:00.922726 1 utils.go:84] GRPC response: {"publish_context":{"LUN":"0"}} ... skipping 22 lines ... I1230 03:28:43.720224 1 azure_controller_common.go:422] azureDisk - detach disk(pvc-5de4b024-0bce-4c3e-a613-4716ff15d5ce, /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-5de4b024-0bce-4c3e-a613-4716ff15d5ce) succeeded I1230 03:28:43.720269 1 controllerserver.go:480] detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-5de4b024-0bce-4c3e-a613-4716ff15d5ce from node k8s-agentpool-17591246-vmss000000 successfully I1230 03:28:43.720321 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=15.534773801 request="azuredisk_csi_driver_controller_unpublish_volume" resource_group="kubetest-du8j4fcj" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-5de4b024-0bce-4c3e-a613-4716ff15d5ce" node="k8s-agentpool-17591246-vmss000000" result_code="succeeded" I1230 03:28:43.720345 1 utils.go:84] GRPC response: {} I1230 03:28:43.720493 1 azure_controller_common.go:255] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-1b45d7aa-e0a8-4d8c-b218-4ab9ad781db8 lun 0 to node k8s-agentpool-17591246-vmss000000, diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-du8j4fcj/providers/microsoft.compute/disks/pvc-1b45d7aa-e0a8-4d8c-b218-4ab9ad781db8:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-1b45d7aa-e0a8-4d8c-b218-4ab9ad781db8 false 0})] I1230 03:28:43.720541 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-du8j4fcj): vm(k8s-agentpool-17591246-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-du8j4fcj/providers/microsoft.compute/disks/pvc-1b45d7aa-e0a8-4d8c-b218-4ab9ad781db8:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-1b45d7aa-e0a8-4d8c-b218-4ab9ad781db8 false 0})]) I1230 03:28:43.954332 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-du8j4fcj): vm(k8s-agentpool-17591246-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-du8j4fcj/providers/microsoft.compute/disks/pvc-1b45d7aa-e0a8-4d8c-b218-4ab9ad781db8:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-1b45d7aa-e0a8-4d8c-b218-4ab9ad781db8 false 0})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I1230 03:28:54.122046 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-du8j4fcj, k8s-agentpool-17591246-vmss, k8s-agentpool-17591246-vmss000000) successfully I1230 03:28:54.122087 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool-17591246-vmss, kubetest-du8j4fcj, k8s-agentpool-17591246-vmss000000) for cacheKey(kubetest-du8j4fcj/k8s-agentpool-17591246-vmss) updated successfully I1230 03:28:54.122109 1 controllerserver.go:413] Attach operation successful: volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-1b45d7aa-e0a8-4d8c-b218-4ab9ad781db8 attached to node k8s-agentpool-17591246-vmss000000. I1230 03:28:54.122124 1 controllerserver.go:433] attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-1b45d7aa-e0a8-4d8c-b218-4ab9ad781db8 to node k8s-agentpool-17591246-vmss000000 successfully I1230 03:28:54.122169 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=23.564514269 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-du8j4fcj" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-1b45d7aa-e0a8-4d8c-b218-4ab9ad781db8" node="k8s-agentpool-17591246-vmss000000" result_code="succeeded" I1230 03:28:54.122194 1 utils.go:84] GRPC response: {"publish_context":{"LUN":"0"}} ... skipping 28 lines ... I1230 03:30:03.265316 1 utils.go:78] GRPC request: {"node_id":"k8s-agentpool-17591246-vmss000000","volume_capability":{"AccessType":{"Mount":{"fs_type":"xfs"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-09cd96fc-cbbc-4d4c-b65a-1b7e53a07aff","csi.storage.k8s.io/pvc/name":"pvc-jthc6","csi.storage.k8s.io/pvc/namespace":"azuredisk-2546","fsType":"xfs","networkAccessPolicy":"DenyAll","requestedsizegib":"10","skuName":"StandardSSD_ZRS","storage.kubernetes.io/csiProvisionerIdentity":"1672369721109-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-09cd96fc-cbbc-4d4c-b65a-1b7e53a07aff"} I1230 03:30:03.356483 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1192 I1230 03:30:03.356980 1 controllerserver.go:383] GetDiskLun returned: cannot find Lun for disk pvc-09cd96fc-cbbc-4d4c-b65a-1b7e53a07aff. Initiating attaching volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-09cd96fc-cbbc-4d4c-b65a-1b7e53a07aff to node k8s-agentpool-17591246-vmss000000 (vmState Succeeded). I1230 03:30:03.357029 1 controllerserver.go:408] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-09cd96fc-cbbc-4d4c-b65a-1b7e53a07aff to node k8s-agentpool-17591246-vmss000000 I1230 03:30:03.357101 1 azure_controller_common.go:255] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-09cd96fc-cbbc-4d4c-b65a-1b7e53a07aff lun 0 to node k8s-agentpool-17591246-vmss000000, diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-du8j4fcj/providers/microsoft.compute/disks/pvc-09cd96fc-cbbc-4d4c-b65a-1b7e53a07aff:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-09cd96fc-cbbc-4d4c-b65a-1b7e53a07aff false 0})] I1230 03:30:03.357474 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-du8j4fcj): vm(k8s-agentpool-17591246-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-du8j4fcj/providers/microsoft.compute/disks/pvc-09cd96fc-cbbc-4d4c-b65a-1b7e53a07aff:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-09cd96fc-cbbc-4d4c-b65a-1b7e53a07aff false 0})]) I1230 03:30:03.533464 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-du8j4fcj): vm(k8s-agentpool-17591246-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-du8j4fcj/providers/microsoft.compute/disks/pvc-09cd96fc-cbbc-4d4c-b65a-1b7e53a07aff:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-09cd96fc-cbbc-4d4c-b65a-1b7e53a07aff false 0})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I1230 03:30:13.666718 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-du8j4fcj, k8s-agentpool-17591246-vmss, k8s-agentpool-17591246-vmss000000) successfully I1230 03:30:13.666775 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool-17591246-vmss, kubetest-du8j4fcj, k8s-agentpool-17591246-vmss000000) for cacheKey(kubetest-du8j4fcj/k8s-agentpool-17591246-vmss) updated successfully I1230 03:30:13.666817 1 controllerserver.go:413] Attach operation successful: volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-09cd96fc-cbbc-4d4c-b65a-1b7e53a07aff attached to node k8s-agentpool-17591246-vmss000000. I1230 03:30:13.666852 1 controllerserver.go:433] attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-09cd96fc-cbbc-4d4c-b65a-1b7e53a07aff to node k8s-agentpool-17591246-vmss000000 successfully I1230 03:30:13.666934 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=10.309968194 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-du8j4fcj" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-09cd96fc-cbbc-4d4c-b65a-1b7e53a07aff" node="k8s-agentpool-17591246-vmss000000" result_code="succeeded" I1230 03:30:13.666956 1 utils.go:84] GRPC response: {"publish_context":{"LUN":"0"}} ... skipping 17 lines ... I1230 03:30:43.519770 1 azure_controller_common.go:255] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-f90b471f-4346-4dd3-bb62-b179b0df7c0f lun 0 to node k8s-agentpool-17591246-vmss000000, diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-du8j4fcj/providers/microsoft.compute/disks/pvc-f90b471f-4346-4dd3-bb62-b179b0df7c0f:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-f90b471f-4346-4dd3-bb62-b179b0df7c0f false 0})] I1230 03:30:43.519779 1 utils.go:84] GRPC response: {} I1230 03:30:43.519824 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-du8j4fcj): vm(k8s-agentpool-17591246-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-du8j4fcj/providers/microsoft.compute/disks/pvc-f90b471f-4346-4dd3-bb62-b179b0df7c0f:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-f90b471f-4346-4dd3-bb62-b179b0df7c0f false 0})]) I1230 03:30:43.539996 1 utils.go:77] GRPC call: /csi.v1.Controller/ControllerUnpublishVolume I1230 03:30:43.540014 1 utils.go:78] GRPC request: {"node_id":"k8s-agentpool-17591246-vmss000000","volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-09cd96fc-cbbc-4d4c-b65a-1b7e53a07aff"} I1230 03:30:43.540136 1 controllerserver.go:471] Trying to detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-09cd96fc-cbbc-4d4c-b65a-1b7e53a07aff from node k8s-agentpool-17591246-vmss000000 I1230 03:30:43.720090 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-du8j4fcj): vm(k8s-agentpool-17591246-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-du8j4fcj/providers/microsoft.compute/disks/pvc-f90b471f-4346-4dd3-bb62-b179b0df7c0f:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-f90b471f-4346-4dd3-bb62-b179b0df7c0f false 0})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I1230 03:30:53.829224 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-du8j4fcj, k8s-agentpool-17591246-vmss, k8s-agentpool-17591246-vmss000000) successfully I1230 03:30:53.829266 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool-17591246-vmss, kubetest-du8j4fcj, k8s-agentpool-17591246-vmss000000) for cacheKey(kubetest-du8j4fcj/k8s-agentpool-17591246-vmss) updated successfully I1230 03:30:53.829299 1 controllerserver.go:413] Attach operation successful: volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-f90b471f-4346-4dd3-bb62-b179b0df7c0f attached to node k8s-agentpool-17591246-vmss000000. I1230 03:30:53.829316 1 controllerserver.go:433] attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-f90b471f-4346-4dd3-bb62-b179b0df7c0f to node k8s-agentpool-17591246-vmss000000 successfully I1230 03:30:53.829364 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=21.201738565 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-du8j4fcj" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-f90b471f-4346-4dd3-bb62-b179b0df7c0f" node="k8s-agentpool-17591246-vmss000000" result_code="succeeded" I1230 03:30:53.829391 1 utils.go:84] GRPC response: {"publish_context":{"LUN":"0"}} ... skipping 55 lines ... I1230 03:32:05.654659 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1193 I1230 03:32:05.654935 1 controllerserver.go:383] GetDiskLun returned: cannot find Lun for disk pvc-1e16c08d-d552-4acb-b5f5-344d58e96833. Initiating attaching volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-1e16c08d-d552-4acb-b5f5-344d58e96833 to node k8s-agentpool-17591246-vmss000000 (vmState Succeeded). I1230 03:32:05.654968 1 controllerserver.go:408] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-1e16c08d-d552-4acb-b5f5-344d58e96833 to node k8s-agentpool-17591246-vmss000000 I1230 03:32:05.705623 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1193 I1230 03:32:05.705870 1 controllerserver.go:383] GetDiskLun returned: cannot find Lun for disk pvc-c6b24d91-d780-466f-b5ff-38b1b95af676. Initiating attaching volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-c6b24d91-d780-466f-b5ff-38b1b95af676 to node k8s-agentpool-17591246-vmss000000 (vmState Succeeded). I1230 03:32:05.705902 1 controllerserver.go:408] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-c6b24d91-d780-466f-b5ff-38b1b95af676 to node k8s-agentpool-17591246-vmss000000 I1230 03:32:05.846743 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-du8j4fcj): vm(k8s-agentpool-17591246-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-du8j4fcj/providers/microsoft.compute/disks/pvc-5b19c2c7-998d-498a-a02b-e92a17f6986e:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-5b19c2c7-998d-498a-a02b-e92a17f6986e false 0})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I1230 03:32:15.953920 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-du8j4fcj, k8s-agentpool-17591246-vmss, k8s-agentpool-17591246-vmss000000) successfully I1230 03:32:15.953959 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool-17591246-vmss, kubetest-du8j4fcj, k8s-agentpool-17591246-vmss000000) for cacheKey(kubetest-du8j4fcj/k8s-agentpool-17591246-vmss) updated successfully I1230 03:32:15.953999 1 controllerserver.go:413] Attach operation successful: volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-5b19c2c7-998d-498a-a02b-e92a17f6986e attached to node k8s-agentpool-17591246-vmss000000. I1230 03:32:15.954014 1 controllerserver.go:433] attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-5b19c2c7-998d-498a-a02b-e92a17f6986e to node k8s-agentpool-17591246-vmss000000 successfully I1230 03:32:15.954057 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=10.315854813 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-du8j4fcj" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-5b19c2c7-998d-498a-a02b-e92a17f6986e" node="k8s-agentpool-17591246-vmss000000" result_code="succeeded" I1230 03:32:15.954090 1 utils.go:84] GRPC response: {"publish_context":{"LUN":"0"}} ... skipping 4 lines ... I1230 03:32:15.996320 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1420 I1230 03:32:15.996689 1 azure_controller_common.go:516] azureDisk - find disk: lun 0 name pvc-5b19c2c7-998d-498a-a02b-e92a17f6986e uri /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-5b19c2c7-998d-498a-a02b-e92a17f6986e I1230 03:32:15.996740 1 controllerserver.go:383] GetDiskLun returned: <nil>. Initiating attaching volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-5b19c2c7-998d-498a-a02b-e92a17f6986e to node k8s-agentpool-17591246-vmss000000 (vmState Succeeded). I1230 03:32:15.996758 1 controllerserver.go:398] Attach operation is successful. volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-5b19c2c7-998d-498a-a02b-e92a17f6986e is already attached to node k8s-agentpool-17591246-vmss000000 at lun 0. I1230 03:32:15.996806 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=0.0001271 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-du8j4fcj" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-5b19c2c7-998d-498a-a02b-e92a17f6986e" node="k8s-agentpool-17591246-vmss000000" result_code="succeeded" I1230 03:32:15.996824 1 utils.go:84] GRPC response: {"publish_context":{"LUN":"0"}} I1230 03:32:16.146388 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-du8j4fcj): vm(k8s-agentpool-17591246-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-du8j4fcj/providers/microsoft.compute/disks/pvc-1e16c08d-d552-4acb-b5f5-344d58e96833:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-1e16c08d-d552-4acb-b5f5-344d58e96833 false 1}) /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-du8j4fcj/providers/microsoft.compute/disks/pvc-c6b24d91-d780-466f-b5ff-38b1b95af676:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-c6b24d91-d780-466f-b5ff-38b1b95af676 false 2})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I1230 03:32:26.252299 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-du8j4fcj, k8s-agentpool-17591246-vmss, k8s-agentpool-17591246-vmss000000) successfully I1230 03:32:26.252353 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool-17591246-vmss, kubetest-du8j4fcj, k8s-agentpool-17591246-vmss000000) for cacheKey(kubetest-du8j4fcj/k8s-agentpool-17591246-vmss) updated successfully I1230 03:32:26.256735 1 controllerserver.go:413] Attach operation successful: volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-1e16c08d-d552-4acb-b5f5-344d58e96833 attached to node k8s-agentpool-17591246-vmss000000. I1230 03:32:26.256767 1 controllerserver.go:433] attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-1e16c08d-d552-4acb-b5f5-344d58e96833 to node k8s-agentpool-17591246-vmss000000 successfully I1230 03:32:26.256822 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=20.601868273 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-du8j4fcj" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-1e16c08d-d552-4acb-b5f5-344d58e96833" node="k8s-agentpool-17591246-vmss000000" result_code="succeeded" I1230 03:32:26.256839 1 utils.go:84] GRPC response: {"publish_context":{"LUN":"1"}} ... skipping 85 lines ... I1230 03:34:17.284936 1 controllerserver.go:408] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-d09842e7-2ad7-48e6-8375-9c8ff141aa66 to node k8s-agentpool-17591246-vmss000000 I1230 03:34:17.285123 1 azure_controller_common.go:255] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-d09842e7-2ad7-48e6-8375-9c8ff141aa66 lun 0 to node k8s-agentpool-17591246-vmss000000, diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-du8j4fcj/providers/microsoft.compute/disks/pvc-d09842e7-2ad7-48e6-8375-9c8ff141aa66:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-d09842e7-2ad7-48e6-8375-9c8ff141aa66 false 0})] I1230 03:34:17.285212 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-du8j4fcj): vm(k8s-agentpool-17591246-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-du8j4fcj/providers/microsoft.compute/disks/pvc-d09842e7-2ad7-48e6-8375-9c8ff141aa66:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-d09842e7-2ad7-48e6-8375-9c8ff141aa66 false 0})]) I1230 03:34:17.311111 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1193 I1230 03:34:17.311473 1 controllerserver.go:383] GetDiskLun returned: cannot find Lun for disk pvc-8e40ac05-1543-4d95-bb2c-c8bdc215c0ae. Initiating attaching volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-8e40ac05-1543-4d95-bb2c-c8bdc215c0ae to node k8s-agentpool-17591246-vmss000000 (vmState Succeeded). I1230 03:34:17.311511 1 controllerserver.go:408] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-8e40ac05-1543-4d95-bb2c-c8bdc215c0ae to node k8s-agentpool-17591246-vmss000000 I1230 03:34:17.431728 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-du8j4fcj): vm(k8s-agentpool-17591246-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-du8j4fcj/providers/microsoft.compute/disks/pvc-d09842e7-2ad7-48e6-8375-9c8ff141aa66:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-d09842e7-2ad7-48e6-8375-9c8ff141aa66 false 0})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I1230 03:34:32.633843 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-du8j4fcj, k8s-agentpool-17591246-vmss, k8s-agentpool-17591246-vmss000000) successfully I1230 03:34:32.633891 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool-17591246-vmss, kubetest-du8j4fcj, k8s-agentpool-17591246-vmss000000) for cacheKey(kubetest-du8j4fcj/k8s-agentpool-17591246-vmss) updated successfully I1230 03:34:32.633933 1 controllerserver.go:413] Attach operation successful: volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-d09842e7-2ad7-48e6-8375-9c8ff141aa66 attached to node k8s-agentpool-17591246-vmss000000. I1230 03:34:32.633952 1 controllerserver.go:433] attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-d09842e7-2ad7-48e6-8375-9c8ff141aa66 to node k8s-agentpool-17591246-vmss000000 successfully I1230 03:34:32.634021 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=15.349107737 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-du8j4fcj" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-d09842e7-2ad7-48e6-8375-9c8ff141aa66" node="k8s-agentpool-17591246-vmss000000" result_code="succeeded" I1230 03:34:32.634039 1 utils.go:84] GRPC response: {"publish_context":{"LUN":"0"}} I1230 03:34:32.634057 1 azure_controller_common.go:255] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-8e40ac05-1543-4d95-bb2c-c8bdc215c0ae lun 1 to node k8s-agentpool-17591246-vmss000000, diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-du8j4fcj/providers/microsoft.compute/disks/pvc-8e40ac05-1543-4d95-bb2c-c8bdc215c0ae:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-8e40ac05-1543-4d95-bb2c-c8bdc215c0ae false 1})] I1230 03:34:32.634159 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-du8j4fcj): vm(k8s-agentpool-17591246-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-du8j4fcj/providers/microsoft.compute/disks/pvc-8e40ac05-1543-4d95-bb2c-c8bdc215c0ae:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-8e40ac05-1543-4d95-bb2c-c8bdc215c0ae false 1})]) I1230 03:34:32.847145 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-du8j4fcj): vm(k8s-agentpool-17591246-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-du8j4fcj/providers/microsoft.compute/disks/pvc-8e40ac05-1543-4d95-bb2c-c8bdc215c0ae:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-8e40ac05-1543-4d95-bb2c-c8bdc215c0ae false 1})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I1230 03:34:42.993292 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-du8j4fcj, k8s-agentpool-17591246-vmss, k8s-agentpool-17591246-vmss000000) successfully I1230 03:34:42.993350 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool-17591246-vmss, kubetest-du8j4fcj, k8s-agentpool-17591246-vmss000000) for cacheKey(kubetest-du8j4fcj/k8s-agentpool-17591246-vmss) updated successfully I1230 03:34:42.993374 1 controllerserver.go:413] Attach operation successful: volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-8e40ac05-1543-4d95-bb2c-c8bdc215c0ae attached to node k8s-agentpool-17591246-vmss000000. I1230 03:34:42.993390 1 controllerserver.go:433] attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-8e40ac05-1543-4d95-bb2c-c8bdc215c0ae to node k8s-agentpool-17591246-vmss000000 successfully I1230 03:34:42.993436 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=25.681977711000002 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-du8j4fcj" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-8e40ac05-1543-4d95-bb2c-c8bdc215c0ae" node="k8s-agentpool-17591246-vmss000000" result_code="succeeded" I1230 03:34:42.993513 1 utils.go:84] GRPC response: {"publish_context":{"LUN":"1"}} ... skipping 48 lines ... I1230 03:36:20.299902 1 utils.go:78] GRPC request: {"node_id":"k8s-agentpool-17591246-vmss000000","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-f824bdf6-18b4-4fcb-9aa9-f2f83a57ea1c","csi.storage.k8s.io/pvc/name":"pvc-sfxbt","csi.storage.k8s.io/pvc/namespace":"azuredisk-8582","requestedsizegib":"10","skuName":"StandardSSD_ZRS","storage.kubernetes.io/csiProvisionerIdentity":"1672369721109-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-f824bdf6-18b4-4fcb-9aa9-f2f83a57ea1c"} I1230 03:36:20.394416 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1193 I1230 03:36:20.394756 1 controllerserver.go:383] GetDiskLun returned: cannot find Lun for disk pvc-f824bdf6-18b4-4fcb-9aa9-f2f83a57ea1c. Initiating attaching volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-f824bdf6-18b4-4fcb-9aa9-f2f83a57ea1c to node k8s-agentpool-17591246-vmss000000 (vmState Succeeded). I1230 03:36:20.394789 1 controllerserver.go:408] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-f824bdf6-18b4-4fcb-9aa9-f2f83a57ea1c to node k8s-agentpool-17591246-vmss000000 I1230 03:36:20.394840 1 azure_controller_common.go:255] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-f824bdf6-18b4-4fcb-9aa9-f2f83a57ea1c lun 0 to node k8s-agentpool-17591246-vmss000000, diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-du8j4fcj/providers/microsoft.compute/disks/pvc-f824bdf6-18b4-4fcb-9aa9-f2f83a57ea1c:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-f824bdf6-18b4-4fcb-9aa9-f2f83a57ea1c false 0})] I1230 03:36:20.394887 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-du8j4fcj): vm(k8s-agentpool-17591246-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-du8j4fcj/providers/microsoft.compute/disks/pvc-f824bdf6-18b4-4fcb-9aa9-f2f83a57ea1c:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-f824bdf6-18b4-4fcb-9aa9-f2f83a57ea1c false 0})]) I1230 03:36:20.564317 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-du8j4fcj): vm(k8s-agentpool-17591246-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-du8j4fcj/providers/microsoft.compute/disks/pvc-f824bdf6-18b4-4fcb-9aa9-f2f83a57ea1c:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-f824bdf6-18b4-4fcb-9aa9-f2f83a57ea1c false 0})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I1230 03:36:30.660927 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-du8j4fcj, k8s-agentpool-17591246-vmss, k8s-agentpool-17591246-vmss000000) successfully I1230 03:36:30.660969 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool-17591246-vmss, kubetest-du8j4fcj, k8s-agentpool-17591246-vmss000000) for cacheKey(kubetest-du8j4fcj/k8s-agentpool-17591246-vmss) updated successfully I1230 03:36:30.660991 1 controllerserver.go:413] Attach operation successful: volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-f824bdf6-18b4-4fcb-9aa9-f2f83a57ea1c attached to node k8s-agentpool-17591246-vmss000000. I1230 03:36:30.661006 1 controllerserver.go:433] attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-f824bdf6-18b4-4fcb-9aa9-f2f83a57ea1c to node k8s-agentpool-17591246-vmss000000 successfully I1230 03:36:30.661050 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=10.266314757 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-du8j4fcj" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-f824bdf6-18b4-4fcb-9aa9-f2f83a57ea1c" node="k8s-agentpool-17591246-vmss000000" result_code="succeeded" I1230 03:36:30.661066 1 utils.go:84] GRPC response: {"publish_context":{"LUN":"0"}} ... skipping 14 lines ... I1230 03:37:03.926027 1 azure_controller_common.go:422] azureDisk - detach disk(pvc-f824bdf6-18b4-4fcb-9aa9-f2f83a57ea1c, /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-f824bdf6-18b4-4fcb-9aa9-f2f83a57ea1c) succeeded I1230 03:37:03.926054 1 controllerserver.go:480] detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-f824bdf6-18b4-4fcb-9aa9-f2f83a57ea1c from node k8s-agentpool-17591246-vmss000000 successfully I1230 03:37:03.926096 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=15.305706311 request="azuredisk_csi_driver_controller_unpublish_volume" resource_group="kubetest-du8j4fcj" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-f824bdf6-18b4-4fcb-9aa9-f2f83a57ea1c" node="k8s-agentpool-17591246-vmss000000" result_code="succeeded" I1230 03:37:03.926114 1 utils.go:84] GRPC response: {} I1230 03:37:03.926187 1 azure_controller_common.go:255] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-ee39ec89-f6e8-48fd-b252-5b8ef9a0d01e lun 0 to node k8s-agentpool-17591246-vmss000000, diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-du8j4fcj/providers/microsoft.compute/disks/pvc-ee39ec89-f6e8-48fd-b252-5b8ef9a0d01e:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-ee39ec89-f6e8-48fd-b252-5b8ef9a0d01e false 0})] I1230 03:37:03.926244 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-du8j4fcj): vm(k8s-agentpool-17591246-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-du8j4fcj/providers/microsoft.compute/disks/pvc-ee39ec89-f6e8-48fd-b252-5b8ef9a0d01e:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-ee39ec89-f6e8-48fd-b252-5b8ef9a0d01e false 0})]) I1230 03:37:04.125236 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-du8j4fcj): vm(k8s-agentpool-17591246-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-du8j4fcj/providers/microsoft.compute/disks/pvc-ee39ec89-f6e8-48fd-b252-5b8ef9a0d01e:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-ee39ec89-f6e8-48fd-b252-5b8ef9a0d01e false 0})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I1230 03:37:14.228969 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-du8j4fcj, k8s-agentpool-17591246-vmss, k8s-agentpool-17591246-vmss000000) successfully I1230 03:37:14.229092 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool-17591246-vmss, kubetest-du8j4fcj, k8s-agentpool-17591246-vmss000000) for cacheKey(kubetest-du8j4fcj/k8s-agentpool-17591246-vmss) updated successfully I1230 03:37:14.229185 1 controllerserver.go:413] Attach operation successful: volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-ee39ec89-f6e8-48fd-b252-5b8ef9a0d01e attached to node k8s-agentpool-17591246-vmss000000. I1230 03:37:14.229297 1 controllerserver.go:433] attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-ee39ec89-f6e8-48fd-b252-5b8ef9a0d01e to node k8s-agentpool-17591246-vmss000000 successfully I1230 03:37:14.229396 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=12.687477651 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-du8j4fcj" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-ee39ec89-f6e8-48fd-b252-5b8ef9a0d01e" node="k8s-agentpool-17591246-vmss000000" result_code="succeeded" I1230 03:37:14.229424 1 utils.go:84] GRPC response: {"publish_context":{"LUN":"0"}} ... skipping 36 lines ... I1230 03:39:35.046390 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1193 I1230 03:39:35.238860 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 24988 I1230 03:39:35.241992 1 controllerserver.go:383] GetDiskLun returned: cannot find Lun for disk pvc-46a1030f-1b72-4b7d-8daa-0ecfb6288da6. Initiating attaching volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-46a1030f-1b72-4b7d-8daa-0ecfb6288da6 to node k8s-agentpool-17591246-vmss000000 (vmState Succeeded). I1230 03:39:35.242028 1 controllerserver.go:408] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-46a1030f-1b72-4b7d-8daa-0ecfb6288da6 to node k8s-agentpool-17591246-vmss000000 I1230 03:39:35.242069 1 azure_controller_common.go:255] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-46a1030f-1b72-4b7d-8daa-0ecfb6288da6 lun 0 to node k8s-agentpool-17591246-vmss000000, diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-du8j4fcj/providers/microsoft.compute/disks/pvc-46a1030f-1b72-4b7d-8daa-0ecfb6288da6:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-46a1030f-1b72-4b7d-8daa-0ecfb6288da6 false 0})] I1230 03:39:35.242120 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-du8j4fcj): vm(k8s-agentpool-17591246-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-du8j4fcj/providers/microsoft.compute/disks/pvc-46a1030f-1b72-4b7d-8daa-0ecfb6288da6:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-46a1030f-1b72-4b7d-8daa-0ecfb6288da6 false 0})]) I1230 03:39:37.078930 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-du8j4fcj): vm(k8s-agentpool-17591246-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-du8j4fcj/providers/microsoft.compute/disks/pvc-46a1030f-1b72-4b7d-8daa-0ecfb6288da6:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-46a1030f-1b72-4b7d-8daa-0ecfb6288da6 false 0})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I1230 03:39:47.174004 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-du8j4fcj, k8s-agentpool-17591246-vmss, k8s-agentpool-17591246-vmss000000) successfully I1230 03:39:47.174054 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool-17591246-vmss, kubetest-du8j4fcj, k8s-agentpool-17591246-vmss000000) for cacheKey(kubetest-du8j4fcj/k8s-agentpool-17591246-vmss) updated successfully I1230 03:39:47.174076 1 controllerserver.go:413] Attach operation successful: volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-46a1030f-1b72-4b7d-8daa-0ecfb6288da6 attached to node k8s-agentpool-17591246-vmss000000. I1230 03:39:47.174091 1 controllerserver.go:433] attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-46a1030f-1b72-4b7d-8daa-0ecfb6288da6 to node k8s-agentpool-17591246-vmss000000 successfully I1230 03:39:47.174146 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=12.127403479 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-du8j4fcj" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-46a1030f-1b72-4b7d-8daa-0ecfb6288da6" node="k8s-agentpool-17591246-vmss000000" result_code="succeeded" I1230 03:39:47.174185 1 utils.go:84] GRPC response: {"publish_context":{"LUN":"0"}} ... skipping 3 lines ... I1230 03:40:17.615142 1 azure_vmss_cache.go:327] refresh the cache of NonVmssUniformNodesCache in rg map[kubetest-du8j4fcj:{}] I1230 03:40:17.643000 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 12 I1230 03:40:17.643243 1 controllerserver.go:383] GetDiskLun returned: cannot find Lun for disk pvc-8f316563-7fb8-4a24-9556-50003d8e9889. Initiating attaching volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-8f316563-7fb8-4a24-9556-50003d8e9889 to node k8s-agentpool-17591246-vmss000001 (vmState Succeeded). I1230 03:40:17.643344 1 controllerserver.go:408] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-8f316563-7fb8-4a24-9556-50003d8e9889 to node k8s-agentpool-17591246-vmss000001 I1230 03:40:17.643532 1 azure_controller_common.go:255] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-8f316563-7fb8-4a24-9556-50003d8e9889 lun 0 to node k8s-agentpool-17591246-vmss000001, diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-du8j4fcj/providers/microsoft.compute/disks/pvc-8f316563-7fb8-4a24-9556-50003d8e9889:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-8f316563-7fb8-4a24-9556-50003d8e9889 false 0})] I1230 03:40:17.643648 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-du8j4fcj): vm(k8s-agentpool-17591246-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-du8j4fcj/providers/microsoft.compute/disks/pvc-8f316563-7fb8-4a24-9556-50003d8e9889:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-8f316563-7fb8-4a24-9556-50003d8e9889 false 0})]) I1230 03:40:17.810849 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-du8j4fcj): vm(k8s-agentpool-17591246-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-du8j4fcj/providers/microsoft.compute/disks/pvc-8f316563-7fb8-4a24-9556-50003d8e9889:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-8f316563-7fb8-4a24-9556-50003d8e9889 false 0})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I1230 03:40:27.935761 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-du8j4fcj, k8s-agentpool-17591246-vmss, k8s-agentpool-17591246-vmss000001) successfully I1230 03:40:27.935817 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool-17591246-vmss, kubetest-du8j4fcj, k8s-agentpool-17591246-vmss000001) for cacheKey(kubetest-du8j4fcj/k8s-agentpool-17591246-vmss) updated successfully I1230 03:40:27.935846 1 controllerserver.go:413] Attach operation successful: volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-8f316563-7fb8-4a24-9556-50003d8e9889 attached to node k8s-agentpool-17591246-vmss000001. I1230 03:40:27.936229 1 controllerserver.go:433] attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-8f316563-7fb8-4a24-9556-50003d8e9889 to node k8s-agentpool-17591246-vmss000001 successfully I1230 03:40:27.936423 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=10.321125584 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-du8j4fcj" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-8f316563-7fb8-4a24-9556-50003d8e9889" node="k8s-agentpool-17591246-vmss000001" result_code="succeeded" I1230 03:40:27.936448 1 utils.go:84] GRPC response: {"publish_context":{"LUN":"0"}} ... skipping 45 lines ... I1230 03:42:51.365274 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1219 I1230 03:42:51.365637 1 controllerserver.go:383] GetDiskLun returned: cannot find Lun for disk pvc-0deacc11-df28-4024-bfc0-35731042f7b3. Initiating attaching volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-0deacc11-df28-4024-bfc0-35731042f7b3 to node k8s-agentpool-17591246-vmss000000 (vmState Succeeded). I1230 03:42:51.365666 1 controllerserver.go:408] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-0deacc11-df28-4024-bfc0-35731042f7b3 to node k8s-agentpool-17591246-vmss000000 I1230 03:42:51.368762 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1236 I1230 03:42:51.368970 1 controllerserver.go:383] GetDiskLun returned: cannot find Lun for disk pvc-86840d9a-64f2-49a5-a419-199a7c1bacd7. Initiating attaching volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-86840d9a-64f2-49a5-a419-199a7c1bacd7 to node k8s-agentpool-17591246-vmss000000 (vmState Succeeded). I1230 03:42:51.369005 1 controllerserver.go:408] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-86840d9a-64f2-49a5-a419-199a7c1bacd7 to node k8s-agentpool-17591246-vmss000000 I1230 03:42:51.525339 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-du8j4fcj): vm(k8s-agentpool-17591246-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-du8j4fcj/providers/microsoft.compute/disks/pvc-90d96477-fef7-4017-aa34-035607fd6467:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-90d96477-fef7-4017-aa34-035607fd6467 false 0})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I1230 03:43:01.651443 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-du8j4fcj, k8s-agentpool-17591246-vmss, k8s-agentpool-17591246-vmss000000) successfully I1230 03:43:01.651488 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool-17591246-vmss, kubetest-du8j4fcj, k8s-agentpool-17591246-vmss000000) for cacheKey(kubetest-du8j4fcj/k8s-agentpool-17591246-vmss) updated successfully I1230 03:43:01.651527 1 controllerserver.go:413] Attach operation successful: volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-90d96477-fef7-4017-aa34-035607fd6467 attached to node k8s-agentpool-17591246-vmss000000. I1230 03:43:01.651546 1 controllerserver.go:433] attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-90d96477-fef7-4017-aa34-035607fd6467 to node k8s-agentpool-17591246-vmss000000 successfully I1230 03:43:01.651593 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=10.319949155 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-du8j4fcj" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-90d96477-fef7-4017-aa34-035607fd6467" node="k8s-agentpool-17591246-vmss000000" result_code="succeeded" I1230 03:43:01.651631 1 utils.go:84] GRPC response: {"publish_context":{"LUN":"0"}} I1230 03:43:01.651762 1 azure_controller_common.go:255] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-0deacc11-df28-4024-bfc0-35731042f7b3 lun 1 to node k8s-agentpool-17591246-vmss000000, diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-du8j4fcj/providers/microsoft.compute/disks/pvc-0deacc11-df28-4024-bfc0-35731042f7b3:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-0deacc11-df28-4024-bfc0-35731042f7b3 false 1}) /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-du8j4fcj/providers/microsoft.compute/disks/pvc-86840d9a-64f2-49a5-a419-199a7c1bacd7:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-86840d9a-64f2-49a5-a419-199a7c1bacd7 false 2})] I1230 03:43:01.651813 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-du8j4fcj): vm(k8s-agentpool-17591246-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-du8j4fcj/providers/microsoft.compute/disks/pvc-0deacc11-df28-4024-bfc0-35731042f7b3:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-0deacc11-df28-4024-bfc0-35731042f7b3 false 1}) /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-du8j4fcj/providers/microsoft.compute/disks/pvc-86840d9a-64f2-49a5-a419-199a7c1bacd7:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-86840d9a-64f2-49a5-a419-199a7c1bacd7 false 2})]) I1230 03:43:01.842639 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-du8j4fcj): vm(k8s-agentpool-17591246-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-du8j4fcj/providers/microsoft.compute/disks/pvc-0deacc11-df28-4024-bfc0-35731042f7b3:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-0deacc11-df28-4024-bfc0-35731042f7b3 false 1}) /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-du8j4fcj/providers/microsoft.compute/disks/pvc-86840d9a-64f2-49a5-a419-199a7c1bacd7:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-86840d9a-64f2-49a5-a419-199a7c1bacd7 false 2})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I1230 03:43:17.103426 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-du8j4fcj, k8s-agentpool-17591246-vmss, k8s-agentpool-17591246-vmss000000) successfully I1230 03:43:17.103471 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool-17591246-vmss, kubetest-du8j4fcj, k8s-agentpool-17591246-vmss000000) for cacheKey(kubetest-du8j4fcj/k8s-agentpool-17591246-vmss) updated successfully I1230 03:43:17.103524 1 controllerserver.go:413] Attach operation successful: volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-0deacc11-df28-4024-bfc0-35731042f7b3 attached to node k8s-agentpool-17591246-vmss000000. I1230 03:43:17.103541 1 controllerserver.go:433] attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-0deacc11-df28-4024-bfc0-35731042f7b3 to node k8s-agentpool-17591246-vmss000000 successfully I1230 03:43:17.103588 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=25.737929932 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-du8j4fcj" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-0deacc11-df28-4024-bfc0-35731042f7b3" node="k8s-agentpool-17591246-vmss000000" result_code="succeeded" I1230 03:43:17.103613 1 utils.go:84] GRPC response: {"publish_context":{"LUN":"1"}} ... skipping 57 lines ... I1230 03:44:39.513338 1 utils.go:78] GRPC request: {"node_id":"k8s-agentpool-17591246-vmss000000","volume_capability":{"AccessType":{"Mount":{"mount_flags":["barrier=1","acl"]}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-0aa3b982-cfa3-4e28-95ee-14463aa95879","csi.storage.k8s.io/pvc/name":"pvc-azuredisk-volume-tester-xfqpc-0","csi.storage.k8s.io/pvc/namespace":"azuredisk-1387","requestedsizegib":"10","skuName":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1672369721109-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-0aa3b982-cfa3-4e28-95ee-14463aa95879"} I1230 03:44:39.566472 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1248 I1230 03:44:39.567006 1 controllerserver.go:383] GetDiskLun returned: cannot find Lun for disk pvc-0aa3b982-cfa3-4e28-95ee-14463aa95879. Initiating attaching volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-0aa3b982-cfa3-4e28-95ee-14463aa95879 to node k8s-agentpool-17591246-vmss000000 (vmState Succeeded). I1230 03:44:39.567088 1 controllerserver.go:408] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-0aa3b982-cfa3-4e28-95ee-14463aa95879 to node k8s-agentpool-17591246-vmss000000 I1230 03:44:39.567130 1 azure_controller_common.go:255] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-0aa3b982-cfa3-4e28-95ee-14463aa95879 lun 0 to node k8s-agentpool-17591246-vmss000000, diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-du8j4fcj/providers/microsoft.compute/disks/pvc-0aa3b982-cfa3-4e28-95ee-14463aa95879:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-0aa3b982-cfa3-4e28-95ee-14463aa95879 false 0})] I1230 03:44:39.567177 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-du8j4fcj): vm(k8s-agentpool-17591246-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-du8j4fcj/providers/microsoft.compute/disks/pvc-0aa3b982-cfa3-4e28-95ee-14463aa95879:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-0aa3b982-cfa3-4e28-95ee-14463aa95879 false 0})]) I1230 03:44:39.799043 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-du8j4fcj): vm(k8s-agentpool-17591246-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-du8j4fcj/providers/microsoft.compute/disks/pvc-0aa3b982-cfa3-4e28-95ee-14463aa95879:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-0aa3b982-cfa3-4e28-95ee-14463aa95879 false 0})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I1230 03:44:54.991958 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-du8j4fcj, k8s-agentpool-17591246-vmss, k8s-agentpool-17591246-vmss000000) successfully I1230 03:44:54.992006 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool-17591246-vmss, kubetest-du8j4fcj, k8s-agentpool-17591246-vmss000000) for cacheKey(kubetest-du8j4fcj/k8s-agentpool-17591246-vmss) updated successfully I1230 03:44:54.992034 1 controllerserver.go:413] Attach operation successful: volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-0aa3b982-cfa3-4e28-95ee-14463aa95879 attached to node k8s-agentpool-17591246-vmss000000. I1230 03:44:54.992050 1 controllerserver.go:433] attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-0aa3b982-cfa3-4e28-95ee-14463aa95879 to node k8s-agentpool-17591246-vmss000000 successfully I1230 03:44:54.992099 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=15.425086727 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-du8j4fcj" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-0aa3b982-cfa3-4e28-95ee-14463aa95879" node="k8s-agentpool-17591246-vmss000000" result_code="succeeded" I1230 03:44:54.992131 1 utils.go:84] GRPC response: {"publish_context":{"LUN":"0"}} ... skipping 14 lines ... I1230 03:47:48.444217 1 utils.go:78] GRPC request: {"node_id":"k8s-agentpool-17591246-vmss000000","volume_capability":{"AccessType":{"Mount":{"mount_flags":["barrier=1","acl"]}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-0aa3b982-cfa3-4e28-95ee-14463aa95879","csi.storage.k8s.io/pvc/name":"pvc-azuredisk-volume-tester-xfqpc-0","csi.storage.k8s.io/pvc/namespace":"azuredisk-1387","requestedsizegib":"10","skuName":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1672369721109-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-0aa3b982-cfa3-4e28-95ee-14463aa95879"} I1230 03:47:48.474098 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1248 I1230 03:47:48.474418 1 controllerserver.go:383] GetDiskLun returned: cannot find Lun for disk pvc-0aa3b982-cfa3-4e28-95ee-14463aa95879. Initiating attaching volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-0aa3b982-cfa3-4e28-95ee-14463aa95879 to node k8s-agentpool-17591246-vmss000000 (vmState Succeeded). I1230 03:47:48.474450 1 controllerserver.go:408] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-0aa3b982-cfa3-4e28-95ee-14463aa95879 to node k8s-agentpool-17591246-vmss000000 I1230 03:47:48.474490 1 azure_controller_common.go:255] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-0aa3b982-cfa3-4e28-95ee-14463aa95879 lun 0 to node k8s-agentpool-17591246-vmss000000, diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-du8j4fcj/providers/microsoft.compute/disks/pvc-0aa3b982-cfa3-4e28-95ee-14463aa95879:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-0aa3b982-cfa3-4e28-95ee-14463aa95879 false 0})] I1230 03:47:48.474533 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-du8j4fcj): vm(k8s-agentpool-17591246-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-du8j4fcj/providers/microsoft.compute/disks/pvc-0aa3b982-cfa3-4e28-95ee-14463aa95879:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-0aa3b982-cfa3-4e28-95ee-14463aa95879 false 0})]) I1230 03:47:48.645089 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-du8j4fcj): vm(k8s-agentpool-17591246-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-du8j4fcj/providers/microsoft.compute/disks/pvc-0aa3b982-cfa3-4e28-95ee-14463aa95879:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-0aa3b982-cfa3-4e28-95ee-14463aa95879 false 0})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I1230 03:47:58.794946 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-du8j4fcj, k8s-agentpool-17591246-vmss, k8s-agentpool-17591246-vmss000000) successfully I1230 03:47:58.794988 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool-17591246-vmss, kubetest-du8j4fcj, k8s-agentpool-17591246-vmss000000) for cacheKey(kubetest-du8j4fcj/k8s-agentpool-17591246-vmss) updated successfully I1230 03:47:58.795009 1 controllerserver.go:413] Attach operation successful: volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-0aa3b982-cfa3-4e28-95ee-14463aa95879 attached to node k8s-agentpool-17591246-vmss000000. I1230 03:47:58.795024 1 controllerserver.go:433] attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-0aa3b982-cfa3-4e28-95ee-14463aa95879 to node k8s-agentpool-17591246-vmss000000 successfully I1230 03:47:58.795085 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=10.320652954 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-du8j4fcj" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-0aa3b982-cfa3-4e28-95ee-14463aa95879" node="k8s-agentpool-17591246-vmss000000" result_code="succeeded" I1230 03:47:58.795113 1 utils.go:84] GRPC response: {"publish_context":{"LUN":"0"}} I1230 03:48:24.810956 1 utils.go:77] GRPC call: /csi.v1.Controller/ControllerPublishVolume I1230 03:48:24.810983 1 utils.go:78] GRPC request: {"node_id":"k8s-agentpool-17591246-vmss000000","volume_capability":{"AccessType":{"Mount":{"mount_flags":["barrier=1","acl"]}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-8600f1cf-f2c8-45ef-93cf-633b1f30cd8b","csi.storage.k8s.io/pvc/name":"pvc-48rjk","csi.storage.k8s.io/pvc/namespace":"azuredisk-4801","requestedsizegib":"10","skuName":"StandardSSD_ZRS","storage.kubernetes.io/csiProvisionerIdentity":"1672369721109-8081-disk.csi.azure.com","tags":"disk=test"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-8600f1cf-f2c8-45ef-93cf-633b1f30cd8b"} I1230 03:48:24.835120 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1214 I1230 03:48:24.835601 1 controllerserver.go:383] GetDiskLun returned: cannot find Lun for disk pvc-8600f1cf-f2c8-45ef-93cf-633b1f30cd8b. Initiating attaching volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-8600f1cf-f2c8-45ef-93cf-633b1f30cd8b to node k8s-agentpool-17591246-vmss000000 (vmState Succeeded). I1230 03:48:24.835642 1 controllerserver.go:408] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-8600f1cf-f2c8-45ef-93cf-633b1f30cd8b to node k8s-agentpool-17591246-vmss000000 I1230 03:48:24.835700 1 azure_controller_common.go:255] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-8600f1cf-f2c8-45ef-93cf-633b1f30cd8b lun 1 to node k8s-agentpool-17591246-vmss000000, diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-du8j4fcj/providers/microsoft.compute/disks/pvc-8600f1cf-f2c8-45ef-93cf-633b1f30cd8b:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-8600f1cf-f2c8-45ef-93cf-633b1f30cd8b false 1})] I1230 03:48:24.835748 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-du8j4fcj): vm(k8s-agentpool-17591246-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-du8j4fcj/providers/microsoft.compute/disks/pvc-8600f1cf-f2c8-45ef-93cf-633b1f30cd8b:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-8600f1cf-f2c8-45ef-93cf-633b1f30cd8b false 1})]) I1230 03:48:24.985575 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-du8j4fcj): vm(k8s-agentpool-17591246-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-du8j4fcj/providers/microsoft.compute/disks/pvc-8600f1cf-f2c8-45ef-93cf-633b1f30cd8b:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-8600f1cf-f2c8-45ef-93cf-633b1f30cd8b false 1})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I1230 03:48:40.114425 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-du8j4fcj, k8s-agentpool-17591246-vmss, k8s-agentpool-17591246-vmss000000) successfully I1230 03:48:40.114472 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool-17591246-vmss, kubetest-du8j4fcj, k8s-agentpool-17591246-vmss000000) for cacheKey(kubetest-du8j4fcj/k8s-agentpool-17591246-vmss) updated successfully I1230 03:48:40.114526 1 controllerserver.go:413] Attach operation successful: volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-8600f1cf-f2c8-45ef-93cf-633b1f30cd8b attached to node k8s-agentpool-17591246-vmss000000. I1230 03:48:40.114636 1 controllerserver.go:433] attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-8600f1cf-f2c8-45ef-93cf-633b1f30cd8b to node k8s-agentpool-17591246-vmss000000 successfully I1230 03:48:40.114713 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=15.279116846 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-du8j4fcj" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-8600f1cf-f2c8-45ef-93cf-633b1f30cd8b" node="k8s-agentpool-17591246-vmss000000" result_code="succeeded" I1230 03:48:40.114731 1 utils.go:84] GRPC response: {"publish_context":{"LUN":"1"}} ... skipping 41 lines ... I1230 03:49:44.701321 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1193 I1230 03:49:44.747479 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 24988 I1230 03:49:44.751029 1 controllerserver.go:383] GetDiskLun returned: cannot find Lun for disk pvc-be55c18f-6945-4445-8f93-5bf7b43d80ce. Initiating attaching volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-be55c18f-6945-4445-8f93-5bf7b43d80ce to node k8s-agentpool-17591246-vmss000000 (vmState Succeeded). I1230 03:49:44.751080 1 controllerserver.go:408] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-be55c18f-6945-4445-8f93-5bf7b43d80ce to node k8s-agentpool-17591246-vmss000000 I1230 03:49:44.751119 1 azure_controller_common.go:255] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-be55c18f-6945-4445-8f93-5bf7b43d80ce lun 0 to node k8s-agentpool-17591246-vmss000000, diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-du8j4fcj/providers/microsoft.compute/disks/pvc-be55c18f-6945-4445-8f93-5bf7b43d80ce:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-be55c18f-6945-4445-8f93-5bf7b43d80ce false 0})] I1230 03:49:44.751175 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-du8j4fcj): vm(k8s-agentpool-17591246-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-du8j4fcj/providers/microsoft.compute/disks/pvc-be55c18f-6945-4445-8f93-5bf7b43d80ce:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-be55c18f-6945-4445-8f93-5bf7b43d80ce false 0})]) I1230 03:49:44.906354 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-du8j4fcj): vm(k8s-agentpool-17591246-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-du8j4fcj/providers/microsoft.compute/disks/pvc-be55c18f-6945-4445-8f93-5bf7b43d80ce:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-be55c18f-6945-4445-8f93-5bf7b43d80ce false 0})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I1230 03:50:00.113112 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-du8j4fcj, k8s-agentpool-17591246-vmss, k8s-agentpool-17591246-vmss000000) successfully I1230 03:50:00.113150 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool-17591246-vmss, kubetest-du8j4fcj, k8s-agentpool-17591246-vmss000000) for cacheKey(kubetest-du8j4fcj/k8s-agentpool-17591246-vmss) updated successfully I1230 03:50:00.113172 1 controllerserver.go:413] Attach operation successful: volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-be55c18f-6945-4445-8f93-5bf7b43d80ce attached to node k8s-agentpool-17591246-vmss000000. I1230 03:50:00.113188 1 controllerserver.go:433] attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-be55c18f-6945-4445-8f93-5bf7b43d80ce to node k8s-agentpool-17591246-vmss000000 successfully I1230 03:50:00.113232 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=15.411434597 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-du8j4fcj" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-be55c18f-6945-4445-8f93-5bf7b43d80ce" node="k8s-agentpool-17591246-vmss000000" result_code="succeeded" I1230 03:50:00.113314 1 utils.go:84] GRPC response: {"publish_context":{"LUN":"0"}} ... skipping 22 lines ... I1230 03:51:37.728912 1 utils.go:78] GRPC request: {"node_id":"k8s-agentpool-17591246-vmss000000","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-712b830f-11b3-4554-97b3-174b9bac7661","csi.storage.k8s.io/pvc/name":"pvc-azuredisk-volume-tester-gltms-0","csi.storage.k8s.io/pvc/namespace":"azuredisk-1166","requestedsizegib":"10","skuName":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1672369721109-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-712b830f-11b3-4554-97b3-174b9bac7661"} I1230 03:51:37.768247 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1248 I1230 03:51:37.768600 1 controllerserver.go:383] GetDiskLun returned: cannot find Lun for disk pvc-712b830f-11b3-4554-97b3-174b9bac7661. Initiating attaching volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-712b830f-11b3-4554-97b3-174b9bac7661 to node k8s-agentpool-17591246-vmss000000 (vmState Succeeded). I1230 03:51:37.768645 1 controllerserver.go:408] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-712b830f-11b3-4554-97b3-174b9bac7661 to node k8s-agentpool-17591246-vmss000000 I1230 03:51:37.768687 1 azure_controller_common.go:255] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-712b830f-11b3-4554-97b3-174b9bac7661 lun 0 to node k8s-agentpool-17591246-vmss000000, diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-du8j4fcj/providers/microsoft.compute/disks/pvc-712b830f-11b3-4554-97b3-174b9bac7661:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-712b830f-11b3-4554-97b3-174b9bac7661 false 0})] I1230 03:51:37.768992 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-du8j4fcj): vm(k8s-agentpool-17591246-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-du8j4fcj/providers/microsoft.compute/disks/pvc-712b830f-11b3-4554-97b3-174b9bac7661:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-712b830f-11b3-4554-97b3-174b9bac7661 false 0})]) I1230 03:51:37.937644 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-du8j4fcj): vm(k8s-agentpool-17591246-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-du8j4fcj/providers/microsoft.compute/disks/pvc-712b830f-11b3-4554-97b3-174b9bac7661:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-712b830f-11b3-4554-97b3-174b9bac7661 false 0})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I1230 03:51:48.042121 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-du8j4fcj, k8s-agentpool-17591246-vmss, k8s-agentpool-17591246-vmss000000) successfully I1230 03:51:48.042165 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool-17591246-vmss, kubetest-du8j4fcj, k8s-agentpool-17591246-vmss000000) for cacheKey(kubetest-du8j4fcj/k8s-agentpool-17591246-vmss) updated successfully I1230 03:51:48.042187 1 controllerserver.go:413] Attach operation successful: volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-712b830f-11b3-4554-97b3-174b9bac7661 attached to node k8s-agentpool-17591246-vmss000000. I1230 03:51:48.042202 1 controllerserver.go:433] attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-712b830f-11b3-4554-97b3-174b9bac7661 to node k8s-agentpool-17591246-vmss000000 successfully I1230 03:51:48.042248 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=10.273654729 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-du8j4fcj" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-712b830f-11b3-4554-97b3-174b9bac7661" node="k8s-agentpool-17591246-vmss000000" result_code="succeeded" I1230 03:51:48.042273 1 utils.go:84] GRPC response: {"publish_context":{"LUN":"0"}} ... skipping 35 lines ... I1230 03:53:05.130675 1 utils.go:78] GRPC request: {"node_id":"k8s-agentpool-17591246-vmss000000","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-712b830f-11b3-4554-97b3-174b9bac7661","csi.storage.k8s.io/pvc/name":"pvc-azuredisk-volume-tester-gltms-0","csi.storage.k8s.io/pvc/namespace":"azuredisk-1166","requestedsizegib":"10","skuName":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1672369721109-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-712b830f-11b3-4554-97b3-174b9bac7661"} I1230 03:53:05.158493 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1248 I1230 03:53:05.159027 1 controllerserver.go:383] GetDiskLun returned: cannot find Lun for disk pvc-712b830f-11b3-4554-97b3-174b9bac7661. Initiating attaching volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-712b830f-11b3-4554-97b3-174b9bac7661 to node k8s-agentpool-17591246-vmss000000 (vmState Succeeded). I1230 03:53:05.159063 1 controllerserver.go:408] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-712b830f-11b3-4554-97b3-174b9bac7661 to node k8s-agentpool-17591246-vmss000000 I1230 03:53:05.159212 1 azure_controller_common.go:255] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-712b830f-11b3-4554-97b3-174b9bac7661 lun 0 to node k8s-agentpool-17591246-vmss000000, diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-du8j4fcj/providers/microsoft.compute/disks/pvc-712b830f-11b3-4554-97b3-174b9bac7661:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-712b830f-11b3-4554-97b3-174b9bac7661 false 0})] I1230 03:53:05.159309 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-du8j4fcj): vm(k8s-agentpool-17591246-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-du8j4fcj/providers/microsoft.compute/disks/pvc-712b830f-11b3-4554-97b3-174b9bac7661:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-712b830f-11b3-4554-97b3-174b9bac7661 false 0})]) I1230 03:53:05.302746 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-du8j4fcj): vm(k8s-agentpool-17591246-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-du8j4fcj/providers/microsoft.compute/disks/pvc-712b830f-11b3-4554-97b3-174b9bac7661:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-712b830f-11b3-4554-97b3-174b9bac7661 false 0})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I1230 03:53:15.445778 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-du8j4fcj, k8s-agentpool-17591246-vmss, k8s-agentpool-17591246-vmss000000) successfully I1230 03:53:15.445822 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool-17591246-vmss, kubetest-du8j4fcj, k8s-agentpool-17591246-vmss000000) for cacheKey(kubetest-du8j4fcj/k8s-agentpool-17591246-vmss) updated successfully I1230 03:53:15.445846 1 controllerserver.go:413] Attach operation successful: volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-712b830f-11b3-4554-97b3-174b9bac7661 attached to node k8s-agentpool-17591246-vmss000000. I1230 03:53:15.445865 1 controllerserver.go:433] attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-712b830f-11b3-4554-97b3-174b9bac7661 to node k8s-agentpool-17591246-vmss000000 successfully I1230 03:53:15.445912 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=10.286880378 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-du8j4fcj" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-712b830f-11b3-4554-97b3-174b9bac7661" node="k8s-agentpool-17591246-vmss000000" result_code="succeeded" I1230 03:53:15.445937 1 utils.go:84] GRPC response: {"publish_context":{"LUN":"0"}} I1230 03:53:55.725119 1 utils.go:77] GRPC call: /csi.v1.Controller/ControllerPublishVolume I1230 03:53:55.725170 1 utils.go:78] GRPC request: {"node_id":"k8s-agentpool-17591246-vmss000000","volume_capability":{"AccessType":{"Mount":{"mount_flags":["barrier=1","acl"]}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-c257fb5d-fc30-4d85-a6a9-6e41b1b9a379","csi.storage.k8s.io/pvc/name":"pvc-fmd9s","csi.storage.k8s.io/pvc/namespace":"azuredisk-783","requestedsizegib":"10","skuName":"StandardSSD_ZRS","storage.kubernetes.io/csiProvisionerIdentity":"1672369721109-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-c257fb5d-fc30-4d85-a6a9-6e41b1b9a379"} I1230 03:53:55.759200 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1192 I1230 03:53:55.759595 1 controllerserver.go:383] GetDiskLun returned: cannot find Lun for disk pvc-c257fb5d-fc30-4d85-a6a9-6e41b1b9a379. Initiating attaching volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-c257fb5d-fc30-4d85-a6a9-6e41b1b9a379 to node k8s-agentpool-17591246-vmss000000 (vmState Succeeded). I1230 03:53:55.759651 1 controllerserver.go:408] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-c257fb5d-fc30-4d85-a6a9-6e41b1b9a379 to node k8s-agentpool-17591246-vmss000000 I1230 03:53:55.759690 1 azure_controller_common.go:255] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-c257fb5d-fc30-4d85-a6a9-6e41b1b9a379 lun 1 to node k8s-agentpool-17591246-vmss000000, diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-du8j4fcj/providers/microsoft.compute/disks/pvc-c257fb5d-fc30-4d85-a6a9-6e41b1b9a379:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-c257fb5d-fc30-4d85-a6a9-6e41b1b9a379 false 1})] I1230 03:53:55.759760 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-du8j4fcj): vm(k8s-agentpool-17591246-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-du8j4fcj/providers/microsoft.compute/disks/pvc-c257fb5d-fc30-4d85-a6a9-6e41b1b9a379:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-c257fb5d-fc30-4d85-a6a9-6e41b1b9a379 false 1})]) I1230 03:53:55.928326 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-du8j4fcj): vm(k8s-agentpool-17591246-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-du8j4fcj/providers/microsoft.compute/disks/pvc-c257fb5d-fc30-4d85-a6a9-6e41b1b9a379:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-c257fb5d-fc30-4d85-a6a9-6e41b1b9a379 false 1})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I1230 03:54:06.032369 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-du8j4fcj, k8s-agentpool-17591246-vmss, k8s-agentpool-17591246-vmss000000) successfully I1230 03:54:06.032416 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool-17591246-vmss, kubetest-du8j4fcj, k8s-agentpool-17591246-vmss000000) for cacheKey(kubetest-du8j4fcj/k8s-agentpool-17591246-vmss) updated successfully I1230 03:54:06.032435 1 controllerserver.go:413] Attach operation successful: volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-c257fb5d-fc30-4d85-a6a9-6e41b1b9a379 attached to node k8s-agentpool-17591246-vmss000000. I1230 03:54:06.032446 1 controllerserver.go:433] attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-c257fb5d-fc30-4d85-a6a9-6e41b1b9a379 to node k8s-agentpool-17591246-vmss000000 successfully I1230 03:54:06.032489 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=10.272917791 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-du8j4fcj" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-c257fb5d-fc30-4d85-a6a9-6e41b1b9a379" node="k8s-agentpool-17591246-vmss000000" result_code="succeeded" I1230 03:54:06.032509 1 utils.go:84] GRPC response: {"publish_context":{"LUN":"1"}} ... skipping 41 lines ... I1230 03:55:05.249469 1 utils.go:78] GRPC request: {"node_id":"k8s-agentpool-17591246-vmss000001","volume_capability":{"AccessType":{"Mount":{"mount_flags":["barrier=1","acl"]}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-c257fb5d-fc30-4d85-a6a9-6e41b1b9a379","csi.storage.k8s.io/pvc/name":"pvc-fmd9s","csi.storage.k8s.io/pvc/namespace":"azuredisk-783","requestedsizegib":"10","skuName":"StandardSSD_ZRS","storage.kubernetes.io/csiProvisionerIdentity":"1672369721109-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-c257fb5d-fc30-4d85-a6a9-6e41b1b9a379"} I1230 03:55:05.299607 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1192 I1230 03:55:05.299969 1 controllerserver.go:383] GetDiskLun returned: cannot find Lun for disk pvc-c257fb5d-fc30-4d85-a6a9-6e41b1b9a379. Initiating attaching volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-c257fb5d-fc30-4d85-a6a9-6e41b1b9a379 to node k8s-agentpool-17591246-vmss000001 (vmState Succeeded). I1230 03:55:05.300015 1 controllerserver.go:408] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-c257fb5d-fc30-4d85-a6a9-6e41b1b9a379 to node k8s-agentpool-17591246-vmss000001 I1230 03:55:05.300053 1 azure_controller_common.go:255] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-c257fb5d-fc30-4d85-a6a9-6e41b1b9a379 lun 0 to node k8s-agentpool-17591246-vmss000001, diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-du8j4fcj/providers/microsoft.compute/disks/pvc-c257fb5d-fc30-4d85-a6a9-6e41b1b9a379:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-c257fb5d-fc30-4d85-a6a9-6e41b1b9a379 false 0})] I1230 03:55:05.300154 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-du8j4fcj): vm(k8s-agentpool-17591246-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-du8j4fcj/providers/microsoft.compute/disks/pvc-c257fb5d-fc30-4d85-a6a9-6e41b1b9a379:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-c257fb5d-fc30-4d85-a6a9-6e41b1b9a379 false 0})]) I1230 03:55:05.446420 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-du8j4fcj): vm(k8s-agentpool-17591246-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-du8j4fcj/providers/microsoft.compute/disks/pvc-c257fb5d-fc30-4d85-a6a9-6e41b1b9a379:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-c257fb5d-fc30-4d85-a6a9-6e41b1b9a379 false 0})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I1230 03:55:10.306375 1 azure_controller_vmss.go:252] azureDisk - update(kubetest-du8j4fcj): vm(k8s-agentpool-17591246-vmss000000) - detach disk(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-du8j4fcj/providers/microsoft.compute/disks/pvc-712b830f-11b3-4554-97b3-174b9bac7661:pvc-712b830f-11b3-4554-97b3-174b9bac7661]) returned with <nil> I1230 03:55:10.306468 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-du8j4fcj, k8s-agentpool-17591246-vmss, k8s-agentpool-17591246-vmss000000) successfully I1230 03:55:10.306488 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool-17591246-vmss, kubetest-du8j4fcj, k8s-agentpool-17591246-vmss000000) for cacheKey(kubetest-du8j4fcj/k8s-agentpool-17591246-vmss) updated successfully I1230 03:55:10.306504 1 azure_controller_common.go:422] azureDisk - detach disk(pvc-712b830f-11b3-4554-97b3-174b9bac7661, /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-712b830f-11b3-4554-97b3-174b9bac7661) succeeded I1230 03:55:10.306518 1 controllerserver.go:480] detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-712b830f-11b3-4554-97b3-174b9bac7661 from node k8s-agentpool-17591246-vmss000000 successfully I1230 03:55:10.306563 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=20.518873888 request="azuredisk_csi_driver_controller_unpublish_volume" resource_group="kubetest-du8j4fcj" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-712b830f-11b3-4554-97b3-174b9bac7661" node="k8s-agentpool-17591246-vmss000000" result_code="succeeded" ... skipping 36 lines ... I1230 03:56:36.855273 1 azure_vmss_cache.go:327] refresh the cache of NonVmssUniformNodesCache in rg map[kubetest-du8j4fcj:{}] I1230 03:56:36.880670 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 12 I1230 03:56:36.881074 1 controllerserver.go:383] GetDiskLun returned: cannot find Lun for disk pvc-270ec856-ccdf-49b4-acde-cda596efc335. Initiating attaching volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-270ec856-ccdf-49b4-acde-cda596efc335 to node k8s-agentpool-17591246-vmss000000 (vmState Succeeded). I1230 03:56:36.881156 1 controllerserver.go:408] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-270ec856-ccdf-49b4-acde-cda596efc335 to node k8s-agentpool-17591246-vmss000000 I1230 03:56:36.881294 1 azure_controller_common.go:255] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-270ec856-ccdf-49b4-acde-cda596efc335 lun 0 to node k8s-agentpool-17591246-vmss000000, diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-du8j4fcj/providers/microsoft.compute/disks/pvc-270ec856-ccdf-49b4-acde-cda596efc335:%!s(*provider.AttachDiskOptions=&{None pvc-270ec856-ccdf-49b4-acde-cda596efc335 false 0})] I1230 03:56:36.881545 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-du8j4fcj): vm(k8s-agentpool-17591246-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-du8j4fcj/providers/microsoft.compute/disks/pvc-270ec856-ccdf-49b4-acde-cda596efc335:%!s(*provider.AttachDiskOptions=&{None pvc-270ec856-ccdf-49b4-acde-cda596efc335 false 0})]) I1230 03:56:37.090853 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-du8j4fcj): vm(k8s-agentpool-17591246-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-du8j4fcj/providers/microsoft.compute/disks/pvc-270ec856-ccdf-49b4-acde-cda596efc335:%!s(*provider.AttachDiskOptions=&{None pvc-270ec856-ccdf-49b4-acde-cda596efc335 false 0})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I1230 03:56:38.419312 1 utils.go:77] GRPC call: /csi.v1.Controller/ControllerPublishVolume I1230 03:56:38.419615 1 utils.go:78] GRPC request: {"node_id":"k8s-agentpool-17591246-vmss000001","volume_capability":{"AccessType":{"Block":{}},"access_mode":{"mode":5}},"volume_context":{"cachingmode":"None","csi.storage.k8s.io/pv/name":"pvc-270ec856-ccdf-49b4-acde-cda596efc335","csi.storage.k8s.io/pvc/name":"pvc-48khc","csi.storage.k8s.io/pvc/namespace":"azuredisk-7920","maxshares":"2","requestedsizegib":"10","skuname":"StandardSSD_ZRS","storage.kubernetes.io/csiProvisionerIdentity":"1672369721109-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-270ec856-ccdf-49b4-acde-cda596efc335"} I1230 03:56:38.444586 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1692 I1230 03:56:38.444926 1 controllerserver.go:383] GetDiskLun returned: cannot find Lun for disk pvc-270ec856-ccdf-49b4-acde-cda596efc335. Initiating attaching volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-270ec856-ccdf-49b4-acde-cda596efc335 to node k8s-agentpool-17591246-vmss000001 (vmState Succeeded). I1230 03:56:38.444959 1 controllerserver.go:408] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-270ec856-ccdf-49b4-acde-cda596efc335 to node k8s-agentpool-17591246-vmss000001 I1230 03:56:38.444998 1 azure_controller_common.go:255] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-270ec856-ccdf-49b4-acde-cda596efc335 lun 0 to node k8s-agentpool-17591246-vmss000001, diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-du8j4fcj/providers/microsoft.compute/disks/pvc-270ec856-ccdf-49b4-acde-cda596efc335:%!s(*provider.AttachDiskOptions=&{None pvc-270ec856-ccdf-49b4-acde-cda596efc335 false 0})] I1230 03:56:38.445038 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-du8j4fcj): vm(k8s-agentpool-17591246-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-du8j4fcj/providers/microsoft.compute/disks/pvc-270ec856-ccdf-49b4-acde-cda596efc335:%!s(*provider.AttachDiskOptions=&{None pvc-270ec856-ccdf-49b4-acde-cda596efc335 false 0})]) I1230 03:56:38.616469 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-du8j4fcj): vm(k8s-agentpool-17591246-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-du8j4fcj/providers/microsoft.compute/disks/pvc-270ec856-ccdf-49b4-acde-cda596efc335:%!s(*provider.AttachDiskOptions=&{None pvc-270ec856-ccdf-49b4-acde-cda596efc335 false 0})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I1230 03:57:07.430890 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-du8j4fcj, k8s-agentpool-17591246-vmss, k8s-agentpool-17591246-vmss000000) successfully I1230 03:57:07.431599 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool-17591246-vmss, kubetest-du8j4fcj, k8s-agentpool-17591246-vmss000000) for cacheKey(kubetest-du8j4fcj/k8s-agentpool-17591246-vmss) updated successfully I1230 03:57:07.433747 1 controllerserver.go:413] Attach operation successful: volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-270ec856-ccdf-49b4-acde-cda596efc335 attached to node k8s-agentpool-17591246-vmss000000. I1230 03:57:07.433801 1 controllerserver.go:433] attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-270ec856-ccdf-49b4-acde-cda596efc335 to node k8s-agentpool-17591246-vmss000000 successfully I1230 03:57:07.433881 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=30.578558321 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-du8j4fcj" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-270ec856-ccdf-49b4-acde-cda596efc335" node="k8s-agentpool-17591246-vmss000000" result_code="succeeded" I1230 03:57:07.433909 1 utils.go:84] GRPC response: {"publish_context":{"LUN":"0"}} ... skipping 46 lines ... I1230 03:59:11.849614 1 utils.go:78] GRPC request: {"node_id":"k8s-agentpool-17591246-vmss000000","volume_capability":{"AccessType":{"Mount":{"mount_flags":["barrier=1","acl"]}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-ac778d07-712a-4d77-9adc-570701d094fc","csi.storage.k8s.io/pvc/name":"pvc-bqvnz","csi.storage.k8s.io/pvc/namespace":"azuredisk-1092","device-setting/device/queue_depth":"17","device-setting/queue/max_sectors_kb":"211","device-setting/queue/nr_requests":"44","device-setting/queue/read_ahead_kb":"256","device-setting/queue/rotational":"0","device-setting/queue/scheduler":"none","device-setting/queue/wbt_lat_usec":"0","perfProfile":"advanced","requestedsizegib":"10","skuname":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1672369721109-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-ac778d07-712a-4d77-9adc-570701d094fc"} I1230 03:59:11.879886 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1222 I1230 03:59:11.880314 1 controllerserver.go:383] GetDiskLun returned: cannot find Lun for disk pvc-ac778d07-712a-4d77-9adc-570701d094fc. Initiating attaching volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-ac778d07-712a-4d77-9adc-570701d094fc to node k8s-agentpool-17591246-vmss000000 (vmState Succeeded). I1230 03:59:11.880342 1 controllerserver.go:408] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-ac778d07-712a-4d77-9adc-570701d094fc to node k8s-agentpool-17591246-vmss000000 I1230 03:59:11.880377 1 azure_controller_common.go:255] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-ac778d07-712a-4d77-9adc-570701d094fc lun 0 to node k8s-agentpool-17591246-vmss000000, diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-du8j4fcj/providers/microsoft.compute/disks/pvc-ac778d07-712a-4d77-9adc-570701d094fc:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-ac778d07-712a-4d77-9adc-570701d094fc false 0})] I1230 03:59:11.880415 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-du8j4fcj): vm(k8s-agentpool-17591246-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-du8j4fcj/providers/microsoft.compute/disks/pvc-ac778d07-712a-4d77-9adc-570701d094fc:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-ac778d07-712a-4d77-9adc-570701d094fc false 0})]) I1230 03:59:12.049463 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-du8j4fcj): vm(k8s-agentpool-17591246-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-du8j4fcj/providers/microsoft.compute/disks/pvc-ac778d07-712a-4d77-9adc-570701d094fc:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-ac778d07-712a-4d77-9adc-570701d094fc false 0})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I1230 03:59:22.154144 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-du8j4fcj, k8s-agentpool-17591246-vmss, k8s-agentpool-17591246-vmss000000) successfully I1230 03:59:22.154272 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool-17591246-vmss, kubetest-du8j4fcj, k8s-agentpool-17591246-vmss000000) for cacheKey(kubetest-du8j4fcj/k8s-agentpool-17591246-vmss) updated successfully I1230 03:59:22.154379 1 controllerserver.go:413] Attach operation successful: volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-ac778d07-712a-4d77-9adc-570701d094fc attached to node k8s-agentpool-17591246-vmss000000. I1230 03:59:22.154455 1 controllerserver.go:433] attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-ac778d07-712a-4d77-9adc-570701d094fc to node k8s-agentpool-17591246-vmss000000 successfully I1230 03:59:22.154671 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=10.274299403 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-du8j4fcj" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-ac778d07-712a-4d77-9adc-570701d094fc" node="k8s-agentpool-17591246-vmss000000" result_code="succeeded" I1230 03:59:22.154780 1 utils.go:84] GRPC response: {"publish_context":{"LUN":"0"}} ... skipping 15 lines ... I1230 04:00:19.383503 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1219 I1230 04:00:19.503263 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 24988 I1230 04:00:19.506578 1 controllerserver.go:383] GetDiskLun returned: cannot find Lun for disk pvc-6943f4cd-3c77-4c0d-95ea-786bd62531f4. Initiating attaching volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-6943f4cd-3c77-4c0d-95ea-786bd62531f4 to node k8s-agentpool-17591246-vmss000000 (vmState Succeeded). I1230 04:00:19.506690 1 controllerserver.go:408] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-6943f4cd-3c77-4c0d-95ea-786bd62531f4 to node k8s-agentpool-17591246-vmss000000 I1230 04:00:19.506827 1 azure_controller_common.go:255] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-6943f4cd-3c77-4c0d-95ea-786bd62531f4 lun 0 to node k8s-agentpool-17591246-vmss000000, diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-du8j4fcj/providers/microsoft.compute/disks/pvc-6943f4cd-3c77-4c0d-95ea-786bd62531f4:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-6943f4cd-3c77-4c0d-95ea-786bd62531f4 false 0})] I1230 04:00:19.506919 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-du8j4fcj): vm(k8s-agentpool-17591246-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-du8j4fcj/providers/microsoft.compute/disks/pvc-6943f4cd-3c77-4c0d-95ea-786bd62531f4:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-6943f4cd-3c77-4c0d-95ea-786bd62531f4 false 0})]) I1230 04:00:19.732476 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-du8j4fcj): vm(k8s-agentpool-17591246-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-du8j4fcj/providers/microsoft.compute/disks/pvc-6943f4cd-3c77-4c0d-95ea-786bd62531f4:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-6943f4cd-3c77-4c0d-95ea-786bd62531f4 false 0})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I1230 04:00:29.883432 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-du8j4fcj, k8s-agentpool-17591246-vmss, k8s-agentpool-17591246-vmss000000) successfully I1230 04:00:29.883473 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool-17591246-vmss, kubetest-du8j4fcj, k8s-agentpool-17591246-vmss000000) for cacheKey(kubetest-du8j4fcj/k8s-agentpool-17591246-vmss) updated successfully I1230 04:00:29.883496 1 controllerserver.go:413] Attach operation successful: volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-6943f4cd-3c77-4c0d-95ea-786bd62531f4 attached to node k8s-agentpool-17591246-vmss000000. I1230 04:00:29.883704 1 controllerserver.go:433] attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-6943f4cd-3c77-4c0d-95ea-786bd62531f4 to node k8s-agentpool-17591246-vmss000000 successfully I1230 04:00:29.883822 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=10.499928276 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-du8j4fcj" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-6943f4cd-3c77-4c0d-95ea-786bd62531f4" node="k8s-agentpool-17591246-vmss000000" result_code="succeeded" I1230 04:00:29.883867 1 utils.go:84] GRPC response: {"publish_context":{"LUN":"0"}} I1230 04:00:44.946899 1 utils.go:77] GRPC call: /csi.v1.Controller/ControllerPublishVolume I1230 04:00:44.946920 1 utils.go:78] GRPC request: {"node_id":"k8s-agentpool-17591246-vmss000001","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-17971628-47f7-4846-9951-4a4f867837c4","csi.storage.k8s.io/pvc/name":"persistent-storage-statefulset-azuredisk-0","csi.storage.k8s.io/pvc/namespace":"default","requestedsizegib":"10","skuName":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1672369721109-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-17971628-47f7-4846-9951-4a4f867837c4"} I1230 04:00:45.007904 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1248 I1230 04:00:45.008368 1 controllerserver.go:383] GetDiskLun returned: cannot find Lun for disk pvc-17971628-47f7-4846-9951-4a4f867837c4. Initiating attaching volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-17971628-47f7-4846-9951-4a4f867837c4 to node k8s-agentpool-17591246-vmss000001 (vmState Succeeded). I1230 04:00:45.008409 1 controllerserver.go:408] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-17971628-47f7-4846-9951-4a4f867837c4 to node k8s-agentpool-17591246-vmss000001 I1230 04:00:45.008448 1 azure_controller_common.go:255] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-17971628-47f7-4846-9951-4a4f867837c4 lun 0 to node k8s-agentpool-17591246-vmss000001, diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-du8j4fcj/providers/microsoft.compute/disks/pvc-17971628-47f7-4846-9951-4a4f867837c4:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-17971628-47f7-4846-9951-4a4f867837c4 false 0})] I1230 04:00:45.008500 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-du8j4fcj): vm(k8s-agentpool-17591246-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-du8j4fcj/providers/microsoft.compute/disks/pvc-17971628-47f7-4846-9951-4a4f867837c4:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-17971628-47f7-4846-9951-4a4f867837c4 false 0})]) I1230 04:00:45.150013 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-du8j4fcj): vm(k8s-agentpool-17591246-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-du8j4fcj/providers/microsoft.compute/disks/pvc-17971628-47f7-4846-9951-4a4f867837c4:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-17971628-47f7-4846-9951-4a4f867837c4 false 0})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I1230 04:00:55.264077 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-du8j4fcj, k8s-agentpool-17591246-vmss, k8s-agentpool-17591246-vmss000001) successfully I1230 04:00:55.264120 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool-17591246-vmss, kubetest-du8j4fcj, k8s-agentpool-17591246-vmss000001) for cacheKey(kubetest-du8j4fcj/k8s-agentpool-17591246-vmss) updated successfully I1230 04:00:55.264143 1 controllerserver.go:413] Attach operation successful: volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-17971628-47f7-4846-9951-4a4f867837c4 attached to node k8s-agentpool-17591246-vmss000001. I1230 04:00:55.264158 1 controllerserver.go:433] attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-17971628-47f7-4846-9951-4a4f867837c4 to node k8s-agentpool-17591246-vmss000001 successfully I1230 04:00:55.264211 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=10.255846927 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-du8j4fcj" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-17971628-47f7-4846-9951-4a4f867837c4" node="k8s-agentpool-17591246-vmss000001" result_code="succeeded" I1230 04:00:55.264230 1 utils.go:84] GRPC response: {"publish_context":{"LUN":"0"}} ... skipping 9 lines ... I1230 04:01:12.747863 1 utils.go:78] GRPC request: {"node_id":"k8s-agentpool-17591246-vmss000000","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-878843e0-00f6-4ee5-b202-be5bd68fb408","csi.storage.k8s.io/pvc/name":"persistent-storage-statefulset-azuredisk-nonroot-0","csi.storage.k8s.io/pvc/namespace":"default","requestedsizegib":"10","skuName":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1672369721109-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-878843e0-00f6-4ee5-b202-be5bd68fb408"} I1230 04:01:12.821916 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1256 I1230 04:01:12.822291 1 controllerserver.go:383] GetDiskLun returned: cannot find Lun for disk pvc-878843e0-00f6-4ee5-b202-be5bd68fb408. Initiating attaching volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-878843e0-00f6-4ee5-b202-be5bd68fb408 to node k8s-agentpool-17591246-vmss000000 (vmState Succeeded). I1230 04:01:12.822318 1 controllerserver.go:408] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-878843e0-00f6-4ee5-b202-be5bd68fb408 to node k8s-agentpool-17591246-vmss000000 I1230 04:01:12.822358 1 azure_controller_common.go:255] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-878843e0-00f6-4ee5-b202-be5bd68fb408 lun 1 to node k8s-agentpool-17591246-vmss000000, diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-du8j4fcj/providers/microsoft.compute/disks/pvc-878843e0-00f6-4ee5-b202-be5bd68fb408:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-878843e0-00f6-4ee5-b202-be5bd68fb408 false 1})] I1230 04:01:12.822407 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-du8j4fcj): vm(k8s-agentpool-17591246-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-du8j4fcj/providers/microsoft.compute/disks/pvc-878843e0-00f6-4ee5-b202-be5bd68fb408:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-878843e0-00f6-4ee5-b202-be5bd68fb408 false 1})]) I1230 04:01:12.975954 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-du8j4fcj): vm(k8s-agentpool-17591246-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-du8j4fcj/providers/microsoft.compute/disks/pvc-878843e0-00f6-4ee5-b202-be5bd68fb408:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-878843e0-00f6-4ee5-b202-be5bd68fb408 false 1})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I1230 04:01:28.139270 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-du8j4fcj, k8s-agentpool-17591246-vmss, k8s-agentpool-17591246-vmss000000) successfully I1230 04:01:28.139312 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool-17591246-vmss, kubetest-du8j4fcj, k8s-agentpool-17591246-vmss000000) for cacheKey(kubetest-du8j4fcj/k8s-agentpool-17591246-vmss) updated successfully I1230 04:01:28.139336 1 controllerserver.go:413] Attach operation successful: volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-878843e0-00f6-4ee5-b202-be5bd68fb408 attached to node k8s-agentpool-17591246-vmss000000. I1230 04:01:28.139353 1 controllerserver.go:433] attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-878843e0-00f6-4ee5-b202-be5bd68fb408 to node k8s-agentpool-17591246-vmss000000 successfully I1230 04:01:28.139400 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=15.317107097 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-du8j4fcj" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-878843e0-00f6-4ee5-b202-be5bd68fb408" node="k8s-agentpool-17591246-vmss000000" result_code="succeeded" I1230 04:01:28.139428 1 utils.go:84] GRPC response: {"publish_context":{"LUN":"1"}} ... skipping 20 lines ... Platform: linux/amd64 Topology Key: topology.disk.csi.azure.com/zone Streaming logs below: I1230 03:08:40.310423 1 azuredisk.go:175] driver userAgent: disk.csi.azure.com/v1.26.0-4cfb4ee1bf05ca9ba058389347ac5904b562074b e2e-test I1230 03:08:40.311073 1 azure_disk_utils.go:162] reading cloud config from secret kube-system/azure-cloud-provider I1230 03:08:40.343624 1 azure_disk_utils.go:169] InitializeCloudFromSecret: failed to get cloud config from secret kube-system/azure-cloud-provider: failed to get secret kube-system/azure-cloud-provider: secrets "azure-cloud-provider" not found I1230 03:08:40.343670 1 azure_disk_utils.go:174] could not read cloud config from secret kube-system/azure-cloud-provider I1230 03:08:40.343682 1 azure_disk_utils.go:184] use default AZURE_CREDENTIAL_FILE env var: /etc/kubernetes/azure.json I1230 03:08:40.343721 1 azure_disk_utils.go:192] read cloud config from file: /etc/kubernetes/azure.json successfully I1230 03:08:40.344717 1 azure_auth.go:245] Using AzurePublicCloud environment I1230 03:08:40.344785 1 azure_auth.go:130] azure: using client_id+client_secret to retrieve access token I1230 03:08:40.344864 1 azure.go:775] Azure cloudprovider using try backoff: retries=6, exponent=1.500000, duration=5, jitter=1.000000 ... skipping 25 lines ... I1230 03:08:40.345457 1 azure_blobclient.go:67] Azure BlobClient using API version: 2021-09-01 I1230 03:08:40.345491 1 azure_vmasclient.go:70] Azure AvailabilitySetsClient (read ops) using rate limit config: QPS=6, bucket=20 I1230 03:08:40.345516 1 azure_vmasclient.go:73] Azure AvailabilitySetsClient (write ops) using rate limit config: QPS=100, bucket=1000 I1230 03:08:40.346213 1 azure.go:1006] attach/detach disk operation rate limit QPS: 6.000000, Bucket: 10 I1230 03:08:40.346251 1 azuredisk.go:192] disable UseInstanceMetadata for controller I1230 03:08:40.346264 1 azuredisk.go:204] cloud: AzurePublicCloud, location: westus2, rg: kubetest-du8j4fcj, VMType: vmss, PrimaryScaleSetName: k8s-agentpool-17591246-vmss, PrimaryAvailabilitySetName: , DisableAvailabilitySetNodes: false I1230 03:08:40.350372 1 mount_linux.go:287] 'umount /tmp/kubelet-detect-safe-umount3927801952' failed with: exit status 32, output: umount: /tmp/kubelet-detect-safe-umount3927801952: must be superuser to unmount. I1230 03:08:40.350404 1 mount_linux.go:289] Detected umount with unsafe 'not mounted' behavior I1230 03:08:40.350485 1 driver.go:81] Enabling controller service capability: CREATE_DELETE_VOLUME I1230 03:08:40.350496 1 driver.go:81] Enabling controller service capability: PUBLISH_UNPUBLISH_VOLUME I1230 03:08:40.350503 1 driver.go:81] Enabling controller service capability: CREATE_DELETE_SNAPSHOT I1230 03:08:40.350511 1 driver.go:81] Enabling controller service capability: CLONE_VOLUME I1230 03:08:40.350517 1 driver.go:81] Enabling controller service capability: EXPAND_VOLUME ... skipping 65 lines ... I1230 03:09:54.497710 1 controllerserver.go:319] delete azure disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-4be03758-bf6c-473a-8086-fad258f56014) returned with <nil> I1230 03:09:54.497765 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=5.176238678 request="azuredisk_csi_driver_controller_delete_volume" resource_group="kubetest-du8j4fcj" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-4be03758-bf6c-473a-8086-fad258f56014" result_code="succeeded" I1230 03:09:54.497786 1 utils.go:84] GRPC response: {} I1230 03:10:00.112864 1 utils.go:77] GRPC call: /csi.v1.Controller/CreateVolume I1230 03:10:00.112927 1 utils.go:78] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.disk.csi.azure.com/zone":"westus2-2","topology.kubernetes.io/zone":"westus2-2"}},{"segments":{"topology.disk.csi.azure.com/zone":"westus2-1","topology.kubernetes.io/zone":"westus2-1"}}],"requisite":[{"segments":{"topology.disk.csi.azure.com/zone":"westus2-1","topology.kubernetes.io/zone":"westus2-1"}},{"segments":{"topology.disk.csi.azure.com/zone":"westus2-2","topology.kubernetes.io/zone":"westus2-2"}}]},"capacity_range":{"required_bytes":10737418240},"name":"pvc-41d31c68-d589-47ea-8d9e-2a7e36969f37","parameters":{"csi.storage.k8s.io/pv/name":"pvc-41d31c68-d589-47ea-8d9e-2a7e36969f37","csi.storage.k8s.io/pvc/name":"pvc-txd99","csi.storage.k8s.io/pvc/namespace":"azuredisk-2540","enableAsyncAttach":"false","networkAccessPolicy":"DenyAll","skuName":"Standard_LRS","userAgent":"azuredisk-e2e-test"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]} I1230 03:10:00.113947 1 azure_disk_utils.go:162] reading cloud config from secret kube-system/azure-cloud-provider I1230 03:10:00.121840 1 azure_disk_utils.go:169] InitializeCloudFromSecret: failed to get cloud config from secret kube-system/azure-cloud-provider: failed to get secret kube-system/azure-cloud-provider: secrets "azure-cloud-provider" not found I1230 03:10:00.122140 1 azure_disk_utils.go:174] could not read cloud config from secret kube-system/azure-cloud-provider I1230 03:10:00.122214 1 azure_disk_utils.go:184] use default AZURE_CREDENTIAL_FILE env var: /etc/kubernetes/azure.json I1230 03:10:00.122302 1 azure_disk_utils.go:192] read cloud config from file: /etc/kubernetes/azure.json successfully I1230 03:10:00.123003 1 azure_auth.go:245] Using AzurePublicCloud environment I1230 03:10:00.123089 1 azure_auth.go:130] azure: using client_id+client_secret to retrieve access token I1230 03:10:00.123189 1 azure.go:775] Azure cloudprovider using try backoff: retries=6, exponent=1.500000, duration=5, jitter=1.000000 ... skipping 41 lines ... I1230 03:11:03.139850 1 controllerserver.go:319] delete azure disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-41d31c68-d589-47ea-8d9e-2a7e36969f37) returned with <nil> I1230 03:11:03.139895 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=5.328185987 request="azuredisk_csi_driver_controller_delete_volume" resource_group="kubetest-du8j4fcj" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-41d31c68-d589-47ea-8d9e-2a7e36969f37" result_code="succeeded" I1230 03:11:03.139944 1 utils.go:84] GRPC response: {} I1230 03:11:08.792634 1 utils.go:77] GRPC call: /csi.v1.Controller/CreateVolume I1230 03:11:08.792668 1 utils.go:78] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.disk.csi.azure.com/zone":"westus2-1","topology.kubernetes.io/zone":"westus2-1"}}],"requisite":[{"segments":{"topology.disk.csi.azure.com/zone":"westus2-1","topology.kubernetes.io/zone":"westus2-1"}}]},"capacity_range":{"required_bytes":1099511627776},"name":"pvc-ef878302-a013-4f6f-a1e8-297da8b3ad19","parameters":{"csi.storage.k8s.io/pv/name":"pvc-ef878302-a013-4f6f-a1e8-297da8b3ad19","csi.storage.k8s.io/pvc/name":"pvc-qbmx9","csi.storage.k8s.io/pvc/namespace":"azuredisk-4728","enableAsyncAttach":"false","enableBursting":"true","perfProfile":"Basic","skuName":"Premium_LRS","userAgent":"azuredisk-e2e-test"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]} I1230 03:11:08.793615 1 azure_disk_utils.go:162] reading cloud config from secret kube-system/azure-cloud-provider I1230 03:11:08.799937 1 azure_disk_utils.go:169] InitializeCloudFromSecret: failed to get cloud config from secret kube-system/azure-cloud-provider: failed to get secret kube-system/azure-cloud-provider: secrets "azure-cloud-provider" not found I1230 03:11:08.799969 1 azure_disk_utils.go:174] could not read cloud config from secret kube-system/azure-cloud-provider I1230 03:11:08.799981 1 azure_disk_utils.go:184] use default AZURE_CREDENTIAL_FILE env var: /etc/kubernetes/azure.json I1230 03:11:08.800014 1 azure_disk_utils.go:192] read cloud config from file: /etc/kubernetes/azure.json successfully I1230 03:11:08.800524 1 azure_auth.go:245] Using AzurePublicCloud environment I1230 03:11:08.800584 1 azure_auth.go:130] azure: using client_id+client_secret to retrieve access token I1230 03:11:08.800616 1 azure.go:775] Azure cloudprovider using try backoff: retries=6, exponent=1.500000, duration=5, jitter=1.000000 ... skipping 742 lines ... Platform: linux/amd64 Topology Key: topology.disk.csi.azure.com/zone Streaming logs below: I1230 03:08:34.604677 1 azuredisk.go:175] driver userAgent: disk.csi.azure.com/v1.26.0-4cfb4ee1bf05ca9ba058389347ac5904b562074b e2e-test I1230 03:08:34.605542 1 azure_disk_utils.go:162] reading cloud config from secret kube-system/azure-cloud-provider I1230 03:08:34.631381 1 azure_disk_utils.go:169] InitializeCloudFromSecret: failed to get cloud config from secret kube-system/azure-cloud-provider: failed to get secret kube-system/azure-cloud-provider: secrets "azure-cloud-provider" not found I1230 03:08:34.631435 1 azure_disk_utils.go:174] could not read cloud config from secret kube-system/azure-cloud-provider I1230 03:08:34.631445 1 azure_disk_utils.go:184] use default AZURE_CREDENTIAL_FILE env var: /etc/kubernetes/azure.json I1230 03:08:34.631481 1 azure_disk_utils.go:192] read cloud config from file: /etc/kubernetes/azure.json successfully I1230 03:08:34.633395 1 azure_auth.go:245] Using AzurePublicCloud environment I1230 03:08:34.633477 1 azure_auth.go:130] azure: using client_id+client_secret to retrieve access token I1230 03:08:34.633533 1 azure.go:775] Azure cloudprovider using try backoff: retries=6, exponent=1.500000, duration=5, jitter=1.000000 ... skipping 147 lines ... I1230 03:12:55.957942 1 mount_linux.go:567] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0]) I1230 03:12:55.972998 1 mount_linux.go:570] Output: "" I1230 03:12:55.973043 1 mount_linux.go:529] Disk "/dev/disk/azure/scsi1/lun0" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/disk/azure/scsi1/lun0] I1230 03:12:56.450580 1 mount_linux.go:539] Disk successfully formatted (mkfs): ext4 - /dev/disk/azure/scsi1/lun0 /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6eed3ff8-9ce1-43fb-a6c8-1a50bff430f5/globalmount I1230 03:12:56.450622 1 mount_linux.go:557] Attempting to mount disk /dev/disk/azure/scsi1/lun0 in ext4 format at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6eed3ff8-9ce1-43fb-a6c8-1a50bff430f5/globalmount I1230 03:12:56.450649 1 mount_linux.go:220] Mounting cmd (mount) with arguments (-t ext4 -o invalid,mount,options,defaults /dev/disk/azure/scsi1/lun0 /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6eed3ff8-9ce1-43fb-a6c8-1a50bff430f5/globalmount) E1230 03:12:56.476399 1 mount_linux.go:232] Mount failed: exit status 32 Mounting command: mount Mounting arguments: -t ext4 -o invalid,mount,options,defaults /dev/disk/azure/scsi1/lun0 /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6eed3ff8-9ce1-43fb-a6c8-1a50bff430f5/globalmount Output: mount: /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6eed3ff8-9ce1-43fb-a6c8-1a50bff430f5/globalmount: wrong fs type, bad option, bad superblock on /dev/sdc, missing codepage or helper program, or other error. E1230 03:12:56.476492 1 utils.go:82] GRPC error: rpc error: code = Internal desc = could not format /dev/disk/azure/scsi1/lun0(lun: 0), and mount it at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6eed3ff8-9ce1-43fb-a6c8-1a50bff430f5/globalmount, failed with mount failed: exit status 32 Mounting command: mount Mounting arguments: -t ext4 -o invalid,mount,options,defaults /dev/disk/azure/scsi1/lun0 /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6eed3ff8-9ce1-43fb-a6c8-1a50bff430f5/globalmount Output: mount: /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6eed3ff8-9ce1-43fb-a6c8-1a50bff430f5/globalmount: wrong fs type, bad option, bad superblock on /dev/sdc, missing codepage or helper program, or other error. I1230 03:12:57.053995 1 utils.go:77] GRPC call: /csi.v1.Node/NodeStageVolume I1230 03:12:57.054063 1 utils.go:78] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6eed3ff8-9ce1-43fb-a6c8-1a50bff430f5/globalmount","volume_capability":{"AccessType":{"Mount":{"mount_flags":["invalid","mount","options"]}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-6eed3ff8-9ce1-43fb-a6c8-1a50bff430f5","csi.storage.k8s.io/pvc/name":"pvc-6862x","csi.storage.k8s.io/pvc/namespace":"azuredisk-5466","requestedsizegib":"10","skuName":"StandardSSD_ZRS","storage.kubernetes.io/csiProvisionerIdentity":"1672369721109-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-6eed3ff8-9ce1-43fb-a6c8-1a50bff430f5"} I1230 03:12:58.958517 1 azure_common_linux.go:185] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/ I1230 03:12:58.958566 1 nodeserver.go:116] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType StandardSSD_ZRS I1230 03:12:58.958946 1 nodeserver.go:157] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6eed3ff8-9ce1-43fb-a6c8-1a50bff430f5/globalmount with mount options([invalid mount options]) I1230 03:12:58.958962 1 mount_linux.go:567] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0]) I1230 03:12:58.985312 1 mount_linux.go:570] Output: "DEVNAME=/dev/disk/azure/scsi1/lun0\nTYPE=ext4\n" I1230 03:12:58.985348 1 mount_linux.go:453] Checking for issues with fsck on disk: /dev/disk/azure/scsi1/lun0 I1230 03:12:59.004542 1 mount_linux.go:557] Attempting to mount disk /dev/disk/azure/scsi1/lun0 in ext4 format at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6eed3ff8-9ce1-43fb-a6c8-1a50bff430f5/globalmount I1230 03:12:59.004634 1 mount_linux.go:220] Mounting cmd (mount) with arguments (-t ext4 -o invalid,mount,options,defaults /dev/disk/azure/scsi1/lun0 /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6eed3ff8-9ce1-43fb-a6c8-1a50bff430f5/globalmount) E1230 03:12:59.022858 1 mount_linux.go:232] Mount failed: exit status 32 Mounting command: mount Mounting arguments: -t ext4 -o invalid,mount,options,defaults /dev/disk/azure/scsi1/lun0 /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6eed3ff8-9ce1-43fb-a6c8-1a50bff430f5/globalmount Output: mount: /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6eed3ff8-9ce1-43fb-a6c8-1a50bff430f5/globalmount: wrong fs type, bad option, bad superblock on /dev/sdc, missing codepage or helper program, or other error. E1230 03:12:59.022925 1 utils.go:82] GRPC error: rpc error: code = Internal desc = could not format /dev/disk/azure/scsi1/lun0(lun: 0), and mount it at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6eed3ff8-9ce1-43fb-a6c8-1a50bff430f5/globalmount, failed with mount failed: exit status 32 Mounting command: mount Mounting arguments: -t ext4 -o invalid,mount,options,defaults /dev/disk/azure/scsi1/lun0 /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6eed3ff8-9ce1-43fb-a6c8-1a50bff430f5/globalmount Output: mount: /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6eed3ff8-9ce1-43fb-a6c8-1a50bff430f5/globalmount: wrong fs type, bad option, bad superblock on /dev/sdc, missing codepage or helper program, or other error. I1230 03:13:00.082786 1 utils.go:77] GRPC call: /csi.v1.Node/NodeStageVolume I1230 03:13:00.082812 1 utils.go:78] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6eed3ff8-9ce1-43fb-a6c8-1a50bff430f5/globalmount","volume_capability":{"AccessType":{"Mount":{"mount_flags":["invalid","mount","options"]}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-6eed3ff8-9ce1-43fb-a6c8-1a50bff430f5","csi.storage.k8s.io/pvc/name":"pvc-6862x","csi.storage.k8s.io/pvc/namespace":"azuredisk-5466","requestedsizegib":"10","skuName":"StandardSSD_ZRS","storage.kubernetes.io/csiProvisionerIdentity":"1672369721109-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-6eed3ff8-9ce1-43fb-a6c8-1a50bff430f5"} I1230 03:13:01.976389 1 azure_common_linux.go:185] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/ I1230 03:13:01.976452 1 nodeserver.go:116] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType StandardSSD_ZRS I1230 03:13:01.977379 1 nodeserver.go:157] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6eed3ff8-9ce1-43fb-a6c8-1a50bff430f5/globalmount with mount options([invalid mount options]) I1230 03:13:01.977411 1 mount_linux.go:567] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0]) I1230 03:13:01.994574 1 mount_linux.go:570] Output: "DEVNAME=/dev/disk/azure/scsi1/lun0\nTYPE=ext4\n" I1230 03:13:01.994600 1 mount_linux.go:453] Checking for issues with fsck on disk: /dev/disk/azure/scsi1/lun0 I1230 03:13:02.017775 1 mount_linux.go:557] Attempting to mount disk /dev/disk/azure/scsi1/lun0 in ext4 format at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6eed3ff8-9ce1-43fb-a6c8-1a50bff430f5/globalmount I1230 03:13:02.017810 1 mount_linux.go:220] Mounting cmd (mount) with arguments (-t ext4 -o invalid,mount,options,defaults /dev/disk/azure/scsi1/lun0 /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6eed3ff8-9ce1-43fb-a6c8-1a50bff430f5/globalmount) E1230 03:13:02.036849 1 mount_linux.go:232] Mount failed: exit status 32 Mounting command: mount Mounting arguments: -t ext4 -o invalid,mount,options,defaults /dev/disk/azure/scsi1/lun0 /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6eed3ff8-9ce1-43fb-a6c8-1a50bff430f5/globalmount Output: mount: /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6eed3ff8-9ce1-43fb-a6c8-1a50bff430f5/globalmount: wrong fs type, bad option, bad superblock on /dev/sdc, missing codepage or helper program, or other error. E1230 03:13:02.036921 1 utils.go:82] GRPC error: rpc error: code = Internal desc = could not format /dev/disk/azure/scsi1/lun0(lun: 0), and mount it at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6eed3ff8-9ce1-43fb-a6c8-1a50bff430f5/globalmount, failed with mount failed: exit status 32 Mounting command: mount Mounting arguments: -t ext4 -o invalid,mount,options,defaults /dev/disk/azure/scsi1/lun0 /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6eed3ff8-9ce1-43fb-a6c8-1a50bff430f5/globalmount Output: mount: /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6eed3ff8-9ce1-43fb-a6c8-1a50bff430f5/globalmount: wrong fs type, bad option, bad superblock on /dev/sdc, missing codepage or helper program, or other error. I1230 03:13:58.475226 1 utils.go:77] GRPC call: /csi.v1.Node/NodeStageVolume I1230 03:13:58.475264 1 utils.go:78] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-5e94aef0-afd5-46ef-b079-f4bad5b9b2e9","volume_capability":{"AccessType":{"Block":{}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-5e94aef0-afd5-46ef-b079-f4bad5b9b2e9","csi.storage.k8s.io/pvc/name":"pvc-czzmh","csi.storage.k8s.io/pvc/namespace":"azuredisk-2790","requestedsizegib":"10","skuName":"StandardSSD_ZRS","storage.kubernetes.io/csiProvisionerIdentity":"1672369721109-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-5e94aef0-afd5-46ef-b079-f4bad5b9b2e9"} I1230 03:14:00.298401 1 azure_common_linux.go:185] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/ I1230 03:14:00.298452 1 nodeserver.go:116] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType StandardSSD_ZRS I1230 03:14:00.298467 1 utils.go:84] GRPC response: {} I1230 03:14:00.308902 1 utils.go:77] GRPC call: /csi.v1.Node/NodePublishVolume ... skipping 16 lines ... I1230 03:14:06.258706 1 utils.go:84] GRPC response: {} I1230 03:14:06.292787 1 utils.go:77] GRPC call: /csi.v1.Node/NodeUnstageVolume I1230 03:14:06.292820 1 utils.go:78] GRPC request: {"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-5e94aef0-afd5-46ef-b079-f4bad5b9b2e9","volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-5e94aef0-afd5-46ef-b079-f4bad5b9b2e9"} I1230 03:14:06.292913 1 nodeserver.go:201] NodeUnstageVolume: unmounting /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-5e94aef0-afd5-46ef-b079-f4bad5b9b2e9 I1230 03:14:06.292961 1 mount_helper_common.go:93] unmounting "/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-5e94aef0-afd5-46ef-b079-f4bad5b9b2e9" (corruptedMount: false, mounterCanSkipMountPointChecks: true) I1230 03:14:06.292975 1 mount_linux.go:362] Unmounting /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-5e94aef0-afd5-46ef-b079-f4bad5b9b2e9 I1230 03:14:06.295197 1 mount_linux.go:375] ignoring 'not mounted' error for /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-5e94aef0-afd5-46ef-b079-f4bad5b9b2e9 I1230 03:14:06.295220 1 mount_helper_common.go:150] Warning: deleting path "/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-5e94aef0-afd5-46ef-b079-f4bad5b9b2e9" I1230 03:14:06.295330 1 nodeserver.go:206] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-5e94aef0-afd5-46ef-b079-f4bad5b9b2e9 successfully I1230 03:14:06.295344 1 utils.go:84] GRPC response: {} I1230 03:15:08.613591 1 utils.go:77] GRPC call: /csi.v1.Node/NodeStageVolume I1230 03:15:08.613618 1 utils.go:78] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-247ef8cf-cacf-43d8-9263-4145cc9afa17/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-247ef8cf-cacf-43d8-9263-4145cc9afa17","csi.storage.k8s.io/pvc/name":"pvc-8hpm9","csi.storage.k8s.io/pvc/namespace":"azuredisk-5356","requestedsizegib":"10","resourceGroup":"azuredisk-csi-driver-test-1da21cf0-87f0-11ed-a90a-762e3f62171e","skuName":"StandardSSD_ZRS","storage.kubernetes.io/csiProvisionerIdentity":"1672369721109-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/azuredisk-csi-driver-test-1da21cf0-87f0-11ed-a90a-762e3f62171e/providers/Microsoft.Compute/disks/pvc-247ef8cf-cacf-43d8-9263-4145cc9afa17"} I1230 03:15:10.424137 1 azure_common_linux.go:185] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/ ... skipping 648 lines ... I1230 03:34:55.353167 1 utils.go:84] GRPC response: {} I1230 03:34:55.377877 1 utils.go:77] GRPC call: /csi.v1.Node/NodeUnstageVolume I1230 03:34:55.377900 1 utils.go:78] GRPC request: {"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-d09842e7-2ad7-48e6-8375-9c8ff141aa66","volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-d09842e7-2ad7-48e6-8375-9c8ff141aa66"} I1230 03:34:55.377971 1 nodeserver.go:201] NodeUnstageVolume: unmounting /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-d09842e7-2ad7-48e6-8375-9c8ff141aa66 I1230 03:34:55.377996 1 mount_helper_common.go:93] unmounting "/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-d09842e7-2ad7-48e6-8375-9c8ff141aa66" (corruptedMount: false, mounterCanSkipMountPointChecks: true) I1230 03:34:55.378005 1 mount_linux.go:362] Unmounting /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-d09842e7-2ad7-48e6-8375-9c8ff141aa66 I1230 03:34:55.379837 1 mount_linux.go:375] ignoring 'not mounted' error for /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-d09842e7-2ad7-48e6-8375-9c8ff141aa66 I1230 03:34:55.379854 1 mount_helper_common.go:150] Warning: deleting path "/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-d09842e7-2ad7-48e6-8375-9c8ff141aa66" I1230 03:34:55.379949 1 nodeserver.go:206] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-d09842e7-2ad7-48e6-8375-9c8ff141aa66 successfully I1230 03:34:55.379962 1 utils.go:84] GRPC response: {} I1230 03:36:36.088482 1 utils.go:77] GRPC call: /csi.v1.Node/NodeStageVolume I1230 03:36:36.088554 1 utils.go:78] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-f824bdf6-18b4-4fcb-9aa9-f2f83a57ea1c/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-f824bdf6-18b4-4fcb-9aa9-f2f83a57ea1c","csi.storage.k8s.io/pvc/name":"pvc-sfxbt","csi.storage.k8s.io/pvc/namespace":"azuredisk-8582","requestedsizegib":"10","skuName":"StandardSSD_ZRS","storage.kubernetes.io/csiProvisionerIdentity":"1672369721109-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-f824bdf6-18b4-4fcb-9aa9-f2f83a57ea1c"} I1230 03:36:37.950912 1 azure_common_linux.go:185] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/ ... skipping 579 lines ... I1230 03:58:22.956385 1 utils.go:84] GRPC response: {} I1230 03:58:22.982509 1 utils.go:77] GRPC call: /csi.v1.Node/NodeUnstageVolume I1230 03:58:22.982529 1 utils.go:78] GRPC request: {"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-270ec856-ccdf-49b4-acde-cda596efc335","volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-270ec856-ccdf-49b4-acde-cda596efc335"} I1230 03:58:22.982580 1 nodeserver.go:201] NodeUnstageVolume: unmounting /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-270ec856-ccdf-49b4-acde-cda596efc335 I1230 03:58:22.982601 1 mount_helper_common.go:93] unmounting "/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-270ec856-ccdf-49b4-acde-cda596efc335" (corruptedMount: false, mounterCanSkipMountPointChecks: true) I1230 03:58:22.982613 1 mount_linux.go:362] Unmounting /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-270ec856-ccdf-49b4-acde-cda596efc335 I1230 03:58:22.985019 1 mount_linux.go:375] ignoring 'not mounted' error for /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-270ec856-ccdf-49b4-acde-cda596efc335 I1230 03:58:22.985031 1 mount_helper_common.go:150] Warning: deleting path "/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-270ec856-ccdf-49b4-acde-cda596efc335" I1230 03:58:22.985175 1 nodeserver.go:206] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-270ec856-ccdf-49b4-acde-cda596efc335 successfully I1230 03:58:22.985190 1 utils.go:84] GRPC response: {} I1230 03:59:27.621507 1 utils.go:77] GRPC call: /csi.v1.Node/NodeStageVolume I1230 03:59:27.621543 1 utils.go:78] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-ac778d07-712a-4d77-9adc-570701d094fc/globalmount","volume_capability":{"AccessType":{"Mount":{"mount_flags":["barrier=1","acl"]}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-ac778d07-712a-4d77-9adc-570701d094fc","csi.storage.k8s.io/pvc/name":"pvc-bqvnz","csi.storage.k8s.io/pvc/namespace":"azuredisk-1092","device-setting/device/queue_depth":"17","device-setting/queue/max_sectors_kb":"211","device-setting/queue/nr_requests":"44","device-setting/queue/read_ahead_kb":"256","device-setting/queue/rotational":"0","device-setting/queue/scheduler":"none","device-setting/queue/wbt_lat_usec":"0","perfProfile":"advanced","requestedsizegib":"10","skuname":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1672369721109-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-ac778d07-712a-4d77-9adc-570701d094fc"} I1230 03:59:29.390392 1 azure_common_linux.go:185] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/ ... skipping 100 lines ... Platform: linux/amd64 Topology Key: topology.disk.csi.azure.com/zone Streaming logs below: I1230 03:08:30.938340 1 azuredisk.go:175] driver userAgent: disk.csi.azure.com/v1.26.0-4cfb4ee1bf05ca9ba058389347ac5904b562074b e2e-test I1230 03:08:30.939091 1 azure_disk_utils.go:162] reading cloud config from secret kube-system/azure-cloud-provider I1230 03:08:30.971496 1 azure_disk_utils.go:169] InitializeCloudFromSecret: failed to get cloud config from secret kube-system/azure-cloud-provider: failed to get secret kube-system/azure-cloud-provider: secrets "azure-cloud-provider" not found I1230 03:08:30.971518 1 azure_disk_utils.go:174] could not read cloud config from secret kube-system/azure-cloud-provider I1230 03:08:30.971527 1 azure_disk_utils.go:184] use default AZURE_CREDENTIAL_FILE env var: /etc/kubernetes/azure.json I1230 03:08:30.971588 1 azure_disk_utils.go:192] read cloud config from file: /etc/kubernetes/azure.json successfully I1230 03:08:30.973055 1 azure_auth.go:245] Using AzurePublicCloud environment I1230 03:08:30.973105 1 azure_auth.go:130] azure: using client_id+client_secret to retrieve access token I1230 03:08:30.973176 1 azure.go:775] Azure cloudprovider using try backoff: retries=6, exponent=1.500000, duration=5, jitter=1.000000 ... skipping 68 lines ... Platform: linux/amd64 Topology Key: topology.disk.csi.azure.com/zone Streaming logs below: I1230 03:08:35.455835 1 azuredisk.go:175] driver userAgent: disk.csi.azure.com/v1.26.0-4cfb4ee1bf05ca9ba058389347ac5904b562074b e2e-test I1230 03:08:35.456523 1 azure_disk_utils.go:162] reading cloud config from secret kube-system/azure-cloud-provider I1230 03:08:35.480377 1 azure_disk_utils.go:169] InitializeCloudFromSecret: failed to get cloud config from secret kube-system/azure-cloud-provider: failed to get secret kube-system/azure-cloud-provider: secrets "azure-cloud-provider" not found I1230 03:08:35.480407 1 azure_disk_utils.go:174] could not read cloud config from secret kube-system/azure-cloud-provider I1230 03:08:35.480417 1 azure_disk_utils.go:184] use default AZURE_CREDENTIAL_FILE env var: /etc/kubernetes/azure.json I1230 03:08:35.480449 1 azure_disk_utils.go:192] read cloud config from file: /etc/kubernetes/azure.json successfully I1230 03:08:35.481730 1 azure_auth.go:245] Using AzurePublicCloud environment I1230 03:08:35.481789 1 azure_auth.go:130] azure: using client_id+client_secret to retrieve access token I1230 03:08:35.481830 1 azure.go:775] Azure cloudprovider using try backoff: retries=6, exponent=1.500000, duration=5, jitter=1.000000 ... skipping 242 lines ... I1230 03:58:22.748538 1 utils.go:84] GRPC response: {} I1230 03:58:22.783971 1 utils.go:77] GRPC call: /csi.v1.Node/NodeUnstageVolume I1230 03:58:22.783993 1 utils.go:78] GRPC request: {"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-270ec856-ccdf-49b4-acde-cda596efc335","volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-270ec856-ccdf-49b4-acde-cda596efc335"} I1230 03:58:22.784068 1 nodeserver.go:201] NodeUnstageVolume: unmounting /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-270ec856-ccdf-49b4-acde-cda596efc335 I1230 03:58:22.784092 1 mount_helper_common.go:93] unmounting "/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-270ec856-ccdf-49b4-acde-cda596efc335" (corruptedMount: false, mounterCanSkipMountPointChecks: true) I1230 03:58:22.784105 1 mount_linux.go:362] Unmounting /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-270ec856-ccdf-49b4-acde-cda596efc335 I1230 03:58:22.786545 1 mount_linux.go:375] ignoring 'not mounted' error for /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-270ec856-ccdf-49b4-acde-cda596efc335 I1230 03:58:22.786580 1 mount_helper_common.go:150] Warning: deleting path "/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-270ec856-ccdf-49b4-acde-cda596efc335" I1230 03:58:22.786683 1 nodeserver.go:206] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-270ec856-ccdf-49b4-acde-cda596efc335 successfully I1230 03:58:22.786722 1 utils.go:84] GRPC response: {} I1230 04:01:00.787153 1 utils.go:77] GRPC call: /csi.v1.Node/NodeStageVolume I1230 04:01:00.787181 1 utils.go:78] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-17971628-47f7-4846-9951-4a4f867837c4/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-17971628-47f7-4846-9951-4a4f867837c4","csi.storage.k8s.io/pvc/name":"persistent-storage-statefulset-azuredisk-0","csi.storage.k8s.io/pvc/namespace":"default","requestedsizegib":"10","skuName":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1672369721109-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-du8j4fcj/providers/Microsoft.Compute/disks/pvc-17971628-47f7-4846-9951-4a4f867837c4"} I1230 04:01:02.735509 1 azure_common_linux.go:185] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/ ... skipping 316 lines ... # HELP go_gc_heap_objects_objects Number of objects, live or unswept, occupying heap memory. # TYPE go_gc_heap_objects_objects gauge go_gc_heap_objects_objects 53100 # HELP go_gc_heap_tiny_allocs_objects_total Count of small allocations that are packed together into blocks. These allocations are counted separately from other allocations because each individual allocation is not tracked by the runtime, only their block. Each block is already accounted for in allocs-by-size and frees-by-size. # TYPE go_gc_heap_tiny_allocs_objects_total counter go_gc_heap_tiny_allocs_objects_total 43936 # HELP go_gc_limiter_last_enabled_gc_cycle GC cycle the last time the GC CPU limiter was enabled. This metric is useful for diagnosing the root cause of an out-of-memory error, because the limiter trades memory for CPU time when the GC's CPU time gets too high. This is most likely to occur with use of SetMemoryLimit. The first GC cycle is cycle 1, so a value of 0 indicates that it was never enabled. # TYPE go_gc_limiter_last_enabled_gc_cycle gauge go_gc_limiter_last_enabled_gc_cycle 0 # HELP go_gc_pauses_seconds Distribution individual GC-related stop-the-world pause latencies. # TYPE go_gc_pauses_seconds histogram go_gc_pauses_seconds_bucket{le="9.999999999999999e-10"} 0 go_gc_pauses_seconds_bucket{le="9.999999999999999e-09"} 0 ... skipping 686 lines ... # HELP go_gc_heap_objects_objects Number of objects, live or unswept, occupying heap memory. # TYPE go_gc_heap_objects_objects gauge go_gc_heap_objects_objects 46234 # HELP go_gc_heap_tiny_allocs_objects_total Count of small allocations that are packed together into blocks. These allocations are counted separately from other allocations because each individual allocation is not tracked by the runtime, only their block. Each block is already accounted for in allocs-by-size and frees-by-size. # TYPE go_gc_heap_tiny_allocs_objects_total counter go_gc_heap_tiny_allocs_objects_total 20190 # HELP go_gc_limiter_last_enabled_gc_cycle GC cycle the last time the GC CPU limiter was enabled. This metric is useful for diagnosing the root cause of an out-of-memory error, because the limiter trades memory for CPU time when the GC's CPU time gets too high. This is most likely to occur with use of SetMemoryLimit. The first GC cycle is cycle 1, so a value of 0 indicates that it was never enabled. # TYPE go_gc_limiter_last_enabled_gc_cycle gauge go_gc_limiter_last_enabled_gc_cycle 0 # HELP go_gc_pauses_seconds Distribution individual GC-related stop-the-world pause latencies. # TYPE go_gc_pauses_seconds histogram go_gc_pauses_seconds_bucket{le="9.999999999999999e-10"} 0 go_gc_pauses_seconds_bucket{le="9.999999999999999e-09"} 0 ... skipping 272 lines ... [AfterSuite] [38;5;243m/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/suite_test.go:165[0m [38;5;243m------------------------------[0m [38;5;9m[1mSummarizing 1 Failure:[0m [38;5;9m[FAIL][0m [0mDynamic Provisioning [38;5;243m[multi-az] [0m[38;5;9m[1m[It] should create a pod, write to its pv, take a volume snapshot, overwrite data in original pv, create another pod from the snapshot, and read unaltered original data from original pv[disk.csi.azure.com][0m[0m [38;5;243m/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/testsuites/testsuites.go:823[0m [38;5;9m[1mRan 26 of 66 Specs in 4029.181 seconds[0m [38;5;9m[1mFAIL![0m -- [38;5;10m[1m25 Passed[0m | [38;5;9m[1m1 Failed[0m | [38;5;11m[1m0 Pending[0m | [38;5;14m[1m40 Skipped[0m [38;5;228mYou're using deprecated Ginkgo functionality:[0m [38;5;228m=============================================[0m [38;5;11mSupport for custom reporters has been removed in V2. Please read the documentation linked to below for Ginkgo's new behavior and for a migration path:[0m [1mLearn more at:[0m [38;5;14m[4mhttps://onsi.github.io/ginkgo/MIGRATING_TO_V2#removed-custom-reporters[0m [38;5;243mTo silence deprecations that can be silenced set the following environment variable:[0m [38;5;243mACK_GINKGO_DEPRECATIONS=2.4.0[0m --- FAIL: TestE2E (4029.19s) FAIL FAIL sigs.k8s.io/azuredisk-csi-driver/test/e2e 4029.357s FAIL make: *** [Makefile:261: e2e-test] Error 1 2022/12/30 04:02:42 process.go:155: Step 'make e2e-test' finished in 1h9m57.964251869s 2022/12/30 04:02:42 aksengine_helpers.go:425: downloading /root/tmp1650667201/log-dump.sh from https://raw.githubusercontent.com/kubernetes-sigs/cloud-provider-azure/master/hack/log-dump/log-dump.sh 2022/12/30 04:02:42 util.go:70: curl https://raw.githubusercontent.com/kubernetes-sigs/cloud-provider-azure/master/hack/log-dump/log-dump.sh 2022/12/30 04:02:43 process.go:153: Running: chmod +x /root/tmp1650667201/log-dump.sh 2022/12/30 04:02:43 process.go:155: Step 'chmod +x /root/tmp1650667201/log-dump.sh' finished in 2.769703ms 2022/12/30 04:02:43 aksengine_helpers.go:425: downloading /root/tmp1650667201/log-dump-daemonset.yaml from https://raw.githubusercontent.com/kubernetes-sigs/cloud-provider-azure/master/hack/log-dump/log-dump-daemonset.yaml ... skipping 63 lines ... ssh key file /root/.ssh/id_rsa does not exist. Exiting. 2022/12/30 04:03:18 process.go:155: Step 'bash -c /root/tmp1650667201/win-ci-logs-collector.sh kubetest-du8j4fcj.westus2.cloudapp.azure.com /root/tmp1650667201 /root/.ssh/id_rsa' finished in 3.90595ms 2022/12/30 04:03:18 aksengine.go:1141: Deleting resource group: kubetest-du8j4fcj. 2022/12/30 04:09:36 process.go:96: Saved XML output to /logs/artifacts/junit_runner.xml. 2022/12/30 04:09:36 process.go:153: Running: bash -c . hack/lib/version.sh && KUBE_ROOT=. kube::version::get_version_vars && echo "${KUBE_GIT_VERSION-}" 2022/12/30 04:09:36 process.go:155: Step 'bash -c . hack/lib/version.sh && KUBE_ROOT=. kube::version::get_version_vars && echo "${KUBE_GIT_VERSION-}"' finished in 321.896504ms 2022/12/30 04:09:36 main.go:328: Something went wrong: encountered 1 errors: [error during make e2e-test: exit status 2] + EXIT_VALUE=1 + set +o xtrace Cleaning up after docker in docker. ================================================================================ Cleaning up after docker 8831d5c81313 ... skipping 4 lines ...