kubernetes-sigs/azuredisk-csi-driver PR #1705 pull-azuredisk-csi-driver-e2e-multi-az #1619537375879237632
Recent runs || View in Spyglass
Recent runs || View in Spyglass
| PR | andyzhangx: [WIP] test |
| Result | FAILURE |
| Tests | 1 failed / 13 succeeded |
| Started | |
| Elapsed | 1h15m |
| Revision | 5916b4686125b493881f175d375f7803f797feda |
| Refs |
1705 |
| job-version | v1.27.0-alpha.1.73+8e642d3d0deab2 |
| kubetest-version | v20230117-50d6df3625 |
| revision | v1.27.0-alpha.1.73+8e642d3d0deab2 |
error during make e2e-test: exit status 2
from junit_runner.xml
Filter through log files | View test history on testgrid
kubetest Check APIReachability
kubetest Deferred TearDown
kubetest DumpClusterLogs
kubetest GetDeployer
kubetest IsUp
kubetest Prepare
kubetest TearDown
kubetest TearDown Previous
kubetest Timeout
kubetest Up
kubetest kubectl version
kubetest list nodes
kubetest test setup
... skipping 107 lines ... 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 11345 100 11345 0 0 181k 0 --:--:-- --:--:-- --:--:-- 181k Downloading https://get.helm.sh/helm-v3.11.0-linux-amd64.tar.gz Verifying checksum... Done. Preparing to install helm into /usr/local/bin helm installed into /usr/local/bin/helm docker pull k8sprow.azurecr.io/azuredisk-csi:v1.27.0-d5369cddb1fe3d1d78e5a517b281425883f14ad4 || make container-all push-manifest Error response from daemon: manifest for k8sprow.azurecr.io/azuredisk-csi:v1.27.0-d5369cddb1fe3d1d78e5a517b281425883f14ad4 not found: manifest unknown: manifest tagged by "v1.27.0-d5369cddb1fe3d1d78e5a517b281425883f14ad4" is not found make[2]: Entering directory '/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver' CGO_ENABLED=0 GOOS=windows go build -a -ldflags "-X sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.driverVersion=v1.27.0-d5369cddb1fe3d1d78e5a517b281425883f14ad4 -X sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.gitCommit=d5369cddb1fe3d1d78e5a517b281425883f14ad4 -X sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.buildDate=2023-01-29T03:34:26Z -extldflags "-static"" -mod vendor -o _output/amd64/azurediskplugin.exe ./pkg/azurediskplugin docker buildx rm container-builder || true ERROR: no builder "container-builder" found docker buildx create --use --name=container-builder container-builder # enable qemu for arm64 build # https://github.com/docker/buildx/issues/464#issuecomment-741507760 docker run --privileged --rm tonistiigi/binfmt --uninstall qemu-aarch64 Unable to find image 'tonistiigi/binfmt:latest' locally ... skipping 1757 lines ... type: string type: object oneOf: - required: ["persistentVolumeClaimName"] - required: ["volumeSnapshotContentName"] volumeSnapshotClassName: description: 'VolumeSnapshotClassName is the name of the VolumeSnapshotClass requested by the VolumeSnapshot. VolumeSnapshotClassName may be left nil to indicate that the default SnapshotClass should be used. A given cluster may have multiple default Volume SnapshotClasses: one default per CSI Driver. If a VolumeSnapshot does not specify a SnapshotClass, VolumeSnapshotSource will be checked to figure out what the associated CSI Driver is, and the default VolumeSnapshotClass associated with that CSI Driver will be used. If more than one VolumeSnapshotClass exist for a given CSI Driver and more than one have been marked as default, CreateSnapshot will fail and generate an event. Empty string is not allowed for this field.' type: string required: - source type: object status: description: status represents the current information of a snapshot. Consumers must verify binding between VolumeSnapshot and VolumeSnapshotContent objects is successful (by validating that both VolumeSnapshot and VolumeSnapshotContent point at each other) before using this object. ... skipping 2 lines ... description: 'boundVolumeSnapshotContentName is the name of the VolumeSnapshotContent object to which this VolumeSnapshot object intends to bind to. If not specified, it indicates that the VolumeSnapshot object has not been successfully bound to a VolumeSnapshotContent object yet. NOTE: To avoid possible security issues, consumers must verify binding between VolumeSnapshot and VolumeSnapshotContent objects is successful (by validating that both VolumeSnapshot and VolumeSnapshotContent point at each other) before using this object.' type: string creationTime: description: creationTime is the timestamp when the point-in-time snapshot is taken by the underlying storage system. In dynamic snapshot creation case, this field will be filled in by the snapshot controller with the "creation_time" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "creation_time" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. If not specified, it may indicate that the creation time of the snapshot is unknown. format: date-time type: string error: description: error is the last observed error during snapshot creation, if any. This field could be helpful to upper level controllers(i.e., application controller) to decide whether they should continue on waiting for the snapshot to be created based on the type of error reported. The snapshot controller will keep retrying when an error occurrs during the snapshot creation. Upon success, this error field will be cleared. properties: message: description: 'message is a string detailing the encountered error during snapshot creation if specified. NOTE: message may be logged, and it should not contain sensitive information.' type: string time: description: time is the timestamp when the error was encountered. format: date-time type: string type: object readyToUse: description: readyToUse indicates if the snapshot is ready to be used to restore a volume. In dynamic snapshot creation case, this field will be filled in by the snapshot controller with the "ready_to_use" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "ready_to_use" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it, otherwise, this field will be set to "True". If not specified, it means the readiness of a snapshot is unknown. type: boolean restoreSize: type: string description: restoreSize represents the minimum size of volume required to create a volume from this snapshot. In dynamic snapshot creation case, this field will be filled in by the snapshot controller with the "size_bytes" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "size_bytes" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. When restoring a volume from this snapshot, the size of the volume MUST NOT be smaller than the restoreSize if it is specified, otherwise the restoration will fail. If not specified, it indicates that the size is unknown. pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ x-kubernetes-int-or-string: true type: object required: - spec type: object ... skipping 60 lines ... type: string volumeSnapshotContentName: description: volumeSnapshotContentName specifies the name of a pre-existing VolumeSnapshotContent object representing an existing volume snapshot. This field should be set if the snapshot already exists and only needs a representation in Kubernetes. This field is immutable. type: string type: object volumeSnapshotClassName: description: 'VolumeSnapshotClassName is the name of the VolumeSnapshotClass requested by the VolumeSnapshot. VolumeSnapshotClassName may be left nil to indicate that the default SnapshotClass should be used. A given cluster may have multiple default Volume SnapshotClasses: one default per CSI Driver. If a VolumeSnapshot does not specify a SnapshotClass, VolumeSnapshotSource will be checked to figure out what the associated CSI Driver is, and the default VolumeSnapshotClass associated with that CSI Driver will be used. If more than one VolumeSnapshotClass exist for a given CSI Driver and more than one have been marked as default, CreateSnapshot will fail and generate an event. Empty string is not allowed for this field.' type: string required: - source type: object status: description: status represents the current information of a snapshot. Consumers must verify binding between VolumeSnapshot and VolumeSnapshotContent objects is successful (by validating that both VolumeSnapshot and VolumeSnapshotContent point at each other) before using this object. ... skipping 2 lines ... description: 'boundVolumeSnapshotContentName is the name of the VolumeSnapshotContent object to which this VolumeSnapshot object intends to bind to. If not specified, it indicates that the VolumeSnapshot object has not been successfully bound to a VolumeSnapshotContent object yet. NOTE: To avoid possible security issues, consumers must verify binding between VolumeSnapshot and VolumeSnapshotContent objects is successful (by validating that both VolumeSnapshot and VolumeSnapshotContent point at each other) before using this object.' type: string creationTime: description: creationTime is the timestamp when the point-in-time snapshot is taken by the underlying storage system. In dynamic snapshot creation case, this field will be filled in by the snapshot controller with the "creation_time" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "creation_time" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. If not specified, it may indicate that the creation time of the snapshot is unknown. format: date-time type: string error: description: error is the last observed error during snapshot creation, if any. This field could be helpful to upper level controllers(i.e., application controller) to decide whether they should continue on waiting for the snapshot to be created based on the type of error reported. The snapshot controller will keep retrying when an error occurrs during the snapshot creation. Upon success, this error field will be cleared. properties: message: description: 'message is a string detailing the encountered error during snapshot creation if specified. NOTE: message may be logged, and it should not contain sensitive information.' type: string time: description: time is the timestamp when the error was encountered. format: date-time type: string type: object readyToUse: description: readyToUse indicates if the snapshot is ready to be used to restore a volume. In dynamic snapshot creation case, this field will be filled in by the snapshot controller with the "ready_to_use" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "ready_to_use" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it, otherwise, this field will be set to "True". If not specified, it means the readiness of a snapshot is unknown. type: boolean restoreSize: type: string description: restoreSize represents the minimum size of volume required to create a volume from this snapshot. In dynamic snapshot creation case, this field will be filled in by the snapshot controller with the "size_bytes" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "size_bytes" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. When restoring a volume from this snapshot, the size of the volume MUST NOT be smaller than the restoreSize if it is specified, otherwise the restoration will fail. If not specified, it indicates that the size is unknown. pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ x-kubernetes-int-or-string: true type: object required: - spec type: object ... skipping 254 lines ... description: status represents the current information of a snapshot. properties: creationTime: description: creationTime is the timestamp when the point-in-time snapshot is taken by the underlying storage system. In dynamic snapshot creation case, this field will be filled in by the CSI snapshotter sidecar with the "creation_time" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "creation_time" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. If not specified, it indicates the creation time is unknown. The format of this field is a Unix nanoseconds time encoded as an int64. On Unix, the command `date +%s%N` returns the current time in nanoseconds since 1970-01-01 00:00:00 UTC. format: int64 type: integer error: description: error is the last observed error during snapshot creation, if any. Upon success after retry, this error field will be cleared. properties: message: description: 'message is a string detailing the encountered error during snapshot creation if specified. NOTE: message may be logged, and it should not contain sensitive information.' type: string time: description: time is the timestamp when the error was encountered. format: date-time type: string type: object readyToUse: description: readyToUse indicates if a snapshot is ready to be used to restore a volume. In dynamic snapshot creation case, this field will be filled in by the CSI snapshotter sidecar with the "ready_to_use" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "ready_to_use" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it, otherwise, this field will be set to "True". If not specified, it means the readiness of a snapshot is unknown. type: boolean restoreSize: description: restoreSize represents the complete size of the snapshot in bytes. In dynamic snapshot creation case, this field will be filled in by the CSI snapshotter sidecar with the "size_bytes" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "size_bytes" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. When restoring a volume from this snapshot, the size of the volume MUST NOT be smaller than the restoreSize if it is specified, otherwise the restoration will fail. If not specified, it indicates that the size is unknown. format: int64 minimum: 0 type: integer snapshotHandle: description: snapshotHandle is the CSI "snapshot_id" of a snapshot on the underlying storage system. If not specified, it indicates that dynamic snapshot creation has either failed or it is still in progress. type: string type: object required: - spec type: object served: true ... skipping 108 lines ... description: status represents the current information of a snapshot. properties: creationTime: description: creationTime is the timestamp when the point-in-time snapshot is taken by the underlying storage system. In dynamic snapshot creation case, this field will be filled in by the CSI snapshotter sidecar with the "creation_time" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "creation_time" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. If not specified, it indicates the creation time is unknown. The format of this field is a Unix nanoseconds time encoded as an int64. On Unix, the command `date +%s%N` returns the current time in nanoseconds since 1970-01-01 00:00:00 UTC. format: int64 type: integer error: description: error is the last observed error during snapshot creation, if any. Upon success after retry, this error field will be cleared. properties: message: description: 'message is a string detailing the encountered error during snapshot creation if specified. NOTE: message may be logged, and it should not contain sensitive information.' type: string time: description: time is the timestamp when the error was encountered. format: date-time type: string type: object readyToUse: description: readyToUse indicates if a snapshot is ready to be used to restore a volume. In dynamic snapshot creation case, this field will be filled in by the CSI snapshotter sidecar with the "ready_to_use" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "ready_to_use" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it, otherwise, this field will be set to "True". If not specified, it means the readiness of a snapshot is unknown. type: boolean restoreSize: description: restoreSize represents the complete size of the snapshot in bytes. In dynamic snapshot creation case, this field will be filled in by the CSI snapshotter sidecar with the "size_bytes" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "size_bytes" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. When restoring a volume from this snapshot, the size of the volume MUST NOT be smaller than the restoreSize if it is specified, otherwise the restoration will fail. If not specified, it indicates that the size is unknown. format: int64 minimum: 0 type: integer snapshotHandle: description: snapshotHandle is the CSI "snapshot_id" of a snapshot on the underlying storage system. If not specified, it indicates that dynamic snapshot creation has either failed or it is still in progress. type: string type: object required: - spec type: object served: true ... skipping 865 lines ... image: "mcr.microsoft.com/oss/kubernetes-csi/csi-resizer:v1.6.0" args: - "-csi-address=$(ADDRESS)" - "-v=2" - "-leader-election" - "--leader-election-namespace=kube-system" - '-handle-volume-inuse-error=false' - '-feature-gates=RecoverVolumeExpansionFailure=true' - "-timeout=240s" env: - name: ADDRESS value: /csi/csi.sock volumeMounts: ... skipping 216 lines ... [1mSTEP:[0m setting up the StorageClass [38;5;243m01/29/23 03:44:12.455[0m [1mSTEP:[0m creating a StorageClass [38;5;243m01/29/23 03:44:12.455[0m [1mSTEP:[0m setting up the PVC and PV [38;5;243m01/29/23 03:44:12.515[0m [1mSTEP:[0m creating a PVC [38;5;243m01/29/23 03:44:12.515[0m [1mSTEP:[0m setting up the pod [38;5;243m01/29/23 03:44:12.577[0m [1mSTEP:[0m deploying the pod [38;5;243m01/29/23 03:44:12.577[0m [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/29/23 03:44:12.637[0m Jan 29 03:44:12.637: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-szbzp" in namespace "azuredisk-8081" to be "Succeeded or Failed" Jan 29 03:44:12.696: INFO: Pod "azuredisk-volume-tester-szbzp": Phase="Pending", Reason="", readiness=false. Elapsed: 58.726867ms Jan 29 03:44:14.756: INFO: Pod "azuredisk-volume-tester-szbzp": Phase="Pending", Reason="", readiness=false. Elapsed: 2.118280967s Jan 29 03:44:16.755: INFO: Pod "azuredisk-volume-tester-szbzp": Phase="Pending", Reason="", readiness=false. Elapsed: 4.117787383s Jan 29 03:44:18.757: INFO: Pod "azuredisk-volume-tester-szbzp": Phase="Pending", Reason="", readiness=false. Elapsed: 6.119618156s Jan 29 03:44:20.764: INFO: Pod "azuredisk-volume-tester-szbzp": Phase="Pending", Reason="", readiness=false. Elapsed: 8.12681443s Jan 29 03:44:22.757: INFO: Pod "azuredisk-volume-tester-szbzp": Phase="Pending", Reason="", readiness=false. Elapsed: 10.119133277s ... skipping 6 lines ... Jan 29 03:44:36.757: INFO: Pod "azuredisk-volume-tester-szbzp": Phase="Pending", Reason="", readiness=false. Elapsed: 24.119137482s Jan 29 03:44:38.756: INFO: Pod "azuredisk-volume-tester-szbzp": Phase="Pending", Reason="", readiness=false. Elapsed: 26.118595325s Jan 29 03:44:40.761: INFO: Pod "azuredisk-volume-tester-szbzp": Phase="Pending", Reason="", readiness=false. Elapsed: 28.123399136s Jan 29 03:44:42.757: INFO: Pod "azuredisk-volume-tester-szbzp": Phase="Pending", Reason="", readiness=false. Elapsed: 30.12001122s Jan 29 03:44:44.756: INFO: Pod "azuredisk-volume-tester-szbzp": Phase="Succeeded", Reason="", readiness=false. Elapsed: 32.118759001s [1mSTEP:[0m Saw pod success [38;5;243m01/29/23 03:44:44.756[0m Jan 29 03:44:44.757: INFO: Pod "azuredisk-volume-tester-szbzp" satisfied condition "Succeeded or Failed" Jan 29 03:44:44.757: INFO: deleting Pod "azuredisk-8081"/"azuredisk-volume-tester-szbzp" Jan 29 03:44:44.853: INFO: Pod azuredisk-volume-tester-szbzp has the following logs: hello world [1mSTEP:[0m Deleting pod azuredisk-volume-tester-szbzp in namespace azuredisk-8081 [38;5;243m01/29/23 03:44:44.853[0m [1mSTEP:[0m validating provisioned PV [38;5;243m01/29/23 03:44:44.977[0m [1mSTEP:[0m checking the PV [38;5;243m01/29/23 03:44:45.034[0m ... skipping 50 lines ... [1mSTEP:[0m setting up the StorageClass [38;5;243m01/29/23 03:44:12.455[0m [1mSTEP:[0m creating a StorageClass [38;5;243m01/29/23 03:44:12.455[0m [1mSTEP:[0m setting up the PVC and PV [38;5;243m01/29/23 03:44:12.515[0m [1mSTEP:[0m creating a PVC [38;5;243m01/29/23 03:44:12.515[0m [1mSTEP:[0m setting up the pod [38;5;243m01/29/23 03:44:12.577[0m [1mSTEP:[0m deploying the pod [38;5;243m01/29/23 03:44:12.577[0m [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/29/23 03:44:12.637[0m Jan 29 03:44:12.637: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-szbzp" in namespace "azuredisk-8081" to be "Succeeded or Failed" Jan 29 03:44:12.696: INFO: Pod "azuredisk-volume-tester-szbzp": Phase="Pending", Reason="", readiness=false. Elapsed: 58.726867ms Jan 29 03:44:14.756: INFO: Pod "azuredisk-volume-tester-szbzp": Phase="Pending", Reason="", readiness=false. Elapsed: 2.118280967s Jan 29 03:44:16.755: INFO: Pod "azuredisk-volume-tester-szbzp": Phase="Pending", Reason="", readiness=false. Elapsed: 4.117787383s Jan 29 03:44:18.757: INFO: Pod "azuredisk-volume-tester-szbzp": Phase="Pending", Reason="", readiness=false. Elapsed: 6.119618156s Jan 29 03:44:20.764: INFO: Pod "azuredisk-volume-tester-szbzp": Phase="Pending", Reason="", readiness=false. Elapsed: 8.12681443s Jan 29 03:44:22.757: INFO: Pod "azuredisk-volume-tester-szbzp": Phase="Pending", Reason="", readiness=false. Elapsed: 10.119133277s ... skipping 6 lines ... Jan 29 03:44:36.757: INFO: Pod "azuredisk-volume-tester-szbzp": Phase="Pending", Reason="", readiness=false. Elapsed: 24.119137482s Jan 29 03:44:38.756: INFO: Pod "azuredisk-volume-tester-szbzp": Phase="Pending", Reason="", readiness=false. Elapsed: 26.118595325s Jan 29 03:44:40.761: INFO: Pod "azuredisk-volume-tester-szbzp": Phase="Pending", Reason="", readiness=false. Elapsed: 28.123399136s Jan 29 03:44:42.757: INFO: Pod "azuredisk-volume-tester-szbzp": Phase="Pending", Reason="", readiness=false. Elapsed: 30.12001122s Jan 29 03:44:44.756: INFO: Pod "azuredisk-volume-tester-szbzp": Phase="Succeeded", Reason="", readiness=false. Elapsed: 32.118759001s [1mSTEP:[0m Saw pod success [38;5;243m01/29/23 03:44:44.756[0m Jan 29 03:44:44.757: INFO: Pod "azuredisk-volume-tester-szbzp" satisfied condition "Succeeded or Failed" Jan 29 03:44:44.757: INFO: deleting Pod "azuredisk-8081"/"azuredisk-volume-tester-szbzp" Jan 29 03:44:44.853: INFO: Pod azuredisk-volume-tester-szbzp has the following logs: hello world [1mSTEP:[0m Deleting pod azuredisk-volume-tester-szbzp in namespace azuredisk-8081 [38;5;243m01/29/23 03:44:44.853[0m [1mSTEP:[0m validating provisioned PV [38;5;243m01/29/23 03:44:44.977[0m [1mSTEP:[0m checking the PV [38;5;243m01/29/23 03:44:45.034[0m ... skipping 45 lines ... Jan 29 03:45:59.339: INFO: PersistentVolumeClaim pvc-8jnrg found but phase is Pending instead of Bound. Jan 29 03:46:01.400: INFO: PersistentVolumeClaim pvc-8jnrg found and phase=Bound (4.178634804s) [1mSTEP:[0m checking the PVC [38;5;243m01/29/23 03:46:01.4[0m [1mSTEP:[0m validating provisioned PV [38;5;243m01/29/23 03:46:01.461[0m [1mSTEP:[0m checking the PV [38;5;243m01/29/23 03:46:01.542[0m [1mSTEP:[0m deploying the pod [38;5;243m01/29/23 03:46:01.542[0m [1mSTEP:[0m checking that the pods command exits with no error [38;5;243m01/29/23 03:46:01.601[0m Jan 29 03:46:01.601: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-9nmp8" in namespace "azuredisk-2540" to be "Succeeded or Failed" Jan 29 03:46:01.659: INFO: Pod "azuredisk-volume-tester-9nmp8": Phase="Pending", Reason="", readiness=false. Elapsed: 58.474242ms Jan 29 03:46:03.720: INFO: Pod "azuredisk-volume-tester-9nmp8": Phase="Pending", Reason="", readiness=false. Elapsed: 2.118693069s Jan 29 03:46:05.719: INFO: Pod "azuredisk-volume-tester-9nmp8": Phase="Pending", Reason="", readiness=false. Elapsed: 4.118071212s Jan 29 03:46:07.720: INFO: Pod "azuredisk-volume-tester-9nmp8": Phase="Pending", Reason="", readiness=false. Elapsed: 6.118836803s Jan 29 03:46:09.720: INFO: Pod "azuredisk-volume-tester-9nmp8": Phase="Pending", Reason="", readiness=false. Elapsed: 8.1186958s Jan 29 03:46:11.720: INFO: Pod "azuredisk-volume-tester-9nmp8": Phase="Pending", Reason="", readiness=false. Elapsed: 10.118758914s Jan 29 03:46:13.718: INFO: Pod "azuredisk-volume-tester-9nmp8": Phase="Pending", Reason="", readiness=false. Elapsed: 12.11730581s Jan 29 03:46:15.720: INFO: Pod "azuredisk-volume-tester-9nmp8": Phase="Pending", Reason="", readiness=false. Elapsed: 14.118973424s Jan 29 03:46:17.719: INFO: Pod "azuredisk-volume-tester-9nmp8": Phase="Pending", Reason="", readiness=false. Elapsed: 16.118053363s Jan 29 03:46:19.719: INFO: Pod "azuredisk-volume-tester-9nmp8": Phase="Pending", Reason="", readiness=false. Elapsed: 18.117877329s Jan 29 03:46:21.719: INFO: Pod "azuredisk-volume-tester-9nmp8": Phase="Pending", Reason="", readiness=false. Elapsed: 20.117554141s Jan 29 03:46:23.719: INFO: Pod "azuredisk-volume-tester-9nmp8": Phase="Succeeded", Reason="", readiness=false. Elapsed: 22.118088539s [1mSTEP:[0m Saw pod success [38;5;243m01/29/23 03:46:23.719[0m Jan 29 03:46:23.719: INFO: Pod "azuredisk-volume-tester-9nmp8" satisfied condition "Succeeded or Failed" Jan 29 03:46:23.719: INFO: deleting Pod "azuredisk-2540"/"azuredisk-volume-tester-9nmp8" Jan 29 03:46:23.806: INFO: Pod azuredisk-volume-tester-9nmp8 has the following logs: hello world [1mSTEP:[0m Deleting pod azuredisk-volume-tester-9nmp8 in namespace azuredisk-2540 [38;5;243m01/29/23 03:46:23.806[0m Jan 29 03:46:23.871: INFO: deleting PVC "azuredisk-2540"/"pvc-8jnrg" Jan 29 03:46:23.871: INFO: Deleting PersistentVolumeClaim "pvc-8jnrg" ... skipping 38 lines ... Jan 29 03:45:59.339: INFO: PersistentVolumeClaim pvc-8jnrg found but phase is Pending instead of Bound. Jan 29 03:46:01.400: INFO: PersistentVolumeClaim pvc-8jnrg found and phase=Bound (4.178634804s) [1mSTEP:[0m checking the PVC [38;5;243m01/29/23 03:46:01.4[0m [1mSTEP:[0m validating provisioned PV [38;5;243m01/29/23 03:46:01.461[0m [1mSTEP:[0m checking the PV [38;5;243m01/29/23 03:46:01.542[0m [1mSTEP:[0m deploying the pod [38;5;243m01/29/23 03:46:01.542[0m [1mSTEP:[0m checking that the pods command exits with no error [38;5;243m01/29/23 03:46:01.601[0m Jan 29 03:46:01.601: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-9nmp8" in namespace "azuredisk-2540" to be "Succeeded or Failed" Jan 29 03:46:01.659: INFO: Pod "azuredisk-volume-tester-9nmp8": Phase="Pending", Reason="", readiness=false. Elapsed: 58.474242ms Jan 29 03:46:03.720: INFO: Pod "azuredisk-volume-tester-9nmp8": Phase="Pending", Reason="", readiness=false. Elapsed: 2.118693069s Jan 29 03:46:05.719: INFO: Pod "azuredisk-volume-tester-9nmp8": Phase="Pending", Reason="", readiness=false. Elapsed: 4.118071212s Jan 29 03:46:07.720: INFO: Pod "azuredisk-volume-tester-9nmp8": Phase="Pending", Reason="", readiness=false. Elapsed: 6.118836803s Jan 29 03:46:09.720: INFO: Pod "azuredisk-volume-tester-9nmp8": Phase="Pending", Reason="", readiness=false. Elapsed: 8.1186958s Jan 29 03:46:11.720: INFO: Pod "azuredisk-volume-tester-9nmp8": Phase="Pending", Reason="", readiness=false. Elapsed: 10.118758914s Jan 29 03:46:13.718: INFO: Pod "azuredisk-volume-tester-9nmp8": Phase="Pending", Reason="", readiness=false. Elapsed: 12.11730581s Jan 29 03:46:15.720: INFO: Pod "azuredisk-volume-tester-9nmp8": Phase="Pending", Reason="", readiness=false. Elapsed: 14.118973424s Jan 29 03:46:17.719: INFO: Pod "azuredisk-volume-tester-9nmp8": Phase="Pending", Reason="", readiness=false. Elapsed: 16.118053363s Jan 29 03:46:19.719: INFO: Pod "azuredisk-volume-tester-9nmp8": Phase="Pending", Reason="", readiness=false. Elapsed: 18.117877329s Jan 29 03:46:21.719: INFO: Pod "azuredisk-volume-tester-9nmp8": Phase="Pending", Reason="", readiness=false. Elapsed: 20.117554141s Jan 29 03:46:23.719: INFO: Pod "azuredisk-volume-tester-9nmp8": Phase="Succeeded", Reason="", readiness=false. Elapsed: 22.118088539s [1mSTEP:[0m Saw pod success [38;5;243m01/29/23 03:46:23.719[0m Jan 29 03:46:23.719: INFO: Pod "azuredisk-volume-tester-9nmp8" satisfied condition "Succeeded or Failed" Jan 29 03:46:23.719: INFO: deleting Pod "azuredisk-2540"/"azuredisk-volume-tester-9nmp8" Jan 29 03:46:23.806: INFO: Pod azuredisk-volume-tester-9nmp8 has the following logs: hello world [1mSTEP:[0m Deleting pod azuredisk-volume-tester-9nmp8 in namespace azuredisk-2540 [38;5;243m01/29/23 03:46:23.806[0m Jan 29 03:46:23.871: INFO: deleting PVC "azuredisk-2540"/"pvc-8jnrg" Jan 29 03:46:23.871: INFO: Deleting PersistentVolumeClaim "pvc-8jnrg" ... skipping 30 lines ... [1mSTEP:[0m setting up the StorageClass [38;5;243m01/29/23 03:47:05.565[0m [1mSTEP:[0m creating a StorageClass [38;5;243m01/29/23 03:47:05.565[0m [1mSTEP:[0m setting up the PVC and PV [38;5;243m01/29/23 03:47:05.623[0m [1mSTEP:[0m creating a PVC [38;5;243m01/29/23 03:47:05.624[0m [1mSTEP:[0m setting up the pod [38;5;243m01/29/23 03:47:05.687[0m [1mSTEP:[0m deploying the pod [38;5;243m01/29/23 03:47:05.687[0m [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/29/23 03:47:05.746[0m Jan 29 03:47:05.746: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-mdxgk" in namespace "azuredisk-4728" to be "Succeeded or Failed" Jan 29 03:47:05.804: INFO: Pod "azuredisk-volume-tester-mdxgk": Phase="Pending", Reason="", readiness=false. Elapsed: 57.437121ms Jan 29 03:47:07.864: INFO: Pod "azuredisk-volume-tester-mdxgk": Phase="Pending", Reason="", readiness=false. Elapsed: 2.117493768s Jan 29 03:47:09.865: INFO: Pod "azuredisk-volume-tester-mdxgk": Phase="Pending", Reason="", readiness=false. Elapsed: 4.119027367s Jan 29 03:47:11.863: INFO: Pod "azuredisk-volume-tester-mdxgk": Phase="Pending", Reason="", readiness=false. Elapsed: 6.11732325s Jan 29 03:47:13.864: INFO: Pod "azuredisk-volume-tester-mdxgk": Phase="Pending", Reason="", readiness=false. Elapsed: 8.117629767s Jan 29 03:47:15.864: INFO: Pod "azuredisk-volume-tester-mdxgk": Phase="Pending", Reason="", readiness=false. Elapsed: 10.117970969s ... skipping 16 lines ... Jan 29 03:47:49.862: INFO: Pod "azuredisk-volume-tester-mdxgk": Phase="Pending", Reason="", readiness=false. Elapsed: 44.116031036s Jan 29 03:47:51.862: INFO: Pod "azuredisk-volume-tester-mdxgk": Phase="Pending", Reason="", readiness=false. Elapsed: 46.11581304s Jan 29 03:47:53.862: INFO: Pod "azuredisk-volume-tester-mdxgk": Phase="Pending", Reason="", readiness=false. Elapsed: 48.115455287s Jan 29 03:47:55.863: INFO: Pod "azuredisk-volume-tester-mdxgk": Phase="Pending", Reason="", readiness=false. Elapsed: 50.116654584s Jan 29 03:47:57.862: INFO: Pod "azuredisk-volume-tester-mdxgk": Phase="Succeeded", Reason="", readiness=false. Elapsed: 52.116082604s [1mSTEP:[0m Saw pod success [38;5;243m01/29/23 03:47:57.862[0m Jan 29 03:47:57.862: INFO: Pod "azuredisk-volume-tester-mdxgk" satisfied condition "Succeeded or Failed" Jan 29 03:47:57.862: INFO: deleting Pod "azuredisk-4728"/"azuredisk-volume-tester-mdxgk" Jan 29 03:47:57.950: INFO: Pod azuredisk-volume-tester-mdxgk has the following logs: hello world [1mSTEP:[0m Deleting pod azuredisk-volume-tester-mdxgk in namespace azuredisk-4728 [38;5;243m01/29/23 03:47:57.95[0m [1mSTEP:[0m validating provisioned PV [38;5;243m01/29/23 03:47:58.075[0m [1mSTEP:[0m checking the PV [38;5;243m01/29/23 03:47:58.133[0m ... skipping 34 lines ... [1mSTEP:[0m setting up the StorageClass [38;5;243m01/29/23 03:47:05.565[0m [1mSTEP:[0m creating a StorageClass [38;5;243m01/29/23 03:47:05.565[0m [1mSTEP:[0m setting up the PVC and PV [38;5;243m01/29/23 03:47:05.623[0m [1mSTEP:[0m creating a PVC [38;5;243m01/29/23 03:47:05.624[0m [1mSTEP:[0m setting up the pod [38;5;243m01/29/23 03:47:05.687[0m [1mSTEP:[0m deploying the pod [38;5;243m01/29/23 03:47:05.687[0m [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/29/23 03:47:05.746[0m Jan 29 03:47:05.746: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-mdxgk" in namespace "azuredisk-4728" to be "Succeeded or Failed" Jan 29 03:47:05.804: INFO: Pod "azuredisk-volume-tester-mdxgk": Phase="Pending", Reason="", readiness=false. Elapsed: 57.437121ms Jan 29 03:47:07.864: INFO: Pod "azuredisk-volume-tester-mdxgk": Phase="Pending", Reason="", readiness=false. Elapsed: 2.117493768s Jan 29 03:47:09.865: INFO: Pod "azuredisk-volume-tester-mdxgk": Phase="Pending", Reason="", readiness=false. Elapsed: 4.119027367s Jan 29 03:47:11.863: INFO: Pod "azuredisk-volume-tester-mdxgk": Phase="Pending", Reason="", readiness=false. Elapsed: 6.11732325s Jan 29 03:47:13.864: INFO: Pod "azuredisk-volume-tester-mdxgk": Phase="Pending", Reason="", readiness=false. Elapsed: 8.117629767s Jan 29 03:47:15.864: INFO: Pod "azuredisk-volume-tester-mdxgk": Phase="Pending", Reason="", readiness=false. Elapsed: 10.117970969s ... skipping 16 lines ... Jan 29 03:47:49.862: INFO: Pod "azuredisk-volume-tester-mdxgk": Phase="Pending", Reason="", readiness=false. Elapsed: 44.116031036s Jan 29 03:47:51.862: INFO: Pod "azuredisk-volume-tester-mdxgk": Phase="Pending", Reason="", readiness=false. Elapsed: 46.11581304s Jan 29 03:47:53.862: INFO: Pod "azuredisk-volume-tester-mdxgk": Phase="Pending", Reason="", readiness=false. Elapsed: 48.115455287s Jan 29 03:47:55.863: INFO: Pod "azuredisk-volume-tester-mdxgk": Phase="Pending", Reason="", readiness=false. Elapsed: 50.116654584s Jan 29 03:47:57.862: INFO: Pod "azuredisk-volume-tester-mdxgk": Phase="Succeeded", Reason="", readiness=false. Elapsed: 52.116082604s [1mSTEP:[0m Saw pod success [38;5;243m01/29/23 03:47:57.862[0m Jan 29 03:47:57.862: INFO: Pod "azuredisk-volume-tester-mdxgk" satisfied condition "Succeeded or Failed" Jan 29 03:47:57.862: INFO: deleting Pod "azuredisk-4728"/"azuredisk-volume-tester-mdxgk" Jan 29 03:47:57.950: INFO: Pod azuredisk-volume-tester-mdxgk has the following logs: hello world [1mSTEP:[0m Deleting pod azuredisk-volume-tester-mdxgk in namespace azuredisk-4728 [38;5;243m01/29/23 03:47:57.95[0m [1mSTEP:[0m validating provisioned PV [38;5;243m01/29/23 03:47:58.075[0m [1mSTEP:[0m checking the PV [38;5;243m01/29/23 03:47:58.133[0m ... skipping 35 lines ... [1mSTEP:[0m setting up the PVC and PV [38;5;243m01/29/23 03:48:44.949[0m [1mSTEP:[0m creating a PVC [38;5;243m01/29/23 03:48:44.949[0m [1mSTEP:[0m setting up the pod [38;5;243m01/29/23 03:48:45.018[0m [1mSTEP:[0m deploying the pod [38;5;243m01/29/23 03:48:45.018[0m [1mSTEP:[0m checking that the pod has 'FailedMount' event [38;5;243m01/29/23 03:48:45.078[0m Jan 29 03:49:07.199: INFO: deleting Pod "azuredisk-5466"/"azuredisk-volume-tester-7pqsn" Jan 29 03:49:07.271: INFO: Error getting logs for pod azuredisk-volume-tester-7pqsn: the server rejected our request for an unknown reason (get pods azuredisk-volume-tester-7pqsn) [1mSTEP:[0m Deleting pod azuredisk-volume-tester-7pqsn in namespace azuredisk-5466 [38;5;243m01/29/23 03:49:07.271[0m [1mSTEP:[0m validating provisioned PV [38;5;243m01/29/23 03:49:07.393[0m [1mSTEP:[0m checking the PV [38;5;243m01/29/23 03:49:07.452[0m Jan 29 03:49:07.452: INFO: deleting PVC "azuredisk-5466"/"pvc-mcr8s" Jan 29 03:49:07.452: INFO: Deleting PersistentVolumeClaim "pvc-mcr8s" [1mSTEP:[0m waiting for claim's PV "pvc-6423556c-e23b-4d33-b659-6d4fe7fc8557" to be deleted [38;5;243m01/29/23 03:49:07.513[0m ... skipping 33 lines ... [1mSTEP:[0m setting up the PVC and PV [38;5;243m01/29/23 03:48:44.949[0m [1mSTEP:[0m creating a PVC [38;5;243m01/29/23 03:48:44.949[0m [1mSTEP:[0m setting up the pod [38;5;243m01/29/23 03:48:45.018[0m [1mSTEP:[0m deploying the pod [38;5;243m01/29/23 03:48:45.018[0m [1mSTEP:[0m checking that the pod has 'FailedMount' event [38;5;243m01/29/23 03:48:45.078[0m Jan 29 03:49:07.199: INFO: deleting Pod "azuredisk-5466"/"azuredisk-volume-tester-7pqsn" Jan 29 03:49:07.271: INFO: Error getting logs for pod azuredisk-volume-tester-7pqsn: the server rejected our request for an unknown reason (get pods azuredisk-volume-tester-7pqsn) [1mSTEP:[0m Deleting pod azuredisk-volume-tester-7pqsn in namespace azuredisk-5466 [38;5;243m01/29/23 03:49:07.271[0m [1mSTEP:[0m validating provisioned PV [38;5;243m01/29/23 03:49:07.393[0m [1mSTEP:[0m checking the PV [38;5;243m01/29/23 03:49:07.452[0m Jan 29 03:49:07.452: INFO: deleting PVC "azuredisk-5466"/"pvc-mcr8s" Jan 29 03:49:07.452: INFO: Deleting PersistentVolumeClaim "pvc-mcr8s" [1mSTEP:[0m waiting for claim's PV "pvc-6423556c-e23b-4d33-b659-6d4fe7fc8557" to be deleted [38;5;243m01/29/23 03:49:07.513[0m ... skipping 30 lines ... [1mSTEP:[0m setting up the StorageClass [38;5;243m01/29/23 03:49:54.192[0m [1mSTEP:[0m creating a StorageClass [38;5;243m01/29/23 03:49:54.192[0m [1mSTEP:[0m setting up the PVC and PV [38;5;243m01/29/23 03:49:54.256[0m [1mSTEP:[0m creating a PVC [38;5;243m01/29/23 03:49:54.256[0m [1mSTEP:[0m setting up the pod [38;5;243m01/29/23 03:49:54.316[0m [1mSTEP:[0m deploying the pod [38;5;243m01/29/23 03:49:54.316[0m [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/29/23 03:49:54.382[0m Jan 29 03:49:54.382: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-2pqrb" in namespace "azuredisk-2790" to be "Succeeded or Failed" Jan 29 03:49:54.447: INFO: Pod "azuredisk-volume-tester-2pqrb": Phase="Pending", Reason="", readiness=false. Elapsed: 64.316746ms Jan 29 03:49:56.505: INFO: Pod "azuredisk-volume-tester-2pqrb": Phase="Pending", Reason="", readiness=false. Elapsed: 2.12291349s Jan 29 03:49:58.507: INFO: Pod "azuredisk-volume-tester-2pqrb": Phase="Pending", Reason="", readiness=false. Elapsed: 4.125224774s Jan 29 03:50:00.507: INFO: Pod "azuredisk-volume-tester-2pqrb": Phase="Pending", Reason="", readiness=false. Elapsed: 6.124391939s Jan 29 03:50:02.506: INFO: Pod "azuredisk-volume-tester-2pqrb": Phase="Pending", Reason="", readiness=false. Elapsed: 8.123365174s Jan 29 03:50:04.505: INFO: Pod "azuredisk-volume-tester-2pqrb": Phase="Pending", Reason="", readiness=false. Elapsed: 10.122799637s ... skipping 3 lines ... Jan 29 03:50:12.505: INFO: Pod "azuredisk-volume-tester-2pqrb": Phase="Pending", Reason="", readiness=false. Elapsed: 18.122351617s Jan 29 03:50:14.505: INFO: Pod "azuredisk-volume-tester-2pqrb": Phase="Pending", Reason="", readiness=false. Elapsed: 20.12228783s Jan 29 03:50:16.505: INFO: Pod "azuredisk-volume-tester-2pqrb": Phase="Pending", Reason="", readiness=false. Elapsed: 22.122804394s Jan 29 03:50:18.506: INFO: Pod "azuredisk-volume-tester-2pqrb": Phase="Pending", Reason="", readiness=false. Elapsed: 24.124018324s Jan 29 03:50:20.508: INFO: Pod "azuredisk-volume-tester-2pqrb": Phase="Succeeded", Reason="", readiness=false. Elapsed: 26.125911927s [1mSTEP:[0m Saw pod success [38;5;243m01/29/23 03:50:20.508[0m Jan 29 03:50:20.508: INFO: Pod "azuredisk-volume-tester-2pqrb" satisfied condition "Succeeded or Failed" Jan 29 03:50:20.508: INFO: deleting Pod "azuredisk-2790"/"azuredisk-volume-tester-2pqrb" Jan 29 03:50:20.574: INFO: Pod azuredisk-volume-tester-2pqrb has the following logs: e2e-test [1mSTEP:[0m Deleting pod azuredisk-volume-tester-2pqrb in namespace azuredisk-2790 [38;5;243m01/29/23 03:50:20.574[0m [1mSTEP:[0m validating provisioned PV [38;5;243m01/29/23 03:50:20.698[0m [1mSTEP:[0m checking the PV [38;5;243m01/29/23 03:50:20.756[0m ... skipping 39 lines ... [1mSTEP:[0m setting up the StorageClass [38;5;243m01/29/23 03:49:54.192[0m [1mSTEP:[0m creating a StorageClass [38;5;243m01/29/23 03:49:54.192[0m [1mSTEP:[0m setting up the PVC and PV [38;5;243m01/29/23 03:49:54.256[0m [1mSTEP:[0m creating a PVC [38;5;243m01/29/23 03:49:54.256[0m [1mSTEP:[0m setting up the pod [38;5;243m01/29/23 03:49:54.316[0m [1mSTEP:[0m deploying the pod [38;5;243m01/29/23 03:49:54.316[0m [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/29/23 03:49:54.382[0m Jan 29 03:49:54.382: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-2pqrb" in namespace "azuredisk-2790" to be "Succeeded or Failed" Jan 29 03:49:54.447: INFO: Pod "azuredisk-volume-tester-2pqrb": Phase="Pending", Reason="", readiness=false. Elapsed: 64.316746ms Jan 29 03:49:56.505: INFO: Pod "azuredisk-volume-tester-2pqrb": Phase="Pending", Reason="", readiness=false. Elapsed: 2.12291349s Jan 29 03:49:58.507: INFO: Pod "azuredisk-volume-tester-2pqrb": Phase="Pending", Reason="", readiness=false. Elapsed: 4.125224774s Jan 29 03:50:00.507: INFO: Pod "azuredisk-volume-tester-2pqrb": Phase="Pending", Reason="", readiness=false. Elapsed: 6.124391939s Jan 29 03:50:02.506: INFO: Pod "azuredisk-volume-tester-2pqrb": Phase="Pending", Reason="", readiness=false. Elapsed: 8.123365174s Jan 29 03:50:04.505: INFO: Pod "azuredisk-volume-tester-2pqrb": Phase="Pending", Reason="", readiness=false. Elapsed: 10.122799637s ... skipping 3 lines ... Jan 29 03:50:12.505: INFO: Pod "azuredisk-volume-tester-2pqrb": Phase="Pending", Reason="", readiness=false. Elapsed: 18.122351617s Jan 29 03:50:14.505: INFO: Pod "azuredisk-volume-tester-2pqrb": Phase="Pending", Reason="", readiness=false. Elapsed: 20.12228783s Jan 29 03:50:16.505: INFO: Pod "azuredisk-volume-tester-2pqrb": Phase="Pending", Reason="", readiness=false. Elapsed: 22.122804394s Jan 29 03:50:18.506: INFO: Pod "azuredisk-volume-tester-2pqrb": Phase="Pending", Reason="", readiness=false. Elapsed: 24.124018324s Jan 29 03:50:20.508: INFO: Pod "azuredisk-volume-tester-2pqrb": Phase="Succeeded", Reason="", readiness=false. Elapsed: 26.125911927s [1mSTEP:[0m Saw pod success [38;5;243m01/29/23 03:50:20.508[0m Jan 29 03:50:20.508: INFO: Pod "azuredisk-volume-tester-2pqrb" satisfied condition "Succeeded or Failed" Jan 29 03:50:20.508: INFO: deleting Pod "azuredisk-2790"/"azuredisk-volume-tester-2pqrb" Jan 29 03:50:20.574: INFO: Pod azuredisk-volume-tester-2pqrb has the following logs: e2e-test [1mSTEP:[0m Deleting pod azuredisk-volume-tester-2pqrb in namespace azuredisk-2790 [38;5;243m01/29/23 03:50:20.574[0m [1mSTEP:[0m validating provisioned PV [38;5;243m01/29/23 03:50:20.698[0m [1mSTEP:[0m checking the PV [38;5;243m01/29/23 03:50:20.756[0m ... skipping 43 lines ... [1mSTEP:[0m creating volume in external rg azuredisk-csi-driver-test-3856d907-9f88-11ed-987b-d279894c5862 [38;5;243m01/29/23 03:51:33.809[0m [1mSTEP:[0m setting up the StorageClass [38;5;243m01/29/23 03:51:33.809[0m [1mSTEP:[0m creating a StorageClass [38;5;243m01/29/23 03:51:33.809[0m [1mSTEP:[0m setting up the PVC and PV [38;5;243m01/29/23 03:51:33.868[0m [1mSTEP:[0m creating a PVC [38;5;243m01/29/23 03:51:33.868[0m [1mSTEP:[0m deploying the pod [38;5;243m01/29/23 03:51:33.932[0m [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/29/23 03:51:33.991[0m Jan 29 03:51:33.991: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-zfxck" in namespace "azuredisk-5356" to be "Succeeded or Failed" Jan 29 03:51:34.049: INFO: Pod "azuredisk-volume-tester-zfxck": Phase="Pending", Reason="", readiness=false. Elapsed: 57.736672ms Jan 29 03:51:36.109: INFO: Pod "azuredisk-volume-tester-zfxck": Phase="Pending", Reason="", readiness=false. Elapsed: 2.117512219s Jan 29 03:51:38.110: INFO: Pod "azuredisk-volume-tester-zfxck": Phase="Pending", Reason="", readiness=false. Elapsed: 4.118131392s Jan 29 03:51:40.108: INFO: Pod "azuredisk-volume-tester-zfxck": Phase="Pending", Reason="", readiness=false. Elapsed: 6.116223089s Jan 29 03:51:42.109: INFO: Pod "azuredisk-volume-tester-zfxck": Phase="Pending", Reason="", readiness=false. Elapsed: 8.117176435s Jan 29 03:51:44.109: INFO: Pod "azuredisk-volume-tester-zfxck": Phase="Pending", Reason="", readiness=false. Elapsed: 10.117585424s ... skipping 2 lines ... Jan 29 03:51:50.109: INFO: Pod "azuredisk-volume-tester-zfxck": Phase="Pending", Reason="", readiness=false. Elapsed: 16.117121787s Jan 29 03:51:52.110: INFO: Pod "azuredisk-volume-tester-zfxck": Phase="Pending", Reason="", readiness=false. Elapsed: 18.118532519s Jan 29 03:51:54.109: INFO: Pod "azuredisk-volume-tester-zfxck": Phase="Pending", Reason="", readiness=false. Elapsed: 20.118019476s Jan 29 03:51:56.108: INFO: Pod "azuredisk-volume-tester-zfxck": Phase="Pending", Reason="", readiness=false. Elapsed: 22.116518666s Jan 29 03:51:58.109: INFO: Pod "azuredisk-volume-tester-zfxck": Phase="Succeeded", Reason="", readiness=false. Elapsed: 24.117800862s [1mSTEP:[0m Saw pod success [38;5;243m01/29/23 03:51:58.11[0m Jan 29 03:51:58.110: INFO: Pod "azuredisk-volume-tester-zfxck" satisfied condition "Succeeded or Failed" Jan 29 03:51:58.110: INFO: deleting Pod "azuredisk-5356"/"azuredisk-volume-tester-zfxck" Jan 29 03:51:58.217: INFO: Pod azuredisk-volume-tester-zfxck has the following logs: hello world [1mSTEP:[0m Deleting pod azuredisk-volume-tester-zfxck in namespace azuredisk-5356 [38;5;243m01/29/23 03:51:58.217[0m [1mSTEP:[0m validating provisioned PV [38;5;243m01/29/23 03:51:58.337[0m [1mSTEP:[0m checking the PV [38;5;243m01/29/23 03:51:58.394[0m ... skipping 37 lines ... [1mSTEP:[0m creating volume in external rg azuredisk-csi-driver-test-3856d907-9f88-11ed-987b-d279894c5862 [38;5;243m01/29/23 03:51:33.809[0m [1mSTEP:[0m setting up the StorageClass [38;5;243m01/29/23 03:51:33.809[0m [1mSTEP:[0m creating a StorageClass [38;5;243m01/29/23 03:51:33.809[0m [1mSTEP:[0m setting up the PVC and PV [38;5;243m01/29/23 03:51:33.868[0m [1mSTEP:[0m creating a PVC [38;5;243m01/29/23 03:51:33.868[0m [1mSTEP:[0m deploying the pod [38;5;243m01/29/23 03:51:33.932[0m [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/29/23 03:51:33.991[0m Jan 29 03:51:33.991: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-zfxck" in namespace "azuredisk-5356" to be "Succeeded or Failed" Jan 29 03:51:34.049: INFO: Pod "azuredisk-volume-tester-zfxck": Phase="Pending", Reason="", readiness=false. Elapsed: 57.736672ms Jan 29 03:51:36.109: INFO: Pod "azuredisk-volume-tester-zfxck": Phase="Pending", Reason="", readiness=false. Elapsed: 2.117512219s Jan 29 03:51:38.110: INFO: Pod "azuredisk-volume-tester-zfxck": Phase="Pending", Reason="", readiness=false. Elapsed: 4.118131392s Jan 29 03:51:40.108: INFO: Pod "azuredisk-volume-tester-zfxck": Phase="Pending", Reason="", readiness=false. Elapsed: 6.116223089s Jan 29 03:51:42.109: INFO: Pod "azuredisk-volume-tester-zfxck": Phase="Pending", Reason="", readiness=false. Elapsed: 8.117176435s Jan 29 03:51:44.109: INFO: Pod "azuredisk-volume-tester-zfxck": Phase="Pending", Reason="", readiness=false. Elapsed: 10.117585424s ... skipping 2 lines ... Jan 29 03:51:50.109: INFO: Pod "azuredisk-volume-tester-zfxck": Phase="Pending", Reason="", readiness=false. Elapsed: 16.117121787s Jan 29 03:51:52.110: INFO: Pod "azuredisk-volume-tester-zfxck": Phase="Pending", Reason="", readiness=false. Elapsed: 18.118532519s Jan 29 03:51:54.109: INFO: Pod "azuredisk-volume-tester-zfxck": Phase="Pending", Reason="", readiness=false. Elapsed: 20.118019476s Jan 29 03:51:56.108: INFO: Pod "azuredisk-volume-tester-zfxck": Phase="Pending", Reason="", readiness=false. Elapsed: 22.116518666s Jan 29 03:51:58.109: INFO: Pod "azuredisk-volume-tester-zfxck": Phase="Succeeded", Reason="", readiness=false. Elapsed: 24.117800862s [1mSTEP:[0m Saw pod success [38;5;243m01/29/23 03:51:58.11[0m Jan 29 03:51:58.110: INFO: Pod "azuredisk-volume-tester-zfxck" satisfied condition "Succeeded or Failed" Jan 29 03:51:58.110: INFO: deleting Pod "azuredisk-5356"/"azuredisk-volume-tester-zfxck" Jan 29 03:51:58.217: INFO: Pod azuredisk-volume-tester-zfxck has the following logs: hello world [1mSTEP:[0m Deleting pod azuredisk-volume-tester-zfxck in namespace azuredisk-5356 [38;5;243m01/29/23 03:51:58.217[0m [1mSTEP:[0m validating provisioned PV [38;5;243m01/29/23 03:51:58.337[0m [1mSTEP:[0m checking the PV [38;5;243m01/29/23 03:51:58.394[0m ... skipping 44 lines ... [1mSTEP:[0m creating volume in external rg azuredisk-csi-driver-test-6a857c0f-9f88-11ed-987b-d279894c5862 [38;5;243m01/29/23 03:52:57.44[0m [1mSTEP:[0m setting up the StorageClass [38;5;243m01/29/23 03:52:57.441[0m [1mSTEP:[0m creating a StorageClass [38;5;243m01/29/23 03:52:57.441[0m [1mSTEP:[0m setting up the PVC and PV [38;5;243m01/29/23 03:52:57.502[0m [1mSTEP:[0m creating a PVC [38;5;243m01/29/23 03:52:57.502[0m [1mSTEP:[0m deploying the pod [38;5;243m01/29/23 03:52:57.561[0m [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/29/23 03:52:57.621[0m Jan 29 03:52:57.621: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-k26jm" in namespace "azuredisk-5194" to be "Succeeded or Failed" Jan 29 03:52:57.679: INFO: Pod "azuredisk-volume-tester-k26jm": Phase="Pending", Reason="", readiness=false. Elapsed: 57.626377ms Jan 29 03:52:59.738: INFO: Pod "azuredisk-volume-tester-k26jm": Phase="Pending", Reason="", readiness=false. Elapsed: 2.116576091s Jan 29 03:53:01.738: INFO: Pod "azuredisk-volume-tester-k26jm": Phase="Pending", Reason="", readiness=false. Elapsed: 4.117013947s Jan 29 03:53:03.738: INFO: Pod "azuredisk-volume-tester-k26jm": Phase="Pending", Reason="", readiness=false. Elapsed: 6.116710708s Jan 29 03:53:05.739: INFO: Pod "azuredisk-volume-tester-k26jm": Phase="Pending", Reason="", readiness=false. Elapsed: 8.118215185s Jan 29 03:53:07.740: INFO: Pod "azuredisk-volume-tester-k26jm": Phase="Pending", Reason="", readiness=false. Elapsed: 10.118457578s ... skipping 2 lines ... Jan 29 03:53:13.738: INFO: Pod "azuredisk-volume-tester-k26jm": Phase="Pending", Reason="", readiness=false. Elapsed: 16.116707141s Jan 29 03:53:15.737: INFO: Pod "azuredisk-volume-tester-k26jm": Phase="Pending", Reason="", readiness=false. Elapsed: 18.116199634s Jan 29 03:53:17.738: INFO: Pod "azuredisk-volume-tester-k26jm": Phase="Pending", Reason="", readiness=false. Elapsed: 20.116733016s Jan 29 03:53:19.738: INFO: Pod "azuredisk-volume-tester-k26jm": Phase="Pending", Reason="", readiness=false. Elapsed: 22.117222567s Jan 29 03:53:21.738: INFO: Pod "azuredisk-volume-tester-k26jm": Phase="Succeeded", Reason="", readiness=false. Elapsed: 24.116386243s [1mSTEP:[0m Saw pod success [38;5;243m01/29/23 03:53:21.738[0m Jan 29 03:53:21.738: INFO: Pod "azuredisk-volume-tester-k26jm" satisfied condition "Succeeded or Failed" Jan 29 03:53:21.738: INFO: deleting Pod "azuredisk-5194"/"azuredisk-volume-tester-k26jm" Jan 29 03:53:21.799: INFO: Pod azuredisk-volume-tester-k26jm has the following logs: hello world hello world [1mSTEP:[0m Deleting pod azuredisk-volume-tester-k26jm in namespace azuredisk-5194 [38;5;243m01/29/23 03:53:21.799[0m [1mSTEP:[0m validating provisioned PV [38;5;243m01/29/23 03:53:21.922[0m ... skipping 57 lines ... [1mSTEP:[0m creating volume in external rg azuredisk-csi-driver-test-6a857c0f-9f88-11ed-987b-d279894c5862 [38;5;243m01/29/23 03:52:57.44[0m [1mSTEP:[0m setting up the StorageClass [38;5;243m01/29/23 03:52:57.441[0m [1mSTEP:[0m creating a StorageClass [38;5;243m01/29/23 03:52:57.441[0m [1mSTEP:[0m setting up the PVC and PV [38;5;243m01/29/23 03:52:57.502[0m [1mSTEP:[0m creating a PVC [38;5;243m01/29/23 03:52:57.502[0m [1mSTEP:[0m deploying the pod [38;5;243m01/29/23 03:52:57.561[0m [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/29/23 03:52:57.621[0m Jan 29 03:52:57.621: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-k26jm" in namespace "azuredisk-5194" to be "Succeeded or Failed" Jan 29 03:52:57.679: INFO: Pod "azuredisk-volume-tester-k26jm": Phase="Pending", Reason="", readiness=false. Elapsed: 57.626377ms Jan 29 03:52:59.738: INFO: Pod "azuredisk-volume-tester-k26jm": Phase="Pending", Reason="", readiness=false. Elapsed: 2.116576091s Jan 29 03:53:01.738: INFO: Pod "azuredisk-volume-tester-k26jm": Phase="Pending", Reason="", readiness=false. Elapsed: 4.117013947s Jan 29 03:53:03.738: INFO: Pod "azuredisk-volume-tester-k26jm": Phase="Pending", Reason="", readiness=false. Elapsed: 6.116710708s Jan 29 03:53:05.739: INFO: Pod "azuredisk-volume-tester-k26jm": Phase="Pending", Reason="", readiness=false. Elapsed: 8.118215185s Jan 29 03:53:07.740: INFO: Pod "azuredisk-volume-tester-k26jm": Phase="Pending", Reason="", readiness=false. Elapsed: 10.118457578s ... skipping 2 lines ... Jan 29 03:53:13.738: INFO: Pod "azuredisk-volume-tester-k26jm": Phase="Pending", Reason="", readiness=false. Elapsed: 16.116707141s Jan 29 03:53:15.737: INFO: Pod "azuredisk-volume-tester-k26jm": Phase="Pending", Reason="", readiness=false. Elapsed: 18.116199634s Jan 29 03:53:17.738: INFO: Pod "azuredisk-volume-tester-k26jm": Phase="Pending", Reason="", readiness=false. Elapsed: 20.116733016s Jan 29 03:53:19.738: INFO: Pod "azuredisk-volume-tester-k26jm": Phase="Pending", Reason="", readiness=false. Elapsed: 22.117222567s Jan 29 03:53:21.738: INFO: Pod "azuredisk-volume-tester-k26jm": Phase="Succeeded", Reason="", readiness=false. Elapsed: 24.116386243s [1mSTEP:[0m Saw pod success [38;5;243m01/29/23 03:53:21.738[0m Jan 29 03:53:21.738: INFO: Pod "azuredisk-volume-tester-k26jm" satisfied condition "Succeeded or Failed" Jan 29 03:53:21.738: INFO: deleting Pod "azuredisk-5194"/"azuredisk-volume-tester-k26jm" Jan 29 03:53:21.799: INFO: Pod azuredisk-volume-tester-k26jm has the following logs: hello world hello world [1mSTEP:[0m Deleting pod azuredisk-volume-tester-k26jm in namespace azuredisk-5194 [38;5;243m01/29/23 03:53:21.799[0m [1mSTEP:[0m validating provisioned PV [38;5;243m01/29/23 03:53:21.922[0m ... skipping 47 lines ... [1mSTEP:[0m setting up the StorageClass [38;5;243m01/29/23 03:54:45.683[0m [1mSTEP:[0m creating a StorageClass [38;5;243m01/29/23 03:54:45.683[0m [1mSTEP:[0m setting up the PVC and PV [38;5;243m01/29/23 03:54:45.742[0m [1mSTEP:[0m creating a PVC [38;5;243m01/29/23 03:54:45.742[0m [1mSTEP:[0m setting up the pod [38;5;243m01/29/23 03:54:45.803[0m [1mSTEP:[0m deploying the pod [38;5;243m01/29/23 03:54:45.803[0m [1mSTEP:[0m checking that the pod's command exits with an error [38;5;243m01/29/23 03:54:45.863[0m Jan 29 03:54:45.864: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-sxgsn" in namespace "azuredisk-1353" to be "Error status code" Jan 29 03:54:45.921: INFO: Pod "azuredisk-volume-tester-sxgsn": Phase="Pending", Reason="", readiness=false. Elapsed: 57.927525ms Jan 29 03:54:47.980: INFO: Pod "azuredisk-volume-tester-sxgsn": Phase="Pending", Reason="", readiness=false. Elapsed: 2.116020733s Jan 29 03:54:49.979: INFO: Pod "azuredisk-volume-tester-sxgsn": Phase="Pending", Reason="", readiness=false. Elapsed: 4.115755603s Jan 29 03:54:51.980: INFO: Pod "azuredisk-volume-tester-sxgsn": Phase="Pending", Reason="", readiness=false. Elapsed: 6.116200314s Jan 29 03:54:53.981: INFO: Pod "azuredisk-volume-tester-sxgsn": Phase="Pending", Reason="", readiness=false. Elapsed: 8.117066572s Jan 29 03:54:55.982: INFO: Pod "azuredisk-volume-tester-sxgsn": Phase="Pending", Reason="", readiness=false. Elapsed: 10.118883519s ... skipping 24 lines ... Jan 29 03:55:45.982: INFO: Pod "azuredisk-volume-tester-sxgsn": Phase="Pending", Reason="", readiness=false. Elapsed: 1m0.118473387s Jan 29 03:55:47.982: INFO: Pod "azuredisk-volume-tester-sxgsn": Phase="Pending", Reason="", readiness=false. Elapsed: 1m2.118532737s Jan 29 03:55:49.982: INFO: Pod "azuredisk-volume-tester-sxgsn": Phase="Pending", Reason="", readiness=false. Elapsed: 1m4.1185153s Jan 29 03:55:51.980: INFO: Pod "azuredisk-volume-tester-sxgsn": Phase="Pending", Reason="", readiness=false. Elapsed: 1m6.116723944s Jan 29 03:55:53.980: INFO: Pod "azuredisk-volume-tester-sxgsn": Phase="Pending", Reason="", readiness=false. Elapsed: 1m8.116642643s Jan 29 03:55:55.982: INFO: Pod "azuredisk-volume-tester-sxgsn": Phase="Pending", Reason="", readiness=false. Elapsed: 1m10.118659894s Jan 29 03:55:57.980: INFO: Pod "azuredisk-volume-tester-sxgsn": Phase="Failed", Reason="", readiness=false. Elapsed: 1m12.116069116s [1mSTEP:[0m Saw pod failure [38;5;243m01/29/23 03:55:57.98[0m Jan 29 03:55:57.980: INFO: Pod "azuredisk-volume-tester-sxgsn" satisfied condition "Error status code" [1mSTEP:[0m checking that pod logs contain expected message [38;5;243m01/29/23 03:55:57.98[0m Jan 29 03:55:58.078: INFO: deleting Pod "azuredisk-1353"/"azuredisk-volume-tester-sxgsn" Jan 29 03:55:58.139: INFO: Pod azuredisk-volume-tester-sxgsn has the following logs: touch: /mnt/test-1/data: Read-only file system [1mSTEP:[0m Deleting pod azuredisk-volume-tester-sxgsn in namespace azuredisk-1353 [38;5;243m01/29/23 03:55:58.139[0m [1mSTEP:[0m validating provisioned PV [38;5;243m01/29/23 03:55:58.26[0m ... skipping 34 lines ... [1mSTEP:[0m setting up the StorageClass [38;5;243m01/29/23 03:54:45.683[0m [1mSTEP:[0m creating a StorageClass [38;5;243m01/29/23 03:54:45.683[0m [1mSTEP:[0m setting up the PVC and PV [38;5;243m01/29/23 03:54:45.742[0m [1mSTEP:[0m creating a PVC [38;5;243m01/29/23 03:54:45.742[0m [1mSTEP:[0m setting up the pod [38;5;243m01/29/23 03:54:45.803[0m [1mSTEP:[0m deploying the pod [38;5;243m01/29/23 03:54:45.803[0m [1mSTEP:[0m checking that the pod's command exits with an error [38;5;243m01/29/23 03:54:45.863[0m Jan 29 03:54:45.864: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-sxgsn" in namespace "azuredisk-1353" to be "Error status code" Jan 29 03:54:45.921: INFO: Pod "azuredisk-volume-tester-sxgsn": Phase="Pending", Reason="", readiness=false. Elapsed: 57.927525ms Jan 29 03:54:47.980: INFO: Pod "azuredisk-volume-tester-sxgsn": Phase="Pending", Reason="", readiness=false. Elapsed: 2.116020733s Jan 29 03:54:49.979: INFO: Pod "azuredisk-volume-tester-sxgsn": Phase="Pending", Reason="", readiness=false. Elapsed: 4.115755603s Jan 29 03:54:51.980: INFO: Pod "azuredisk-volume-tester-sxgsn": Phase="Pending", Reason="", readiness=false. Elapsed: 6.116200314s Jan 29 03:54:53.981: INFO: Pod "azuredisk-volume-tester-sxgsn": Phase="Pending", Reason="", readiness=false. Elapsed: 8.117066572s Jan 29 03:54:55.982: INFO: Pod "azuredisk-volume-tester-sxgsn": Phase="Pending", Reason="", readiness=false. Elapsed: 10.118883519s ... skipping 24 lines ... Jan 29 03:55:45.982: INFO: Pod "azuredisk-volume-tester-sxgsn": Phase="Pending", Reason="", readiness=false. Elapsed: 1m0.118473387s Jan 29 03:55:47.982: INFO: Pod "azuredisk-volume-tester-sxgsn": Phase="Pending", Reason="", readiness=false. Elapsed: 1m2.118532737s Jan 29 03:55:49.982: INFO: Pod "azuredisk-volume-tester-sxgsn": Phase="Pending", Reason="", readiness=false. Elapsed: 1m4.1185153s Jan 29 03:55:51.980: INFO: Pod "azuredisk-volume-tester-sxgsn": Phase="Pending", Reason="", readiness=false. Elapsed: 1m6.116723944s Jan 29 03:55:53.980: INFO: Pod "azuredisk-volume-tester-sxgsn": Phase="Pending", Reason="", readiness=false. Elapsed: 1m8.116642643s Jan 29 03:55:55.982: INFO: Pod "azuredisk-volume-tester-sxgsn": Phase="Pending", Reason="", readiness=false. Elapsed: 1m10.118659894s Jan 29 03:55:57.980: INFO: Pod "azuredisk-volume-tester-sxgsn": Phase="Failed", Reason="", readiness=false. Elapsed: 1m12.116069116s [1mSTEP:[0m Saw pod failure [38;5;243m01/29/23 03:55:57.98[0m Jan 29 03:55:57.980: INFO: Pod "azuredisk-volume-tester-sxgsn" satisfied condition "Error status code" [1mSTEP:[0m checking that pod logs contain expected message [38;5;243m01/29/23 03:55:57.98[0m Jan 29 03:55:58.078: INFO: deleting Pod "azuredisk-1353"/"azuredisk-volume-tester-sxgsn" Jan 29 03:55:58.139: INFO: Pod azuredisk-volume-tester-sxgsn has the following logs: touch: /mnt/test-1/data: Read-only file system [1mSTEP:[0m Deleting pod azuredisk-volume-tester-sxgsn in namespace azuredisk-1353 [38;5;243m01/29/23 03:55:58.139[0m [1mSTEP:[0m validating provisioned PV [38;5;243m01/29/23 03:55:58.26[0m ... skipping 659 lines ... [1mSTEP:[0m setting up the StorageClass [38;5;243m01/29/23 04:04:09.241[0m [1mSTEP:[0m creating a StorageClass [38;5;243m01/29/23 04:04:09.241[0m [1mSTEP:[0m setting up the PVC and PV [38;5;243m01/29/23 04:04:09.302[0m [1mSTEP:[0m creating a PVC [38;5;243m01/29/23 04:04:09.302[0m [1mSTEP:[0m setting up the pod [38;5;243m01/29/23 04:04:09.364[0m [1mSTEP:[0m deploying the pod [38;5;243m01/29/23 04:04:09.365[0m [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/29/23 04:04:09.431[0m Jan 29 04:04:09.431: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-jbmsp" in namespace "azuredisk-59" to be "Succeeded or Failed" Jan 29 04:04:09.490: INFO: Pod "azuredisk-volume-tester-jbmsp": Phase="Pending", Reason="", readiness=false. Elapsed: 58.273077ms Jan 29 04:04:11.548: INFO: Pod "azuredisk-volume-tester-jbmsp": Phase="Pending", Reason="", readiness=false. Elapsed: 2.116999902s Jan 29 04:04:13.549: INFO: Pod "azuredisk-volume-tester-jbmsp": Phase="Pending", Reason="", readiness=false. Elapsed: 4.117592591s Jan 29 04:04:15.549: INFO: Pod "azuredisk-volume-tester-jbmsp": Phase="Pending", Reason="", readiness=false. Elapsed: 6.117250492s Jan 29 04:04:17.551: INFO: Pod "azuredisk-volume-tester-jbmsp": Phase="Pending", Reason="", readiness=false. Elapsed: 8.119221734s Jan 29 04:04:19.550: INFO: Pod "azuredisk-volume-tester-jbmsp": Phase="Pending", Reason="", readiness=false. Elapsed: 10.118867402s ... skipping 2 lines ... Jan 29 04:04:25.548: INFO: Pod "azuredisk-volume-tester-jbmsp": Phase="Pending", Reason="", readiness=false. Elapsed: 16.116340692s Jan 29 04:04:27.548: INFO: Pod "azuredisk-volume-tester-jbmsp": Phase="Pending", Reason="", readiness=false. Elapsed: 18.11688976s Jan 29 04:04:29.549: INFO: Pod "azuredisk-volume-tester-jbmsp": Phase="Pending", Reason="", readiness=false. Elapsed: 20.117211863s Jan 29 04:04:31.548: INFO: Pod "azuredisk-volume-tester-jbmsp": Phase="Pending", Reason="", readiness=false. Elapsed: 22.116942704s Jan 29 04:04:33.549: INFO: Pod "azuredisk-volume-tester-jbmsp": Phase="Succeeded", Reason="", readiness=false. Elapsed: 24.117794654s [1mSTEP:[0m Saw pod success [38;5;243m01/29/23 04:04:33.549[0m Jan 29 04:04:33.550: INFO: Pod "azuredisk-volume-tester-jbmsp" satisfied condition "Succeeded or Failed" [1mSTEP:[0m sleep 5s and then clone volume [38;5;243m01/29/23 04:04:33.55[0m [1mSTEP:[0m cloning existing volume [38;5;243m01/29/23 04:04:38.55[0m [1mSTEP:[0m setting up the PVC and PV [38;5;243m01/29/23 04:04:38.666[0m [1mSTEP:[0m creating a PVC [38;5;243m01/29/23 04:04:38.666[0m [1mSTEP:[0m setting up the pod [38;5;243m01/29/23 04:04:38.728[0m [1mSTEP:[0m deploying a second pod with cloned volume [38;5;243m01/29/23 04:04:38.728[0m [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/29/23 04:04:38.787[0m Jan 29 04:04:38.788: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-dt2gf" in namespace "azuredisk-59" to be "Succeeded or Failed" Jan 29 04:04:38.846: INFO: Pod "azuredisk-volume-tester-dt2gf": Phase="Pending", Reason="", readiness=false. Elapsed: 58.010767ms Jan 29 04:04:40.907: INFO: Pod "azuredisk-volume-tester-dt2gf": Phase="Pending", Reason="", readiness=false. Elapsed: 2.11975456s Jan 29 04:04:42.905: INFO: Pod "azuredisk-volume-tester-dt2gf": Phase="Pending", Reason="", readiness=false. Elapsed: 4.11741617s Jan 29 04:04:44.906: INFO: Pod "azuredisk-volume-tester-dt2gf": Phase="Pending", Reason="", readiness=false. Elapsed: 6.118169487s Jan 29 04:04:46.907: INFO: Pod "azuredisk-volume-tester-dt2gf": Phase="Pending", Reason="", readiness=false. Elapsed: 8.118959985s Jan 29 04:04:48.905: INFO: Pod "azuredisk-volume-tester-dt2gf": Phase="Pending", Reason="", readiness=false. Elapsed: 10.117582612s ... skipping 9 lines ... Jan 29 04:05:08.910: INFO: Pod "azuredisk-volume-tester-dt2gf": Phase="Pending", Reason="", readiness=false. Elapsed: 30.122069825s Jan 29 04:05:10.904: INFO: Pod "azuredisk-volume-tester-dt2gf": Phase="Pending", Reason="", readiness=false. Elapsed: 32.116120157s Jan 29 04:05:12.904: INFO: Pod "azuredisk-volume-tester-dt2gf": Phase="Pending", Reason="", readiness=false. Elapsed: 34.116542543s Jan 29 04:05:14.905: INFO: Pod "azuredisk-volume-tester-dt2gf": Phase="Pending", Reason="", readiness=false. Elapsed: 36.117546058s Jan 29 04:05:16.904: INFO: Pod "azuredisk-volume-tester-dt2gf": Phase="Succeeded", Reason="", readiness=false. Elapsed: 38.116612919s [1mSTEP:[0m Saw pod success [38;5;243m01/29/23 04:05:16.904[0m Jan 29 04:05:16.904: INFO: Pod "azuredisk-volume-tester-dt2gf" satisfied condition "Succeeded or Failed" Jan 29 04:05:16.904: INFO: deleting Pod "azuredisk-59"/"azuredisk-volume-tester-dt2gf" Jan 29 04:05:16.990: INFO: Pod azuredisk-volume-tester-dt2gf has the following logs: hello world [1mSTEP:[0m Deleting pod azuredisk-volume-tester-dt2gf in namespace azuredisk-59 [38;5;243m01/29/23 04:05:16.99[0m [1mSTEP:[0m validating provisioned PV [38;5;243m01/29/23 04:05:17.112[0m [1mSTEP:[0m checking the PV [38;5;243m01/29/23 04:05:17.17[0m ... skipping 47 lines ... [1mSTEP:[0m setting up the StorageClass [38;5;243m01/29/23 04:04:09.241[0m [1mSTEP:[0m creating a StorageClass [38;5;243m01/29/23 04:04:09.241[0m [1mSTEP:[0m setting up the PVC and PV [38;5;243m01/29/23 04:04:09.302[0m [1mSTEP:[0m creating a PVC [38;5;243m01/29/23 04:04:09.302[0m [1mSTEP:[0m setting up the pod [38;5;243m01/29/23 04:04:09.364[0m [1mSTEP:[0m deploying the pod [38;5;243m01/29/23 04:04:09.365[0m [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/29/23 04:04:09.431[0m Jan 29 04:04:09.431: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-jbmsp" in namespace "azuredisk-59" to be "Succeeded or Failed" Jan 29 04:04:09.490: INFO: Pod "azuredisk-volume-tester-jbmsp": Phase="Pending", Reason="", readiness=false. Elapsed: 58.273077ms Jan 29 04:04:11.548: INFO: Pod "azuredisk-volume-tester-jbmsp": Phase="Pending", Reason="", readiness=false. Elapsed: 2.116999902s Jan 29 04:04:13.549: INFO: Pod "azuredisk-volume-tester-jbmsp": Phase="Pending", Reason="", readiness=false. Elapsed: 4.117592591s Jan 29 04:04:15.549: INFO: Pod "azuredisk-volume-tester-jbmsp": Phase="Pending", Reason="", readiness=false. Elapsed: 6.117250492s Jan 29 04:04:17.551: INFO: Pod "azuredisk-volume-tester-jbmsp": Phase="Pending", Reason="", readiness=false. Elapsed: 8.119221734s Jan 29 04:04:19.550: INFO: Pod "azuredisk-volume-tester-jbmsp": Phase="Pending", Reason="", readiness=false. Elapsed: 10.118867402s ... skipping 2 lines ... Jan 29 04:04:25.548: INFO: Pod "azuredisk-volume-tester-jbmsp": Phase="Pending", Reason="", readiness=false. Elapsed: 16.116340692s Jan 29 04:04:27.548: INFO: Pod "azuredisk-volume-tester-jbmsp": Phase="Pending", Reason="", readiness=false. Elapsed: 18.11688976s Jan 29 04:04:29.549: INFO: Pod "azuredisk-volume-tester-jbmsp": Phase="Pending", Reason="", readiness=false. Elapsed: 20.117211863s Jan 29 04:04:31.548: INFO: Pod "azuredisk-volume-tester-jbmsp": Phase="Pending", Reason="", readiness=false. Elapsed: 22.116942704s Jan 29 04:04:33.549: INFO: Pod "azuredisk-volume-tester-jbmsp": Phase="Succeeded", Reason="", readiness=false. Elapsed: 24.117794654s [1mSTEP:[0m Saw pod success [38;5;243m01/29/23 04:04:33.549[0m Jan 29 04:04:33.550: INFO: Pod "azuredisk-volume-tester-jbmsp" satisfied condition "Succeeded or Failed" [1mSTEP:[0m sleep 5s and then clone volume [38;5;243m01/29/23 04:04:33.55[0m [1mSTEP:[0m cloning existing volume [38;5;243m01/29/23 04:04:38.55[0m [1mSTEP:[0m setting up the PVC and PV [38;5;243m01/29/23 04:04:38.666[0m [1mSTEP:[0m creating a PVC [38;5;243m01/29/23 04:04:38.666[0m [1mSTEP:[0m setting up the pod [38;5;243m01/29/23 04:04:38.728[0m [1mSTEP:[0m deploying a second pod with cloned volume [38;5;243m01/29/23 04:04:38.728[0m [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/29/23 04:04:38.787[0m Jan 29 04:04:38.788: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-dt2gf" in namespace "azuredisk-59" to be "Succeeded or Failed" Jan 29 04:04:38.846: INFO: Pod "azuredisk-volume-tester-dt2gf": Phase="Pending", Reason="", readiness=false. Elapsed: 58.010767ms Jan 29 04:04:40.907: INFO: Pod "azuredisk-volume-tester-dt2gf": Phase="Pending", Reason="", readiness=false. Elapsed: 2.11975456s Jan 29 04:04:42.905: INFO: Pod "azuredisk-volume-tester-dt2gf": Phase="Pending", Reason="", readiness=false. Elapsed: 4.11741617s Jan 29 04:04:44.906: INFO: Pod "azuredisk-volume-tester-dt2gf": Phase="Pending", Reason="", readiness=false. Elapsed: 6.118169487s Jan 29 04:04:46.907: INFO: Pod "azuredisk-volume-tester-dt2gf": Phase="Pending", Reason="", readiness=false. Elapsed: 8.118959985s Jan 29 04:04:48.905: INFO: Pod "azuredisk-volume-tester-dt2gf": Phase="Pending", Reason="", readiness=false. Elapsed: 10.117582612s ... skipping 9 lines ... Jan 29 04:05:08.910: INFO: Pod "azuredisk-volume-tester-dt2gf": Phase="Pending", Reason="", readiness=false. Elapsed: 30.122069825s Jan 29 04:05:10.904: INFO: Pod "azuredisk-volume-tester-dt2gf": Phase="Pending", Reason="", readiness=false. Elapsed: 32.116120157s Jan 29 04:05:12.904: INFO: Pod "azuredisk-volume-tester-dt2gf": Phase="Pending", Reason="", readiness=false. Elapsed: 34.116542543s Jan 29 04:05:14.905: INFO: Pod "azuredisk-volume-tester-dt2gf": Phase="Pending", Reason="", readiness=false. Elapsed: 36.117546058s Jan 29 04:05:16.904: INFO: Pod "azuredisk-volume-tester-dt2gf": Phase="Succeeded", Reason="", readiness=false. Elapsed: 38.116612919s [1mSTEP:[0m Saw pod success [38;5;243m01/29/23 04:05:16.904[0m Jan 29 04:05:16.904: INFO: Pod "azuredisk-volume-tester-dt2gf" satisfied condition "Succeeded or Failed" Jan 29 04:05:16.904: INFO: deleting Pod "azuredisk-59"/"azuredisk-volume-tester-dt2gf" Jan 29 04:05:16.990: INFO: Pod azuredisk-volume-tester-dt2gf has the following logs: hello world [1mSTEP:[0m Deleting pod azuredisk-volume-tester-dt2gf in namespace azuredisk-59 [38;5;243m01/29/23 04:05:16.99[0m [1mSTEP:[0m validating provisioned PV [38;5;243m01/29/23 04:05:17.112[0m [1mSTEP:[0m checking the PV [38;5;243m01/29/23 04:05:17.17[0m ... skipping 46 lines ... [1mSTEP:[0m setting up the StorageClass [38;5;243m01/29/23 04:06:09.45[0m [1mSTEP:[0m creating a StorageClass [38;5;243m01/29/23 04:06:09.45[0m [1mSTEP:[0m setting up the PVC and PV [38;5;243m01/29/23 04:06:09.519[0m [1mSTEP:[0m creating a PVC [38;5;243m01/29/23 04:06:09.519[0m [1mSTEP:[0m setting up the pod [38;5;243m01/29/23 04:06:09.581[0m [1mSTEP:[0m deploying the pod [38;5;243m01/29/23 04:06:09.581[0m [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/29/23 04:06:09.641[0m Jan 29 04:06:09.641: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-22jxr" in namespace "azuredisk-2546" to be "Succeeded or Failed" Jan 29 04:06:09.700: INFO: Pod "azuredisk-volume-tester-22jxr": Phase="Pending", Reason="", readiness=false. Elapsed: 59.123555ms Jan 29 04:06:11.758: INFO: Pod "azuredisk-volume-tester-22jxr": Phase="Pending", Reason="", readiness=false. Elapsed: 2.117587728s Jan 29 04:06:13.770: INFO: Pod "azuredisk-volume-tester-22jxr": Phase="Pending", Reason="", readiness=false. Elapsed: 4.128992371s Jan 29 04:06:15.759: INFO: Pod "azuredisk-volume-tester-22jxr": Phase="Pending", Reason="", readiness=false. Elapsed: 6.117738567s Jan 29 04:06:17.759: INFO: Pod "azuredisk-volume-tester-22jxr": Phase="Pending", Reason="", readiness=false. Elapsed: 8.118277284s Jan 29 04:06:19.759: INFO: Pod "azuredisk-volume-tester-22jxr": Phase="Pending", Reason="", readiness=false. Elapsed: 10.118272583s ... skipping 2 lines ... Jan 29 04:06:25.760: INFO: Pod "azuredisk-volume-tester-22jxr": Phase="Pending", Reason="", readiness=false. Elapsed: 16.119117811s Jan 29 04:06:27.759: INFO: Pod "azuredisk-volume-tester-22jxr": Phase="Pending", Reason="", readiness=false. Elapsed: 18.118565213s Jan 29 04:06:29.759: INFO: Pod "azuredisk-volume-tester-22jxr": Phase="Pending", Reason="", readiness=false. Elapsed: 20.117792413s Jan 29 04:06:31.759: INFO: Pod "azuredisk-volume-tester-22jxr": Phase="Pending", Reason="", readiness=false. Elapsed: 22.117834842s Jan 29 04:06:33.759: INFO: Pod "azuredisk-volume-tester-22jxr": Phase="Succeeded", Reason="", readiness=false. Elapsed: 24.117928567s [1mSTEP:[0m Saw pod success [38;5;243m01/29/23 04:06:33.759[0m Jan 29 04:06:33.759: INFO: Pod "azuredisk-volume-tester-22jxr" satisfied condition "Succeeded or Failed" [1mSTEP:[0m sleep 5s and then clone volume [38;5;243m01/29/23 04:06:33.759[0m [1mSTEP:[0m cloning existing volume [38;5;243m01/29/23 04:06:38.76[0m [1mSTEP:[0m setting up the PVC and PV [38;5;243m01/29/23 04:06:38.882[0m [1mSTEP:[0m creating a PVC [38;5;243m01/29/23 04:06:38.882[0m [1mSTEP:[0m setting up the pod [38;5;243m01/29/23 04:06:38.946[0m [1mSTEP:[0m deploying a second pod with cloned volume [38;5;243m01/29/23 04:06:38.946[0m [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/29/23 04:06:39.006[0m Jan 29 04:06:39.006: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-mt557" in namespace "azuredisk-2546" to be "Succeeded or Failed" Jan 29 04:06:39.065: INFO: Pod "azuredisk-volume-tester-mt557": Phase="Pending", Reason="", readiness=false. Elapsed: 59.091221ms Jan 29 04:06:41.123: INFO: Pod "azuredisk-volume-tester-mt557": Phase="Pending", Reason="", readiness=false. Elapsed: 2.117469453s Jan 29 04:06:43.131: INFO: Pod "azuredisk-volume-tester-mt557": Phase="Pending", Reason="", readiness=false. Elapsed: 4.125482767s Jan 29 04:06:45.123: INFO: Pod "azuredisk-volume-tester-mt557": Phase="Pending", Reason="", readiness=false. Elapsed: 6.116968706s Jan 29 04:06:47.123: INFO: Pod "azuredisk-volume-tester-mt557": Phase="Pending", Reason="", readiness=false. Elapsed: 8.117356898s Jan 29 04:06:49.124: INFO: Pod "azuredisk-volume-tester-mt557": Phase="Pending", Reason="", readiness=false. Elapsed: 10.118532405s ... skipping 9 lines ... Jan 29 04:07:09.124: INFO: Pod "azuredisk-volume-tester-mt557": Phase="Pending", Reason="", readiness=false. Elapsed: 30.117901488s Jan 29 04:07:11.129: INFO: Pod "azuredisk-volume-tester-mt557": Phase="Pending", Reason="", readiness=false. Elapsed: 32.123090324s Jan 29 04:07:13.125: INFO: Pod "azuredisk-volume-tester-mt557": Phase="Pending", Reason="", readiness=false. Elapsed: 34.11958705s Jan 29 04:07:15.125: INFO: Pod "azuredisk-volume-tester-mt557": Phase="Pending", Reason="", readiness=false. Elapsed: 36.119579258s Jan 29 04:07:17.125: INFO: Pod "azuredisk-volume-tester-mt557": Phase="Succeeded", Reason="", readiness=false. Elapsed: 38.118855945s [1mSTEP:[0m Saw pod success [38;5;243m01/29/23 04:07:17.125[0m Jan 29 04:07:17.125: INFO: Pod "azuredisk-volume-tester-mt557" satisfied condition "Succeeded or Failed" Jan 29 04:07:17.125: INFO: deleting Pod "azuredisk-2546"/"azuredisk-volume-tester-mt557" Jan 29 04:07:17.185: INFO: Pod azuredisk-volume-tester-mt557 has the following logs: 20.0G [1mSTEP:[0m Deleting pod azuredisk-volume-tester-mt557 in namespace azuredisk-2546 [38;5;243m01/29/23 04:07:17.185[0m [1mSTEP:[0m validating provisioned PV [38;5;243m01/29/23 04:07:17.318[0m [1mSTEP:[0m checking the PV [38;5;243m01/29/23 04:07:17.376[0m ... skipping 47 lines ... [1mSTEP:[0m setting up the StorageClass [38;5;243m01/29/23 04:06:09.45[0m [1mSTEP:[0m creating a StorageClass [38;5;243m01/29/23 04:06:09.45[0m [1mSTEP:[0m setting up the PVC and PV [38;5;243m01/29/23 04:06:09.519[0m [1mSTEP:[0m creating a PVC [38;5;243m01/29/23 04:06:09.519[0m [1mSTEP:[0m setting up the pod [38;5;243m01/29/23 04:06:09.581[0m [1mSTEP:[0m deploying the pod [38;5;243m01/29/23 04:06:09.581[0m [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/29/23 04:06:09.641[0m Jan 29 04:06:09.641: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-22jxr" in namespace "azuredisk-2546" to be "Succeeded or Failed" Jan 29 04:06:09.700: INFO: Pod "azuredisk-volume-tester-22jxr": Phase="Pending", Reason="", readiness=false. Elapsed: 59.123555ms Jan 29 04:06:11.758: INFO: Pod "azuredisk-volume-tester-22jxr": Phase="Pending", Reason="", readiness=false. Elapsed: 2.117587728s Jan 29 04:06:13.770: INFO: Pod "azuredisk-volume-tester-22jxr": Phase="Pending", Reason="", readiness=false. Elapsed: 4.128992371s Jan 29 04:06:15.759: INFO: Pod "azuredisk-volume-tester-22jxr": Phase="Pending", Reason="", readiness=false. Elapsed: 6.117738567s Jan 29 04:06:17.759: INFO: Pod "azuredisk-volume-tester-22jxr": Phase="Pending", Reason="", readiness=false. Elapsed: 8.118277284s Jan 29 04:06:19.759: INFO: Pod "azuredisk-volume-tester-22jxr": Phase="Pending", Reason="", readiness=false. Elapsed: 10.118272583s ... skipping 2 lines ... Jan 29 04:06:25.760: INFO: Pod "azuredisk-volume-tester-22jxr": Phase="Pending", Reason="", readiness=false. Elapsed: 16.119117811s Jan 29 04:06:27.759: INFO: Pod "azuredisk-volume-tester-22jxr": Phase="Pending", Reason="", readiness=false. Elapsed: 18.118565213s Jan 29 04:06:29.759: INFO: Pod "azuredisk-volume-tester-22jxr": Phase="Pending", Reason="", readiness=false. Elapsed: 20.117792413s Jan 29 04:06:31.759: INFO: Pod "azuredisk-volume-tester-22jxr": Phase="Pending", Reason="", readiness=false. Elapsed: 22.117834842s Jan 29 04:06:33.759: INFO: Pod "azuredisk-volume-tester-22jxr": Phase="Succeeded", Reason="", readiness=false. Elapsed: 24.117928567s [1mSTEP:[0m Saw pod success [38;5;243m01/29/23 04:06:33.759[0m Jan 29 04:06:33.759: INFO: Pod "azuredisk-volume-tester-22jxr" satisfied condition "Succeeded or Failed" [1mSTEP:[0m sleep 5s and then clone volume [38;5;243m01/29/23 04:06:33.759[0m [1mSTEP:[0m cloning existing volume [38;5;243m01/29/23 04:06:38.76[0m [1mSTEP:[0m setting up the PVC and PV [38;5;243m01/29/23 04:06:38.882[0m [1mSTEP:[0m creating a PVC [38;5;243m01/29/23 04:06:38.882[0m [1mSTEP:[0m setting up the pod [38;5;243m01/29/23 04:06:38.946[0m [1mSTEP:[0m deploying a second pod with cloned volume [38;5;243m01/29/23 04:06:38.946[0m [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/29/23 04:06:39.006[0m Jan 29 04:06:39.006: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-mt557" in namespace "azuredisk-2546" to be "Succeeded or Failed" Jan 29 04:06:39.065: INFO: Pod "azuredisk-volume-tester-mt557": Phase="Pending", Reason="", readiness=false. Elapsed: 59.091221ms Jan 29 04:06:41.123: INFO: Pod "azuredisk-volume-tester-mt557": Phase="Pending", Reason="", readiness=false. Elapsed: 2.117469453s Jan 29 04:06:43.131: INFO: Pod "azuredisk-volume-tester-mt557": Phase="Pending", Reason="", readiness=false. Elapsed: 4.125482767s Jan 29 04:06:45.123: INFO: Pod "azuredisk-volume-tester-mt557": Phase="Pending", Reason="", readiness=false. Elapsed: 6.116968706s Jan 29 04:06:47.123: INFO: Pod "azuredisk-volume-tester-mt557": Phase="Pending", Reason="", readiness=false. Elapsed: 8.117356898s Jan 29 04:06:49.124: INFO: Pod "azuredisk-volume-tester-mt557": Phase="Pending", Reason="", readiness=false. Elapsed: 10.118532405s ... skipping 9 lines ... Jan 29 04:07:09.124: INFO: Pod "azuredisk-volume-tester-mt557": Phase="Pending", Reason="", readiness=false. Elapsed: 30.117901488s Jan 29 04:07:11.129: INFO: Pod "azuredisk-volume-tester-mt557": Phase="Pending", Reason="", readiness=false. Elapsed: 32.123090324s Jan 29 04:07:13.125: INFO: Pod "azuredisk-volume-tester-mt557": Phase="Pending", Reason="", readiness=false. Elapsed: 34.11958705s Jan 29 04:07:15.125: INFO: Pod "azuredisk-volume-tester-mt557": Phase="Pending", Reason="", readiness=false. Elapsed: 36.119579258s Jan 29 04:07:17.125: INFO: Pod "azuredisk-volume-tester-mt557": Phase="Succeeded", Reason="", readiness=false. Elapsed: 38.118855945s [1mSTEP:[0m Saw pod success [38;5;243m01/29/23 04:07:17.125[0m Jan 29 04:07:17.125: INFO: Pod "azuredisk-volume-tester-mt557" satisfied condition "Succeeded or Failed" Jan 29 04:07:17.125: INFO: deleting Pod "azuredisk-2546"/"azuredisk-volume-tester-mt557" Jan 29 04:07:17.185: INFO: Pod azuredisk-volume-tester-mt557 has the following logs: 20.0G [1mSTEP:[0m Deleting pod azuredisk-volume-tester-mt557 in namespace azuredisk-2546 [38;5;243m01/29/23 04:07:17.185[0m [1mSTEP:[0m validating provisioned PV [38;5;243m01/29/23 04:07:17.318[0m [1mSTEP:[0m checking the PV [38;5;243m01/29/23 04:07:17.376[0m ... skipping 56 lines ... [1mSTEP:[0m setting up the StorageClass [38;5;243m01/29/23 04:08:09.868[0m [1mSTEP:[0m creating a StorageClass [38;5;243m01/29/23 04:08:09.868[0m [1mSTEP:[0m setting up the PVC and PV [38;5;243m01/29/23 04:08:09.94[0m [1mSTEP:[0m creating a PVC [38;5;243m01/29/23 04:08:09.941[0m [1mSTEP:[0m setting up the pod [38;5;243m01/29/23 04:08:10.007[0m [1mSTEP:[0m deploying the pod [38;5;243m01/29/23 04:08:10.007[0m [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/29/23 04:08:10.066[0m Jan 29 04:08:10.067: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-cxtd2" in namespace "azuredisk-1598" to be "Succeeded or Failed" Jan 29 04:08:10.124: INFO: Pod "azuredisk-volume-tester-cxtd2": Phase="Pending", Reason="", readiness=false. Elapsed: 57.635738ms Jan 29 04:08:12.185: INFO: Pod "azuredisk-volume-tester-cxtd2": Phase="Pending", Reason="", readiness=false. Elapsed: 2.117930948s Jan 29 04:08:14.185: INFO: Pod "azuredisk-volume-tester-cxtd2": Phase="Pending", Reason="", readiness=false. Elapsed: 4.118285317s Jan 29 04:08:16.185: INFO: Pod "azuredisk-volume-tester-cxtd2": Phase="Pending", Reason="", readiness=false. Elapsed: 6.118135713s Jan 29 04:08:18.182: INFO: Pod "azuredisk-volume-tester-cxtd2": Phase="Pending", Reason="", readiness=false. Elapsed: 8.1156457s Jan 29 04:08:20.184: INFO: Pod "azuredisk-volume-tester-cxtd2": Phase="Pending", Reason="", readiness=false. Elapsed: 10.11707499s ... skipping 2 lines ... Jan 29 04:08:26.183: INFO: Pod "azuredisk-volume-tester-cxtd2": Phase="Pending", Reason="", readiness=false. Elapsed: 16.116918272s Jan 29 04:08:28.184: INFO: Pod "azuredisk-volume-tester-cxtd2": Phase="Pending", Reason="", readiness=false. Elapsed: 18.117883081s Jan 29 04:08:30.186: INFO: Pod "azuredisk-volume-tester-cxtd2": Phase="Pending", Reason="", readiness=false. Elapsed: 20.119193484s Jan 29 04:08:32.184: INFO: Pod "azuredisk-volume-tester-cxtd2": Phase="Pending", Reason="", readiness=false. Elapsed: 22.117025881s Jan 29 04:08:34.188: INFO: Pod "azuredisk-volume-tester-cxtd2": Phase="Succeeded", Reason="", readiness=false. Elapsed: 24.121821781s [1mSTEP:[0m Saw pod success [38;5;243m01/29/23 04:08:34.188[0m Jan 29 04:08:34.189: INFO: Pod "azuredisk-volume-tester-cxtd2" satisfied condition "Succeeded or Failed" Jan 29 04:08:34.189: INFO: deleting Pod "azuredisk-1598"/"azuredisk-volume-tester-cxtd2" Jan 29 04:08:34.250: INFO: Pod azuredisk-volume-tester-cxtd2 has the following logs: hello world hello world hello world [1mSTEP:[0m Deleting pod azuredisk-volume-tester-cxtd2 in namespace azuredisk-1598 [38;5;243m01/29/23 04:08:34.25[0m ... skipping 68 lines ... [1mSTEP:[0m setting up the StorageClass [38;5;243m01/29/23 04:08:09.868[0m [1mSTEP:[0m creating a StorageClass [38;5;243m01/29/23 04:08:09.868[0m [1mSTEP:[0m setting up the PVC and PV [38;5;243m01/29/23 04:08:09.94[0m [1mSTEP:[0m creating a PVC [38;5;243m01/29/23 04:08:09.941[0m [1mSTEP:[0m setting up the pod [38;5;243m01/29/23 04:08:10.007[0m [1mSTEP:[0m deploying the pod [38;5;243m01/29/23 04:08:10.007[0m [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/29/23 04:08:10.066[0m Jan 29 04:08:10.067: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-cxtd2" in namespace "azuredisk-1598" to be "Succeeded or Failed" Jan 29 04:08:10.124: INFO: Pod "azuredisk-volume-tester-cxtd2": Phase="Pending", Reason="", readiness=false. Elapsed: 57.635738ms Jan 29 04:08:12.185: INFO: Pod "azuredisk-volume-tester-cxtd2": Phase="Pending", Reason="", readiness=false. Elapsed: 2.117930948s Jan 29 04:08:14.185: INFO: Pod "azuredisk-volume-tester-cxtd2": Phase="Pending", Reason="", readiness=false. Elapsed: 4.118285317s Jan 29 04:08:16.185: INFO: Pod "azuredisk-volume-tester-cxtd2": Phase="Pending", Reason="", readiness=false. Elapsed: 6.118135713s Jan 29 04:08:18.182: INFO: Pod "azuredisk-volume-tester-cxtd2": Phase="Pending", Reason="", readiness=false. Elapsed: 8.1156457s Jan 29 04:08:20.184: INFO: Pod "azuredisk-volume-tester-cxtd2": Phase="Pending", Reason="", readiness=false. Elapsed: 10.11707499s ... skipping 2 lines ... Jan 29 04:08:26.183: INFO: Pod "azuredisk-volume-tester-cxtd2": Phase="Pending", Reason="", readiness=false. Elapsed: 16.116918272s Jan 29 04:08:28.184: INFO: Pod "azuredisk-volume-tester-cxtd2": Phase="Pending", Reason="", readiness=false. Elapsed: 18.117883081s Jan 29 04:08:30.186: INFO: Pod "azuredisk-volume-tester-cxtd2": Phase="Pending", Reason="", readiness=false. Elapsed: 20.119193484s Jan 29 04:08:32.184: INFO: Pod "azuredisk-volume-tester-cxtd2": Phase="Pending", Reason="", readiness=false. Elapsed: 22.117025881s Jan 29 04:08:34.188: INFO: Pod "azuredisk-volume-tester-cxtd2": Phase="Succeeded", Reason="", readiness=false. Elapsed: 24.121821781s [1mSTEP:[0m Saw pod success [38;5;243m01/29/23 04:08:34.188[0m Jan 29 04:08:34.189: INFO: Pod "azuredisk-volume-tester-cxtd2" satisfied condition "Succeeded or Failed" Jan 29 04:08:34.189: INFO: deleting Pod "azuredisk-1598"/"azuredisk-volume-tester-cxtd2" Jan 29 04:08:34.250: INFO: Pod azuredisk-volume-tester-cxtd2 has the following logs: hello world hello world hello world [1mSTEP:[0m Deleting pod azuredisk-volume-tester-cxtd2 in namespace azuredisk-1598 [38;5;243m01/29/23 04:08:34.25[0m ... skipping 62 lines ... [1mSTEP:[0m setting up the StorageClass [38;5;243m01/29/23 04:09:32.125[0m [1mSTEP:[0m creating a StorageClass [38;5;243m01/29/23 04:09:32.125[0m [1mSTEP:[0m setting up the PVC and PV [38;5;243m01/29/23 04:09:32.183[0m [1mSTEP:[0m creating a PVC [38;5;243m01/29/23 04:09:32.184[0m [1mSTEP:[0m setting up the pod [38;5;243m01/29/23 04:09:32.243[0m [1mSTEP:[0m deploying the pod [38;5;243m01/29/23 04:09:32.244[0m [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/29/23 04:09:32.303[0m Jan 29 04:09:32.303: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-zqgcm" in namespace "azuredisk-3410" to be "Succeeded or Failed" Jan 29 04:09:32.362: INFO: Pod "azuredisk-volume-tester-zqgcm": Phase="Pending", Reason="", readiness=false. Elapsed: 58.55048ms Jan 29 04:09:34.421: INFO: Pod "azuredisk-volume-tester-zqgcm": Phase="Pending", Reason="", readiness=false. Elapsed: 2.117388799s Jan 29 04:09:36.421: INFO: Pod "azuredisk-volume-tester-zqgcm": Phase="Pending", Reason="", readiness=false. Elapsed: 4.117219482s Jan 29 04:09:38.423: INFO: Pod "azuredisk-volume-tester-zqgcm": Phase="Pending", Reason="", readiness=false. Elapsed: 6.119376844s Jan 29 04:09:40.420: INFO: Pod "azuredisk-volume-tester-zqgcm": Phase="Pending", Reason="", readiness=false. Elapsed: 8.117079632s Jan 29 04:09:42.420: INFO: Pod "azuredisk-volume-tester-zqgcm": Phase="Pending", Reason="", readiness=false. Elapsed: 10.117049712s ... skipping 10 lines ... Jan 29 04:10:04.421: INFO: Pod "azuredisk-volume-tester-zqgcm": Phase="Pending", Reason="", readiness=false. Elapsed: 32.117454475s Jan 29 04:10:06.423: INFO: Pod "azuredisk-volume-tester-zqgcm": Phase="Pending", Reason="", readiness=false. Elapsed: 34.119562221s Jan 29 04:10:08.421: INFO: Pod "azuredisk-volume-tester-zqgcm": Phase="Pending", Reason="", readiness=false. Elapsed: 36.117789173s Jan 29 04:10:10.422: INFO: Pod "azuredisk-volume-tester-zqgcm": Phase="Pending", Reason="", readiness=false. Elapsed: 38.118695961s Jan 29 04:10:12.421: INFO: Pod "azuredisk-volume-tester-zqgcm": Phase="Succeeded", Reason="", readiness=false. Elapsed: 40.117227736s [1mSTEP:[0m Saw pod success [38;5;243m01/29/23 04:10:12.421[0m Jan 29 04:10:12.421: INFO: Pod "azuredisk-volume-tester-zqgcm" satisfied condition "Succeeded or Failed" Jan 29 04:10:12.421: INFO: deleting Pod "azuredisk-3410"/"azuredisk-volume-tester-zqgcm" Jan 29 04:10:12.510: INFO: Pod azuredisk-volume-tester-zqgcm has the following logs: 100+0 records in 100+0 records out 104857600 bytes (100.0MB) copied, 0.069986 seconds, 1.4GB/s hello world ... skipping 52 lines ... [1mSTEP:[0m setting up the StorageClass [38;5;243m01/29/23 04:09:32.125[0m [1mSTEP:[0m creating a StorageClass [38;5;243m01/29/23 04:09:32.125[0m [1mSTEP:[0m setting up the PVC and PV [38;5;243m01/29/23 04:09:32.183[0m [1mSTEP:[0m creating a PVC [38;5;243m01/29/23 04:09:32.184[0m [1mSTEP:[0m setting up the pod [38;5;243m01/29/23 04:09:32.243[0m [1mSTEP:[0m deploying the pod [38;5;243m01/29/23 04:09:32.244[0m [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/29/23 04:09:32.303[0m Jan 29 04:09:32.303: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-zqgcm" in namespace "azuredisk-3410" to be "Succeeded or Failed" Jan 29 04:09:32.362: INFO: Pod "azuredisk-volume-tester-zqgcm": Phase="Pending", Reason="", readiness=false. Elapsed: 58.55048ms Jan 29 04:09:34.421: INFO: Pod "azuredisk-volume-tester-zqgcm": Phase="Pending", Reason="", readiness=false. Elapsed: 2.117388799s Jan 29 04:09:36.421: INFO: Pod "azuredisk-volume-tester-zqgcm": Phase="Pending", Reason="", readiness=false. Elapsed: 4.117219482s Jan 29 04:09:38.423: INFO: Pod "azuredisk-volume-tester-zqgcm": Phase="Pending", Reason="", readiness=false. Elapsed: 6.119376844s Jan 29 04:09:40.420: INFO: Pod "azuredisk-volume-tester-zqgcm": Phase="Pending", Reason="", readiness=false. Elapsed: 8.117079632s Jan 29 04:09:42.420: INFO: Pod "azuredisk-volume-tester-zqgcm": Phase="Pending", Reason="", readiness=false. Elapsed: 10.117049712s ... skipping 10 lines ... Jan 29 04:10:04.421: INFO: Pod "azuredisk-volume-tester-zqgcm": Phase="Pending", Reason="", readiness=false. Elapsed: 32.117454475s Jan 29 04:10:06.423: INFO: Pod "azuredisk-volume-tester-zqgcm": Phase="Pending", Reason="", readiness=false. Elapsed: 34.119562221s Jan 29 04:10:08.421: INFO: Pod "azuredisk-volume-tester-zqgcm": Phase="Pending", Reason="", readiness=false. Elapsed: 36.117789173s Jan 29 04:10:10.422: INFO: Pod "azuredisk-volume-tester-zqgcm": Phase="Pending", Reason="", readiness=false. Elapsed: 38.118695961s Jan 29 04:10:12.421: INFO: Pod "azuredisk-volume-tester-zqgcm": Phase="Succeeded", Reason="", readiness=false. Elapsed: 40.117227736s [1mSTEP:[0m Saw pod success [38;5;243m01/29/23 04:10:12.421[0m Jan 29 04:10:12.421: INFO: Pod "azuredisk-volume-tester-zqgcm" satisfied condition "Succeeded or Failed" Jan 29 04:10:12.421: INFO: deleting Pod "azuredisk-3410"/"azuredisk-volume-tester-zqgcm" Jan 29 04:10:12.510: INFO: Pod azuredisk-volume-tester-zqgcm has the following logs: 100+0 records in 100+0 records out 104857600 bytes (100.0MB) copied, 0.069986 seconds, 1.4GB/s hello world ... skipping 45 lines ... Jan 29 04:10:59.804: INFO: >>> kubeConfig: /root/tmp1482758792/kubeconfig/kubeconfig.westus2.json [1mSTEP:[0m setting up the StorageClass [38;5;243m01/29/23 04:10:59.805[0m [1mSTEP:[0m creating a StorageClass [38;5;243m01/29/23 04:10:59.805[0m [1mSTEP:[0m setting up the PVC and PV [38;5;243m01/29/23 04:10:59.865[0m [1mSTEP:[0m creating a PVC [38;5;243m01/29/23 04:10:59.865[0m [1mSTEP:[0m deploying the pod [38;5;243m01/29/23 04:10:59.928[0m [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/29/23 04:10:59.992[0m Jan 29 04:10:59.992: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-48j6j" in namespace "azuredisk-8582" to be "Succeeded or Failed" Jan 29 04:11:00.051: INFO: Pod "azuredisk-volume-tester-48j6j": Phase="Pending", Reason="", readiness=false. Elapsed: 58.61558ms Jan 29 04:11:02.110: INFO: Pod "azuredisk-volume-tester-48j6j": Phase="Pending", Reason="", readiness=false. Elapsed: 2.118007183s Jan 29 04:11:04.113: INFO: Pod "azuredisk-volume-tester-48j6j": Phase="Pending", Reason="", readiness=false. Elapsed: 4.120885786s Jan 29 04:11:06.110: INFO: Pod "azuredisk-volume-tester-48j6j": Phase="Pending", Reason="", readiness=false. Elapsed: 6.117644415s Jan 29 04:11:08.117: INFO: Pod "azuredisk-volume-tester-48j6j": Phase="Pending", Reason="", readiness=false. Elapsed: 8.124674335s Jan 29 04:11:10.109: INFO: Pod "azuredisk-volume-tester-48j6j": Phase="Pending", Reason="", readiness=false. Elapsed: 10.116742918s ... skipping 2 lines ... Jan 29 04:11:16.112: INFO: Pod "azuredisk-volume-tester-48j6j": Phase="Pending", Reason="", readiness=false. Elapsed: 16.119457867s Jan 29 04:11:18.112: INFO: Pod "azuredisk-volume-tester-48j6j": Phase="Pending", Reason="", readiness=false. Elapsed: 18.119757629s Jan 29 04:11:20.113: INFO: Pod "azuredisk-volume-tester-48j6j": Phase="Pending", Reason="", readiness=false. Elapsed: 20.120527888s Jan 29 04:11:22.110: INFO: Pod "azuredisk-volume-tester-48j6j": Phase="Pending", Reason="", readiness=false. Elapsed: 22.117577826s Jan 29 04:11:24.110: INFO: Pod "azuredisk-volume-tester-48j6j": Phase="Succeeded", Reason="", readiness=false. Elapsed: 24.118400485s [1mSTEP:[0m Saw pod success [38;5;243m01/29/23 04:11:24.111[0m Jan 29 04:11:24.111: INFO: Pod "azuredisk-volume-tester-48j6j" satisfied condition "Succeeded or Failed" [1mSTEP:[0m Checking Prow test resource group [38;5;243m01/29/23 04:11:24.111[0m 2023/01/29 04:11:24 Running in Prow, converting AZURE_CREDENTIALS to AZURE_CREDENTIAL_FILE 2023/01/29 04:11:24 Reading credentials file /etc/azure-cred/credentials [1mSTEP:[0m Prow test resource group: kubetest-l6t2jkxe [38;5;243m01/29/23 04:11:24.112[0m [1mSTEP:[0m Creating external resource group: azuredisk-csi-driver-test-fe6ba519-9f8a-11ed-987b-d279894c5862 [38;5;243m01/29/23 04:11:24.112[0m [1mSTEP:[0m creating volume snapshot class with external rg azuredisk-csi-driver-test-fe6ba519-9f8a-11ed-987b-d279894c5862 [38;5;243m01/29/23 04:11:25.712[0m ... skipping 5 lines ... [1mSTEP:[0m setting up the StorageClass [38;5;243m01/29/23 04:11:40.911[0m [1mSTEP:[0m creating a StorageClass [38;5;243m01/29/23 04:11:40.911[0m [1mSTEP:[0m setting up the PVC and PV [38;5;243m01/29/23 04:11:40.975[0m [1mSTEP:[0m creating a PVC [38;5;243m01/29/23 04:11:40.975[0m [1mSTEP:[0m setting up the pod [38;5;243m01/29/23 04:11:41.056[0m [1mSTEP:[0m deploying a pod with a volume restored from the snapshot [38;5;243m01/29/23 04:11:41.056[0m [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/29/23 04:11:41.141[0m Jan 29 04:11:41.141: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-4qmfw" in namespace "azuredisk-8582" to be "Succeeded or Failed" Jan 29 04:11:41.199: INFO: Pod "azuredisk-volume-tester-4qmfw": Phase="Pending", Reason="", readiness=false. Elapsed: 58.01566ms Jan 29 04:11:43.258: INFO: Pod "azuredisk-volume-tester-4qmfw": Phase="Pending", Reason="", readiness=false. Elapsed: 2.117228685s Jan 29 04:11:45.258: INFO: Pod "azuredisk-volume-tester-4qmfw": Phase="Pending", Reason="", readiness=false. Elapsed: 4.116786076s Jan 29 04:11:47.258: INFO: Pod "azuredisk-volume-tester-4qmfw": Phase="Pending", Reason="", readiness=false. Elapsed: 6.116923883s Jan 29 04:11:49.267: INFO: Pod "azuredisk-volume-tester-4qmfw": Phase="Pending", Reason="", readiness=false. Elapsed: 8.125566233s Jan 29 04:11:51.257: INFO: Pod "azuredisk-volume-tester-4qmfw": Phase="Pending", Reason="", readiness=false. Elapsed: 10.116019528s ... skipping 17 lines ... Jan 29 04:12:27.259: INFO: Pod "azuredisk-volume-tester-4qmfw": Phase="Pending", Reason="", readiness=false. Elapsed: 46.118342473s Jan 29 04:12:29.260: INFO: Pod "azuredisk-volume-tester-4qmfw": Phase="Pending", Reason="", readiness=false. Elapsed: 48.118732179s Jan 29 04:12:31.257: INFO: Pod "azuredisk-volume-tester-4qmfw": Phase="Pending", Reason="", readiness=false. Elapsed: 50.116092132s Jan 29 04:12:33.262: INFO: Pod "azuredisk-volume-tester-4qmfw": Phase="Pending", Reason="", readiness=false. Elapsed: 52.120922833s Jan 29 04:12:35.258: INFO: Pod "azuredisk-volume-tester-4qmfw": Phase="Succeeded", Reason="", readiness=false. Elapsed: 54.116854621s [1mSTEP:[0m Saw pod success [38;5;243m01/29/23 04:12:35.258[0m Jan 29 04:12:35.258: INFO: Pod "azuredisk-volume-tester-4qmfw" satisfied condition "Succeeded or Failed" Jan 29 04:12:35.258: INFO: deleting Pod "azuredisk-8582"/"azuredisk-volume-tester-4qmfw" Jan 29 04:12:35.346: INFO: Pod azuredisk-volume-tester-4qmfw has the following logs: hello world [1mSTEP:[0m Deleting pod azuredisk-volume-tester-4qmfw in namespace azuredisk-8582 [38;5;243m01/29/23 04:12:35.346[0m [1mSTEP:[0m validating provisioned PV [38;5;243m01/29/23 04:12:35.466[0m [1mSTEP:[0m checking the PV [38;5;243m01/29/23 04:12:35.524[0m ... skipping 50 lines ... Jan 29 04:10:59.804: INFO: >>> kubeConfig: /root/tmp1482758792/kubeconfig/kubeconfig.westus2.json [1mSTEP:[0m setting up the StorageClass [38;5;243m01/29/23 04:10:59.805[0m [1mSTEP:[0m creating a StorageClass [38;5;243m01/29/23 04:10:59.805[0m [1mSTEP:[0m setting up the PVC and PV [38;5;243m01/29/23 04:10:59.865[0m [1mSTEP:[0m creating a PVC [38;5;243m01/29/23 04:10:59.865[0m [1mSTEP:[0m deploying the pod [38;5;243m01/29/23 04:10:59.928[0m [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/29/23 04:10:59.992[0m Jan 29 04:10:59.992: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-48j6j" in namespace "azuredisk-8582" to be "Succeeded or Failed" Jan 29 04:11:00.051: INFO: Pod "azuredisk-volume-tester-48j6j": Phase="Pending", Reason="", readiness=false. Elapsed: 58.61558ms Jan 29 04:11:02.110: INFO: Pod "azuredisk-volume-tester-48j6j": Phase="Pending", Reason="", readiness=false. Elapsed: 2.118007183s Jan 29 04:11:04.113: INFO: Pod "azuredisk-volume-tester-48j6j": Phase="Pending", Reason="", readiness=false. Elapsed: 4.120885786s Jan 29 04:11:06.110: INFO: Pod "azuredisk-volume-tester-48j6j": Phase="Pending", Reason="", readiness=false. Elapsed: 6.117644415s Jan 29 04:11:08.117: INFO: Pod "azuredisk-volume-tester-48j6j": Phase="Pending", Reason="", readiness=false. Elapsed: 8.124674335s Jan 29 04:11:10.109: INFO: Pod "azuredisk-volume-tester-48j6j": Phase="Pending", Reason="", readiness=false. Elapsed: 10.116742918s ... skipping 2 lines ... Jan 29 04:11:16.112: INFO: Pod "azuredisk-volume-tester-48j6j": Phase="Pending", Reason="", readiness=false. Elapsed: 16.119457867s Jan 29 04:11:18.112: INFO: Pod "azuredisk-volume-tester-48j6j": Phase="Pending", Reason="", readiness=false. Elapsed: 18.119757629s Jan 29 04:11:20.113: INFO: Pod "azuredisk-volume-tester-48j6j": Phase="Pending", Reason="", readiness=false. Elapsed: 20.120527888s Jan 29 04:11:22.110: INFO: Pod "azuredisk-volume-tester-48j6j": Phase="Pending", Reason="", readiness=false. Elapsed: 22.117577826s Jan 29 04:11:24.110: INFO: Pod "azuredisk-volume-tester-48j6j": Phase="Succeeded", Reason="", readiness=false. Elapsed: 24.118400485s [1mSTEP:[0m Saw pod success [38;5;243m01/29/23 04:11:24.111[0m Jan 29 04:11:24.111: INFO: Pod "azuredisk-volume-tester-48j6j" satisfied condition "Succeeded or Failed" [1mSTEP:[0m Checking Prow test resource group [38;5;243m01/29/23 04:11:24.111[0m [1mSTEP:[0m Prow test resource group: kubetest-l6t2jkxe [38;5;243m01/29/23 04:11:24.112[0m [1mSTEP:[0m Creating external resource group: azuredisk-csi-driver-test-fe6ba519-9f8a-11ed-987b-d279894c5862 [38;5;243m01/29/23 04:11:24.112[0m [1mSTEP:[0m creating volume snapshot class with external rg azuredisk-csi-driver-test-fe6ba519-9f8a-11ed-987b-d279894c5862 [38;5;243m01/29/23 04:11:25.712[0m [1mSTEP:[0m setting up the VolumeSnapshotClass [38;5;243m01/29/23 04:11:25.712[0m [1mSTEP:[0m creating a VolumeSnapshotClass [38;5;243m01/29/23 04:11:25.712[0m ... skipping 3 lines ... [1mSTEP:[0m setting up the StorageClass [38;5;243m01/29/23 04:11:40.911[0m [1mSTEP:[0m creating a StorageClass [38;5;243m01/29/23 04:11:40.911[0m [1mSTEP:[0m setting up the PVC and PV [38;5;243m01/29/23 04:11:40.975[0m [1mSTEP:[0m creating a PVC [38;5;243m01/29/23 04:11:40.975[0m [1mSTEP:[0m setting up the pod [38;5;243m01/29/23 04:11:41.056[0m [1mSTEP:[0m deploying a pod with a volume restored from the snapshot [38;5;243m01/29/23 04:11:41.056[0m [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/29/23 04:11:41.141[0m Jan 29 04:11:41.141: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-4qmfw" in namespace "azuredisk-8582" to be "Succeeded or Failed" Jan 29 04:11:41.199: INFO: Pod "azuredisk-volume-tester-4qmfw": Phase="Pending", Reason="", readiness=false. Elapsed: 58.01566ms Jan 29 04:11:43.258: INFO: Pod "azuredisk-volume-tester-4qmfw": Phase="Pending", Reason="", readiness=false. Elapsed: 2.117228685s Jan 29 04:11:45.258: INFO: Pod "azuredisk-volume-tester-4qmfw": Phase="Pending", Reason="", readiness=false. Elapsed: 4.116786076s Jan 29 04:11:47.258: INFO: Pod "azuredisk-volume-tester-4qmfw": Phase="Pending", Reason="", readiness=false. Elapsed: 6.116923883s Jan 29 04:11:49.267: INFO: Pod "azuredisk-volume-tester-4qmfw": Phase="Pending", Reason="", readiness=false. Elapsed: 8.125566233s Jan 29 04:11:51.257: INFO: Pod "azuredisk-volume-tester-4qmfw": Phase="Pending", Reason="", readiness=false. Elapsed: 10.116019528s ... skipping 17 lines ... Jan 29 04:12:27.259: INFO: Pod "azuredisk-volume-tester-4qmfw": Phase="Pending", Reason="", readiness=false. Elapsed: 46.118342473s Jan 29 04:12:29.260: INFO: Pod "azuredisk-volume-tester-4qmfw": Phase="Pending", Reason="", readiness=false. Elapsed: 48.118732179s Jan 29 04:12:31.257: INFO: Pod "azuredisk-volume-tester-4qmfw": Phase="Pending", Reason="", readiness=false. Elapsed: 50.116092132s Jan 29 04:12:33.262: INFO: Pod "azuredisk-volume-tester-4qmfw": Phase="Pending", Reason="", readiness=false. Elapsed: 52.120922833s Jan 29 04:12:35.258: INFO: Pod "azuredisk-volume-tester-4qmfw": Phase="Succeeded", Reason="", readiness=false. Elapsed: 54.116854621s [1mSTEP:[0m Saw pod success [38;5;243m01/29/23 04:12:35.258[0m Jan 29 04:12:35.258: INFO: Pod "azuredisk-volume-tester-4qmfw" satisfied condition "Succeeded or Failed" Jan 29 04:12:35.258: INFO: deleting Pod "azuredisk-8582"/"azuredisk-volume-tester-4qmfw" Jan 29 04:12:35.346: INFO: Pod azuredisk-volume-tester-4qmfw has the following logs: hello world [1mSTEP:[0m Deleting pod azuredisk-volume-tester-4qmfw in namespace azuredisk-8582 [38;5;243m01/29/23 04:12:35.346[0m [1mSTEP:[0m validating provisioned PV [38;5;243m01/29/23 04:12:35.466[0m [1mSTEP:[0m checking the PV [38;5;243m01/29/23 04:12:35.524[0m ... skipping 49 lines ... Jan 29 04:14:44.837: INFO: >>> kubeConfig: /root/tmp1482758792/kubeconfig/kubeconfig.westus2.json [1mSTEP:[0m setting up the StorageClass [38;5;243m01/29/23 04:14:44.838[0m [1mSTEP:[0m creating a StorageClass [38;5;243m01/29/23 04:14:44.838[0m [1mSTEP:[0m setting up the PVC and PV [38;5;243m01/29/23 04:14:44.898[0m [1mSTEP:[0m creating a PVC [38;5;243m01/29/23 04:14:44.898[0m [1mSTEP:[0m deploying the pod [38;5;243m01/29/23 04:14:44.96[0m [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/29/23 04:14:45.02[0m Jan 29 04:14:45.020: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-gh8xj" in namespace "azuredisk-7726" to be "Succeeded or Failed" Jan 29 04:14:45.077: INFO: Pod "azuredisk-volume-tester-gh8xj": Phase="Pending", Reason="", readiness=false. Elapsed: 57.526813ms Jan 29 04:14:47.136: INFO: Pod "azuredisk-volume-tester-gh8xj": Phase="Pending", Reason="", readiness=false. Elapsed: 2.116188411s Jan 29 04:14:49.137: INFO: Pod "azuredisk-volume-tester-gh8xj": Phase="Pending", Reason="", readiness=false. Elapsed: 4.117369618s Jan 29 04:14:51.139: INFO: Pod "azuredisk-volume-tester-gh8xj": Phase="Pending", Reason="", readiness=false. Elapsed: 6.119188281s Jan 29 04:14:53.137: INFO: Pod "azuredisk-volume-tester-gh8xj": Phase="Pending", Reason="", readiness=false. Elapsed: 8.117089459s Jan 29 04:14:55.137: INFO: Pod "azuredisk-volume-tester-gh8xj": Phase="Pending", Reason="", readiness=false. Elapsed: 10.117307367s Jan 29 04:14:57.137: INFO: Pod "azuredisk-volume-tester-gh8xj": Phase="Pending", Reason="", readiness=false. Elapsed: 12.116799867s Jan 29 04:14:59.138: INFO: Pod "azuredisk-volume-tester-gh8xj": Phase="Pending", Reason="", readiness=false. Elapsed: 14.117601107s Jan 29 04:15:01.139: INFO: Pod "azuredisk-volume-tester-gh8xj": Phase="Pending", Reason="", readiness=false. Elapsed: 16.119214023s Jan 29 04:15:03.138: INFO: Pod "azuredisk-volume-tester-gh8xj": Phase="Pending", Reason="", readiness=false. Elapsed: 18.117805361s Jan 29 04:15:05.139: INFO: Pod "azuredisk-volume-tester-gh8xj": Phase="Pending", Reason="", readiness=false. Elapsed: 20.118832864s Jan 29 04:15:07.136: INFO: Pod "azuredisk-volume-tester-gh8xj": Phase="Succeeded", Reason="", readiness=false. Elapsed: 22.115988363s [1mSTEP:[0m Saw pod success [38;5;243m01/29/23 04:15:07.136[0m Jan 29 04:15:07.136: INFO: Pod "azuredisk-volume-tester-gh8xj" satisfied condition "Succeeded or Failed" [1mSTEP:[0m Checking Prow test resource group [38;5;243m01/29/23 04:15:07.136[0m 2023/01/29 04:15:07 Running in Prow, converting AZURE_CREDENTIALS to AZURE_CREDENTIAL_FILE 2023/01/29 04:15:07 Reading credentials file /etc/azure-cred/credentials [1mSTEP:[0m Prow test resource group: kubetest-l6t2jkxe [38;5;243m01/29/23 04:15:07.137[0m [1mSTEP:[0m Creating external resource group: azuredisk-csi-driver-test-835a9b8c-9f8b-11ed-987b-d279894c5862 [38;5;243m01/29/23 04:15:07.137[0m [1mSTEP:[0m creating volume snapshot class with external rg azuredisk-csi-driver-test-835a9b8c-9f8b-11ed-987b-d279894c5862 [38;5;243m01/29/23 04:15:08.241[0m ... skipping 12 lines ... [1mSTEP:[0m creating a StorageClass [38;5;243m01/29/23 04:15:25.624[0m [1mSTEP:[0m setting up the PVC and PV [38;5;243m01/29/23 04:15:25.682[0m [1mSTEP:[0m creating a PVC [38;5;243m01/29/23 04:15:25.682[0m [1mSTEP:[0m setting up the pod [38;5;243m01/29/23 04:15:25.745[0m [1mSTEP:[0m Set pod anti-affinity to make sure two pods are scheduled on different nodes [38;5;243m01/29/23 04:15:25.745[0m [1mSTEP:[0m deploying a pod with a volume restored from the snapshot [38;5;243m01/29/23 04:15:25.745[0m [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/29/23 04:15:25.804[0m Jan 29 04:15:25.804: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-spzxl" in namespace "azuredisk-7726" to be "Succeeded or Failed" Jan 29 04:15:25.862: INFO: Pod "azuredisk-volume-tester-spzxl": Phase="Pending", Reason="", readiness=false. Elapsed: 57.474182ms Jan 29 04:15:27.920: INFO: Pod "azuredisk-volume-tester-spzxl": Phase="Pending", Reason="", readiness=false. Elapsed: 2.115915053s Jan 29 04:15:29.921: INFO: Pod "azuredisk-volume-tester-spzxl": Phase="Pending", Reason="", readiness=false. Elapsed: 4.11628579s Jan 29 04:15:31.921: INFO: Pod "azuredisk-volume-tester-spzxl": Phase="Pending", Reason="", readiness=false. Elapsed: 6.116417165s Jan 29 04:15:33.921: INFO: Pod "azuredisk-volume-tester-spzxl": Phase="Pending", Reason="", readiness=false. Elapsed: 8.1163837s Jan 29 04:15:35.921: INFO: Pod "azuredisk-volume-tester-spzxl": Phase="Pending", Reason="", readiness=false. Elapsed: 10.116985379s Jan 29 04:15:37.920: INFO: Pod "azuredisk-volume-tester-spzxl": Phase="Pending", Reason="", readiness=false. Elapsed: 12.116008405s Jan 29 04:15:39.920: INFO: Pod "azuredisk-volume-tester-spzxl": Phase="Pending", Reason="", readiness=false. Elapsed: 14.116064119s Jan 29 04:15:41.922: INFO: Pod "azuredisk-volume-tester-spzxl": Phase="Pending", Reason="", readiness=false. Elapsed: 16.117509254s Jan 29 04:15:43.924: INFO: Pod "azuredisk-volume-tester-spzxl": Phase="Pending", Reason="", readiness=false. Elapsed: 18.119304704s Jan 29 04:15:45.922: INFO: Pod "azuredisk-volume-tester-spzxl": Phase="Pending", Reason="", readiness=false. Elapsed: 20.117727387s Jan 29 04:15:47.920: INFO: Pod "azuredisk-volume-tester-spzxl": Phase="Pending", Reason="", readiness=false. Elapsed: 22.115661182s Jan 29 04:15:49.922: INFO: Pod "azuredisk-volume-tester-spzxl": Phase="Failed", Reason="", readiness=false. Elapsed: 24.117631218s Jan 29 04:15:49.922: INFO: Unexpected error: <*fmt.wrapError | 0xc0000b3920>: { msg: "error while waiting for pod azuredisk-7726/azuredisk-volume-tester-spzxl to be Succeeded or Failed: pod \"azuredisk-volume-tester-spzxl\" failed with status: {Phase:Failed Conditions:[{Type:Initialized Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2023-01-29 04:15:28 +0000 UTC Reason: Message:} {Type:Ready Status:False LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2023-01-29 04:15:28 +0000 UTC Reason:ContainersNotReady Message:containers with unready status: [volume-tester]} {Type:ContainersReady Status:False LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2023-01-29 04:15:28 +0000 UTC Reason:ContainersNotReady Message:containers with unready status: [volume-tester]} {Type:PodScheduled Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2023-01-29 04:15:28 +0000 UTC Reason: Message:}] Message: Reason: NominatedNodeName: HostIP:10.248.0.32 PodIP:10.248.0.50 PodIPs:[{IP:10.248.0.50}] StartTime:2023-01-29 04:15:28 +0000 UTC InitContainerStatuses:[] ContainerStatuses:[{Name:volume-tester State:{Waiting:nil Running:nil Terminated:&ContainerStateTerminated{ExitCode:2,Signal:0,Reason:Error,Message:,StartedAt:2023-01-29 04:15:47 +0000 UTC,FinishedAt:2023-01-29 04:15:47 +0000 UTC,ContainerID:containerd://98c497fd9b000ed870a528bb200e50d08ae75f8c808ebc2a6714b0a570c9d64b,}} LastTerminationState:{Waiting:nil Running:nil Terminated:nil} Ready:false RestartCount:0 Image:registry.k8s.io/e2e-test-images/busybox:1.29-4 ImageID:registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 ContainerID:containerd://98c497fd9b000ed870a528bb200e50d08ae75f8c808ebc2a6714b0a570c9d64b Started:0xc00084ab80}] QOSClass:BestEffort EphemeralContainerStatuses:[]}", err: <*errors.errorString | 0xc000a62f80>{ s: "pod \"azuredisk-volume-tester-spzxl\" failed with status: {Phase:Failed Conditions:[{Type:Initialized Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2023-01-29 04:15:28 +0000 UTC Reason: Message:} {Type:Ready Status:False LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2023-01-29 04:15:28 +0000 UTC Reason:ContainersNotReady Message:containers with unready status: [volume-tester]} {Type:ContainersReady Status:False LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2023-01-29 04:15:28 +0000 UTC Reason:ContainersNotReady Message:containers with unready status: [volume-tester]} {Type:PodScheduled Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2023-01-29 04:15:28 +0000 UTC Reason: Message:}] Message: Reason: NominatedNodeName: HostIP:10.248.0.32 PodIP:10.248.0.50 PodIPs:[{IP:10.248.0.50}] StartTime:2023-01-29 04:15:28 +0000 UTC InitContainerStatuses:[] ContainerStatuses:[{Name:volume-tester State:{Waiting:nil Running:nil Terminated:&ContainerStateTerminated{ExitCode:2,Signal:0,Reason:Error,Message:,StartedAt:2023-01-29 04:15:47 +0000 UTC,FinishedAt:2023-01-29 04:15:47 +0000 UTC,ContainerID:containerd://98c497fd9b000ed870a528bb200e50d08ae75f8c808ebc2a6714b0a570c9d64b,}} LastTerminationState:{Waiting:nil Running:nil Terminated:nil} Ready:false RestartCount:0 Image:registry.k8s.io/e2e-test-images/busybox:1.29-4 ImageID:registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 ContainerID:containerd://98c497fd9b000ed870a528bb200e50d08ae75f8c808ebc2a6714b0a570c9d64b Started:0xc00084ab80}] QOSClass:BestEffort EphemeralContainerStatuses:[]}", }, } Jan 29 04:15:49.922: FAIL: error while waiting for pod azuredisk-7726/azuredisk-volume-tester-spzxl to be Succeeded or Failed: pod "azuredisk-volume-tester-spzxl" failed with status: {Phase:Failed Conditions:[{Type:Initialized Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2023-01-29 04:15:28 +0000 UTC Reason: Message:} {Type:Ready Status:False LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2023-01-29 04:15:28 +0000 UTC Reason:ContainersNotReady Message:containers with unready status: [volume-tester]} {Type:ContainersReady Status:False LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2023-01-29 04:15:28 +0000 UTC Reason:ContainersNotReady Message:containers with unready status: [volume-tester]} {Type:PodScheduled Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2023-01-29 04:15:28 +0000 UTC Reason: Message:}] Message: Reason: NominatedNodeName: HostIP:10.248.0.32 PodIP:10.248.0.50 PodIPs:[{IP:10.248.0.50}] StartTime:2023-01-29 04:15:28 +0000 UTC InitContainerStatuses:[] ContainerStatuses:[{Name:volume-tester State:{Waiting:nil Running:nil Terminated:&ContainerStateTerminated{ExitCode:2,Signal:0,Reason:Error,Message:,StartedAt:2023-01-29 04:15:47 +0000 UTC,FinishedAt:2023-01-29 04:15:47 +0000 UTC,ContainerID:containerd://98c497fd9b000ed870a528bb200e50d08ae75f8c808ebc2a6714b0a570c9d64b,}} LastTerminationState:{Waiting:nil Running:nil Terminated:nil} Ready:false RestartCount:0 Image:registry.k8s.io/e2e-test-images/busybox:1.29-4 ImageID:registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 ContainerID:containerd://98c497fd9b000ed870a528bb200e50d08ae75f8c808ebc2a6714b0a570c9d64b Started:0xc00084ab80}] QOSClass:BestEffort EphemeralContainerStatuses:[]} Full Stack Trace sigs.k8s.io/azuredisk-csi-driver/test/e2e/testsuites.(*TestPod).WaitForSuccess(0x2253857?) /home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/testsuites/testsuites.go:823 +0x5d sigs.k8s.io/azuredisk-csi-driver/test/e2e/testsuites.(*DynamicallyProvisionedVolumeSnapshotTest).Run(0xc000bbfd78, {0x270de60, 0xc000447520}, {0x26f9060, 0xc0005330e0}, 0xc0008974a0?) /home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/testsuites/dynamically_provisioned_volume_snapshot_tester.go:142 +0x1358 ... skipping 42 lines ... Jan 29 04:17:58.159: INFO: Waiting up to 5m0s for PersistentVolumeClaim azuredisk-7726 to be removed Jan 29 04:17:58.220: INFO: Claim "azuredisk-7726" in namespace "pvc-qbnhp" doesn't exist in the system Jan 29 04:17:58.221: INFO: deleting StorageClass azuredisk-7726-disk.csi.azure.com-dynamic-sc-cxk59 [1mSTEP:[0m dump namespace information after failure [38;5;243m01/29/23 04:17:58.28[0m [1mSTEP:[0m Destroying namespace "azuredisk-7726" for this suite. [38;5;243m01/29/23 04:17:58.281[0m [38;5;243m------------------------------[0m [38;5;9m• [FAILED] [194.407 seconds][0m Dynamic Provisioning [38;5;243m/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning_test.go:41[0m [multi-az] [38;5;243m/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning_test.go:48[0m [38;5;9m[1m[It] should create a pod, write to its pv, take a volume snapshot, overwrite data in original pv, create another pod from the snapshot, and read unaltered original data from original pv[disk.csi.azure.com][0m [38;5;243m/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning_test.go:747[0m ... skipping 7 lines ... Jan 29 04:14:44.837: INFO: >>> kubeConfig: /root/tmp1482758792/kubeconfig/kubeconfig.westus2.json [1mSTEP:[0m setting up the StorageClass [38;5;243m01/29/23 04:14:44.838[0m [1mSTEP:[0m creating a StorageClass [38;5;243m01/29/23 04:14:44.838[0m [1mSTEP:[0m setting up the PVC and PV [38;5;243m01/29/23 04:14:44.898[0m [1mSTEP:[0m creating a PVC [38;5;243m01/29/23 04:14:44.898[0m [1mSTEP:[0m deploying the pod [38;5;243m01/29/23 04:14:44.96[0m [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/29/23 04:14:45.02[0m Jan 29 04:14:45.020: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-gh8xj" in namespace "azuredisk-7726" to be "Succeeded or Failed" Jan 29 04:14:45.077: INFO: Pod "azuredisk-volume-tester-gh8xj": Phase="Pending", Reason="", readiness=false. Elapsed: 57.526813ms Jan 29 04:14:47.136: INFO: Pod "azuredisk-volume-tester-gh8xj": Phase="Pending", Reason="", readiness=false. Elapsed: 2.116188411s Jan 29 04:14:49.137: INFO: Pod "azuredisk-volume-tester-gh8xj": Phase="Pending", Reason="", readiness=false. Elapsed: 4.117369618s Jan 29 04:14:51.139: INFO: Pod "azuredisk-volume-tester-gh8xj": Phase="Pending", Reason="", readiness=false. Elapsed: 6.119188281s Jan 29 04:14:53.137: INFO: Pod "azuredisk-volume-tester-gh8xj": Phase="Pending", Reason="", readiness=false. Elapsed: 8.117089459s Jan 29 04:14:55.137: INFO: Pod "azuredisk-volume-tester-gh8xj": Phase="Pending", Reason="", readiness=false. Elapsed: 10.117307367s Jan 29 04:14:57.137: INFO: Pod "azuredisk-volume-tester-gh8xj": Phase="Pending", Reason="", readiness=false. Elapsed: 12.116799867s Jan 29 04:14:59.138: INFO: Pod "azuredisk-volume-tester-gh8xj": Phase="Pending", Reason="", readiness=false. Elapsed: 14.117601107s Jan 29 04:15:01.139: INFO: Pod "azuredisk-volume-tester-gh8xj": Phase="Pending", Reason="", readiness=false. Elapsed: 16.119214023s Jan 29 04:15:03.138: INFO: Pod "azuredisk-volume-tester-gh8xj": Phase="Pending", Reason="", readiness=false. Elapsed: 18.117805361s Jan 29 04:15:05.139: INFO: Pod "azuredisk-volume-tester-gh8xj": Phase="Pending", Reason="", readiness=false. Elapsed: 20.118832864s Jan 29 04:15:07.136: INFO: Pod "azuredisk-volume-tester-gh8xj": Phase="Succeeded", Reason="", readiness=false. Elapsed: 22.115988363s [1mSTEP:[0m Saw pod success [38;5;243m01/29/23 04:15:07.136[0m Jan 29 04:15:07.136: INFO: Pod "azuredisk-volume-tester-gh8xj" satisfied condition "Succeeded or Failed" [1mSTEP:[0m Checking Prow test resource group [38;5;243m01/29/23 04:15:07.136[0m [1mSTEP:[0m Prow test resource group: kubetest-l6t2jkxe [38;5;243m01/29/23 04:15:07.137[0m [1mSTEP:[0m Creating external resource group: azuredisk-csi-driver-test-835a9b8c-9f8b-11ed-987b-d279894c5862 [38;5;243m01/29/23 04:15:07.137[0m [1mSTEP:[0m creating volume snapshot class with external rg azuredisk-csi-driver-test-835a9b8c-9f8b-11ed-987b-d279894c5862 [38;5;243m01/29/23 04:15:08.241[0m [1mSTEP:[0m setting up the VolumeSnapshotClass [38;5;243m01/29/23 04:15:08.241[0m [1mSTEP:[0m creating a VolumeSnapshotClass [38;5;243m01/29/23 04:15:08.241[0m ... skipping 10 lines ... [1mSTEP:[0m creating a StorageClass [38;5;243m01/29/23 04:15:25.624[0m [1mSTEP:[0m setting up the PVC and PV [38;5;243m01/29/23 04:15:25.682[0m [1mSTEP:[0m creating a PVC [38;5;243m01/29/23 04:15:25.682[0m [1mSTEP:[0m setting up the pod [38;5;243m01/29/23 04:15:25.745[0m [1mSTEP:[0m Set pod anti-affinity to make sure two pods are scheduled on different nodes [38;5;243m01/29/23 04:15:25.745[0m [1mSTEP:[0m deploying a pod with a volume restored from the snapshot [38;5;243m01/29/23 04:15:25.745[0m [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/29/23 04:15:25.804[0m Jan 29 04:15:25.804: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-spzxl" in namespace "azuredisk-7726" to be "Succeeded or Failed" Jan 29 04:15:25.862: INFO: Pod "azuredisk-volume-tester-spzxl": Phase="Pending", Reason="", readiness=false. Elapsed: 57.474182ms Jan 29 04:15:27.920: INFO: Pod "azuredisk-volume-tester-spzxl": Phase="Pending", Reason="", readiness=false. Elapsed: 2.115915053s Jan 29 04:15:29.921: INFO: Pod "azuredisk-volume-tester-spzxl": Phase="Pending", Reason="", readiness=false. Elapsed: 4.11628579s Jan 29 04:15:31.921: INFO: Pod "azuredisk-volume-tester-spzxl": Phase="Pending", Reason="", readiness=false. Elapsed: 6.116417165s Jan 29 04:15:33.921: INFO: Pod "azuredisk-volume-tester-spzxl": Phase="Pending", Reason="", readiness=false. Elapsed: 8.1163837s Jan 29 04:15:35.921: INFO: Pod "azuredisk-volume-tester-spzxl": Phase="Pending", Reason="", readiness=false. Elapsed: 10.116985379s Jan 29 04:15:37.920: INFO: Pod "azuredisk-volume-tester-spzxl": Phase="Pending", Reason="", readiness=false. Elapsed: 12.116008405s Jan 29 04:15:39.920: INFO: Pod "azuredisk-volume-tester-spzxl": Phase="Pending", Reason="", readiness=false. Elapsed: 14.116064119s Jan 29 04:15:41.922: INFO: Pod "azuredisk-volume-tester-spzxl": Phase="Pending", Reason="", readiness=false. Elapsed: 16.117509254s Jan 29 04:15:43.924: INFO: Pod "azuredisk-volume-tester-spzxl": Phase="Pending", Reason="", readiness=false. Elapsed: 18.119304704s Jan 29 04:15:45.922: INFO: Pod "azuredisk-volume-tester-spzxl": Phase="Pending", Reason="", readiness=false. Elapsed: 20.117727387s Jan 29 04:15:47.920: INFO: Pod "azuredisk-volume-tester-spzxl": Phase="Pending", Reason="", readiness=false. Elapsed: 22.115661182s Jan 29 04:15:49.922: INFO: Pod "azuredisk-volume-tester-spzxl": Phase="Failed", Reason="", readiness=false. Elapsed: 24.117631218s Jan 29 04:15:49.922: INFO: Unexpected error: <*fmt.wrapError | 0xc0000b3920>: { msg: "error while waiting for pod azuredisk-7726/azuredisk-volume-tester-spzxl to be Succeeded or Failed: pod \"azuredisk-volume-tester-spzxl\" failed with status: {Phase:Failed Conditions:[{Type:Initialized Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2023-01-29 04:15:28 +0000 UTC Reason: Message:} {Type:Ready Status:False LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2023-01-29 04:15:28 +0000 UTC Reason:ContainersNotReady Message:containers with unready status: [volume-tester]} {Type:ContainersReady Status:False LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2023-01-29 04:15:28 +0000 UTC Reason:ContainersNotReady Message:containers with unready status: [volume-tester]} {Type:PodScheduled Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2023-01-29 04:15:28 +0000 UTC Reason: Message:}] Message: Reason: NominatedNodeName: HostIP:10.248.0.32 PodIP:10.248.0.50 PodIPs:[{IP:10.248.0.50}] StartTime:2023-01-29 04:15:28 +0000 UTC InitContainerStatuses:[] ContainerStatuses:[{Name:volume-tester State:{Waiting:nil Running:nil Terminated:&ContainerStateTerminated{ExitCode:2,Signal:0,Reason:Error,Message:,StartedAt:2023-01-29 04:15:47 +0000 UTC,FinishedAt:2023-01-29 04:15:47 +0000 UTC,ContainerID:containerd://98c497fd9b000ed870a528bb200e50d08ae75f8c808ebc2a6714b0a570c9d64b,}} LastTerminationState:{Waiting:nil Running:nil Terminated:nil} Ready:false RestartCount:0 Image:registry.k8s.io/e2e-test-images/busybox:1.29-4 ImageID:registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 ContainerID:containerd://98c497fd9b000ed870a528bb200e50d08ae75f8c808ebc2a6714b0a570c9d64b Started:0xc00084ab80}] QOSClass:BestEffort EphemeralContainerStatuses:[]}", err: <*errors.errorString | 0xc000a62f80>{ s: "pod \"azuredisk-volume-tester-spzxl\" failed with status: {Phase:Failed Conditions:[{Type:Initialized Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2023-01-29 04:15:28 +0000 UTC Reason: Message:} {Type:Ready Status:False LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2023-01-29 04:15:28 +0000 UTC Reason:ContainersNotReady Message:containers with unready status: [volume-tester]} {Type:ContainersReady Status:False LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2023-01-29 04:15:28 +0000 UTC Reason:ContainersNotReady Message:containers with unready status: [volume-tester]} {Type:PodScheduled Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2023-01-29 04:15:28 +0000 UTC Reason: Message:}] Message: Reason: NominatedNodeName: HostIP:10.248.0.32 PodIP:10.248.0.50 PodIPs:[{IP:10.248.0.50}] StartTime:2023-01-29 04:15:28 +0000 UTC InitContainerStatuses:[] ContainerStatuses:[{Name:volume-tester State:{Waiting:nil Running:nil Terminated:&ContainerStateTerminated{ExitCode:2,Signal:0,Reason:Error,Message:,StartedAt:2023-01-29 04:15:47 +0000 UTC,FinishedAt:2023-01-29 04:15:47 +0000 UTC,ContainerID:containerd://98c497fd9b000ed870a528bb200e50d08ae75f8c808ebc2a6714b0a570c9d64b,}} LastTerminationState:{Waiting:nil Running:nil Terminated:nil} Ready:false RestartCount:0 Image:registry.k8s.io/e2e-test-images/busybox:1.29-4 ImageID:registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 ContainerID:containerd://98c497fd9b000ed870a528bb200e50d08ae75f8c808ebc2a6714b0a570c9d64b Started:0xc00084ab80}] QOSClass:BestEffort EphemeralContainerStatuses:[]}", }, } Jan 29 04:15:49.922: FAIL: error while waiting for pod azuredisk-7726/azuredisk-volume-tester-spzxl to be Succeeded or Failed: pod "azuredisk-volume-tester-spzxl" failed with status: {Phase:Failed Conditions:[{Type:Initialized Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2023-01-29 04:15:28 +0000 UTC Reason: Message:} {Type:Ready Status:False LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2023-01-29 04:15:28 +0000 UTC Reason:ContainersNotReady Message:containers with unready status: [volume-tester]} {Type:ContainersReady Status:False LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2023-01-29 04:15:28 +0000 UTC Reason:ContainersNotReady Message:containers with unready status: [volume-tester]} {Type:PodScheduled Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2023-01-29 04:15:28 +0000 UTC Reason: Message:}] Message: Reason: NominatedNodeName: HostIP:10.248.0.32 PodIP:10.248.0.50 PodIPs:[{IP:10.248.0.50}] StartTime:2023-01-29 04:15:28 +0000 UTC InitContainerStatuses:[] ContainerStatuses:[{Name:volume-tester State:{Waiting:nil Running:nil Terminated:&ContainerStateTerminated{ExitCode:2,Signal:0,Reason:Error,Message:,StartedAt:2023-01-29 04:15:47 +0000 UTC,FinishedAt:2023-01-29 04:15:47 +0000 UTC,ContainerID:containerd://98c497fd9b000ed870a528bb200e50d08ae75f8c808ebc2a6714b0a570c9d64b,}} LastTerminationState:{Waiting:nil Running:nil Terminated:nil} Ready:false RestartCount:0 Image:registry.k8s.io/e2e-test-images/busybox:1.29-4 ImageID:registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 ContainerID:containerd://98c497fd9b000ed870a528bb200e50d08ae75f8c808ebc2a6714b0a570c9d64b Started:0xc00084ab80}] QOSClass:BestEffort EphemeralContainerStatuses:[]} Full Stack Trace sigs.k8s.io/azuredisk-csi-driver/test/e2e/testsuites.(*TestPod).WaitForSuccess(0x2253857?) /home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/testsuites/testsuites.go:823 +0x5d sigs.k8s.io/azuredisk-csi-driver/test/e2e/testsuites.(*DynamicallyProvisionedVolumeSnapshotTest).Run(0xc000bbfd78, {0x270de60, 0xc000447520}, {0x26f9060, 0xc0005330e0}, 0xc0008974a0?) /home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/testsuites/dynamically_provisioned_volume_snapshot_tester.go:142 +0x1358 ... skipping 43 lines ... Jan 29 04:17:58.220: INFO: Claim "azuredisk-7726" in namespace "pvc-qbnhp" doesn't exist in the system Jan 29 04:17:58.221: INFO: deleting StorageClass azuredisk-7726-disk.csi.azure.com-dynamic-sc-cxk59 [1mSTEP:[0m dump namespace information after failure [38;5;243m01/29/23 04:17:58.28[0m [1mSTEP:[0m Destroying namespace "azuredisk-7726" for this suite. [38;5;243m01/29/23 04:17:58.281[0m [38;5;243m<< End Captured GinkgoWriter Output[0m [38;5;9mJan 29 04:15:49.922: error while waiting for pod azuredisk-7726/azuredisk-volume-tester-spzxl to be Succeeded or Failed: pod "azuredisk-volume-tester-spzxl" failed with status: {Phase:Failed Conditions:[{Type:Initialized Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2023-01-29 04:15:28 +0000 UTC Reason: Message:} {Type:Ready Status:False LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2023-01-29 04:15:28 +0000 UTC Reason:ContainersNotReady Message:containers with unready status: [volume-tester]} {Type:ContainersReady Status:False LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2023-01-29 04:15:28 +0000 UTC Reason:ContainersNotReady Message:containers with unready status: [volume-tester]} {Type:PodScheduled Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2023-01-29 04:15:28 +0000 UTC Reason: Message:}] Message: Reason: NominatedNodeName: HostIP:10.248.0.32 PodIP:10.248.0.50 PodIPs:[{IP:10.248.0.50}] StartTime:2023-01-29 04:15:28 +0000 UTC InitContainerStatuses:[] ContainerStatuses:[{Name:volume-tester State:{Waiting:nil Running:nil Terminated:&ContainerStateTerminated{ExitCode:2,Signal:0,Reason:Error,Message:,StartedAt:2023-01-29 04:15:47 +0000 UTC,FinishedAt:2023-01-29 04:15:47 +0000 UTC,ContainerID:containerd://98c497fd9b000ed870a528bb200e50d08ae75f8c808ebc2a6714b0a570c9d64b,}} LastTerminationState:{Waiting:nil Running:nil Terminated:nil} Ready:false RestartCount:0 Image:registry.k8s.io/e2e-test-images/busybox:1.29-4 ImageID:registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 ContainerID:containerd://98c497fd9b000ed870a528bb200e50d08ae75f8c808ebc2a6714b0a570c9d64b Started:0xc00084ab80}] QOSClass:BestEffort EphemeralContainerStatuses:[]}[0m [38;5;9mIn [1m[It][0m[38;5;9m at: [1m/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/testsuites/testsuites.go:823[0m [1mThere were additional failures detected after the initial failure:[0m [38;5;13m[PANICKED][0m [38;5;13mTest Panicked[0m [38;5;13mIn [1m[DeferCleanup (Each)][0m[38;5;13m at: [1m/usr/local/go/src/runtime/panic.go:260[0m [38;5;13mruntime error: invalid memory address or nil pointer dereference[0m [38;5;13mFull Stack Trace[0m k8s.io/kubernetes/test/e2e/framework.(*Framework).dumpNamespaceInfo.func1() /home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:274 +0x5c k8s.io/kubernetes/test/e2e/framework.(*Framework).dumpNamespaceInfo(0xc0000343c0) /home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:271 +0x179 ... skipping 25 lines ... [1mSTEP:[0m creating a PVC [38;5;243m01/29/23 04:17:59.414[0m [1mSTEP:[0m setting up the StorageClass [38;5;243m01/29/23 04:17:59.474[0m [1mSTEP:[0m creating a StorageClass [38;5;243m01/29/23 04:17:59.475[0m [1mSTEP:[0m setting up the PVC and PV [38;5;243m01/29/23 04:17:59.538[0m [1mSTEP:[0m creating a PVC [38;5;243m01/29/23 04:17:59.538[0m [1mSTEP:[0m deploying the pod [38;5;243m01/29/23 04:17:59.598[0m [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/29/23 04:17:59.658[0m Jan 29 04:17:59.658: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-vqwx7" in namespace "azuredisk-3086" to be "Succeeded or Failed" Jan 29 04:17:59.717: INFO: Pod "azuredisk-volume-tester-vqwx7": Phase="Pending", Reason="", readiness=false. Elapsed: 58.647566ms Jan 29 04:18:01.777: INFO: Pod "azuredisk-volume-tester-vqwx7": Phase="Pending", Reason="", readiness=false. Elapsed: 2.118772814s Jan 29 04:18:03.777: INFO: Pod "azuredisk-volume-tester-vqwx7": Phase="Pending", Reason="", readiness=false. Elapsed: 4.119052646s Jan 29 04:18:05.775: INFO: Pod "azuredisk-volume-tester-vqwx7": Phase="Pending", Reason="", readiness=false. Elapsed: 6.117116632s Jan 29 04:18:07.777: INFO: Pod "azuredisk-volume-tester-vqwx7": Phase="Pending", Reason="", readiness=false. Elapsed: 8.118766676s Jan 29 04:18:09.775: INFO: Pod "azuredisk-volume-tester-vqwx7": Phase="Pending", Reason="", readiness=false. Elapsed: 10.116992849s Jan 29 04:18:11.775: INFO: Pod "azuredisk-volume-tester-vqwx7": Phase="Pending", Reason="", readiness=false. Elapsed: 12.117095162s Jan 29 04:18:13.777: INFO: Pod "azuredisk-volume-tester-vqwx7": Phase="Pending", Reason="", readiness=false. Elapsed: 14.118873394s Jan 29 04:18:15.777: INFO: Pod "azuredisk-volume-tester-vqwx7": Phase="Pending", Reason="", readiness=false. Elapsed: 16.118346391s Jan 29 04:18:17.777: INFO: Pod "azuredisk-volume-tester-vqwx7": Phase="Pending", Reason="", readiness=false. Elapsed: 18.118641042s Jan 29 04:18:19.776: INFO: Pod "azuredisk-volume-tester-vqwx7": Phase="Pending", Reason="", readiness=false. Elapsed: 20.117645334s Jan 29 04:18:21.776: INFO: Pod "azuredisk-volume-tester-vqwx7": Phase="Succeeded", Reason="", readiness=false. Elapsed: 22.117902255s [1mSTEP:[0m Saw pod success [38;5;243m01/29/23 04:18:21.776[0m Jan 29 04:18:21.777: INFO: Pod "azuredisk-volume-tester-vqwx7" satisfied condition "Succeeded or Failed" Jan 29 04:18:21.777: INFO: deleting Pod "azuredisk-3086"/"azuredisk-volume-tester-vqwx7" Jan 29 04:18:21.838: INFO: Pod azuredisk-volume-tester-vqwx7 has the following logs: hello world [1mSTEP:[0m Deleting pod azuredisk-volume-tester-vqwx7 in namespace azuredisk-3086 [38;5;243m01/29/23 04:18:21.838[0m [1mSTEP:[0m validating provisioned PV [38;5;243m01/29/23 04:18:21.958[0m [1mSTEP:[0m checking the PV [38;5;243m01/29/23 04:18:22.016[0m ... skipping 64 lines ... [1mSTEP:[0m creating a PVC [38;5;243m01/29/23 04:17:59.414[0m [1mSTEP:[0m setting up the StorageClass [38;5;243m01/29/23 04:17:59.474[0m [1mSTEP:[0m creating a StorageClass [38;5;243m01/29/23 04:17:59.475[0m [1mSTEP:[0m setting up the PVC and PV [38;5;243m01/29/23 04:17:59.538[0m [1mSTEP:[0m creating a PVC [38;5;243m01/29/23 04:17:59.538[0m [1mSTEP:[0m deploying the pod [38;5;243m01/29/23 04:17:59.598[0m [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/29/23 04:17:59.658[0m Jan 29 04:17:59.658: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-vqwx7" in namespace "azuredisk-3086" to be "Succeeded or Failed" Jan 29 04:17:59.717: INFO: Pod "azuredisk-volume-tester-vqwx7": Phase="Pending", Reason="", readiness=false. Elapsed: 58.647566ms Jan 29 04:18:01.777: INFO: Pod "azuredisk-volume-tester-vqwx7": Phase="Pending", Reason="", readiness=false. Elapsed: 2.118772814s Jan 29 04:18:03.777: INFO: Pod "azuredisk-volume-tester-vqwx7": Phase="Pending", Reason="", readiness=false. Elapsed: 4.119052646s Jan 29 04:18:05.775: INFO: Pod "azuredisk-volume-tester-vqwx7": Phase="Pending", Reason="", readiness=false. Elapsed: 6.117116632s Jan 29 04:18:07.777: INFO: Pod "azuredisk-volume-tester-vqwx7": Phase="Pending", Reason="", readiness=false. Elapsed: 8.118766676s Jan 29 04:18:09.775: INFO: Pod "azuredisk-volume-tester-vqwx7": Phase="Pending", Reason="", readiness=false. Elapsed: 10.116992849s Jan 29 04:18:11.775: INFO: Pod "azuredisk-volume-tester-vqwx7": Phase="Pending", Reason="", readiness=false. Elapsed: 12.117095162s Jan 29 04:18:13.777: INFO: Pod "azuredisk-volume-tester-vqwx7": Phase="Pending", Reason="", readiness=false. Elapsed: 14.118873394s Jan 29 04:18:15.777: INFO: Pod "azuredisk-volume-tester-vqwx7": Phase="Pending", Reason="", readiness=false. Elapsed: 16.118346391s Jan 29 04:18:17.777: INFO: Pod "azuredisk-volume-tester-vqwx7": Phase="Pending", Reason="", readiness=false. Elapsed: 18.118641042s Jan 29 04:18:19.776: INFO: Pod "azuredisk-volume-tester-vqwx7": Phase="Pending", Reason="", readiness=false. Elapsed: 20.117645334s Jan 29 04:18:21.776: INFO: Pod "azuredisk-volume-tester-vqwx7": Phase="Succeeded", Reason="", readiness=false. Elapsed: 22.117902255s [1mSTEP:[0m Saw pod success [38;5;243m01/29/23 04:18:21.776[0m Jan 29 04:18:21.777: INFO: Pod "azuredisk-volume-tester-vqwx7" satisfied condition "Succeeded or Failed" Jan 29 04:18:21.777: INFO: deleting Pod "azuredisk-3086"/"azuredisk-volume-tester-vqwx7" Jan 29 04:18:21.838: INFO: Pod azuredisk-volume-tester-vqwx7 has the following logs: hello world [1mSTEP:[0m Deleting pod azuredisk-volume-tester-vqwx7 in namespace azuredisk-3086 [38;5;243m01/29/23 04:18:21.838[0m [1mSTEP:[0m validating provisioned PV [38;5;243m01/29/23 04:18:21.958[0m [1mSTEP:[0m checking the PV [38;5;243m01/29/23 04:18:22.016[0m ... skipping 938 lines ... [1mSTEP:[0m setting up the StorageClass [38;5;243m01/29/23 04:31:58.298[0m [1mSTEP:[0m creating a StorageClass [38;5;243m01/29/23 04:31:58.298[0m [1mSTEP:[0m setting up the PVC and PV [38;5;243m01/29/23 04:31:58.356[0m [1mSTEP:[0m creating a PVC [38;5;243m01/29/23 04:31:58.357[0m [1mSTEP:[0m setting up the pod [38;5;243m01/29/23 04:31:58.417[0m [1mSTEP:[0m deploying the pod [38;5;243m01/29/23 04:31:58.417[0m [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/29/23 04:31:58.475[0m Jan 29 04:31:58.475: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-mf945" in namespace "azuredisk-1092" to be "Succeeded or Failed" Jan 29 04:31:58.532: INFO: Pod "azuredisk-volume-tester-mf945": Phase="Pending", Reason="", readiness=false. Elapsed: 56.26606ms Jan 29 04:32:00.589: INFO: Pod "azuredisk-volume-tester-mf945": Phase="Pending", Reason="", readiness=false. Elapsed: 2.113693188s Jan 29 04:32:02.590: INFO: Pod "azuredisk-volume-tester-mf945": Phase="Pending", Reason="", readiness=false. Elapsed: 4.114973611s Jan 29 04:32:04.590: INFO: Pod "azuredisk-volume-tester-mf945": Phase="Pending", Reason="", readiness=false. Elapsed: 6.114262506s Jan 29 04:32:06.591: INFO: Pod "azuredisk-volume-tester-mf945": Phase="Pending", Reason="", readiness=false. Elapsed: 8.115067191s Jan 29 04:32:08.591: INFO: Pod "azuredisk-volume-tester-mf945": Phase="Pending", Reason="", readiness=false. Elapsed: 10.115177919s ... skipping 10 lines ... Jan 29 04:32:30.590: INFO: Pod "azuredisk-volume-tester-mf945": Phase="Pending", Reason="", readiness=false. Elapsed: 32.114139029s Jan 29 04:32:32.590: INFO: Pod "azuredisk-volume-tester-mf945": Phase="Pending", Reason="", readiness=false. Elapsed: 34.114073375s Jan 29 04:32:34.590: INFO: Pod "azuredisk-volume-tester-mf945": Phase="Pending", Reason="", readiness=false. Elapsed: 36.114748866s Jan 29 04:32:36.589: INFO: Pod "azuredisk-volume-tester-mf945": Phase="Pending", Reason="", readiness=false. Elapsed: 38.113768057s Jan 29 04:32:38.592: INFO: Pod "azuredisk-volume-tester-mf945": Phase="Succeeded", Reason="", readiness=false. Elapsed: 40.116091495s [1mSTEP:[0m Saw pod success [38;5;243m01/29/23 04:32:38.592[0m Jan 29 04:32:38.592: INFO: Pod "azuredisk-volume-tester-mf945" satisfied condition "Succeeded or Failed" Jan 29 04:32:38.592: INFO: deleting Pod "azuredisk-1092"/"azuredisk-volume-tester-mf945" Jan 29 04:32:38.678: INFO: Pod azuredisk-volume-tester-mf945 has the following logs: hello world [1mSTEP:[0m Deleting pod azuredisk-volume-tester-mf945 in namespace azuredisk-1092 [38;5;243m01/29/23 04:32:38.678[0m [1mSTEP:[0m validating provisioned PV [38;5;243m01/29/23 04:32:38.796[0m [1mSTEP:[0m checking the PV [38;5;243m01/29/23 04:32:38.853[0m ... skipping 33 lines ... [1mSTEP:[0m setting up the StorageClass [38;5;243m01/29/23 04:31:58.298[0m [1mSTEP:[0m creating a StorageClass [38;5;243m01/29/23 04:31:58.298[0m [1mSTEP:[0m setting up the PVC and PV [38;5;243m01/29/23 04:31:58.356[0m [1mSTEP:[0m creating a PVC [38;5;243m01/29/23 04:31:58.357[0m [1mSTEP:[0m setting up the pod [38;5;243m01/29/23 04:31:58.417[0m [1mSTEP:[0m deploying the pod [38;5;243m01/29/23 04:31:58.417[0m [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/29/23 04:31:58.475[0m Jan 29 04:31:58.475: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-mf945" in namespace "azuredisk-1092" to be "Succeeded or Failed" Jan 29 04:31:58.532: INFO: Pod "azuredisk-volume-tester-mf945": Phase="Pending", Reason="", readiness=false. Elapsed: 56.26606ms Jan 29 04:32:00.589: INFO: Pod "azuredisk-volume-tester-mf945": Phase="Pending", Reason="", readiness=false. Elapsed: 2.113693188s Jan 29 04:32:02.590: INFO: Pod "azuredisk-volume-tester-mf945": Phase="Pending", Reason="", readiness=false. Elapsed: 4.114973611s Jan 29 04:32:04.590: INFO: Pod "azuredisk-volume-tester-mf945": Phase="Pending", Reason="", readiness=false. Elapsed: 6.114262506s Jan 29 04:32:06.591: INFO: Pod "azuredisk-volume-tester-mf945": Phase="Pending", Reason="", readiness=false. Elapsed: 8.115067191s Jan 29 04:32:08.591: INFO: Pod "azuredisk-volume-tester-mf945": Phase="Pending", Reason="", readiness=false. Elapsed: 10.115177919s ... skipping 10 lines ... Jan 29 04:32:30.590: INFO: Pod "azuredisk-volume-tester-mf945": Phase="Pending", Reason="", readiness=false. Elapsed: 32.114139029s Jan 29 04:32:32.590: INFO: Pod "azuredisk-volume-tester-mf945": Phase="Pending", Reason="", readiness=false. Elapsed: 34.114073375s Jan 29 04:32:34.590: INFO: Pod "azuredisk-volume-tester-mf945": Phase="Pending", Reason="", readiness=false. Elapsed: 36.114748866s Jan 29 04:32:36.589: INFO: Pod "azuredisk-volume-tester-mf945": Phase="Pending", Reason="", readiness=false. Elapsed: 38.113768057s Jan 29 04:32:38.592: INFO: Pod "azuredisk-volume-tester-mf945": Phase="Succeeded", Reason="", readiness=false. Elapsed: 40.116091495s [1mSTEP:[0m Saw pod success [38;5;243m01/29/23 04:32:38.592[0m Jan 29 04:32:38.592: INFO: Pod "azuredisk-volume-tester-mf945" satisfied condition "Succeeded or Failed" Jan 29 04:32:38.592: INFO: deleting Pod "azuredisk-1092"/"azuredisk-volume-tester-mf945" Jan 29 04:32:38.678: INFO: Pod azuredisk-volume-tester-mf945 has the following logs: hello world [1mSTEP:[0m Deleting pod azuredisk-volume-tester-mf945 in namespace azuredisk-1092 [38;5;243m01/29/23 04:32:38.678[0m [1mSTEP:[0m validating provisioned PV [38;5;243m01/29/23 04:32:38.796[0m [1mSTEP:[0m checking the PV [38;5;243m01/29/23 04:32:38.853[0m ... skipping 93 lines ... Platform: linux/amd64 Topology Key: topology.disk.csi.azure.com/zone Streaming logs below: I0129 03:44:08.326358 1 azuredisk.go:175] driver userAgent: disk.csi.azure.com/v1.27.0-d5369cddb1fe3d1d78e5a517b281425883f14ad4 e2e-test I0129 03:44:08.326961 1 azure_disk_utils.go:162] reading cloud config from secret kube-system/azure-cloud-provider I0129 03:44:08.355666 1 azure_disk_utils.go:169] InitializeCloudFromSecret: failed to get cloud config from secret kube-system/azure-cloud-provider: failed to get secret kube-system/azure-cloud-provider: secrets "azure-cloud-provider" not found I0129 03:44:08.355695 1 azure_disk_utils.go:174] could not read cloud config from secret kube-system/azure-cloud-provider I0129 03:44:08.355706 1 azure_disk_utils.go:184] use default AZURE_CREDENTIAL_FILE env var: /etc/kubernetes/azure.json I0129 03:44:08.355747 1 azure_disk_utils.go:192] read cloud config from file: /etc/kubernetes/azure.json successfully I0129 03:44:08.356654 1 azure_auth.go:253] Using AzurePublicCloud environment I0129 03:44:08.356721 1 azure_auth.go:138] azure: using client_id+client_secret to retrieve access token I0129 03:44:08.356799 1 azure.go:776] Azure cloudprovider using try backoff: retries=6, exponent=1.500000, duration=5, jitter=1.000000 ... skipping 25 lines ... I0129 03:44:08.357298 1 azure_blobclient.go:67] Azure BlobClient using API version: 2021-09-01 I0129 03:44:08.357316 1 azure_vmasclient.go:70] Azure AvailabilitySetsClient (read ops) using rate limit config: QPS=6, bucket=20 I0129 03:44:08.357325 1 azure_vmasclient.go:73] Azure AvailabilitySetsClient (write ops) using rate limit config: QPS=100, bucket=1000 I0129 03:44:08.357432 1 azure.go:1007] attach/detach disk operation rate limit QPS: 6.000000, Bucket: 10 I0129 03:44:08.357464 1 azuredisk.go:192] disable UseInstanceMetadata for controller I0129 03:44:08.357475 1 azuredisk.go:204] cloud: AzurePublicCloud, location: westus2, rg: kubetest-l6t2jkxe, VMType: vmss, PrimaryScaleSetName: k8s-agentpool-14513215-vmss, PrimaryAvailabilitySetName: , DisableAvailabilitySetNodes: false I0129 03:44:08.364900 1 mount_linux.go:287] 'umount /tmp/kubelet-detect-safe-umount1110916499' failed with: exit status 32, output: umount: /tmp/kubelet-detect-safe-umount1110916499: must be superuser to unmount. I0129 03:44:08.364942 1 mount_linux.go:289] Detected umount with unsafe 'not mounted' behavior I0129 03:44:08.365132 1 driver.go:81] Enabling controller service capability: CREATE_DELETE_VOLUME I0129 03:44:08.365156 1 driver.go:81] Enabling controller service capability: PUBLISH_UNPUBLISH_VOLUME I0129 03:44:08.365165 1 driver.go:81] Enabling controller service capability: CREATE_DELETE_SNAPSHOT I0129 03:44:08.365176 1 driver.go:81] Enabling controller service capability: CLONE_VOLUME I0129 03:44:08.365303 1 driver.go:81] Enabling controller service capability: EXPAND_VOLUME ... skipping 61 lines ... Platform: linux/amd64 Topology Key: topology.disk.csi.azure.com/zone Streaming logs below: I0129 03:44:05.904530 1 azuredisk.go:175] driver userAgent: disk.csi.azure.com/v1.27.0-d5369cddb1fe3d1d78e5a517b281425883f14ad4 e2e-test I0129 03:44:05.905149 1 azure_disk_utils.go:162] reading cloud config from secret kube-system/azure-cloud-provider I0129 03:44:05.932753 1 azure_disk_utils.go:169] InitializeCloudFromSecret: failed to get cloud config from secret kube-system/azure-cloud-provider: failed to get secret kube-system/azure-cloud-provider: secrets "azure-cloud-provider" not found I0129 03:44:05.932795 1 azure_disk_utils.go:174] could not read cloud config from secret kube-system/azure-cloud-provider I0129 03:44:05.932804 1 azure_disk_utils.go:184] use default AZURE_CREDENTIAL_FILE env var: /etc/kubernetes/azure.json I0129 03:44:05.932836 1 azure_disk_utils.go:192] read cloud config from file: /etc/kubernetes/azure.json successfully I0129 03:44:05.933643 1 azure_auth.go:253] Using AzurePublicCloud environment I0129 03:44:05.933699 1 azure_auth.go:138] azure: using client_id+client_secret to retrieve access token I0129 03:44:05.933732 1 azure.go:776] Azure cloudprovider using try backoff: retries=6, exponent=1.500000, duration=5, jitter=1.000000 ... skipping 25 lines ... I0129 03:44:05.934205 1 azure_blobclient.go:67] Azure BlobClient using API version: 2021-09-01 I0129 03:44:05.934223 1 azure_vmasclient.go:70] Azure AvailabilitySetsClient (read ops) using rate limit config: QPS=6, bucket=20 I0129 03:44:05.934231 1 azure_vmasclient.go:73] Azure AvailabilitySetsClient (write ops) using rate limit config: QPS=100, bucket=1000 I0129 03:44:05.934311 1 azure.go:1007] attach/detach disk operation rate limit QPS: 6.000000, Bucket: 10 I0129 03:44:05.934337 1 azuredisk.go:192] disable UseInstanceMetadata for controller I0129 03:44:05.934347 1 azuredisk.go:204] cloud: AzurePublicCloud, location: westus2, rg: kubetest-l6t2jkxe, VMType: vmss, PrimaryScaleSetName: k8s-agentpool-14513215-vmss, PrimaryAvailabilitySetName: , DisableAvailabilitySetNodes: false I0129 03:44:05.938358 1 mount_linux.go:287] 'umount /tmp/kubelet-detect-safe-umount2532073634' failed with: exit status 32, output: umount: /tmp/kubelet-detect-safe-umount2532073634: must be superuser to unmount. I0129 03:44:05.938393 1 mount_linux.go:289] Detected umount with unsafe 'not mounted' behavior I0129 03:44:05.938561 1 driver.go:81] Enabling controller service capability: CREATE_DELETE_VOLUME I0129 03:44:05.938579 1 driver.go:81] Enabling controller service capability: PUBLISH_UNPUBLISH_VOLUME I0129 03:44:05.938586 1 driver.go:81] Enabling controller service capability: CREATE_DELETE_SNAPSHOT I0129 03:44:05.938592 1 driver.go:81] Enabling controller service capability: CLONE_VOLUME I0129 03:44:05.938598 1 driver.go:81] Enabling controller service capability: EXPAND_VOLUME ... skipping 69 lines ... I0129 03:44:15.928682 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 32358 I0129 03:44:15.931790 1 controllerserver.go:383] GetDiskLun returned: cannot find Lun for disk pvc-bac0c1f2-9486-47f7-8776-c6f85194ad9b. Initiating attaching volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-bac0c1f2-9486-47f7-8776-c6f85194ad9b to node k8s-agentpool-14513215-vmss000000 (vmState Succeeded). I0129 03:44:15.931844 1 controllerserver.go:408] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-bac0c1f2-9486-47f7-8776-c6f85194ad9b to node k8s-agentpool-14513215-vmss000000 I0129 03:44:15.931871 1 azure_controller_common.go:247] waiting for more requests on node k8s-agentpool-14513215-vmss000000, current disk attach: &{{%!s(*http.Response=&{200 OK 200 HTTP/1.1 1 1 map[Cache-Control:[no-cache] Content-Type:[application/json; charset=utf-8] Date:[Sun, 29 Jan 2023 03:44:15 GMT] Expires:[-1] Pragma:[no-cache] Server:[Microsoft-HTTPAPI/2.0 Microsoft-HTTPAPI/2.0] Strict-Transport-Security:[max-age=31536000; includeSubDomains] Vary:[Accept-Encoding] X-Content-Type-Options:[nosniff] X-Ms-Correlation-Request-Id:[de8ae470-48d4-4903-a4b1-c245a5dd34f3] X-Ms-Ratelimit-Remaining-Resource:[Microsoft.Compute/LowCostGet3Min;14997,Microsoft.Compute/LowCostGet30Min;119979] X-Ms-Ratelimit-Remaining-Subscription-Reads:[11996] X-Ms-Request-Id:[86a68d5a-1595-4b1c-bce6-35747548f756] X-Ms-Routing-Request-Id:[WESTUS2:20230129T034415Z:de8ae470-48d4-4903-a4b1-c245a5dd34f3] X-Ms-Served-By:[43c1a32f-0327-4d8b-a820-a28f01b4195d_133148057271908261]] {0xc00095c300} -1 [chunked] false true map[] 0xc000662a00 0xc0005fb550})} %!s(*string=<nil>) %!s(*[]string=<nil>) %!s(*compute.DiskSku=&{UltraSSD_LRS 0xc000193b50}) %!s(*[]string=&[1]) %!s(*compute.ExtendedLocation=<nil>) %!s(*compute.DiskProperties=&{0xc00095a4b0 <nil> <nil> 0xc00092c8a0 0xc000913e88 0xc000913f10 0xc000193bc0 <nil> 0xc000193bb0 0xc000913e90 0xc000913ea0 0xc000913eb0 0xc000913ec0 Unattached 0xc00095a498 0xc000913ecc <nil> AllowAll <nil> <nil> <nil> <nil> <nil> <nil> <nil> Enabled }) %!s(*string=0xc000193be0) %!s(*string=0xc000193bd0) %!s(*string=0xc000193bf0) %!s(*string=0xc000193c00) map[k8s-azure-created-by:%!s(*string=0xc000193c10) kubernetes.io-created-for-pv-name:%!s(*string=0xc000193c30) kubernetes.io-created-for-pvc-name:%!s(*string=0xc000193ca0) kubernetes.io-created-for-pvc-namespace:%!s(*string=0xc000193d20)]} I0129 03:44:16.932126 1 azure_controller_common.go:261] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-bac0c1f2-9486-47f7-8776-c6f85194ad9b lun 0 to node k8s-agentpool-14513215-vmss000000, diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-bac0c1f2-9486-47f7-8776-c6f85194ad9b:%!s(*provider.AttachDiskOptions=&{None pvc-bac0c1f2-9486-47f7-8776-c6f85194ad9b false 0})] I0129 03:44:16.932216 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-l6t2jkxe): vm(k8s-agentpool-14513215-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-bac0c1f2-9486-47f7-8776-c6f85194ad9b:%!s(*provider.AttachDiskOptions=&{None pvc-bac0c1f2-9486-47f7-8776-c6f85194ad9b false 0})]) I0129 03:44:17.590845 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-l6t2jkxe): vm(k8s-agentpool-14513215-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-bac0c1f2-9486-47f7-8776-c6f85194ad9b:%!s(*provider.AttachDiskOptions=&{None pvc-bac0c1f2-9486-47f7-8776-c6f85194ad9b false 0})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I0129 03:44:27.683936 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-l6t2jkxe, k8s-agentpool-14513215-vmss, k8s-agentpool-14513215-vmss000000) successfully I0129 03:44:27.683974 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool-14513215-vmss, kubetest-l6t2jkxe, k8s-agentpool-14513215-vmss000000) for cacheKey(kubetest-l6t2jkxe/k8s-agentpool-14513215-vmss) updated successfully I0129 03:44:27.684042 1 controllerserver.go:413] Attach operation successful: volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-bac0c1f2-9486-47f7-8776-c6f85194ad9b attached to node k8s-agentpool-14513215-vmss000000. I0129 03:44:27.684251 1 controllerserver.go:433] attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-bac0c1f2-9486-47f7-8776-c6f85194ad9b to node k8s-agentpool-14513215-vmss000000 successfully I0129 03:44:27.684406 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=11.90904118 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-l6t2jkxe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-bac0c1f2-9486-47f7-8776-c6f85194ad9b" node="k8s-agentpool-14513215-vmss000000" result_code="succeeded" I0129 03:44:27.684433 1 utils.go:84] GRPC response: {"publish_context":{"LUN":"0"}} ... skipping 19 lines ... I0129 03:45:53.288139 1 controllerserver.go:319] delete azure disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-bac0c1f2-9486-47f7-8776-c6f85194ad9b) returned with <nil> I0129 03:45:53.288185 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=5.168174121 request="azuredisk_csi_driver_controller_delete_volume" resource_group="kubetest-l6t2jkxe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-bac0c1f2-9486-47f7-8776-c6f85194ad9b" result_code="succeeded" I0129 03:45:53.288202 1 utils.go:84] GRPC response: {} I0129 03:45:57.211669 1 utils.go:77] GRPC call: /csi.v1.Controller/CreateVolume I0129 03:45:57.211698 1 utils.go:78] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.disk.csi.azure.com/zone":"westus2-1","topology.kubernetes.io/zone":"westus2-1"}},{"segments":{"topology.disk.csi.azure.com/zone":"westus2-2","topology.kubernetes.io/zone":"westus2-2"}}],"requisite":[{"segments":{"topology.disk.csi.azure.com/zone":"westus2-1","topology.kubernetes.io/zone":"westus2-1"}},{"segments":{"topology.disk.csi.azure.com/zone":"westus2-2","topology.kubernetes.io/zone":"westus2-2"}}]},"capacity_range":{"required_bytes":10737418240},"name":"pvc-ddefd3ef-a9a4-485e-8d04-aae7cce90b03","parameters":{"csi.storage.k8s.io/pv/name":"pvc-ddefd3ef-a9a4-485e-8d04-aae7cce90b03","csi.storage.k8s.io/pvc/name":"pvc-8jnrg","csi.storage.k8s.io/pvc/namespace":"azuredisk-2540","enableAsyncAttach":"false","networkAccessPolicy":"DenyAll","skuName":"Standard_LRS","userAgent":"azuredisk-e2e-test"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]} I0129 03:45:57.212734 1 azure_disk_utils.go:162] reading cloud config from secret kube-system/azure-cloud-provider I0129 03:45:57.252306 1 azure_disk_utils.go:169] InitializeCloudFromSecret: failed to get cloud config from secret kube-system/azure-cloud-provider: failed to get secret kube-system/azure-cloud-provider: secrets "azure-cloud-provider" not found I0129 03:45:57.252333 1 azure_disk_utils.go:174] could not read cloud config from secret kube-system/azure-cloud-provider I0129 03:45:57.252345 1 azure_disk_utils.go:184] use default AZURE_CREDENTIAL_FILE env var: /etc/kubernetes/azure.json I0129 03:45:57.252383 1 azure_disk_utils.go:192] read cloud config from file: /etc/kubernetes/azure.json successfully I0129 03:45:57.252838 1 azure_auth.go:253] Using AzurePublicCloud environment I0129 03:45:57.252880 1 azure_auth.go:138] azure: using client_id+client_secret to retrieve access token I0129 03:45:57.252899 1 azure.go:776] Azure cloudprovider using try backoff: retries=6, exponent=1.500000, duration=5, jitter=1.000000 ... skipping 38 lines ... I0129 03:46:01.660270 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1218 I0129 03:46:01.660776 1 controllerserver.go:383] GetDiskLun returned: cannot find Lun for disk pvc-ddefd3ef-a9a4-485e-8d04-aae7cce90b03. Initiating attaching volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-ddefd3ef-a9a4-485e-8d04-aae7cce90b03 to node k8s-agentpool-14513215-vmss000000 (vmState Succeeded). I0129 03:46:01.660843 1 controllerserver.go:408] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-ddefd3ef-a9a4-485e-8d04-aae7cce90b03 to node k8s-agentpool-14513215-vmss000000 I0129 03:46:01.660971 1 azure_controller_common.go:247] waiting for more requests on node k8s-agentpool-14513215-vmss000000, current disk attach: &{{%!s(*http.Response=&{200 OK 200 HTTP/1.1 1 1 map[Cache-Control:[no-cache] Content-Type:[application/json; charset=utf-8] Date:[Sun, 29 Jan 2023 03:46:01 GMT] Expires:[-1] Pragma:[no-cache] Server:[Microsoft-HTTPAPI/2.0 Microsoft-HTTPAPI/2.0] Strict-Transport-Security:[max-age=31536000; includeSubDomains] Vary:[Accept-Encoding] X-Content-Type-Options:[nosniff] X-Ms-Correlation-Request-Id:[8f58a62b-0724-4b82-b6b9-770ee41d5cc7] X-Ms-Ratelimit-Remaining-Resource:[Microsoft.Compute/LowCostGet3Min;14983,Microsoft.Compute/LowCostGet30Min;119965] X-Ms-Ratelimit-Remaining-Subscription-Reads:[11975] X-Ms-Request-Id:[9909f369-8fda-4051-beb0-a933014e306c] X-Ms-Routing-Request-Id:[WESTUS2:20230129T034601Z:8f58a62b-0724-4b82-b6b9-770ee41d5cc7] X-Ms-Served-By:[43c1a32f-0327-4d8b-a820-a28f01b4195d_133148057271908261]] {0xc000986ab0} -1 [chunked] false true map[] 0xc000a8c000 0xc0005fb550})} %!s(*string=<nil>) %!s(*[]string=<nil>) %!s(*compute.DiskSku=&{Standard_LRS 0xc000097ab0}) %!s(*[]string=&[1]) %!s(*compute.ExtendedLocation=<nil>) %!s(*compute.DiskProperties=&{0xc0007054a0 <nil> <nil> 0xc0002581e0 0xc000740a50 0xc000740ac0 0xc000097c30 <nil> 0xc000097bf0 0xc000740a58 0xc000740a68 <nil> <nil> Unattached 0xc000705488 <nil> <nil> DenyAll <nil> <nil> <nil> <nil> <nil> <nil> <nil> Enabled }) %!s(*string=0xc000097cb0) %!s(*string=0xc000097c70) %!s(*string=0xc000097cf0) %!s(*string=0xc000097d30) map[k8s-azure-created-by:%!s(*string=0xc000097d70) kubernetes.io-created-for-pv-name:%!s(*string=0xc000097df0) kubernetes.io-created-for-pvc-name:%!s(*string=0xc000097e70) kubernetes.io-created-for-pvc-namespace:%!s(*string=0xc000097ed0)]} I0129 03:46:02.661371 1 azure_controller_common.go:261] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-ddefd3ef-a9a4-485e-8d04-aae7cce90b03 lun 0 to node k8s-agentpool-14513215-vmss000000, diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-ddefd3ef-a9a4-485e-8d04-aae7cce90b03:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-ddefd3ef-a9a4-485e-8d04-aae7cce90b03 false 0})] I0129 03:46:02.661446 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-l6t2jkxe): vm(k8s-agentpool-14513215-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-ddefd3ef-a9a4-485e-8d04-aae7cce90b03:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-ddefd3ef-a9a4-485e-8d04-aae7cce90b03 false 0})]) I0129 03:46:02.800966 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-l6t2jkxe): vm(k8s-agentpool-14513215-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-ddefd3ef-a9a4-485e-8d04-aae7cce90b03:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-ddefd3ef-a9a4-485e-8d04-aae7cce90b03 false 0})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I0129 03:46:12.910778 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-l6t2jkxe, k8s-agentpool-14513215-vmss, k8s-agentpool-14513215-vmss000000) successfully I0129 03:46:12.910813 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool-14513215-vmss, kubetest-l6t2jkxe, k8s-agentpool-14513215-vmss000000) for cacheKey(kubetest-l6t2jkxe/k8s-agentpool-14513215-vmss) updated successfully I0129 03:46:12.910834 1 controllerserver.go:413] Attach operation successful: volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-ddefd3ef-a9a4-485e-8d04-aae7cce90b03 attached to node k8s-agentpool-14513215-vmss000000. I0129 03:46:12.910845 1 controllerserver.go:433] attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-ddefd3ef-a9a4-485e-8d04-aae7cce90b03 to node k8s-agentpool-14513215-vmss000000 successfully I0129 03:46:12.910886 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=11.250111439 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-l6t2jkxe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-ddefd3ef-a9a4-485e-8d04-aae7cce90b03" node="k8s-agentpool-14513215-vmss000000" result_code="succeeded" I0129 03:46:12.910908 1 utils.go:84] GRPC response: {"publish_context":{"LUN":"0"}} ... skipping 19 lines ... I0129 03:47:00.128395 1 controllerserver.go:319] delete azure disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-ddefd3ef-a9a4-485e-8d04-aae7cce90b03) returned with <nil> I0129 03:47:00.128434 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=5.201033517 request="azuredisk_csi_driver_controller_delete_volume" resource_group="kubetest-l6t2jkxe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-ddefd3ef-a9a4-485e-8d04-aae7cce90b03" result_code="succeeded" I0129 03:47:00.128451 1 utils.go:84] GRPC response: {} I0129 03:47:05.736752 1 utils.go:77] GRPC call: /csi.v1.Controller/CreateVolume I0129 03:47:05.736773 1 utils.go:78] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.disk.csi.azure.com/zone":"westus2-1","topology.kubernetes.io/zone":"westus2-1"}}],"requisite":[{"segments":{"topology.disk.csi.azure.com/zone":"westus2-1","topology.kubernetes.io/zone":"westus2-1"}}]},"capacity_range":{"required_bytes":1099511627776},"name":"pvc-0aa3fb46-6c1a-482a-b7fd-6d814adb1cd2","parameters":{"csi.storage.k8s.io/pv/name":"pvc-0aa3fb46-6c1a-482a-b7fd-6d814adb1cd2","csi.storage.k8s.io/pvc/name":"pvc-mx86l","csi.storage.k8s.io/pvc/namespace":"azuredisk-4728","enableAsyncAttach":"false","enableBursting":"true","perfProfile":"Basic","skuName":"Premium_LRS","userAgent":"azuredisk-e2e-test"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]} I0129 03:47:05.737370 1 azure_disk_utils.go:162] reading cloud config from secret kube-system/azure-cloud-provider I0129 03:47:05.743020 1 azure_disk_utils.go:169] InitializeCloudFromSecret: failed to get cloud config from secret kube-system/azure-cloud-provider: failed to get secret kube-system/azure-cloud-provider: secrets "azure-cloud-provider" not found I0129 03:47:05.743039 1 azure_disk_utils.go:174] could not read cloud config from secret kube-system/azure-cloud-provider I0129 03:47:05.743046 1 azure_disk_utils.go:184] use default AZURE_CREDENTIAL_FILE env var: /etc/kubernetes/azure.json I0129 03:47:05.743101 1 azure_disk_utils.go:192] read cloud config from file: /etc/kubernetes/azure.json successfully I0129 03:47:05.743769 1 azure_auth.go:253] Using AzurePublicCloud environment I0129 03:47:05.743805 1 azure_auth.go:138] azure: using client_id+client_secret to retrieve access token I0129 03:47:05.743823 1 azure.go:776] Azure cloudprovider using try backoff: retries=6, exponent=1.500000, duration=5, jitter=1.000000 ... skipping 38 lines ... I0129 03:47:08.920669 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1338 I0129 03:47:08.921232 1 controllerserver.go:383] GetDiskLun returned: cannot find Lun for disk pvc-0aa3fb46-6c1a-482a-b7fd-6d814adb1cd2. Initiating attaching volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-0aa3fb46-6c1a-482a-b7fd-6d814adb1cd2 to node k8s-agentpool-14513215-vmss000000 (vmState Succeeded). I0129 03:47:08.921263 1 controllerserver.go:408] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-0aa3fb46-6c1a-482a-b7fd-6d814adb1cd2 to node k8s-agentpool-14513215-vmss000000 I0129 03:47:08.921289 1 azure_controller_common.go:247] waiting for more requests on node k8s-agentpool-14513215-vmss000000, current disk attach: &{{%!s(*http.Response=&{200 OK 200 HTTP/1.1 1 1 map[Cache-Control:[no-cache] Content-Type:[application/json; charset=utf-8] Date:[Sun, 29 Jan 2023 03:47:08 GMT] Expires:[-1] Pragma:[no-cache] Server:[Microsoft-HTTPAPI/2.0 Microsoft-HTTPAPI/2.0] Strict-Transport-Security:[max-age=31536000; includeSubDomains] Vary:[Accept-Encoding] X-Content-Type-Options:[nosniff] X-Ms-Correlation-Request-Id:[556b8656-fd7f-47e2-887e-c797794137de] X-Ms-Ratelimit-Remaining-Resource:[Microsoft.Compute/LowCostGet3Min;14981,Microsoft.Compute/LowCostGet30Min;119956] X-Ms-Ratelimit-Remaining-Subscription-Reads:[11959] X-Ms-Request-Id:[3c00b48d-5870-41ae-931f-349e70b6a316] X-Ms-Routing-Request-Id:[WESTUS2:20230129T034708Z:556b8656-fd7f-47e2-887e-c797794137de] X-Ms-Served-By:[43c1a32f-0327-4d8b-a820-a28f01b4195d_133148057271908261]] {0xc000660720} -1 [chunked] false true map[] 0xc0008abf00 0xc0005fb550})} %!s(*string=<nil>) %!s(*[]string=<nil>) %!s(*compute.DiskSku=&{Premium_LRS 0xc0004fcc40}) %!s(*[]string=&[1]) %!s(*compute.ExtendedLocation=<nil>) %!s(*compute.DiskProperties=&{0xc000201560 <nil> <nil> 0xc000413e00 0xc000a95388 0xc000a95410 0xc0004fcd80 <nil> 0xc0004fcd40 0xc000a95390 0xc000a953a0 <nil> <nil> Unattached 0xc000201548 <nil> <nil> AllowAll <nil> 0xc0004fcdc0 0xc000a953ab <nil> <nil> <nil> <nil> Enabled }) %!s(*string=0xc0004fce40) %!s(*string=0xc0004fce00) %!s(*string=0xc0004fce80) %!s(*string=0xc0004fcea0) map[k8s-azure-created-by:%!s(*string=0xc0004fced0) kubernetes.io-created-for-pv-name:%!s(*string=0xc0004fcf50) kubernetes.io-created-for-pvc-name:%!s(*string=0xc0004fcfd0) kubernetes.io-created-for-pvc-namespace:%!s(*string=0xc0004fd030)]} I0129 03:47:09.921678 1 azure_controller_common.go:261] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-0aa3fb46-6c1a-482a-b7fd-6d814adb1cd2 lun 0 to node k8s-agentpool-14513215-vmss000000, diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-0aa3fb46-6c1a-482a-b7fd-6d814adb1cd2:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-0aa3fb46-6c1a-482a-b7fd-6d814adb1cd2 false 0})] I0129 03:47:09.921766 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-l6t2jkxe): vm(k8s-agentpool-14513215-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-0aa3fb46-6c1a-482a-b7fd-6d814adb1cd2:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-0aa3fb46-6c1a-482a-b7fd-6d814adb1cd2 false 0})]) I0129 03:47:10.052581 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-l6t2jkxe): vm(k8s-agentpool-14513215-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-0aa3fb46-6c1a-482a-b7fd-6d814adb1cd2:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-0aa3fb46-6c1a-482a-b7fd-6d814adb1cd2 false 0})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I0129 03:47:20.133262 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-l6t2jkxe, k8s-agentpool-14513215-vmss, k8s-agentpool-14513215-vmss000000) successfully I0129 03:47:20.133298 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool-14513215-vmss, kubetest-l6t2jkxe, k8s-agentpool-14513215-vmss000000) for cacheKey(kubetest-l6t2jkxe/k8s-agentpool-14513215-vmss) updated successfully I0129 03:47:20.133316 1 controllerserver.go:413] Attach operation successful: volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-0aa3fb46-6c1a-482a-b7fd-6d814adb1cd2 attached to node k8s-agentpool-14513215-vmss000000. I0129 03:47:20.133329 1 controllerserver.go:433] attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-0aa3fb46-6c1a-482a-b7fd-6d814adb1cd2 to node k8s-agentpool-14513215-vmss000000 successfully I0129 03:47:20.133370 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=11.212143293 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-l6t2jkxe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-0aa3fb46-6c1a-482a-b7fd-6d814adb1cd2" node="k8s-agentpool-14513215-vmss000000" result_code="succeeded" I0129 03:47:20.133389 1 utils.go:84] GRPC response: {"publish_context":{"LUN":"0"}} ... skipping 34 lines ... I0129 03:48:48.188549 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1193 I0129 03:48:48.188835 1 controllerserver.go:383] GetDiskLun returned: cannot find Lun for disk pvc-6423556c-e23b-4d33-b659-6d4fe7fc8557. Initiating attaching volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-6423556c-e23b-4d33-b659-6d4fe7fc8557 to node k8s-agentpool-14513215-vmss000000 (vmState Succeeded). I0129 03:48:48.188865 1 controllerserver.go:408] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-6423556c-e23b-4d33-b659-6d4fe7fc8557 to node k8s-agentpool-14513215-vmss000000 I0129 03:48:48.188887 1 azure_controller_common.go:247] waiting for more requests on node k8s-agentpool-14513215-vmss000000, current disk attach: &{{%!s(*http.Response=&{200 OK 200 HTTP/1.1 1 1 map[Cache-Control:[no-cache] Content-Type:[application/json; charset=utf-8] Date:[Sun, 29 Jan 2023 03:48:48 GMT] Expires:[-1] Pragma:[no-cache] Server:[Microsoft-HTTPAPI/2.0 Microsoft-HTTPAPI/2.0] Strict-Transport-Security:[max-age=31536000; includeSubDomains] Vary:[Accept-Encoding] X-Content-Type-Options:[nosniff] X-Ms-Correlation-Request-Id:[513872d0-9c8a-4b2b-a1ad-a3093520ca0f] X-Ms-Ratelimit-Remaining-Resource:[Microsoft.Compute/LowCostGet3Min;14972,Microsoft.Compute/LowCostGet30Min;119938] X-Ms-Ratelimit-Remaining-Subscription-Reads:[11942] X-Ms-Request-Id:[6c011135-ac82-48b7-a5cb-0c2c9ca3f6f1] X-Ms-Routing-Request-Id:[WESTUS2:20230129T034848Z:513872d0-9c8a-4b2b-a1ad-a3093520ca0f] X-Ms-Served-By:[43c1a32f-0327-4d8b-a820-a28f01b4195d_133148057271908261]] {0xc0001c74a0} -1 [chunked] false true map[] 0xc000663b00 0xc0005fb550})} %!s(*string=<nil>) %!s(*[]string=<nil>) %!s(*compute.DiskSku=&{StandardSSD_ZRS 0xc0001b9290}) %!s(*[]string=<nil>) %!s(*compute.ExtendedLocation=<nil>) %!s(*compute.DiskProperties=&{0xc000a996e0 <nil> <nil> 0xc0006e8f60 0xc000a95948 0xc000a959d0 0xc0001b92f0 <nil> 0xc0001b92e0 0xc000a95950 0xc000a95960 <nil> <nil> Unattached 0xc000a996c8 <nil> <nil> AllowAll <nil> <nil> <nil> <nil> <nil> <nil> <nil> Enabled }) %!s(*string=0xc0001b9320) %!s(*string=0xc0001b9310) %!s(*string=0xc0001b9330) %!s(*string=0xc0001b9340) map[k8s-azure-created-by:%!s(*string=0xc0001b9350) kubernetes.io-created-for-pv-name:%!s(*string=0xc0001b9370) kubernetes.io-created-for-pvc-name:%!s(*string=0xc0001b9390) kubernetes.io-created-for-pvc-namespace:%!s(*string=0xc0001b93b0)]} I0129 03:48:49.189325 1 azure_controller_common.go:261] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-6423556c-e23b-4d33-b659-6d4fe7fc8557 lun 0 to node k8s-agentpool-14513215-vmss000000, diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-6423556c-e23b-4d33-b659-6d4fe7fc8557:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-6423556c-e23b-4d33-b659-6d4fe7fc8557 false 0})] I0129 03:48:49.189408 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-l6t2jkxe): vm(k8s-agentpool-14513215-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-6423556c-e23b-4d33-b659-6d4fe7fc8557:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-6423556c-e23b-4d33-b659-6d4fe7fc8557 false 0})]) I0129 03:48:49.358861 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-l6t2jkxe): vm(k8s-agentpool-14513215-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-6423556c-e23b-4d33-b659-6d4fe7fc8557:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-6423556c-e23b-4d33-b659-6d4fe7fc8557 false 0})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I0129 03:48:59.455462 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-l6t2jkxe, k8s-agentpool-14513215-vmss, k8s-agentpool-14513215-vmss000000) successfully I0129 03:48:59.455499 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool-14513215-vmss, kubetest-l6t2jkxe, k8s-agentpool-14513215-vmss000000) for cacheKey(kubetest-l6t2jkxe/k8s-agentpool-14513215-vmss) updated successfully I0129 03:48:59.455519 1 controllerserver.go:413] Attach operation successful: volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-6423556c-e23b-4d33-b659-6d4fe7fc8557 attached to node k8s-agentpool-14513215-vmss000000. I0129 03:48:59.455534 1 controllerserver.go:433] attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-6423556c-e23b-4d33-b659-6d4fe7fc8557 to node k8s-agentpool-14513215-vmss000000 successfully I0129 03:48:59.455575 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=11.266742942 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-l6t2jkxe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-6423556c-e23b-4d33-b659-6d4fe7fc8557" node="k8s-agentpool-14513215-vmss000000" result_code="succeeded" I0129 03:48:59.455594 1 utils.go:84] GRPC response: {"publish_context":{"LUN":"0"}} ... skipping 34 lines ... I0129 03:49:57.499698 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1193 I0129 03:49:57.500128 1 controllerserver.go:383] GetDiskLun returned: cannot find Lun for disk pvc-840f6b42-69f6-4ce0-ad7e-3b7494e2b2db. Initiating attaching volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-840f6b42-69f6-4ce0-ad7e-3b7494e2b2db to node k8s-agentpool-14513215-vmss000000 (vmState Succeeded). I0129 03:49:57.500156 1 controllerserver.go:408] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-840f6b42-69f6-4ce0-ad7e-3b7494e2b2db to node k8s-agentpool-14513215-vmss000000 I0129 03:49:57.500277 1 azure_controller_common.go:247] waiting for more requests on node k8s-agentpool-14513215-vmss000000, current disk attach: &{{%!s(*http.Response=&{200 OK 200 HTTP/1.1 1 1 map[Cache-Control:[no-cache] Content-Type:[application/json; charset=utf-8] Date:[Sun, 29 Jan 2023 03:49:57 GMT] Expires:[-1] Pragma:[no-cache] Server:[Microsoft-HTTPAPI/2.0 Microsoft-HTTPAPI/2.0] Strict-Transport-Security:[max-age=31536000; includeSubDomains] Vary:[Accept-Encoding] X-Content-Type-Options:[nosniff] X-Ms-Correlation-Request-Id:[fcd8f99a-77cc-4bc5-8ac0-6c660e9beda0] X-Ms-Ratelimit-Remaining-Resource:[Microsoft.Compute/LowCostGet3Min;14971,Microsoft.Compute/LowCostGet30Min;119930] X-Ms-Ratelimit-Remaining-Subscription-Reads:[11926] X-Ms-Request-Id:[1896e7cf-18aa-45eb-8b44-02a61c2b8895] X-Ms-Routing-Request-Id:[WESTUS2:20230129T034957Z:fcd8f99a-77cc-4bc5-8ac0-6c660e9beda0] X-Ms-Served-By:[43c1a32f-0327-4d8b-a820-a28f01b4195d_133148057271908261]] {0xc000a8a150} -1 [chunked] false true map[] 0xc0008ab900 0xc0005fb550})} %!s(*string=<nil>) %!s(*[]string=<nil>) %!s(*compute.DiskSku=&{StandardSSD_ZRS 0xc0004f3130}) %!s(*[]string=<nil>) %!s(*compute.ExtendedLocation=<nil>) %!s(*compute.DiskProperties=&{0xc0007cb6c8 <nil> <nil> 0xc000259140 0xc000807b10 0xc000807c70 0xc0004f3330 <nil> 0xc0004f32f0 0xc000807b18 0xc000807b28 <nil> <nil> Unattached 0xc0007cb6b0 <nil> <nil> AllowAll <nil> <nil> <nil> <nil> <nil> <nil> <nil> Enabled }) %!s(*string=0xc0004f33f0) %!s(*string=0xc0004f3370) %!s(*string=0xc0004f3470) %!s(*string=0xc0004f3490) map[k8s-azure-created-by:%!s(*string=0xc0004f34d0) kubernetes.io-created-for-pv-name:%!s(*string=0xc0004f3590) kubernetes.io-created-for-pvc-name:%!s(*string=0xc0004f3620) kubernetes.io-created-for-pvc-namespace:%!s(*string=0xc0004f36b0)]} I0129 03:49:58.500621 1 azure_controller_common.go:261] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-840f6b42-69f6-4ce0-ad7e-3b7494e2b2db lun 0 to node k8s-agentpool-14513215-vmss000000, diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-840f6b42-69f6-4ce0-ad7e-3b7494e2b2db:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-840f6b42-69f6-4ce0-ad7e-3b7494e2b2db false 0})] I0129 03:49:58.500705 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-l6t2jkxe): vm(k8s-agentpool-14513215-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-840f6b42-69f6-4ce0-ad7e-3b7494e2b2db:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-840f6b42-69f6-4ce0-ad7e-3b7494e2b2db false 0})]) I0129 03:49:58.697457 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-l6t2jkxe): vm(k8s-agentpool-14513215-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-840f6b42-69f6-4ce0-ad7e-3b7494e2b2db:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-840f6b42-69f6-4ce0-ad7e-3b7494e2b2db false 0})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I0129 03:50:08.784995 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-l6t2jkxe, k8s-agentpool-14513215-vmss, k8s-agentpool-14513215-vmss000000) successfully I0129 03:50:08.785035 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool-14513215-vmss, kubetest-l6t2jkxe, k8s-agentpool-14513215-vmss000000) for cacheKey(kubetest-l6t2jkxe/k8s-agentpool-14513215-vmss) updated successfully I0129 03:50:08.785057 1 controllerserver.go:413] Attach operation successful: volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-840f6b42-69f6-4ce0-ad7e-3b7494e2b2db attached to node k8s-agentpool-14513215-vmss000000. I0129 03:50:08.785073 1 controllerserver.go:433] attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-840f6b42-69f6-4ce0-ad7e-3b7494e2b2db to node k8s-agentpool-14513215-vmss000000 successfully I0129 03:50:08.785117 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=11.284983999 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-l6t2jkxe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-840f6b42-69f6-4ce0-ad7e-3b7494e2b2db" node="k8s-agentpool-14513215-vmss000000" result_code="succeeded" I0129 03:50:08.785136 1 utils.go:84] GRPC response: {"publish_context":{"LUN":"0"}} ... skipping 18 lines ... I0129 03:50:47.643361 1 azure_controller_common.go:410] Trying to detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-840f6b42-69f6-4ce0-ad7e-3b7494e2b2db from node k8s-agentpool-14513215-vmss000000, diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-840f6b42-69f6-4ce0-ad7e-3b7494e2b2db:pvc-840f6b42-69f6-4ce0-ad7e-3b7494e2b2db] E0129 03:50:47.643434 1 azure_controller_vmss.go:202] detach azure disk on node(k8s-agentpool-14513215-vmss000000): disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-840f6b42-69f6-4ce0-ad7e-3b7494e2b2db:pvc-840f6b42-69f6-4ce0-ad7e-3b7494e2b2db]) not found I0129 03:50:47.643495 1 azure_controller_vmss.go:239] azureDisk - update(kubetest-l6t2jkxe): vm(k8s-agentpool-14513215-vmss000000) - detach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-840f6b42-69f6-4ce0-ad7e-3b7494e2b2db:pvc-840f6b42-69f6-4ce0-ad7e-3b7494e2b2db]) I0129 03:50:51.821698 1 utils.go:77] GRPC call: /csi.v1.Controller/DeleteVolume I0129 03:50:51.821727 1 utils.go:78] GRPC request: {"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-840f6b42-69f6-4ce0-ad7e-3b7494e2b2db"} I0129 03:50:51.821824 1 controllerserver.go:317] deleting azure disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-840f6b42-69f6-4ce0-ad7e-3b7494e2b2db) I0129 03:50:51.821839 1 controllerserver.go:319] delete azure disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-840f6b42-69f6-4ce0-ad7e-3b7494e2b2db) returned with failed to delete disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-840f6b42-69f6-4ce0-ad7e-3b7494e2b2db) since it's in attaching or detaching state I0129 03:50:51.821897 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=3.08e-05 request="azuredisk_csi_driver_controller_delete_volume" resource_group="kubetest-l6t2jkxe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-840f6b42-69f6-4ce0-ad7e-3b7494e2b2db" result_code="failed_csi_driver_controller_delete_volume" E0129 03:50:51.821912 1 utils.go:82] GRPC error: failed to delete disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-840f6b42-69f6-4ce0-ad7e-3b7494e2b2db) since it's in attaching or detaching state I0129 03:50:52.797281 1 azure_controller_vmss.go:252] azureDisk - update(kubetest-l6t2jkxe): vm(k8s-agentpool-14513215-vmss000000) - detach disk(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-840f6b42-69f6-4ce0-ad7e-3b7494e2b2db:pvc-840f6b42-69f6-4ce0-ad7e-3b7494e2b2db]) returned with <nil> I0129 03:50:52.797340 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-l6t2jkxe, k8s-agentpool-14513215-vmss, k8s-agentpool-14513215-vmss000000) successfully I0129 03:50:52.797359 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool-14513215-vmss, kubetest-l6t2jkxe, k8s-agentpool-14513215-vmss000000) for cacheKey(kubetest-l6t2jkxe/k8s-agentpool-14513215-vmss) updated successfully I0129 03:50:52.797371 1 azure_controller_common.go:434] azureDisk - detach disk(pvc-840f6b42-69f6-4ce0-ad7e-3b7494e2b2db, /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-840f6b42-69f6-4ce0-ad7e-3b7494e2b2db) succeeded I0129 03:50:52.797383 1 controllerserver.go:480] detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-840f6b42-69f6-4ce0-ad7e-3b7494e2b2db from node k8s-agentpool-14513215-vmss000000 successfully I0129 03:50:52.797452 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=6.155038117 request="azuredisk_csi_driver_controller_unpublish_volume" resource_group="kubetest-l6t2jkxe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-840f6b42-69f6-4ce0-ad7e-3b7494e2b2db" node="k8s-agentpool-14513215-vmss000000" result_code="succeeded" ... skipping 21 lines ... I0129 03:51:37.035497 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1238 I0129 03:51:37.035809 1 controllerserver.go:383] GetDiskLun returned: cannot find Lun for disk pvc-cad3d7b2-3e89-4b27-bdb3-9953079be191. Initiating attaching volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/azuredisk-csi-driver-test-3856d907-9f88-11ed-987b-d279894c5862/providers/Microsoft.Compute/disks/pvc-cad3d7b2-3e89-4b27-bdb3-9953079be191 to node k8s-agentpool-14513215-vmss000000 (vmState Succeeded). I0129 03:51:37.035847 1 controllerserver.go:408] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/azuredisk-csi-driver-test-3856d907-9f88-11ed-987b-d279894c5862/providers/Microsoft.Compute/disks/pvc-cad3d7b2-3e89-4b27-bdb3-9953079be191 to node k8s-agentpool-14513215-vmss000000 I0129 03:51:37.035875 1 azure_controller_common.go:247] waiting for more requests on node k8s-agentpool-14513215-vmss000000, current disk attach: &{{%!s(*http.Response=&{200 OK 200 HTTP/1.1 1 1 map[Cache-Control:[no-cache] Content-Type:[application/json; charset=utf-8] Date:[Sun, 29 Jan 2023 03:51:36 GMT] Expires:[-1] Pragma:[no-cache] Server:[Microsoft-HTTPAPI/2.0 Microsoft-HTTPAPI/2.0] Strict-Transport-Security:[max-age=31536000; includeSubDomains] Vary:[Accept-Encoding] X-Content-Type-Options:[nosniff] X-Ms-Correlation-Request-Id:[d91bc367-b0e6-4a26-b719-f973de65d8bd] X-Ms-Ratelimit-Remaining-Resource:[Microsoft.Compute/LowCostGet3Min;14983,Microsoft.Compute/LowCostGet30Min;119917] X-Ms-Ratelimit-Remaining-Subscription-Reads:[11906] X-Ms-Request-Id:[27ace08d-08ad-4200-9ecf-112641e2446f] X-Ms-Routing-Request-Id:[WESTUS2:20230129T035137Z:d91bc367-b0e6-4a26-b719-f973de65d8bd] X-Ms-Served-By:[43c1a32f-0327-4d8b-a820-a28f01b4195d_133148057271908261]] {0xc00037bf80} -1 [chunked] false true map[] 0xc0005f6500 0xc0005fb550})} %!s(*string=<nil>) %!s(*[]string=<nil>) %!s(*compute.DiskSku=&{StandardSSD_ZRS 0xc0004ffbf0}) %!s(*[]string=<nil>) %!s(*compute.ExtendedLocation=<nil>) %!s(*compute.DiskProperties=&{0xc000804db0 <nil> <nil> 0xc0008be7e0 0xc0007d4bc0 0xc0007d4c30 0xc0004ffd60 <nil> 0xc0004ffd20 0xc0007d4bc8 0xc0007d4bd8 <nil> <nil> Unattached 0xc000804d98 <nil> <nil> AllowAll <nil> <nil> <nil> <nil> <nil> <nil> <nil> Enabled }) %!s(*string=0xc0004ffde0) %!s(*string=0xc0004ffda0) %!s(*string=0xc0004ffe20) %!s(*string=0xc0004ffe60) map[k8s-azure-created-by:%!s(*string=0xc0004ffea0) kubernetes.io-created-for-pv-name:%!s(*string=0xc0004fff00) kubernetes.io-created-for-pvc-name:%!s(*string=0xc0004fff20) kubernetes.io-created-for-pvc-namespace:%!s(*string=0xc0004fff60)]} I0129 03:51:38.036169 1 azure_controller_common.go:261] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/azuredisk-csi-driver-test-3856d907-9f88-11ed-987b-d279894c5862/providers/Microsoft.Compute/disks/pvc-cad3d7b2-3e89-4b27-bdb3-9953079be191 lun 0 to node k8s-agentpool-14513215-vmss000000, diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/azuredisk-csi-driver-test-3856d907-9f88-11ed-987b-d279894c5862/providers/microsoft.compute/disks/pvc-cad3d7b2-3e89-4b27-bdb3-9953079be191:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-cad3d7b2-3e89-4b27-bdb3-9953079be191 false 0})] I0129 03:51:38.036271 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-l6t2jkxe): vm(k8s-agentpool-14513215-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/azuredisk-csi-driver-test-3856d907-9f88-11ed-987b-d279894c5862/providers/microsoft.compute/disks/pvc-cad3d7b2-3e89-4b27-bdb3-9953079be191:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-cad3d7b2-3e89-4b27-bdb3-9953079be191 false 0})]) I0129 03:51:38.195852 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-l6t2jkxe): vm(k8s-agentpool-14513215-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/azuredisk-csi-driver-test-3856d907-9f88-11ed-987b-d279894c5862/providers/microsoft.compute/disks/pvc-cad3d7b2-3e89-4b27-bdb3-9953079be191:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-cad3d7b2-3e89-4b27-bdb3-9953079be191 false 0})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I0129 03:51:48.270852 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-l6t2jkxe, k8s-agentpool-14513215-vmss, k8s-agentpool-14513215-vmss000000) successfully I0129 03:51:48.270894 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool-14513215-vmss, kubetest-l6t2jkxe, k8s-agentpool-14513215-vmss000000) for cacheKey(kubetest-l6t2jkxe/k8s-agentpool-14513215-vmss) updated successfully I0129 03:51:48.270915 1 controllerserver.go:413] Attach operation successful: volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/azuredisk-csi-driver-test-3856d907-9f88-11ed-987b-d279894c5862/providers/Microsoft.Compute/disks/pvc-cad3d7b2-3e89-4b27-bdb3-9953079be191 attached to node k8s-agentpool-14513215-vmss000000. I0129 03:51:48.270930 1 controllerserver.go:433] attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/azuredisk-csi-driver-test-3856d907-9f88-11ed-987b-d279894c5862/providers/Microsoft.Compute/disks/pvc-cad3d7b2-3e89-4b27-bdb3-9953079be191 to node k8s-agentpool-14513215-vmss000000 successfully I0129 03:51:48.270974 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=11.235160709 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-l6t2jkxe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/azuredisk-csi-driver-test-3856d907-9f88-11ed-987b-d279894c5862/providers/Microsoft.Compute/disks/pvc-cad3d7b2-3e89-4b27-bdb3-9953079be191" node="k8s-agentpool-14513215-vmss000000" result_code="succeeded" I0129 03:51:48.271003 1 utils.go:84] GRPC response: {"publish_context":{"LUN":"0"}} ... skipping 71 lines ... I0129 03:53:00.726930 1 azure_controller_common.go:247] waiting for more requests on node k8s-agentpool-14513215-vmss000000, current disk attach: &{{%!s(*http.Response=&{200 OK 200 HTTP/1.1 1 1 map[Cache-Control:[no-cache] Content-Type:[application/json; charset=utf-8] Date:[Sun, 29 Jan 2023 03:53:00 GMT] Expires:[-1] Pragma:[no-cache] Server:[Microsoft-HTTPAPI/2.0 Microsoft-HTTPAPI/2.0] Strict-Transport-Security:[max-age=31536000; includeSubDomains] Vary:[Accept-Encoding] X-Content-Type-Options:[nosniff] X-Ms-Correlation-Request-Id:[e2d9d2b3-e470-4896-a271-410e8fdae41e] X-Ms-Ratelimit-Remaining-Resource:[Microsoft.Compute/LowCostGet3Min;14983,Microsoft.Compute/LowCostGet30Min;119904] X-Ms-Ratelimit-Remaining-Subscription-Reads:[11993] X-Ms-Request-Id:[63ec45d6-2022-4fe9-94db-66442ce50d61] X-Ms-Routing-Request-Id:[WESTUS2:20230129T035300Z:e2d9d2b3-e470-4896-a271-410e8fdae41e] X-Ms-Served-By:[43c1a32f-0327-4d8b-a820-a28f01b4195d_133148057271908261]] {0xc000a8f110} -1 [chunked] false true map[] 0xc000a8d500 0xc0005da840})} %!s(*string=<nil>) %!s(*[]string=<nil>) %!s(*compute.DiskSku=&{StandardSSD_ZRS 0xc000171280}) %!s(*[]string=<nil>) %!s(*compute.ExtendedLocation=<nil>) %!s(*compute.DiskProperties=&{0xc000745c50 <nil> <nil> 0xc00092c8a0 0xc000741f98 0xc0005c0000 0xc000171550 <nil> 0xc000171520 0xc000741fa0 0xc000741fb0 <nil> <nil> Unattached 0xc000745c38 <nil> <nil> AllowAll <nil> <nil> <nil> <nil> <nil> <nil> <nil> Enabled }) %!s(*string=0xc000170980) %!s(*string=0xc0001715e0) %!s(*string=0xc000170990) %!s(*string=0xc000170b30) map[k8s-azure-created-by:%!s(*string=0xc000170b70) kubernetes.io-created-for-pv-name:%!s(*string=0xc000170ed0) kubernetes.io-created-for-pvc-name:%!s(*string=0xc000170f00) kubernetes.io-created-for-pvc-namespace:%!s(*string=0xc000171090)]} I0129 03:53:00.728520 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1238 I0129 03:53:00.728753 1 controllerserver.go:383] GetDiskLun returned: cannot find Lun for disk pvc-39415a47-e400-409b-9abf-d2f6a0768edc. Initiating attaching volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/azuredisk-csi-driver-test-6a11d7a6-9f88-11ed-987b-d279894c5862/providers/Microsoft.Compute/disks/pvc-39415a47-e400-409b-9abf-d2f6a0768edc to node k8s-agentpool-14513215-vmss000000 (vmState Succeeded). I0129 03:53:00.728772 1 controllerserver.go:408] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/azuredisk-csi-driver-test-6a11d7a6-9f88-11ed-987b-d279894c5862/providers/Microsoft.Compute/disks/pvc-39415a47-e400-409b-9abf-d2f6a0768edc to node k8s-agentpool-14513215-vmss000000 I0129 03:53:01.727244 1 azure_controller_common.go:261] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/azuredisk-csi-driver-test-6a857c0f-9f88-11ed-987b-d279894c5862/providers/Microsoft.Compute/disks/pvc-d7630243-7ff3-4bff-b5b2-b8089e386b5d lun 1 to node k8s-agentpool-14513215-vmss000000, diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/azuredisk-csi-driver-test-6a11d7a6-9f88-11ed-987b-d279894c5862/providers/microsoft.compute/disks/pvc-39415a47-e400-409b-9abf-d2f6a0768edc:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-39415a47-e400-409b-9abf-d2f6a0768edc false 0}) /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/azuredisk-csi-driver-test-6a857c0f-9f88-11ed-987b-d279894c5862/providers/microsoft.compute/disks/pvc-d7630243-7ff3-4bff-b5b2-b8089e386b5d:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-d7630243-7ff3-4bff-b5b2-b8089e386b5d false 1})] I0129 03:53:01.727315 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-l6t2jkxe): vm(k8s-agentpool-14513215-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/azuredisk-csi-driver-test-6a11d7a6-9f88-11ed-987b-d279894c5862/providers/microsoft.compute/disks/pvc-39415a47-e400-409b-9abf-d2f6a0768edc:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-39415a47-e400-409b-9abf-d2f6a0768edc false 0}) /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/azuredisk-csi-driver-test-6a857c0f-9f88-11ed-987b-d279894c5862/providers/microsoft.compute/disks/pvc-d7630243-7ff3-4bff-b5b2-b8089e386b5d:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-d7630243-7ff3-4bff-b5b2-b8089e386b5d false 1})]) I0129 03:53:01.932096 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-l6t2jkxe): vm(k8s-agentpool-14513215-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/azuredisk-csi-driver-test-6a11d7a6-9f88-11ed-987b-d279894c5862/providers/microsoft.compute/disks/pvc-39415a47-e400-409b-9abf-d2f6a0768edc:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-39415a47-e400-409b-9abf-d2f6a0768edc false 0}) /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/azuredisk-csi-driver-test-6a857c0f-9f88-11ed-987b-d279894c5862/providers/microsoft.compute/disks/pvc-d7630243-7ff3-4bff-b5b2-b8089e386b5d:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-d7630243-7ff3-4bff-b5b2-b8089e386b5d false 1})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I0129 03:53:12.006994 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-l6t2jkxe, k8s-agentpool-14513215-vmss, k8s-agentpool-14513215-vmss000000) successfully I0129 03:53:12.007029 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool-14513215-vmss, kubetest-l6t2jkxe, k8s-agentpool-14513215-vmss000000) for cacheKey(kubetest-l6t2jkxe/k8s-agentpool-14513215-vmss) updated successfully I0129 03:53:12.007059 1 controllerserver.go:413] Attach operation successful: volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/azuredisk-csi-driver-test-6a857c0f-9f88-11ed-987b-d279894c5862/providers/Microsoft.Compute/disks/pvc-d7630243-7ff3-4bff-b5b2-b8089e386b5d attached to node k8s-agentpool-14513215-vmss000000. I0129 03:53:12.007074 1 controllerserver.go:433] attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/azuredisk-csi-driver-test-6a857c0f-9f88-11ed-987b-d279894c5862/providers/Microsoft.Compute/disks/pvc-d7630243-7ff3-4bff-b5b2-b8089e386b5d to node k8s-agentpool-14513215-vmss000000 successfully I0129 03:53:12.007119 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=11.280228865 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-l6t2jkxe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/azuredisk-csi-driver-test-6a857c0f-9f88-11ed-987b-d279894c5862/providers/Microsoft.Compute/disks/pvc-d7630243-7ff3-4bff-b5b2-b8089e386b5d" node="k8s-agentpool-14513215-vmss000000" result_code="succeeded" I0129 03:53:12.007138 1 utils.go:84] GRPC response: {"publish_context":{"LUN":"1"}} ... skipping 57 lines ... I0129 03:54:49.054153 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 24989 I0129 03:54:49.058552 1 controllerserver.go:383] GetDiskLun returned: cannot find Lun for disk pvc-63602f55-f07f-4d89-ab63-f93303966afc. Initiating attaching volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-63602f55-f07f-4d89-ab63-f93303966afc to node k8s-agentpool-14513215-vmss000000 (vmState Succeeded). I0129 03:54:49.058586 1 controllerserver.go:408] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-63602f55-f07f-4d89-ab63-f93303966afc to node k8s-agentpool-14513215-vmss000000 I0129 03:54:49.058613 1 azure_controller_common.go:247] waiting for more requests on node k8s-agentpool-14513215-vmss000000, current disk attach: &{{%!s(*http.Response=&{200 OK 200 HTTP/1.1 1 1 map[Cache-Control:[no-cache] Content-Type:[application/json; charset=utf-8] Date:[Sun, 29 Jan 2023 03:54:48 GMT] Expires:[-1] Pragma:[no-cache] Server:[Microsoft-HTTPAPI/2.0 Microsoft-HTTPAPI/2.0] Strict-Transport-Security:[max-age=31536000; includeSubDomains] Vary:[Accept-Encoding] X-Content-Type-Options:[nosniff] X-Ms-Correlation-Request-Id:[19d3c954-46dc-4e4a-84fc-d60de680215a] X-Ms-Ratelimit-Remaining-Resource:[Microsoft.Compute/LowCostGet3Min;14972,Microsoft.Compute/LowCostGet30Min;119883] X-Ms-Ratelimit-Remaining-Subscription-Reads:[11871] X-Ms-Request-Id:[97cbc7b4-31c2-45db-acee-f0ce6e6e3f95] X-Ms-Routing-Request-Id:[WESTUS2:20230129T035448Z:19d3c954-46dc-4e4a-84fc-d60de680215a] X-Ms-Served-By:[43c1a32f-0327-4d8b-a820-a28f01b4195d_133148057271908261]] {0xc0001e9e00} -1 [chunked] false true map[] 0xc0008aa800 0xc0005fb550})} %!s(*string=<nil>) %!s(*[]string=<nil>) %!s(*compute.DiskSku=&{Premium_ZRS 0xc000137630}) %!s(*[]string=<nil>) %!s(*compute.ExtendedLocation=<nil>) %!s(*compute.DiskProperties=&{0xc000805980 <nil> <nil> 0xc00092d500 0xc0007d4e70 0xc0007d4ed0 0xc000137710 <nil> 0xc0001376e0 0xc0007d4e78 0xc0007d4e88 <nil> <nil> Unattached 0xc000805968 <nil> <nil> AllowAll <nil> 0xc000137750 <nil> <nil> <nil> <nil> <nil> Enabled }) %!s(*string=0xc000137400) %!s(*string=0xc0001373e0) %!s(*string=0xc000137410) %!s(*string=0xc000137440) map[k8s-azure-created-by:%!s(*string=0xc000137480) kubernetes.io-created-for-pv-name:%!s(*string=0xc0001374c0) kubernetes.io-created-for-pvc-name:%!s(*string=0xc000137530) kubernetes.io-created-for-pvc-namespace:%!s(*string=0xc0001375b0)]} I0129 03:54:50.062598 1 azure_controller_common.go:261] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-63602f55-f07f-4d89-ab63-f93303966afc lun 0 to node k8s-agentpool-14513215-vmss000000, diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-63602f55-f07f-4d89-ab63-f93303966afc:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-63602f55-f07f-4d89-ab63-f93303966afc false 0})] I0129 03:54:50.062675 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-l6t2jkxe): vm(k8s-agentpool-14513215-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-63602f55-f07f-4d89-ab63-f93303966afc:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-63602f55-f07f-4d89-ab63-f93303966afc false 0})]) I0129 03:54:50.244907 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-l6t2jkxe): vm(k8s-agentpool-14513215-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-63602f55-f07f-4d89-ab63-f93303966afc:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-63602f55-f07f-4d89-ab63-f93303966afc false 0})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I0129 03:55:25.488039 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-l6t2jkxe, k8s-agentpool-14513215-vmss, k8s-agentpool-14513215-vmss000000) successfully I0129 03:55:25.488079 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool-14513215-vmss, kubetest-l6t2jkxe, k8s-agentpool-14513215-vmss000000) for cacheKey(kubetest-l6t2jkxe/k8s-agentpool-14513215-vmss) updated successfully I0129 03:55:25.488100 1 controllerserver.go:413] Attach operation successful: volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-63602f55-f07f-4d89-ab63-f93303966afc attached to node k8s-agentpool-14513215-vmss000000. I0129 03:55:25.488115 1 controllerserver.go:433] attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-63602f55-f07f-4d89-ab63-f93303966afc to node k8s-agentpool-14513215-vmss000000 successfully I0129 03:55:25.488157 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=36.495971103 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-l6t2jkxe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-63602f55-f07f-4d89-ab63-f93303966afc" node="k8s-agentpool-14513215-vmss000000" result_code="succeeded" I0129 03:55:25.488172 1 utils.go:84] GRPC response: {"publish_context":{"LUN":"0"}} ... skipping 48 lines ... I0129 03:56:43.247299 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1193 I0129 03:56:43.247588 1 controllerserver.go:383] GetDiskLun returned: cannot find Lun for disk pvc-71e6a6cd-f320-41bd-9667-f0534b913f79. Initiating attaching volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-71e6a6cd-f320-41bd-9667-f0534b913f79 to node k8s-agentpool-14513215-vmss000000 (vmState Succeeded). I0129 03:56:43.247626 1 controllerserver.go:408] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-71e6a6cd-f320-41bd-9667-f0534b913f79 to node k8s-agentpool-14513215-vmss000000 I0129 03:56:43.247648 1 azure_controller_common.go:247] waiting for more requests on node k8s-agentpool-14513215-vmss000000, current disk attach: &{{%!s(*http.Response=&{200 OK 200 HTTP/1.1 1 1 map[Cache-Control:[no-cache] Content-Type:[application/json; charset=utf-8] Date:[Sun, 29 Jan 2023 03:56:43 GMT] Expires:[-1] Pragma:[no-cache] Server:[Microsoft-HTTPAPI/2.0 Microsoft-HTTPAPI/2.0] Strict-Transport-Security:[max-age=31536000; includeSubDomains] Vary:[Accept-Encoding] X-Content-Type-Options:[nosniff] X-Ms-Correlation-Request-Id:[821e3175-fe20-4a54-b04b-af30cada390d] X-Ms-Ratelimit-Remaining-Resource:[Microsoft.Compute/LowCostGet3Min;14961,Microsoft.Compute/LowCostGet30Min;119865] X-Ms-Ratelimit-Remaining-Subscription-Reads:[11846] X-Ms-Request-Id:[b746d989-1f1b-4e69-b3c9-140a46daa43d] X-Ms-Routing-Request-Id:[WESTUS2:20230129T035643Z:821e3175-fe20-4a54-b04b-af30cada390d] X-Ms-Served-By:[43c1a32f-0327-4d8b-a820-a28f01b4195d_133148057271908261]] {0xc00001af90} -1 [chunked] false true map[] 0xc000a8d700 0xc0005fb550})} %!s(*string=<nil>) %!s(*[]string=<nil>) %!s(*compute.DiskSku=&{StandardSSD_ZRS 0xc000031e90}) %!s(*[]string=<nil>) %!s(*compute.ExtendedLocation=<nil>) %!s(*compute.DiskProperties=&{0xc0008054a0 <nil> <nil> 0xc000412360 0xc000484110 0xc0004849b0 0xc000031fc0 <nil> 0xc000031fa0 0xc000484118 0xc0004841a8 <nil> <nil> Unattached 0xc000805488 <nil> <nil> AllowAll <nil> <nil> <nil> <nil> <nil> <nil> <nil> Enabled }) %!s(*string=0xc000031ff0) %!s(*string=0xc000031fe0) %!s(*string=0xc0000ac000) %!s(*string=0xc0000ac040) map[k8s-azure-created-by:%!s(*string=0xc000031d20) kubernetes.io-created-for-pv-name:%!s(*string=0xc000031da0) kubernetes.io-created-for-pvc-name:%!s(*string=0xc000031dd0) kubernetes.io-created-for-pvc-namespace:%!s(*string=0xc000031e40)]} I0129 03:56:44.247919 1 azure_controller_common.go:261] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-71e6a6cd-f320-41bd-9667-f0534b913f79 lun 0 to node k8s-agentpool-14513215-vmss000000, diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-71e6a6cd-f320-41bd-9667-f0534b913f79:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-71e6a6cd-f320-41bd-9667-f0534b913f79 false 0})] I0129 03:56:44.248044 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-l6t2jkxe): vm(k8s-agentpool-14513215-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-71e6a6cd-f320-41bd-9667-f0534b913f79:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-71e6a6cd-f320-41bd-9667-f0534b913f79 false 0})]) I0129 03:56:44.497684 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-l6t2jkxe): vm(k8s-agentpool-14513215-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-71e6a6cd-f320-41bd-9667-f0534b913f79:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-71e6a6cd-f320-41bd-9667-f0534b913f79 false 0})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I0129 03:56:54.573354 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-l6t2jkxe, k8s-agentpool-14513215-vmss, k8s-agentpool-14513215-vmss000000) successfully I0129 03:56:54.573390 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool-14513215-vmss, kubetest-l6t2jkxe, k8s-agentpool-14513215-vmss000000) for cacheKey(kubetest-l6t2jkxe/k8s-agentpool-14513215-vmss) updated successfully I0129 03:56:54.573409 1 controllerserver.go:413] Attach operation successful: volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-71e6a6cd-f320-41bd-9667-f0534b913f79 attached to node k8s-agentpool-14513215-vmss000000. I0129 03:56:54.573422 1 controllerserver.go:433] attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-71e6a6cd-f320-41bd-9667-f0534b913f79 to node k8s-agentpool-14513215-vmss000000 successfully I0129 03:56:54.573466 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=11.325875434 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-l6t2jkxe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-71e6a6cd-f320-41bd-9667-f0534b913f79" node="k8s-agentpool-14513215-vmss000000" result_code="succeeded" I0129 03:56:54.573505 1 utils.go:84] GRPC response: {"publish_context":{"LUN":"0"}} ... skipping 12 lines ... I0129 03:57:07.541658 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1193 I0129 03:57:07.541933 1 controllerserver.go:383] GetDiskLun returned: cannot find Lun for disk pvc-b9c3702f-e71e-45e5-b2e7-69d73d6d723c. Initiating attaching volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-b9c3702f-e71e-45e5-b2e7-69d73d6d723c to node k8s-agentpool-14513215-vmss000001 (vmState Succeeded). I0129 03:57:07.541961 1 controllerserver.go:408] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-b9c3702f-e71e-45e5-b2e7-69d73d6d723c to node k8s-agentpool-14513215-vmss000001 I0129 03:57:07.541987 1 azure_controller_common.go:247] waiting for more requests on node k8s-agentpool-14513215-vmss000001, current disk attach: &{{%!s(*http.Response=&{200 OK 200 HTTP/1.1 1 1 map[Cache-Control:[no-cache] Content-Type:[application/json; charset=utf-8] Date:[Sun, 29 Jan 2023 03:57:07 GMT] Expires:[-1] Pragma:[no-cache] Server:[Microsoft-HTTPAPI/2.0 Microsoft-HTTPAPI/2.0] Strict-Transport-Security:[max-age=31536000; includeSubDomains] Vary:[Accept-Encoding] X-Content-Type-Options:[nosniff] X-Ms-Correlation-Request-Id:[4b83ef18-e982-4b7b-9f95-72ce541fe8aa] X-Ms-Ratelimit-Remaining-Resource:[Microsoft.Compute/LowCostGet3Min;14956,Microsoft.Compute/LowCostGet30Min;119857] X-Ms-Ratelimit-Remaining-Subscription-Reads:[11838] X-Ms-Request-Id:[631029aa-fdb4-426b-96a1-993187418a19] X-Ms-Routing-Request-Id:[WESTUS2:20230129T035707Z:4b83ef18-e982-4b7b-9f95-72ce541fe8aa] X-Ms-Served-By:[43c1a32f-0327-4d8b-a820-a28f01b4195d_133148057271908261]] {0xc000a8aa50} -1 [chunked] false true map[] 0xc000a8ca00 0xc0005fb550})} %!s(*string=<nil>) %!s(*[]string=<nil>) %!s(*compute.DiskSku=&{StandardSSD_ZRS 0xc000035580}) %!s(*[]string=<nil>) %!s(*compute.ExtendedLocation=<nil>) %!s(*compute.DiskProperties=&{0xc0007ca288 <nil> <nil> 0xc000412ea0 0xc000719c78 0xc000719cf0 0xc0000356f0 <nil> 0xc0000356b0 0xc000719c80 0xc000719c90 <nil> <nil> Unattached 0xc0007ca270 <nil> <nil> AllowAll <nil> <nil> <nil> <nil> <nil> <nil> <nil> Enabled }) %!s(*string=0xc000035770) %!s(*string=0xc000035730) %!s(*string=0xc0000357b0) %!s(*string=0xc0000357f0) map[k8s-azure-created-by:%!s(*string=0xc000035830) kubernetes.io-created-for-pv-name:%!s(*string=0xc0000358b0) kubernetes.io-created-for-pvc-name:%!s(*string=0xc000035930) kubernetes.io-created-for-pvc-namespace:%!s(*string=0xc000035960)]} I0129 03:57:08.542232 1 azure_controller_common.go:261] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-b9c3702f-e71e-45e5-b2e7-69d73d6d723c lun 0 to node k8s-agentpool-14513215-vmss000001, diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-b9c3702f-e71e-45e5-b2e7-69d73d6d723c:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-b9c3702f-e71e-45e5-b2e7-69d73d6d723c false 0})] I0129 03:57:08.542300 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-l6t2jkxe): vm(k8s-agentpool-14513215-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-b9c3702f-e71e-45e5-b2e7-69d73d6d723c:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-b9c3702f-e71e-45e5-b2e7-69d73d6d723c false 0})]) I0129 03:57:08.681310 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-l6t2jkxe): vm(k8s-agentpool-14513215-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-b9c3702f-e71e-45e5-b2e7-69d73d6d723c:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-b9c3702f-e71e-45e5-b2e7-69d73d6d723c false 0})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I0129 03:57:18.824440 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-l6t2jkxe, k8s-agentpool-14513215-vmss, k8s-agentpool-14513215-vmss000001) successfully I0129 03:57:18.824474 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool-14513215-vmss, kubetest-l6t2jkxe, k8s-agentpool-14513215-vmss000001) for cacheKey(kubetest-l6t2jkxe/k8s-agentpool-14513215-vmss) updated successfully I0129 03:57:18.824495 1 controllerserver.go:413] Attach operation successful: volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-b9c3702f-e71e-45e5-b2e7-69d73d6d723c attached to node k8s-agentpool-14513215-vmss000001. I0129 03:57:18.824508 1 controllerserver.go:433] attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-b9c3702f-e71e-45e5-b2e7-69d73d6d723c to node k8s-agentpool-14513215-vmss000001 successfully I0129 03:57:18.824548 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=11.282615126 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-l6t2jkxe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-b9c3702f-e71e-45e5-b2e7-69d73d6d723c" node="k8s-agentpool-14513215-vmss000001" result_code="succeeded" I0129 03:57:18.824571 1 utils.go:84] GRPC response: {"publish_context":{"LUN":"0"}} ... skipping 12 lines ... I0129 03:57:31.931473 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1193 I0129 03:57:31.931884 1 controllerserver.go:383] GetDiskLun returned: cannot find Lun for disk pvc-7c684e10-1d12-40c3-af55-4412ca6bed42. Initiating attaching volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-7c684e10-1d12-40c3-af55-4412ca6bed42 to node k8s-agentpool-14513215-vmss000000 (vmState Succeeded). I0129 03:57:31.931923 1 controllerserver.go:408] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-7c684e10-1d12-40c3-af55-4412ca6bed42 to node k8s-agentpool-14513215-vmss000000 I0129 03:57:31.931951 1 azure_controller_common.go:247] waiting for more requests on node k8s-agentpool-14513215-vmss000000, current disk attach: &{{%!s(*http.Response=&{200 OK 200 HTTP/1.1 1 1 map[Cache-Control:[no-cache] Content-Type:[application/json; charset=utf-8] Date:[Sun, 29 Jan 2023 03:57:31 GMT] Expires:[-1] Pragma:[no-cache] Server:[Microsoft-HTTPAPI/2.0 Microsoft-HTTPAPI/2.0] Strict-Transport-Security:[max-age=31536000; includeSubDomains] Vary:[Accept-Encoding] X-Content-Type-Options:[nosniff] X-Ms-Correlation-Request-Id:[fb284024-afcd-428f-a8ba-ba177e8b79d0] X-Ms-Ratelimit-Remaining-Resource:[Microsoft.Compute/LowCostGet3Min;14960,Microsoft.Compute/LowCostGet30Min;119848] X-Ms-Ratelimit-Remaining-Subscription-Reads:[11830] X-Ms-Request-Id:[13bd7c13-5c69-47b6-b508-c77fde7e0b75] X-Ms-Routing-Request-Id:[WESTUS2:20230129T035731Z:fb284024-afcd-428f-a8ba-ba177e8b79d0] X-Ms-Served-By:[43c1a32f-0327-4d8b-a820-a28f01b4195d_133148057271908261]] {0xc0006baed0} -1 [chunked] false true map[] 0xc0005f6700 0xc0005fb550})} %!s(*string=<nil>) %!s(*[]string=<nil>) %!s(*compute.DiskSku=&{StandardSSD_ZRS 0xc0000efb70}) %!s(*[]string=<nil>) %!s(*compute.ExtendedLocation=<nil>) %!s(*compute.DiskProperties=&{0xc0007cbdb8 <nil> <nil> 0xc0008be360 0xc000a95798 0xc000a95820 0xc0000efcb0 <nil> 0xc0000efc70 0xc000a957a0 0xc000a957b0 <nil> <nil> Unattached 0xc0007cbd88 <nil> <nil> AllowAll <nil> <nil> <nil> <nil> <nil> <nil> <nil> Enabled }) %!s(*string=0xc0000ef8f0) %!s(*string=0xc0000ef8d0) %!s(*string=0xc0000ef900) %!s(*string=0xc0000ef910) map[k8s-azure-created-by:%!s(*string=0xc0000ef940) kubernetes.io-created-for-pv-name:%!s(*string=0xc0000ef9c0) kubernetes.io-created-for-pvc-name:%!s(*string=0xc0000efa50) kubernetes.io-created-for-pvc-namespace:%!s(*string=0xc0000efad0)]} I0129 03:57:32.932281 1 azure_controller_common.go:261] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-7c684e10-1d12-40c3-af55-4412ca6bed42 lun 1 to node k8s-agentpool-14513215-vmss000000, diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-7c684e10-1d12-40c3-af55-4412ca6bed42:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-7c684e10-1d12-40c3-af55-4412ca6bed42 false 1})] I0129 03:57:32.932384 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-l6t2jkxe): vm(k8s-agentpool-14513215-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-7c684e10-1d12-40c3-af55-4412ca6bed42:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-7c684e10-1d12-40c3-af55-4412ca6bed42 false 1})]) I0129 03:57:33.058087 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-l6t2jkxe): vm(k8s-agentpool-14513215-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-7c684e10-1d12-40c3-af55-4412ca6bed42:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-7c684e10-1d12-40c3-af55-4412ca6bed42 false 1})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I0129 03:57:43.146967 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-l6t2jkxe, k8s-agentpool-14513215-vmss, k8s-agentpool-14513215-vmss000000) successfully I0129 03:57:43.147012 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool-14513215-vmss, kubetest-l6t2jkxe, k8s-agentpool-14513215-vmss000000) for cacheKey(kubetest-l6t2jkxe/k8s-agentpool-14513215-vmss) updated successfully I0129 03:57:43.147032 1 controllerserver.go:413] Attach operation successful: volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-7c684e10-1d12-40c3-af55-4412ca6bed42 attached to node k8s-agentpool-14513215-vmss000000. I0129 03:57:43.147109 1 controllerserver.go:433] attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-7c684e10-1d12-40c3-af55-4412ca6bed42 to node k8s-agentpool-14513215-vmss000000 successfully I0129 03:57:43.147154 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=11.215346623 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-l6t2jkxe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-7c684e10-1d12-40c3-af55-4412ca6bed42" node="k8s-agentpool-14513215-vmss000000" result_code="succeeded" I0129 03:57:43.147170 1 utils.go:84] GRPC response: {"publish_context":{"LUN":"1"}} ... skipping 48 lines ... I0129 03:59:53.015426 1 azure_controller_common.go:410] Trying to detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-b9c3702f-e71e-45e5-b2e7-69d73d6d723c from node k8s-agentpool-14513215-vmss000001, diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-b9c3702f-e71e-45e5-b2e7-69d73d6d723c:pvc-b9c3702f-e71e-45e5-b2e7-69d73d6d723c] E0129 03:59:53.015521 1 azure_controller_vmss.go:202] detach azure disk on node(k8s-agentpool-14513215-vmss000001): disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-b9c3702f-e71e-45e5-b2e7-69d73d6d723c:pvc-b9c3702f-e71e-45e5-b2e7-69d73d6d723c]) not found I0129 03:59:53.015542 1 azure_controller_vmss.go:239] azureDisk - update(kubetest-l6t2jkxe): vm(k8s-agentpool-14513215-vmss000001) - detach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-b9c3702f-e71e-45e5-b2e7-69d73d6d723c:pvc-b9c3702f-e71e-45e5-b2e7-69d73d6d723c]) I0129 03:59:53.057211 1 utils.go:77] GRPC call: /csi.v1.Controller/DeleteVolume I0129 03:59:53.057230 1 utils.go:78] GRPC request: {"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-b9c3702f-e71e-45e5-b2e7-69d73d6d723c"} I0129 03:59:53.057355 1 controllerserver.go:317] deleting azure disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-b9c3702f-e71e-45e5-b2e7-69d73d6d723c) I0129 03:59:53.057369 1 controllerserver.go:319] delete azure disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-b9c3702f-e71e-45e5-b2e7-69d73d6d723c) returned with failed to delete disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-b9c3702f-e71e-45e5-b2e7-69d73d6d723c) since it's in attaching or detaching state I0129 03:59:53.057435 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=2.74e-05 request="azuredisk_csi_driver_controller_delete_volume" resource_group="kubetest-l6t2jkxe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-b9c3702f-e71e-45e5-b2e7-69d73d6d723c" result_code="failed_csi_driver_controller_delete_volume" E0129 03:59:53.057448 1 utils.go:82] GRPC error: failed to delete disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-b9c3702f-e71e-45e5-b2e7-69d73d6d723c) since it's in attaching or detaching state I0129 03:59:58.177227 1 azure_controller_vmss.go:252] azureDisk - update(kubetest-l6t2jkxe): vm(k8s-agentpool-14513215-vmss000001) - detach disk(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-b9c3702f-e71e-45e5-b2e7-69d73d6d723c:pvc-b9c3702f-e71e-45e5-b2e7-69d73d6d723c]) returned with <nil> I0129 03:59:58.177359 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-l6t2jkxe, k8s-agentpool-14513215-vmss, k8s-agentpool-14513215-vmss000001) successfully I0129 03:59:58.177379 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool-14513215-vmss, kubetest-l6t2jkxe, k8s-agentpool-14513215-vmss000001) for cacheKey(kubetest-l6t2jkxe/k8s-agentpool-14513215-vmss) updated successfully I0129 03:59:58.177390 1 azure_controller_common.go:434] azureDisk - detach disk(pvc-b9c3702f-e71e-45e5-b2e7-69d73d6d723c, /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-b9c3702f-e71e-45e5-b2e7-69d73d6d723c) succeeded I0129 03:59:58.177404 1 controllerserver.go:480] detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-b9c3702f-e71e-45e5-b2e7-69d73d6d723c from node k8s-agentpool-14513215-vmss000001 successfully I0129 03:59:58.177464 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=6.162349602 request="azuredisk_csi_driver_controller_unpublish_volume" resource_group="kubetest-l6t2jkxe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-b9c3702f-e71e-45e5-b2e7-69d73d6d723c" node="k8s-agentpool-14513215-vmss000001" result_code="succeeded" ... skipping 44 lines ... I0129 04:01:29.255612 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 12 I0129 04:01:29.255766 1 controllerserver.go:383] GetDiskLun returned: cannot find Lun for disk pvc-1ed40fbe-c9a2-43ae-a1a3-c739839b9944. Initiating attaching volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-1ed40fbe-c9a2-43ae-a1a3-c739839b9944 to node k8s-agentpool-14513215-vmss000000 (vmState Succeeded). I0129 04:01:29.255832 1 controllerserver.go:408] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-1ed40fbe-c9a2-43ae-a1a3-c739839b9944 to node k8s-agentpool-14513215-vmss000000 I0129 04:01:29.255895 1 azure_controller_common.go:247] waiting for more requests on node k8s-agentpool-14513215-vmss000000, current disk attach: &{{%!s(*http.Response=&{200 OK 200 HTTP/1.1 1 1 map[Cache-Control:[no-cache] Content-Type:[application/json; charset=utf-8] Date:[Sun, 29 Jan 2023 04:01:28 GMT] Expires:[-1] Pragma:[no-cache] Server:[Microsoft-HTTPAPI/2.0 Microsoft-HTTPAPI/2.0] Strict-Transport-Security:[max-age=31536000; includeSubDomains] Vary:[Accept-Encoding] X-Content-Type-Options:[nosniff] X-Ms-Correlation-Request-Id:[a055544f-f707-4162-a798-780dc3e89e01] X-Ms-Ratelimit-Remaining-Resource:[Microsoft.Compute/LowCostGet3Min;14957,Microsoft.Compute/LowCostGet30Min;119792] X-Ms-Ratelimit-Remaining-Subscription-Reads:[11793] X-Ms-Request-Id:[31666275-d72d-4f9a-8fb4-ca03f97c735b] X-Ms-Routing-Request-Id:[WESTUS2:20230129T040129Z:a055544f-f707-4162-a798-780dc3e89e01] X-Ms-Served-By:[43c1a32f-0327-4d8b-a820-a28f01b4195d_133148057271908261]] {0xc000808240} -1 [chunked] false true map[] 0xc000a8dc00 0xc0005fb550})} %!s(*string=<nil>) %!s(*[]string=<nil>) %!s(*compute.DiskSku=&{StandardSSD_LRS 0xc0001968c0}) %!s(*[]string=&[1]) %!s(*compute.ExtendedLocation=<nil>) %!s(*compute.DiskProperties=&{0xc0002006f0 <nil> <nil> 0xc0008befc0 0xc000741470 0xc0007414e0 0xc000196990 <nil> 0xc000196980 0xc000741478 0xc000741488 <nil> <nil> Unattached 0xc0002006d8 <nil> <nil> AllowAll <nil> <nil> <nil> <nil> <nil> <nil> <nil> Enabled }) %!s(*string=0xc000196410) %!s(*string=0xc0001969c0) %!s(*string=0xc000196450) %!s(*string=0xc000196490) map[k8s-azure-created-by:%!s(*string=0xc0001964b0) kubernetes.io-created-for-pv-name:%!s(*string=0xc0001964f0) kubernetes.io-created-for-pvc-name:%!s(*string=0xc0001966a0) kubernetes.io-created-for-pvc-namespace:%!s(*string=0xc000196820)]} I0129 04:01:30.256401 1 azure_controller_common.go:261] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-1ed40fbe-c9a2-43ae-a1a3-c739839b9944 lun 0 to node k8s-agentpool-14513215-vmss000000, diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-1ed40fbe-c9a2-43ae-a1a3-c739839b9944:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-1ed40fbe-c9a2-43ae-a1a3-c739839b9944 false 0})] I0129 04:01:30.256524 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-l6t2jkxe): vm(k8s-agentpool-14513215-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-1ed40fbe-c9a2-43ae-a1a3-c739839b9944:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-1ed40fbe-c9a2-43ae-a1a3-c739839b9944 false 0})]) I0129 04:01:30.408636 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-l6t2jkxe): vm(k8s-agentpool-14513215-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-1ed40fbe-c9a2-43ae-a1a3-c739839b9944:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-1ed40fbe-c9a2-43ae-a1a3-c739839b9944 false 0})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I0129 04:01:40.502695 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-l6t2jkxe, k8s-agentpool-14513215-vmss, k8s-agentpool-14513215-vmss000000) successfully I0129 04:01:40.502752 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool-14513215-vmss, kubetest-l6t2jkxe, k8s-agentpool-14513215-vmss000000) for cacheKey(kubetest-l6t2jkxe/k8s-agentpool-14513215-vmss) updated successfully I0129 04:01:40.502772 1 controllerserver.go:413] Attach operation successful: volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-1ed40fbe-c9a2-43ae-a1a3-c739839b9944 attached to node k8s-agentpool-14513215-vmss000000. I0129 04:01:40.502785 1 controllerserver.go:433] attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-1ed40fbe-c9a2-43ae-a1a3-c739839b9944 to node k8s-agentpool-14513215-vmss000000 successfully I0129 04:01:40.502826 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=11.288684362 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-l6t2jkxe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-1ed40fbe-c9a2-43ae-a1a3-c739839b9944" node="k8s-agentpool-14513215-vmss000000" result_code="succeeded" I0129 04:01:40.502846 1 utils.go:84] GRPC response: {"publish_context":{"LUN":"0"}} ... skipping 67 lines ... I0129 04:04:12.489603 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1217 I0129 04:04:12.489850 1 controllerserver.go:383] GetDiskLun returned: cannot find Lun for disk pvc-4600ae24-362b-48dd-9148-9dd381910a40. Initiating attaching volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-4600ae24-362b-48dd-9148-9dd381910a40 to node k8s-agentpool-14513215-vmss000000 (vmState Succeeded). I0129 04:04:12.489881 1 controllerserver.go:408] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-4600ae24-362b-48dd-9148-9dd381910a40 to node k8s-agentpool-14513215-vmss000000 I0129 04:04:12.489895 1 azure_controller_common.go:247] waiting for more requests on node k8s-agentpool-14513215-vmss000000, current disk attach: &{{%!s(*http.Response=&{200 OK 200 HTTP/1.1 1 1 map[Cache-Control:[no-cache] Content-Type:[application/json; charset=utf-8] Date:[Sun, 29 Jan 2023 04:04:11 GMT] Expires:[-1] Pragma:[no-cache] Server:[Microsoft-HTTPAPI/2.0 Microsoft-HTTPAPI/2.0] Strict-Transport-Security:[max-age=31536000; includeSubDomains] Vary:[Accept-Encoding] X-Content-Type-Options:[nosniff] X-Ms-Correlation-Request-Id:[633f7f59-451c-4201-89c2-0796438f86e3] X-Ms-Ratelimit-Remaining-Resource:[Microsoft.Compute/LowCostGet3Min;14968,Microsoft.Compute/LowCostGet30Min;119760] X-Ms-Ratelimit-Remaining-Subscription-Reads:[11767] X-Ms-Request-Id:[1ef4b7c8-6003-444c-92ad-6d6eeb7faa42] X-Ms-Routing-Request-Id:[WESTUS2:20230129T040412Z:633f7f59-451c-4201-89c2-0796438f86e3] X-Ms-Served-By:[43c1a32f-0327-4d8b-a820-a28f01b4195d_133148057271908261]] {0xc000aeb050} -1 [chunked] false true map[] 0xc0005f7200 0xc0005fb550})} %!s(*string=<nil>) %!s(*[]string=<nil>) %!s(*compute.DiskSku=&{Standard_LRS 0xc0009d2900}) %!s(*[]string=&[1]) %!s(*compute.ExtendedLocation=<nil>) %!s(*compute.DiskProperties=&{0xc000200b88 <nil> <nil> 0xc0008be660 0xc000740f18 0xc000740f80 0xc0009d2960 <nil> 0xc0009d2950 0xc000740f20 0xc000740f30 <nil> <nil> Unattached 0xc000200b70 <nil> <nil> AllowAll <nil> <nil> <nil> <nil> <nil> <nil> <nil> Enabled }) %!s(*string=0xc0009d2a00) %!s(*string=0xc0009d29f0) %!s(*string=0xc0009d2a10) %!s(*string=0xc0009d2860) map[k8s-azure-created-by:%!s(*string=0xc0009d2870) kubernetes.io-created-for-pv-name:%!s(*string=0xc0009d2890) kubernetes.io-created-for-pvc-name:%!s(*string=0xc0009d28b0) kubernetes.io-created-for-pvc-namespace:%!s(*string=0xc0009d28d0)]} I0129 04:04:13.490132 1 azure_controller_common.go:261] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-4600ae24-362b-48dd-9148-9dd381910a40 lun 0 to node k8s-agentpool-14513215-vmss000000, diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-4600ae24-362b-48dd-9148-9dd381910a40:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-4600ae24-362b-48dd-9148-9dd381910a40 false 0})] I0129 04:04:13.490216 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-l6t2jkxe): vm(k8s-agentpool-14513215-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-4600ae24-362b-48dd-9148-9dd381910a40:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-4600ae24-362b-48dd-9148-9dd381910a40 false 0})]) I0129 04:04:13.612422 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-l6t2jkxe): vm(k8s-agentpool-14513215-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-4600ae24-362b-48dd-9148-9dd381910a40:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-4600ae24-362b-48dd-9148-9dd381910a40 false 0})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I0129 04:04:23.703816 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-l6t2jkxe, k8s-agentpool-14513215-vmss, k8s-agentpool-14513215-vmss000000) successfully I0129 04:04:23.703916 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool-14513215-vmss, kubetest-l6t2jkxe, k8s-agentpool-14513215-vmss000000) for cacheKey(kubetest-l6t2jkxe/k8s-agentpool-14513215-vmss) updated successfully I0129 04:04:23.703947 1 controllerserver.go:413] Attach operation successful: volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-4600ae24-362b-48dd-9148-9dd381910a40 attached to node k8s-agentpool-14513215-vmss000000. I0129 04:04:23.703963 1 controllerserver.go:433] attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-4600ae24-362b-48dd-9148-9dd381910a40 to node k8s-agentpool-14513215-vmss000000 successfully I0129 04:04:23.704039 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=11.214148848 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-l6t2jkxe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-4600ae24-362b-48dd-9148-9dd381910a40" node="k8s-agentpool-14513215-vmss000000" result_code="succeeded" I0129 04:04:23.704067 1 utils.go:84] GRPC response: {"publish_context":{"LUN":"0"}} ... skipping 27 lines ... I0129 04:04:52.355614 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=16.307119613 request="azuredisk_csi_driver_controller_unpublish_volume" resource_group="kubetest-l6t2jkxe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-4600ae24-362b-48dd-9148-9dd381910a40" node="k8s-agentpool-14513215-vmss000000" result_code="succeeded" I0129 04:04:52.355630 1 utils.go:84] GRPC response: {} I0129 04:04:52.355740 1 azure_controller_common.go:247] waiting for more requests on node k8s-agentpool-14513215-vmss000000, current disk attach: &{{%!s(*http.Response=&{200 OK 200 HTTP/1.1 1 1 map[Cache-Control:[no-cache] Content-Type:[application/json; charset=utf-8] Date:[Sun, 29 Jan 2023 04:04:41 GMT] Expires:[-1] Pragma:[no-cache] Server:[Microsoft-HTTPAPI/2.0 Microsoft-HTTPAPI/2.0] Strict-Transport-Security:[max-age=31536000; includeSubDomains] Vary:[Accept-Encoding] X-Content-Type-Options:[nosniff] X-Ms-Correlation-Request-Id:[d7d641b1-bf5b-4b96-9b26-12935105b225] X-Ms-Ratelimit-Remaining-Resource:[Microsoft.Compute/LowCostGet3Min;14965,Microsoft.Compute/LowCostGet30Min;119749] X-Ms-Ratelimit-Remaining-Subscription-Reads:[11757] X-Ms-Request-Id:[13d64d09-ff24-4162-b1bc-157499ffbb0f] X-Ms-Routing-Request-Id:[WESTUS2:20230129T040441Z:d7d641b1-bf5b-4b96-9b26-12935105b225] X-Ms-Served-By:[43c1a32f-0327-4d8b-a820-a28f01b4195d_133148057271908261]] {0xc00001aed0} -1 [chunked] false true map[] 0xc000ac2600 0xc0005fb550})} %!s(*string=<nil>) %!s(*[]string=<nil>) %!s(*compute.DiskSku=&{Standard_LRS 0xc0001426d0}) %!s(*[]string=&[1]) %!s(*compute.ExtendedLocation=<nil>) %!s(*compute.DiskProperties=&{0xc0000ea150 <nil> <nil> 0xc0008bfbc0 0xc000741e2c 0xc000741ee0 0xc0001428d0 <nil> 0xc000142890 0xc000741e48 0xc000741e58 <nil> <nil> Unattached 0xc0000ea138 <nil> <nil> AllowAll <nil> <nil> <nil> <nil> <nil> <nil> <nil> Enabled }) %!s(*string=0xc000142950) %!s(*string=0xc000142910) %!s(*string=0xc000142990) %!s(*string=0xc0001429d0) map[k8s-azure-created-by:%!s(*string=0xc000142490) kubernetes.io-created-for-pv-name:%!s(*string=0xc000142510) kubernetes.io-created-for-pvc-name:%!s(*string=0xc000142590) kubernetes.io-created-for-pvc-namespace:%!s(*string=0xc000142610)]} I0129 04:04:53.421792 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 24989 I0129 04:04:53.424632 1 azure_controller_common.go:261] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-dc70cac4-dc80-4ab5-9320-a356c899b8de lun 0 to node k8s-agentpool-14513215-vmss000000, diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-dc70cac4-dc80-4ab5-9320-a356c899b8de:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-dc70cac4-dc80-4ab5-9320-a356c899b8de false 0})] I0129 04:04:53.424691 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-l6t2jkxe): vm(k8s-agentpool-14513215-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-dc70cac4-dc80-4ab5-9320-a356c899b8de:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-dc70cac4-dc80-4ab5-9320-a356c899b8de false 0})]) I0129 04:04:53.558048 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-l6t2jkxe): vm(k8s-agentpool-14513215-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-dc70cac4-dc80-4ab5-9320-a356c899b8de:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-dc70cac4-dc80-4ab5-9320-a356c899b8de false 0})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I0129 04:05:03.638488 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-l6t2jkxe, k8s-agentpool-14513215-vmss, k8s-agentpool-14513215-vmss000000) successfully I0129 04:05:03.638643 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool-14513215-vmss, kubetest-l6t2jkxe, k8s-agentpool-14513215-vmss000000) for cacheKey(kubetest-l6t2jkxe/k8s-agentpool-14513215-vmss) updated successfully I0129 04:05:03.638723 1 controllerserver.go:413] Attach operation successful: volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-dc70cac4-dc80-4ab5-9320-a356c899b8de attached to node k8s-agentpool-14513215-vmss000000. I0129 04:05:03.638781 1 controllerserver.go:433] attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-dc70cac4-dc80-4ab5-9320-a356c899b8de to node k8s-agentpool-14513215-vmss000000 successfully I0129 04:05:03.638880 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=21.816806737 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-l6t2jkxe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-dc70cac4-dc80-4ab5-9320-a356c899b8de" node="k8s-agentpool-14513215-vmss000000" result_code="succeeded" I0129 04:05:03.638940 1 utils.go:84] GRPC response: {"publish_context":{"LUN":"0"}} ... skipping 42 lines ... I0129 04:06:12.687880 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1192 I0129 04:06:12.688262 1 controllerserver.go:383] GetDiskLun returned: cannot find Lun for disk pvc-f849b8e4-e30b-4e95-b3db-e1a1a6ec0955. Initiating attaching volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-f849b8e4-e30b-4e95-b3db-e1a1a6ec0955 to node k8s-agentpool-14513215-vmss000000 (vmState Succeeded). I0129 04:06:12.688325 1 controllerserver.go:408] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-f849b8e4-e30b-4e95-b3db-e1a1a6ec0955 to node k8s-agentpool-14513215-vmss000000 I0129 04:06:12.688351 1 azure_controller_common.go:247] waiting for more requests on node k8s-agentpool-14513215-vmss000000, current disk attach: &{{%!s(*http.Response=&{200 OK 200 HTTP/1.1 1 1 map[Cache-Control:[no-cache] Content-Type:[application/json; charset=utf-8] Date:[Sun, 29 Jan 2023 04:06:12 GMT] Expires:[-1] Pragma:[no-cache] Server:[Microsoft-HTTPAPI/2.0 Microsoft-HTTPAPI/2.0] Strict-Transport-Security:[max-age=31536000; includeSubDomains] Vary:[Accept-Encoding] X-Content-Type-Options:[nosniff] X-Ms-Correlation-Request-Id:[ef108df7-aa1f-4103-91f7-813d9a5f0873] X-Ms-Ratelimit-Remaining-Resource:[Microsoft.Compute/LowCostGet3Min;14946,Microsoft.Compute/LowCostGet30Min;119727] X-Ms-Ratelimit-Remaining-Subscription-Reads:[11732] X-Ms-Request-Id:[867023eb-731a-4ec6-93de-00770ee43980] X-Ms-Routing-Request-Id:[WESTUS2:20230129T040612Z:ef108df7-aa1f-4103-91f7-813d9a5f0873] X-Ms-Served-By:[43c1a32f-0327-4d8b-a820-a28f01b4195d_133148057271908261]] {0xc000aeadb0} -1 [chunked] false true map[] 0xc00095e700 0xc0005fb550})} %!s(*string=<nil>) %!s(*[]string=<nil>) %!s(*compute.DiskSku=&{StandardSSD_ZRS 0xc0000ad1e0}) %!s(*[]string=<nil>) %!s(*compute.ExtendedLocation=<nil>) %!s(*compute.DiskProperties=&{0xc000a987b0 <nil> <nil> 0xc0006e82a0 0xc0006e4738 0xc0006e47a0 0xc0000ad320 <nil> 0xc0000ad2e0 0xc0006e4740 0xc0006e4750 <nil> <nil> Unattached 0xc000a98798 <nil> <nil> DenyAll <nil> <nil> <nil> <nil> <nil> <nil> <nil> Enabled }) %!s(*string=0xc0000ad3a0) %!s(*string=0xc0000ad360) %!s(*string=0xc0000ad3e0) %!s(*string=0xc0000ad000) map[k8s-azure-created-by:%!s(*string=0xc0000ad050) kubernetes.io-created-for-pv-name:%!s(*string=0xc0000ad0d0) kubernetes.io-created-for-pvc-name:%!s(*string=0xc0000ad100) kubernetes.io-created-for-pvc-namespace:%!s(*string=0xc0000ad140)]} I0129 04:06:13.688638 1 azure_controller_common.go:261] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-f849b8e4-e30b-4e95-b3db-e1a1a6ec0955 lun 0 to node k8s-agentpool-14513215-vmss000000, diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-f849b8e4-e30b-4e95-b3db-e1a1a6ec0955:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-f849b8e4-e30b-4e95-b3db-e1a1a6ec0955 false 0})] I0129 04:06:13.688726 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-l6t2jkxe): vm(k8s-agentpool-14513215-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-f849b8e4-e30b-4e95-b3db-e1a1a6ec0955:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-f849b8e4-e30b-4e95-b3db-e1a1a6ec0955 false 0})]) I0129 04:06:13.847531 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-l6t2jkxe): vm(k8s-agentpool-14513215-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-f849b8e4-e30b-4e95-b3db-e1a1a6ec0955:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-f849b8e4-e30b-4e95-b3db-e1a1a6ec0955 false 0})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I0129 04:06:23.934198 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-l6t2jkxe, k8s-agentpool-14513215-vmss, k8s-agentpool-14513215-vmss000000) successfully I0129 04:06:23.934241 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool-14513215-vmss, kubetest-l6t2jkxe, k8s-agentpool-14513215-vmss000000) for cacheKey(kubetest-l6t2jkxe/k8s-agentpool-14513215-vmss) updated successfully I0129 04:06:23.934259 1 controllerserver.go:413] Attach operation successful: volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-f849b8e4-e30b-4e95-b3db-e1a1a6ec0955 attached to node k8s-agentpool-14513215-vmss000000. I0129 04:06:23.934268 1 controllerserver.go:433] attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-f849b8e4-e30b-4e95-b3db-e1a1a6ec0955 to node k8s-agentpool-14513215-vmss000000 successfully I0129 04:06:23.934308 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=11.246056454 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-l6t2jkxe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-f849b8e4-e30b-4e95-b3db-e1a1a6ec0955" node="k8s-agentpool-14513215-vmss000000" result_code="succeeded" I0129 04:06:23.934327 1 utils.go:84] GRPC response: {"publish_context":{"LUN":"0"}} ... skipping 30 lines ... I0129 04:06:57.399811 1 azure_controller_common.go:247] waiting for more requests on node k8s-agentpool-14513215-vmss000000, current disk attach: &{{%!s(*http.Response=&{200 OK 200 HTTP/1.1 1 1 map[Cache-Control:[no-cache] Content-Type:[application/json; charset=utf-8] Date:[Sun, 29 Jan 2023 04:06:41 GMT] Expires:[-1] Pragma:[no-cache] Server:[Microsoft-HTTPAPI/2.0 Microsoft-HTTPAPI/2.0] Strict-Transport-Security:[max-age=31536000; includeSubDomains] Vary:[Accept-Encoding] X-Content-Type-Options:[nosniff] X-Ms-Correlation-Request-Id:[974c77a4-8d45-4951-aac1-26a77ddecfce] X-Ms-Ratelimit-Remaining-Resource:[Microsoft.Compute/LowCostGet3Min;14942,Microsoft.Compute/LowCostGet30Min;119715] X-Ms-Ratelimit-Remaining-Subscription-Reads:[11722] X-Ms-Request-Id:[faf1fc9c-8f26-4404-b39c-fbaf86d840d8] X-Ms-Routing-Request-Id:[WESTUS2:20230129T040642Z:974c77a4-8d45-4951-aac1-26a77ddecfce] X-Ms-Served-By:[43c1a32f-0327-4d8b-a820-a28f01b4195d_133148057271908261]] {0xc0005be3c0} -1 [chunked] false true map[] 0xc00095fc00 0xc0005fb550})} %!s(*string=<nil>) %!s(*[]string=<nil>) %!s(*compute.DiskSku=&{StandardSSD_ZRS 0xc0001b8c70}) %!s(*[]string=<nil>) %!s(*compute.ExtendedLocation=<nil>) %!s(*compute.DiskProperties=&{0xc0007cb380 <nil> <nil> 0xc0006e97a0 0xc00080767c 0xc000807770 0xc0001b8cf0 <nil> 0xc0001b8ce0 0xc0008076b8 0xc0008076e8 <nil> <nil> Unattached 0xc0007cb368 <nil> <nil> DenyAll <nil> <nil> <nil> <nil> <nil> <nil> <nil> Enabled }) %!s(*string=0xc0001b8d10) %!s(*string=0xc0001b8d00) %!s(*string=0xc0001b8bc0) %!s(*string=0xc0001b8bd0) map[k8s-azure-created-by:%!s(*string=0xc0001b8be0) kubernetes.io-created-for-pv-name:%!s(*string=0xc0001b8c00) kubernetes.io-created-for-pvc-name:%!s(*string=0xc0001b8c20) kubernetes.io-created-for-pvc-namespace:%!s(*string=0xc0001b8c40)]} I0129 04:06:57.420666 1 utils.go:77] GRPC call: /csi.v1.Controller/ControllerUnpublishVolume I0129 04:06:57.420688 1 utils.go:78] GRPC request: {"node_id":"k8s-agentpool-14513215-vmss000000","volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-f849b8e4-e30b-4e95-b3db-e1a1a6ec0955"} I0129 04:06:57.420819 1 controllerserver.go:471] Trying to detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-f849b8e4-e30b-4e95-b3db-e1a1a6ec0955 from node k8s-agentpool-14513215-vmss000000 I0129 04:06:58.400224 1 azure_controller_common.go:261] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-2422a02c-de2c-44b0-8fc7-4b0402a5f598 lun 0 to node k8s-agentpool-14513215-vmss000000, diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-2422a02c-de2c-44b0-8fc7-4b0402a5f598:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-2422a02c-de2c-44b0-8fc7-4b0402a5f598 false 0})] I0129 04:06:58.400311 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-l6t2jkxe): vm(k8s-agentpool-14513215-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-2422a02c-de2c-44b0-8fc7-4b0402a5f598:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-2422a02c-de2c-44b0-8fc7-4b0402a5f598 false 0})]) I0129 04:06:58.526582 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-l6t2jkxe): vm(k8s-agentpool-14513215-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-2422a02c-de2c-44b0-8fc7-4b0402a5f598:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-2422a02c-de2c-44b0-8fc7-4b0402a5f598 false 0})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I0129 04:07:08.639173 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-l6t2jkxe, k8s-agentpool-14513215-vmss, k8s-agentpool-14513215-vmss000000) successfully I0129 04:07:08.639215 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool-14513215-vmss, kubetest-l6t2jkxe, k8s-agentpool-14513215-vmss000000) for cacheKey(kubetest-l6t2jkxe/k8s-agentpool-14513215-vmss) updated successfully I0129 04:07:08.639253 1 controllerserver.go:413] Attach operation successful: volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-2422a02c-de2c-44b0-8fc7-4b0402a5f598 attached to node k8s-agentpool-14513215-vmss000000. I0129 04:07:08.639270 1 controllerserver.go:433] attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-2422a02c-de2c-44b0-8fc7-4b0402a5f598 to node k8s-agentpool-14513215-vmss000000 successfully I0129 04:07:08.639314 1 azure_controller_common.go:402] waiting for more requests on node k8s-agentpool-14513215-vmss000000, current disk detach: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-f849b8e4-e30b-4e95-b3db-e1a1a6ec0955 I0129 04:07:08.639314 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=26.513218181 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-l6t2jkxe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-2422a02c-de2c-44b0-8fc7-4b0402a5f598" node="k8s-agentpool-14513215-vmss000000" result_code="succeeded" ... skipping 83 lines ... I0129 04:08:13.233516 1 controllerserver.go:408] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-524a8b61-7ff4-48ef-b27e-7b37c344745e to node k8s-agentpool-14513215-vmss000000 I0129 04:08:13.295592 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1193 I0129 04:08:13.295824 1 controllerserver.go:383] GetDiskLun returned: cannot find Lun for disk pvc-f8d27713-c85b-4d66-807c-67592c3244f0. Initiating attaching volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-f8d27713-c85b-4d66-807c-67592c3244f0 to node k8s-agentpool-14513215-vmss000000 (vmState Succeeded). I0129 04:08:13.295872 1 controllerserver.go:408] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-f8d27713-c85b-4d66-807c-67592c3244f0 to node k8s-agentpool-14513215-vmss000000 I0129 04:08:14.233530 1 azure_controller_common.go:261] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-b493d5a2-76ba-4e4e-859c-0147e8131d15 lun 0 to node k8s-agentpool-14513215-vmss000000, diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-524a8b61-7ff4-48ef-b27e-7b37c344745e:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-524a8b61-7ff4-48ef-b27e-7b37c344745e false 1}) /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-b493d5a2-76ba-4e4e-859c-0147e8131d15:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-b493d5a2-76ba-4e4e-859c-0147e8131d15 false 0}) /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-f8d27713-c85b-4d66-807c-67592c3244f0:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-f8d27713-c85b-4d66-807c-67592c3244f0 false 2})] I0129 04:08:14.233610 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-l6t2jkxe): vm(k8s-agentpool-14513215-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-524a8b61-7ff4-48ef-b27e-7b37c344745e:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-524a8b61-7ff4-48ef-b27e-7b37c344745e false 1}) /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-b493d5a2-76ba-4e4e-859c-0147e8131d15:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-b493d5a2-76ba-4e4e-859c-0147e8131d15 false 0}) /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-f8d27713-c85b-4d66-807c-67592c3244f0:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-f8d27713-c85b-4d66-807c-67592c3244f0 false 2})]) I0129 04:08:14.411460 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-l6t2jkxe): vm(k8s-agentpool-14513215-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-524a8b61-7ff4-48ef-b27e-7b37c344745e:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-524a8b61-7ff4-48ef-b27e-7b37c344745e false 1}) /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-b493d5a2-76ba-4e4e-859c-0147e8131d15:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-b493d5a2-76ba-4e4e-859c-0147e8131d15 false 0}) /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-f8d27713-c85b-4d66-807c-67592c3244f0:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-f8d27713-c85b-4d66-807c-67592c3244f0 false 2})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I0129 04:08:24.520693 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-l6t2jkxe, k8s-agentpool-14513215-vmss, k8s-agentpool-14513215-vmss000000) successfully I0129 04:08:24.523476 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool-14513215-vmss, kubetest-l6t2jkxe, k8s-agentpool-14513215-vmss000000) for cacheKey(kubetest-l6t2jkxe/k8s-agentpool-14513215-vmss) updated successfully I0129 04:08:24.523668 1 controllerserver.go:413] Attach operation successful: volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-b493d5a2-76ba-4e4e-859c-0147e8131d15 attached to node k8s-agentpool-14513215-vmss000000. I0129 04:08:24.523866 1 controllerserver.go:433] attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-b493d5a2-76ba-4e4e-859c-0147e8131d15 to node k8s-agentpool-14513215-vmss000000 successfully I0129 04:08:24.524079 1 azure_controller_common.go:261] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-524a8b61-7ff4-48ef-b27e-7b37c344745e lun 1 to node k8s-agentpool-14513215-vmss000000, diskMap: map[] I0129 04:08:24.524303 1 controllerserver.go:413] Attach operation successful: volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-524a8b61-7ff4-48ef-b27e-7b37c344745e attached to node k8s-agentpool-14513215-vmss000000. ... skipping 123 lines ... I0129 04:09:35.370964 1 azure_controller_common.go:247] waiting for more requests on node k8s-agentpool-14513215-vmss000000, current disk attach: &{{%!s(*http.Response=&{200 OK 200 HTTP/1.1 1 1 map[Cache-Control:[no-cache] Content-Type:[application/json; charset=utf-8] Date:[Sun, 29 Jan 2023 04:09:35 GMT] Expires:[-1] Pragma:[no-cache] Server:[Microsoft-HTTPAPI/2.0 Microsoft-HTTPAPI/2.0] Strict-Transport-Security:[max-age=31536000; includeSubDomains] Vary:[Accept-Encoding] X-Content-Type-Options:[nosniff] X-Ms-Correlation-Request-Id:[56445661-2b01-4ced-8867-06a508c90958] X-Ms-Ratelimit-Remaining-Resource:[Microsoft.Compute/LowCostGet3Min;14934,Microsoft.Compute/LowCostGet30Min;119643] X-Ms-Ratelimit-Remaining-Subscription-Reads:[11686] X-Ms-Request-Id:[e19ed925-8f9a-467b-8b55-e9db2c3a116c] X-Ms-Routing-Request-Id:[WESTUS2:20230129T040935Z:56445661-2b01-4ced-8867-06a508c90958] X-Ms-Served-By:[43c1a32f-0327-4d8b-a820-a28f01b4195d_133148057271908261]] {0xc00095cd50} -1 [chunked] false true map[] 0xc000663900 0xc0005fb550})} %!s(*string=<nil>) %!s(*[]string=<nil>) %!s(*compute.DiskSku=&{StandardSSD_ZRS 0xc0001b9640}) %!s(*[]string=<nil>) %!s(*compute.ExtendedLocation=<nil>) %!s(*compute.DiskProperties=&{0xc00047db00 <nil> <nil> 0xc00092dc80 0xc0007413c8 0xc000741430 0xc0001b96b0 <nil> 0xc0001b96a0 0xc0007413d0 0xc0007413e0 <nil> <nil> Unattached 0xc00047dae8 <nil> <nil> AllowAll <nil> <nil> <nil> <nil> <nil> <nil> <nil> Enabled }) %!s(*string=0xc0001b96e0) %!s(*string=0xc0001b96c0) %!s(*string=0xc0001b96f0) %!s(*string=0xc0001b9700) map[k8s-azure-created-by:%!s(*string=0xc0001b95a0) kubernetes.io-created-for-pv-name:%!s(*string=0xc0001b95c0) kubernetes.io-created-for-pvc-name:%!s(*string=0xc0001b95e0) kubernetes.io-created-for-pvc-namespace:%!s(*string=0xc0001b9610)]} I0129 04:09:35.382507 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1193 I0129 04:09:35.382882 1 controllerserver.go:383] GetDiskLun returned: cannot find Lun for disk pvc-24197d05-7c1e-49f4-8fdb-3e7de0e165f1. Initiating attaching volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-24197d05-7c1e-49f4-8fdb-3e7de0e165f1 to node k8s-agentpool-14513215-vmss000000 (vmState Succeeded). I0129 04:09:35.382907 1 controllerserver.go:408] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-24197d05-7c1e-49f4-8fdb-3e7de0e165f1 to node k8s-agentpool-14513215-vmss000000 I0129 04:09:36.371649 1 azure_controller_common.go:261] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-66358d3a-0797-40fb-a0fd-78855f7c0503 lun 0 to node k8s-agentpool-14513215-vmss000000, diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-24197d05-7c1e-49f4-8fdb-3e7de0e165f1:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-24197d05-7c1e-49f4-8fdb-3e7de0e165f1 false 1}) /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-66358d3a-0797-40fb-a0fd-78855f7c0503:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-66358d3a-0797-40fb-a0fd-78855f7c0503 false 0})] I0129 04:09:36.371720 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-l6t2jkxe): vm(k8s-agentpool-14513215-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-24197d05-7c1e-49f4-8fdb-3e7de0e165f1:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-24197d05-7c1e-49f4-8fdb-3e7de0e165f1 false 1}) /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-66358d3a-0797-40fb-a0fd-78855f7c0503:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-66358d3a-0797-40fb-a0fd-78855f7c0503 false 0})]) I0129 04:09:36.553024 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-l6t2jkxe): vm(k8s-agentpool-14513215-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-24197d05-7c1e-49f4-8fdb-3e7de0e165f1:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-24197d05-7c1e-49f4-8fdb-3e7de0e165f1 false 1}) /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-66358d3a-0797-40fb-a0fd-78855f7c0503:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-66358d3a-0797-40fb-a0fd-78855f7c0503 false 0})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I0129 04:09:51.757327 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-l6t2jkxe, k8s-agentpool-14513215-vmss, k8s-agentpool-14513215-vmss000000) successfully I0129 04:09:51.757368 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool-14513215-vmss, kubetest-l6t2jkxe, k8s-agentpool-14513215-vmss000000) for cacheKey(kubetest-l6t2jkxe/k8s-agentpool-14513215-vmss) updated successfully I0129 04:09:51.757392 1 controllerserver.go:413] Attach operation successful: volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-66358d3a-0797-40fb-a0fd-78855f7c0503 attached to node k8s-agentpool-14513215-vmss000000. I0129 04:09:51.757416 1 controllerserver.go:433] attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-66358d3a-0797-40fb-a0fd-78855f7c0503 to node k8s-agentpool-14513215-vmss000000 successfully I0129 04:09:51.757459 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=16.386546487 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-l6t2jkxe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-66358d3a-0797-40fb-a0fd-78855f7c0503" node="k8s-agentpool-14513215-vmss000000" result_code="succeeded" I0129 04:09:51.757474 1 utils.go:84] GRPC response: {"publish_context":{"LUN":"0"}} ... skipping 70 lines ... I0129 04:11:03.083258 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1193 I0129 04:11:03.083692 1 controllerserver.go:383] GetDiskLun returned: cannot find Lun for disk pvc-de8db5a3-f4eb-4343-a916-0b26fffd27e7. Initiating attaching volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-de8db5a3-f4eb-4343-a916-0b26fffd27e7 to node k8s-agentpool-14513215-vmss000000 (vmState Succeeded). I0129 04:11:03.083728 1 controllerserver.go:408] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-de8db5a3-f4eb-4343-a916-0b26fffd27e7 to node k8s-agentpool-14513215-vmss000000 I0129 04:11:03.083752 1 azure_controller_common.go:247] waiting for more requests on node k8s-agentpool-14513215-vmss000000, current disk attach: &{{%!s(*http.Response=&{200 OK 200 HTTP/1.1 1 1 map[Cache-Control:[no-cache] Content-Type:[application/json; charset=utf-8] Date:[Sun, 29 Jan 2023 04:11:02 GMT] Expires:[-1] Pragma:[no-cache] Server:[Microsoft-HTTPAPI/2.0 Microsoft-HTTPAPI/2.0] Strict-Transport-Security:[max-age=31536000; includeSubDomains] Vary:[Accept-Encoding] X-Content-Type-Options:[nosniff] X-Ms-Correlation-Request-Id:[2edb9345-595c-4220-992c-3da806059056] X-Ms-Ratelimit-Remaining-Resource:[Microsoft.Compute/LowCostGet3Min;14951,Microsoft.Compute/LowCostGet30Min;119625] X-Ms-Ratelimit-Remaining-Subscription-Reads:[11947] X-Ms-Request-Id:[50c6d1d8-ca41-414c-9db1-e34436a2107a] X-Ms-Routing-Request-Id:[WESTUS2:20230129T041103Z:2edb9345-595c-4220-992c-3da806059056] X-Ms-Served-By:[43c1a32f-0327-4d8b-a820-a28f01b4195d_133148057271908261]] {0xc00001b1d0} -1 [chunked] false true map[] 0xc000ac2c00 0xc0005da790})} %!s(*string=<nil>) %!s(*[]string=<nil>) %!s(*compute.DiskSku=&{StandardSSD_ZRS 0xc000193be0}) %!s(*[]string=<nil>) %!s(*compute.ExtendedLocation=<nil>) %!s(*compute.DiskProperties=&{0xc0002006d8 <nil> <nil> 0xc0002591a0 0xc0007d5db0 0xc0007d5e10 0xc0001939d0 <nil> 0xc0001939b0 0xc0007d5db8 0xc0007d5dc8 <nil> <nil> Unattached 0xc0002006c0 <nil> <nil> AllowAll <nil> <nil> <nil> <nil> <nil> <nil> <nil> Enabled }) %!s(*string=0xc000193a00) %!s(*string=0xc0001939f0) %!s(*string=0xc000193a70) %!s(*string=0xc000193a90) map[k8s-azure-created-by:%!s(*string=0xc000193ad0) kubernetes.io-created-for-pv-name:%!s(*string=0xc000193b50) kubernetes.io-created-for-pvc-name:%!s(*string=0xc000193b80) kubernetes.io-created-for-pvc-namespace:%!s(*string=0xc000193ba0)]} I0129 04:11:04.084064 1 azure_controller_common.go:261] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-de8db5a3-f4eb-4343-a916-0b26fffd27e7 lun 0 to node k8s-agentpool-14513215-vmss000000, diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-de8db5a3-f4eb-4343-a916-0b26fffd27e7:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-de8db5a3-f4eb-4343-a916-0b26fffd27e7 false 0})] I0129 04:11:04.084191 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-l6t2jkxe): vm(k8s-agentpool-14513215-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-de8db5a3-f4eb-4343-a916-0b26fffd27e7:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-de8db5a3-f4eb-4343-a916-0b26fffd27e7 false 0})]) I0129 04:11:04.263764 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-l6t2jkxe): vm(k8s-agentpool-14513215-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-de8db5a3-f4eb-4343-a916-0b26fffd27e7:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-de8db5a3-f4eb-4343-a916-0b26fffd27e7 false 0})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I0129 04:11:14.415569 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-l6t2jkxe, k8s-agentpool-14513215-vmss, k8s-agentpool-14513215-vmss000000) successfully I0129 04:11:14.415610 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool-14513215-vmss, kubetest-l6t2jkxe, k8s-agentpool-14513215-vmss000000) for cacheKey(kubetest-l6t2jkxe/k8s-agentpool-14513215-vmss) updated successfully I0129 04:11:14.415648 1 controllerserver.go:413] Attach operation successful: volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-de8db5a3-f4eb-4343-a916-0b26fffd27e7 attached to node k8s-agentpool-14513215-vmss000000. I0129 04:11:14.415663 1 controllerserver.go:433] attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-de8db5a3-f4eb-4343-a916-0b26fffd27e7 to node k8s-agentpool-14513215-vmss000000 successfully I0129 04:11:14.415720 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=11.332021203 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-l6t2jkxe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-de8db5a3-f4eb-4343-a916-0b26fffd27e7" node="k8s-agentpool-14513215-vmss000000" result_code="succeeded" I0129 04:11:14.415743 1 utils.go:84] GRPC response: {"publish_context":{"LUN":"0"}} ... skipping 37 lines ... I0129 04:11:49.224641 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-l6t2jkxe, k8s-agentpool-14513215-vmss, k8s-agentpool-14513215-vmss000000) successfully I0129 04:11:49.224662 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool-14513215-vmss, kubetest-l6t2jkxe, k8s-agentpool-14513215-vmss000000) for cacheKey(kubetest-l6t2jkxe/k8s-agentpool-14513215-vmss) updated successfully I0129 04:11:49.224675 1 azure_controller_common.go:434] azureDisk - detach disk(pvc-de8db5a3-f4eb-4343-a916-0b26fffd27e7, /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-de8db5a3-f4eb-4343-a916-0b26fffd27e7) succeeded I0129 04:11:49.224689 1 controllerserver.go:480] detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-de8db5a3-f4eb-4343-a916-0b26fffd27e7 from node k8s-agentpool-14513215-vmss000000 successfully I0129 04:11:49.224729 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=6.252471016 request="azuredisk_csi_driver_controller_unpublish_volume" resource_group="kubetest-l6t2jkxe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-de8db5a3-f4eb-4343-a916-0b26fffd27e7" node="k8s-agentpool-14513215-vmss000000" result_code="succeeded" I0129 04:11:49.224746 1 utils.go:84] GRPC response: {} I0129 04:12:11.149220 1 azure_armclient.go:291] Received error in WaitForAsyncOperationCompletion: 'Future#WaitForCompletion: context has been cancelled: StatusCode=200 -- Original Error: context deadline exceeded' I0129 04:12:11.149277 1 azure_armclient.go:381] Received error in WaitForAsyncOperationResult: 'Future#WaitForCompletion: context has been cancelled: StatusCode=200 -- Original Error: context deadline exceeded', no response I0129 04:12:11.149312 1 azure_diskclient.go:202] Received error in disk.put.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-b76c0f7a-a721-4f1e-bf67-356764b6488a, error: Retriable: true, RetryAfter: 0s, HTTPStatusCode: -1, RawError: Future#WaitForCompletion: context has been cancelled: StatusCode=200 -- Original Error: context deadline exceeded I0129 04:12:11.149403 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=30.000141041 request="azuredisk_csi_driver_controller_create_volume" resource_group="kubetest-l6t2jkxe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="" result_code="failed_csi_driver_controller_create_volume" E0129 04:12:11.149424 1 utils.go:82] GRPC error: rpc error: code = Internal desc = Retriable: true, RetryAfter: 0s, HTTPStatusCode: -1, RawError: Future#WaitForCompletion: context has been cancelled: StatusCode=200 -- Original Error: context deadline exceeded I0129 04:12:12.160309 1 utils.go:77] GRPC call: /csi.v1.Controller/CreateVolume I0129 04:12:12.160386 1 utils.go:78] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.disk.csi.azure.com/zone":"westus2-1","topology.kubernetes.io/zone":"westus2-1"}}],"requisite":[{"segments":{"topology.disk.csi.azure.com/zone":"westus2-1","topology.kubernetes.io/zone":"westus2-1"}}]},"capacity_range":{"required_bytes":10737418240},"name":"pvc-b76c0f7a-a721-4f1e-bf67-356764b6488a","parameters":{"csi.storage.k8s.io/pv/name":"pvc-b76c0f7a-a721-4f1e-bf67-356764b6488a","csi.storage.k8s.io/pvc/name":"pvc-qfn79","csi.storage.k8s.io/pvc/namespace":"azuredisk-8582","skuName":"StandardSSD_ZRS"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}],"volume_content_source":{"Type":{"Snapshot":{"snapshot_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/azuredisk-csi-driver-test-fe6ba519-9f8a-11ed-987b-d279894c5862/providers/Microsoft.Compute/snapshots/snapshot-189d0881-09d2-4e67-ad38-37e1a2aeeddd"}}}} I0129 04:12:12.160828 1 controllerserver.go:154] diskZone(westus2-1) is reset as empty since disk(pvc-b76c0f7a-a721-4f1e-bf67-356764b6488a) is ZRS(StandardSSD_ZRS) I0129 04:12:12.160869 1 controllerserver.go:182] begin to create azure disk(pvc-b76c0f7a-a721-4f1e-bf67-356764b6488a) account type(StandardSSD_ZRS) rg(kubetest-l6t2jkxe) location(westus2) size(10) diskZone() maxShares(0) I0129 04:12:12.160894 1 azure_managedDiskController.go:94] azureDisk - creating new managed Name:pvc-b76c0f7a-a721-4f1e-bf67-356764b6488a StorageAccountType:StandardSSD_ZRS Size:10 I0129 04:12:14.374975 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1501 ... skipping 6 lines ... I0129 04:12:15.211240 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1501 I0129 04:12:15.211607 1 controllerserver.go:383] GetDiskLun returned: cannot find Lun for disk pvc-b76c0f7a-a721-4f1e-bf67-356764b6488a. Initiating attaching volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-b76c0f7a-a721-4f1e-bf67-356764b6488a to node k8s-agentpool-14513215-vmss000000 (vmState Succeeded). I0129 04:12:15.211642 1 controllerserver.go:408] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-b76c0f7a-a721-4f1e-bf67-356764b6488a to node k8s-agentpool-14513215-vmss000000 I0129 04:12:15.211689 1 azure_controller_common.go:247] waiting for more requests on node k8s-agentpool-14513215-vmss000000, current disk attach: &{{%!s(*http.Response=&{200 OK 200 HTTP/1.1 1 1 map[Cache-Control:[no-cache] Content-Type:[application/json; charset=utf-8] Date:[Sun, 29 Jan 2023 04:12:14 GMT] Expires:[-1] Pragma:[no-cache] Server:[Microsoft-HTTPAPI/2.0 Microsoft-HTTPAPI/2.0] Strict-Transport-Security:[max-age=31536000; includeSubDomains] Vary:[Accept-Encoding] X-Content-Type-Options:[nosniff] X-Ms-Correlation-Request-Id:[7fa37fcc-e6f3-4767-93ca-e34c858db4d3] X-Ms-Ratelimit-Remaining-Resource:[Microsoft.Compute/LowCostGet3Min;14941,Microsoft.Compute/LowCostGet30Min;119599] X-Ms-Ratelimit-Remaining-Subscription-Reads:[11923] X-Ms-Request-Id:[1316af4d-4713-4503-86e2-996099baa852] X-Ms-Routing-Request-Id:[WESTUS2:20230129T041215Z:7fa37fcc-e6f3-4767-93ca-e34c858db4d3] X-Ms-Served-By:[43c1a32f-0327-4d8b-a820-a28f01b4195d_133148057271908261]] {0xc0009d4720} -1 [chunked] false true map[] 0xc0005f7100 0xc0005da790})} %!s(*string=<nil>) %!s(*[]string=<nil>) %!s(*compute.DiskSku=&{StandardSSD_ZRS 0xc000175b30}) %!s(*[]string=<nil>) %!s(*compute.ExtendedLocation=<nil>) %!s(*compute.DiskProperties=&{0xc000804978 <nil> <nil> 0xc0006e93e0 0xc0005c05d4 0xc0005c0640 0xc000175bb0 <nil> 0xc000175ba0 0xc0005c05e0 0xc0005c05f0 <nil> <nil> Unattached 0xc000804960 <nil> <nil> AllowAll <nil> <nil> <nil> <nil> <nil> <nil> <nil> Enabled }) %!s(*string=0xc000175880) %!s(*string=0xc000175640) %!s(*string=0xc0001758c0) %!s(*string=0xc000175900) map[k8s-azure-created-by:%!s(*string=0xc000175940) kubernetes.io-created-for-pv-name:%!s(*string=0xc0001759c0) kubernetes.io-created-for-pvc-name:%!s(*string=0xc000175a40) kubernetes.io-created-for-pvc-namespace:%!s(*string=0xc000175ac0)]} I0129 04:12:16.212956 1 azure_controller_common.go:261] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-b76c0f7a-a721-4f1e-bf67-356764b6488a lun 0 to node k8s-agentpool-14513215-vmss000000, diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-b76c0f7a-a721-4f1e-bf67-356764b6488a:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-b76c0f7a-a721-4f1e-bf67-356764b6488a false 0})] I0129 04:12:16.213022 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-l6t2jkxe): vm(k8s-agentpool-14513215-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-b76c0f7a-a721-4f1e-bf67-356764b6488a:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-b76c0f7a-a721-4f1e-bf67-356764b6488a false 0})]) I0129 04:12:16.424769 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-l6t2jkxe): vm(k8s-agentpool-14513215-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-b76c0f7a-a721-4f1e-bf67-356764b6488a:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-b76c0f7a-a721-4f1e-bf67-356764b6488a false 0})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I0129 04:12:26.663131 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-l6t2jkxe, k8s-agentpool-14513215-vmss, k8s-agentpool-14513215-vmss000000) successfully I0129 04:12:26.663179 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool-14513215-vmss, kubetest-l6t2jkxe, k8s-agentpool-14513215-vmss000000) for cacheKey(kubetest-l6t2jkxe/k8s-agentpool-14513215-vmss) updated successfully I0129 04:12:26.663200 1 controllerserver.go:413] Attach operation successful: volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-b76c0f7a-a721-4f1e-bf67-356764b6488a attached to node k8s-agentpool-14513215-vmss000000. I0129 04:12:26.663214 1 controllerserver.go:433] attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-b76c0f7a-a721-4f1e-bf67-356764b6488a to node k8s-agentpool-14513215-vmss000000 successfully I0129 04:12:26.663258 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=11.451647132 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-l6t2jkxe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-b76c0f7a-a721-4f1e-bf67-356764b6488a" node="k8s-agentpool-14513215-vmss000000" result_code="succeeded" I0129 04:12:26.663282 1 utils.go:84] GRPC response: {"publish_context":{"LUN":"0"}} ... skipping 48 lines ... I0129 04:14:48.099243 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1192 I0129 04:14:48.099728 1 controllerserver.go:383] GetDiskLun returned: cannot find Lun for disk pvc-34255196-7cd4-4401-af6b-0dba3427c9bb. Initiating attaching volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-34255196-7cd4-4401-af6b-0dba3427c9bb to node k8s-agentpool-14513215-vmss000000 (vmState Succeeded). I0129 04:14:48.099761 1 controllerserver.go:408] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-34255196-7cd4-4401-af6b-0dba3427c9bb to node k8s-agentpool-14513215-vmss000000 I0129 04:14:48.099818 1 azure_controller_common.go:247] waiting for more requests on node k8s-agentpool-14513215-vmss000000, current disk attach: &{{%!s(*http.Response=&{200 OK 200 HTTP/1.1 1 1 map[Cache-Control:[no-cache] Content-Type:[application/json; charset=utf-8] Date:[Sun, 29 Jan 2023 04:14:47 GMT] Expires:[-1] Pragma:[no-cache] Server:[Microsoft-HTTPAPI/2.0 Microsoft-HTTPAPI/2.0] Strict-Transport-Security:[max-age=31536000; includeSubDomains] Vary:[Accept-Encoding] X-Content-Type-Options:[nosniff] X-Ms-Correlation-Request-Id:[62f5a371-b9cb-46aa-b9ea-e33c928ee256] X-Ms-Ratelimit-Remaining-Resource:[Microsoft.Compute/LowCostGet3Min;14971,Microsoft.Compute/LowCostGet30Min;119573] X-Ms-Ratelimit-Remaining-Subscription-Reads:[11902] X-Ms-Request-Id:[08860b86-8548-40be-85c2-d0eedfee3712] X-Ms-Routing-Request-Id:[WESTUS2:20230129T041448Z:62f5a371-b9cb-46aa-b9ea-e33c928ee256] X-Ms-Served-By:[43c1a32f-0327-4d8b-a820-a28f01b4195d_133148057271908261]] {0xc000926180} -1 [chunked] false true map[] 0xc000ac3e00 0xc0005da790})} %!s(*string=<nil>) %!s(*[]string=<nil>) %!s(*compute.DiskSku=&{StandardSSD_ZRS 0xc0009d26a0}) %!s(*[]string=<nil>) %!s(*compute.ExtendedLocation=<nil>) %!s(*compute.DiskProperties=&{0xc000a98468 <nil> <nil> 0xc000258300 0xc0007d56f8 0xc0007d5760 0xc0009d2700 <nil> 0xc0009d26f0 0xc0007d5700 0xc0007d5710 <nil> <nil> Unattached 0xc000a98450 <nil> <nil> AllowAll <nil> <nil> <nil> <nil> <nil> <nil> <nil> Enabled }) %!s(*string=0xc0009d25e0) %!s(*string=0xc0009d25d0) %!s(*string=0xc0009d25f0) %!s(*string=0xc0009d2600) map[k8s-azure-created-by:%!s(*string=0xc0009d2610) kubernetes.io-created-for-pv-name:%!s(*string=0xc0009d2630) kubernetes.io-created-for-pvc-name:%!s(*string=0xc0009d2650) kubernetes.io-created-for-pvc-namespace:%!s(*string=0xc0009d2670)]} I0129 04:14:49.100182 1 azure_controller_common.go:261] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-34255196-7cd4-4401-af6b-0dba3427c9bb lun 0 to node k8s-agentpool-14513215-vmss000000, diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-34255196-7cd4-4401-af6b-0dba3427c9bb:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-34255196-7cd4-4401-af6b-0dba3427c9bb false 0})] I0129 04:14:49.100287 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-l6t2jkxe): vm(k8s-agentpool-14513215-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-34255196-7cd4-4401-af6b-0dba3427c9bb:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-34255196-7cd4-4401-af6b-0dba3427c9bb false 0})]) I0129 04:14:49.235974 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-l6t2jkxe): vm(k8s-agentpool-14513215-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-34255196-7cd4-4401-af6b-0dba3427c9bb:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-34255196-7cd4-4401-af6b-0dba3427c9bb false 0})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I0129 04:14:59.498239 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 24989 I0129 04:14:59.501202 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-l6t2jkxe, k8s-agentpool-14513215-vmss, k8s-agentpool-14513215-vmss000000) successfully I0129 04:14:59.501235 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool-14513215-vmss, kubetest-l6t2jkxe, k8s-agentpool-14513215-vmss000000) for cacheKey(kubetest-l6t2jkxe/k8s-agentpool-14513215-vmss) updated successfully I0129 04:14:59.501258 1 controllerserver.go:413] Attach operation successful: volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-34255196-7cd4-4401-af6b-0dba3427c9bb attached to node k8s-agentpool-14513215-vmss000000. I0129 04:14:59.501277 1 controllerserver.go:433] attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-34255196-7cd4-4401-af6b-0dba3427c9bb to node k8s-agentpool-14513215-vmss000000 successfully I0129 04:14:59.501323 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=11.401588094 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-l6t2jkxe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-34255196-7cd4-4401-af6b-0dba3427c9bb" node="k8s-agentpool-14513215-vmss000000" result_code="succeeded" ... skipping 23 lines ... I0129 04:15:28.868494 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1501 I0129 04:15:28.868961 1 controllerserver.go:383] GetDiskLun returned: cannot find Lun for disk pvc-e33d15b3-b2b2-431e-8476-ad43bb707bbb. Initiating attaching volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-e33d15b3-b2b2-431e-8476-ad43bb707bbb to node k8s-agentpool-14513215-vmss000001 (vmState Succeeded). I0129 04:15:28.868996 1 controllerserver.go:408] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-e33d15b3-b2b2-431e-8476-ad43bb707bbb to node k8s-agentpool-14513215-vmss000001 I0129 04:15:28.869021 1 azure_controller_common.go:247] waiting for more requests on node k8s-agentpool-14513215-vmss000001, current disk attach: &{{%!s(*http.Response=&{200 OK 200 HTTP/1.1 1 1 map[Cache-Control:[no-cache] Content-Type:[application/json; charset=utf-8] Date:[Sun, 29 Jan 2023 04:15:28 GMT] Expires:[-1] Pragma:[no-cache] Server:[Microsoft-HTTPAPI/2.0 Microsoft-HTTPAPI/2.0] Strict-Transport-Security:[max-age=31536000; includeSubDomains] Vary:[Accept-Encoding] X-Content-Type-Options:[nosniff] X-Ms-Correlation-Request-Id:[189dc0cf-75a8-42b3-bb47-b477f28dc9ca] X-Ms-Ratelimit-Remaining-Resource:[Microsoft.Compute/LowCostGet3Min;14963,Microsoft.Compute/LowCostGet30Min;119602] X-Ms-Ratelimit-Remaining-Subscription-Reads:[11890] X-Ms-Request-Id:[2164c93b-62e2-4fd3-8ee5-7774c836ce5d] X-Ms-Routing-Request-Id:[WESTUS2:20230129T041528Z:189dc0cf-75a8-42b3-bb47-b477f28dc9ca] X-Ms-Served-By:[43c1a32f-0327-4d8b-a820-a28f01b4195d_133148057271908261]] {0xc00095cba0} -1 [chunked] false true map[] 0xc000a8cf00 0xc0005da790})} %!s(*string=<nil>) %!s(*[]string=<nil>) %!s(*compute.DiskSku=&{StandardSSD_ZRS 0xc0001958e0}) %!s(*[]string=<nil>) %!s(*compute.ExtendedLocation=<nil>) %!s(*compute.DiskProperties=&{0xc00047da10 <nil> <nil> 0xc0002593e0 0xc000a953a4 0xc000a95410 0xc000195a20 <nil> 0xc000195a00 0xc000a953b0 0xc000a953c0 <nil> <nil> Unattached 0xc00047d9f8 <nil> <nil> AllowAll <nil> <nil> <nil> <nil> <nil> <nil> <nil> Enabled }) %!s(*string=0xc000195a70) %!s(*string=0xc000195a40) %!s(*string=0xc000195ab0) %!s(*string=0xc000195af0) map[k8s-azure-created-by:%!s(*string=0xc000195b10) kubernetes.io-created-for-pv-name:%!s(*string=0xc000195b50) kubernetes.io-created-for-pvc-name:%!s(*string=0xc000195bb0) kubernetes.io-created-for-pvc-namespace:%!s(*string=0xc000195be0)]} I0129 04:15:29.869571 1 azure_controller_common.go:261] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-e33d15b3-b2b2-431e-8476-ad43bb707bbb lun 0 to node k8s-agentpool-14513215-vmss000001, diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-e33d15b3-b2b2-431e-8476-ad43bb707bbb:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-e33d15b3-b2b2-431e-8476-ad43bb707bbb false 0})] I0129 04:15:29.869644 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-l6t2jkxe): vm(k8s-agentpool-14513215-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-e33d15b3-b2b2-431e-8476-ad43bb707bbb:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-e33d15b3-b2b2-431e-8476-ad43bb707bbb false 0})]) I0129 04:15:30.119596 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-l6t2jkxe): vm(k8s-agentpool-14513215-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-e33d15b3-b2b2-431e-8476-ad43bb707bbb:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-e33d15b3-b2b2-431e-8476-ad43bb707bbb false 0})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I0129 04:15:40.396211 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-l6t2jkxe, k8s-agentpool-14513215-vmss, k8s-agentpool-14513215-vmss000001) successfully I0129 04:15:40.396248 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool-14513215-vmss, kubetest-l6t2jkxe, k8s-agentpool-14513215-vmss000001) for cacheKey(kubetest-l6t2jkxe/k8s-agentpool-14513215-vmss) updated successfully I0129 04:15:40.396268 1 controllerserver.go:413] Attach operation successful: volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-e33d15b3-b2b2-431e-8476-ad43bb707bbb attached to node k8s-agentpool-14513215-vmss000001. I0129 04:15:40.396281 1 controllerserver.go:433] attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-e33d15b3-b2b2-431e-8476-ad43bb707bbb to node k8s-agentpool-14513215-vmss000001 successfully I0129 04:15:40.396325 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=11.527360915 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-l6t2jkxe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-e33d15b3-b2b2-431e-8476-ad43bb707bbb" node="k8s-agentpool-14513215-vmss000001" result_code="succeeded" I0129 04:15:40.396348 1 utils.go:84] GRPC response: {"publish_context":{"LUN":"0"}} ... skipping 113 lines ... I0129 04:18:02.795640 1 controllerserver.go:383] GetDiskLun returned: cannot find Lun for disk pvc-603a1296-03a0-4eae-88a4-d0710c4cf876. Initiating attaching volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-603a1296-03a0-4eae-88a4-d0710c4cf876 to node k8s-agentpool-14513215-vmss000000 (vmState Succeeded). I0129 04:18:02.795658 1 controllerserver.go:408] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-603a1296-03a0-4eae-88a4-d0710c4cf876 to node k8s-agentpool-14513215-vmss000000 I0129 04:18:02.795688 1 controllerserver.go:383] GetDiskLun returned: cannot find Lun for disk pvc-8d63c977-bddf-49eb-a85b-70995d906eef. Initiating attaching volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-8d63c977-bddf-49eb-a85b-70995d906eef to node k8s-agentpool-14513215-vmss000000 (vmState Succeeded). I0129 04:18:02.795704 1 controllerserver.go:408] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-8d63c977-bddf-49eb-a85b-70995d906eef to node k8s-agentpool-14513215-vmss000000 I0129 04:18:03.795921 1 azure_controller_common.go:261] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-88ab9e22-abe4-45b6-bd8b-9e1eb7f82007 lun 0 to node k8s-agentpool-14513215-vmss000000, diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-603a1296-03a0-4eae-88a4-d0710c4cf876:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-603a1296-03a0-4eae-88a4-d0710c4cf876 false 1}) /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-88ab9e22-abe4-45b6-bd8b-9e1eb7f82007:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-88ab9e22-abe4-45b6-bd8b-9e1eb7f82007 false 0}) /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-8d63c977-bddf-49eb-a85b-70995d906eef:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-8d63c977-bddf-49eb-a85b-70995d906eef false 2})] I0129 04:18:03.796024 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-l6t2jkxe): vm(k8s-agentpool-14513215-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-603a1296-03a0-4eae-88a4-d0710c4cf876:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-603a1296-03a0-4eae-88a4-d0710c4cf876 false 1}) /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-88ab9e22-abe4-45b6-bd8b-9e1eb7f82007:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-88ab9e22-abe4-45b6-bd8b-9e1eb7f82007 false 0}) /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-8d63c977-bddf-49eb-a85b-70995d906eef:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-8d63c977-bddf-49eb-a85b-70995d906eef false 2})]) I0129 04:18:04.715885 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-l6t2jkxe): vm(k8s-agentpool-14513215-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-603a1296-03a0-4eae-88a4-d0710c4cf876:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-603a1296-03a0-4eae-88a4-d0710c4cf876 false 1}) /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-88ab9e22-abe4-45b6-bd8b-9e1eb7f82007:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-88ab9e22-abe4-45b6-bd8b-9e1eb7f82007 false 0}) /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-8d63c977-bddf-49eb-a85b-70995d906eef:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-8d63c977-bddf-49eb-a85b-70995d906eef false 2})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I0129 04:18:14.819439 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-l6t2jkxe, k8s-agentpool-14513215-vmss, k8s-agentpool-14513215-vmss000000) successfully I0129 04:18:14.819487 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool-14513215-vmss, kubetest-l6t2jkxe, k8s-agentpool-14513215-vmss000000) for cacheKey(kubetest-l6t2jkxe/k8s-agentpool-14513215-vmss) updated successfully I0129 04:18:14.819518 1 controllerserver.go:413] Attach operation successful: volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-88ab9e22-abe4-45b6-bd8b-9e1eb7f82007 attached to node k8s-agentpool-14513215-vmss000000. I0129 04:18:14.819535 1 controllerserver.go:433] attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-88ab9e22-abe4-45b6-bd8b-9e1eb7f82007 to node k8s-agentpool-14513215-vmss000000 successfully I0129 04:18:14.819607 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=12.050101502 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-l6t2jkxe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-88ab9e22-abe4-45b6-bd8b-9e1eb7f82007" node="k8s-agentpool-14513215-vmss000000" result_code="succeeded" I0129 04:18:14.819613 1 azure_controller_common.go:261] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-603a1296-03a0-4eae-88a4-d0710c4cf876 lun 1 to node k8s-agentpool-14513215-vmss000000, diskMap: map[] ... skipping 115 lines ... I0129 04:19:27.947937 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1248 I0129 04:19:27.948423 1 controllerserver.go:383] GetDiskLun returned: cannot find Lun for disk pvc-8bba27cc-8143-46f5-b1d8-ba639323604e. Initiating attaching volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-8bba27cc-8143-46f5-b1d8-ba639323604e to node k8s-agentpool-14513215-vmss000000 (vmState Succeeded). I0129 04:19:27.948556 1 controllerserver.go:408] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-8bba27cc-8143-46f5-b1d8-ba639323604e to node k8s-agentpool-14513215-vmss000000 I0129 04:19:27.948591 1 azure_controller_common.go:247] waiting for more requests on node k8s-agentpool-14513215-vmss000000, current disk attach: &{{%!s(*http.Response=&{200 OK 200 HTTP/1.1 1 1 map[Cache-Control:[no-cache] Content-Type:[application/json; charset=utf-8] Date:[Sun, 29 Jan 2023 04:19:27 GMT] Expires:[-1] Pragma:[no-cache] Server:[Microsoft-HTTPAPI/2.0 Microsoft-HTTPAPI/2.0] Strict-Transport-Security:[max-age=31536000; includeSubDomains] Vary:[Accept-Encoding] X-Content-Type-Options:[nosniff] X-Ms-Correlation-Request-Id:[13e5aaea-7ef0-4cb0-b466-7a7dfd919408] X-Ms-Ratelimit-Remaining-Resource:[Microsoft.Compute/LowCostGet3Min;14920,Microsoft.Compute/LowCostGet30Min;119511] X-Ms-Ratelimit-Remaining-Subscription-Reads:[11968] X-Ms-Request-Id:[0fc86532-1f11-442d-891d-2cf6563ab1df] X-Ms-Routing-Request-Id:[WESTUS2:20230129T041927Z:13e5aaea-7ef0-4cb0-b466-7a7dfd919408] X-Ms-Served-By:[43c1a32f-0327-4d8b-a820-a28f01b4195d_133148057271908261]] {0xc00095c240} -1 [chunked] false true map[] 0xc00095f300 0xc0005dae70})} %!s(*string=<nil>) %!s(*[]string=<nil>) %!s(*compute.DiskSku=&{StandardSSD_LRS 0xc000137f20}) %!s(*[]string=&[1]) %!s(*compute.ExtendedLocation=<nil>) %!s(*compute.DiskProperties=&{0xc00077f4a0 <nil> <nil> 0xc0006e9e60 0xc0007194a8 0xc000719520 0xc000137ff0 <nil> 0xc000137fe0 0xc0007194b0 0xc0007194c0 <nil> <nil> Unattached 0xc00077f458 <nil> <nil> AllowAll <nil> <nil> <nil> <nil> <nil> <nil> <nil> Enabled }) %!s(*string=0xc000138030) %!s(*string=0xc000138000) %!s(*string=0xc000138070) %!s(*string=0xc000137de0) map[k8s-azure-created-by:%!s(*string=0xc000137e00) kubernetes.io-created-for-pv-name:%!s(*string=0xc000137e40) kubernetes.io-created-for-pvc-name:%!s(*string=0xc000137ea0) kubernetes.io-created-for-pvc-namespace:%!s(*string=0xc000137ed0)]} I0129 04:19:28.948906 1 azure_controller_common.go:261] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-8bba27cc-8143-46f5-b1d8-ba639323604e lun 0 to node k8s-agentpool-14513215-vmss000000, diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-8bba27cc-8143-46f5-b1d8-ba639323604e:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-8bba27cc-8143-46f5-b1d8-ba639323604e false 0})] I0129 04:19:28.949039 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-l6t2jkxe): vm(k8s-agentpool-14513215-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-8bba27cc-8143-46f5-b1d8-ba639323604e:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-8bba27cc-8143-46f5-b1d8-ba639323604e false 0})]) I0129 04:19:29.154771 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-l6t2jkxe): vm(k8s-agentpool-14513215-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-8bba27cc-8143-46f5-b1d8-ba639323604e:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-8bba27cc-8143-46f5-b1d8-ba639323604e false 0})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I0129 04:19:39.257016 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-l6t2jkxe, k8s-agentpool-14513215-vmss, k8s-agentpool-14513215-vmss000000) successfully I0129 04:19:39.257114 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool-14513215-vmss, kubetest-l6t2jkxe, k8s-agentpool-14513215-vmss000000) for cacheKey(kubetest-l6t2jkxe/k8s-agentpool-14513215-vmss) updated successfully I0129 04:19:39.257133 1 controllerserver.go:413] Attach operation successful: volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-8bba27cc-8143-46f5-b1d8-ba639323604e attached to node k8s-agentpool-14513215-vmss000000. I0129 04:19:39.257147 1 controllerserver.go:433] attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-8bba27cc-8143-46f5-b1d8-ba639323604e to node k8s-agentpool-14513215-vmss000000 successfully I0129 04:19:39.257194 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=11.308764465 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-l6t2jkxe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-8bba27cc-8143-46f5-b1d8-ba639323604e" node="k8s-agentpool-14513215-vmss000000" result_code="succeeded" I0129 04:19:39.257222 1 utils.go:84] GRPC response: {"publish_context":{"LUN":"0"}} ... skipping 89 lines ... I0129 04:22:20.456494 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1248 I0129 04:22:20.456784 1 controllerserver.go:383] GetDiskLun returned: cannot find Lun for disk pvc-8bba27cc-8143-46f5-b1d8-ba639323604e. Initiating attaching volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-8bba27cc-8143-46f5-b1d8-ba639323604e to node k8s-agentpool-14513215-vmss000000 (vmState Succeeded). I0129 04:22:20.456814 1 controllerserver.go:408] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-8bba27cc-8143-46f5-b1d8-ba639323604e to node k8s-agentpool-14513215-vmss000000 I0129 04:22:20.456838 1 azure_controller_common.go:247] waiting for more requests on node k8s-agentpool-14513215-vmss000000, current disk attach: &{{%!s(*http.Response=&{200 OK 200 HTTP/1.1 1 1 map[Cache-Control:[no-cache] Content-Type:[application/json; charset=utf-8] Date:[Sun, 29 Jan 2023 04:22:20 GMT] Expires:[-1] Pragma:[no-cache] Server:[Microsoft-HTTPAPI/2.0 Microsoft-HTTPAPI/2.0] Strict-Transport-Security:[max-age=31536000; includeSubDomains] Vary:[Accept-Encoding] X-Content-Type-Options:[nosniff] X-Ms-Correlation-Request-Id:[05445143-5c7a-451e-ae33-160bcf939ada] X-Ms-Ratelimit-Remaining-Resource:[Microsoft.Compute/LowCostGet3Min;14916,Microsoft.Compute/LowCostGet30Min;119472] X-Ms-Ratelimit-Remaining-Subscription-Reads:[11944] X-Ms-Request-Id:[ace635c8-fa36-479d-8aef-755c378486d6] X-Ms-Routing-Request-Id:[WESTUS2:20230129T042220Z:05445143-5c7a-451e-ae33-160bcf939ada] X-Ms-Served-By:[43c1a32f-0327-4d8b-a820-a28f01b4195d_133148057271908261]] {0xc00001b050} -1 [chunked] false true map[] 0xc000ac2f00 0xc0005dae70})} %!s(*string=<nil>) %!s(*[]string=<nil>) %!s(*compute.DiskSku=&{StandardSSD_LRS 0xc000142950}) %!s(*[]string=&[1]) %!s(*compute.ExtendedLocation=<nil>) %!s(*compute.DiskProperties=&{0xc0007ca660 <nil> <nil> 0xc000412ea0 0xc000719398 0xc000719430 0xc000142aa0 <nil> 0xc000142a70 0xc0007193a0 0xc0007193b0 <nil> <nil> Unattached 0xc0007ca648 <nil> <nil> AllowAll <nil> <nil> <nil> <nil> <nil> <nil> <nil> Enabled }) %!s(*string=0xc000142610) %!s(*string=0xc000142ae0) %!s(*string=0xc000142650) %!s(*string=0xc000142690) map[k8s-azure-created-by:%!s(*string=0xc0001426d0) kubernetes.io-created-for-pv-name:%!s(*string=0xc000142750) kubernetes.io-created-for-pvc-name:%!s(*string=0xc0001427d0) kubernetes.io-created-for-pvc-namespace:%!s(*string=0xc000142850)]} I0129 04:22:21.457058 1 azure_controller_common.go:261] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-8bba27cc-8143-46f5-b1d8-ba639323604e lun 0 to node k8s-agentpool-14513215-vmss000000, diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-8bba27cc-8143-46f5-b1d8-ba639323604e:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-8bba27cc-8143-46f5-b1d8-ba639323604e false 0})] I0129 04:22:21.457196 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-l6t2jkxe): vm(k8s-agentpool-14513215-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-8bba27cc-8143-46f5-b1d8-ba639323604e:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-8bba27cc-8143-46f5-b1d8-ba639323604e false 0})]) I0129 04:22:21.638577 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-l6t2jkxe): vm(k8s-agentpool-14513215-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-8bba27cc-8143-46f5-b1d8-ba639323604e:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-8bba27cc-8143-46f5-b1d8-ba639323604e false 0})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I0129 04:22:31.769521 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-l6t2jkxe, k8s-agentpool-14513215-vmss, k8s-agentpool-14513215-vmss000000) successfully I0129 04:22:31.769577 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool-14513215-vmss, kubetest-l6t2jkxe, k8s-agentpool-14513215-vmss000000) for cacheKey(kubetest-l6t2jkxe/k8s-agentpool-14513215-vmss) updated successfully I0129 04:22:31.769614 1 controllerserver.go:413] Attach operation successful: volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-8bba27cc-8143-46f5-b1d8-ba639323604e attached to node k8s-agentpool-14513215-vmss000000. I0129 04:22:31.769630 1 controllerserver.go:433] attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-8bba27cc-8143-46f5-b1d8-ba639323604e to node k8s-agentpool-14513215-vmss000000 successfully I0129 04:22:31.769707 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=11.312892696 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-l6t2jkxe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-8bba27cc-8143-46f5-b1d8-ba639323604e" node="k8s-agentpool-14513215-vmss000000" result_code="succeeded" I0129 04:22:31.769753 1 utils.go:84] GRPC response: {"publish_context":{"LUN":"0"}} ... skipping 20 lines ... I0129 04:22:56.783211 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1214 I0129 04:22:56.783537 1 controllerserver.go:383] GetDiskLun returned: cannot find Lun for disk pvc-4639d67a-bf06-4b53-9a63-cc446cdba25e. Initiating attaching volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-4639d67a-bf06-4b53-9a63-cc446cdba25e to node k8s-agentpool-14513215-vmss000001 (vmState Succeeded). I0129 04:22:56.783565 1 controllerserver.go:408] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-4639d67a-bf06-4b53-9a63-cc446cdba25e to node k8s-agentpool-14513215-vmss000001 I0129 04:22:56.783586 1 azure_controller_common.go:247] waiting for more requests on node k8s-agentpool-14513215-vmss000001, current disk attach: &{{%!s(*http.Response=&{200 OK 200 HTTP/1.1 1 1 map[Cache-Control:[no-cache] Content-Type:[application/json; charset=utf-8] Date:[Sun, 29 Jan 2023 04:22:56 GMT] Expires:[-1] Pragma:[no-cache] Server:[Microsoft-HTTPAPI/2.0 Microsoft-HTTPAPI/2.0] Strict-Transport-Security:[max-age=31536000; includeSubDomains] Vary:[Accept-Encoding] X-Content-Type-Options:[nosniff] X-Ms-Correlation-Request-Id:[b56d12cf-887a-4fac-bbb3-b7915192b5c3] X-Ms-Ratelimit-Remaining-Resource:[Microsoft.Compute/LowCostGet3Min;14920,Microsoft.Compute/LowCostGet30Min;119460] X-Ms-Ratelimit-Remaining-Subscription-Reads:[11935] X-Ms-Request-Id:[1c38c767-6779-4fff-9cc7-a3739b0acdf9] X-Ms-Routing-Request-Id:[WESTUS2:20230129T042256Z:b56d12cf-887a-4fac-bbb3-b7915192b5c3] X-Ms-Served-By:[43c1a32f-0327-4d8b-a820-a28f01b4195d_133148057271908261]] {0xc0004fb5c0} -1 [chunked] false true map[] 0xc000662700 0xc0005dae70})} %!s(*string=<nil>) %!s(*[]string=<nil>) %!s(*compute.DiskSku=&{StandardSSD_ZRS 0xc000034d20}) %!s(*[]string=<nil>) %!s(*compute.ExtendedLocation=<nil>) %!s(*compute.DiskProperties=&{0xc00047cd20 <nil> <nil> 0xc0006e9260 0xc000775688 0xc000775750 0xc000034aa0 <nil> 0xc000034a90 0xc000775690 0xc0007756b0 <nil> <nil> Unattached 0xc00047cd08 <nil> <nil> AllowAll <nil> <nil> <nil> <nil> <nil> <nil> <nil> Enabled }) %!s(*string=0xc000034ae0) %!s(*string=0xc000034ab0) %!s(*string=0xc000034b20) %!s(*string=0xc000034b40) map[disk:%!s(*string=0xc000034b60) k8s-azure-created-by:%!s(*string=0xc000034ba0) kubernetes.io-created-for-pv-name:%!s(*string=0xc000034c00) kubernetes.io-created-for-pvc-name:%!s(*string=0xc000034c40) kubernetes.io-created-for-pvc-namespace:%!s(*string=0xc000034ca0)]} I0129 04:22:57.784200 1 azure_controller_common.go:261] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-4639d67a-bf06-4b53-9a63-cc446cdba25e lun 0 to node k8s-agentpool-14513215-vmss000001, diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-4639d67a-bf06-4b53-9a63-cc446cdba25e:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-4639d67a-bf06-4b53-9a63-cc446cdba25e false 0})] I0129 04:22:57.784272 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-l6t2jkxe): vm(k8s-agentpool-14513215-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-4639d67a-bf06-4b53-9a63-cc446cdba25e:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-4639d67a-bf06-4b53-9a63-cc446cdba25e false 0})]) I0129 04:22:57.951873 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-l6t2jkxe): vm(k8s-agentpool-14513215-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-4639d67a-bf06-4b53-9a63-cc446cdba25e:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-4639d67a-bf06-4b53-9a63-cc446cdba25e false 0})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I0129 04:23:08.093157 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-l6t2jkxe, k8s-agentpool-14513215-vmss, k8s-agentpool-14513215-vmss000001) successfully I0129 04:23:08.093199 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool-14513215-vmss, kubetest-l6t2jkxe, k8s-agentpool-14513215-vmss000001) for cacheKey(kubetest-l6t2jkxe/k8s-agentpool-14513215-vmss) updated successfully I0129 04:23:08.093215 1 controllerserver.go:413] Attach operation successful: volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-4639d67a-bf06-4b53-9a63-cc446cdba25e attached to node k8s-agentpool-14513215-vmss000001. I0129 04:23:08.093224 1 controllerserver.go:433] attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-4639d67a-bf06-4b53-9a63-cc446cdba25e to node k8s-agentpool-14513215-vmss000001 successfully I0129 04:23:08.093256 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=11.309725559 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-l6t2jkxe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-4639d67a-bf06-4b53-9a63-cc446cdba25e" node="k8s-agentpool-14513215-vmss000001" result_code="succeeded" I0129 04:23:08.093271 1 utils.go:84] GRPC response: {"publish_context":{"LUN":"0"}} ... skipping 70 lines ... I0129 04:24:38.798463 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1193 I0129 04:24:38.798833 1 controllerserver.go:383] GetDiskLun returned: cannot find Lun for disk pvc-a2393c9e-43a9-46c4-9a76-4a85b6222356. Initiating attaching volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-a2393c9e-43a9-46c4-9a76-4a85b6222356 to node k8s-agentpool-14513215-vmss000000 (vmState Succeeded). I0129 04:24:38.798861 1 controllerserver.go:408] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-a2393c9e-43a9-46c4-9a76-4a85b6222356 to node k8s-agentpool-14513215-vmss000000 I0129 04:24:38.798921 1 azure_controller_common.go:247] waiting for more requests on node k8s-agentpool-14513215-vmss000000, current disk attach: &{{%!s(*http.Response=&{200 OK 200 HTTP/1.1 1 1 map[Cache-Control:[no-cache] Content-Type:[application/json; charset=utf-8] Date:[Sun, 29 Jan 2023 04:24:38 GMT] Expires:[-1] Pragma:[no-cache] Server:[Microsoft-HTTPAPI/2.0 Microsoft-HTTPAPI/2.0] Strict-Transport-Security:[max-age=31536000; includeSubDomains] Vary:[Accept-Encoding] X-Content-Type-Options:[nosniff] X-Ms-Correlation-Request-Id:[912d917c-0fa5-4314-bda0-27c30e28541b] X-Ms-Ratelimit-Remaining-Resource:[Microsoft.Compute/LowCostGet3Min;14922,Microsoft.Compute/LowCostGet30Min;119399] X-Ms-Ratelimit-Remaining-Subscription-Reads:[11908] X-Ms-Request-Id:[07b88d0b-ddbd-44c5-9110-5c82163ec6d3] X-Ms-Routing-Request-Id:[WESTUS2:20230129T042438Z:912d917c-0fa5-4314-bda0-27c30e28541b] X-Ms-Served-By:[43c1a32f-0327-4d8b-a820-a28f01b4195d_133148057271908261]] {0xc000a8f800} -1 [chunked] false true map[] 0xc000a8cc00 0xc0005dae70})} %!s(*string=<nil>) %!s(*[]string=<nil>) %!s(*compute.DiskSku=&{StandardSSD_ZRS 0xc0004ec9a0}) %!s(*[]string=<nil>) %!s(*compute.ExtendedLocation=<nil>) %!s(*compute.DiskProperties=&{0xc0007053b0 <nil> <nil> 0xc0002584e0 0xc0007191a8 0xc000719210 0xc0004ecaf0 <nil> 0xc0004ecac0 0xc0007191b0 0xc0007191c0 <nil> <nil> Unattached 0xc000705350 <nil> <nil> AllowAll <nil> <nil> <nil> <nil> <nil> <nil> <nil> Enabled }) %!s(*string=0xc0004ecb70) %!s(*string=0xc0004ecb30) %!s(*string=0xc0004ecbb0) %!s(*string=0xc0004ec610) map[k8s-azure-created-by:%!s(*string=0xc0004ec640) kubernetes.io-created-for-pv-name:%!s(*string=0xc0004ec6a0) kubernetes.io-created-for-pvc-name:%!s(*string=0xc0004ec6d0) kubernetes.io-created-for-pvc-namespace:%!s(*string=0xc0004ec710)]} I0129 04:24:39.799288 1 azure_controller_common.go:261] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-a2393c9e-43a9-46c4-9a76-4a85b6222356 lun 0 to node k8s-agentpool-14513215-vmss000000, diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-a2393c9e-43a9-46c4-9a76-4a85b6222356:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-a2393c9e-43a9-46c4-9a76-4a85b6222356 false 0})] I0129 04:24:39.799413 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-l6t2jkxe): vm(k8s-agentpool-14513215-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-a2393c9e-43a9-46c4-9a76-4a85b6222356:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-a2393c9e-43a9-46c4-9a76-4a85b6222356 false 0})]) I0129 04:24:39.972635 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-l6t2jkxe): vm(k8s-agentpool-14513215-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-a2393c9e-43a9-46c4-9a76-4a85b6222356:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-a2393c9e-43a9-46c4-9a76-4a85b6222356 false 0})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I0129 04:24:50.064445 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-l6t2jkxe, k8s-agentpool-14513215-vmss, k8s-agentpool-14513215-vmss000000) successfully I0129 04:24:50.064783 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool-14513215-vmss, kubetest-l6t2jkxe, k8s-agentpool-14513215-vmss000000) for cacheKey(kubetest-l6t2jkxe/k8s-agentpool-14513215-vmss) updated successfully I0129 04:24:50.064914 1 controllerserver.go:413] Attach operation successful: volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-a2393c9e-43a9-46c4-9a76-4a85b6222356 attached to node k8s-agentpool-14513215-vmss000000. I0129 04:24:50.064937 1 controllerserver.go:433] attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-a2393c9e-43a9-46c4-9a76-4a85b6222356 to node k8s-agentpool-14513215-vmss000000 successfully I0129 04:24:50.065007 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=11.266151759 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-l6t2jkxe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-a2393c9e-43a9-46c4-9a76-4a85b6222356" node="k8s-agentpool-14513215-vmss000000" result_code="succeeded" I0129 04:24:50.065051 1 utils.go:84] GRPC response: {"publish_context":{"LUN":"0"}} ... skipping 34 lines ... I0129 04:26:15.784372 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1248 I0129 04:26:15.784701 1 controllerserver.go:383] GetDiskLun returned: cannot find Lun for disk pvc-d08b0a5e-3696-48c6-888a-2ab81ae3788f. Initiating attaching volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-d08b0a5e-3696-48c6-888a-2ab81ae3788f to node k8s-agentpool-14513215-vmss000000 (vmState Succeeded). I0129 04:26:15.784728 1 controllerserver.go:408] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-d08b0a5e-3696-48c6-888a-2ab81ae3788f to node k8s-agentpool-14513215-vmss000000 I0129 04:26:15.784857 1 azure_controller_common.go:247] waiting for more requests on node k8s-agentpool-14513215-vmss000000, current disk attach: &{{%!s(*http.Response=&{200 OK 200 HTTP/1.1 1 1 map[Cache-Control:[no-cache] Content-Type:[application/json; charset=utf-8] Date:[Sun, 29 Jan 2023 04:26:15 GMT] Expires:[-1] Pragma:[no-cache] Server:[Microsoft-HTTPAPI/2.0 Microsoft-HTTPAPI/2.0] Strict-Transport-Security:[max-age=31536000; includeSubDomains] Vary:[Accept-Encoding] X-Content-Type-Options:[nosniff] X-Ms-Correlation-Request-Id:[45056e95-7cd5-4df5-a75f-9a1610be2630] X-Ms-Ratelimit-Remaining-Resource:[Microsoft.Compute/LowCostGet3Min;14930,Microsoft.Compute/LowCostGet30Min;119437] X-Ms-Ratelimit-Remaining-Subscription-Reads:[11891] X-Ms-Request-Id:[1309daab-c19f-4ca1-9700-5d26ecf5119c] X-Ms-Routing-Request-Id:[WESTUS2:20230129T042615Z:45056e95-7cd5-4df5-a75f-9a1610be2630] X-Ms-Served-By:[43c1a32f-0327-4d8b-a820-a28f01b4195d_133148057271908261]] {0xc000aa8a50} -1 [chunked] false true map[] 0xc000662800 0xc0005dae70})} %!s(*string=<nil>) %!s(*[]string=<nil>) %!s(*compute.DiskSku=&{StandardSSD_LRS 0xc0004dcf00}) %!s(*[]string=&[1]) %!s(*compute.ExtendedLocation=<nil>) %!s(*compute.DiskProperties=&{0xc000a98150 <nil> <nil> 0xc0008bff80 0xc000a95ca0 0xc000a95d00 0xc0004dd000 <nil> 0xc0004dcff0 0xc000a95ca8 0xc000a95cb8 <nil> <nil> Unattached 0xc000a98138 <nil> <nil> AllowAll <nil> <nil> <nil> <nil> <nil> <nil> <nil> Enabled }) %!s(*string=0xc0004dd040) %!s(*string=0xc0004dd010) %!s(*string=0xc0004dd080) %!s(*string=0xc0004dd0c0) map[k8s-azure-created-by:%!s(*string=0xc0004dcd80) kubernetes.io-created-for-pv-name:%!s(*string=0xc0004dce00) kubernetes.io-created-for-pvc-name:%!s(*string=0xc0004dce30) kubernetes.io-created-for-pvc-namespace:%!s(*string=0xc0004dce50)]} I0129 04:26:16.785115 1 azure_controller_common.go:261] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-d08b0a5e-3696-48c6-888a-2ab81ae3788f lun 0 to node k8s-agentpool-14513215-vmss000000, diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-d08b0a5e-3696-48c6-888a-2ab81ae3788f:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-d08b0a5e-3696-48c6-888a-2ab81ae3788f false 0})] I0129 04:26:16.785210 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-l6t2jkxe): vm(k8s-agentpool-14513215-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-d08b0a5e-3696-48c6-888a-2ab81ae3788f:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-d08b0a5e-3696-48c6-888a-2ab81ae3788f false 0})]) I0129 04:26:16.945293 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-l6t2jkxe): vm(k8s-agentpool-14513215-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-d08b0a5e-3696-48c6-888a-2ab81ae3788f:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-d08b0a5e-3696-48c6-888a-2ab81ae3788f false 0})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I0129 04:26:27.045614 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-l6t2jkxe, k8s-agentpool-14513215-vmss, k8s-agentpool-14513215-vmss000000) successfully I0129 04:26:27.045657 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool-14513215-vmss, kubetest-l6t2jkxe, k8s-agentpool-14513215-vmss000000) for cacheKey(kubetest-l6t2jkxe/k8s-agentpool-14513215-vmss) updated successfully I0129 04:26:27.045682 1 controllerserver.go:413] Attach operation successful: volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-d08b0a5e-3696-48c6-888a-2ab81ae3788f attached to node k8s-agentpool-14513215-vmss000000. I0129 04:26:27.045699 1 controllerserver.go:433] attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-d08b0a5e-3696-48c6-888a-2ab81ae3788f to node k8s-agentpool-14513215-vmss000000 successfully I0129 04:26:27.045749 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=11.261033005 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-l6t2jkxe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-d08b0a5e-3696-48c6-888a-2ab81ae3788f" node="k8s-agentpool-14513215-vmss000000" result_code="succeeded" I0129 04:26:27.045777 1 utils.go:84] GRPC response: {"publish_context":{"LUN":"0"}} ... skipping 12 lines ... I0129 04:27:45.540537 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1192 I0129 04:27:45.541100 1 controllerserver.go:383] GetDiskLun returned: cannot find Lun for disk pvc-27531535-f336-44c6-8b81-6706d20c1395. Initiating attaching volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-27531535-f336-44c6-8b81-6706d20c1395 to node k8s-agentpool-14513215-vmss000001 (vmState Succeeded). I0129 04:27:45.541149 1 controllerserver.go:408] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-27531535-f336-44c6-8b81-6706d20c1395 to node k8s-agentpool-14513215-vmss000001 I0129 04:27:45.541200 1 azure_controller_common.go:247] waiting for more requests on node k8s-agentpool-14513215-vmss000001, current disk attach: &{{%!s(*http.Response=&{200 OK 200 HTTP/1.1 1 1 map[Cache-Control:[no-cache] Content-Type:[application/json; charset=utf-8] Date:[Sun, 29 Jan 2023 04:27:44 GMT] Expires:[-1] Pragma:[no-cache] Server:[Microsoft-HTTPAPI/2.0 Microsoft-HTTPAPI/2.0] Strict-Transport-Security:[max-age=31536000; includeSubDomains] Vary:[Accept-Encoding] X-Content-Type-Options:[nosniff] X-Ms-Correlation-Request-Id:[1a10f60f-4a49-4b58-a874-2e6794fd6e62] X-Ms-Ratelimit-Remaining-Resource:[Microsoft.Compute/LowCostGet3Min;14949,Microsoft.Compute/LowCostGet30Min;119413] X-Ms-Ratelimit-Remaining-Subscription-Reads:[11883] X-Ms-Request-Id:[935efe8e-5da5-4c20-abeb-9e370a8ecf38] X-Ms-Routing-Request-Id:[WESTUS2:20230129T042745Z:1a10f60f-4a49-4b58-a874-2e6794fd6e62] X-Ms-Served-By:[43c1a32f-0327-4d8b-a820-a28f01b4195d_133148057271908261]] {0xc000809380} -1 [chunked] false true map[] 0xc000662700 0xc0005dae70})} %!s(*string=<nil>) %!s(*[]string=<nil>) %!s(*compute.DiskSku=&{StandardSSD_ZRS 0xc0000f00d0}) %!s(*[]string=<nil>) %!s(*compute.ExtendedLocation=<nil>) %!s(*compute.DiskProperties=&{0xc000a991e8 <nil> <nil> 0xc00092cde0 0xc000719200 0xc000719280 0xc0000f0250 <nil> 0xc0000f0210 0xc000719208 0xc000719218 <nil> <nil> Unattached 0xc000a991d0 <nil> <nil> AllowAll <nil> <nil> <nil> <nil> <nil> <nil> <nil> Enabled }) %!s(*string=0xc0000f02e0) %!s(*string=0xc0000f02d0) %!s(*string=0xc0009d3f10) %!s(*string=0xc0009d3f30) map[k8s-azure-created-by:%!s(*string=0xc0009d3f40) kubernetes.io-created-for-pv-name:%!s(*string=0xc0009d3f80) kubernetes.io-created-for-pvc-name:%!s(*string=0xc0009d3fa0) kubernetes.io-created-for-pvc-namespace:%!s(*string=0xc0009d3fe0)]} I0129 04:27:46.541643 1 azure_controller_common.go:261] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-27531535-f336-44c6-8b81-6706d20c1395 lun 0 to node k8s-agentpool-14513215-vmss000001, diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-27531535-f336-44c6-8b81-6706d20c1395:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-27531535-f336-44c6-8b81-6706d20c1395 false 0})] I0129 04:27:46.541741 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-l6t2jkxe): vm(k8s-agentpool-14513215-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-27531535-f336-44c6-8b81-6706d20c1395:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-27531535-f336-44c6-8b81-6706d20c1395 false 0})]) I0129 04:27:46.745020 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-l6t2jkxe): vm(k8s-agentpool-14513215-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-27531535-f336-44c6-8b81-6706d20c1395:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-27531535-f336-44c6-8b81-6706d20c1395 false 0})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I0129 04:27:56.888915 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-l6t2jkxe, k8s-agentpool-14513215-vmss, k8s-agentpool-14513215-vmss000001) successfully I0129 04:27:56.888956 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool-14513215-vmss, kubetest-l6t2jkxe, k8s-agentpool-14513215-vmss000001) for cacheKey(kubetest-l6t2jkxe/k8s-agentpool-14513215-vmss) updated successfully I0129 04:27:56.888976 1 controllerserver.go:413] Attach operation successful: volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-27531535-f336-44c6-8b81-6706d20c1395 attached to node k8s-agentpool-14513215-vmss000001. I0129 04:27:56.888990 1 controllerserver.go:433] attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-27531535-f336-44c6-8b81-6706d20c1395 to node k8s-agentpool-14513215-vmss000001 successfully I0129 04:27:56.889036 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=11.347931963 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-l6t2jkxe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-27531535-f336-44c6-8b81-6706d20c1395" node="k8s-agentpool-14513215-vmss000001" result_code="succeeded" I0129 04:27:56.889058 1 utils.go:84] GRPC response: {"publish_context":{"LUN":"0"}} ... skipping 62 lines ... I0129 04:30:11.603185 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1214 I0129 04:30:11.603592 1 controllerserver.go:383] GetDiskLun returned: cannot find Lun for disk pvc-790bf2bc-e463-458e-8305-f6f1f1e06e34. Initiating attaching volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-790bf2bc-e463-458e-8305-f6f1f1e06e34 to node k8s-agentpool-14513215-vmss000001 (vmState Succeeded). I0129 04:30:11.603642 1 controllerserver.go:408] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-790bf2bc-e463-458e-8305-f6f1f1e06e34 to node k8s-agentpool-14513215-vmss000001 I0129 04:30:11.603667 1 azure_controller_common.go:247] waiting for more requests on node k8s-agentpool-14513215-vmss000001, current disk attach: &{{%!s(*http.Response=&{200 OK 200 HTTP/1.1 1 1 map[Cache-Control:[no-cache] Content-Type:[application/json; charset=utf-8] Date:[Sun, 29 Jan 2023 04:30:11 GMT] Expires:[-1] Pragma:[no-cache] Server:[Microsoft-HTTPAPI/2.0 Microsoft-HTTPAPI/2.0] Strict-Transport-Security:[max-age=31536000; includeSubDomains] Vary:[Accept-Encoding] X-Content-Type-Options:[nosniff] X-Ms-Correlation-Request-Id:[4025aef8-d5c2-4d78-86aa-2ec39ffdde14] X-Ms-Ratelimit-Remaining-Resource:[Microsoft.Compute/LowCostGet3Min;14941,Microsoft.Compute/LowCostGet30Min;119427] X-Ms-Ratelimit-Remaining-Subscription-Reads:[11858] X-Ms-Request-Id:[5f5a0d79-e837-449a-b912-9f36566d2ec4] X-Ms-Routing-Request-Id:[WESTUS2:20230129T043011Z:4025aef8-d5c2-4d78-86aa-2ec39ffdde14] X-Ms-Served-By:[43c1a32f-0327-4d8b-a820-a28f01b4195d_133148057271908261]] {0xc0009d5f20} -1 [chunked] false true map[] 0xc0005f6900 0xc0005dae70})} %!s(*string=<nil>) %!s(*[]string=<nil>) %!s(*compute.DiskSku=&{StandardSSD_ZRS 0xc000193ba0}) %!s(*[]string=<nil>) %!s(*compute.ExtendedLocation=<nil>) %!s(*compute.DiskProperties=&{0xc000805c80 <nil> <nil> 0xc00092c5a0 0xc000740600 0xc000740660 0xc000193c00 <nil> 0xc000193bf0 0xc000740608 0xc000740618 <nil> <nil> Unattached 0xc000805c68 0xc000740624 <nil> AllowAll <nil> <nil> <nil> <nil> <nil> <nil> <nil> Enabled }) %!s(*string=0xc0001939b0) %!s(*string=0xc000193970) %!s(*string=0xc0001939d0) %!s(*string=0xc0001939f0) map[k8s-azure-created-by:%!s(*string=0xc000193a00) kubernetes.io-created-for-pv-name:%!s(*string=0xc000193a90) kubernetes.io-created-for-pvc-name:%!s(*string=0xc000193b10) kubernetes.io-created-for-pvc-namespace:%!s(*string=0xc000193b70)]} I0129 04:30:11.809392 1 azure_controller_common.go:261] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-790bf2bc-e463-458e-8305-f6f1f1e06e34 lun 0 to node k8s-agentpool-14513215-vmss000000, diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-790bf2bc-e463-458e-8305-f6f1f1e06e34:%!s(*provider.AttachDiskOptions=&{None pvc-790bf2bc-e463-458e-8305-f6f1f1e06e34 false 0})] I0129 04:30:11.809487 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-l6t2jkxe): vm(k8s-agentpool-14513215-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-790bf2bc-e463-458e-8305-f6f1f1e06e34:%!s(*provider.AttachDiskOptions=&{None pvc-790bf2bc-e463-458e-8305-f6f1f1e06e34 false 0})]) I0129 04:30:11.974596 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-l6t2jkxe): vm(k8s-agentpool-14513215-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-790bf2bc-e463-458e-8305-f6f1f1e06e34:%!s(*provider.AttachDiskOptions=&{None pvc-790bf2bc-e463-458e-8305-f6f1f1e06e34 false 0})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I0129 04:30:12.604106 1 azure_controller_common.go:261] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-790bf2bc-e463-458e-8305-f6f1f1e06e34 lun 0 to node k8s-agentpool-14513215-vmss000001, diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-790bf2bc-e463-458e-8305-f6f1f1e06e34:%!s(*provider.AttachDiskOptions=&{None pvc-790bf2bc-e463-458e-8305-f6f1f1e06e34 false 0})] I0129 04:30:12.604213 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-l6t2jkxe): vm(k8s-agentpool-14513215-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-790bf2bc-e463-458e-8305-f6f1f1e06e34:%!s(*provider.AttachDiskOptions=&{None pvc-790bf2bc-e463-458e-8305-f6f1f1e06e34 false 0})]) I0129 04:30:12.767055 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-l6t2jkxe): vm(k8s-agentpool-14513215-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-790bf2bc-e463-458e-8305-f6f1f1e06e34:%!s(*provider.AttachDiskOptions=&{None pvc-790bf2bc-e463-458e-8305-f6f1f1e06e34 false 0})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I0129 04:30:32.164883 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-l6t2jkxe, k8s-agentpool-14513215-vmss, k8s-agentpool-14513215-vmss000000) successfully I0129 04:30:32.164926 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool-14513215-vmss, kubetest-l6t2jkxe, k8s-agentpool-14513215-vmss000000) for cacheKey(kubetest-l6t2jkxe/k8s-agentpool-14513215-vmss) updated successfully I0129 04:30:32.164950 1 controllerserver.go:413] Attach operation successful: volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-790bf2bc-e463-458e-8305-f6f1f1e06e34 attached to node k8s-agentpool-14513215-vmss000000. I0129 04:30:32.164968 1 controllerserver.go:433] attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-790bf2bc-e463-458e-8305-f6f1f1e06e34 to node k8s-agentpool-14513215-vmss000000 successfully I0129 04:30:32.165034 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=21.35608344 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-l6t2jkxe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-790bf2bc-e463-458e-8305-f6f1f1e06e34" node="k8s-agentpool-14513215-vmss000000" result_code="succeeded" I0129 04:30:32.165075 1 utils.go:84] GRPC response: {"publish_context":{"LUN":"0"}} ... skipping 69 lines ... I0129 04:32:01.628075 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1222 I0129 04:32:01.628617 1 controllerserver.go:383] GetDiskLun returned: cannot find Lun for disk pvc-82e0ee7d-996f-49fb-9f3a-5f408c0368e8. Initiating attaching volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-82e0ee7d-996f-49fb-9f3a-5f408c0368e8 to node k8s-agentpool-14513215-vmss000000 (vmState Succeeded). I0129 04:32:01.628657 1 controllerserver.go:408] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-82e0ee7d-996f-49fb-9f3a-5f408c0368e8 to node k8s-agentpool-14513215-vmss000000 I0129 04:32:01.628682 1 azure_controller_common.go:247] waiting for more requests on node k8s-agentpool-14513215-vmss000000, current disk attach: &{{%!s(*http.Response=&{200 OK 200 HTTP/1.1 1 1 map[Cache-Control:[no-cache] Content-Type:[application/json; charset=utf-8] Date:[Sun, 29 Jan 2023 04:32:00 GMT] Expires:[-1] Pragma:[no-cache] Server:[Microsoft-HTTPAPI/2.0 Microsoft-HTTPAPI/2.0] Strict-Transport-Security:[max-age=31536000; includeSubDomains] Vary:[Accept-Encoding] X-Content-Type-Options:[nosniff] X-Ms-Correlation-Request-Id:[a72dbdf1-6054-469d-b158-ce6cd4bb8739] X-Ms-Ratelimit-Remaining-Resource:[Microsoft.Compute/LowCostGet3Min;14929,Microsoft.Compute/LowCostGet30Min;119365] X-Ms-Ratelimit-Remaining-Subscription-Reads:[11842] X-Ms-Request-Id:[114b9a76-f15b-4e16-8859-e5a642242db4] X-Ms-Routing-Request-Id:[WESTUS2:20230129T043201Z:a72dbdf1-6054-469d-b158-ce6cd4bb8739] X-Ms-Served-By:[43c1a32f-0327-4d8b-a820-a28f01b4195d_133148057271908261]] {0xc000092d80} -1 [chunked] false true map[] 0xc000a47400 0xc0001df340})} %!s(*string=<nil>) %!s(*[]string=<nil>) %!s(*compute.DiskSku=&{StandardSSD_LRS 0xc0004ce3d0}) %!s(*[]string=&[1]) %!s(*compute.ExtendedLocation=<nil>) %!s(*compute.DiskProperties=&{0xc0008049d8 <nil> <nil> 0xc0006e9260 0xc00004be18 0xc00004beb0 0xc0004ce8e0 <nil> 0xc0004ce8b0 0xc00004be20 0xc00004be30 <nil> <nil> Unattached 0xc000804990 <nil> <nil> AllowAll <nil> <nil> <nil> <nil> <nil> <nil> <nil> Enabled }) %!s(*string=0xc0004ce900) %!s(*string=0xc0004ce8f0) %!s(*string=0xc0004ce910) %!s(*string=0xc0004ce950) map[k8s-azure-created-by:%!s(*string=0xc0004ce960) kubernetes.io-created-for-pv-name:%!s(*string=0xc0004ce980) kubernetes.io-created-for-pvc-name:%!s(*string=0xc0004cea50) kubernetes.io-created-for-pvc-namespace:%!s(*string=0xc0004ceab0)]} I0129 04:32:02.629172 1 azure_controller_common.go:261] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-82e0ee7d-996f-49fb-9f3a-5f408c0368e8 lun 0 to node k8s-agentpool-14513215-vmss000000, diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-82e0ee7d-996f-49fb-9f3a-5f408c0368e8:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-82e0ee7d-996f-49fb-9f3a-5f408c0368e8 false 0})] I0129 04:32:02.629243 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-l6t2jkxe): vm(k8s-agentpool-14513215-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-82e0ee7d-996f-49fb-9f3a-5f408c0368e8:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-82e0ee7d-996f-49fb-9f3a-5f408c0368e8 false 0})]) I0129 04:32:02.812609 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-l6t2jkxe): vm(k8s-agentpool-14513215-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-82e0ee7d-996f-49fb-9f3a-5f408c0368e8:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-82e0ee7d-996f-49fb-9f3a-5f408c0368e8 false 0})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I0129 04:32:17.963206 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-l6t2jkxe, k8s-agentpool-14513215-vmss, k8s-agentpool-14513215-vmss000000) successfully I0129 04:32:17.963244 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool-14513215-vmss, kubetest-l6t2jkxe, k8s-agentpool-14513215-vmss000000) for cacheKey(kubetest-l6t2jkxe/k8s-agentpool-14513215-vmss) updated successfully I0129 04:32:17.963266 1 controllerserver.go:413] Attach operation successful: volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-82e0ee7d-996f-49fb-9f3a-5f408c0368e8 attached to node k8s-agentpool-14513215-vmss000000. I0129 04:32:17.963281 1 controllerserver.go:433] attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-82e0ee7d-996f-49fb-9f3a-5f408c0368e8 to node k8s-agentpool-14513215-vmss000000 successfully I0129 04:32:17.963328 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=16.334730117 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-l6t2jkxe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-82e0ee7d-996f-49fb-9f3a-5f408c0368e8" node="k8s-agentpool-14513215-vmss000000" result_code="succeeded" I0129 04:32:17.963351 1 utils.go:84] GRPC response: {"publish_context":{"LUN":"0"}} ... skipping 43 lines ... I0129 04:33:24.917264 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 12 I0129 04:33:24.917337 1 controllerserver.go:383] GetDiskLun returned: cannot find Lun for disk pvc-a5ef90d8-51ed-4acb-8a0e-276320377ca1. Initiating attaching volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-a5ef90d8-51ed-4acb-8a0e-276320377ca1 to node k8s-agentpool-14513215-vmss000000 (vmState Succeeded). I0129 04:33:24.917387 1 controllerserver.go:408] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-a5ef90d8-51ed-4acb-8a0e-276320377ca1 to node k8s-agentpool-14513215-vmss000000 I0129 04:33:24.917437 1 azure_controller_common.go:247] waiting for more requests on node k8s-agentpool-14513215-vmss000000, current disk attach: &{{%!s(*http.Response=&{200 OK 200 HTTP/1.1 1 1 map[Cache-Control:[no-cache] Content-Type:[application/json; charset=utf-8] Date:[Sun, 29 Jan 2023 04:33:24 GMT] Expires:[-1] Pragma:[no-cache] Server:[Microsoft-HTTPAPI/2.0 Microsoft-HTTPAPI/2.0] Strict-Transport-Security:[max-age=31536000; includeSubDomains] Vary:[Accept-Encoding] X-Content-Type-Options:[nosniff] X-Ms-Correlation-Request-Id:[0ac5223f-67d3-4e01-a34d-06222b23f967] X-Ms-Ratelimit-Remaining-Resource:[Microsoft.Compute/LowCostGet3Min;14898,Microsoft.Compute/LowCostGet30Min;119312] X-Ms-Ratelimit-Remaining-Subscription-Reads:[11823] X-Ms-Request-Id:[08f0b1be-b22c-45aa-a06c-e012bae6f605] X-Ms-Routing-Request-Id:[WESTUS2:20230129T043324Z:0ac5223f-67d3-4e01-a34d-06222b23f967] X-Ms-Served-By:[43c1a32f-0327-4d8b-a820-a28f01b4195d_133148057271908261]] {0xc000aa99b0} -1 [chunked] false true map[] 0xc000ac3000 0xc0001df340})} %!s(*string=<nil>) %!s(*[]string=<nil>) %!s(*compute.DiskSku=&{StandardSSD_LRS 0xc00003b370}) %!s(*[]string=&[1]) %!s(*compute.ExtendedLocation=<nil>) %!s(*compute.DiskProperties=&{0xc00077eb58 <nil> <nil> 0xc0004135c0 0xc000807570 0xc0008075d0 0xc00003b1c0 <nil> 0xc00003b180 0xc000807578 0xc000807588 <nil> <nil> Unattached 0xc00077eb40 <nil> <nil> AllowAll <nil> <nil> <nil> <nil> <nil> <nil> <nil> Enabled }) %!s(*string=0xc00003b240) %!s(*string=0xc00003b200) %!s(*string=0xc00003b280) %!s(*string=0xc00003b2c0) map[k8s-azure-created-by:%!s(*string=0xc00003b2e0) kubernetes.io-created-for-pv-name:%!s(*string=0xc00003b300) kubernetes.io-created-for-pvc-name:%!s(*string=0xc00003b320) kubernetes.io-created-for-pvc-namespace:%!s(*string=0xc00003b340)]} I0129 04:33:25.917690 1 azure_controller_common.go:261] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-a5ef90d8-51ed-4acb-8a0e-276320377ca1 lun 0 to node k8s-agentpool-14513215-vmss000000, diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-a5ef90d8-51ed-4acb-8a0e-276320377ca1:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-a5ef90d8-51ed-4acb-8a0e-276320377ca1 false 0})] I0129 04:33:25.917785 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-l6t2jkxe): vm(k8s-agentpool-14513215-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-a5ef90d8-51ed-4acb-8a0e-276320377ca1:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-a5ef90d8-51ed-4acb-8a0e-276320377ca1 false 0})]) I0129 04:33:26.082907 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-l6t2jkxe): vm(k8s-agentpool-14513215-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-a5ef90d8-51ed-4acb-8a0e-276320377ca1:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-a5ef90d8-51ed-4acb-8a0e-276320377ca1 false 0})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I0129 04:33:41.246810 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-l6t2jkxe, k8s-agentpool-14513215-vmss, k8s-agentpool-14513215-vmss000000) successfully I0129 04:33:41.246848 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool-14513215-vmss, kubetest-l6t2jkxe, k8s-agentpool-14513215-vmss000000) for cacheKey(kubetest-l6t2jkxe/k8s-agentpool-14513215-vmss) updated successfully I0129 04:33:41.246869 1 controllerserver.go:413] Attach operation successful: volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-a5ef90d8-51ed-4acb-8a0e-276320377ca1 attached to node k8s-agentpool-14513215-vmss000000. I0129 04:33:41.246883 1 controllerserver.go:433] attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-a5ef90d8-51ed-4acb-8a0e-276320377ca1 to node k8s-agentpool-14513215-vmss000000 successfully I0129 04:33:41.246925 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=16.357226372 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-l6t2jkxe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-a5ef90d8-51ed-4acb-8a0e-276320377ca1" node="k8s-agentpool-14513215-vmss000000" result_code="succeeded" I0129 04:33:41.246946 1 utils.go:84] GRPC response: {"publish_context":{"LUN":"0"}} ... skipping 11 lines ... I0129 04:34:05.673664 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1247 I0129 04:34:05.674101 1 controllerserver.go:383] GetDiskLun returned: cannot find Lun for disk pvc-b66d9e08-0eb7-42a8-bd71-c9cee29f4066. Initiating attaching volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-b66d9e08-0eb7-42a8-bd71-c9cee29f4066 to node k8s-agentpool-14513215-vmss000001 (vmState Succeeded). I0129 04:34:05.674133 1 controllerserver.go:408] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-b66d9e08-0eb7-42a8-bd71-c9cee29f4066 to node k8s-agentpool-14513215-vmss000001 I0129 04:34:05.674179 1 azure_controller_common.go:247] waiting for more requests on node k8s-agentpool-14513215-vmss000001, current disk attach: &{{%!s(*http.Response=&{200 OK 200 HTTP/1.1 1 1 map[Cache-Control:[no-cache] Content-Type:[application/json; charset=utf-8] Date:[Sun, 29 Jan 2023 04:34:04 GMT] Expires:[-1] Pragma:[no-cache] Server:[Microsoft-HTTPAPI/2.0 Microsoft-HTTPAPI/2.0] Strict-Transport-Security:[max-age=31536000; includeSubDomains] Vary:[Accept-Encoding] X-Content-Type-Options:[nosniff] X-Ms-Correlation-Request-Id:[905a280e-828e-452b-891e-6f9f81532722] X-Ms-Ratelimit-Remaining-Resource:[Microsoft.Compute/LowCostGet3Min;14921,Microsoft.Compute/LowCostGet30Min;119294] X-Ms-Ratelimit-Remaining-Subscription-Reads:[11813] X-Ms-Request-Id:[09bbdde2-6f4e-4cfc-82cb-d3966bc8f7fd] X-Ms-Routing-Request-Id:[WESTUS2:20230129T043405Z:905a280e-828e-452b-891e-6f9f81532722] X-Ms-Served-By:[43c1a32f-0327-4d8b-a820-a28f01b4195d_133148057271908261]] {0xc0008091a0} -1 [chunked] false true map[] 0xc000a46200 0xc0001df340})} %!s(*string=<nil>) %!s(*[]string=<nil>) %!s(*compute.DiskSku=&{StandardSSD_LRS 0xc0000ef0f0}) %!s(*[]string=&[2]) %!s(*compute.ExtendedLocation=<nil>) %!s(*compute.DiskProperties=&{0xc00077fe48 <nil> <nil> 0xc0008be3c0 0xc0005c1060 0xc0005c10c0 0xc0000ef1d0 <nil> 0xc0000ef1c0 0xc0005c1068 0xc0005c1078 <nil> <nil> Unattached 0xc00077fe18 <nil> <nil> AllowAll <nil> <nil> <nil> <nil> <nil> <nil> <nil> Enabled }) %!s(*string=0xc0000ef230) %!s(*string=0xc0000ef200) %!s(*string=0xc0000eef00) %!s(*string=0xc0000eef40) map[k8s-azure-created-by:%!s(*string=0xc0000eef80) kubernetes.io-created-for-pv-name:%!s(*string=0xc0000eefd0) kubernetes.io-created-for-pvc-name:%!s(*string=0xc0000ef030) kubernetes.io-created-for-pvc-namespace:%!s(*string=0xc0000ef060)]} I0129 04:34:06.674503 1 azure_controller_common.go:261] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-b66d9e08-0eb7-42a8-bd71-c9cee29f4066 lun 0 to node k8s-agentpool-14513215-vmss000001, diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-b66d9e08-0eb7-42a8-bd71-c9cee29f4066:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-b66d9e08-0eb7-42a8-bd71-c9cee29f4066 false 0})] I0129 04:34:06.674620 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-l6t2jkxe): vm(k8s-agentpool-14513215-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-b66d9e08-0eb7-42a8-bd71-c9cee29f4066:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-b66d9e08-0eb7-42a8-bd71-c9cee29f4066 false 0})]) I0129 04:34:06.840761 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-l6t2jkxe): vm(k8s-agentpool-14513215-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-b66d9e08-0eb7-42a8-bd71-c9cee29f4066:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-b66d9e08-0eb7-42a8-bd71-c9cee29f4066 false 0})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I0129 04:34:16.975415 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-l6t2jkxe, k8s-agentpool-14513215-vmss, k8s-agentpool-14513215-vmss000001) successfully I0129 04:34:16.975461 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool-14513215-vmss, kubetest-l6t2jkxe, k8s-agentpool-14513215-vmss000001) for cacheKey(kubetest-l6t2jkxe/k8s-agentpool-14513215-vmss) updated successfully I0129 04:34:16.975626 1 controllerserver.go:413] Attach operation successful: volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-b66d9e08-0eb7-42a8-bd71-c9cee29f4066 attached to node k8s-agentpool-14513215-vmss000001. I0129 04:34:16.975647 1 controllerserver.go:433] attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-b66d9e08-0eb7-42a8-bd71-c9cee29f4066 to node k8s-agentpool-14513215-vmss000001 successfully I0129 04:34:16.975695 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=11.301583725 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-l6t2jkxe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-b66d9e08-0eb7-42a8-bd71-c9cee29f4066" node="k8s-agentpool-14513215-vmss000001" result_code="succeeded" I0129 04:34:16.975713 1 utils.go:84] GRPC response: {"publish_context":{"LUN":"0"}} ... skipping 11 lines ... I0129 04:34:33.896588 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1256 I0129 04:34:33.897057 1 controllerserver.go:383] GetDiskLun returned: cannot find Lun for disk pvc-a65e4d1f-e1f3-4c58-b011-e9e150ef6d21. Initiating attaching volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-a65e4d1f-e1f3-4c58-b011-e9e150ef6d21 to node k8s-agentpool-14513215-vmss000000 (vmState Succeeded). I0129 04:34:33.897096 1 controllerserver.go:408] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-a65e4d1f-e1f3-4c58-b011-e9e150ef6d21 to node k8s-agentpool-14513215-vmss000000 I0129 04:34:33.902556 1 azure_controller_common.go:247] waiting for more requests on node k8s-agentpool-14513215-vmss000000, current disk attach: &{{%!s(*http.Response=&{200 OK 200 HTTP/1.1 1 1 map[Cache-Control:[no-cache] Content-Type:[application/json; charset=utf-8] Date:[Sun, 29 Jan 2023 04:34:33 GMT] Expires:[-1] Pragma:[no-cache] Server:[Microsoft-HTTPAPI/2.0 Microsoft-HTTPAPI/2.0] Strict-Transport-Security:[max-age=31536000; includeSubDomains] Vary:[Accept-Encoding] X-Content-Type-Options:[nosniff] X-Ms-Correlation-Request-Id:[3d292074-0c93-4fdc-81d0-0f506d0196c7] X-Ms-Ratelimit-Remaining-Resource:[Microsoft.Compute/LowCostGet3Min;14914,Microsoft.Compute/LowCostGet30Min;119282] X-Ms-Ratelimit-Remaining-Subscription-Reads:[11805] X-Ms-Request-Id:[8052af1c-e8a0-4941-bf24-be0492651ecb] X-Ms-Routing-Request-Id:[WESTUS2:20230129T043433Z:3d292074-0c93-4fdc-81d0-0f506d0196c7] X-Ms-Served-By:[43c1a32f-0327-4d8b-a820-a28f01b4195d_133148057271908261]] {0xc000b6b3e0} -1 [chunked] false true map[] 0xc000a47300 0xc0001df340})} %!s(*string=<nil>) %!s(*[]string=<nil>) %!s(*compute.DiskSku=&{StandardSSD_LRS 0xc0004ed550}) %!s(*[]string=&[1]) %!s(*compute.ExtendedLocation=<nil>) %!s(*compute.DiskProperties=&{0xc0002017a0 <nil> <nil> 0xc0008bfec0 0xc0005c1b90 0xc0005c1c00 0xc0004ed710 <nil> 0xc0004ed6d0 0xc0005c1b98 0xc0005c1ba8 <nil> <nil> Unattached 0xc000201788 <nil> <nil> AllowAll <nil> <nil> <nil> <nil> <nil> <nil> <nil> Enabled }) %!s(*string=0xc0004ed2d0) %!s(*string=0xc0004ed2a0) %!s(*string=0xc0004ed310) %!s(*string=0xc0004ed330) map[k8s-azure-created-by:%!s(*string=0xc0004ed340) kubernetes.io-created-for-pv-name:%!s(*string=0xc0004ed3b0) kubernetes.io-created-for-pvc-name:%!s(*string=0xc0004ed410) kubernetes.io-created-for-pvc-namespace:%!s(*string=0xc0004ed490)]} I0129 04:34:34.903077 1 azure_controller_common.go:261] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-a65e4d1f-e1f3-4c58-b011-e9e150ef6d21 lun 1 to node k8s-agentpool-14513215-vmss000000, diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-a65e4d1f-e1f3-4c58-b011-e9e150ef6d21:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-a65e4d1f-e1f3-4c58-b011-e9e150ef6d21 false 1})] I0129 04:34:34.903177 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-l6t2jkxe): vm(k8s-agentpool-14513215-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-a65e4d1f-e1f3-4c58-b011-e9e150ef6d21:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-a65e4d1f-e1f3-4c58-b011-e9e150ef6d21 false 1})]) I0129 04:34:35.080967 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-l6t2jkxe): vm(k8s-agentpool-14513215-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-l6t2jkxe/providers/microsoft.compute/disks/pvc-a65e4d1f-e1f3-4c58-b011-e9e150ef6d21:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-a65e4d1f-e1f3-4c58-b011-e9e150ef6d21 false 1})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I0129 04:34:45.201908 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-l6t2jkxe, k8s-agentpool-14513215-vmss, k8s-agentpool-14513215-vmss000000) successfully I0129 04:34:45.201962 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool-14513215-vmss, kubetest-l6t2jkxe, k8s-agentpool-14513215-vmss000000) for cacheKey(kubetest-l6t2jkxe/k8s-agentpool-14513215-vmss) updated successfully I0129 04:34:45.202003 1 controllerserver.go:413] Attach operation successful: volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-a65e4d1f-e1f3-4c58-b011-e9e150ef6d21 attached to node k8s-agentpool-14513215-vmss000000. I0129 04:34:45.202037 1 controllerserver.go:433] attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-a65e4d1f-e1f3-4c58-b011-e9e150ef6d21 to node k8s-agentpool-14513215-vmss000000 successfully I0129 04:34:45.202082 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=11.305038656 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-l6t2jkxe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-a65e4d1f-e1f3-4c58-b011-e9e150ef6d21" node="k8s-agentpool-14513215-vmss000000" result_code="succeeded" I0129 04:34:45.202243 1 utils.go:84] GRPC response: {"publish_context":{"LUN":"1"}} ... skipping 13 lines ... Platform: linux/amd64 Topology Key: topology.disk.csi.azure.com/zone Streaming logs below: I0129 03:44:01.978439 1 azuredisk.go:175] driver userAgent: disk.csi.azure.com/v1.27.0-d5369cddb1fe3d1d78e5a517b281425883f14ad4 e2e-test I0129 03:44:01.979061 1 azure_disk_utils.go:162] reading cloud config from secret kube-system/azure-cloud-provider I0129 03:44:02.006280 1 azure_disk_utils.go:169] InitializeCloudFromSecret: failed to get cloud config from secret kube-system/azure-cloud-provider: failed to get secret kube-system/azure-cloud-provider: secrets "azure-cloud-provider" not found I0129 03:44:02.006307 1 azure_disk_utils.go:174] could not read cloud config from secret kube-system/azure-cloud-provider I0129 03:44:02.006318 1 azure_disk_utils.go:184] use default AZURE_CREDENTIAL_FILE env var: /etc/kubernetes/azure.json I0129 03:44:02.006345 1 azure_disk_utils.go:192] read cloud config from file: /etc/kubernetes/azure.json successfully I0129 03:44:02.007207 1 azure_auth.go:253] Using AzurePublicCloud environment I0129 03:44:02.007262 1 azure_auth.go:138] azure: using client_id+client_secret to retrieve access token I0129 03:44:02.007297 1 azure.go:776] Azure cloudprovider using try backoff: retries=6, exponent=1.500000, duration=5, jitter=1.000000 ... skipping 188 lines ... I0129 03:49:05.659904 1 mount_linux.go:567] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0]) I0129 03:49:05.677424 1 mount_linux.go:570] Output: "" I0129 03:49:05.677483 1 mount_linux.go:529] Disk "/dev/disk/azure/scsi1/lun0" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/disk/azure/scsi1/lun0] I0129 03:49:06.193118 1 mount_linux.go:539] Disk successfully formatted (mkfs): ext4 - /dev/disk/azure/scsi1/lun0 /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6423556c-e23b-4d33-b659-6d4fe7fc8557/globalmount I0129 03:49:06.193523 1 mount_linux.go:557] Attempting to mount disk /dev/disk/azure/scsi1/lun0 in ext4 format at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6423556c-e23b-4d33-b659-6d4fe7fc8557/globalmount I0129 03:49:06.193597 1 mount_linux.go:220] Mounting cmd (mount) with arguments (-t ext4 -o invalid,mount,options,defaults /dev/disk/azure/scsi1/lun0 /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6423556c-e23b-4d33-b659-6d4fe7fc8557/globalmount) E0129 03:49:06.213025 1 mount_linux.go:232] Mount failed: exit status 32 Mounting command: mount Mounting arguments: -t ext4 -o invalid,mount,options,defaults /dev/disk/azure/scsi1/lun0 /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6423556c-e23b-4d33-b659-6d4fe7fc8557/globalmount Output: mount: /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6423556c-e23b-4d33-b659-6d4fe7fc8557/globalmount: wrong fs type, bad option, bad superblock on /dev/sdc, missing codepage or helper program, or other error. E0129 03:49:06.213527 1 utils.go:82] GRPC error: rpc error: code = Internal desc = could not format /dev/disk/azure/scsi1/lun0(lun: 0), and mount it at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6423556c-e23b-4d33-b659-6d4fe7fc8557/globalmount, failed with mount failed: exit status 32 Mounting command: mount Mounting arguments: -t ext4 -o invalid,mount,options,defaults /dev/disk/azure/scsi1/lun0 /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6423556c-e23b-4d33-b659-6d4fe7fc8557/globalmount Output: mount: /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6423556c-e23b-4d33-b659-6d4fe7fc8557/globalmount: wrong fs type, bad option, bad superblock on /dev/sdc, missing codepage or helper program, or other error. I0129 03:49:06.828546 1 utils.go:77] GRPC call: /csi.v1.Node/NodeStageVolume I0129 03:49:06.828581 1 utils.go:78] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6423556c-e23b-4d33-b659-6d4fe7fc8557/globalmount","volume_capability":{"AccessType":{"Mount":{"mount_flags":["invalid","mount","options"]}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-6423556c-e23b-4d33-b659-6d4fe7fc8557","csi.storage.k8s.io/pvc/name":"pvc-mcr8s","csi.storage.k8s.io/pvc/namespace":"azuredisk-5466","requestedsizegib":"10","skuName":"StandardSSD_ZRS","storage.kubernetes.io/csiProvisionerIdentity":"1674963846395-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-6423556c-e23b-4d33-b659-6d4fe7fc8557"} I0129 03:49:08.623675 1 azure_common_linux.go:185] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/ I0129 03:49:08.623722 1 nodeserver.go:116] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType StandardSSD_ZRS I0129 03:49:08.624139 1 nodeserver.go:157] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6423556c-e23b-4d33-b659-6d4fe7fc8557/globalmount with mount options([invalid mount options]) I0129 03:49:08.624163 1 mount_linux.go:567] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0]) I0129 03:49:08.632885 1 mount_linux.go:570] Output: "DEVNAME=/dev/disk/azure/scsi1/lun0\nTYPE=ext4\n" I0129 03:49:08.632907 1 mount_linux.go:453] Checking for issues with fsck on disk: /dev/disk/azure/scsi1/lun0 I0129 03:49:08.650306 1 mount_linux.go:557] Attempting to mount disk /dev/disk/azure/scsi1/lun0 in ext4 format at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6423556c-e23b-4d33-b659-6d4fe7fc8557/globalmount I0129 03:49:08.650342 1 mount_linux.go:220] Mounting cmd (mount) with arguments (-t ext4 -o invalid,mount,options,defaults /dev/disk/azure/scsi1/lun0 /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6423556c-e23b-4d33-b659-6d4fe7fc8557/globalmount) E0129 03:49:08.668977 1 mount_linux.go:232] Mount failed: exit status 32 Mounting command: mount Mounting arguments: -t ext4 -o invalid,mount,options,defaults /dev/disk/azure/scsi1/lun0 /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6423556c-e23b-4d33-b659-6d4fe7fc8557/globalmount Output: mount: /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6423556c-e23b-4d33-b659-6d4fe7fc8557/globalmount: wrong fs type, bad option, bad superblock on /dev/sdc, missing codepage or helper program, or other error. E0129 03:49:08.669172 1 utils.go:82] GRPC error: rpc error: code = Internal desc = could not format /dev/disk/azure/scsi1/lun0(lun: 0), and mount it at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6423556c-e23b-4d33-b659-6d4fe7fc8557/globalmount, failed with mount failed: exit status 32 Mounting command: mount Mounting arguments: -t ext4 -o invalid,mount,options,defaults /dev/disk/azure/scsi1/lun0 /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6423556c-e23b-4d33-b659-6d4fe7fc8557/globalmount Output: mount: /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6423556c-e23b-4d33-b659-6d4fe7fc8557/globalmount: wrong fs type, bad option, bad superblock on /dev/sdc, missing codepage or helper program, or other error. I0129 03:49:09.687560 1 utils.go:77] GRPC call: /csi.v1.Node/NodeStageVolume I0129 03:49:09.687589 1 utils.go:78] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6423556c-e23b-4d33-b659-6d4fe7fc8557/globalmount","volume_capability":{"AccessType":{"Mount":{"mount_flags":["invalid","mount","options"]}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-6423556c-e23b-4d33-b659-6d4fe7fc8557","csi.storage.k8s.io/pvc/name":"pvc-mcr8s","csi.storage.k8s.io/pvc/namespace":"azuredisk-5466","requestedsizegib":"10","skuName":"StandardSSD_ZRS","storage.kubernetes.io/csiProvisionerIdentity":"1674963846395-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-6423556c-e23b-4d33-b659-6d4fe7fc8557"} I0129 03:49:11.510831 1 azure_common_linux.go:185] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/ I0129 03:49:11.510871 1 nodeserver.go:116] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType StandardSSD_ZRS I0129 03:49:11.511333 1 nodeserver.go:157] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6423556c-e23b-4d33-b659-6d4fe7fc8557/globalmount with mount options([invalid mount options]) I0129 03:49:11.511360 1 mount_linux.go:567] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0]) I0129 03:49:11.521624 1 mount_linux.go:570] Output: "DEVNAME=/dev/disk/azure/scsi1/lun0\nTYPE=ext4\n" I0129 03:49:11.521645 1 mount_linux.go:453] Checking for issues with fsck on disk: /dev/disk/azure/scsi1/lun0 I0129 03:49:11.537488 1 mount_linux.go:557] Attempting to mount disk /dev/disk/azure/scsi1/lun0 in ext4 format at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6423556c-e23b-4d33-b659-6d4fe7fc8557/globalmount I0129 03:49:11.537520 1 mount_linux.go:220] Mounting cmd (mount) with arguments (-t ext4 -o invalid,mount,options,defaults /dev/disk/azure/scsi1/lun0 /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6423556c-e23b-4d33-b659-6d4fe7fc8557/globalmount) E0129 03:49:11.555612 1 mount_linux.go:232] Mount failed: exit status 32 Mounting command: mount Mounting arguments: -t ext4 -o invalid,mount,options,defaults /dev/disk/azure/scsi1/lun0 /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6423556c-e23b-4d33-b659-6d4fe7fc8557/globalmount Output: mount: /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6423556c-e23b-4d33-b659-6d4fe7fc8557/globalmount: wrong fs type, bad option, bad superblock on /dev/sdc, missing codepage or helper program, or other error. E0129 03:49:11.555662 1 utils.go:82] GRPC error: rpc error: code = Internal desc = could not format /dev/disk/azure/scsi1/lun0(lun: 0), and mount it at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6423556c-e23b-4d33-b659-6d4fe7fc8557/globalmount, failed with mount failed: exit status 32 Mounting command: mount Mounting arguments: -t ext4 -o invalid,mount,options,defaults /dev/disk/azure/scsi1/lun0 /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6423556c-e23b-4d33-b659-6d4fe7fc8557/globalmount Output: mount: /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6423556c-e23b-4d33-b659-6d4fe7fc8557/globalmount: wrong fs type, bad option, bad superblock on /dev/sdc, missing codepage or helper program, or other error. I0129 03:50:13.294014 1 utils.go:77] GRPC call: /csi.v1.Node/NodeStageVolume I0129 03:50:13.294057 1 utils.go:78] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-840f6b42-69f6-4ce0-ad7e-3b7494e2b2db","volume_capability":{"AccessType":{"Block":{}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-840f6b42-69f6-4ce0-ad7e-3b7494e2b2db","csi.storage.k8s.io/pvc/name":"pvc-s25c8","csi.storage.k8s.io/pvc/namespace":"azuredisk-2790","requestedsizegib":"10","skuName":"StandardSSD_ZRS","storage.kubernetes.io/csiProvisionerIdentity":"1674963846395-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-840f6b42-69f6-4ce0-ad7e-3b7494e2b2db"} I0129 03:50:15.094868 1 azure_common_linux.go:185] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/ I0129 03:50:15.094913 1 nodeserver.go:116] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType StandardSSD_ZRS I0129 03:50:15.094930 1 utils.go:84] GRPC response: {} I0129 03:50:15.101649 1 utils.go:77] GRPC call: /csi.v1.Node/NodePublishVolume ... skipping 16 lines ... I0129 03:50:20.691130 1 utils.go:84] GRPC response: {} I0129 03:50:20.718593 1 utils.go:77] GRPC call: /csi.v1.Node/NodeUnstageVolume I0129 03:50:20.718617 1 utils.go:78] GRPC request: {"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-840f6b42-69f6-4ce0-ad7e-3b7494e2b2db","volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-840f6b42-69f6-4ce0-ad7e-3b7494e2b2db"} I0129 03:50:20.718687 1 nodeserver.go:201] NodeUnstageVolume: unmounting /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-840f6b42-69f6-4ce0-ad7e-3b7494e2b2db I0129 03:50:20.718711 1 mount_helper_common.go:93] unmounting "/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-840f6b42-69f6-4ce0-ad7e-3b7494e2b2db" (corruptedMount: false, mounterCanSkipMountPointChecks: true) I0129 03:50:20.718727 1 mount_linux.go:362] Unmounting /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-840f6b42-69f6-4ce0-ad7e-3b7494e2b2db I0129 03:50:20.721421 1 mount_linux.go:375] ignoring 'not mounted' error for /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-840f6b42-69f6-4ce0-ad7e-3b7494e2b2db I0129 03:50:20.721442 1 mount_helper_common.go:150] Warning: deleting path "/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-840f6b42-69f6-4ce0-ad7e-3b7494e2b2db" I0129 03:50:20.721542 1 nodeserver.go:206] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-840f6b42-69f6-4ce0-ad7e-3b7494e2b2db successfully I0129 03:50:20.721581 1 utils.go:84] GRPC response: {} I0129 03:51:52.850344 1 utils.go:77] GRPC call: /csi.v1.Node/NodeStageVolume I0129 03:51:52.850370 1 utils.go:78] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-cad3d7b2-3e89-4b27-bdb3-9953079be191/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-cad3d7b2-3e89-4b27-bdb3-9953079be191","csi.storage.k8s.io/pvc/name":"pvc-59wkt","csi.storage.k8s.io/pvc/namespace":"azuredisk-5356","requestedsizegib":"10","resourceGroup":"azuredisk-csi-driver-test-3856d907-9f88-11ed-987b-d279894c5862","skuName":"StandardSSD_ZRS","storage.kubernetes.io/csiProvisionerIdentity":"1674963846395-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/azuredisk-csi-driver-test-3856d907-9f88-11ed-987b-d279894c5862/providers/Microsoft.Compute/disks/pvc-cad3d7b2-3e89-4b27-bdb3-9953079be191"} I0129 03:51:54.580011 1 azure_common_linux.go:185] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/ ... skipping 648 lines ... I0129 04:10:13.429648 1 utils.go:84] GRPC response: {} I0129 04:10:13.438254 1 utils.go:77] GRPC call: /csi.v1.Node/NodeUnstageVolume I0129 04:10:13.438272 1 utils.go:78] GRPC request: {"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-66358d3a-0797-40fb-a0fd-78855f7c0503","volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-66358d3a-0797-40fb-a0fd-78855f7c0503"} I0129 04:10:13.438348 1 nodeserver.go:201] NodeUnstageVolume: unmounting /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-66358d3a-0797-40fb-a0fd-78855f7c0503 I0129 04:10:13.438368 1 mount_helper_common.go:93] unmounting "/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-66358d3a-0797-40fb-a0fd-78855f7c0503" (corruptedMount: false, mounterCanSkipMountPointChecks: true) I0129 04:10:13.438380 1 mount_linux.go:362] Unmounting /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-66358d3a-0797-40fb-a0fd-78855f7c0503 I0129 04:10:13.440870 1 mount_linux.go:375] ignoring 'not mounted' error for /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-66358d3a-0797-40fb-a0fd-78855f7c0503 I0129 04:10:13.440882 1 mount_helper_common.go:150] Warning: deleting path "/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-66358d3a-0797-40fb-a0fd-78855f7c0503" I0129 04:10:13.440957 1 nodeserver.go:206] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-66358d3a-0797-40fb-a0fd-78855f7c0503 successfully I0129 04:10:13.440971 1 utils.go:84] GRPC response: {} I0129 04:11:18.890047 1 utils.go:77] GRPC call: /csi.v1.Node/NodeStageVolume I0129 04:11:18.890073 1 utils.go:78] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-de8db5a3-f4eb-4343-a916-0b26fffd27e7/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-de8db5a3-f4eb-4343-a916-0b26fffd27e7","csi.storage.k8s.io/pvc/name":"pvc-2tzrw","csi.storage.k8s.io/pvc/namespace":"azuredisk-8582","requestedsizegib":"10","skuName":"StandardSSD_ZRS","storage.kubernetes.io/csiProvisionerIdentity":"1674963846395-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-de8db5a3-f4eb-4343-a916-0b26fffd27e7"} I0129 04:11:20.665924 1 azure_common_linux.go:185] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/ ... skipping 497 lines ... I0129 04:31:22.689686 1 utils.go:84] GRPC response: {} I0129 04:31:22.731604 1 utils.go:77] GRPC call: /csi.v1.Node/NodeUnstageVolume I0129 04:31:22.731626 1 utils.go:78] GRPC request: {"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-790bf2bc-e463-458e-8305-f6f1f1e06e34","volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-790bf2bc-e463-458e-8305-f6f1f1e06e34"} I0129 04:31:22.731684 1 nodeserver.go:201] NodeUnstageVolume: unmounting /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-790bf2bc-e463-458e-8305-f6f1f1e06e34 I0129 04:31:22.731717 1 mount_helper_common.go:93] unmounting "/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-790bf2bc-e463-458e-8305-f6f1f1e06e34" (corruptedMount: false, mounterCanSkipMountPointChecks: true) I0129 04:31:22.731731 1 mount_linux.go:362] Unmounting /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-790bf2bc-e463-458e-8305-f6f1f1e06e34 I0129 04:31:22.733509 1 mount_linux.go:375] ignoring 'not mounted' error for /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-790bf2bc-e463-458e-8305-f6f1f1e06e34 I0129 04:31:22.733526 1 mount_helper_common.go:150] Warning: deleting path "/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-790bf2bc-e463-458e-8305-f6f1f1e06e34" I0129 04:31:22.733605 1 nodeserver.go:206] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-790bf2bc-e463-458e-8305-f6f1f1e06e34 successfully I0129 04:31:22.733619 1 utils.go:84] GRPC response: {} I0129 04:32:33.406535 1 utils.go:77] GRPC call: /csi.v1.Node/NodeStageVolume I0129 04:32:33.406562 1 utils.go:78] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-82e0ee7d-996f-49fb-9f3a-5f408c0368e8/globalmount","volume_capability":{"AccessType":{"Mount":{"mount_flags":["barrier=1","acl"]}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-82e0ee7d-996f-49fb-9f3a-5f408c0368e8","csi.storage.k8s.io/pvc/name":"pvc-9cwrt","csi.storage.k8s.io/pvc/namespace":"azuredisk-1092","device-setting/device/queue_depth":"17","device-setting/queue/max_sectors_kb":"211","device-setting/queue/nr_requests":"44","device-setting/queue/read_ahead_kb":"256","device-setting/queue/rotational":"0","device-setting/queue/scheduler":"none","device-setting/queue/wbt_lat_usec":"0","perfProfile":"advanced","requestedsizegib":"10","skuname":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1674963846395-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-82e0ee7d-996f-49fb-9f3a-5f408c0368e8"} I0129 04:32:35.183368 1 azure_common_linux.go:185] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/ ... skipping 105 lines ... Platform: linux/amd64 Topology Key: topology.disk.csi.azure.com/zone Streaming logs below: I0129 03:43:57.253523 1 azuredisk.go:175] driver userAgent: disk.csi.azure.com/v1.27.0-d5369cddb1fe3d1d78e5a517b281425883f14ad4 e2e-test I0129 03:43:57.254036 1 azure_disk_utils.go:162] reading cloud config from secret kube-system/azure-cloud-provider I0129 03:43:57.287509 1 azure_disk_utils.go:169] InitializeCloudFromSecret: failed to get cloud config from secret kube-system/azure-cloud-provider: failed to get secret kube-system/azure-cloud-provider: secrets "azure-cloud-provider" not found I0129 03:43:57.287536 1 azure_disk_utils.go:174] could not read cloud config from secret kube-system/azure-cloud-provider I0129 03:43:57.287546 1 azure_disk_utils.go:184] use default AZURE_CREDENTIAL_FILE env var: /etc/kubernetes/azure.json I0129 03:43:57.287572 1 azure_disk_utils.go:192] read cloud config from file: /etc/kubernetes/azure.json successfully I0129 03:43:57.288444 1 azure_auth.go:253] Using AzurePublicCloud environment I0129 03:43:57.288504 1 azure_auth.go:138] azure: using client_id+client_secret to retrieve access token I0129 03:43:57.288543 1 azure.go:776] Azure cloudprovider using try backoff: retries=6, exponent=1.500000, duration=5, jitter=1.000000 ... skipping 68 lines ... Platform: linux/amd64 Topology Key: topology.disk.csi.azure.com/zone Streaming logs below: I0129 03:44:04.168691 1 azuredisk.go:175] driver userAgent: disk.csi.azure.com/v1.27.0-d5369cddb1fe3d1d78e5a517b281425883f14ad4 e2e-test I0129 03:44:04.169246 1 azure_disk_utils.go:162] reading cloud config from secret kube-system/azure-cloud-provider I0129 03:44:04.199566 1 azure_disk_utils.go:169] InitializeCloudFromSecret: failed to get cloud config from secret kube-system/azure-cloud-provider: failed to get secret kube-system/azure-cloud-provider: secrets "azure-cloud-provider" not found I0129 03:44:04.199594 1 azure_disk_utils.go:174] could not read cloud config from secret kube-system/azure-cloud-provider I0129 03:44:04.199604 1 azure_disk_utils.go:184] use default AZURE_CREDENTIAL_FILE env var: /etc/kubernetes/azure.json I0129 03:44:04.199631 1 azure_disk_utils.go:192] read cloud config from file: /etc/kubernetes/azure.json successfully I0129 03:44:04.200729 1 azure_auth.go:253] Using AzurePublicCloud environment I0129 03:44:04.200778 1 azure_auth.go:138] azure: using client_id+client_secret to retrieve access token I0129 03:44:04.200827 1 azure.go:776] Azure cloudprovider using try backoff: retries=6, exponent=1.500000, duration=5, jitter=1.000000 ... skipping 258 lines ... I0129 04:31:21.904201 1 utils.go:84] GRPC response: {} I0129 04:31:21.921973 1 utils.go:77] GRPC call: /csi.v1.Node/NodeUnstageVolume I0129 04:31:21.922020 1 utils.go:78] GRPC request: {"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-790bf2bc-e463-458e-8305-f6f1f1e06e34","volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-790bf2bc-e463-458e-8305-f6f1f1e06e34"} I0129 04:31:21.922103 1 nodeserver.go:201] NodeUnstageVolume: unmounting /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-790bf2bc-e463-458e-8305-f6f1f1e06e34 I0129 04:31:21.922129 1 mount_helper_common.go:93] unmounting "/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-790bf2bc-e463-458e-8305-f6f1f1e06e34" (corruptedMount: false, mounterCanSkipMountPointChecks: true) I0129 04:31:21.922142 1 mount_linux.go:362] Unmounting /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-790bf2bc-e463-458e-8305-f6f1f1e06e34 I0129 04:31:21.924562 1 mount_linux.go:375] ignoring 'not mounted' error for /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-790bf2bc-e463-458e-8305-f6f1f1e06e34 I0129 04:31:21.924581 1 mount_helper_common.go:150] Warning: deleting path "/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-790bf2bc-e463-458e-8305-f6f1f1e06e34" I0129 04:31:21.924706 1 nodeserver.go:206] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-790bf2bc-e463-458e-8305-f6f1f1e06e34 successfully I0129 04:31:21.924736 1 utils.go:84] GRPC response: {} I0129 04:34:21.484974 1 utils.go:77] GRPC call: /csi.v1.Node/NodeStageVolume I0129 04:34:21.485011 1 utils.go:78] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-b66d9e08-0eb7-42a8-bd71-c9cee29f4066/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-b66d9e08-0eb7-42a8-bd71-c9cee29f4066","csi.storage.k8s.io/pvc/name":"persistent-storage-statefulset-azuredisk-0","csi.storage.k8s.io/pvc/namespace":"default","requestedsizegib":"10","skuName":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1674963846395-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-l6t2jkxe/providers/Microsoft.Compute/disks/pvc-b66d9e08-0eb7-42a8-bd71-c9cee29f4066"} I0129 04:34:23.562833 1 azure_common_linux.go:185] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/ ... skipping 102 lines ... # HELP go_gc_heap_objects_objects Number of objects, live or unswept, occupying heap memory. # TYPE go_gc_heap_objects_objects gauge go_gc_heap_objects_objects 35964 # HELP go_gc_heap_tiny_allocs_objects_total Count of small allocations that are packed together into blocks. These allocations are counted separately from other allocations because each individual allocation is not tracked by the runtime, only their block. Each block is already accounted for in allocs-by-size and frees-by-size. # TYPE go_gc_heap_tiny_allocs_objects_total counter go_gc_heap_tiny_allocs_objects_total 4658 # HELP go_gc_limiter_last_enabled_gc_cycle GC cycle the last time the GC CPU limiter was enabled. This metric is useful for diagnosing the root cause of an out-of-memory error, because the limiter trades memory for CPU time when the GC's CPU time gets too high. This is most likely to occur with use of SetMemoryLimit. The first GC cycle is cycle 1, so a value of 0 indicates that it was never enabled. # TYPE go_gc_limiter_last_enabled_gc_cycle gauge go_gc_limiter_last_enabled_gc_cycle 0 # HELP go_gc_pauses_seconds Distribution individual GC-related stop-the-world pause latencies. # TYPE go_gc_pauses_seconds histogram go_gc_pauses_seconds_bucket{le="9.999999999999999e-10"} 0 go_gc_pauses_seconds_bucket{le="9.999999999999999e-09"} 0 ... skipping 754 lines ... cloudprovider_azure_op_duration_seconds_bucket{request="azuredisk_csi_driver_controller_unpublish_volume",resource_group="kubetest-l6t2jkxe",source="disk.csi.azure.com",subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e",le="300"} 49 cloudprovider_azure_op_duration_seconds_bucket{request="azuredisk_csi_driver_controller_unpublish_volume",resource_group="kubetest-l6t2jkxe",source="disk.csi.azure.com",subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e",le="600"} 49 cloudprovider_azure_op_duration_seconds_bucket{request="azuredisk_csi_driver_controller_unpublish_volume",resource_group="kubetest-l6t2jkxe",source="disk.csi.azure.com",subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e",le="1200"} 49 cloudprovider_azure_op_duration_seconds_bucket{request="azuredisk_csi_driver_controller_unpublish_volume",resource_group="kubetest-l6t2jkxe",source="disk.csi.azure.com",subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e",le="+Inf"} 49 cloudprovider_azure_op_duration_seconds_sum{request="azuredisk_csi_driver_controller_unpublish_volume",resource_group="kubetest-l6t2jkxe",source="disk.csi.azure.com",subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e"} 752.9288070340001 cloudprovider_azure_op_duration_seconds_count{request="azuredisk_csi_driver_controller_unpublish_volume",resource_group="kubetest-l6t2jkxe",source="disk.csi.azure.com",subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e"} 49 # HELP cloudprovider_azure_op_failure_count [ALPHA] Number of failed Azure service operations # TYPE cloudprovider_azure_op_failure_count counter cloudprovider_azure_op_failure_count{request="azuredisk_csi_driver_controller_create_volume",resource_group="kubetest-l6t2jkxe",source="disk.csi.azure.com",subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e"} 1 cloudprovider_azure_op_failure_count{request="azuredisk_csi_driver_controller_delete_volume",resource_group="kubetest-l6t2jkxe",source="disk.csi.azure.com",subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e"} 2 # HELP disabled_metric_total [ALPHA] The count of disabled metrics. # TYPE disabled_metric_total counter disabled_metric_total 0 ... skipping 68 lines ... # HELP go_gc_heap_objects_objects Number of objects, live or unswept, occupying heap memory. # TYPE go_gc_heap_objects_objects gauge go_gc_heap_objects_objects 53048 # HELP go_gc_heap_tiny_allocs_objects_total Count of small allocations that are packed together into blocks. These allocations are counted separately from other allocations because each individual allocation is not tracked by the runtime, only their block. Each block is already accounted for in allocs-by-size and frees-by-size. # TYPE go_gc_heap_tiny_allocs_objects_total counter go_gc_heap_tiny_allocs_objects_total 45592 # HELP go_gc_limiter_last_enabled_gc_cycle GC cycle the last time the GC CPU limiter was enabled. This metric is useful for diagnosing the root cause of an out-of-memory error, because the limiter trades memory for CPU time when the GC's CPU time gets too high. This is most likely to occur with use of SetMemoryLimit. The first GC cycle is cycle 1, so a value of 0 indicates that it was never enabled. # TYPE go_gc_limiter_last_enabled_gc_cycle gauge go_gc_limiter_last_enabled_gc_cycle 0 # HELP go_gc_pauses_seconds Distribution individual GC-related stop-the-world pause latencies. # TYPE go_gc_pauses_seconds histogram go_gc_pauses_seconds_bucket{le="9.999999999999999e-10"} 0 go_gc_pauses_seconds_bucket{le="9.999999999999999e-09"} 0 ... skipping 272 lines ... [AfterSuite] [38;5;243m/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/suite_test.go:165[0m [38;5;243m------------------------------[0m [38;5;9m[1mSummarizing 1 Failure:[0m [38;5;9m[FAIL][0m [0mDynamic Provisioning [38;5;243m[multi-az] [0m[38;5;9m[1m[It] should create a pod, write to its pv, take a volume snapshot, overwrite data in original pv, create another pod from the snapshot, and read unaltered original data from original pv[disk.csi.azure.com][0m[0m [38;5;243m/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/testsuites/testsuites.go:823[0m [38;5;9m[1mRan 26 of 66 Specs in 3684.753 seconds[0m [38;5;9m[1mFAIL![0m -- [38;5;10m[1m25 Passed[0m | [38;5;9m[1m1 Failed[0m | [38;5;11m[1m0 Pending[0m | [38;5;14m[1m40 Skipped[0m [38;5;228mYou're using deprecated Ginkgo functionality:[0m [38;5;228m=============================================[0m [38;5;11mSupport for custom reporters has been removed in V2. Please read the documentation linked to below for Ginkgo's new behavior and for a migration path:[0m [1mLearn more at:[0m [38;5;14m[4mhttps://onsi.github.io/ginkgo/MIGRATING_TO_V2#removed-custom-reporters[0m [38;5;243mTo silence deprecations that can be silenced set the following environment variable:[0m [38;5;243mACK_GINKGO_DEPRECATIONS=2.4.0[0m --- FAIL: TestE2E (3684.75s) FAIL FAIL sigs.k8s.io/azuredisk-csi-driver/test/e2e 3684.820s FAIL make: *** [Makefile:261: e2e-test] Error 1 2023/01/29 04:35:47 process.go:155: Step 'make e2e-test' finished in 1h3m1.073155362s 2023/01/29 04:35:47 aksengine_helpers.go:425: downloading /root/tmp1482758792/log-dump.sh from https://raw.githubusercontent.com/kubernetes-sigs/cloud-provider-azure/master/hack/log-dump/log-dump.sh 2023/01/29 04:35:47 util.go:70: curl https://raw.githubusercontent.com/kubernetes-sigs/cloud-provider-azure/master/hack/log-dump/log-dump.sh 2023/01/29 04:35:47 process.go:153: Running: chmod +x /root/tmp1482758792/log-dump.sh 2023/01/29 04:35:47 process.go:155: Step 'chmod +x /root/tmp1482758792/log-dump.sh' finished in 1.529753ms 2023/01/29 04:35:47 aksengine_helpers.go:425: downloading /root/tmp1482758792/log-dump-daemonset.yaml from https://raw.githubusercontent.com/kubernetes-sigs/cloud-provider-azure/master/hack/log-dump/log-dump-daemonset.yaml ... skipping 63 lines ... ssh key file /root/.ssh/id_rsa does not exist. Exiting. 2023/01/29 04:36:20 process.go:155: Step 'bash -c /root/tmp1482758792/win-ci-logs-collector.sh kubetest-l6t2jkxe.westus2.cloudapp.azure.com /root/tmp1482758792 /root/.ssh/id_rsa' finished in 4.099464ms 2023/01/29 04:36:20 aksengine.go:1141: Deleting resource group: kubetest-l6t2jkxe. 2023/01/29 04:43:26 process.go:96: Saved XML output to /logs/artifacts/junit_runner.xml. 2023/01/29 04:43:26 process.go:153: Running: bash -c . hack/lib/version.sh && KUBE_ROOT=. kube::version::get_version_vars && echo "${KUBE_GIT_VERSION-}" 2023/01/29 04:43:26 process.go:155: Step 'bash -c . hack/lib/version.sh && KUBE_ROOT=. kube::version::get_version_vars && echo "${KUBE_GIT_VERSION-}"' finished in 241.616899ms 2023/01/29 04:43:26 main.go:328: Something went wrong: encountered 1 errors: [error during make e2e-test: exit status 2] + EXIT_VALUE=1 + set +o xtrace Cleaning up after docker in docker. ================================================================================ Cleaning up after docker 1fb9508afe00 ... skipping 4 lines ...