kubernetes-sigs/azuredisk-csi-driver PR #1685 pull-azuredisk-csi-driver-e2e-multi-az #1612378339039776768
Recent runs || View in Spyglass
Recent runs || View in Spyglass
| PR | andyzhangx: [WIP] test |
| Result | FAILURE |
| Tests | 1 failed / 13 succeeded |
| Started | |
| Elapsed | 1h16m |
| Revision | 0f6f26196850f19d8bc2d41d0c07658694c5896f |
| Refs |
1685 |
| job-version | v1.27.0-alpha.0.834+cf7a3c5bbbebf2 |
| kubetest-version | v20230105-c1beb6cf98 |
| revision | v1.27.0-alpha.0.834+cf7a3c5bbbebf2 |
error during make e2e-test: exit status 2
from junit_runner.xml
Filter through log files | View test history on testgrid
kubetest Check APIReachability
kubetest Deferred TearDown
kubetest DumpClusterLogs
kubetest GetDeployer
kubetest IsUp
kubetest Prepare
kubetest TearDown
kubetest TearDown Previous
kubetest Timeout
kubetest Up
kubetest kubectl version
kubetest list nodes
kubetest test setup
... skipping 107 lines ... 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 11345 100 11345 0 0 197k 0 --:--:-- --:--:-- --:--:-- 197k Downloading https://get.helm.sh/helm-v3.10.3-linux-amd64.tar.gz Verifying checksum... Done. Preparing to install helm into /usr/local/bin helm installed into /usr/local/bin/helm docker pull k8sprow.azurecr.io/azuredisk-csi:v1.27.0-24f0924abd1b960083c095e7d894758dec8252c3 || make container-all push-manifest Error response from daemon: manifest for k8sprow.azurecr.io/azuredisk-csi:v1.27.0-24f0924abd1b960083c095e7d894758dec8252c3 not found: manifest unknown: manifest tagged by "v1.27.0-24f0924abd1b960083c095e7d894758dec8252c3" is not found make[2]: Entering directory '/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver' CGO_ENABLED=0 GOOS=windows go build -a -ldflags "-X sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.driverVersion=v1.27.0-24f0924abd1b960083c095e7d894758dec8252c3 -X sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.gitCommit=24f0924abd1b960083c095e7d894758dec8252c3 -X sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.buildDate=2023-01-09T09:26:30Z -extldflags "-static"" -mod vendor -o _output/amd64/azurediskplugin.exe ./pkg/azurediskplugin docker buildx rm container-builder || true ERROR: no builder "container-builder" found docker buildx create --use --name=container-builder container-builder # enable qemu for arm64 build # https://github.com/docker/buildx/issues/464#issuecomment-741507760 docker run --privileged --rm tonistiigi/binfmt --uninstall qemu-aarch64 Unable to find image 'tonistiigi/binfmt:latest' locally ... skipping 1757 lines ... type: string type: object oneOf: - required: ["persistentVolumeClaimName"] - required: ["volumeSnapshotContentName"] volumeSnapshotClassName: description: 'VolumeSnapshotClassName is the name of the VolumeSnapshotClass requested by the VolumeSnapshot. VolumeSnapshotClassName may be left nil to indicate that the default SnapshotClass should be used. A given cluster may have multiple default Volume SnapshotClasses: one default per CSI Driver. If a VolumeSnapshot does not specify a SnapshotClass, VolumeSnapshotSource will be checked to figure out what the associated CSI Driver is, and the default VolumeSnapshotClass associated with that CSI Driver will be used. If more than one VolumeSnapshotClass exist for a given CSI Driver and more than one have been marked as default, CreateSnapshot will fail and generate an event. Empty string is not allowed for this field.' type: string required: - source type: object status: description: status represents the current information of a snapshot. Consumers must verify binding between VolumeSnapshot and VolumeSnapshotContent objects is successful (by validating that both VolumeSnapshot and VolumeSnapshotContent point at each other) before using this object. ... skipping 2 lines ... description: 'boundVolumeSnapshotContentName is the name of the VolumeSnapshotContent object to which this VolumeSnapshot object intends to bind to. If not specified, it indicates that the VolumeSnapshot object has not been successfully bound to a VolumeSnapshotContent object yet. NOTE: To avoid possible security issues, consumers must verify binding between VolumeSnapshot and VolumeSnapshotContent objects is successful (by validating that both VolumeSnapshot and VolumeSnapshotContent point at each other) before using this object.' type: string creationTime: description: creationTime is the timestamp when the point-in-time snapshot is taken by the underlying storage system. In dynamic snapshot creation case, this field will be filled in by the snapshot controller with the "creation_time" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "creation_time" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. If not specified, it may indicate that the creation time of the snapshot is unknown. format: date-time type: string error: description: error is the last observed error during snapshot creation, if any. This field could be helpful to upper level controllers(i.e., application controller) to decide whether they should continue on waiting for the snapshot to be created based on the type of error reported. The snapshot controller will keep retrying when an error occurrs during the snapshot creation. Upon success, this error field will be cleared. properties: message: description: 'message is a string detailing the encountered error during snapshot creation if specified. NOTE: message may be logged, and it should not contain sensitive information.' type: string time: description: time is the timestamp when the error was encountered. format: date-time type: string type: object readyToUse: description: readyToUse indicates if the snapshot is ready to be used to restore a volume. In dynamic snapshot creation case, this field will be filled in by the snapshot controller with the "ready_to_use" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "ready_to_use" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it, otherwise, this field will be set to "True". If not specified, it means the readiness of a snapshot is unknown. type: boolean restoreSize: type: string description: restoreSize represents the minimum size of volume required to create a volume from this snapshot. In dynamic snapshot creation case, this field will be filled in by the snapshot controller with the "size_bytes" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "size_bytes" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. When restoring a volume from this snapshot, the size of the volume MUST NOT be smaller than the restoreSize if it is specified, otherwise the restoration will fail. If not specified, it indicates that the size is unknown. pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ x-kubernetes-int-or-string: true type: object required: - spec type: object ... skipping 60 lines ... type: string volumeSnapshotContentName: description: volumeSnapshotContentName specifies the name of a pre-existing VolumeSnapshotContent object representing an existing volume snapshot. This field should be set if the snapshot already exists and only needs a representation in Kubernetes. This field is immutable. type: string type: object volumeSnapshotClassName: description: 'VolumeSnapshotClassName is the name of the VolumeSnapshotClass requested by the VolumeSnapshot. VolumeSnapshotClassName may be left nil to indicate that the default SnapshotClass should be used. A given cluster may have multiple default Volume SnapshotClasses: one default per CSI Driver. If a VolumeSnapshot does not specify a SnapshotClass, VolumeSnapshotSource will be checked to figure out what the associated CSI Driver is, and the default VolumeSnapshotClass associated with that CSI Driver will be used. If more than one VolumeSnapshotClass exist for a given CSI Driver and more than one have been marked as default, CreateSnapshot will fail and generate an event. Empty string is not allowed for this field.' type: string required: - source type: object status: description: status represents the current information of a snapshot. Consumers must verify binding between VolumeSnapshot and VolumeSnapshotContent objects is successful (by validating that both VolumeSnapshot and VolumeSnapshotContent point at each other) before using this object. ... skipping 2 lines ... description: 'boundVolumeSnapshotContentName is the name of the VolumeSnapshotContent object to which this VolumeSnapshot object intends to bind to. If not specified, it indicates that the VolumeSnapshot object has not been successfully bound to a VolumeSnapshotContent object yet. NOTE: To avoid possible security issues, consumers must verify binding between VolumeSnapshot and VolumeSnapshotContent objects is successful (by validating that both VolumeSnapshot and VolumeSnapshotContent point at each other) before using this object.' type: string creationTime: description: creationTime is the timestamp when the point-in-time snapshot is taken by the underlying storage system. In dynamic snapshot creation case, this field will be filled in by the snapshot controller with the "creation_time" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "creation_time" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. If not specified, it may indicate that the creation time of the snapshot is unknown. format: date-time type: string error: description: error is the last observed error during snapshot creation, if any. This field could be helpful to upper level controllers(i.e., application controller) to decide whether they should continue on waiting for the snapshot to be created based on the type of error reported. The snapshot controller will keep retrying when an error occurrs during the snapshot creation. Upon success, this error field will be cleared. properties: message: description: 'message is a string detailing the encountered error during snapshot creation if specified. NOTE: message may be logged, and it should not contain sensitive information.' type: string time: description: time is the timestamp when the error was encountered. format: date-time type: string type: object readyToUse: description: readyToUse indicates if the snapshot is ready to be used to restore a volume. In dynamic snapshot creation case, this field will be filled in by the snapshot controller with the "ready_to_use" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "ready_to_use" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it, otherwise, this field will be set to "True". If not specified, it means the readiness of a snapshot is unknown. type: boolean restoreSize: type: string description: restoreSize represents the minimum size of volume required to create a volume from this snapshot. In dynamic snapshot creation case, this field will be filled in by the snapshot controller with the "size_bytes" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "size_bytes" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. When restoring a volume from this snapshot, the size of the volume MUST NOT be smaller than the restoreSize if it is specified, otherwise the restoration will fail. If not specified, it indicates that the size is unknown. pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ x-kubernetes-int-or-string: true type: object required: - spec type: object ... skipping 254 lines ... description: status represents the current information of a snapshot. properties: creationTime: description: creationTime is the timestamp when the point-in-time snapshot is taken by the underlying storage system. In dynamic snapshot creation case, this field will be filled in by the CSI snapshotter sidecar with the "creation_time" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "creation_time" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. If not specified, it indicates the creation time is unknown. The format of this field is a Unix nanoseconds time encoded as an int64. On Unix, the command `date +%s%N` returns the current time in nanoseconds since 1970-01-01 00:00:00 UTC. format: int64 type: integer error: description: error is the last observed error during snapshot creation, if any. Upon success after retry, this error field will be cleared. properties: message: description: 'message is a string detailing the encountered error during snapshot creation if specified. NOTE: message may be logged, and it should not contain sensitive information.' type: string time: description: time is the timestamp when the error was encountered. format: date-time type: string type: object readyToUse: description: readyToUse indicates if a snapshot is ready to be used to restore a volume. In dynamic snapshot creation case, this field will be filled in by the CSI snapshotter sidecar with the "ready_to_use" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "ready_to_use" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it, otherwise, this field will be set to "True". If not specified, it means the readiness of a snapshot is unknown. type: boolean restoreSize: description: restoreSize represents the complete size of the snapshot in bytes. In dynamic snapshot creation case, this field will be filled in by the CSI snapshotter sidecar with the "size_bytes" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "size_bytes" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. When restoring a volume from this snapshot, the size of the volume MUST NOT be smaller than the restoreSize if it is specified, otherwise the restoration will fail. If not specified, it indicates that the size is unknown. format: int64 minimum: 0 type: integer snapshotHandle: description: snapshotHandle is the CSI "snapshot_id" of a snapshot on the underlying storage system. If not specified, it indicates that dynamic snapshot creation has either failed or it is still in progress. type: string type: object required: - spec type: object served: true ... skipping 108 lines ... description: status represents the current information of a snapshot. properties: creationTime: description: creationTime is the timestamp when the point-in-time snapshot is taken by the underlying storage system. In dynamic snapshot creation case, this field will be filled in by the CSI snapshotter sidecar with the "creation_time" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "creation_time" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. If not specified, it indicates the creation time is unknown. The format of this field is a Unix nanoseconds time encoded as an int64. On Unix, the command `date +%s%N` returns the current time in nanoseconds since 1970-01-01 00:00:00 UTC. format: int64 type: integer error: description: error is the last observed error during snapshot creation, if any. Upon success after retry, this error field will be cleared. properties: message: description: 'message is a string detailing the encountered error during snapshot creation if specified. NOTE: message may be logged, and it should not contain sensitive information.' type: string time: description: time is the timestamp when the error was encountered. format: date-time type: string type: object readyToUse: description: readyToUse indicates if a snapshot is ready to be used to restore a volume. In dynamic snapshot creation case, this field will be filled in by the CSI snapshotter sidecar with the "ready_to_use" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "ready_to_use" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it, otherwise, this field will be set to "True". If not specified, it means the readiness of a snapshot is unknown. type: boolean restoreSize: description: restoreSize represents the complete size of the snapshot in bytes. In dynamic snapshot creation case, this field will be filled in by the CSI snapshotter sidecar with the "size_bytes" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "size_bytes" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. When restoring a volume from this snapshot, the size of the volume MUST NOT be smaller than the restoreSize if it is specified, otherwise the restoration will fail. If not specified, it indicates that the size is unknown. format: int64 minimum: 0 type: integer snapshotHandle: description: snapshotHandle is the CSI "snapshot_id" of a snapshot on the underlying storage system. If not specified, it indicates that dynamic snapshot creation has either failed or it is still in progress. type: string type: object required: - spec type: object served: true ... skipping 865 lines ... image: "mcr.microsoft.com/oss/kubernetes-csi/csi-resizer:v1.6.0" args: - "-csi-address=$(ADDRESS)" - "-v=2" - "-leader-election" - "--leader-election-namespace=kube-system" - '-handle-volume-inuse-error=false' - '-feature-gates=RecoverVolumeExpansionFailure=true' - "-timeout=240s" env: - name: ADDRESS value: /csi/csi.sock volumeMounts: ... skipping 216 lines ... [1mSTEP:[0m setting up the StorageClass [38;5;243m01/09/23 09:36:35.403[0m [1mSTEP:[0m creating a StorageClass [38;5;243m01/09/23 09:36:35.403[0m [1mSTEP:[0m setting up the PVC and PV [38;5;243m01/09/23 09:36:35.461[0m [1mSTEP:[0m creating a PVC [38;5;243m01/09/23 09:36:35.462[0m [1mSTEP:[0m setting up the pod [38;5;243m01/09/23 09:36:35.522[0m [1mSTEP:[0m deploying the pod [38;5;243m01/09/23 09:36:35.523[0m [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/09/23 09:36:35.586[0m Jan 9 09:36:35.586: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-2t5vz" in namespace "azuredisk-8081" to be "Succeeded or Failed" Jan 9 09:36:35.643: INFO: Pod "azuredisk-volume-tester-2t5vz": Phase="Pending", Reason="", readiness=false. Elapsed: 56.926941ms Jan 9 09:36:37.701: INFO: Pod "azuredisk-volume-tester-2t5vz": Phase="Pending", Reason="", readiness=false. Elapsed: 2.115078895s Jan 9 09:36:39.701: INFO: Pod "azuredisk-volume-tester-2t5vz": Phase="Pending", Reason="", readiness=false. Elapsed: 4.115446803s Jan 9 09:36:41.703: INFO: Pod "azuredisk-volume-tester-2t5vz": Phase="Pending", Reason="", readiness=false. Elapsed: 6.116679732s Jan 9 09:36:43.701: INFO: Pod "azuredisk-volume-tester-2t5vz": Phase="Pending", Reason="", readiness=false. Elapsed: 8.115306837s Jan 9 09:36:45.701: INFO: Pod "azuredisk-volume-tester-2t5vz": Phase="Pending", Reason="", readiness=false. Elapsed: 10.115129157s ... skipping 6 lines ... Jan 9 09:36:59.701: INFO: Pod "azuredisk-volume-tester-2t5vz": Phase="Pending", Reason="", readiness=false. Elapsed: 24.115529895s Jan 9 09:37:01.702: INFO: Pod "azuredisk-volume-tester-2t5vz": Phase="Pending", Reason="", readiness=false. Elapsed: 26.116321252s Jan 9 09:37:03.702: INFO: Pod "azuredisk-volume-tester-2t5vz": Phase="Pending", Reason="", readiness=false. Elapsed: 28.116050198s Jan 9 09:37:05.706: INFO: Pod "azuredisk-volume-tester-2t5vz": Phase="Pending", Reason="", readiness=false. Elapsed: 30.119769632s Jan 9 09:37:07.703: INFO: Pod "azuredisk-volume-tester-2t5vz": Phase="Succeeded", Reason="", readiness=false. Elapsed: 32.116652339s [1mSTEP:[0m Saw pod success [38;5;243m01/09/23 09:37:07.703[0m Jan 9 09:37:07.703: INFO: Pod "azuredisk-volume-tester-2t5vz" satisfied condition "Succeeded or Failed" Jan 9 09:37:07.703: INFO: deleting Pod "azuredisk-8081"/"azuredisk-volume-tester-2t5vz" Jan 9 09:37:07.826: INFO: Pod azuredisk-volume-tester-2t5vz has the following logs: hello world [1mSTEP:[0m Deleting pod azuredisk-volume-tester-2t5vz in namespace azuredisk-8081 [38;5;243m01/09/23 09:37:07.826[0m [1mSTEP:[0m validating provisioned PV [38;5;243m01/09/23 09:37:07.947[0m [1mSTEP:[0m checking the PV [38;5;243m01/09/23 09:37:08.004[0m ... skipping 57 lines ... [1mSTEP:[0m setting up the StorageClass [38;5;243m01/09/23 09:36:35.403[0m [1mSTEP:[0m creating a StorageClass [38;5;243m01/09/23 09:36:35.403[0m [1mSTEP:[0m setting up the PVC and PV [38;5;243m01/09/23 09:36:35.461[0m [1mSTEP:[0m creating a PVC [38;5;243m01/09/23 09:36:35.462[0m [1mSTEP:[0m setting up the pod [38;5;243m01/09/23 09:36:35.522[0m [1mSTEP:[0m deploying the pod [38;5;243m01/09/23 09:36:35.523[0m [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/09/23 09:36:35.586[0m Jan 9 09:36:35.586: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-2t5vz" in namespace "azuredisk-8081" to be "Succeeded or Failed" Jan 9 09:36:35.643: INFO: Pod "azuredisk-volume-tester-2t5vz": Phase="Pending", Reason="", readiness=false. Elapsed: 56.926941ms Jan 9 09:36:37.701: INFO: Pod "azuredisk-volume-tester-2t5vz": Phase="Pending", Reason="", readiness=false. Elapsed: 2.115078895s Jan 9 09:36:39.701: INFO: Pod "azuredisk-volume-tester-2t5vz": Phase="Pending", Reason="", readiness=false. Elapsed: 4.115446803s Jan 9 09:36:41.703: INFO: Pod "azuredisk-volume-tester-2t5vz": Phase="Pending", Reason="", readiness=false. Elapsed: 6.116679732s Jan 9 09:36:43.701: INFO: Pod "azuredisk-volume-tester-2t5vz": Phase="Pending", Reason="", readiness=false. Elapsed: 8.115306837s Jan 9 09:36:45.701: INFO: Pod "azuredisk-volume-tester-2t5vz": Phase="Pending", Reason="", readiness=false. Elapsed: 10.115129157s ... skipping 6 lines ... Jan 9 09:36:59.701: INFO: Pod "azuredisk-volume-tester-2t5vz": Phase="Pending", Reason="", readiness=false. Elapsed: 24.115529895s Jan 9 09:37:01.702: INFO: Pod "azuredisk-volume-tester-2t5vz": Phase="Pending", Reason="", readiness=false. Elapsed: 26.116321252s Jan 9 09:37:03.702: INFO: Pod "azuredisk-volume-tester-2t5vz": Phase="Pending", Reason="", readiness=false. Elapsed: 28.116050198s Jan 9 09:37:05.706: INFO: Pod "azuredisk-volume-tester-2t5vz": Phase="Pending", Reason="", readiness=false. Elapsed: 30.119769632s Jan 9 09:37:07.703: INFO: Pod "azuredisk-volume-tester-2t5vz": Phase="Succeeded", Reason="", readiness=false. Elapsed: 32.116652339s [1mSTEP:[0m Saw pod success [38;5;243m01/09/23 09:37:07.703[0m Jan 9 09:37:07.703: INFO: Pod "azuredisk-volume-tester-2t5vz" satisfied condition "Succeeded or Failed" Jan 9 09:37:07.703: INFO: deleting Pod "azuredisk-8081"/"azuredisk-volume-tester-2t5vz" Jan 9 09:37:07.826: INFO: Pod azuredisk-volume-tester-2t5vz has the following logs: hello world [1mSTEP:[0m Deleting pod azuredisk-volume-tester-2t5vz in namespace azuredisk-8081 [38;5;243m01/09/23 09:37:07.826[0m [1mSTEP:[0m validating provisioned PV [38;5;243m01/09/23 09:37:07.947[0m [1mSTEP:[0m checking the PV [38;5;243m01/09/23 09:37:08.004[0m ... skipping 39 lines ... Jan 9 09:37:51.909: INFO: PersistentVolumeClaim pvc-pszvh found but phase is Pending instead of Bound. Jan 9 09:37:53.968: INFO: PersistentVolumeClaim pvc-pszvh found and phase=Bound (4.174470721s) [1mSTEP:[0m checking the PVC [38;5;243m01/09/23 09:37:53.968[0m [1mSTEP:[0m validating provisioned PV [38;5;243m01/09/23 09:37:54.027[0m [1mSTEP:[0m checking the PV [38;5;243m01/09/23 09:37:54.084[0m [1mSTEP:[0m deploying the pod [38;5;243m01/09/23 09:37:54.084[0m [1mSTEP:[0m checking that the pods command exits with no error [38;5;243m01/09/23 09:37:54.144[0m Jan 9 09:37:54.144: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-sl2t7" in namespace "azuredisk-2540" to be "Succeeded or Failed" Jan 9 09:37:54.200: INFO: Pod "azuredisk-volume-tester-sl2t7": Phase="Pending", Reason="", readiness=false. Elapsed: 56.366497ms Jan 9 09:37:56.258: INFO: Pod "azuredisk-volume-tester-sl2t7": Phase="Pending", Reason="", readiness=false. Elapsed: 2.113911633s Jan 9 09:37:58.259: INFO: Pod "azuredisk-volume-tester-sl2t7": Phase="Pending", Reason="", readiness=false. Elapsed: 4.115564242s Jan 9 09:38:00.260: INFO: Pod "azuredisk-volume-tester-sl2t7": Phase="Pending", Reason="", readiness=false. Elapsed: 6.115695228s Jan 9 09:38:02.260: INFO: Pod "azuredisk-volume-tester-sl2t7": Phase="Pending", Reason="", readiness=false. Elapsed: 8.115834197s Jan 9 09:38:04.258: INFO: Pod "azuredisk-volume-tester-sl2t7": Phase="Pending", Reason="", readiness=false. Elapsed: 10.113907753s Jan 9 09:38:06.260: INFO: Pod "azuredisk-volume-tester-sl2t7": Phase="Pending", Reason="", readiness=false. Elapsed: 12.115729839s Jan 9 09:38:08.260: INFO: Pod "azuredisk-volume-tester-sl2t7": Phase="Pending", Reason="", readiness=false. Elapsed: 14.115656215s Jan 9 09:38:10.259: INFO: Pod "azuredisk-volume-tester-sl2t7": Phase="Pending", Reason="", readiness=false. Elapsed: 16.115509138s Jan 9 09:38:12.261: INFO: Pod "azuredisk-volume-tester-sl2t7": Phase="Pending", Reason="", readiness=false. Elapsed: 18.11699098s Jan 9 09:38:14.259: INFO: Pod "azuredisk-volume-tester-sl2t7": Phase="Running", Reason="", readiness=true. Elapsed: 20.115521236s Jan 9 09:38:16.258: INFO: Pod "azuredisk-volume-tester-sl2t7": Phase="Succeeded", Reason="", readiness=false. Elapsed: 22.114511084s [1mSTEP:[0m Saw pod success [38;5;243m01/09/23 09:38:16.258[0m Jan 9 09:38:16.259: INFO: Pod "azuredisk-volume-tester-sl2t7" satisfied condition "Succeeded or Failed" Jan 9 09:38:16.259: INFO: deleting Pod "azuredisk-2540"/"azuredisk-volume-tester-sl2t7" Jan 9 09:38:16.349: INFO: Pod azuredisk-volume-tester-sl2t7 has the following logs: hello world [1mSTEP:[0m Deleting pod azuredisk-volume-tester-sl2t7 in namespace azuredisk-2540 [38;5;243m01/09/23 09:38:16.349[0m Jan 9 09:38:16.421: INFO: deleting PVC "azuredisk-2540"/"pvc-pszvh" Jan 9 09:38:16.421: INFO: Deleting PersistentVolumeClaim "pvc-pszvh" ... skipping 44 lines ... Jan 9 09:37:51.909: INFO: PersistentVolumeClaim pvc-pszvh found but phase is Pending instead of Bound. Jan 9 09:37:53.968: INFO: PersistentVolumeClaim pvc-pszvh found and phase=Bound (4.174470721s) [1mSTEP:[0m checking the PVC [38;5;243m01/09/23 09:37:53.968[0m [1mSTEP:[0m validating provisioned PV [38;5;243m01/09/23 09:37:54.027[0m [1mSTEP:[0m checking the PV [38;5;243m01/09/23 09:37:54.084[0m [1mSTEP:[0m deploying the pod [38;5;243m01/09/23 09:37:54.084[0m [1mSTEP:[0m checking that the pods command exits with no error [38;5;243m01/09/23 09:37:54.144[0m Jan 9 09:37:54.144: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-sl2t7" in namespace "azuredisk-2540" to be "Succeeded or Failed" Jan 9 09:37:54.200: INFO: Pod "azuredisk-volume-tester-sl2t7": Phase="Pending", Reason="", readiness=false. Elapsed: 56.366497ms Jan 9 09:37:56.258: INFO: Pod "azuredisk-volume-tester-sl2t7": Phase="Pending", Reason="", readiness=false. Elapsed: 2.113911633s Jan 9 09:37:58.259: INFO: Pod "azuredisk-volume-tester-sl2t7": Phase="Pending", Reason="", readiness=false. Elapsed: 4.115564242s Jan 9 09:38:00.260: INFO: Pod "azuredisk-volume-tester-sl2t7": Phase="Pending", Reason="", readiness=false. Elapsed: 6.115695228s Jan 9 09:38:02.260: INFO: Pod "azuredisk-volume-tester-sl2t7": Phase="Pending", Reason="", readiness=false. Elapsed: 8.115834197s Jan 9 09:38:04.258: INFO: Pod "azuredisk-volume-tester-sl2t7": Phase="Pending", Reason="", readiness=false. Elapsed: 10.113907753s Jan 9 09:38:06.260: INFO: Pod "azuredisk-volume-tester-sl2t7": Phase="Pending", Reason="", readiness=false. Elapsed: 12.115729839s Jan 9 09:38:08.260: INFO: Pod "azuredisk-volume-tester-sl2t7": Phase="Pending", Reason="", readiness=false. Elapsed: 14.115656215s Jan 9 09:38:10.259: INFO: Pod "azuredisk-volume-tester-sl2t7": Phase="Pending", Reason="", readiness=false. Elapsed: 16.115509138s Jan 9 09:38:12.261: INFO: Pod "azuredisk-volume-tester-sl2t7": Phase="Pending", Reason="", readiness=false. Elapsed: 18.11699098s Jan 9 09:38:14.259: INFO: Pod "azuredisk-volume-tester-sl2t7": Phase="Running", Reason="", readiness=true. Elapsed: 20.115521236s Jan 9 09:38:16.258: INFO: Pod "azuredisk-volume-tester-sl2t7": Phase="Succeeded", Reason="", readiness=false. Elapsed: 22.114511084s [1mSTEP:[0m Saw pod success [38;5;243m01/09/23 09:38:16.258[0m Jan 9 09:38:16.259: INFO: Pod "azuredisk-volume-tester-sl2t7" satisfied condition "Succeeded or Failed" Jan 9 09:38:16.259: INFO: deleting Pod "azuredisk-2540"/"azuredisk-volume-tester-sl2t7" Jan 9 09:38:16.349: INFO: Pod azuredisk-volume-tester-sl2t7 has the following logs: hello world [1mSTEP:[0m Deleting pod azuredisk-volume-tester-sl2t7 in namespace azuredisk-2540 [38;5;243m01/09/23 09:38:16.349[0m Jan 9 09:38:16.421: INFO: deleting PVC "azuredisk-2540"/"pvc-pszvh" Jan 9 09:38:16.421: INFO: Deleting PersistentVolumeClaim "pvc-pszvh" ... skipping 36 lines ... [1mSTEP:[0m setting up the StorageClass [38;5;243m01/09/23 09:39:28.481[0m [1mSTEP:[0m creating a StorageClass [38;5;243m01/09/23 09:39:28.481[0m [1mSTEP:[0m setting up the PVC and PV [38;5;243m01/09/23 09:39:28.544[0m [1mSTEP:[0m creating a PVC [38;5;243m01/09/23 09:39:28.544[0m [1mSTEP:[0m setting up the pod [38;5;243m01/09/23 09:39:28.606[0m [1mSTEP:[0m deploying the pod [38;5;243m01/09/23 09:39:28.606[0m [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/09/23 09:39:28.666[0m Jan 9 09:39:28.666: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-88r7c" in namespace "azuredisk-4728" to be "Succeeded or Failed" Jan 9 09:39:28.724: INFO: Pod "azuredisk-volume-tester-88r7c": Phase="Pending", Reason="", readiness=false. Elapsed: 57.385146ms Jan 9 09:39:30.782: INFO: Pod "azuredisk-volume-tester-88r7c": Phase="Pending", Reason="", readiness=false. Elapsed: 2.116174816s Jan 9 09:39:32.783: INFO: Pod "azuredisk-volume-tester-88r7c": Phase="Pending", Reason="", readiness=false. Elapsed: 4.116358112s Jan 9 09:39:34.783: INFO: Pod "azuredisk-volume-tester-88r7c": Phase="Pending", Reason="", readiness=false. Elapsed: 6.117135017s Jan 9 09:39:36.782: INFO: Pod "azuredisk-volume-tester-88r7c": Phase="Pending", Reason="", readiness=false. Elapsed: 8.116039319s Jan 9 09:39:38.782: INFO: Pod "azuredisk-volume-tester-88r7c": Phase="Pending", Reason="", readiness=false. Elapsed: 10.115652493s ... skipping 14 lines ... Jan 9 09:40:08.783: INFO: Pod "azuredisk-volume-tester-88r7c": Phase="Pending", Reason="", readiness=false. Elapsed: 40.117160811s Jan 9 09:40:10.784: INFO: Pod "azuredisk-volume-tester-88r7c": Phase="Pending", Reason="", readiness=false. Elapsed: 42.117563048s Jan 9 09:40:12.782: INFO: Pod "azuredisk-volume-tester-88r7c": Phase="Pending", Reason="", readiness=false. Elapsed: 44.115717487s Jan 9 09:40:14.781: INFO: Pod "azuredisk-volume-tester-88r7c": Phase="Pending", Reason="", readiness=false. Elapsed: 46.114650543s Jan 9 09:40:16.782: INFO: Pod "azuredisk-volume-tester-88r7c": Phase="Succeeded", Reason="", readiness=false. Elapsed: 48.115513099s [1mSTEP:[0m Saw pod success [38;5;243m01/09/23 09:40:16.782[0m Jan 9 09:40:16.782: INFO: Pod "azuredisk-volume-tester-88r7c" satisfied condition "Succeeded or Failed" Jan 9 09:40:16.782: INFO: deleting Pod "azuredisk-4728"/"azuredisk-volume-tester-88r7c" Jan 9 09:40:16.872: INFO: Pod azuredisk-volume-tester-88r7c has the following logs: hello world [1mSTEP:[0m Deleting pod azuredisk-volume-tester-88r7c in namespace azuredisk-4728 [38;5;243m01/09/23 09:40:16.872[0m [1mSTEP:[0m validating provisioned PV [38;5;243m01/09/23 09:40:16.993[0m [1mSTEP:[0m checking the PV [38;5;243m01/09/23 09:40:17.05[0m ... skipping 33 lines ... [1mSTEP:[0m setting up the StorageClass [38;5;243m01/09/23 09:39:28.481[0m [1mSTEP:[0m creating a StorageClass [38;5;243m01/09/23 09:39:28.481[0m [1mSTEP:[0m setting up the PVC and PV [38;5;243m01/09/23 09:39:28.544[0m [1mSTEP:[0m creating a PVC [38;5;243m01/09/23 09:39:28.544[0m [1mSTEP:[0m setting up the pod [38;5;243m01/09/23 09:39:28.606[0m [1mSTEP:[0m deploying the pod [38;5;243m01/09/23 09:39:28.606[0m [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/09/23 09:39:28.666[0m Jan 9 09:39:28.666: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-88r7c" in namespace "azuredisk-4728" to be "Succeeded or Failed" Jan 9 09:39:28.724: INFO: Pod "azuredisk-volume-tester-88r7c": Phase="Pending", Reason="", readiness=false. Elapsed: 57.385146ms Jan 9 09:39:30.782: INFO: Pod "azuredisk-volume-tester-88r7c": Phase="Pending", Reason="", readiness=false. Elapsed: 2.116174816s Jan 9 09:39:32.783: INFO: Pod "azuredisk-volume-tester-88r7c": Phase="Pending", Reason="", readiness=false. Elapsed: 4.116358112s Jan 9 09:39:34.783: INFO: Pod "azuredisk-volume-tester-88r7c": Phase="Pending", Reason="", readiness=false. Elapsed: 6.117135017s Jan 9 09:39:36.782: INFO: Pod "azuredisk-volume-tester-88r7c": Phase="Pending", Reason="", readiness=false. Elapsed: 8.116039319s Jan 9 09:39:38.782: INFO: Pod "azuredisk-volume-tester-88r7c": Phase="Pending", Reason="", readiness=false. Elapsed: 10.115652493s ... skipping 14 lines ... Jan 9 09:40:08.783: INFO: Pod "azuredisk-volume-tester-88r7c": Phase="Pending", Reason="", readiness=false. Elapsed: 40.117160811s Jan 9 09:40:10.784: INFO: Pod "azuredisk-volume-tester-88r7c": Phase="Pending", Reason="", readiness=false. Elapsed: 42.117563048s Jan 9 09:40:12.782: INFO: Pod "azuredisk-volume-tester-88r7c": Phase="Pending", Reason="", readiness=false. Elapsed: 44.115717487s Jan 9 09:40:14.781: INFO: Pod "azuredisk-volume-tester-88r7c": Phase="Pending", Reason="", readiness=false. Elapsed: 46.114650543s Jan 9 09:40:16.782: INFO: Pod "azuredisk-volume-tester-88r7c": Phase="Succeeded", Reason="", readiness=false. Elapsed: 48.115513099s [1mSTEP:[0m Saw pod success [38;5;243m01/09/23 09:40:16.782[0m Jan 9 09:40:16.782: INFO: Pod "azuredisk-volume-tester-88r7c" satisfied condition "Succeeded or Failed" Jan 9 09:40:16.782: INFO: deleting Pod "azuredisk-4728"/"azuredisk-volume-tester-88r7c" Jan 9 09:40:16.872: INFO: Pod azuredisk-volume-tester-88r7c has the following logs: hello world [1mSTEP:[0m Deleting pod azuredisk-volume-tester-88r7c in namespace azuredisk-4728 [38;5;243m01/09/23 09:40:16.872[0m [1mSTEP:[0m validating provisioned PV [38;5;243m01/09/23 09:40:16.993[0m [1mSTEP:[0m checking the PV [38;5;243m01/09/23 09:40:17.05[0m ... skipping 34 lines ... [1mSTEP:[0m setting up the PVC and PV [38;5;243m01/09/23 09:40:58.786[0m [1mSTEP:[0m creating a PVC [38;5;243m01/09/23 09:40:58.786[0m [1mSTEP:[0m setting up the pod [38;5;243m01/09/23 09:40:58.849[0m [1mSTEP:[0m deploying the pod [38;5;243m01/09/23 09:40:58.849[0m [1mSTEP:[0m checking that the pod has 'FailedMount' event [38;5;243m01/09/23 09:40:58.912[0m Jan 9 09:41:21.028: INFO: deleting Pod "azuredisk-5466"/"azuredisk-volume-tester-8sg5k" Jan 9 09:41:21.088: INFO: Error getting logs for pod azuredisk-volume-tester-8sg5k: the server rejected our request for an unknown reason (get pods azuredisk-volume-tester-8sg5k) [1mSTEP:[0m Deleting pod azuredisk-volume-tester-8sg5k in namespace azuredisk-5466 [38;5;243m01/09/23 09:41:21.089[0m [1mSTEP:[0m validating provisioned PV [38;5;243m01/09/23 09:41:21.207[0m [1mSTEP:[0m checking the PV [38;5;243m01/09/23 09:41:21.265[0m Jan 9 09:41:21.265: INFO: deleting PVC "azuredisk-5466"/"pvc-s6b42" Jan 9 09:41:21.265: INFO: Deleting PersistentVolumeClaim "pvc-s6b42" [1mSTEP:[0m waiting for claim's PV "pvc-f62b55d7-4648-4389-b72a-610edf2d30c0" to be deleted [38;5;243m01/09/23 09:41:21.325[0m ... skipping 33 lines ... [1mSTEP:[0m setting up the PVC and PV [38;5;243m01/09/23 09:40:58.786[0m [1mSTEP:[0m creating a PVC [38;5;243m01/09/23 09:40:58.786[0m [1mSTEP:[0m setting up the pod [38;5;243m01/09/23 09:40:58.849[0m [1mSTEP:[0m deploying the pod [38;5;243m01/09/23 09:40:58.849[0m [1mSTEP:[0m checking that the pod has 'FailedMount' event [38;5;243m01/09/23 09:40:58.912[0m Jan 9 09:41:21.028: INFO: deleting Pod "azuredisk-5466"/"azuredisk-volume-tester-8sg5k" Jan 9 09:41:21.088: INFO: Error getting logs for pod azuredisk-volume-tester-8sg5k: the server rejected our request for an unknown reason (get pods azuredisk-volume-tester-8sg5k) [1mSTEP:[0m Deleting pod azuredisk-volume-tester-8sg5k in namespace azuredisk-5466 [38;5;243m01/09/23 09:41:21.089[0m [1mSTEP:[0m validating provisioned PV [38;5;243m01/09/23 09:41:21.207[0m [1mSTEP:[0m checking the PV [38;5;243m01/09/23 09:41:21.265[0m Jan 9 09:41:21.265: INFO: deleting PVC "azuredisk-5466"/"pvc-s6b42" Jan 9 09:41:21.265: INFO: Deleting PersistentVolumeClaim "pvc-s6b42" [1mSTEP:[0m waiting for claim's PV "pvc-f62b55d7-4648-4389-b72a-610edf2d30c0" to be deleted [38;5;243m01/09/23 09:41:21.325[0m ... skipping 30 lines ... [1mSTEP:[0m setting up the StorageClass [38;5;243m01/09/23 09:42:08.038[0m [1mSTEP:[0m creating a StorageClass [38;5;243m01/09/23 09:42:08.038[0m [1mSTEP:[0m setting up the PVC and PV [38;5;243m01/09/23 09:42:08.098[0m [1mSTEP:[0m creating a PVC [38;5;243m01/09/23 09:42:08.098[0m [1mSTEP:[0m setting up the pod [38;5;243m01/09/23 09:42:08.159[0m [1mSTEP:[0m deploying the pod [38;5;243m01/09/23 09:42:08.159[0m [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/09/23 09:42:08.222[0m Jan 9 09:42:08.222: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-j89tr" in namespace "azuredisk-2790" to be "Succeeded or Failed" Jan 9 09:42:08.279: INFO: Pod "azuredisk-volume-tester-j89tr": Phase="Pending", Reason="", readiness=false. Elapsed: 57.089906ms Jan 9 09:42:10.339: INFO: Pod "azuredisk-volume-tester-j89tr": Phase="Pending", Reason="", readiness=false. Elapsed: 2.116141008s Jan 9 09:42:12.339: INFO: Pod "azuredisk-volume-tester-j89tr": Phase="Pending", Reason="", readiness=false. Elapsed: 4.116161383s Jan 9 09:42:14.339: INFO: Pod "azuredisk-volume-tester-j89tr": Phase="Pending", Reason="", readiness=false. Elapsed: 6.116602308s Jan 9 09:42:16.338: INFO: Pod "azuredisk-volume-tester-j89tr": Phase="Pending", Reason="", readiness=false. Elapsed: 8.11577227s Jan 9 09:42:18.339: INFO: Pod "azuredisk-volume-tester-j89tr": Phase="Pending", Reason="", readiness=false. Elapsed: 10.116711053s ... skipping 3 lines ... Jan 9 09:42:26.340: INFO: Pod "azuredisk-volume-tester-j89tr": Phase="Pending", Reason="", readiness=false. Elapsed: 18.117532398s Jan 9 09:42:28.338: INFO: Pod "azuredisk-volume-tester-j89tr": Phase="Pending", Reason="", readiness=false. Elapsed: 20.115357392s Jan 9 09:42:30.337: INFO: Pod "azuredisk-volume-tester-j89tr": Phase="Pending", Reason="", readiness=false. Elapsed: 22.11487342s Jan 9 09:42:32.338: INFO: Pod "azuredisk-volume-tester-j89tr": Phase="Running", Reason="", readiness=true. Elapsed: 24.115728291s Jan 9 09:42:34.339: INFO: Pod "azuredisk-volume-tester-j89tr": Phase="Succeeded", Reason="", readiness=false. Elapsed: 26.11650131s [1mSTEP:[0m Saw pod success [38;5;243m01/09/23 09:42:34.339[0m Jan 9 09:42:34.339: INFO: Pod "azuredisk-volume-tester-j89tr" satisfied condition "Succeeded or Failed" Jan 9 09:42:34.339: INFO: deleting Pod "azuredisk-2790"/"azuredisk-volume-tester-j89tr" Jan 9 09:42:34.405: INFO: Pod azuredisk-volume-tester-j89tr has the following logs: e2e-test [1mSTEP:[0m Deleting pod azuredisk-volume-tester-j89tr in namespace azuredisk-2790 [38;5;243m01/09/23 09:42:34.405[0m [1mSTEP:[0m validating provisioned PV [38;5;243m01/09/23 09:42:34.528[0m [1mSTEP:[0m checking the PV [38;5;243m01/09/23 09:42:34.585[0m ... skipping 33 lines ... [1mSTEP:[0m setting up the StorageClass [38;5;243m01/09/23 09:42:08.038[0m [1mSTEP:[0m creating a StorageClass [38;5;243m01/09/23 09:42:08.038[0m [1mSTEP:[0m setting up the PVC and PV [38;5;243m01/09/23 09:42:08.098[0m [1mSTEP:[0m creating a PVC [38;5;243m01/09/23 09:42:08.098[0m [1mSTEP:[0m setting up the pod [38;5;243m01/09/23 09:42:08.159[0m [1mSTEP:[0m deploying the pod [38;5;243m01/09/23 09:42:08.159[0m [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/09/23 09:42:08.222[0m Jan 9 09:42:08.222: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-j89tr" in namespace "azuredisk-2790" to be "Succeeded or Failed" Jan 9 09:42:08.279: INFO: Pod "azuredisk-volume-tester-j89tr": Phase="Pending", Reason="", readiness=false. Elapsed: 57.089906ms Jan 9 09:42:10.339: INFO: Pod "azuredisk-volume-tester-j89tr": Phase="Pending", Reason="", readiness=false. Elapsed: 2.116141008s Jan 9 09:42:12.339: INFO: Pod "azuredisk-volume-tester-j89tr": Phase="Pending", Reason="", readiness=false. Elapsed: 4.116161383s Jan 9 09:42:14.339: INFO: Pod "azuredisk-volume-tester-j89tr": Phase="Pending", Reason="", readiness=false. Elapsed: 6.116602308s Jan 9 09:42:16.338: INFO: Pod "azuredisk-volume-tester-j89tr": Phase="Pending", Reason="", readiness=false. Elapsed: 8.11577227s Jan 9 09:42:18.339: INFO: Pod "azuredisk-volume-tester-j89tr": Phase="Pending", Reason="", readiness=false. Elapsed: 10.116711053s ... skipping 3 lines ... Jan 9 09:42:26.340: INFO: Pod "azuredisk-volume-tester-j89tr": Phase="Pending", Reason="", readiness=false. Elapsed: 18.117532398s Jan 9 09:42:28.338: INFO: Pod "azuredisk-volume-tester-j89tr": Phase="Pending", Reason="", readiness=false. Elapsed: 20.115357392s Jan 9 09:42:30.337: INFO: Pod "azuredisk-volume-tester-j89tr": Phase="Pending", Reason="", readiness=false. Elapsed: 22.11487342s Jan 9 09:42:32.338: INFO: Pod "azuredisk-volume-tester-j89tr": Phase="Running", Reason="", readiness=true. Elapsed: 24.115728291s Jan 9 09:42:34.339: INFO: Pod "azuredisk-volume-tester-j89tr": Phase="Succeeded", Reason="", readiness=false. Elapsed: 26.11650131s [1mSTEP:[0m Saw pod success [38;5;243m01/09/23 09:42:34.339[0m Jan 9 09:42:34.339: INFO: Pod "azuredisk-volume-tester-j89tr" satisfied condition "Succeeded or Failed" Jan 9 09:42:34.339: INFO: deleting Pod "azuredisk-2790"/"azuredisk-volume-tester-j89tr" Jan 9 09:42:34.405: INFO: Pod azuredisk-volume-tester-j89tr has the following logs: e2e-test [1mSTEP:[0m Deleting pod azuredisk-volume-tester-j89tr in namespace azuredisk-2790 [38;5;243m01/09/23 09:42:34.405[0m [1mSTEP:[0m validating provisioned PV [38;5;243m01/09/23 09:42:34.528[0m [1mSTEP:[0m checking the PV [38;5;243m01/09/23 09:42:34.585[0m ... skipping 37 lines ... [1mSTEP:[0m creating volume in external rg azuredisk-csi-driver-test-0ab45119-9002-11ed-9173-829d2d0c1a9d [38;5;243m01/09/23 09:43:17.624[0m [1mSTEP:[0m setting up the StorageClass [38;5;243m01/09/23 09:43:17.624[0m [1mSTEP:[0m creating a StorageClass [38;5;243m01/09/23 09:43:17.625[0m [1mSTEP:[0m setting up the PVC and PV [38;5;243m01/09/23 09:43:17.683[0m [1mSTEP:[0m creating a PVC [38;5;243m01/09/23 09:43:17.683[0m [1mSTEP:[0m deploying the pod [38;5;243m01/09/23 09:43:17.748[0m [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/09/23 09:43:17.807[0m Jan 9 09:43:17.807: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-qxg7b" in namespace "azuredisk-5356" to be "Succeeded or Failed" Jan 9 09:43:17.865: INFO: Pod "azuredisk-volume-tester-qxg7b": Phase="Pending", Reason="", readiness=false. Elapsed: 57.52375ms Jan 9 09:43:19.923: INFO: Pod "azuredisk-volume-tester-qxg7b": Phase="Pending", Reason="", readiness=false. Elapsed: 2.115534679s Jan 9 09:43:21.922: INFO: Pod "azuredisk-volume-tester-qxg7b": Phase="Pending", Reason="", readiness=false. Elapsed: 4.115254104s Jan 9 09:43:23.923: INFO: Pod "azuredisk-volume-tester-qxg7b": Phase="Pending", Reason="", readiness=false. Elapsed: 6.115723661s Jan 9 09:43:25.924: INFO: Pod "azuredisk-volume-tester-qxg7b": Phase="Pending", Reason="", readiness=false. Elapsed: 8.116618048s Jan 9 09:43:27.924: INFO: Pod "azuredisk-volume-tester-qxg7b": Phase="Pending", Reason="", readiness=false. Elapsed: 10.11649832s ... skipping 2 lines ... Jan 9 09:43:33.923: INFO: Pod "azuredisk-volume-tester-qxg7b": Phase="Pending", Reason="", readiness=false. Elapsed: 16.115981924s Jan 9 09:43:35.924: INFO: Pod "azuredisk-volume-tester-qxg7b": Phase="Pending", Reason="", readiness=false. Elapsed: 18.116727724s Jan 9 09:43:37.926: INFO: Pod "azuredisk-volume-tester-qxg7b": Phase="Pending", Reason="", readiness=false. Elapsed: 20.118561688s Jan 9 09:43:39.923: INFO: Pod "azuredisk-volume-tester-qxg7b": Phase="Pending", Reason="", readiness=false. Elapsed: 22.115995419s Jan 9 09:43:41.924: INFO: Pod "azuredisk-volume-tester-qxg7b": Phase="Succeeded", Reason="", readiness=false. Elapsed: 24.116865426s [1mSTEP:[0m Saw pod success [38;5;243m01/09/23 09:43:41.924[0m Jan 9 09:43:41.924: INFO: Pod "azuredisk-volume-tester-qxg7b" satisfied condition "Succeeded or Failed" Jan 9 09:43:41.925: INFO: deleting Pod "azuredisk-5356"/"azuredisk-volume-tester-qxg7b" Jan 9 09:43:41.986: INFO: Pod azuredisk-volume-tester-qxg7b has the following logs: hello world [1mSTEP:[0m Deleting pod azuredisk-volume-tester-qxg7b in namespace azuredisk-5356 [38;5;243m01/09/23 09:43:41.986[0m [1mSTEP:[0m validating provisioned PV [38;5;243m01/09/23 09:43:42.108[0m [1mSTEP:[0m checking the PV [38;5;243m01/09/23 09:43:42.165[0m ... skipping 37 lines ... [1mSTEP:[0m creating volume in external rg azuredisk-csi-driver-test-0ab45119-9002-11ed-9173-829d2d0c1a9d [38;5;243m01/09/23 09:43:17.624[0m [1mSTEP:[0m setting up the StorageClass [38;5;243m01/09/23 09:43:17.624[0m [1mSTEP:[0m creating a StorageClass [38;5;243m01/09/23 09:43:17.625[0m [1mSTEP:[0m setting up the PVC and PV [38;5;243m01/09/23 09:43:17.683[0m [1mSTEP:[0m creating a PVC [38;5;243m01/09/23 09:43:17.683[0m [1mSTEP:[0m deploying the pod [38;5;243m01/09/23 09:43:17.748[0m [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/09/23 09:43:17.807[0m Jan 9 09:43:17.807: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-qxg7b" in namespace "azuredisk-5356" to be "Succeeded or Failed" Jan 9 09:43:17.865: INFO: Pod "azuredisk-volume-tester-qxg7b": Phase="Pending", Reason="", readiness=false. Elapsed: 57.52375ms Jan 9 09:43:19.923: INFO: Pod "azuredisk-volume-tester-qxg7b": Phase="Pending", Reason="", readiness=false. Elapsed: 2.115534679s Jan 9 09:43:21.922: INFO: Pod "azuredisk-volume-tester-qxg7b": Phase="Pending", Reason="", readiness=false. Elapsed: 4.115254104s Jan 9 09:43:23.923: INFO: Pod "azuredisk-volume-tester-qxg7b": Phase="Pending", Reason="", readiness=false. Elapsed: 6.115723661s Jan 9 09:43:25.924: INFO: Pod "azuredisk-volume-tester-qxg7b": Phase="Pending", Reason="", readiness=false. Elapsed: 8.116618048s Jan 9 09:43:27.924: INFO: Pod "azuredisk-volume-tester-qxg7b": Phase="Pending", Reason="", readiness=false. Elapsed: 10.11649832s ... skipping 2 lines ... Jan 9 09:43:33.923: INFO: Pod "azuredisk-volume-tester-qxg7b": Phase="Pending", Reason="", readiness=false. Elapsed: 16.115981924s Jan 9 09:43:35.924: INFO: Pod "azuredisk-volume-tester-qxg7b": Phase="Pending", Reason="", readiness=false. Elapsed: 18.116727724s Jan 9 09:43:37.926: INFO: Pod "azuredisk-volume-tester-qxg7b": Phase="Pending", Reason="", readiness=false. Elapsed: 20.118561688s Jan 9 09:43:39.923: INFO: Pod "azuredisk-volume-tester-qxg7b": Phase="Pending", Reason="", readiness=false. Elapsed: 22.115995419s Jan 9 09:43:41.924: INFO: Pod "azuredisk-volume-tester-qxg7b": Phase="Succeeded", Reason="", readiness=false. Elapsed: 24.116865426s [1mSTEP:[0m Saw pod success [38;5;243m01/09/23 09:43:41.924[0m Jan 9 09:43:41.924: INFO: Pod "azuredisk-volume-tester-qxg7b" satisfied condition "Succeeded or Failed" Jan 9 09:43:41.925: INFO: deleting Pod "azuredisk-5356"/"azuredisk-volume-tester-qxg7b" Jan 9 09:43:41.986: INFO: Pod azuredisk-volume-tester-qxg7b has the following logs: hello world [1mSTEP:[0m Deleting pod azuredisk-volume-tester-qxg7b in namespace azuredisk-5356 [38;5;243m01/09/23 09:43:41.986[0m [1mSTEP:[0m validating provisioned PV [38;5;243m01/09/23 09:43:42.108[0m [1mSTEP:[0m checking the PV [38;5;243m01/09/23 09:43:42.165[0m ... skipping 44 lines ... [1mSTEP:[0m creating volume in external rg azuredisk-csi-driver-test-3d342f14-9002-11ed-9173-829d2d0c1a9d [38;5;243m01/09/23 09:44:41.407[0m [1mSTEP:[0m setting up the StorageClass [38;5;243m01/09/23 09:44:41.407[0m [1mSTEP:[0m creating a StorageClass [38;5;243m01/09/23 09:44:41.407[0m [1mSTEP:[0m setting up the PVC and PV [38;5;243m01/09/23 09:44:41.468[0m [1mSTEP:[0m creating a PVC [38;5;243m01/09/23 09:44:41.468[0m [1mSTEP:[0m deploying the pod [38;5;243m01/09/23 09:44:41.532[0m [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/09/23 09:44:41.592[0m Jan 9 09:44:41.592: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-k2xqw" in namespace "azuredisk-5194" to be "Succeeded or Failed" Jan 9 09:44:41.649: INFO: Pod "azuredisk-volume-tester-k2xqw": Phase="Pending", Reason="", readiness=false. Elapsed: 56.665452ms Jan 9 09:44:43.708: INFO: Pod "azuredisk-volume-tester-k2xqw": Phase="Pending", Reason="", readiness=false. Elapsed: 2.116178102s Jan 9 09:44:45.708: INFO: Pod "azuredisk-volume-tester-k2xqw": Phase="Pending", Reason="", readiness=false. Elapsed: 4.116050077s Jan 9 09:44:47.708: INFO: Pod "azuredisk-volume-tester-k2xqw": Phase="Pending", Reason="", readiness=false. Elapsed: 6.115702568s Jan 9 09:44:49.708: INFO: Pod "azuredisk-volume-tester-k2xqw": Phase="Pending", Reason="", readiness=false. Elapsed: 8.116178045s Jan 9 09:44:51.709: INFO: Pod "azuredisk-volume-tester-k2xqw": Phase="Pending", Reason="", readiness=false. Elapsed: 10.116667654s ... skipping 9 lines ... Jan 9 09:45:11.707: INFO: Pod "azuredisk-volume-tester-k2xqw": Phase="Pending", Reason="", readiness=false. Elapsed: 30.115193141s Jan 9 09:45:13.708: INFO: Pod "azuredisk-volume-tester-k2xqw": Phase="Pending", Reason="", readiness=false. Elapsed: 32.116209402s Jan 9 09:45:15.707: INFO: Pod "azuredisk-volume-tester-k2xqw": Phase="Pending", Reason="", readiness=false. Elapsed: 34.115175049s Jan 9 09:45:17.708: INFO: Pod "azuredisk-volume-tester-k2xqw": Phase="Pending", Reason="", readiness=false. Elapsed: 36.11579017s Jan 9 09:45:19.706: INFO: Pod "azuredisk-volume-tester-k2xqw": Phase="Succeeded", Reason="", readiness=false. Elapsed: 38.114478807s [1mSTEP:[0m Saw pod success [38;5;243m01/09/23 09:45:19.706[0m Jan 9 09:45:19.707: INFO: Pod "azuredisk-volume-tester-k2xqw" satisfied condition "Succeeded or Failed" Jan 9 09:45:19.707: INFO: deleting Pod "azuredisk-5194"/"azuredisk-volume-tester-k2xqw" Jan 9 09:45:19.796: INFO: Pod azuredisk-volume-tester-k2xqw has the following logs: hello world hello world [1mSTEP:[0m Deleting pod azuredisk-volume-tester-k2xqw in namespace azuredisk-5194 [38;5;243m01/09/23 09:45:19.796[0m [1mSTEP:[0m validating provisioned PV [38;5;243m01/09/23 09:45:19.918[0m ... skipping 63 lines ... [1mSTEP:[0m creating volume in external rg azuredisk-csi-driver-test-3d342f14-9002-11ed-9173-829d2d0c1a9d [38;5;243m01/09/23 09:44:41.407[0m [1mSTEP:[0m setting up the StorageClass [38;5;243m01/09/23 09:44:41.407[0m [1mSTEP:[0m creating a StorageClass [38;5;243m01/09/23 09:44:41.407[0m [1mSTEP:[0m setting up the PVC and PV [38;5;243m01/09/23 09:44:41.468[0m [1mSTEP:[0m creating a PVC [38;5;243m01/09/23 09:44:41.468[0m [1mSTEP:[0m deploying the pod [38;5;243m01/09/23 09:44:41.532[0m [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/09/23 09:44:41.592[0m Jan 9 09:44:41.592: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-k2xqw" in namespace "azuredisk-5194" to be "Succeeded or Failed" Jan 9 09:44:41.649: INFO: Pod "azuredisk-volume-tester-k2xqw": Phase="Pending", Reason="", readiness=false. Elapsed: 56.665452ms Jan 9 09:44:43.708: INFO: Pod "azuredisk-volume-tester-k2xqw": Phase="Pending", Reason="", readiness=false. Elapsed: 2.116178102s Jan 9 09:44:45.708: INFO: Pod "azuredisk-volume-tester-k2xqw": Phase="Pending", Reason="", readiness=false. Elapsed: 4.116050077s Jan 9 09:44:47.708: INFO: Pod "azuredisk-volume-tester-k2xqw": Phase="Pending", Reason="", readiness=false. Elapsed: 6.115702568s Jan 9 09:44:49.708: INFO: Pod "azuredisk-volume-tester-k2xqw": Phase="Pending", Reason="", readiness=false. Elapsed: 8.116178045s Jan 9 09:44:51.709: INFO: Pod "azuredisk-volume-tester-k2xqw": Phase="Pending", Reason="", readiness=false. Elapsed: 10.116667654s ... skipping 9 lines ... Jan 9 09:45:11.707: INFO: Pod "azuredisk-volume-tester-k2xqw": Phase="Pending", Reason="", readiness=false. Elapsed: 30.115193141s Jan 9 09:45:13.708: INFO: Pod "azuredisk-volume-tester-k2xqw": Phase="Pending", Reason="", readiness=false. Elapsed: 32.116209402s Jan 9 09:45:15.707: INFO: Pod "azuredisk-volume-tester-k2xqw": Phase="Pending", Reason="", readiness=false. Elapsed: 34.115175049s Jan 9 09:45:17.708: INFO: Pod "azuredisk-volume-tester-k2xqw": Phase="Pending", Reason="", readiness=false. Elapsed: 36.11579017s Jan 9 09:45:19.706: INFO: Pod "azuredisk-volume-tester-k2xqw": Phase="Succeeded", Reason="", readiness=false. Elapsed: 38.114478807s [1mSTEP:[0m Saw pod success [38;5;243m01/09/23 09:45:19.706[0m Jan 9 09:45:19.707: INFO: Pod "azuredisk-volume-tester-k2xqw" satisfied condition "Succeeded or Failed" Jan 9 09:45:19.707: INFO: deleting Pod "azuredisk-5194"/"azuredisk-volume-tester-k2xqw" Jan 9 09:45:19.796: INFO: Pod azuredisk-volume-tester-k2xqw has the following logs: hello world hello world [1mSTEP:[0m Deleting pod azuredisk-volume-tester-k2xqw in namespace azuredisk-5194 [38;5;243m01/09/23 09:45:19.796[0m [1mSTEP:[0m validating provisioned PV [38;5;243m01/09/23 09:45:19.918[0m ... skipping 53 lines ... [1mSTEP:[0m setting up the StorageClass [38;5;243m01/09/23 09:48:15.866[0m [1mSTEP:[0m creating a StorageClass [38;5;243m01/09/23 09:48:15.866[0m [1mSTEP:[0m setting up the PVC and PV [38;5;243m01/09/23 09:48:15.921[0m [1mSTEP:[0m creating a PVC [38;5;243m01/09/23 09:48:15.921[0m [1mSTEP:[0m setting up the pod [38;5;243m01/09/23 09:48:15.979[0m [1mSTEP:[0m deploying the pod [38;5;243m01/09/23 09:48:15.98[0m [1mSTEP:[0m checking that the pod's command exits with an error [38;5;243m01/09/23 09:48:16.036[0m Jan 9 09:48:16.036: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-rdvzh" in namespace "azuredisk-1353" to be "Error status code" Jan 9 09:48:16.091: INFO: Pod "azuredisk-volume-tester-rdvzh": Phase="Pending", Reason="", readiness=false. Elapsed: 55.469907ms Jan 9 09:48:18.147: INFO: Pod "azuredisk-volume-tester-rdvzh": Phase="Pending", Reason="", readiness=false. Elapsed: 2.110742998s Jan 9 09:48:20.149: INFO: Pod "azuredisk-volume-tester-rdvzh": Phase="Pending", Reason="", readiness=false. Elapsed: 4.112727078s Jan 9 09:48:22.148: INFO: Pod "azuredisk-volume-tester-rdvzh": Phase="Pending", Reason="", readiness=false. Elapsed: 6.111599542s Jan 9 09:48:24.147: INFO: Pod "azuredisk-volume-tester-rdvzh": Phase="Pending", Reason="", readiness=false. Elapsed: 8.110561818s Jan 9 09:48:26.148: INFO: Pod "azuredisk-volume-tester-rdvzh": Phase="Pending", Reason="", readiness=false. Elapsed: 10.111818738s ... skipping 24 lines ... Jan 9 09:49:16.148: INFO: Pod "azuredisk-volume-tester-rdvzh": Phase="Pending", Reason="", readiness=false. Elapsed: 1m0.112272231s Jan 9 09:49:18.148: INFO: Pod "azuredisk-volume-tester-rdvzh": Phase="Pending", Reason="", readiness=false. Elapsed: 1m2.11242951s Jan 9 09:49:20.149: INFO: Pod "azuredisk-volume-tester-rdvzh": Phase="Pending", Reason="", readiness=false. Elapsed: 1m4.113488726s Jan 9 09:49:22.148: INFO: Pod "azuredisk-volume-tester-rdvzh": Phase="Pending", Reason="", readiness=false. Elapsed: 1m6.1124552s Jan 9 09:49:24.149: INFO: Pod "azuredisk-volume-tester-rdvzh": Phase="Pending", Reason="", readiness=false. Elapsed: 1m8.112742704s Jan 9 09:49:26.155: INFO: Pod "azuredisk-volume-tester-rdvzh": Phase="Pending", Reason="", readiness=false. Elapsed: 1m10.118808877s Jan 9 09:49:28.151: INFO: Pod "azuredisk-volume-tester-rdvzh": Phase="Failed", Reason="", readiness=false. Elapsed: 1m12.114685886s [1mSTEP:[0m Saw pod failure [38;5;243m01/09/23 09:49:28.151[0m Jan 9 09:49:28.151: INFO: Pod "azuredisk-volume-tester-rdvzh" satisfied condition "Error status code" [1mSTEP:[0m checking that pod logs contain expected message [38;5;243m01/09/23 09:49:28.151[0m Jan 9 09:49:28.267: INFO: deleting Pod "azuredisk-1353"/"azuredisk-volume-tester-rdvzh" Jan 9 09:49:28.336: INFO: Pod azuredisk-volume-tester-rdvzh has the following logs: touch: /mnt/test-1/data: Read-only file system [1mSTEP:[0m Deleting pod azuredisk-volume-tester-rdvzh in namespace azuredisk-1353 [38;5;243m01/09/23 09:49:28.336[0m [1mSTEP:[0m validating provisioned PV [38;5;243m01/09/23 09:49:28.452[0m ... skipping 34 lines ... [1mSTEP:[0m setting up the StorageClass [38;5;243m01/09/23 09:48:15.866[0m [1mSTEP:[0m creating a StorageClass [38;5;243m01/09/23 09:48:15.866[0m [1mSTEP:[0m setting up the PVC and PV [38;5;243m01/09/23 09:48:15.921[0m [1mSTEP:[0m creating a PVC [38;5;243m01/09/23 09:48:15.921[0m [1mSTEP:[0m setting up the pod [38;5;243m01/09/23 09:48:15.979[0m [1mSTEP:[0m deploying the pod [38;5;243m01/09/23 09:48:15.98[0m [1mSTEP:[0m checking that the pod's command exits with an error [38;5;243m01/09/23 09:48:16.036[0m Jan 9 09:48:16.036: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-rdvzh" in namespace "azuredisk-1353" to be "Error status code" Jan 9 09:48:16.091: INFO: Pod "azuredisk-volume-tester-rdvzh": Phase="Pending", Reason="", readiness=false. Elapsed: 55.469907ms Jan 9 09:48:18.147: INFO: Pod "azuredisk-volume-tester-rdvzh": Phase="Pending", Reason="", readiness=false. Elapsed: 2.110742998s Jan 9 09:48:20.149: INFO: Pod "azuredisk-volume-tester-rdvzh": Phase="Pending", Reason="", readiness=false. Elapsed: 4.112727078s Jan 9 09:48:22.148: INFO: Pod "azuredisk-volume-tester-rdvzh": Phase="Pending", Reason="", readiness=false. Elapsed: 6.111599542s Jan 9 09:48:24.147: INFO: Pod "azuredisk-volume-tester-rdvzh": Phase="Pending", Reason="", readiness=false. Elapsed: 8.110561818s Jan 9 09:48:26.148: INFO: Pod "azuredisk-volume-tester-rdvzh": Phase="Pending", Reason="", readiness=false. Elapsed: 10.111818738s ... skipping 24 lines ... Jan 9 09:49:16.148: INFO: Pod "azuredisk-volume-tester-rdvzh": Phase="Pending", Reason="", readiness=false. Elapsed: 1m0.112272231s Jan 9 09:49:18.148: INFO: Pod "azuredisk-volume-tester-rdvzh": Phase="Pending", Reason="", readiness=false. Elapsed: 1m2.11242951s Jan 9 09:49:20.149: INFO: Pod "azuredisk-volume-tester-rdvzh": Phase="Pending", Reason="", readiness=false. Elapsed: 1m4.113488726s Jan 9 09:49:22.148: INFO: Pod "azuredisk-volume-tester-rdvzh": Phase="Pending", Reason="", readiness=false. Elapsed: 1m6.1124552s Jan 9 09:49:24.149: INFO: Pod "azuredisk-volume-tester-rdvzh": Phase="Pending", Reason="", readiness=false. Elapsed: 1m8.112742704s Jan 9 09:49:26.155: INFO: Pod "azuredisk-volume-tester-rdvzh": Phase="Pending", Reason="", readiness=false. Elapsed: 1m10.118808877s Jan 9 09:49:28.151: INFO: Pod "azuredisk-volume-tester-rdvzh": Phase="Failed", Reason="", readiness=false. Elapsed: 1m12.114685886s [1mSTEP:[0m Saw pod failure [38;5;243m01/09/23 09:49:28.151[0m Jan 9 09:49:28.151: INFO: Pod "azuredisk-volume-tester-rdvzh" satisfied condition "Error status code" [1mSTEP:[0m checking that pod logs contain expected message [38;5;243m01/09/23 09:49:28.151[0m Jan 9 09:49:28.267: INFO: deleting Pod "azuredisk-1353"/"azuredisk-volume-tester-rdvzh" Jan 9 09:49:28.336: INFO: Pod azuredisk-volume-tester-rdvzh has the following logs: touch: /mnt/test-1/data: Read-only file system [1mSTEP:[0m Deleting pod azuredisk-volume-tester-rdvzh in namespace azuredisk-1353 [38;5;243m01/09/23 09:49:28.336[0m [1mSTEP:[0m validating provisioned PV [38;5;243m01/09/23 09:49:28.452[0m ... skipping 659 lines ... [1mSTEP:[0m setting up the StorageClass [38;5;243m01/09/23 09:57:37.01[0m [1mSTEP:[0m creating a StorageClass [38;5;243m01/09/23 09:57:37.01[0m [1mSTEP:[0m setting up the PVC and PV [38;5;243m01/09/23 09:57:37.066[0m [1mSTEP:[0m creating a PVC [38;5;243m01/09/23 09:57:37.066[0m [1mSTEP:[0m setting up the pod [38;5;243m01/09/23 09:57:37.125[0m [1mSTEP:[0m deploying the pod [38;5;243m01/09/23 09:57:37.125[0m [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/09/23 09:57:37.186[0m Jan 9 09:57:37.186: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-svr8v" in namespace "azuredisk-59" to be "Succeeded or Failed" Jan 9 09:57:37.240: INFO: Pod "azuredisk-volume-tester-svr8v": Phase="Pending", Reason="", readiness=false. Elapsed: 53.710144ms Jan 9 09:57:39.295: INFO: Pod "azuredisk-volume-tester-svr8v": Phase="Pending", Reason="", readiness=false. Elapsed: 2.108539799s Jan 9 09:57:41.296: INFO: Pod "azuredisk-volume-tester-svr8v": Phase="Pending", Reason="", readiness=false. Elapsed: 4.109618877s Jan 9 09:57:43.296: INFO: Pod "azuredisk-volume-tester-svr8v": Phase="Pending", Reason="", readiness=false. Elapsed: 6.109758568s Jan 9 09:57:45.295: INFO: Pod "azuredisk-volume-tester-svr8v": Phase="Pending", Reason="", readiness=false. Elapsed: 8.108276148s Jan 9 09:57:47.296: INFO: Pod "azuredisk-volume-tester-svr8v": Phase="Pending", Reason="", readiness=false. Elapsed: 10.109288768s ... skipping 2 lines ... Jan 9 09:57:53.295: INFO: Pod "azuredisk-volume-tester-svr8v": Phase="Pending", Reason="", readiness=false. Elapsed: 16.108792177s Jan 9 09:57:55.295: INFO: Pod "azuredisk-volume-tester-svr8v": Phase="Pending", Reason="", readiness=false. Elapsed: 18.109138838s Jan 9 09:57:57.296: INFO: Pod "azuredisk-volume-tester-svr8v": Phase="Pending", Reason="", readiness=false. Elapsed: 20.109880573s Jan 9 09:57:59.298: INFO: Pod "azuredisk-volume-tester-svr8v": Phase="Pending", Reason="", readiness=false. Elapsed: 22.111233673s Jan 9 09:58:01.295: INFO: Pod "azuredisk-volume-tester-svr8v": Phase="Succeeded", Reason="", readiness=false. Elapsed: 24.108629489s [1mSTEP:[0m Saw pod success [38;5;243m01/09/23 09:58:01.295[0m Jan 9 09:58:01.295: INFO: Pod "azuredisk-volume-tester-svr8v" satisfied condition "Succeeded or Failed" [1mSTEP:[0m sleep 5s and then clone volume [38;5;243m01/09/23 09:58:01.295[0m [1mSTEP:[0m cloning existing volume [38;5;243m01/09/23 09:58:06.297[0m [1mSTEP:[0m setting up the PVC and PV [38;5;243m01/09/23 09:58:06.407[0m [1mSTEP:[0m creating a PVC [38;5;243m01/09/23 09:58:06.407[0m [1mSTEP:[0m setting up the pod [38;5;243m01/09/23 09:58:06.471[0m [1mSTEP:[0m deploying a second pod with cloned volume [38;5;243m01/09/23 09:58:06.471[0m [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/09/23 09:58:06.532[0m Jan 9 09:58:06.532: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-j49b6" in namespace "azuredisk-59" to be "Succeeded or Failed" Jan 9 09:58:06.587: INFO: Pod "azuredisk-volume-tester-j49b6": Phase="Pending", Reason="", readiness=false. Elapsed: 54.124296ms Jan 9 09:58:08.643: INFO: Pod "azuredisk-volume-tester-j49b6": Phase="Pending", Reason="", readiness=false. Elapsed: 2.110514259s Jan 9 09:58:10.641: INFO: Pod "azuredisk-volume-tester-j49b6": Phase="Pending", Reason="", readiness=false. Elapsed: 4.108959736s Jan 9 09:58:12.645: INFO: Pod "azuredisk-volume-tester-j49b6": Phase="Pending", Reason="", readiness=false. Elapsed: 6.112673436s Jan 9 09:58:14.643: INFO: Pod "azuredisk-volume-tester-j49b6": Phase="Pending", Reason="", readiness=false. Elapsed: 8.110544937s Jan 9 09:58:16.644: INFO: Pod "azuredisk-volume-tester-j49b6": Phase="Pending", Reason="", readiness=false. Elapsed: 10.111773068s ... skipping 26 lines ... Jan 9 09:59:10.642: INFO: Pod "azuredisk-volume-tester-j49b6": Phase="Pending", Reason="", readiness=false. Elapsed: 1m4.10907698s Jan 9 09:59:12.643: INFO: Pod "azuredisk-volume-tester-j49b6": Phase="Pending", Reason="", readiness=false. Elapsed: 1m6.110568858s Jan 9 09:59:14.642: INFO: Pod "azuredisk-volume-tester-j49b6": Phase="Pending", Reason="", readiness=false. Elapsed: 1m8.109289918s Jan 9 09:59:16.643: INFO: Pod "azuredisk-volume-tester-j49b6": Phase="Pending", Reason="", readiness=false. Elapsed: 1m10.110743811s Jan 9 09:59:18.643: INFO: Pod "azuredisk-volume-tester-j49b6": Phase="Succeeded", Reason="", readiness=false. Elapsed: 1m12.110158892s [1mSTEP:[0m Saw pod success [38;5;243m01/09/23 09:59:18.643[0m Jan 9 09:59:18.643: INFO: Pod "azuredisk-volume-tester-j49b6" satisfied condition "Succeeded or Failed" Jan 9 09:59:18.643: INFO: deleting Pod "azuredisk-59"/"azuredisk-volume-tester-j49b6" Jan 9 09:59:18.727: INFO: Pod azuredisk-volume-tester-j49b6 has the following logs: hello world [1mSTEP:[0m Deleting pod azuredisk-volume-tester-j49b6 in namespace azuredisk-59 [38;5;243m01/09/23 09:59:18.727[0m [1mSTEP:[0m validating provisioned PV [38;5;243m01/09/23 09:59:18.848[0m [1mSTEP:[0m checking the PV [38;5;243m01/09/23 09:59:18.903[0m ... skipping 47 lines ... [1mSTEP:[0m setting up the StorageClass [38;5;243m01/09/23 09:57:37.01[0m [1mSTEP:[0m creating a StorageClass [38;5;243m01/09/23 09:57:37.01[0m [1mSTEP:[0m setting up the PVC and PV [38;5;243m01/09/23 09:57:37.066[0m [1mSTEP:[0m creating a PVC [38;5;243m01/09/23 09:57:37.066[0m [1mSTEP:[0m setting up the pod [38;5;243m01/09/23 09:57:37.125[0m [1mSTEP:[0m deploying the pod [38;5;243m01/09/23 09:57:37.125[0m [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/09/23 09:57:37.186[0m Jan 9 09:57:37.186: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-svr8v" in namespace "azuredisk-59" to be "Succeeded or Failed" Jan 9 09:57:37.240: INFO: Pod "azuredisk-volume-tester-svr8v": Phase="Pending", Reason="", readiness=false. Elapsed: 53.710144ms Jan 9 09:57:39.295: INFO: Pod "azuredisk-volume-tester-svr8v": Phase="Pending", Reason="", readiness=false. Elapsed: 2.108539799s Jan 9 09:57:41.296: INFO: Pod "azuredisk-volume-tester-svr8v": Phase="Pending", Reason="", readiness=false. Elapsed: 4.109618877s Jan 9 09:57:43.296: INFO: Pod "azuredisk-volume-tester-svr8v": Phase="Pending", Reason="", readiness=false. Elapsed: 6.109758568s Jan 9 09:57:45.295: INFO: Pod "azuredisk-volume-tester-svr8v": Phase="Pending", Reason="", readiness=false. Elapsed: 8.108276148s Jan 9 09:57:47.296: INFO: Pod "azuredisk-volume-tester-svr8v": Phase="Pending", Reason="", readiness=false. Elapsed: 10.109288768s ... skipping 2 lines ... Jan 9 09:57:53.295: INFO: Pod "azuredisk-volume-tester-svr8v": Phase="Pending", Reason="", readiness=false. Elapsed: 16.108792177s Jan 9 09:57:55.295: INFO: Pod "azuredisk-volume-tester-svr8v": Phase="Pending", Reason="", readiness=false. Elapsed: 18.109138838s Jan 9 09:57:57.296: INFO: Pod "azuredisk-volume-tester-svr8v": Phase="Pending", Reason="", readiness=false. Elapsed: 20.109880573s Jan 9 09:57:59.298: INFO: Pod "azuredisk-volume-tester-svr8v": Phase="Pending", Reason="", readiness=false. Elapsed: 22.111233673s Jan 9 09:58:01.295: INFO: Pod "azuredisk-volume-tester-svr8v": Phase="Succeeded", Reason="", readiness=false. Elapsed: 24.108629489s [1mSTEP:[0m Saw pod success [38;5;243m01/09/23 09:58:01.295[0m Jan 9 09:58:01.295: INFO: Pod "azuredisk-volume-tester-svr8v" satisfied condition "Succeeded or Failed" [1mSTEP:[0m sleep 5s and then clone volume [38;5;243m01/09/23 09:58:01.295[0m [1mSTEP:[0m cloning existing volume [38;5;243m01/09/23 09:58:06.297[0m [1mSTEP:[0m setting up the PVC and PV [38;5;243m01/09/23 09:58:06.407[0m [1mSTEP:[0m creating a PVC [38;5;243m01/09/23 09:58:06.407[0m [1mSTEP:[0m setting up the pod [38;5;243m01/09/23 09:58:06.471[0m [1mSTEP:[0m deploying a second pod with cloned volume [38;5;243m01/09/23 09:58:06.471[0m [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/09/23 09:58:06.532[0m Jan 9 09:58:06.532: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-j49b6" in namespace "azuredisk-59" to be "Succeeded or Failed" Jan 9 09:58:06.587: INFO: Pod "azuredisk-volume-tester-j49b6": Phase="Pending", Reason="", readiness=false. Elapsed: 54.124296ms Jan 9 09:58:08.643: INFO: Pod "azuredisk-volume-tester-j49b6": Phase="Pending", Reason="", readiness=false. Elapsed: 2.110514259s Jan 9 09:58:10.641: INFO: Pod "azuredisk-volume-tester-j49b6": Phase="Pending", Reason="", readiness=false. Elapsed: 4.108959736s Jan 9 09:58:12.645: INFO: Pod "azuredisk-volume-tester-j49b6": Phase="Pending", Reason="", readiness=false. Elapsed: 6.112673436s Jan 9 09:58:14.643: INFO: Pod "azuredisk-volume-tester-j49b6": Phase="Pending", Reason="", readiness=false. Elapsed: 8.110544937s Jan 9 09:58:16.644: INFO: Pod "azuredisk-volume-tester-j49b6": Phase="Pending", Reason="", readiness=false. Elapsed: 10.111773068s ... skipping 26 lines ... Jan 9 09:59:10.642: INFO: Pod "azuredisk-volume-tester-j49b6": Phase="Pending", Reason="", readiness=false. Elapsed: 1m4.10907698s Jan 9 09:59:12.643: INFO: Pod "azuredisk-volume-tester-j49b6": Phase="Pending", Reason="", readiness=false. Elapsed: 1m6.110568858s Jan 9 09:59:14.642: INFO: Pod "azuredisk-volume-tester-j49b6": Phase="Pending", Reason="", readiness=false. Elapsed: 1m8.109289918s Jan 9 09:59:16.643: INFO: Pod "azuredisk-volume-tester-j49b6": Phase="Pending", Reason="", readiness=false. Elapsed: 1m10.110743811s Jan 9 09:59:18.643: INFO: Pod "azuredisk-volume-tester-j49b6": Phase="Succeeded", Reason="", readiness=false. Elapsed: 1m12.110158892s [1mSTEP:[0m Saw pod success [38;5;243m01/09/23 09:59:18.643[0m Jan 9 09:59:18.643: INFO: Pod "azuredisk-volume-tester-j49b6" satisfied condition "Succeeded or Failed" Jan 9 09:59:18.643: INFO: deleting Pod "azuredisk-59"/"azuredisk-volume-tester-j49b6" Jan 9 09:59:18.727: INFO: Pod azuredisk-volume-tester-j49b6 has the following logs: hello world [1mSTEP:[0m Deleting pod azuredisk-volume-tester-j49b6 in namespace azuredisk-59 [38;5;243m01/09/23 09:59:18.727[0m [1mSTEP:[0m validating provisioned PV [38;5;243m01/09/23 09:59:18.848[0m [1mSTEP:[0m checking the PV [38;5;243m01/09/23 09:59:18.903[0m ... skipping 46 lines ... [1mSTEP:[0m setting up the StorageClass [38;5;243m01/09/23 10:00:11.062[0m [1mSTEP:[0m creating a StorageClass [38;5;243m01/09/23 10:00:11.062[0m [1mSTEP:[0m setting up the PVC and PV [38;5;243m01/09/23 10:00:11.117[0m [1mSTEP:[0m creating a PVC [38;5;243m01/09/23 10:00:11.117[0m [1mSTEP:[0m setting up the pod [38;5;243m01/09/23 10:00:11.177[0m [1mSTEP:[0m deploying the pod [38;5;243m01/09/23 10:00:11.177[0m [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/09/23 10:00:11.234[0m Jan 9 10:00:11.234: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-d9j5b" in namespace "azuredisk-2546" to be "Succeeded or Failed" Jan 9 10:00:11.289: INFO: Pod "azuredisk-volume-tester-d9j5b": Phase="Pending", Reason="", readiness=false. Elapsed: 55.624099ms Jan 9 10:00:13.345: INFO: Pod "azuredisk-volume-tester-d9j5b": Phase="Pending", Reason="", readiness=false. Elapsed: 2.111440782s Jan 9 10:00:15.346: INFO: Pod "azuredisk-volume-tester-d9j5b": Phase="Pending", Reason="", readiness=false. Elapsed: 4.111782699s Jan 9 10:00:17.345: INFO: Pod "azuredisk-volume-tester-d9j5b": Phase="Pending", Reason="", readiness=false. Elapsed: 6.111509179s Jan 9 10:00:19.347: INFO: Pod "azuredisk-volume-tester-d9j5b": Phase="Pending", Reason="", readiness=false. Elapsed: 8.112747959s Jan 9 10:00:21.345: INFO: Pod "azuredisk-volume-tester-d9j5b": Phase="Pending", Reason="", readiness=false. Elapsed: 10.110862923s ... skipping 2 lines ... Jan 9 10:00:27.349: INFO: Pod "azuredisk-volume-tester-d9j5b": Phase="Pending", Reason="", readiness=false. Elapsed: 16.115031615s Jan 9 10:00:29.349: INFO: Pod "azuredisk-volume-tester-d9j5b": Phase="Pending", Reason="", readiness=false. Elapsed: 18.114922147s Jan 9 10:00:31.344: INFO: Pod "azuredisk-volume-tester-d9j5b": Phase="Pending", Reason="", readiness=false. Elapsed: 20.110573041s Jan 9 10:00:33.350: INFO: Pod "azuredisk-volume-tester-d9j5b": Phase="Pending", Reason="", readiness=false. Elapsed: 22.115932346s Jan 9 10:00:35.346: INFO: Pod "azuredisk-volume-tester-d9j5b": Phase="Succeeded", Reason="", readiness=false. Elapsed: 24.111694181s [1mSTEP:[0m Saw pod success [38;5;243m01/09/23 10:00:35.346[0m Jan 9 10:00:35.346: INFO: Pod "azuredisk-volume-tester-d9j5b" satisfied condition "Succeeded or Failed" [1mSTEP:[0m sleep 5s and then clone volume [38;5;243m01/09/23 10:00:35.346[0m [1mSTEP:[0m cloning existing volume [38;5;243m01/09/23 10:00:40.346[0m [1mSTEP:[0m setting up the PVC and PV [38;5;243m01/09/23 10:00:40.458[0m [1mSTEP:[0m creating a PVC [38;5;243m01/09/23 10:00:40.458[0m [1mSTEP:[0m setting up the pod [38;5;243m01/09/23 10:00:40.517[0m [1mSTEP:[0m deploying a second pod with cloned volume [38;5;243m01/09/23 10:00:40.517[0m [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/09/23 10:00:40.574[0m Jan 9 10:00:40.575: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-wck89" in namespace "azuredisk-2546" to be "Succeeded or Failed" Jan 9 10:00:40.630: INFO: Pod "azuredisk-volume-tester-wck89": Phase="Pending", Reason="", readiness=false. Elapsed: 55.20471ms Jan 9 10:00:42.685: INFO: Pod "azuredisk-volume-tester-wck89": Phase="Pending", Reason="", readiness=false. Elapsed: 2.110726484s Jan 9 10:00:44.687: INFO: Pod "azuredisk-volume-tester-wck89": Phase="Pending", Reason="", readiness=false. Elapsed: 4.112210675s Jan 9 10:00:46.685: INFO: Pod "azuredisk-volume-tester-wck89": Phase="Pending", Reason="", readiness=false. Elapsed: 6.110825596s Jan 9 10:00:48.687: INFO: Pod "azuredisk-volume-tester-wck89": Phase="Pending", Reason="", readiness=false. Elapsed: 8.112308395s Jan 9 10:00:50.687: INFO: Pod "azuredisk-volume-tester-wck89": Phase="Pending", Reason="", readiness=false. Elapsed: 10.112484531s ... skipping 2 lines ... Jan 9 10:00:56.689: INFO: Pod "azuredisk-volume-tester-wck89": Phase="Pending", Reason="", readiness=false. Elapsed: 16.114598037s Jan 9 10:00:58.686: INFO: Pod "azuredisk-volume-tester-wck89": Phase="Pending", Reason="", readiness=false. Elapsed: 18.111365834s Jan 9 10:01:00.688: INFO: Pod "azuredisk-volume-tester-wck89": Phase="Pending", Reason="", readiness=false. Elapsed: 20.113017818s Jan 9 10:01:02.685: INFO: Pod "azuredisk-volume-tester-wck89": Phase="Pending", Reason="", readiness=false. Elapsed: 22.110920682s Jan 9 10:01:04.685: INFO: Pod "azuredisk-volume-tester-wck89": Phase="Succeeded", Reason="", readiness=false. Elapsed: 24.110833272s [1mSTEP:[0m Saw pod success [38;5;243m01/09/23 10:01:04.685[0m Jan 9 10:01:04.686: INFO: Pod "azuredisk-volume-tester-wck89" satisfied condition "Succeeded or Failed" Jan 9 10:01:04.686: INFO: deleting Pod "azuredisk-2546"/"azuredisk-volume-tester-wck89" Jan 9 10:01:04.753: INFO: Pod azuredisk-volume-tester-wck89 has the following logs: 20.0G [1mSTEP:[0m Deleting pod azuredisk-volume-tester-wck89 in namespace azuredisk-2546 [38;5;243m01/09/23 10:01:04.753[0m [1mSTEP:[0m validating provisioned PV [38;5;243m01/09/23 10:01:04.87[0m [1mSTEP:[0m checking the PV [38;5;243m01/09/23 10:01:04.924[0m ... skipping 47 lines ... [1mSTEP:[0m setting up the StorageClass [38;5;243m01/09/23 10:00:11.062[0m [1mSTEP:[0m creating a StorageClass [38;5;243m01/09/23 10:00:11.062[0m [1mSTEP:[0m setting up the PVC and PV [38;5;243m01/09/23 10:00:11.117[0m [1mSTEP:[0m creating a PVC [38;5;243m01/09/23 10:00:11.117[0m [1mSTEP:[0m setting up the pod [38;5;243m01/09/23 10:00:11.177[0m [1mSTEP:[0m deploying the pod [38;5;243m01/09/23 10:00:11.177[0m [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/09/23 10:00:11.234[0m Jan 9 10:00:11.234: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-d9j5b" in namespace "azuredisk-2546" to be "Succeeded or Failed" Jan 9 10:00:11.289: INFO: Pod "azuredisk-volume-tester-d9j5b": Phase="Pending", Reason="", readiness=false. Elapsed: 55.624099ms Jan 9 10:00:13.345: INFO: Pod "azuredisk-volume-tester-d9j5b": Phase="Pending", Reason="", readiness=false. Elapsed: 2.111440782s Jan 9 10:00:15.346: INFO: Pod "azuredisk-volume-tester-d9j5b": Phase="Pending", Reason="", readiness=false. Elapsed: 4.111782699s Jan 9 10:00:17.345: INFO: Pod "azuredisk-volume-tester-d9j5b": Phase="Pending", Reason="", readiness=false. Elapsed: 6.111509179s Jan 9 10:00:19.347: INFO: Pod "azuredisk-volume-tester-d9j5b": Phase="Pending", Reason="", readiness=false. Elapsed: 8.112747959s Jan 9 10:00:21.345: INFO: Pod "azuredisk-volume-tester-d9j5b": Phase="Pending", Reason="", readiness=false. Elapsed: 10.110862923s ... skipping 2 lines ... Jan 9 10:00:27.349: INFO: Pod "azuredisk-volume-tester-d9j5b": Phase="Pending", Reason="", readiness=false. Elapsed: 16.115031615s Jan 9 10:00:29.349: INFO: Pod "azuredisk-volume-tester-d9j5b": Phase="Pending", Reason="", readiness=false. Elapsed: 18.114922147s Jan 9 10:00:31.344: INFO: Pod "azuredisk-volume-tester-d9j5b": Phase="Pending", Reason="", readiness=false. Elapsed: 20.110573041s Jan 9 10:00:33.350: INFO: Pod "azuredisk-volume-tester-d9j5b": Phase="Pending", Reason="", readiness=false. Elapsed: 22.115932346s Jan 9 10:00:35.346: INFO: Pod "azuredisk-volume-tester-d9j5b": Phase="Succeeded", Reason="", readiness=false. Elapsed: 24.111694181s [1mSTEP:[0m Saw pod success [38;5;243m01/09/23 10:00:35.346[0m Jan 9 10:00:35.346: INFO: Pod "azuredisk-volume-tester-d9j5b" satisfied condition "Succeeded or Failed" [1mSTEP:[0m sleep 5s and then clone volume [38;5;243m01/09/23 10:00:35.346[0m [1mSTEP:[0m cloning existing volume [38;5;243m01/09/23 10:00:40.346[0m [1mSTEP:[0m setting up the PVC and PV [38;5;243m01/09/23 10:00:40.458[0m [1mSTEP:[0m creating a PVC [38;5;243m01/09/23 10:00:40.458[0m [1mSTEP:[0m setting up the pod [38;5;243m01/09/23 10:00:40.517[0m [1mSTEP:[0m deploying a second pod with cloned volume [38;5;243m01/09/23 10:00:40.517[0m [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/09/23 10:00:40.574[0m Jan 9 10:00:40.575: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-wck89" in namespace "azuredisk-2546" to be "Succeeded or Failed" Jan 9 10:00:40.630: INFO: Pod "azuredisk-volume-tester-wck89": Phase="Pending", Reason="", readiness=false. Elapsed: 55.20471ms Jan 9 10:00:42.685: INFO: Pod "azuredisk-volume-tester-wck89": Phase="Pending", Reason="", readiness=false. Elapsed: 2.110726484s Jan 9 10:00:44.687: INFO: Pod "azuredisk-volume-tester-wck89": Phase="Pending", Reason="", readiness=false. Elapsed: 4.112210675s Jan 9 10:00:46.685: INFO: Pod "azuredisk-volume-tester-wck89": Phase="Pending", Reason="", readiness=false. Elapsed: 6.110825596s Jan 9 10:00:48.687: INFO: Pod "azuredisk-volume-tester-wck89": Phase="Pending", Reason="", readiness=false. Elapsed: 8.112308395s Jan 9 10:00:50.687: INFO: Pod "azuredisk-volume-tester-wck89": Phase="Pending", Reason="", readiness=false. Elapsed: 10.112484531s ... skipping 2 lines ... Jan 9 10:00:56.689: INFO: Pod "azuredisk-volume-tester-wck89": Phase="Pending", Reason="", readiness=false. Elapsed: 16.114598037s Jan 9 10:00:58.686: INFO: Pod "azuredisk-volume-tester-wck89": Phase="Pending", Reason="", readiness=false. Elapsed: 18.111365834s Jan 9 10:01:00.688: INFO: Pod "azuredisk-volume-tester-wck89": Phase="Pending", Reason="", readiness=false. Elapsed: 20.113017818s Jan 9 10:01:02.685: INFO: Pod "azuredisk-volume-tester-wck89": Phase="Pending", Reason="", readiness=false. Elapsed: 22.110920682s Jan 9 10:01:04.685: INFO: Pod "azuredisk-volume-tester-wck89": Phase="Succeeded", Reason="", readiness=false. Elapsed: 24.110833272s [1mSTEP:[0m Saw pod success [38;5;243m01/09/23 10:01:04.685[0m Jan 9 10:01:04.686: INFO: Pod "azuredisk-volume-tester-wck89" satisfied condition "Succeeded or Failed" Jan 9 10:01:04.686: INFO: deleting Pod "azuredisk-2546"/"azuredisk-volume-tester-wck89" Jan 9 10:01:04.753: INFO: Pod azuredisk-volume-tester-wck89 has the following logs: 20.0G [1mSTEP:[0m Deleting pod azuredisk-volume-tester-wck89 in namespace azuredisk-2546 [38;5;243m01/09/23 10:01:04.753[0m [1mSTEP:[0m validating provisioned PV [38;5;243m01/09/23 10:01:04.87[0m [1mSTEP:[0m checking the PV [38;5;243m01/09/23 10:01:04.924[0m ... skipping 56 lines ... [1mSTEP:[0m setting up the StorageClass [38;5;243m01/09/23 10:01:57.284[0m [1mSTEP:[0m creating a StorageClass [38;5;243m01/09/23 10:01:57.284[0m [1mSTEP:[0m setting up the PVC and PV [38;5;243m01/09/23 10:01:57.339[0m [1mSTEP:[0m creating a PVC [38;5;243m01/09/23 10:01:57.339[0m [1mSTEP:[0m setting up the pod [38;5;243m01/09/23 10:01:57.397[0m [1mSTEP:[0m deploying the pod [38;5;243m01/09/23 10:01:57.397[0m [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/09/23 10:01:57.458[0m Jan 9 10:01:57.458: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-gj9gz" in namespace "azuredisk-1598" to be "Succeeded or Failed" Jan 9 10:01:57.513: INFO: Pod "azuredisk-volume-tester-gj9gz": Phase="Pending", Reason="", readiness=false. Elapsed: 55.03467ms Jan 9 10:01:59.570: INFO: Pod "azuredisk-volume-tester-gj9gz": Phase="Pending", Reason="", readiness=false. Elapsed: 2.111188688s Jan 9 10:02:01.569: INFO: Pod "azuredisk-volume-tester-gj9gz": Phase="Pending", Reason="", readiness=false. Elapsed: 4.110281803s Jan 9 10:02:03.568: INFO: Pod "azuredisk-volume-tester-gj9gz": Phase="Pending", Reason="", readiness=false. Elapsed: 6.109667163s Jan 9 10:02:05.570: INFO: Pod "azuredisk-volume-tester-gj9gz": Phase="Pending", Reason="", readiness=false. Elapsed: 8.11161149s Jan 9 10:02:07.571: INFO: Pod "azuredisk-volume-tester-gj9gz": Phase="Pending", Reason="", readiness=false. Elapsed: 10.112787451s ... skipping 10 lines ... Jan 9 10:02:29.570: INFO: Pod "azuredisk-volume-tester-gj9gz": Phase="Pending", Reason="", readiness=false. Elapsed: 32.111605105s Jan 9 10:02:31.569: INFO: Pod "azuredisk-volume-tester-gj9gz": Phase="Pending", Reason="", readiness=false. Elapsed: 34.110432055s Jan 9 10:02:33.568: INFO: Pod "azuredisk-volume-tester-gj9gz": Phase="Pending", Reason="", readiness=false. Elapsed: 36.109755419s Jan 9 10:02:35.568: INFO: Pod "azuredisk-volume-tester-gj9gz": Phase="Pending", Reason="", readiness=false. Elapsed: 38.109922697s Jan 9 10:02:37.570: INFO: Pod "azuredisk-volume-tester-gj9gz": Phase="Succeeded", Reason="", readiness=false. Elapsed: 40.111292302s [1mSTEP:[0m Saw pod success [38;5;243m01/09/23 10:02:37.57[0m Jan 9 10:02:37.570: INFO: Pod "azuredisk-volume-tester-gj9gz" satisfied condition "Succeeded or Failed" Jan 9 10:02:37.570: INFO: deleting Pod "azuredisk-1598"/"azuredisk-volume-tester-gj9gz" Jan 9 10:02:37.628: INFO: Pod azuredisk-volume-tester-gj9gz has the following logs: hello world hello world hello world [1mSTEP:[0m Deleting pod azuredisk-volume-tester-gj9gz in namespace azuredisk-1598 [38;5;243m01/09/23 10:02:37.628[0m ... skipping 71 lines ... [1mSTEP:[0m setting up the StorageClass [38;5;243m01/09/23 10:01:57.284[0m [1mSTEP:[0m creating a StorageClass [38;5;243m01/09/23 10:01:57.284[0m [1mSTEP:[0m setting up the PVC and PV [38;5;243m01/09/23 10:01:57.339[0m [1mSTEP:[0m creating a PVC [38;5;243m01/09/23 10:01:57.339[0m [1mSTEP:[0m setting up the pod [38;5;243m01/09/23 10:01:57.397[0m [1mSTEP:[0m deploying the pod [38;5;243m01/09/23 10:01:57.397[0m [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/09/23 10:01:57.458[0m Jan 9 10:01:57.458: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-gj9gz" in namespace "azuredisk-1598" to be "Succeeded or Failed" Jan 9 10:01:57.513: INFO: Pod "azuredisk-volume-tester-gj9gz": Phase="Pending", Reason="", readiness=false. Elapsed: 55.03467ms Jan 9 10:01:59.570: INFO: Pod "azuredisk-volume-tester-gj9gz": Phase="Pending", Reason="", readiness=false. Elapsed: 2.111188688s Jan 9 10:02:01.569: INFO: Pod "azuredisk-volume-tester-gj9gz": Phase="Pending", Reason="", readiness=false. Elapsed: 4.110281803s Jan 9 10:02:03.568: INFO: Pod "azuredisk-volume-tester-gj9gz": Phase="Pending", Reason="", readiness=false. Elapsed: 6.109667163s Jan 9 10:02:05.570: INFO: Pod "azuredisk-volume-tester-gj9gz": Phase="Pending", Reason="", readiness=false. Elapsed: 8.11161149s Jan 9 10:02:07.571: INFO: Pod "azuredisk-volume-tester-gj9gz": Phase="Pending", Reason="", readiness=false. Elapsed: 10.112787451s ... skipping 10 lines ... Jan 9 10:02:29.570: INFO: Pod "azuredisk-volume-tester-gj9gz": Phase="Pending", Reason="", readiness=false. Elapsed: 32.111605105s Jan 9 10:02:31.569: INFO: Pod "azuredisk-volume-tester-gj9gz": Phase="Pending", Reason="", readiness=false. Elapsed: 34.110432055s Jan 9 10:02:33.568: INFO: Pod "azuredisk-volume-tester-gj9gz": Phase="Pending", Reason="", readiness=false. Elapsed: 36.109755419s Jan 9 10:02:35.568: INFO: Pod "azuredisk-volume-tester-gj9gz": Phase="Pending", Reason="", readiness=false. Elapsed: 38.109922697s Jan 9 10:02:37.570: INFO: Pod "azuredisk-volume-tester-gj9gz": Phase="Succeeded", Reason="", readiness=false. Elapsed: 40.111292302s [1mSTEP:[0m Saw pod success [38;5;243m01/09/23 10:02:37.57[0m Jan 9 10:02:37.570: INFO: Pod "azuredisk-volume-tester-gj9gz" satisfied condition "Succeeded or Failed" Jan 9 10:02:37.570: INFO: deleting Pod "azuredisk-1598"/"azuredisk-volume-tester-gj9gz" Jan 9 10:02:37.628: INFO: Pod azuredisk-volume-tester-gj9gz has the following logs: hello world hello world hello world [1mSTEP:[0m Deleting pod azuredisk-volume-tester-gj9gz in namespace azuredisk-1598 [38;5;243m01/09/23 10:02:37.628[0m ... skipping 65 lines ... [1mSTEP:[0m setting up the StorageClass [38;5;243m01/09/23 10:03:50.541[0m [1mSTEP:[0m creating a StorageClass [38;5;243m01/09/23 10:03:50.541[0m [1mSTEP:[0m setting up the PVC and PV [38;5;243m01/09/23 10:03:50.596[0m [1mSTEP:[0m creating a PVC [38;5;243m01/09/23 10:03:50.596[0m [1mSTEP:[0m setting up the pod [38;5;243m01/09/23 10:03:50.651[0m [1mSTEP:[0m deploying the pod [38;5;243m01/09/23 10:03:50.651[0m [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/09/23 10:03:50.706[0m Jan 9 10:03:50.706: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-rq4h8" in namespace "azuredisk-3410" to be "Succeeded or Failed" Jan 9 10:03:50.760: INFO: Pod "azuredisk-volume-tester-rq4h8": Phase="Pending", Reason="", readiness=false. Elapsed: 53.516835ms Jan 9 10:03:52.815: INFO: Pod "azuredisk-volume-tester-rq4h8": Phase="Pending", Reason="", readiness=false. Elapsed: 2.108412711s Jan 9 10:03:54.815: INFO: Pod "azuredisk-volume-tester-rq4h8": Phase="Pending", Reason="", readiness=false. Elapsed: 4.108358219s Jan 9 10:03:56.816: INFO: Pod "azuredisk-volume-tester-rq4h8": Phase="Pending", Reason="", readiness=false. Elapsed: 6.109894702s Jan 9 10:03:58.816: INFO: Pod "azuredisk-volume-tester-rq4h8": Phase="Pending", Reason="", readiness=false. Elapsed: 8.109226684s Jan 9 10:04:00.815: INFO: Pod "azuredisk-volume-tester-rq4h8": Phase="Pending", Reason="", readiness=false. Elapsed: 10.108261004s ... skipping 10 lines ... Jan 9 10:04:22.815: INFO: Pod "azuredisk-volume-tester-rq4h8": Phase="Pending", Reason="", readiness=false. Elapsed: 32.108757617s Jan 9 10:04:24.815: INFO: Pod "azuredisk-volume-tester-rq4h8": Phase="Pending", Reason="", readiness=false. Elapsed: 34.109060737s Jan 9 10:04:26.816: INFO: Pod "azuredisk-volume-tester-rq4h8": Phase="Pending", Reason="", readiness=false. Elapsed: 36.110063631s Jan 9 10:04:28.816: INFO: Pod "azuredisk-volume-tester-rq4h8": Phase="Pending", Reason="", readiness=false. Elapsed: 38.109548619s Jan 9 10:04:30.815: INFO: Pod "azuredisk-volume-tester-rq4h8": Phase="Succeeded", Reason="", readiness=false. Elapsed: 40.108887634s [1mSTEP:[0m Saw pod success [38;5;243m01/09/23 10:04:30.815[0m Jan 9 10:04:30.815: INFO: Pod "azuredisk-volume-tester-rq4h8" satisfied condition "Succeeded or Failed" Jan 9 10:04:30.816: INFO: deleting Pod "azuredisk-3410"/"azuredisk-volume-tester-rq4h8" Jan 9 10:04:30.907: INFO: Pod azuredisk-volume-tester-rq4h8 has the following logs: 100+0 records in 100+0 records out 104857600 bytes (100.0MB) copied, 0.078045 seconds, 1.3GB/s hello world ... skipping 59 lines ... [1mSTEP:[0m setting up the StorageClass [38;5;243m01/09/23 10:03:50.541[0m [1mSTEP:[0m creating a StorageClass [38;5;243m01/09/23 10:03:50.541[0m [1mSTEP:[0m setting up the PVC and PV [38;5;243m01/09/23 10:03:50.596[0m [1mSTEP:[0m creating a PVC [38;5;243m01/09/23 10:03:50.596[0m [1mSTEP:[0m setting up the pod [38;5;243m01/09/23 10:03:50.651[0m [1mSTEP:[0m deploying the pod [38;5;243m01/09/23 10:03:50.651[0m [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/09/23 10:03:50.706[0m Jan 9 10:03:50.706: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-rq4h8" in namespace "azuredisk-3410" to be "Succeeded or Failed" Jan 9 10:03:50.760: INFO: Pod "azuredisk-volume-tester-rq4h8": Phase="Pending", Reason="", readiness=false. Elapsed: 53.516835ms Jan 9 10:03:52.815: INFO: Pod "azuredisk-volume-tester-rq4h8": Phase="Pending", Reason="", readiness=false. Elapsed: 2.108412711s Jan 9 10:03:54.815: INFO: Pod "azuredisk-volume-tester-rq4h8": Phase="Pending", Reason="", readiness=false. Elapsed: 4.108358219s Jan 9 10:03:56.816: INFO: Pod "azuredisk-volume-tester-rq4h8": Phase="Pending", Reason="", readiness=false. Elapsed: 6.109894702s Jan 9 10:03:58.816: INFO: Pod "azuredisk-volume-tester-rq4h8": Phase="Pending", Reason="", readiness=false. Elapsed: 8.109226684s Jan 9 10:04:00.815: INFO: Pod "azuredisk-volume-tester-rq4h8": Phase="Pending", Reason="", readiness=false. Elapsed: 10.108261004s ... skipping 10 lines ... Jan 9 10:04:22.815: INFO: Pod "azuredisk-volume-tester-rq4h8": Phase="Pending", Reason="", readiness=false. Elapsed: 32.108757617s Jan 9 10:04:24.815: INFO: Pod "azuredisk-volume-tester-rq4h8": Phase="Pending", Reason="", readiness=false. Elapsed: 34.109060737s Jan 9 10:04:26.816: INFO: Pod "azuredisk-volume-tester-rq4h8": Phase="Pending", Reason="", readiness=false. Elapsed: 36.110063631s Jan 9 10:04:28.816: INFO: Pod "azuredisk-volume-tester-rq4h8": Phase="Pending", Reason="", readiness=false. Elapsed: 38.109548619s Jan 9 10:04:30.815: INFO: Pod "azuredisk-volume-tester-rq4h8": Phase="Succeeded", Reason="", readiness=false. Elapsed: 40.108887634s [1mSTEP:[0m Saw pod success [38;5;243m01/09/23 10:04:30.815[0m Jan 9 10:04:30.815: INFO: Pod "azuredisk-volume-tester-rq4h8" satisfied condition "Succeeded or Failed" Jan 9 10:04:30.816: INFO: deleting Pod "azuredisk-3410"/"azuredisk-volume-tester-rq4h8" Jan 9 10:04:30.907: INFO: Pod azuredisk-volume-tester-rq4h8 has the following logs: 100+0 records in 100+0 records out 104857600 bytes (100.0MB) copied, 0.078045 seconds, 1.3GB/s hello world ... skipping 52 lines ... Jan 9 10:05:53.539: INFO: >>> kubeConfig: /root/tmp2587121816/kubeconfig/kubeconfig.westus2.json [1mSTEP:[0m setting up the StorageClass [38;5;243m01/09/23 10:05:53.54[0m [1mSTEP:[0m creating a StorageClass [38;5;243m01/09/23 10:05:53.54[0m [1mSTEP:[0m setting up the PVC and PV [38;5;243m01/09/23 10:05:53.595[0m [1mSTEP:[0m creating a PVC [38;5;243m01/09/23 10:05:53.596[0m [1mSTEP:[0m deploying the pod [38;5;243m01/09/23 10:05:53.67[0m [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/09/23 10:05:53.726[0m Jan 9 10:05:53.726: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-t5zxh" in namespace "azuredisk-8582" to be "Succeeded or Failed" Jan 9 10:05:53.780: INFO: Pod "azuredisk-volume-tester-t5zxh": Phase="Pending", Reason="", readiness=false. Elapsed: 54.121483ms Jan 9 10:05:55.837: INFO: Pod "azuredisk-volume-tester-t5zxh": Phase="Pending", Reason="", readiness=false. Elapsed: 2.110497395s Jan 9 10:05:57.838: INFO: Pod "azuredisk-volume-tester-t5zxh": Phase="Pending", Reason="", readiness=false. Elapsed: 4.112292598s Jan 9 10:05:59.836: INFO: Pod "azuredisk-volume-tester-t5zxh": Phase="Pending", Reason="", readiness=false. Elapsed: 6.109759811s Jan 9 10:06:01.837: INFO: Pod "azuredisk-volume-tester-t5zxh": Phase="Pending", Reason="", readiness=false. Elapsed: 8.110782987s Jan 9 10:06:03.836: INFO: Pod "azuredisk-volume-tester-t5zxh": Phase="Pending", Reason="", readiness=false. Elapsed: 10.109520798s ... skipping 2 lines ... Jan 9 10:06:09.836: INFO: Pod "azuredisk-volume-tester-t5zxh": Phase="Pending", Reason="", readiness=false. Elapsed: 16.109430222s Jan 9 10:06:11.836: INFO: Pod "azuredisk-volume-tester-t5zxh": Phase="Pending", Reason="", readiness=false. Elapsed: 18.110141981s Jan 9 10:06:13.837: INFO: Pod "azuredisk-volume-tester-t5zxh": Phase="Pending", Reason="", readiness=false. Elapsed: 20.111105896s Jan 9 10:06:15.836: INFO: Pod "azuredisk-volume-tester-t5zxh": Phase="Pending", Reason="", readiness=false. Elapsed: 22.109540333s Jan 9 10:06:17.835: INFO: Pod "azuredisk-volume-tester-t5zxh": Phase="Succeeded", Reason="", readiness=false. Elapsed: 24.108942503s [1mSTEP:[0m Saw pod success [38;5;243m01/09/23 10:06:17.835[0m Jan 9 10:06:17.835: INFO: Pod "azuredisk-volume-tester-t5zxh" satisfied condition "Succeeded or Failed" [1mSTEP:[0m Checking Prow test resource group [38;5;243m01/09/23 10:06:17.835[0m 2023/01/09 10:06:17 Running in Prow, converting AZURE_CREDENTIALS to AZURE_CREDENTIAL_FILE 2023/01/09 10:06:17 Reading credentials file /etc/azure-cred/credentials [1mSTEP:[0m Prow test resource group: kubetest-ht4culj6 [38;5;243m01/09/23 10:06:17.836[0m [1mSTEP:[0m Creating external resource group: azuredisk-csi-driver-test-42352b8f-9005-11ed-9173-829d2d0c1a9d [38;5;243m01/09/23 10:06:17.837[0m [1mSTEP:[0m creating volume snapshot class with external rg azuredisk-csi-driver-test-42352b8f-9005-11ed-9173-829d2d0c1a9d [38;5;243m01/09/23 10:06:19.099[0m ... skipping 5 lines ... [1mSTEP:[0m setting up the StorageClass [38;5;243m01/09/23 10:06:34.285[0m [1mSTEP:[0m creating a StorageClass [38;5;243m01/09/23 10:06:34.285[0m [1mSTEP:[0m setting up the PVC and PV [38;5;243m01/09/23 10:06:34.341[0m [1mSTEP:[0m creating a PVC [38;5;243m01/09/23 10:06:34.341[0m [1mSTEP:[0m setting up the pod [38;5;243m01/09/23 10:06:34.4[0m [1mSTEP:[0m deploying a pod with a volume restored from the snapshot [38;5;243m01/09/23 10:06:34.4[0m [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/09/23 10:06:34.456[0m Jan 9 10:06:34.456: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-94vkb" in namespace "azuredisk-8582" to be "Succeeded or Failed" Jan 9 10:06:34.511: INFO: Pod "azuredisk-volume-tester-94vkb": Phase="Pending", Reason="", readiness=false. Elapsed: 54.628847ms Jan 9 10:06:36.566: INFO: Pod "azuredisk-volume-tester-94vkb": Phase="Pending", Reason="", readiness=false. Elapsed: 2.110327679s Jan 9 10:06:38.568: INFO: Pod "azuredisk-volume-tester-94vkb": Phase="Pending", Reason="", readiness=false. Elapsed: 4.111608131s Jan 9 10:06:40.566: INFO: Pod "azuredisk-volume-tester-94vkb": Phase="Pending", Reason="", readiness=false. Elapsed: 6.110054129s Jan 9 10:06:42.567: INFO: Pod "azuredisk-volume-tester-94vkb": Phase="Pending", Reason="", readiness=false. Elapsed: 8.111359293s Jan 9 10:06:44.566: INFO: Pod "azuredisk-volume-tester-94vkb": Phase="Pending", Reason="", readiness=false. Elapsed: 10.109616118s Jan 9 10:06:46.567: INFO: Pod "azuredisk-volume-tester-94vkb": Phase="Pending", Reason="", readiness=false. Elapsed: 12.111321313s Jan 9 10:06:48.566: INFO: Pod "azuredisk-volume-tester-94vkb": Phase="Pending", Reason="", readiness=false. Elapsed: 14.109577712s Jan 9 10:06:50.567: INFO: Pod "azuredisk-volume-tester-94vkb": Phase="Pending", Reason="", readiness=false. Elapsed: 16.111236624s Jan 9 10:06:52.567: INFO: Pod "azuredisk-volume-tester-94vkb": Phase="Pending", Reason="", readiness=false. Elapsed: 18.110829368s Jan 9 10:06:54.567: INFO: Pod "azuredisk-volume-tester-94vkb": Phase="Pending", Reason="", readiness=false. Elapsed: 20.111353677s Jan 9 10:06:56.567: INFO: Pod "azuredisk-volume-tester-94vkb": Phase="Succeeded", Reason="", readiness=false. Elapsed: 22.111432947s [1mSTEP:[0m Saw pod success [38;5;243m01/09/23 10:06:56.568[0m Jan 9 10:06:56.568: INFO: Pod "azuredisk-volume-tester-94vkb" satisfied condition "Succeeded or Failed" Jan 9 10:06:56.568: INFO: deleting Pod "azuredisk-8582"/"azuredisk-volume-tester-94vkb" Jan 9 10:06:56.661: INFO: Pod azuredisk-volume-tester-94vkb has the following logs: hello world [1mSTEP:[0m Deleting pod azuredisk-volume-tester-94vkb in namespace azuredisk-8582 [38;5;243m01/09/23 10:06:56.661[0m [1mSTEP:[0m validating provisioned PV [38;5;243m01/09/23 10:06:56.778[0m [1mSTEP:[0m checking the PV [38;5;243m01/09/23 10:06:56.832[0m ... skipping 50 lines ... Jan 9 10:05:53.539: INFO: >>> kubeConfig: /root/tmp2587121816/kubeconfig/kubeconfig.westus2.json [1mSTEP:[0m setting up the StorageClass [38;5;243m01/09/23 10:05:53.54[0m [1mSTEP:[0m creating a StorageClass [38;5;243m01/09/23 10:05:53.54[0m [1mSTEP:[0m setting up the PVC and PV [38;5;243m01/09/23 10:05:53.595[0m [1mSTEP:[0m creating a PVC [38;5;243m01/09/23 10:05:53.596[0m [1mSTEP:[0m deploying the pod [38;5;243m01/09/23 10:05:53.67[0m [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/09/23 10:05:53.726[0m Jan 9 10:05:53.726: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-t5zxh" in namespace "azuredisk-8582" to be "Succeeded or Failed" Jan 9 10:05:53.780: INFO: Pod "azuredisk-volume-tester-t5zxh": Phase="Pending", Reason="", readiness=false. Elapsed: 54.121483ms Jan 9 10:05:55.837: INFO: Pod "azuredisk-volume-tester-t5zxh": Phase="Pending", Reason="", readiness=false. Elapsed: 2.110497395s Jan 9 10:05:57.838: INFO: Pod "azuredisk-volume-tester-t5zxh": Phase="Pending", Reason="", readiness=false. Elapsed: 4.112292598s Jan 9 10:05:59.836: INFO: Pod "azuredisk-volume-tester-t5zxh": Phase="Pending", Reason="", readiness=false. Elapsed: 6.109759811s Jan 9 10:06:01.837: INFO: Pod "azuredisk-volume-tester-t5zxh": Phase="Pending", Reason="", readiness=false. Elapsed: 8.110782987s Jan 9 10:06:03.836: INFO: Pod "azuredisk-volume-tester-t5zxh": Phase="Pending", Reason="", readiness=false. Elapsed: 10.109520798s ... skipping 2 lines ... Jan 9 10:06:09.836: INFO: Pod "azuredisk-volume-tester-t5zxh": Phase="Pending", Reason="", readiness=false. Elapsed: 16.109430222s Jan 9 10:06:11.836: INFO: Pod "azuredisk-volume-tester-t5zxh": Phase="Pending", Reason="", readiness=false. Elapsed: 18.110141981s Jan 9 10:06:13.837: INFO: Pod "azuredisk-volume-tester-t5zxh": Phase="Pending", Reason="", readiness=false. Elapsed: 20.111105896s Jan 9 10:06:15.836: INFO: Pod "azuredisk-volume-tester-t5zxh": Phase="Pending", Reason="", readiness=false. Elapsed: 22.109540333s Jan 9 10:06:17.835: INFO: Pod "azuredisk-volume-tester-t5zxh": Phase="Succeeded", Reason="", readiness=false. Elapsed: 24.108942503s [1mSTEP:[0m Saw pod success [38;5;243m01/09/23 10:06:17.835[0m Jan 9 10:06:17.835: INFO: Pod "azuredisk-volume-tester-t5zxh" satisfied condition "Succeeded or Failed" [1mSTEP:[0m Checking Prow test resource group [38;5;243m01/09/23 10:06:17.835[0m [1mSTEP:[0m Prow test resource group: kubetest-ht4culj6 [38;5;243m01/09/23 10:06:17.836[0m [1mSTEP:[0m Creating external resource group: azuredisk-csi-driver-test-42352b8f-9005-11ed-9173-829d2d0c1a9d [38;5;243m01/09/23 10:06:17.837[0m [1mSTEP:[0m creating volume snapshot class with external rg azuredisk-csi-driver-test-42352b8f-9005-11ed-9173-829d2d0c1a9d [38;5;243m01/09/23 10:06:19.099[0m [1mSTEP:[0m setting up the VolumeSnapshotClass [38;5;243m01/09/23 10:06:19.099[0m [1mSTEP:[0m creating a VolumeSnapshotClass [38;5;243m01/09/23 10:06:19.099[0m ... skipping 3 lines ... [1mSTEP:[0m setting up the StorageClass [38;5;243m01/09/23 10:06:34.285[0m [1mSTEP:[0m creating a StorageClass [38;5;243m01/09/23 10:06:34.285[0m [1mSTEP:[0m setting up the PVC and PV [38;5;243m01/09/23 10:06:34.341[0m [1mSTEP:[0m creating a PVC [38;5;243m01/09/23 10:06:34.341[0m [1mSTEP:[0m setting up the pod [38;5;243m01/09/23 10:06:34.4[0m [1mSTEP:[0m deploying a pod with a volume restored from the snapshot [38;5;243m01/09/23 10:06:34.4[0m [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/09/23 10:06:34.456[0m Jan 9 10:06:34.456: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-94vkb" in namespace "azuredisk-8582" to be "Succeeded or Failed" Jan 9 10:06:34.511: INFO: Pod "azuredisk-volume-tester-94vkb": Phase="Pending", Reason="", readiness=false. Elapsed: 54.628847ms Jan 9 10:06:36.566: INFO: Pod "azuredisk-volume-tester-94vkb": Phase="Pending", Reason="", readiness=false. Elapsed: 2.110327679s Jan 9 10:06:38.568: INFO: Pod "azuredisk-volume-tester-94vkb": Phase="Pending", Reason="", readiness=false. Elapsed: 4.111608131s Jan 9 10:06:40.566: INFO: Pod "azuredisk-volume-tester-94vkb": Phase="Pending", Reason="", readiness=false. Elapsed: 6.110054129s Jan 9 10:06:42.567: INFO: Pod "azuredisk-volume-tester-94vkb": Phase="Pending", Reason="", readiness=false. Elapsed: 8.111359293s Jan 9 10:06:44.566: INFO: Pod "azuredisk-volume-tester-94vkb": Phase="Pending", Reason="", readiness=false. Elapsed: 10.109616118s Jan 9 10:06:46.567: INFO: Pod "azuredisk-volume-tester-94vkb": Phase="Pending", Reason="", readiness=false. Elapsed: 12.111321313s Jan 9 10:06:48.566: INFO: Pod "azuredisk-volume-tester-94vkb": Phase="Pending", Reason="", readiness=false. Elapsed: 14.109577712s Jan 9 10:06:50.567: INFO: Pod "azuredisk-volume-tester-94vkb": Phase="Pending", Reason="", readiness=false. Elapsed: 16.111236624s Jan 9 10:06:52.567: INFO: Pod "azuredisk-volume-tester-94vkb": Phase="Pending", Reason="", readiness=false. Elapsed: 18.110829368s Jan 9 10:06:54.567: INFO: Pod "azuredisk-volume-tester-94vkb": Phase="Pending", Reason="", readiness=false. Elapsed: 20.111353677s Jan 9 10:06:56.567: INFO: Pod "azuredisk-volume-tester-94vkb": Phase="Succeeded", Reason="", readiness=false. Elapsed: 22.111432947s [1mSTEP:[0m Saw pod success [38;5;243m01/09/23 10:06:56.568[0m Jan 9 10:06:56.568: INFO: Pod "azuredisk-volume-tester-94vkb" satisfied condition "Succeeded or Failed" Jan 9 10:06:56.568: INFO: deleting Pod "azuredisk-8582"/"azuredisk-volume-tester-94vkb" Jan 9 10:06:56.661: INFO: Pod azuredisk-volume-tester-94vkb has the following logs: hello world [1mSTEP:[0m Deleting pod azuredisk-volume-tester-94vkb in namespace azuredisk-8582 [38;5;243m01/09/23 10:06:56.661[0m [1mSTEP:[0m validating provisioned PV [38;5;243m01/09/23 10:06:56.778[0m [1mSTEP:[0m checking the PV [38;5;243m01/09/23 10:06:56.832[0m ... skipping 49 lines ... Jan 9 10:09:05.595: INFO: >>> kubeConfig: /root/tmp2587121816/kubeconfig/kubeconfig.westus2.json [1mSTEP:[0m setting up the StorageClass [38;5;243m01/09/23 10:09:05.596[0m [1mSTEP:[0m creating a StorageClass [38;5;243m01/09/23 10:09:05.596[0m [1mSTEP:[0m setting up the PVC and PV [38;5;243m01/09/23 10:09:05.653[0m [1mSTEP:[0m creating a PVC [38;5;243m01/09/23 10:09:05.653[0m [1mSTEP:[0m deploying the pod [38;5;243m01/09/23 10:09:05.712[0m [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/09/23 10:09:05.769[0m Jan 9 10:09:05.769: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-ljcdx" in namespace "azuredisk-7726" to be "Succeeded or Failed" Jan 9 10:09:05.823: INFO: Pod "azuredisk-volume-tester-ljcdx": Phase="Pending", Reason="", readiness=false. Elapsed: 54.289432ms Jan 9 10:09:07.879: INFO: Pod "azuredisk-volume-tester-ljcdx": Phase="Pending", Reason="", readiness=false. Elapsed: 2.109656854s Jan 9 10:09:09.881: INFO: Pod "azuredisk-volume-tester-ljcdx": Phase="Pending", Reason="", readiness=false. Elapsed: 4.112354924s Jan 9 10:09:11.881: INFO: Pod "azuredisk-volume-tester-ljcdx": Phase="Pending", Reason="", readiness=false. Elapsed: 6.111880646s Jan 9 10:09:13.881: INFO: Pod "azuredisk-volume-tester-ljcdx": Phase="Pending", Reason="", readiness=false. Elapsed: 8.111845905s Jan 9 10:09:15.878: INFO: Pod "azuredisk-volume-tester-ljcdx": Phase="Pending", Reason="", readiness=false. Elapsed: 10.109166697s ... skipping 2 lines ... Jan 9 10:09:21.879: INFO: Pod "azuredisk-volume-tester-ljcdx": Phase="Pending", Reason="", readiness=false. Elapsed: 16.110050847s Jan 9 10:09:23.878: INFO: Pod "azuredisk-volume-tester-ljcdx": Phase="Pending", Reason="", readiness=false. Elapsed: 18.109138436s Jan 9 10:09:25.880: INFO: Pod "azuredisk-volume-tester-ljcdx": Phase="Pending", Reason="", readiness=false. Elapsed: 20.111544528s Jan 9 10:09:27.879: INFO: Pod "azuredisk-volume-tester-ljcdx": Phase="Pending", Reason="", readiness=false. Elapsed: 22.110543847s Jan 9 10:09:29.887: INFO: Pod "azuredisk-volume-tester-ljcdx": Phase="Succeeded", Reason="", readiness=false. Elapsed: 24.11807628s [1mSTEP:[0m Saw pod success [38;5;243m01/09/23 10:09:29.887[0m Jan 9 10:09:29.887: INFO: Pod "azuredisk-volume-tester-ljcdx" satisfied condition "Succeeded or Failed" [1mSTEP:[0m Checking Prow test resource group [38;5;243m01/09/23 10:09:29.887[0m 2023/01/09 10:09:29 Running in Prow, converting AZURE_CREDENTIALS to AZURE_CREDENTIAL_FILE 2023/01/09 10:09:29 Reading credentials file /etc/azure-cred/credentials [1mSTEP:[0m Prow test resource group: kubetest-ht4culj6 [38;5;243m01/09/23 10:09:29.888[0m [1mSTEP:[0m Creating external resource group: azuredisk-csi-driver-test-b4adeef9-9005-11ed-9173-829d2d0c1a9d [38;5;243m01/09/23 10:09:29.888[0m [1mSTEP:[0m creating volume snapshot class with external rg azuredisk-csi-driver-test-b4adeef9-9005-11ed-9173-829d2d0c1a9d [38;5;243m01/09/23 10:09:30.708[0m ... skipping 12 lines ... [1mSTEP:[0m creating a StorageClass [38;5;243m01/09/23 10:09:48.066[0m [1mSTEP:[0m setting up the PVC and PV [38;5;243m01/09/23 10:09:48.123[0m [1mSTEP:[0m creating a PVC [38;5;243m01/09/23 10:09:48.123[0m [1mSTEP:[0m setting up the pod [38;5;243m01/09/23 10:09:48.182[0m [1mSTEP:[0m Set pod anti-affinity to make sure two pods are scheduled on different nodes [38;5;243m01/09/23 10:09:48.182[0m [1mSTEP:[0m deploying a pod with a volume restored from the snapshot [38;5;243m01/09/23 10:09:48.182[0m [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/09/23 10:09:48.239[0m Jan 9 10:09:48.239: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-dd9cm" in namespace "azuredisk-7726" to be "Succeeded or Failed" Jan 9 10:09:48.293: INFO: Pod "azuredisk-volume-tester-dd9cm": Phase="Pending", Reason="", readiness=false. Elapsed: 54.39833ms Jan 9 10:09:50.351: INFO: Pod "azuredisk-volume-tester-dd9cm": Phase="Pending", Reason="", readiness=false. Elapsed: 2.111911828s Jan 9 10:09:52.354: INFO: Pod "azuredisk-volume-tester-dd9cm": Phase="Pending", Reason="", readiness=false. Elapsed: 4.114738874s Jan 9 10:09:54.349: INFO: Pod "azuredisk-volume-tester-dd9cm": Phase="Pending", Reason="", readiness=false. Elapsed: 6.110140794s Jan 9 10:09:56.351: INFO: Pod "azuredisk-volume-tester-dd9cm": Phase="Pending", Reason="", readiness=false. Elapsed: 8.112313729s Jan 9 10:09:58.351: INFO: Pod "azuredisk-volume-tester-dd9cm": Phase="Pending", Reason="", readiness=false. Elapsed: 10.112122126s Jan 9 10:10:00.350: INFO: Pod "azuredisk-volume-tester-dd9cm": Phase="Pending", Reason="", readiness=false. Elapsed: 12.110775324s Jan 9 10:10:02.351: INFO: Pod "azuredisk-volume-tester-dd9cm": Phase="Pending", Reason="", readiness=false. Elapsed: 14.112216557s Jan 9 10:10:04.348: INFO: Pod "azuredisk-volume-tester-dd9cm": Phase="Pending", Reason="", readiness=false. Elapsed: 16.10931375s Jan 9 10:10:06.348: INFO: Pod "azuredisk-volume-tester-dd9cm": Phase="Pending", Reason="", readiness=false. Elapsed: 18.109665355s Jan 9 10:10:08.350: INFO: Pod "azuredisk-volume-tester-dd9cm": Phase="Pending", Reason="", readiness=false. Elapsed: 20.110841246s Jan 9 10:10:10.349: INFO: Pod "azuredisk-volume-tester-dd9cm": Phase="Pending", Reason="", readiness=false. Elapsed: 22.110613738s Jan 9 10:10:12.350: INFO: Pod "azuredisk-volume-tester-dd9cm": Phase="Failed", Reason="", readiness=false. Elapsed: 24.111330956s Jan 9 10:10:12.351: INFO: Unexpected error: <*fmt.wrapError | 0xc000cccc60>: { msg: "error while waiting for pod azuredisk-7726/azuredisk-volume-tester-dd9cm to be Succeeded or Failed: pod \"azuredisk-volume-tester-dd9cm\" failed with status: {Phase:Failed Conditions:[{Type:Initialized Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2023-01-09 10:09:51 +0000 UTC Reason: Message:} {Type:Ready Status:False LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2023-01-09 10:09:51 +0000 UTC Reason:ContainersNotReady Message:containers with unready status: [volume-tester]} {Type:ContainersReady Status:False LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2023-01-09 10:09:51 +0000 UTC Reason:ContainersNotReady Message:containers with unready status: [volume-tester]} {Type:PodScheduled Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2023-01-09 10:09:51 +0000 UTC Reason: Message:}] Message: Reason: NominatedNodeName: HostIP:10.248.0.4 PodIP:10.248.0.21 PodIPs:[{IP:10.248.0.21}] StartTime:2023-01-09 10:09:51 +0000 UTC InitContainerStatuses:[] ContainerStatuses:[{Name:volume-tester State:{Waiting:nil Running:nil Terminated:&ContainerStateTerminated{ExitCode:2,Signal:0,Reason:Error,Message:,StartedAt:2023-01-09 10:10:10 +0000 UTC,FinishedAt:2023-01-09 10:10:10 +0000 UTC,ContainerID:containerd://0febcbf3523961e126b3b3bae04dd2e9fe2c13cf52f1037a08b3eb09862f399f,}} LastTerminationState:{Waiting:nil Running:nil Terminated:nil} Ready:false RestartCount:0 Image:registry.k8s.io/e2e-test-images/busybox:1.29-4 ImageID:registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 ContainerID:containerd://0febcbf3523961e126b3b3bae04dd2e9fe2c13cf52f1037a08b3eb09862f399f Started:0xc000c9905f}] QOSClass:BestEffort EphemeralContainerStatuses:[]}", err: <*errors.errorString | 0xc000298f30>{ s: "pod \"azuredisk-volume-tester-dd9cm\" failed with status: {Phase:Failed Conditions:[{Type:Initialized Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2023-01-09 10:09:51 +0000 UTC Reason: Message:} {Type:Ready Status:False LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2023-01-09 10:09:51 +0000 UTC Reason:ContainersNotReady Message:containers with unready status: [volume-tester]} {Type:ContainersReady Status:False LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2023-01-09 10:09:51 +0000 UTC Reason:ContainersNotReady Message:containers with unready status: [volume-tester]} {Type:PodScheduled Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2023-01-09 10:09:51 +0000 UTC Reason: Message:}] Message: Reason: NominatedNodeName: HostIP:10.248.0.4 PodIP:10.248.0.21 PodIPs:[{IP:10.248.0.21}] StartTime:2023-01-09 10:09:51 +0000 UTC InitContainerStatuses:[] ContainerStatuses:[{Name:volume-tester State:{Waiting:nil Running:nil Terminated:&ContainerStateTerminated{ExitCode:2,Signal:0,Reason:Error,Message:,StartedAt:2023-01-09 10:10:10 +0000 UTC,FinishedAt:2023-01-09 10:10:10 +0000 UTC,ContainerID:containerd://0febcbf3523961e126b3b3bae04dd2e9fe2c13cf52f1037a08b3eb09862f399f,}} LastTerminationState:{Waiting:nil Running:nil Terminated:nil} Ready:false RestartCount:0 Image:registry.k8s.io/e2e-test-images/busybox:1.29-4 ImageID:registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 ContainerID:containerd://0febcbf3523961e126b3b3bae04dd2e9fe2c13cf52f1037a08b3eb09862f399f Started:0xc000c9905f}] QOSClass:BestEffort EphemeralContainerStatuses:[]}", }, } Jan 9 10:10:12.351: FAIL: error while waiting for pod azuredisk-7726/azuredisk-volume-tester-dd9cm to be Succeeded or Failed: pod "azuredisk-volume-tester-dd9cm" failed with status: {Phase:Failed Conditions:[{Type:Initialized Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2023-01-09 10:09:51 +0000 UTC Reason: Message:} {Type:Ready Status:False LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2023-01-09 10:09:51 +0000 UTC Reason:ContainersNotReady Message:containers with unready status: [volume-tester]} {Type:ContainersReady Status:False LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2023-01-09 10:09:51 +0000 UTC Reason:ContainersNotReady Message:containers with unready status: [volume-tester]} {Type:PodScheduled Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2023-01-09 10:09:51 +0000 UTC Reason: Message:}] Message: Reason: NominatedNodeName: HostIP:10.248.0.4 PodIP:10.248.0.21 PodIPs:[{IP:10.248.0.21}] StartTime:2023-01-09 10:09:51 +0000 UTC InitContainerStatuses:[] ContainerStatuses:[{Name:volume-tester State:{Waiting:nil Running:nil Terminated:&ContainerStateTerminated{ExitCode:2,Signal:0,Reason:Error,Message:,StartedAt:2023-01-09 10:10:10 +0000 UTC,FinishedAt:2023-01-09 10:10:10 +0000 UTC,ContainerID:containerd://0febcbf3523961e126b3b3bae04dd2e9fe2c13cf52f1037a08b3eb09862f399f,}} LastTerminationState:{Waiting:nil Running:nil Terminated:nil} Ready:false RestartCount:0 Image:registry.k8s.io/e2e-test-images/busybox:1.29-4 ImageID:registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 ContainerID:containerd://0febcbf3523961e126b3b3bae04dd2e9fe2c13cf52f1037a08b3eb09862f399f Started:0xc000c9905f}] QOSClass:BestEffort EphemeralContainerStatuses:[]} Full Stack Trace sigs.k8s.io/azuredisk-csi-driver/test/e2e/testsuites.(*TestPod).WaitForSuccess(0x225059f?) /home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/testsuites/testsuites.go:823 +0x5d sigs.k8s.io/azuredisk-csi-driver/test/e2e/testsuites.(*DynamicallyProvisionedVolumeSnapshotTest).Run(0xc000bf1d78, {0x270a860, 0xc0006dad00}, {0x26f5a60, 0xc000cfc0a0}, 0xc000bf42c0?) /home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/testsuites/dynamically_provisioned_volume_snapshot_tester.go:142 +0x1358 ... skipping 42 lines ... Jan 9 10:12:20.362: INFO: Waiting up to 5m0s for PersistentVolumeClaim azuredisk-7726 to be removed Jan 9 10:12:20.416: INFO: Claim "azuredisk-7726" in namespace "pvc-l6cgd" doesn't exist in the system Jan 9 10:12:20.416: INFO: deleting StorageClass azuredisk-7726-disk.csi.azure.com-dynamic-sc-sz9nz [1mSTEP:[0m dump namespace information after failure [38;5;243m01/09/23 10:12:20.476[0m [1mSTEP:[0m Destroying namespace "azuredisk-7726" for this suite. [38;5;243m01/09/23 10:12:20.476[0m [38;5;243m------------------------------[0m [38;5;9m• [FAILED] [195.823 seconds][0m Dynamic Provisioning [38;5;243m/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning_test.go:41[0m [multi-az] [38;5;243m/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning_test.go:48[0m [38;5;9m[1m[It] should create a pod, write to its pv, take a volume snapshot, overwrite data in original pv, create another pod from the snapshot, and read unaltered original data from original pv[disk.csi.azure.com][0m [38;5;243m/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning_test.go:747[0m ... skipping 7 lines ... Jan 9 10:09:05.595: INFO: >>> kubeConfig: /root/tmp2587121816/kubeconfig/kubeconfig.westus2.json [1mSTEP:[0m setting up the StorageClass [38;5;243m01/09/23 10:09:05.596[0m [1mSTEP:[0m creating a StorageClass [38;5;243m01/09/23 10:09:05.596[0m [1mSTEP:[0m setting up the PVC and PV [38;5;243m01/09/23 10:09:05.653[0m [1mSTEP:[0m creating a PVC [38;5;243m01/09/23 10:09:05.653[0m [1mSTEP:[0m deploying the pod [38;5;243m01/09/23 10:09:05.712[0m [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/09/23 10:09:05.769[0m Jan 9 10:09:05.769: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-ljcdx" in namespace "azuredisk-7726" to be "Succeeded or Failed" Jan 9 10:09:05.823: INFO: Pod "azuredisk-volume-tester-ljcdx": Phase="Pending", Reason="", readiness=false. Elapsed: 54.289432ms Jan 9 10:09:07.879: INFO: Pod "azuredisk-volume-tester-ljcdx": Phase="Pending", Reason="", readiness=false. Elapsed: 2.109656854s Jan 9 10:09:09.881: INFO: Pod "azuredisk-volume-tester-ljcdx": Phase="Pending", Reason="", readiness=false. Elapsed: 4.112354924s Jan 9 10:09:11.881: INFO: Pod "azuredisk-volume-tester-ljcdx": Phase="Pending", Reason="", readiness=false. Elapsed: 6.111880646s Jan 9 10:09:13.881: INFO: Pod "azuredisk-volume-tester-ljcdx": Phase="Pending", Reason="", readiness=false. Elapsed: 8.111845905s Jan 9 10:09:15.878: INFO: Pod "azuredisk-volume-tester-ljcdx": Phase="Pending", Reason="", readiness=false. Elapsed: 10.109166697s ... skipping 2 lines ... Jan 9 10:09:21.879: INFO: Pod "azuredisk-volume-tester-ljcdx": Phase="Pending", Reason="", readiness=false. Elapsed: 16.110050847s Jan 9 10:09:23.878: INFO: Pod "azuredisk-volume-tester-ljcdx": Phase="Pending", Reason="", readiness=false. Elapsed: 18.109138436s Jan 9 10:09:25.880: INFO: Pod "azuredisk-volume-tester-ljcdx": Phase="Pending", Reason="", readiness=false. Elapsed: 20.111544528s Jan 9 10:09:27.879: INFO: Pod "azuredisk-volume-tester-ljcdx": Phase="Pending", Reason="", readiness=false. Elapsed: 22.110543847s Jan 9 10:09:29.887: INFO: Pod "azuredisk-volume-tester-ljcdx": Phase="Succeeded", Reason="", readiness=false. Elapsed: 24.11807628s [1mSTEP:[0m Saw pod success [38;5;243m01/09/23 10:09:29.887[0m Jan 9 10:09:29.887: INFO: Pod "azuredisk-volume-tester-ljcdx" satisfied condition "Succeeded or Failed" [1mSTEP:[0m Checking Prow test resource group [38;5;243m01/09/23 10:09:29.887[0m [1mSTEP:[0m Prow test resource group: kubetest-ht4culj6 [38;5;243m01/09/23 10:09:29.888[0m [1mSTEP:[0m Creating external resource group: azuredisk-csi-driver-test-b4adeef9-9005-11ed-9173-829d2d0c1a9d [38;5;243m01/09/23 10:09:29.888[0m [1mSTEP:[0m creating volume snapshot class with external rg azuredisk-csi-driver-test-b4adeef9-9005-11ed-9173-829d2d0c1a9d [38;5;243m01/09/23 10:09:30.708[0m [1mSTEP:[0m setting up the VolumeSnapshotClass [38;5;243m01/09/23 10:09:30.708[0m [1mSTEP:[0m creating a VolumeSnapshotClass [38;5;243m01/09/23 10:09:30.708[0m ... skipping 10 lines ... [1mSTEP:[0m creating a StorageClass [38;5;243m01/09/23 10:09:48.066[0m [1mSTEP:[0m setting up the PVC and PV [38;5;243m01/09/23 10:09:48.123[0m [1mSTEP:[0m creating a PVC [38;5;243m01/09/23 10:09:48.123[0m [1mSTEP:[0m setting up the pod [38;5;243m01/09/23 10:09:48.182[0m [1mSTEP:[0m Set pod anti-affinity to make sure two pods are scheduled on different nodes [38;5;243m01/09/23 10:09:48.182[0m [1mSTEP:[0m deploying a pod with a volume restored from the snapshot [38;5;243m01/09/23 10:09:48.182[0m [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/09/23 10:09:48.239[0m Jan 9 10:09:48.239: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-dd9cm" in namespace "azuredisk-7726" to be "Succeeded or Failed" Jan 9 10:09:48.293: INFO: Pod "azuredisk-volume-tester-dd9cm": Phase="Pending", Reason="", readiness=false. Elapsed: 54.39833ms Jan 9 10:09:50.351: INFO: Pod "azuredisk-volume-tester-dd9cm": Phase="Pending", Reason="", readiness=false. Elapsed: 2.111911828s Jan 9 10:09:52.354: INFO: Pod "azuredisk-volume-tester-dd9cm": Phase="Pending", Reason="", readiness=false. Elapsed: 4.114738874s Jan 9 10:09:54.349: INFO: Pod "azuredisk-volume-tester-dd9cm": Phase="Pending", Reason="", readiness=false. Elapsed: 6.110140794s Jan 9 10:09:56.351: INFO: Pod "azuredisk-volume-tester-dd9cm": Phase="Pending", Reason="", readiness=false. Elapsed: 8.112313729s Jan 9 10:09:58.351: INFO: Pod "azuredisk-volume-tester-dd9cm": Phase="Pending", Reason="", readiness=false. Elapsed: 10.112122126s Jan 9 10:10:00.350: INFO: Pod "azuredisk-volume-tester-dd9cm": Phase="Pending", Reason="", readiness=false. Elapsed: 12.110775324s Jan 9 10:10:02.351: INFO: Pod "azuredisk-volume-tester-dd9cm": Phase="Pending", Reason="", readiness=false. Elapsed: 14.112216557s Jan 9 10:10:04.348: INFO: Pod "azuredisk-volume-tester-dd9cm": Phase="Pending", Reason="", readiness=false. Elapsed: 16.10931375s Jan 9 10:10:06.348: INFO: Pod "azuredisk-volume-tester-dd9cm": Phase="Pending", Reason="", readiness=false. Elapsed: 18.109665355s Jan 9 10:10:08.350: INFO: Pod "azuredisk-volume-tester-dd9cm": Phase="Pending", Reason="", readiness=false. Elapsed: 20.110841246s Jan 9 10:10:10.349: INFO: Pod "azuredisk-volume-tester-dd9cm": Phase="Pending", Reason="", readiness=false. Elapsed: 22.110613738s Jan 9 10:10:12.350: INFO: Pod "azuredisk-volume-tester-dd9cm": Phase="Failed", Reason="", readiness=false. Elapsed: 24.111330956s Jan 9 10:10:12.351: INFO: Unexpected error: <*fmt.wrapError | 0xc000cccc60>: { msg: "error while waiting for pod azuredisk-7726/azuredisk-volume-tester-dd9cm to be Succeeded or Failed: pod \"azuredisk-volume-tester-dd9cm\" failed with status: {Phase:Failed Conditions:[{Type:Initialized Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2023-01-09 10:09:51 +0000 UTC Reason: Message:} {Type:Ready Status:False LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2023-01-09 10:09:51 +0000 UTC Reason:ContainersNotReady Message:containers with unready status: [volume-tester]} {Type:ContainersReady Status:False LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2023-01-09 10:09:51 +0000 UTC Reason:ContainersNotReady Message:containers with unready status: [volume-tester]} {Type:PodScheduled Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2023-01-09 10:09:51 +0000 UTC Reason: Message:}] Message: Reason: NominatedNodeName: HostIP:10.248.0.4 PodIP:10.248.0.21 PodIPs:[{IP:10.248.0.21}] StartTime:2023-01-09 10:09:51 +0000 UTC InitContainerStatuses:[] ContainerStatuses:[{Name:volume-tester State:{Waiting:nil Running:nil Terminated:&ContainerStateTerminated{ExitCode:2,Signal:0,Reason:Error,Message:,StartedAt:2023-01-09 10:10:10 +0000 UTC,FinishedAt:2023-01-09 10:10:10 +0000 UTC,ContainerID:containerd://0febcbf3523961e126b3b3bae04dd2e9fe2c13cf52f1037a08b3eb09862f399f,}} LastTerminationState:{Waiting:nil Running:nil Terminated:nil} Ready:false RestartCount:0 Image:registry.k8s.io/e2e-test-images/busybox:1.29-4 ImageID:registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 ContainerID:containerd://0febcbf3523961e126b3b3bae04dd2e9fe2c13cf52f1037a08b3eb09862f399f Started:0xc000c9905f}] QOSClass:BestEffort EphemeralContainerStatuses:[]}", err: <*errors.errorString | 0xc000298f30>{ s: "pod \"azuredisk-volume-tester-dd9cm\" failed with status: {Phase:Failed Conditions:[{Type:Initialized Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2023-01-09 10:09:51 +0000 UTC Reason: Message:} {Type:Ready Status:False LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2023-01-09 10:09:51 +0000 UTC Reason:ContainersNotReady Message:containers with unready status: [volume-tester]} {Type:ContainersReady Status:False LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2023-01-09 10:09:51 +0000 UTC Reason:ContainersNotReady Message:containers with unready status: [volume-tester]} {Type:PodScheduled Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2023-01-09 10:09:51 +0000 UTC Reason: Message:}] Message: Reason: NominatedNodeName: HostIP:10.248.0.4 PodIP:10.248.0.21 PodIPs:[{IP:10.248.0.21}] StartTime:2023-01-09 10:09:51 +0000 UTC InitContainerStatuses:[] ContainerStatuses:[{Name:volume-tester State:{Waiting:nil Running:nil Terminated:&ContainerStateTerminated{ExitCode:2,Signal:0,Reason:Error,Message:,StartedAt:2023-01-09 10:10:10 +0000 UTC,FinishedAt:2023-01-09 10:10:10 +0000 UTC,ContainerID:containerd://0febcbf3523961e126b3b3bae04dd2e9fe2c13cf52f1037a08b3eb09862f399f,}} LastTerminationState:{Waiting:nil Running:nil Terminated:nil} Ready:false RestartCount:0 Image:registry.k8s.io/e2e-test-images/busybox:1.29-4 ImageID:registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 ContainerID:containerd://0febcbf3523961e126b3b3bae04dd2e9fe2c13cf52f1037a08b3eb09862f399f Started:0xc000c9905f}] QOSClass:BestEffort EphemeralContainerStatuses:[]}", }, } Jan 9 10:10:12.351: FAIL: error while waiting for pod azuredisk-7726/azuredisk-volume-tester-dd9cm to be Succeeded or Failed: pod "azuredisk-volume-tester-dd9cm" failed with status: {Phase:Failed Conditions:[{Type:Initialized Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2023-01-09 10:09:51 +0000 UTC Reason: Message:} {Type:Ready Status:False LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2023-01-09 10:09:51 +0000 UTC Reason:ContainersNotReady Message:containers with unready status: [volume-tester]} {Type:ContainersReady Status:False LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2023-01-09 10:09:51 +0000 UTC Reason:ContainersNotReady Message:containers with unready status: [volume-tester]} {Type:PodScheduled Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2023-01-09 10:09:51 +0000 UTC Reason: Message:}] Message: Reason: NominatedNodeName: HostIP:10.248.0.4 PodIP:10.248.0.21 PodIPs:[{IP:10.248.0.21}] StartTime:2023-01-09 10:09:51 +0000 UTC InitContainerStatuses:[] ContainerStatuses:[{Name:volume-tester State:{Waiting:nil Running:nil Terminated:&ContainerStateTerminated{ExitCode:2,Signal:0,Reason:Error,Message:,StartedAt:2023-01-09 10:10:10 +0000 UTC,FinishedAt:2023-01-09 10:10:10 +0000 UTC,ContainerID:containerd://0febcbf3523961e126b3b3bae04dd2e9fe2c13cf52f1037a08b3eb09862f399f,}} LastTerminationState:{Waiting:nil Running:nil Terminated:nil} Ready:false RestartCount:0 Image:registry.k8s.io/e2e-test-images/busybox:1.29-4 ImageID:registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 ContainerID:containerd://0febcbf3523961e126b3b3bae04dd2e9fe2c13cf52f1037a08b3eb09862f399f Started:0xc000c9905f}] QOSClass:BestEffort EphemeralContainerStatuses:[]} Full Stack Trace sigs.k8s.io/azuredisk-csi-driver/test/e2e/testsuites.(*TestPod).WaitForSuccess(0x225059f?) /home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/testsuites/testsuites.go:823 +0x5d sigs.k8s.io/azuredisk-csi-driver/test/e2e/testsuites.(*DynamicallyProvisionedVolumeSnapshotTest).Run(0xc000bf1d78, {0x270a860, 0xc0006dad00}, {0x26f5a60, 0xc000cfc0a0}, 0xc000bf42c0?) /home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/testsuites/dynamically_provisioned_volume_snapshot_tester.go:142 +0x1358 ... skipping 43 lines ... Jan 9 10:12:20.416: INFO: Claim "azuredisk-7726" in namespace "pvc-l6cgd" doesn't exist in the system Jan 9 10:12:20.416: INFO: deleting StorageClass azuredisk-7726-disk.csi.azure.com-dynamic-sc-sz9nz [1mSTEP:[0m dump namespace information after failure [38;5;243m01/09/23 10:12:20.476[0m [1mSTEP:[0m Destroying namespace "azuredisk-7726" for this suite. [38;5;243m01/09/23 10:12:20.476[0m [38;5;243m<< End Captured GinkgoWriter Output[0m [38;5;9mJan 9 10:10:12.351: error while waiting for pod azuredisk-7726/azuredisk-volume-tester-dd9cm to be Succeeded or Failed: pod "azuredisk-volume-tester-dd9cm" failed with status: {Phase:Failed Conditions:[{Type:Initialized Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2023-01-09 10:09:51 +0000 UTC Reason: Message:} {Type:Ready Status:False LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2023-01-09 10:09:51 +0000 UTC Reason:ContainersNotReady Message:containers with unready status: [volume-tester]} {Type:ContainersReady Status:False LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2023-01-09 10:09:51 +0000 UTC Reason:ContainersNotReady Message:containers with unready status: [volume-tester]} {Type:PodScheduled Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2023-01-09 10:09:51 +0000 UTC Reason: Message:}] Message: Reason: NominatedNodeName: HostIP:10.248.0.4 PodIP:10.248.0.21 PodIPs:[{IP:10.248.0.21}] StartTime:2023-01-09 10:09:51 +0000 UTC InitContainerStatuses:[] ContainerStatuses:[{Name:volume-tester State:{Waiting:nil Running:nil Terminated:&ContainerStateTerminated{ExitCode:2,Signal:0,Reason:Error,Message:,StartedAt:2023-01-09 10:10:10 +0000 UTC,FinishedAt:2023-01-09 10:10:10 +0000 UTC,ContainerID:containerd://0febcbf3523961e126b3b3bae04dd2e9fe2c13cf52f1037a08b3eb09862f399f,}} LastTerminationState:{Waiting:nil Running:nil Terminated:nil} Ready:false RestartCount:0 Image:registry.k8s.io/e2e-test-images/busybox:1.29-4 ImageID:registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 ContainerID:containerd://0febcbf3523961e126b3b3bae04dd2e9fe2c13cf52f1037a08b3eb09862f399f Started:0xc000c9905f}] QOSClass:BestEffort EphemeralContainerStatuses:[]}[0m [38;5;9mIn [1m[It][0m[38;5;9m at: [1m/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/testsuites/testsuites.go:823[0m [1mThere were additional failures detected after the initial failure:[0m [38;5;13m[PANICKED][0m [38;5;13mTest Panicked[0m [38;5;13mIn [1m[DeferCleanup (Each)][0m[38;5;13m at: [1m/usr/local/go/src/runtime/panic.go:260[0m [38;5;13mruntime error: invalid memory address or nil pointer dereference[0m [38;5;13mFull Stack Trace[0m k8s.io/kubernetes/test/e2e/framework.(*Framework).dumpNamespaceInfo.func1() /home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:274 +0x5c k8s.io/kubernetes/test/e2e/framework.(*Framework).dumpNamespaceInfo(0xc0000203c0) /home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:271 +0x179 ... skipping 25 lines ... [1mSTEP:[0m creating a PVC [38;5;243m01/09/23 10:12:21.581[0m [1mSTEP:[0m setting up the StorageClass [38;5;243m01/09/23 10:12:21.636[0m [1mSTEP:[0m creating a StorageClass [38;5;243m01/09/23 10:12:21.636[0m [1mSTEP:[0m setting up the PVC and PV [38;5;243m01/09/23 10:12:21.691[0m [1mSTEP:[0m creating a PVC [38;5;243m01/09/23 10:12:21.691[0m [1mSTEP:[0m deploying the pod [38;5;243m01/09/23 10:12:21.747[0m [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/09/23 10:12:21.803[0m Jan 9 10:12:21.803: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-d4skp" in namespace "azuredisk-3086" to be "Succeeded or Failed" Jan 9 10:12:21.856: INFO: Pod "azuredisk-volume-tester-d4skp": Phase="Pending", Reason="", readiness=false. Elapsed: 53.538101ms Jan 9 10:12:23.912: INFO: Pod "azuredisk-volume-tester-d4skp": Phase="Pending", Reason="", readiness=false. Elapsed: 2.108709978s Jan 9 10:12:25.913: INFO: Pod "azuredisk-volume-tester-d4skp": Phase="Pending", Reason="", readiness=false. Elapsed: 4.109672996s Jan 9 10:12:27.912: INFO: Pod "azuredisk-volume-tester-d4skp": Phase="Pending", Reason="", readiness=false. Elapsed: 6.109575982s Jan 9 10:12:29.915: INFO: Pod "azuredisk-volume-tester-d4skp": Phase="Pending", Reason="", readiness=false. Elapsed: 8.111817172s Jan 9 10:12:31.913: INFO: Pod "azuredisk-volume-tester-d4skp": Phase="Pending", Reason="", readiness=false. Elapsed: 10.109829198s ... skipping 10 lines ... Jan 9 10:12:53.912: INFO: Pod "azuredisk-volume-tester-d4skp": Phase="Pending", Reason="", readiness=false. Elapsed: 32.109260219s Jan 9 10:12:55.914: INFO: Pod "azuredisk-volume-tester-d4skp": Phase="Pending", Reason="", readiness=false. Elapsed: 34.110668258s Jan 9 10:12:57.913: INFO: Pod "azuredisk-volume-tester-d4skp": Phase="Pending", Reason="", readiness=false. Elapsed: 36.110043981s Jan 9 10:12:59.913: INFO: Pod "azuredisk-volume-tester-d4skp": Phase="Pending", Reason="", readiness=false. Elapsed: 38.110121278s Jan 9 10:13:01.913: INFO: Pod "azuredisk-volume-tester-d4skp": Phase="Succeeded", Reason="", readiness=false. Elapsed: 40.110397725s [1mSTEP:[0m Saw pod success [38;5;243m01/09/23 10:13:01.913[0m Jan 9 10:13:01.913: INFO: Pod "azuredisk-volume-tester-d4skp" satisfied condition "Succeeded or Failed" Jan 9 10:13:01.913: INFO: deleting Pod "azuredisk-3086"/"azuredisk-volume-tester-d4skp" Jan 9 10:13:01.971: INFO: Pod azuredisk-volume-tester-d4skp has the following logs: hello world [1mSTEP:[0m Deleting pod azuredisk-volume-tester-d4skp in namespace azuredisk-3086 [38;5;243m01/09/23 10:13:01.972[0m [1mSTEP:[0m validating provisioned PV [38;5;243m01/09/23 10:13:02.089[0m [1mSTEP:[0m checking the PV [38;5;243m01/09/23 10:13:02.144[0m ... skipping 70 lines ... [1mSTEP:[0m creating a PVC [38;5;243m01/09/23 10:12:21.581[0m [1mSTEP:[0m setting up the StorageClass [38;5;243m01/09/23 10:12:21.636[0m [1mSTEP:[0m creating a StorageClass [38;5;243m01/09/23 10:12:21.636[0m [1mSTEP:[0m setting up the PVC and PV [38;5;243m01/09/23 10:12:21.691[0m [1mSTEP:[0m creating a PVC [38;5;243m01/09/23 10:12:21.691[0m [1mSTEP:[0m deploying the pod [38;5;243m01/09/23 10:12:21.747[0m [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/09/23 10:12:21.803[0m Jan 9 10:12:21.803: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-d4skp" in namespace "azuredisk-3086" to be "Succeeded or Failed" Jan 9 10:12:21.856: INFO: Pod "azuredisk-volume-tester-d4skp": Phase="Pending", Reason="", readiness=false. Elapsed: 53.538101ms Jan 9 10:12:23.912: INFO: Pod "azuredisk-volume-tester-d4skp": Phase="Pending", Reason="", readiness=false. Elapsed: 2.108709978s Jan 9 10:12:25.913: INFO: Pod "azuredisk-volume-tester-d4skp": Phase="Pending", Reason="", readiness=false. Elapsed: 4.109672996s Jan 9 10:12:27.912: INFO: Pod "azuredisk-volume-tester-d4skp": Phase="Pending", Reason="", readiness=false. Elapsed: 6.109575982s Jan 9 10:12:29.915: INFO: Pod "azuredisk-volume-tester-d4skp": Phase="Pending", Reason="", readiness=false. Elapsed: 8.111817172s Jan 9 10:12:31.913: INFO: Pod "azuredisk-volume-tester-d4skp": Phase="Pending", Reason="", readiness=false. Elapsed: 10.109829198s ... skipping 10 lines ... Jan 9 10:12:53.912: INFO: Pod "azuredisk-volume-tester-d4skp": Phase="Pending", Reason="", readiness=false. Elapsed: 32.109260219s Jan 9 10:12:55.914: INFO: Pod "azuredisk-volume-tester-d4skp": Phase="Pending", Reason="", readiness=false. Elapsed: 34.110668258s Jan 9 10:12:57.913: INFO: Pod "azuredisk-volume-tester-d4skp": Phase="Pending", Reason="", readiness=false. Elapsed: 36.110043981s Jan 9 10:12:59.913: INFO: Pod "azuredisk-volume-tester-d4skp": Phase="Pending", Reason="", readiness=false. Elapsed: 38.110121278s Jan 9 10:13:01.913: INFO: Pod "azuredisk-volume-tester-d4skp": Phase="Succeeded", Reason="", readiness=false. Elapsed: 40.110397725s [1mSTEP:[0m Saw pod success [38;5;243m01/09/23 10:13:01.913[0m Jan 9 10:13:01.913: INFO: Pod "azuredisk-volume-tester-d4skp" satisfied condition "Succeeded or Failed" Jan 9 10:13:01.913: INFO: deleting Pod "azuredisk-3086"/"azuredisk-volume-tester-d4skp" Jan 9 10:13:01.971: INFO: Pod azuredisk-volume-tester-d4skp has the following logs: hello world [1mSTEP:[0m Deleting pod azuredisk-volume-tester-d4skp in namespace azuredisk-3086 [38;5;243m01/09/23 10:13:01.972[0m [1mSTEP:[0m validating provisioned PV [38;5;243m01/09/23 10:13:02.089[0m [1mSTEP:[0m checking the PV [38;5;243m01/09/23 10:13:02.144[0m ... skipping 932 lines ... [1mSTEP:[0m setting up the StorageClass [38;5;243m01/09/23 10:26:58.385[0m [1mSTEP:[0m creating a StorageClass [38;5;243m01/09/23 10:26:58.386[0m [1mSTEP:[0m setting up the PVC and PV [38;5;243m01/09/23 10:26:58.445[0m [1mSTEP:[0m creating a PVC [38;5;243m01/09/23 10:26:58.445[0m [1mSTEP:[0m setting up the pod [38;5;243m01/09/23 10:26:58.506[0m [1mSTEP:[0m deploying the pod [38;5;243m01/09/23 10:26:58.506[0m [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/09/23 10:26:58.567[0m Jan 9 10:26:58.567: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-kvjwc" in namespace "azuredisk-1092" to be "Succeeded or Failed" Jan 9 10:26:58.624: INFO: Pod "azuredisk-volume-tester-kvjwc": Phase="Pending", Reason="", readiness=false. Elapsed: 57.597056ms Jan 9 10:27:00.684: INFO: Pod "azuredisk-volume-tester-kvjwc": Phase="Pending", Reason="", readiness=false. Elapsed: 2.116690703s Jan 9 10:27:02.684: INFO: Pod "azuredisk-volume-tester-kvjwc": Phase="Pending", Reason="", readiness=false. Elapsed: 4.117392681s Jan 9 10:27:04.683: INFO: Pod "azuredisk-volume-tester-kvjwc": Phase="Pending", Reason="", readiness=false. Elapsed: 6.115910142s Jan 9 10:27:06.683: INFO: Pod "azuredisk-volume-tester-kvjwc": Phase="Pending", Reason="", readiness=false. Elapsed: 8.116425987s Jan 9 10:27:08.684: INFO: Pod "azuredisk-volume-tester-kvjwc": Phase="Pending", Reason="", readiness=false. Elapsed: 10.116712806s ... skipping 2 lines ... Jan 9 10:27:14.684: INFO: Pod "azuredisk-volume-tester-kvjwc": Phase="Pending", Reason="", readiness=false. Elapsed: 16.116767119s Jan 9 10:27:16.684: INFO: Pod "azuredisk-volume-tester-kvjwc": Phase="Pending", Reason="", readiness=false. Elapsed: 18.117441694s Jan 9 10:27:18.684: INFO: Pod "azuredisk-volume-tester-kvjwc": Phase="Pending", Reason="", readiness=false. Elapsed: 20.116909725s Jan 9 10:27:20.686: INFO: Pod "azuredisk-volume-tester-kvjwc": Phase="Pending", Reason="", readiness=false. Elapsed: 22.118638444s Jan 9 10:27:22.685: INFO: Pod "azuredisk-volume-tester-kvjwc": Phase="Succeeded", Reason="", readiness=false. Elapsed: 24.118105579s [1mSTEP:[0m Saw pod success [38;5;243m01/09/23 10:27:22.685[0m Jan 9 10:27:22.685: INFO: Pod "azuredisk-volume-tester-kvjwc" satisfied condition "Succeeded or Failed" Jan 9 10:27:22.685: INFO: deleting Pod "azuredisk-1092"/"azuredisk-volume-tester-kvjwc" Jan 9 10:27:22.780: INFO: Pod azuredisk-volume-tester-kvjwc has the following logs: hello world [1mSTEP:[0m Deleting pod azuredisk-volume-tester-kvjwc in namespace azuredisk-1092 [38;5;243m01/09/23 10:27:22.78[0m [1mSTEP:[0m validating provisioned PV [38;5;243m01/09/23 10:27:22.899[0m [1mSTEP:[0m checking the PV [38;5;243m01/09/23 10:27:22.959[0m ... skipping 33 lines ... [1mSTEP:[0m setting up the StorageClass [38;5;243m01/09/23 10:26:58.385[0m [1mSTEP:[0m creating a StorageClass [38;5;243m01/09/23 10:26:58.386[0m [1mSTEP:[0m setting up the PVC and PV [38;5;243m01/09/23 10:26:58.445[0m [1mSTEP:[0m creating a PVC [38;5;243m01/09/23 10:26:58.445[0m [1mSTEP:[0m setting up the pod [38;5;243m01/09/23 10:26:58.506[0m [1mSTEP:[0m deploying the pod [38;5;243m01/09/23 10:26:58.506[0m [1mSTEP:[0m checking that the pod's command exits with no error [38;5;243m01/09/23 10:26:58.567[0m Jan 9 10:26:58.567: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-kvjwc" in namespace "azuredisk-1092" to be "Succeeded or Failed" Jan 9 10:26:58.624: INFO: Pod "azuredisk-volume-tester-kvjwc": Phase="Pending", Reason="", readiness=false. Elapsed: 57.597056ms Jan 9 10:27:00.684: INFO: Pod "azuredisk-volume-tester-kvjwc": Phase="Pending", Reason="", readiness=false. Elapsed: 2.116690703s Jan 9 10:27:02.684: INFO: Pod "azuredisk-volume-tester-kvjwc": Phase="Pending", Reason="", readiness=false. Elapsed: 4.117392681s Jan 9 10:27:04.683: INFO: Pod "azuredisk-volume-tester-kvjwc": Phase="Pending", Reason="", readiness=false. Elapsed: 6.115910142s Jan 9 10:27:06.683: INFO: Pod "azuredisk-volume-tester-kvjwc": Phase="Pending", Reason="", readiness=false. Elapsed: 8.116425987s Jan 9 10:27:08.684: INFO: Pod "azuredisk-volume-tester-kvjwc": Phase="Pending", Reason="", readiness=false. Elapsed: 10.116712806s ... skipping 2 lines ... Jan 9 10:27:14.684: INFO: Pod "azuredisk-volume-tester-kvjwc": Phase="Pending", Reason="", readiness=false. Elapsed: 16.116767119s Jan 9 10:27:16.684: INFO: Pod "azuredisk-volume-tester-kvjwc": Phase="Pending", Reason="", readiness=false. Elapsed: 18.117441694s Jan 9 10:27:18.684: INFO: Pod "azuredisk-volume-tester-kvjwc": Phase="Pending", Reason="", readiness=false. Elapsed: 20.116909725s Jan 9 10:27:20.686: INFO: Pod "azuredisk-volume-tester-kvjwc": Phase="Pending", Reason="", readiness=false. Elapsed: 22.118638444s Jan 9 10:27:22.685: INFO: Pod "azuredisk-volume-tester-kvjwc": Phase="Succeeded", Reason="", readiness=false. Elapsed: 24.118105579s [1mSTEP:[0m Saw pod success [38;5;243m01/09/23 10:27:22.685[0m Jan 9 10:27:22.685: INFO: Pod "azuredisk-volume-tester-kvjwc" satisfied condition "Succeeded or Failed" Jan 9 10:27:22.685: INFO: deleting Pod "azuredisk-1092"/"azuredisk-volume-tester-kvjwc" Jan 9 10:27:22.780: INFO: Pod azuredisk-volume-tester-kvjwc has the following logs: hello world [1mSTEP:[0m Deleting pod azuredisk-volume-tester-kvjwc in namespace azuredisk-1092 [38;5;243m01/09/23 10:27:22.78[0m [1mSTEP:[0m validating provisioned PV [38;5;243m01/09/23 10:27:22.899[0m [1mSTEP:[0m checking the PV [38;5;243m01/09/23 10:27:22.959[0m ... skipping 95 lines ... Platform: linux/amd64 Topology Key: topology.disk.csi.azure.com/zone Streaming logs below: I0109 09:36:30.166906 1 azuredisk.go:175] driver userAgent: disk.csi.azure.com/v1.27.0-24f0924abd1b960083c095e7d894758dec8252c3 e2e-test I0109 09:36:30.167574 1 azure_disk_utils.go:162] reading cloud config from secret kube-system/azure-cloud-provider I0109 09:36:30.206235 1 azure_disk_utils.go:169] InitializeCloudFromSecret: failed to get cloud config from secret kube-system/azure-cloud-provider: failed to get secret kube-system/azure-cloud-provider: secrets "azure-cloud-provider" not found I0109 09:36:30.206260 1 azure_disk_utils.go:174] could not read cloud config from secret kube-system/azure-cloud-provider I0109 09:36:30.206269 1 azure_disk_utils.go:184] use default AZURE_CREDENTIAL_FILE env var: /etc/kubernetes/azure.json I0109 09:36:30.206300 1 azure_disk_utils.go:192] read cloud config from file: /etc/kubernetes/azure.json successfully I0109 09:36:30.207078 1 azure_auth.go:245] Using AzurePublicCloud environment I0109 09:36:30.207129 1 azure_auth.go:130] azure: using client_id+client_secret to retrieve access token I0109 09:36:30.207159 1 azure.go:777] Azure cloudprovider using try backoff: retries=6, exponent=1.500000, duration=5, jitter=1.000000 ... skipping 25 lines ... I0109 09:36:30.207553 1 azure_blobclient.go:67] Azure BlobClient using API version: 2021-09-01 I0109 09:36:30.207568 1 azure_vmasclient.go:70] Azure AvailabilitySetsClient (read ops) using rate limit config: QPS=6, bucket=20 I0109 09:36:30.207576 1 azure_vmasclient.go:73] Azure AvailabilitySetsClient (write ops) using rate limit config: QPS=100, bucket=1000 I0109 09:36:30.207644 1 azure.go:1008] attach/detach disk operation rate limit QPS: 6.000000, Bucket: 10 I0109 09:36:30.207682 1 azuredisk.go:192] disable UseInstanceMetadata for controller I0109 09:36:30.207688 1 azuredisk.go:204] cloud: AzurePublicCloud, location: westus2, rg: kubetest-ht4culj6, VMType: vmss, PrimaryScaleSetName: k8s-agentpool-27319582-vmss, PrimaryAvailabilitySetName: , DisableAvailabilitySetNodes: false I0109 09:36:30.212326 1 mount_linux.go:287] 'umount /tmp/kubelet-detect-safe-umount3050680674' failed with: exit status 32, output: umount: /tmp/kubelet-detect-safe-umount3050680674: must be superuser to unmount. I0109 09:36:30.212368 1 mount_linux.go:289] Detected umount with unsafe 'not mounted' behavior I0109 09:36:30.212459 1 driver.go:81] Enabling controller service capability: CREATE_DELETE_VOLUME I0109 09:36:30.212475 1 driver.go:81] Enabling controller service capability: PUBLISH_UNPUBLISH_VOLUME I0109 09:36:30.212481 1 driver.go:81] Enabling controller service capability: CREATE_DELETE_SNAPSHOT I0109 09:36:30.212487 1 driver.go:81] Enabling controller service capability: CLONE_VOLUME I0109 09:36:30.212494 1 driver.go:81] Enabling controller service capability: EXPAND_VOLUME ... skipping 68 lines ... I0109 09:36:38.813182 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 24988 I0109 09:36:38.895540 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 32354 I0109 09:36:38.898962 1 controllerserver.go:383] GetDiskLun returned: cannot find Lun for disk pvc-fcad8486-6f29-4394-8743-a9f55fb97bc1. Initiating attaching volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-fcad8486-6f29-4394-8743-a9f55fb97bc1 to node k8s-agentpool-27319582-vmss000001 (vmState Succeeded). I0109 09:36:38.899025 1 controllerserver.go:408] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-fcad8486-6f29-4394-8743-a9f55fb97bc1 to node k8s-agentpool-27319582-vmss000001 I0109 09:36:38.899109 1 azure_controller_common.go:255] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-fcad8486-6f29-4394-8743-a9f55fb97bc1 lun 0 to node k8s-agentpool-27319582-vmss000001, diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-ht4culj6/providers/microsoft.compute/disks/pvc-fcad8486-6f29-4394-8743-a9f55fb97bc1:%!s(*provider.AttachDiskOptions=&{None pvc-fcad8486-6f29-4394-8743-a9f55fb97bc1 false 0})] I0109 09:36:38.899164 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-ht4culj6): vm(k8s-agentpool-27319582-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-ht4culj6/providers/microsoft.compute/disks/pvc-fcad8486-6f29-4394-8743-a9f55fb97bc1:%!s(*provider.AttachDiskOptions=&{None pvc-fcad8486-6f29-4394-8743-a9f55fb97bc1 false 0})]) I0109 09:36:39.478464 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-ht4culj6): vm(k8s-agentpool-27319582-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-ht4culj6/providers/microsoft.compute/disks/pvc-fcad8486-6f29-4394-8743-a9f55fb97bc1:%!s(*provider.AttachDiskOptions=&{None pvc-fcad8486-6f29-4394-8743-a9f55fb97bc1 false 0})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I0109 09:36:49.594054 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-ht4culj6, k8s-agentpool-27319582-vmss, k8s-agentpool-27319582-vmss000001) successfully I0109 09:36:49.594091 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool-27319582-vmss, kubetest-ht4culj6, k8s-agentpool-27319582-vmss000001) for cacheKey(kubetest-ht4culj6/k8s-agentpool-27319582-vmss) updated successfully I0109 09:36:49.594111 1 controllerserver.go:413] Attach operation successful: volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-fcad8486-6f29-4394-8743-a9f55fb97bc1 attached to node k8s-agentpool-27319582-vmss000001. I0109 09:36:49.594125 1 controllerserver.go:433] attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-fcad8486-6f29-4394-8743-a9f55fb97bc1 to node k8s-agentpool-27319582-vmss000001 successfully I0109 09:36:49.594177 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=10.86771875 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-ht4culj6" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-fcad8486-6f29-4394-8743-a9f55fb97bc1" node="k8s-agentpool-27319582-vmss000001" result_code="succeeded" I0109 09:36:49.594197 1 utils.go:84] GRPC response: {"publish_context":{"LUN":"0"}} ... skipping 18 lines ... I0109 09:37:44.255196 1 controllerserver.go:319] delete azure disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-fcad8486-6f29-4394-8743-a9f55fb97bc1) returned with <nil> I0109 09:37:44.255244 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=5.15819575 request="azuredisk_csi_driver_controller_delete_volume" resource_group="kubetest-ht4culj6" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-fcad8486-6f29-4394-8743-a9f55fb97bc1" result_code="succeeded" I0109 09:37:44.255265 1 utils.go:84] GRPC response: {} I0109 09:37:49.775699 1 utils.go:77] GRPC call: /csi.v1.Controller/CreateVolume I0109 09:37:49.775719 1 utils.go:78] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.disk.csi.azure.com/zone":"westus2-1","topology.kubernetes.io/zone":"westus2-1"}},{"segments":{"topology.disk.csi.azure.com/zone":"westus2-2","topology.kubernetes.io/zone":"westus2-2"}}],"requisite":[{"segments":{"topology.disk.csi.azure.com/zone":"westus2-2","topology.kubernetes.io/zone":"westus2-2"}},{"segments":{"topology.disk.csi.azure.com/zone":"westus2-1","topology.kubernetes.io/zone":"westus2-1"}}]},"capacity_range":{"required_bytes":10737418240},"name":"pvc-dcd9bef3-f84d-4eaf-b806-cc663cf3eee7","parameters":{"csi.storage.k8s.io/pv/name":"pvc-dcd9bef3-f84d-4eaf-b806-cc663cf3eee7","csi.storage.k8s.io/pvc/name":"pvc-pszvh","csi.storage.k8s.io/pvc/namespace":"azuredisk-2540","enableAsyncAttach":"false","networkAccessPolicy":"DenyAll","skuName":"Standard_LRS","userAgent":"azuredisk-e2e-test"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]} I0109 09:37:49.776605 1 azure_disk_utils.go:162] reading cloud config from secret kube-system/azure-cloud-provider I0109 09:37:49.783118 1 azure_disk_utils.go:169] InitializeCloudFromSecret: failed to get cloud config from secret kube-system/azure-cloud-provider: failed to get secret kube-system/azure-cloud-provider: secrets "azure-cloud-provider" not found I0109 09:37:49.783184 1 azure_disk_utils.go:174] could not read cloud config from secret kube-system/azure-cloud-provider I0109 09:37:49.783194 1 azure_disk_utils.go:184] use default AZURE_CREDENTIAL_FILE env var: /etc/kubernetes/azure.json I0109 09:37:49.783264 1 azure_disk_utils.go:192] read cloud config from file: /etc/kubernetes/azure.json successfully I0109 09:37:49.784017 1 azure_auth.go:245] Using AzurePublicCloud environment I0109 09:37:49.784082 1 azure_auth.go:130] azure: using client_id+client_secret to retrieve access token I0109 09:37:49.784133 1 azure.go:777] Azure cloudprovider using try backoff: retries=6, exponent=1.500000, duration=5, jitter=1.000000 ... skipping 37 lines ... I0109 09:37:54.198384 1 utils.go:78] GRPC request: {"node_id":"k8s-agentpool-27319582-vmss000000","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-dcd9bef3-f84d-4eaf-b806-cc663cf3eee7","csi.storage.k8s.io/pvc/name":"pvc-pszvh","csi.storage.k8s.io/pvc/namespace":"azuredisk-2540","enableAsyncAttach":"false","enableasyncattach":"false","networkAccessPolicy":"DenyAll","requestedsizegib":"10","skuName":"Standard_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1673256990696-8081-disk.csi.azure.com","userAgent":"azuredisk-e2e-test"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-dcd9bef3-f84d-4eaf-b806-cc663cf3eee7"} I0109 09:37:54.273075 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1218 I0109 09:37:54.273634 1 controllerserver.go:383] GetDiskLun returned: cannot find Lun for disk pvc-dcd9bef3-f84d-4eaf-b806-cc663cf3eee7. Initiating attaching volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-dcd9bef3-f84d-4eaf-b806-cc663cf3eee7 to node k8s-agentpool-27319582-vmss000000 (vmState Succeeded). I0109 09:37:54.273669 1 controllerserver.go:408] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-dcd9bef3-f84d-4eaf-b806-cc663cf3eee7 to node k8s-agentpool-27319582-vmss000000 I0109 09:37:54.273806 1 azure_controller_common.go:255] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-dcd9bef3-f84d-4eaf-b806-cc663cf3eee7 lun 0 to node k8s-agentpool-27319582-vmss000000, diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-ht4culj6/providers/microsoft.compute/disks/pvc-dcd9bef3-f84d-4eaf-b806-cc663cf3eee7:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-dcd9bef3-f84d-4eaf-b806-cc663cf3eee7 false 0})] I0109 09:37:54.273889 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-ht4culj6): vm(k8s-agentpool-27319582-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-ht4culj6/providers/microsoft.compute/disks/pvc-dcd9bef3-f84d-4eaf-b806-cc663cf3eee7:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-dcd9bef3-f84d-4eaf-b806-cc663cf3eee7 false 0})]) I0109 09:37:54.426049 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-ht4culj6): vm(k8s-agentpool-27319582-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-ht4culj6/providers/microsoft.compute/disks/pvc-dcd9bef3-f84d-4eaf-b806-cc663cf3eee7:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-dcd9bef3-f84d-4eaf-b806-cc663cf3eee7 false 0})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I0109 09:38:04.497797 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-ht4culj6, k8s-agentpool-27319582-vmss, k8s-agentpool-27319582-vmss000000) successfully I0109 09:38:04.497838 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool-27319582-vmss, kubetest-ht4culj6, k8s-agentpool-27319582-vmss000000) for cacheKey(kubetest-ht4culj6/k8s-agentpool-27319582-vmss) updated successfully I0109 09:38:04.497859 1 controllerserver.go:413] Attach operation successful: volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-dcd9bef3-f84d-4eaf-b806-cc663cf3eee7 attached to node k8s-agentpool-27319582-vmss000000. I0109 09:38:04.497872 1 controllerserver.go:433] attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-dcd9bef3-f84d-4eaf-b806-cc663cf3eee7 to node k8s-agentpool-27319582-vmss000000 successfully I0109 09:38:04.497915 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=10.224293726 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-ht4culj6" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-dcd9bef3-f84d-4eaf-b806-cc663cf3eee7" node="k8s-agentpool-27319582-vmss000000" result_code="succeeded" I0109 09:38:04.497929 1 utils.go:84] GRPC response: {"publish_context":{"LUN":"0"}} ... skipping 39 lines ... I0109 09:39:24.638120 1 controllerserver.go:319] delete azure disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-dcd9bef3-f84d-4eaf-b806-cc663cf3eee7) returned with <nil> I0109 09:39:24.638161 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=5.153355585 request="azuredisk_csi_driver_controller_delete_volume" resource_group="kubetest-ht4culj6" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-dcd9bef3-f84d-4eaf-b806-cc663cf3eee7" result_code="succeeded" I0109 09:39:24.638178 1 utils.go:84] GRPC response: {} I0109 09:39:28.652698 1 utils.go:77] GRPC call: /csi.v1.Controller/CreateVolume I0109 09:39:28.652718 1 utils.go:78] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.disk.csi.azure.com/zone":"westus2-2","topology.kubernetes.io/zone":"westus2-2"}}],"requisite":[{"segments":{"topology.disk.csi.azure.com/zone":"westus2-2","topology.kubernetes.io/zone":"westus2-2"}}]},"capacity_range":{"required_bytes":1099511627776},"name":"pvc-ba413786-16b6-49a7-a3a3-f9f41fe5ce85","parameters":{"csi.storage.k8s.io/pv/name":"pvc-ba413786-16b6-49a7-a3a3-f9f41fe5ce85","csi.storage.k8s.io/pvc/name":"pvc-skjsl","csi.storage.k8s.io/pvc/namespace":"azuredisk-4728","enableAsyncAttach":"false","enableBursting":"true","perfProfile":"Basic","skuName":"Premium_LRS","userAgent":"azuredisk-e2e-test"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]} I0109 09:39:28.653578 1 azure_disk_utils.go:162] reading cloud config from secret kube-system/azure-cloud-provider I0109 09:39:28.703763 1 azure_disk_utils.go:169] InitializeCloudFromSecret: failed to get cloud config from secret kube-system/azure-cloud-provider: failed to get secret kube-system/azure-cloud-provider: secrets "azure-cloud-provider" not found I0109 09:39:28.703784 1 azure_disk_utils.go:174] could not read cloud config from secret kube-system/azure-cloud-provider I0109 09:39:28.703792 1 azure_disk_utils.go:184] use default AZURE_CREDENTIAL_FILE env var: /etc/kubernetes/azure.json I0109 09:39:28.703827 1 azure_disk_utils.go:192] read cloud config from file: /etc/kubernetes/azure.json successfully I0109 09:39:28.704436 1 azure_auth.go:245] Using AzurePublicCloud environment I0109 09:39:28.704485 1 azure_auth.go:130] azure: using client_id+client_secret to retrieve access token I0109 09:39:28.704608 1 azure.go:777] Azure cloudprovider using try backoff: retries=6, exponent=1.500000, duration=5, jitter=1.000000 ... skipping 37 lines ... I0109 09:39:31.707294 1 utils.go:78] GRPC request: {"node_id":"k8s-agentpool-27319582-vmss000001","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-ba413786-16b6-49a7-a3a3-f9f41fe5ce85","csi.storage.k8s.io/pvc/name":"pvc-skjsl","csi.storage.k8s.io/pvc/namespace":"azuredisk-4728","enableAsyncAttach":"false","enableBursting":"true","enableasyncattach":"false","perfProfile":"Basic","requestedsizegib":"1024","skuName":"Premium_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1673256990696-8081-disk.csi.azure.com","userAgent":"azuredisk-e2e-test"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-ba413786-16b6-49a7-a3a3-f9f41fe5ce85"} I0109 09:39:31.728937 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1338 I0109 09:39:31.729276 1 controllerserver.go:383] GetDiskLun returned: cannot find Lun for disk pvc-ba413786-16b6-49a7-a3a3-f9f41fe5ce85. Initiating attaching volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-ba413786-16b6-49a7-a3a3-f9f41fe5ce85 to node k8s-agentpool-27319582-vmss000001 (vmState Succeeded). I0109 09:39:31.729327 1 controllerserver.go:408] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-ba413786-16b6-49a7-a3a3-f9f41fe5ce85 to node k8s-agentpool-27319582-vmss000001 I0109 09:39:31.729407 1 azure_controller_common.go:255] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-ba413786-16b6-49a7-a3a3-f9f41fe5ce85 lun 0 to node k8s-agentpool-27319582-vmss000001, diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-ht4culj6/providers/microsoft.compute/disks/pvc-ba413786-16b6-49a7-a3a3-f9f41fe5ce85:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-ba413786-16b6-49a7-a3a3-f9f41fe5ce85 false 0})] I0109 09:39:31.729444 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-ht4culj6): vm(k8s-agentpool-27319582-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-ht4culj6/providers/microsoft.compute/disks/pvc-ba413786-16b6-49a7-a3a3-f9f41fe5ce85:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-ba413786-16b6-49a7-a3a3-f9f41fe5ce85 false 0})]) I0109 09:39:31.905215 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-ht4culj6): vm(k8s-agentpool-27319582-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-ht4culj6/providers/microsoft.compute/disks/pvc-ba413786-16b6-49a7-a3a3-f9f41fe5ce85:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-ba413786-16b6-49a7-a3a3-f9f41fe5ce85 false 0})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I0109 09:39:42.050189 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-ht4culj6, k8s-agentpool-27319582-vmss, k8s-agentpool-27319582-vmss000001) successfully I0109 09:39:42.050228 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool-27319582-vmss, kubetest-ht4culj6, k8s-agentpool-27319582-vmss000001) for cacheKey(kubetest-ht4culj6/k8s-agentpool-27319582-vmss) updated successfully I0109 09:39:42.050249 1 controllerserver.go:413] Attach operation successful: volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-ba413786-16b6-49a7-a3a3-f9f41fe5ce85 attached to node k8s-agentpool-27319582-vmss000001. I0109 09:39:42.050264 1 controllerserver.go:433] attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-ba413786-16b6-49a7-a3a3-f9f41fe5ce85 to node k8s-agentpool-27319582-vmss000001 successfully I0109 09:39:42.050307 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=10.321046816 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-ht4culj6" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-ba413786-16b6-49a7-a3a3-f9f41fe5ce85" node="k8s-agentpool-27319582-vmss000001" result_code="succeeded" I0109 09:39:42.050332 1 utils.go:84] GRPC response: {"publish_context":{"LUN":"0"}} ... skipping 32 lines ... I0109 09:41:01.989949 1 utils.go:78] GRPC request: {"node_id":"k8s-agentpool-27319582-vmss000001","volume_capability":{"AccessType":{"Mount":{"mount_flags":["invalid","mount","options"]}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-f62b55d7-4648-4389-b72a-610edf2d30c0","csi.storage.k8s.io/pvc/name":"pvc-s6b42","csi.storage.k8s.io/pvc/namespace":"azuredisk-5466","requestedsizegib":"10","skuName":"StandardSSD_ZRS","storage.kubernetes.io/csiProvisionerIdentity":"1673256990696-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-f62b55d7-4648-4389-b72a-610edf2d30c0"} I0109 09:41:02.015527 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1193 I0109 09:41:02.015831 1 controllerserver.go:383] GetDiskLun returned: cannot find Lun for disk pvc-f62b55d7-4648-4389-b72a-610edf2d30c0. Initiating attaching volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-f62b55d7-4648-4389-b72a-610edf2d30c0 to node k8s-agentpool-27319582-vmss000001 (vmState Succeeded). I0109 09:41:02.015853 1 controllerserver.go:408] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-f62b55d7-4648-4389-b72a-610edf2d30c0 to node k8s-agentpool-27319582-vmss000001 I0109 09:41:02.015884 1 azure_controller_common.go:255] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-f62b55d7-4648-4389-b72a-610edf2d30c0 lun 0 to node k8s-agentpool-27319582-vmss000001, diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-ht4culj6/providers/microsoft.compute/disks/pvc-f62b55d7-4648-4389-b72a-610edf2d30c0:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-f62b55d7-4648-4389-b72a-610edf2d30c0 false 0})] I0109 09:41:02.015917 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-ht4culj6): vm(k8s-agentpool-27319582-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-ht4culj6/providers/microsoft.compute/disks/pvc-f62b55d7-4648-4389-b72a-610edf2d30c0:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-f62b55d7-4648-4389-b72a-610edf2d30c0 false 0})]) I0109 09:41:02.157614 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-ht4culj6): vm(k8s-agentpool-27319582-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-ht4culj6/providers/microsoft.compute/disks/pvc-f62b55d7-4648-4389-b72a-610edf2d30c0:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-f62b55d7-4648-4389-b72a-610edf2d30c0 false 0})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I0109 09:41:12.241767 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-ht4culj6, k8s-agentpool-27319582-vmss, k8s-agentpool-27319582-vmss000001) successfully I0109 09:41:12.242322 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool-27319582-vmss, kubetest-ht4culj6, k8s-agentpool-27319582-vmss000001) for cacheKey(kubetest-ht4culj6/k8s-agentpool-27319582-vmss) updated successfully I0109 09:41:12.242421 1 controllerserver.go:413] Attach operation successful: volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-f62b55d7-4648-4389-b72a-610edf2d30c0 attached to node k8s-agentpool-27319582-vmss000001. I0109 09:41:12.242460 1 controllerserver.go:433] attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-f62b55d7-4648-4389-b72a-610edf2d30c0 to node k8s-agentpool-27319582-vmss000001 successfully I0109 09:41:12.242613 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=10.226729317 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-ht4culj6" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-f62b55d7-4648-4389-b72a-610edf2d30c0" node="k8s-agentpool-27319582-vmss000001" result_code="succeeded" I0109 09:41:12.242644 1 utils.go:84] GRPC response: {"publish_context":{"LUN":"0"}} ... skipping 53 lines ... I0109 09:42:11.238481 1 utils.go:78] GRPC request: {"node_id":"k8s-agentpool-27319582-vmss000001","volume_capability":{"AccessType":{"Block":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-508b6639-f8a0-4ce2-9ab0-38324ebb14e2","csi.storage.k8s.io/pvc/name":"pvc-8db69","csi.storage.k8s.io/pvc/namespace":"azuredisk-2790","requestedsizegib":"10","skuName":"StandardSSD_ZRS","storage.kubernetes.io/csiProvisionerIdentity":"1673256990696-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-508b6639-f8a0-4ce2-9ab0-38324ebb14e2"} I0109 09:42:11.257275 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1193 I0109 09:42:11.257668 1 controllerserver.go:383] GetDiskLun returned: cannot find Lun for disk pvc-508b6639-f8a0-4ce2-9ab0-38324ebb14e2. Initiating attaching volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-508b6639-f8a0-4ce2-9ab0-38324ebb14e2 to node k8s-agentpool-27319582-vmss000001 (vmState Succeeded). I0109 09:42:11.257701 1 controllerserver.go:408] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-508b6639-f8a0-4ce2-9ab0-38324ebb14e2 to node k8s-agentpool-27319582-vmss000001 I0109 09:42:11.257745 1 azure_controller_common.go:255] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-508b6639-f8a0-4ce2-9ab0-38324ebb14e2 lun 0 to node k8s-agentpool-27319582-vmss000001, diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-ht4culj6/providers/microsoft.compute/disks/pvc-508b6639-f8a0-4ce2-9ab0-38324ebb14e2:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-508b6639-f8a0-4ce2-9ab0-38324ebb14e2 false 0})] I0109 09:42:11.257788 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-ht4culj6): vm(k8s-agentpool-27319582-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-ht4culj6/providers/microsoft.compute/disks/pvc-508b6639-f8a0-4ce2-9ab0-38324ebb14e2:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-508b6639-f8a0-4ce2-9ab0-38324ebb14e2 false 0})]) I0109 09:42:11.380441 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-ht4culj6): vm(k8s-agentpool-27319582-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-ht4culj6/providers/microsoft.compute/disks/pvc-508b6639-f8a0-4ce2-9ab0-38324ebb14e2:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-508b6639-f8a0-4ce2-9ab0-38324ebb14e2 false 0})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I0109 09:42:21.509425 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-ht4culj6, k8s-agentpool-27319582-vmss, k8s-agentpool-27319582-vmss000001) successfully I0109 09:42:21.509459 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool-27319582-vmss, kubetest-ht4culj6, k8s-agentpool-27319582-vmss000001) for cacheKey(kubetest-ht4culj6/k8s-agentpool-27319582-vmss) updated successfully I0109 09:42:21.509479 1 controllerserver.go:413] Attach operation successful: volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-508b6639-f8a0-4ce2-9ab0-38324ebb14e2 attached to node k8s-agentpool-27319582-vmss000001. I0109 09:42:21.509494 1 controllerserver.go:433] attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-508b6639-f8a0-4ce2-9ab0-38324ebb14e2 to node k8s-agentpool-27319582-vmss000001 successfully I0109 09:42:21.509578 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=10.251891514 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-ht4culj6" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-508b6639-f8a0-4ce2-9ab0-38324ebb14e2" node="k8s-agentpool-27319582-vmss000001" result_code="succeeded" I0109 09:42:21.509662 1 utils.go:84] GRPC response: {"publish_context":{"LUN":"0"}} ... skipping 45 lines ... I0109 09:43:20.852574 1 utils.go:78] GRPC request: {"node_id":"k8s-agentpool-27319582-vmss000001","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-261268db-1c68-4f25-a4c1-ecd4067cd0fe","csi.storage.k8s.io/pvc/name":"pvc-wpj5b","csi.storage.k8s.io/pvc/namespace":"azuredisk-5356","requestedsizegib":"10","resourceGroup":"azuredisk-csi-driver-test-0ab45119-9002-11ed-9173-829d2d0c1a9d","skuName":"StandardSSD_ZRS","storage.kubernetes.io/csiProvisionerIdentity":"1673256990696-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/azuredisk-csi-driver-test-0ab45119-9002-11ed-9173-829d2d0c1a9d/providers/Microsoft.Compute/disks/pvc-261268db-1c68-4f25-a4c1-ecd4067cd0fe"} I0109 09:43:20.872763 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1238 I0109 09:43:20.873157 1 controllerserver.go:383] GetDiskLun returned: cannot find Lun for disk pvc-261268db-1c68-4f25-a4c1-ecd4067cd0fe. Initiating attaching volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/azuredisk-csi-driver-test-0ab45119-9002-11ed-9173-829d2d0c1a9d/providers/Microsoft.Compute/disks/pvc-261268db-1c68-4f25-a4c1-ecd4067cd0fe to node k8s-agentpool-27319582-vmss000001 (vmState Succeeded). I0109 09:43:20.873189 1 controllerserver.go:408] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/azuredisk-csi-driver-test-0ab45119-9002-11ed-9173-829d2d0c1a9d/providers/Microsoft.Compute/disks/pvc-261268db-1c68-4f25-a4c1-ecd4067cd0fe to node k8s-agentpool-27319582-vmss000001 I0109 09:43:20.873323 1 azure_controller_common.go:255] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/azuredisk-csi-driver-test-0ab45119-9002-11ed-9173-829d2d0c1a9d/providers/Microsoft.Compute/disks/pvc-261268db-1c68-4f25-a4c1-ecd4067cd0fe lun 0 to node k8s-agentpool-27319582-vmss000001, diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/azuredisk-csi-driver-test-0ab45119-9002-11ed-9173-829d2d0c1a9d/providers/microsoft.compute/disks/pvc-261268db-1c68-4f25-a4c1-ecd4067cd0fe:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-261268db-1c68-4f25-a4c1-ecd4067cd0fe false 0})] I0109 09:43:20.873508 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-ht4culj6): vm(k8s-agentpool-27319582-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/azuredisk-csi-driver-test-0ab45119-9002-11ed-9173-829d2d0c1a9d/providers/microsoft.compute/disks/pvc-261268db-1c68-4f25-a4c1-ecd4067cd0fe:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-261268db-1c68-4f25-a4c1-ecd4067cd0fe false 0})]) I0109 09:43:21.027901 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-ht4culj6): vm(k8s-agentpool-27319582-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/azuredisk-csi-driver-test-0ab45119-9002-11ed-9173-829d2d0c1a9d/providers/microsoft.compute/disks/pvc-261268db-1c68-4f25-a4c1-ecd4067cd0fe:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-261268db-1c68-4f25-a4c1-ecd4067cd0fe false 0})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I0109 09:43:31.107100 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-ht4culj6, k8s-agentpool-27319582-vmss, k8s-agentpool-27319582-vmss000001) successfully I0109 09:43:31.107143 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool-27319582-vmss, kubetest-ht4culj6, k8s-agentpool-27319582-vmss000001) for cacheKey(kubetest-ht4culj6/k8s-agentpool-27319582-vmss) updated successfully I0109 09:43:31.107163 1 controllerserver.go:413] Attach operation successful: volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/azuredisk-csi-driver-test-0ab45119-9002-11ed-9173-829d2d0c1a9d/providers/Microsoft.Compute/disks/pvc-261268db-1c68-4f25-a4c1-ecd4067cd0fe attached to node k8s-agentpool-27319582-vmss000001. I0109 09:43:31.107178 1 controllerserver.go:433] attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/azuredisk-csi-driver-test-0ab45119-9002-11ed-9173-829d2d0c1a9d/providers/Microsoft.Compute/disks/pvc-261268db-1c68-4f25-a4c1-ecd4067cd0fe to node k8s-agentpool-27319582-vmss000001 successfully I0109 09:43:31.107222 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=10.234062777 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-ht4culj6" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/azuredisk-csi-driver-test-0ab45119-9002-11ed-9173-829d2d0c1a9d/providers/Microsoft.Compute/disks/pvc-261268db-1c68-4f25-a4c1-ecd4067cd0fe" node="k8s-agentpool-27319582-vmss000001" result_code="succeeded" I0109 09:43:31.107243 1 utils.go:84] GRPC response: {"publish_context":{"LUN":"0"}} ... skipping 47 lines ... I0109 09:44:44.691207 1 controllerserver.go:408] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/azuredisk-csi-driver-test-3cab6831-9002-11ed-9173-829d2d0c1a9d/providers/Microsoft.Compute/disks/pvc-9795461e-8b9e-424b-8af0-c59f83dd3499 to node k8s-agentpool-27319582-vmss000001 I0109 09:44:44.691245 1 azure_controller_common.go:255] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/azuredisk-csi-driver-test-3cab6831-9002-11ed-9173-829d2d0c1a9d/providers/Microsoft.Compute/disks/pvc-9795461e-8b9e-424b-8af0-c59f83dd3499 lun 0 to node k8s-agentpool-27319582-vmss000001, diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/azuredisk-csi-driver-test-3cab6831-9002-11ed-9173-829d2d0c1a9d/providers/microsoft.compute/disks/pvc-9795461e-8b9e-424b-8af0-c59f83dd3499:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-9795461e-8b9e-424b-8af0-c59f83dd3499 false 0})] I0109 09:44:44.691291 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-ht4culj6): vm(k8s-agentpool-27319582-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/azuredisk-csi-driver-test-3cab6831-9002-11ed-9173-829d2d0c1a9d/providers/microsoft.compute/disks/pvc-9795461e-8b9e-424b-8af0-c59f83dd3499:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-9795461e-8b9e-424b-8af0-c59f83dd3499 false 0})]) I0109 09:44:44.693762 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1238 I0109 09:44:44.694129 1 controllerserver.go:383] GetDiskLun returned: cannot find Lun for disk pvc-bb894f62-94c4-47a7-9b43-92deea8b50cc. Initiating attaching volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/azuredisk-csi-driver-test-3d342f14-9002-11ed-9173-829d2d0c1a9d/providers/Microsoft.Compute/disks/pvc-bb894f62-94c4-47a7-9b43-92deea8b50cc to node k8s-agentpool-27319582-vmss000001 (vmState Succeeded). I0109 09:44:44.694166 1 controllerserver.go:408] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/azuredisk-csi-driver-test-3d342f14-9002-11ed-9173-829d2d0c1a9d/providers/Microsoft.Compute/disks/pvc-bb894f62-94c4-47a7-9b43-92deea8b50cc to node k8s-agentpool-27319582-vmss000001 I0109 09:44:44.810412 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-ht4culj6): vm(k8s-agentpool-27319582-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/azuredisk-csi-driver-test-3cab6831-9002-11ed-9173-829d2d0c1a9d/providers/microsoft.compute/disks/pvc-9795461e-8b9e-424b-8af0-c59f83dd3499:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-9795461e-8b9e-424b-8af0-c59f83dd3499 false 0})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I0109 09:44:54.889344 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-ht4culj6, k8s-agentpool-27319582-vmss, k8s-agentpool-27319582-vmss000001) successfully I0109 09:44:54.889379 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool-27319582-vmss, kubetest-ht4culj6, k8s-agentpool-27319582-vmss000001) for cacheKey(kubetest-ht4culj6/k8s-agentpool-27319582-vmss) updated successfully I0109 09:44:54.889411 1 controllerserver.go:413] Attach operation successful: volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/azuredisk-csi-driver-test-3cab6831-9002-11ed-9173-829d2d0c1a9d/providers/Microsoft.Compute/disks/pvc-9795461e-8b9e-424b-8af0-c59f83dd3499 attached to node k8s-agentpool-27319582-vmss000001. I0109 09:44:54.889440 1 controllerserver.go:433] attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/azuredisk-csi-driver-test-3cab6831-9002-11ed-9173-829d2d0c1a9d/providers/Microsoft.Compute/disks/pvc-9795461e-8b9e-424b-8af0-c59f83dd3499 to node k8s-agentpool-27319582-vmss000001 successfully I0109 09:44:54.889478 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=10.198313132 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-ht4culj6" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/azuredisk-csi-driver-test-3cab6831-9002-11ed-9173-829d2d0c1a9d/providers/Microsoft.Compute/disks/pvc-9795461e-8b9e-424b-8af0-c59f83dd3499" node="k8s-agentpool-27319582-vmss000001" result_code="succeeded" I0109 09:44:54.889499 1 utils.go:84] GRPC response: {"publish_context":{"LUN":"0"}} I0109 09:44:54.889640 1 azure_controller_common.go:255] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/azuredisk-csi-driver-test-3d342f14-9002-11ed-9173-829d2d0c1a9d/providers/Microsoft.Compute/disks/pvc-bb894f62-94c4-47a7-9b43-92deea8b50cc lun 1 to node k8s-agentpool-27319582-vmss000001, diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/azuredisk-csi-driver-test-3d342f14-9002-11ed-9173-829d2d0c1a9d/providers/microsoft.compute/disks/pvc-bb894f62-94c4-47a7-9b43-92deea8b50cc:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-bb894f62-94c4-47a7-9b43-92deea8b50cc false 1})] I0109 09:44:54.889690 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-ht4culj6): vm(k8s-agentpool-27319582-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/azuredisk-csi-driver-test-3d342f14-9002-11ed-9173-829d2d0c1a9d/providers/microsoft.compute/disks/pvc-bb894f62-94c4-47a7-9b43-92deea8b50cc:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-bb894f62-94c4-47a7-9b43-92deea8b50cc false 1})]) I0109 09:44:55.056278 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-ht4culj6): vm(k8s-agentpool-27319582-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/azuredisk-csi-driver-test-3d342f14-9002-11ed-9173-829d2d0c1a9d/providers/microsoft.compute/disks/pvc-bb894f62-94c4-47a7-9b43-92deea8b50cc:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-bb894f62-94c4-47a7-9b43-92deea8b50cc false 1})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I0109 09:45:05.174696 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-ht4culj6, k8s-agentpool-27319582-vmss, k8s-agentpool-27319582-vmss000001) successfully I0109 09:45:05.174730 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool-27319582-vmss, kubetest-ht4culj6, k8s-agentpool-27319582-vmss000001) for cacheKey(kubetest-ht4culj6/k8s-agentpool-27319582-vmss) updated successfully I0109 09:45:05.174753 1 controllerserver.go:413] Attach operation successful: volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/azuredisk-csi-driver-test-3d342f14-9002-11ed-9173-829d2d0c1a9d/providers/Microsoft.Compute/disks/pvc-bb894f62-94c4-47a7-9b43-92deea8b50cc attached to node k8s-agentpool-27319582-vmss000001. I0109 09:45:05.174767 1 controllerserver.go:433] attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/azuredisk-csi-driver-test-3d342f14-9002-11ed-9173-829d2d0c1a9d/providers/Microsoft.Compute/disks/pvc-bb894f62-94c4-47a7-9b43-92deea8b50cc to node k8s-agentpool-27319582-vmss000001 successfully I0109 09:45:05.174813 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=20.480683172 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-ht4culj6" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/azuredisk-csi-driver-test-3d342f14-9002-11ed-9173-829d2d0c1a9d/providers/Microsoft.Compute/disks/pvc-bb894f62-94c4-47a7-9b43-92deea8b50cc" node="k8s-agentpool-27319582-vmss000001" result_code="succeeded" I0109 09:45:05.174834 1 utils.go:84] GRPC response: {"publish_context":{"LUN":"1"}} ... skipping 54 lines ... I0109 09:48:19.139076 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1207 I0109 09:48:19.176967 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 24988 I0109 09:48:19.179513 1 controllerserver.go:383] GetDiskLun returned: cannot find Lun for disk pvc-73a37549-54ad-4030-bad8-d8244d0eb839. Initiating attaching volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-73a37549-54ad-4030-bad8-d8244d0eb839 to node k8s-agentpool-27319582-vmss000001 (vmState Succeeded). I0109 09:48:19.179556 1 controllerserver.go:408] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-73a37549-54ad-4030-bad8-d8244d0eb839 to node k8s-agentpool-27319582-vmss000001 I0109 09:48:19.179630 1 azure_controller_common.go:255] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-73a37549-54ad-4030-bad8-d8244d0eb839 lun 0 to node k8s-agentpool-27319582-vmss000001, diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-ht4culj6/providers/microsoft.compute/disks/pvc-73a37549-54ad-4030-bad8-d8244d0eb839:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-73a37549-54ad-4030-bad8-d8244d0eb839 false 0})] I0109 09:48:19.179694 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-ht4culj6): vm(k8s-agentpool-27319582-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-ht4culj6/providers/microsoft.compute/disks/pvc-73a37549-54ad-4030-bad8-d8244d0eb839:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-73a37549-54ad-4030-bad8-d8244d0eb839 false 0})]) I0109 09:48:19.326836 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-ht4culj6): vm(k8s-agentpool-27319582-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-ht4culj6/providers/microsoft.compute/disks/pvc-73a37549-54ad-4030-bad8-d8244d0eb839:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-73a37549-54ad-4030-bad8-d8244d0eb839 false 0})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I0109 09:48:54.512361 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-ht4culj6, k8s-agentpool-27319582-vmss, k8s-agentpool-27319582-vmss000001) successfully I0109 09:48:54.512414 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool-27319582-vmss, kubetest-ht4culj6, k8s-agentpool-27319582-vmss000001) for cacheKey(kubetest-ht4culj6/k8s-agentpool-27319582-vmss) updated successfully I0109 09:48:54.512435 1 controllerserver.go:413] Attach operation successful: volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-73a37549-54ad-4030-bad8-d8244d0eb839 attached to node k8s-agentpool-27319582-vmss000001. I0109 09:48:54.512465 1 controllerserver.go:433] attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-73a37549-54ad-4030-bad8-d8244d0eb839 to node k8s-agentpool-27319582-vmss000001 successfully I0109 09:48:54.512542 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=35.373068293 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-ht4culj6" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-73a37549-54ad-4030-bad8-d8244d0eb839" node="k8s-agentpool-27319582-vmss000001" result_code="succeeded" I0109 09:48:54.512558 1 utils.go:84] GRPC response: {"publish_context":{"LUN":"0"}} ... skipping 45 lines ... I0109 09:50:13.365677 1 utils.go:78] GRPC request: {"node_id":"k8s-agentpool-27319582-vmss000001","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-4308824b-cf4f-4939-9d82-39d7d8fb6084","csi.storage.k8s.io/pvc/name":"pvc-4rb24","csi.storage.k8s.io/pvc/namespace":"azuredisk-2888","requestedsizegib":"10","skuName":"StandardSSD_ZRS","storage.kubernetes.io/csiProvisionerIdentity":"1673256990696-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-4308824b-cf4f-4939-9d82-39d7d8fb6084"} I0109 09:50:13.385499 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1193 I0109 09:50:13.385840 1 controllerserver.go:383] GetDiskLun returned: cannot find Lun for disk pvc-4308824b-cf4f-4939-9d82-39d7d8fb6084. Initiating attaching volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-4308824b-cf4f-4939-9d82-39d7d8fb6084 to node k8s-agentpool-27319582-vmss000001 (vmState Succeeded). I0109 09:50:13.385875 1 controllerserver.go:408] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-4308824b-cf4f-4939-9d82-39d7d8fb6084 to node k8s-agentpool-27319582-vmss000001 I0109 09:50:13.385909 1 azure_controller_common.go:255] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-4308824b-cf4f-4939-9d82-39d7d8fb6084 lun 0 to node k8s-agentpool-27319582-vmss000001, diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-ht4culj6/providers/microsoft.compute/disks/pvc-4308824b-cf4f-4939-9d82-39d7d8fb6084:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-4308824b-cf4f-4939-9d82-39d7d8fb6084 false 0})] I0109 09:50:13.385964 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-ht4culj6): vm(k8s-agentpool-27319582-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-ht4culj6/providers/microsoft.compute/disks/pvc-4308824b-cf4f-4939-9d82-39d7d8fb6084:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-4308824b-cf4f-4939-9d82-39d7d8fb6084 false 0})]) I0109 09:50:13.516244 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-ht4culj6): vm(k8s-agentpool-27319582-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-ht4culj6/providers/microsoft.compute/disks/pvc-4308824b-cf4f-4939-9d82-39d7d8fb6084:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-4308824b-cf4f-4939-9d82-39d7d8fb6084 false 0})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I0109 09:50:23.600517 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-ht4culj6, k8s-agentpool-27319582-vmss, k8s-agentpool-27319582-vmss000001) successfully I0109 09:50:23.600555 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool-27319582-vmss, kubetest-ht4culj6, k8s-agentpool-27319582-vmss000001) for cacheKey(kubetest-ht4culj6/k8s-agentpool-27319582-vmss) updated successfully I0109 09:50:23.600600 1 controllerserver.go:413] Attach operation successful: volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-4308824b-cf4f-4939-9d82-39d7d8fb6084 attached to node k8s-agentpool-27319582-vmss000001. I0109 09:50:23.600615 1 controllerserver.go:433] attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-4308824b-cf4f-4939-9d82-39d7d8fb6084 to node k8s-agentpool-27319582-vmss000001 successfully I0109 09:50:23.601014 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=10.214830521 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-ht4culj6" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-4308824b-cf4f-4939-9d82-39d7d8fb6084" node="k8s-agentpool-27319582-vmss000001" result_code="succeeded" I0109 09:50:23.601041 1 utils.go:84] GRPC response: {"publish_context":{"LUN":"0"}} ... skipping 19 lines ... I0109 09:50:37.671064 1 utils.go:78] GRPC request: {"node_id":"k8s-agentpool-27319582-vmss000001","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-00f6a5fd-590d-4eb1-9092-ee35e8314f41","csi.storage.k8s.io/pvc/name":"pvc-mrwkf","csi.storage.k8s.io/pvc/namespace":"azuredisk-2888","requestedsizegib":"10","skuName":"StandardSSD_ZRS","storage.kubernetes.io/csiProvisionerIdentity":"1673256990696-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-00f6a5fd-590d-4eb1-9092-ee35e8314f41"} I0109 09:50:37.782124 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1193 I0109 09:50:37.782508 1 controllerserver.go:383] GetDiskLun returned: cannot find Lun for disk pvc-00f6a5fd-590d-4eb1-9092-ee35e8314f41. Initiating attaching volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-00f6a5fd-590d-4eb1-9092-ee35e8314f41 to node k8s-agentpool-27319582-vmss000001 (vmState Succeeded). I0109 09:50:37.782545 1 controllerserver.go:408] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-00f6a5fd-590d-4eb1-9092-ee35e8314f41 to node k8s-agentpool-27319582-vmss000001 I0109 09:50:37.782600 1 azure_controller_common.go:255] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-00f6a5fd-590d-4eb1-9092-ee35e8314f41 lun 1 to node k8s-agentpool-27319582-vmss000001, diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-ht4culj6/providers/microsoft.compute/disks/pvc-00f6a5fd-590d-4eb1-9092-ee35e8314f41:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-00f6a5fd-590d-4eb1-9092-ee35e8314f41 false 1})] I0109 09:50:37.782651 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-ht4culj6): vm(k8s-agentpool-27319582-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-ht4culj6/providers/microsoft.compute/disks/pvc-00f6a5fd-590d-4eb1-9092-ee35e8314f41:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-00f6a5fd-590d-4eb1-9092-ee35e8314f41 false 1})]) I0109 09:50:37.947148 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-ht4culj6): vm(k8s-agentpool-27319582-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-ht4culj6/providers/microsoft.compute/disks/pvc-00f6a5fd-590d-4eb1-9092-ee35e8314f41:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-00f6a5fd-590d-4eb1-9092-ee35e8314f41 false 1})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I0109 09:50:48.043527 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-ht4culj6, k8s-agentpool-27319582-vmss, k8s-agentpool-27319582-vmss000001) successfully I0109 09:50:48.043575 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool-27319582-vmss, kubetest-ht4culj6, k8s-agentpool-27319582-vmss000001) for cacheKey(kubetest-ht4culj6/k8s-agentpool-27319582-vmss) updated successfully I0109 09:50:48.043639 1 controllerserver.go:413] Attach operation successful: volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-00f6a5fd-590d-4eb1-9092-ee35e8314f41 attached to node k8s-agentpool-27319582-vmss000001. I0109 09:50:48.043656 1 controllerserver.go:433] attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-00f6a5fd-590d-4eb1-9092-ee35e8314f41 to node k8s-agentpool-27319582-vmss000001 successfully I0109 09:50:48.043732 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=10.261186352 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-ht4culj6" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-00f6a5fd-590d-4eb1-9092-ee35e8314f41" node="k8s-agentpool-27319582-vmss000001" result_code="succeeded" I0109 09:50:48.043796 1 utils.go:84] GRPC response: {"publish_context":{"LUN":"1"}} ... skipping 19 lines ... I0109 09:50:59.919609 1 utils.go:78] GRPC request: {"node_id":"k8s-agentpool-27319582-vmss000000","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-9368e88b-80fb-4151-b9d6-b67f16c63fd6","csi.storage.k8s.io/pvc/name":"pvc-hgpj8","csi.storage.k8s.io/pvc/namespace":"azuredisk-2888","requestedsizegib":"10","skuName":"StandardSSD_ZRS","storage.kubernetes.io/csiProvisionerIdentity":"1673256990696-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-9368e88b-80fb-4151-b9d6-b67f16c63fd6"} I0109 09:50:59.939313 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1193 I0109 09:50:59.939685 1 controllerserver.go:383] GetDiskLun returned: cannot find Lun for disk pvc-9368e88b-80fb-4151-b9d6-b67f16c63fd6. Initiating attaching volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-9368e88b-80fb-4151-b9d6-b67f16c63fd6 to node k8s-agentpool-27319582-vmss000000 (vmState Succeeded). I0109 09:50:59.939715 1 controllerserver.go:408] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-9368e88b-80fb-4151-b9d6-b67f16c63fd6 to node k8s-agentpool-27319582-vmss000000 I0109 09:50:59.939749 1 azure_controller_common.go:255] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-9368e88b-80fb-4151-b9d6-b67f16c63fd6 lun 0 to node k8s-agentpool-27319582-vmss000000, diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-ht4culj6/providers/microsoft.compute/disks/pvc-9368e88b-80fb-4151-b9d6-b67f16c63fd6:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-9368e88b-80fb-4151-b9d6-b67f16c63fd6 false 0})] I0109 09:50:59.939789 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-ht4culj6): vm(k8s-agentpool-27319582-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-ht4culj6/providers/microsoft.compute/disks/pvc-9368e88b-80fb-4151-b9d6-b67f16c63fd6:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-9368e88b-80fb-4151-b9d6-b67f16c63fd6 false 0})]) I0109 09:51:00.119623 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-ht4culj6): vm(k8s-agentpool-27319582-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-ht4culj6/providers/microsoft.compute/disks/pvc-9368e88b-80fb-4151-b9d6-b67f16c63fd6:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-9368e88b-80fb-4151-b9d6-b67f16c63fd6 false 0})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I0109 09:51:10.223280 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-ht4culj6, k8s-agentpool-27319582-vmss, k8s-agentpool-27319582-vmss000000) successfully I0109 09:51:10.223330 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool-27319582-vmss, kubetest-ht4culj6, k8s-agentpool-27319582-vmss000000) for cacheKey(kubetest-ht4culj6/k8s-agentpool-27319582-vmss) updated successfully I0109 09:51:10.223353 1 controllerserver.go:413] Attach operation successful: volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-9368e88b-80fb-4151-b9d6-b67f16c63fd6 attached to node k8s-agentpool-27319582-vmss000000. I0109 09:51:10.223369 1 controllerserver.go:433] attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-9368e88b-80fb-4151-b9d6-b67f16c63fd6 to node k8s-agentpool-27319582-vmss000000 successfully I0109 09:51:10.223412 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=10.283718955 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-ht4culj6" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-9368e88b-80fb-4151-b9d6-b67f16c63fd6" node="k8s-agentpool-27319582-vmss000000" result_code="succeeded" I0109 09:51:10.223429 1 utils.go:84] GRPC response: {"publish_context":{"LUN":"0"}} ... skipping 75 lines ... I0109 09:54:48.830983 1 azure_vmss_cache.go:327] refresh the cache of NonVmssUniformNodesCache in rg map[kubetest-ht4culj6:{}] I0109 09:54:48.868293 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 12 I0109 09:54:48.868407 1 controllerserver.go:383] GetDiskLun returned: cannot find Lun for disk pvc-897be4b9-71d0-4822-8dc4-e93639c68b83. Initiating attaching volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-897be4b9-71d0-4822-8dc4-e93639c68b83 to node k8s-agentpool-27319582-vmss000001 (vmState Succeeded). I0109 09:54:48.868442 1 controllerserver.go:408] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-897be4b9-71d0-4822-8dc4-e93639c68b83 to node k8s-agentpool-27319582-vmss000001 I0109 09:54:48.868482 1 azure_controller_common.go:255] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-897be4b9-71d0-4822-8dc4-e93639c68b83 lun 0 to node k8s-agentpool-27319582-vmss000001, diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-ht4culj6/providers/microsoft.compute/disks/pvc-897be4b9-71d0-4822-8dc4-e93639c68b83:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-897be4b9-71d0-4822-8dc4-e93639c68b83 false 0})] I0109 09:54:48.868528 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-ht4culj6): vm(k8s-agentpool-27319582-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-ht4culj6/providers/microsoft.compute/disks/pvc-897be4b9-71d0-4822-8dc4-e93639c68b83:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-897be4b9-71d0-4822-8dc4-e93639c68b83 false 0})]) I0109 09:54:49.026526 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-ht4culj6): vm(k8s-agentpool-27319582-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-ht4culj6/providers/microsoft.compute/disks/pvc-897be4b9-71d0-4822-8dc4-e93639c68b83:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-897be4b9-71d0-4822-8dc4-e93639c68b83 false 0})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I0109 09:54:59.105008 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-ht4culj6, k8s-agentpool-27319582-vmss, k8s-agentpool-27319582-vmss000001) successfully I0109 09:54:59.105052 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool-27319582-vmss, kubetest-ht4culj6, k8s-agentpool-27319582-vmss000001) for cacheKey(kubetest-ht4culj6/k8s-agentpool-27319582-vmss) updated successfully I0109 09:54:59.105088 1 controllerserver.go:413] Attach operation successful: volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-897be4b9-71d0-4822-8dc4-e93639c68b83 attached to node k8s-agentpool-27319582-vmss000001. I0109 09:54:59.105102 1 controllerserver.go:433] attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-897be4b9-71d0-4822-8dc4-e93639c68b83 to node k8s-agentpool-27319582-vmss000001 successfully I0109 09:54:59.105161 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=10.274153756 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-ht4culj6" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-897be4b9-71d0-4822-8dc4-e93639c68b83" node="k8s-agentpool-27319582-vmss000001" result_code="succeeded" I0109 09:54:59.105175 1 utils.go:84] GRPC response: {"publish_context":{"LUN":"0"}} ... skipping 57 lines ... I0109 09:57:40.288718 1 utils.go:78] GRPC request: {"node_id":"k8s-agentpool-27319582-vmss000001","volume_capability":{"AccessType":{"Mount":{"fs_type":"xfs"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-584ec444-2d95-413c-b3ae-aa2ec2a7c063","csi.storage.k8s.io/pvc/name":"pvc-w8nxc","csi.storage.k8s.io/pvc/namespace":"azuredisk-59","fsType":"xfs","requestedsizegib":"10","skuName":"Standard_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1673256990696-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-584ec444-2d95-413c-b3ae-aa2ec2a7c063"} I0109 09:57:40.308307 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1217 I0109 09:57:40.308720 1 controllerserver.go:383] GetDiskLun returned: cannot find Lun for disk pvc-584ec444-2d95-413c-b3ae-aa2ec2a7c063. Initiating attaching volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-584ec444-2d95-413c-b3ae-aa2ec2a7c063 to node k8s-agentpool-27319582-vmss000001 (vmState Succeeded). I0109 09:57:40.308751 1 controllerserver.go:408] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-584ec444-2d95-413c-b3ae-aa2ec2a7c063 to node k8s-agentpool-27319582-vmss000001 I0109 09:57:40.308820 1 azure_controller_common.go:255] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-584ec444-2d95-413c-b3ae-aa2ec2a7c063 lun 0 to node k8s-agentpool-27319582-vmss000001, diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-ht4culj6/providers/microsoft.compute/disks/pvc-584ec444-2d95-413c-b3ae-aa2ec2a7c063:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-584ec444-2d95-413c-b3ae-aa2ec2a7c063 false 0})] I0109 09:57:40.308862 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-ht4culj6): vm(k8s-agentpool-27319582-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-ht4culj6/providers/microsoft.compute/disks/pvc-584ec444-2d95-413c-b3ae-aa2ec2a7c063:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-584ec444-2d95-413c-b3ae-aa2ec2a7c063 false 0})]) I0109 09:57:40.450507 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-ht4culj6): vm(k8s-agentpool-27319582-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-ht4culj6/providers/microsoft.compute/disks/pvc-584ec444-2d95-413c-b3ae-aa2ec2a7c063:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-584ec444-2d95-413c-b3ae-aa2ec2a7c063 false 0})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I0109 09:57:50.528517 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-ht4culj6, k8s-agentpool-27319582-vmss, k8s-agentpool-27319582-vmss000001) successfully I0109 09:57:50.528556 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool-27319582-vmss, kubetest-ht4culj6, k8s-agentpool-27319582-vmss000001) for cacheKey(kubetest-ht4culj6/k8s-agentpool-27319582-vmss) updated successfully I0109 09:57:50.528578 1 controllerserver.go:413] Attach operation successful: volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-584ec444-2d95-413c-b3ae-aa2ec2a7c063 attached to node k8s-agentpool-27319582-vmss000001. I0109 09:57:50.528594 1 controllerserver.go:433] attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-584ec444-2d95-413c-b3ae-aa2ec2a7c063 to node k8s-agentpool-27319582-vmss000001 successfully I0109 09:57:50.528845 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=10.219951748 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-ht4culj6" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-584ec444-2d95-413c-b3ae-aa2ec2a7c063" node="k8s-agentpool-27319582-vmss000001" result_code="succeeded" I0109 09:57:50.528941 1 utils.go:84] GRPC response: {"publish_context":{"LUN":"0"}} ... skipping 28 lines ... I0109 09:58:25.327520 1 utils.go:77] GRPC call: /csi.v1.Controller/ControllerUnpublishVolume I0109 09:58:25.327563 1 utils.go:78] GRPC request: {"node_id":"k8s-agentpool-27319582-vmss000001","volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-584ec444-2d95-413c-b3ae-aa2ec2a7c063"} I0109 09:58:25.327739 1 controllerserver.go:471] Trying to detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-584ec444-2d95-413c-b3ae-aa2ec2a7c063 from node k8s-agentpool-27319582-vmss000001 I0109 09:58:25.346107 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 24988 I0109 09:58:25.348882 1 azure_controller_common.go:255] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-fd0dd139-c95c-4c6f-b6aa-4b7e02e7ff1a lun 0 to node k8s-agentpool-27319582-vmss000001, diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-ht4culj6/providers/microsoft.compute/disks/pvc-fd0dd139-c95c-4c6f-b6aa-4b7e02e7ff1a:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-fd0dd139-c95c-4c6f-b6aa-4b7e02e7ff1a false 0})] I0109 09:58:25.348924 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-ht4culj6): vm(k8s-agentpool-27319582-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-ht4culj6/providers/microsoft.compute/disks/pvc-fd0dd139-c95c-4c6f-b6aa-4b7e02e7ff1a:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-fd0dd139-c95c-4c6f-b6aa-4b7e02e7ff1a false 0})]) I0109 09:58:25.619334 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-ht4culj6): vm(k8s-agentpool-27319582-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-ht4culj6/providers/microsoft.compute/disks/pvc-fd0dd139-c95c-4c6f-b6aa-4b7e02e7ff1a:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-fd0dd139-c95c-4c6f-b6aa-4b7e02e7ff1a false 0})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I0109 09:58:40.728185 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-ht4culj6, k8s-agentpool-27319582-vmss, k8s-agentpool-27319582-vmss000001) successfully I0109 09:58:40.728220 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool-27319582-vmss, kubetest-ht4culj6, k8s-agentpool-27319582-vmss000001) for cacheKey(kubetest-ht4culj6/k8s-agentpool-27319582-vmss) updated successfully I0109 09:58:40.728281 1 controllerserver.go:413] Attach operation successful: volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-fd0dd139-c95c-4c6f-b6aa-4b7e02e7ff1a attached to node k8s-agentpool-27319582-vmss000001. I0109 09:58:40.728302 1 controllerserver.go:433] attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-fd0dd139-c95c-4c6f-b6aa-4b7e02e7ff1a to node k8s-agentpool-27319582-vmss000001 successfully I0109 09:58:40.728362 1 azure_controller_common.go:398] Trying to detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-584ec444-2d95-413c-b3ae-aa2ec2a7c063 from node k8s-agentpool-27319582-vmss000001, diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-ht4culj6/providers/microsoft.compute/disks/pvc-584ec444-2d95-413c-b3ae-aa2ec2a7c063:pvc-584ec444-2d95-413c-b3ae-aa2ec2a7c063] E0109 09:58:40.728407 1 azure_controller_vmss.go:202] detach azure disk on node(k8s-agentpool-27319582-vmss000001): disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-ht4culj6/providers/microsoft.compute/disks/pvc-584ec444-2d95-413c-b3ae-aa2ec2a7c063:pvc-584ec444-2d95-413c-b3ae-aa2ec2a7c063]) not found ... skipping 50 lines ... I0109 10:00:14.324978 1 utils.go:78] GRPC request: {"node_id":"k8s-agentpool-27319582-vmss000001","volume_capability":{"AccessType":{"Mount":{"fs_type":"xfs"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-e8cabb73-ab02-4a5f-b1e7-f3a709038f7a","csi.storage.k8s.io/pvc/name":"pvc-xrkzd","csi.storage.k8s.io/pvc/namespace":"azuredisk-2546","fsType":"xfs","networkAccessPolicy":"DenyAll","requestedsizegib":"10","skuName":"StandardSSD_ZRS","storage.kubernetes.io/csiProvisionerIdentity":"1673256990696-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-e8cabb73-ab02-4a5f-b1e7-f3a709038f7a"} I0109 10:00:14.345231 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1192 I0109 10:00:14.345566 1 controllerserver.go:383] GetDiskLun returned: cannot find Lun for disk pvc-e8cabb73-ab02-4a5f-b1e7-f3a709038f7a. Initiating attaching volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-e8cabb73-ab02-4a5f-b1e7-f3a709038f7a to node k8s-agentpool-27319582-vmss000001 (vmState Succeeded). I0109 10:00:14.345593 1 controllerserver.go:408] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-e8cabb73-ab02-4a5f-b1e7-f3a709038f7a to node k8s-agentpool-27319582-vmss000001 I0109 10:00:14.345627 1 azure_controller_common.go:255] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-e8cabb73-ab02-4a5f-b1e7-f3a709038f7a lun 0 to node k8s-agentpool-27319582-vmss000001, diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-ht4culj6/providers/microsoft.compute/disks/pvc-e8cabb73-ab02-4a5f-b1e7-f3a709038f7a:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-e8cabb73-ab02-4a5f-b1e7-f3a709038f7a false 0})] I0109 10:00:14.345663 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-ht4culj6): vm(k8s-agentpool-27319582-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-ht4culj6/providers/microsoft.compute/disks/pvc-e8cabb73-ab02-4a5f-b1e7-f3a709038f7a:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-e8cabb73-ab02-4a5f-b1e7-f3a709038f7a false 0})]) I0109 10:00:14.462624 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-ht4culj6): vm(k8s-agentpool-27319582-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-ht4culj6/providers/microsoft.compute/disks/pvc-e8cabb73-ab02-4a5f-b1e7-f3a709038f7a:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-e8cabb73-ab02-4a5f-b1e7-f3a709038f7a false 0})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I0109 10:00:24.583972 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-ht4culj6, k8s-agentpool-27319582-vmss, k8s-agentpool-27319582-vmss000001) successfully I0109 10:00:24.584015 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool-27319582-vmss, kubetest-ht4culj6, k8s-agentpool-27319582-vmss000001) for cacheKey(kubetest-ht4culj6/k8s-agentpool-27319582-vmss) updated successfully I0109 10:00:24.584033 1 controllerserver.go:413] Attach operation successful: volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-e8cabb73-ab02-4a5f-b1e7-f3a709038f7a attached to node k8s-agentpool-27319582-vmss000001. I0109 10:00:24.584043 1 controllerserver.go:433] attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-e8cabb73-ab02-4a5f-b1e7-f3a709038f7a to node k8s-agentpool-27319582-vmss000001 successfully I0109 10:00:24.584085 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=10.238524174 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-ht4culj6" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-e8cabb73-ab02-4a5f-b1e7-f3a709038f7a" node="k8s-agentpool-27319582-vmss000001" result_code="succeeded" I0109 10:00:24.584100 1 utils.go:84] GRPC response: {"publish_context":{"LUN":"0"}} ... skipping 20 lines ... I0109 10:00:43.655086 1 utils.go:78] GRPC request: {"node_id":"k8s-agentpool-27319582-vmss000001","volume_capability":{"AccessType":{"Mount":{"fs_type":"xfs"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-94e7935f-a640-446c-8b98-cfbc3188bd4b","csi.storage.k8s.io/pvc/name":"pvc-59k5s","csi.storage.k8s.io/pvc/namespace":"azuredisk-2546","fsType":"xfs","networkAccessPolicy":"DenyAll","requestedsizegib":"20","resizeRequired":"true","skuName":"StandardSSD_ZRS","storage.kubernetes.io/csiProvisionerIdentity":"1673256990696-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-94e7935f-a640-446c-8b98-cfbc3188bd4b"} I0109 10:00:43.676137 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1446 I0109 10:00:43.676641 1 controllerserver.go:383] GetDiskLun returned: cannot find Lun for disk pvc-94e7935f-a640-446c-8b98-cfbc3188bd4b. Initiating attaching volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-94e7935f-a640-446c-8b98-cfbc3188bd4b to node k8s-agentpool-27319582-vmss000001 (vmState Succeeded). I0109 10:00:43.676677 1 controllerserver.go:408] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-94e7935f-a640-446c-8b98-cfbc3188bd4b to node k8s-agentpool-27319582-vmss000001 I0109 10:00:43.676741 1 azure_controller_common.go:255] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-94e7935f-a640-446c-8b98-cfbc3188bd4b lun 1 to node k8s-agentpool-27319582-vmss000001, diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-ht4culj6/providers/microsoft.compute/disks/pvc-94e7935f-a640-446c-8b98-cfbc3188bd4b:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-94e7935f-a640-446c-8b98-cfbc3188bd4b false 1})] I0109 10:00:43.676962 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-ht4culj6): vm(k8s-agentpool-27319582-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-ht4culj6/providers/microsoft.compute/disks/pvc-94e7935f-a640-446c-8b98-cfbc3188bd4b:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-94e7935f-a640-446c-8b98-cfbc3188bd4b false 1})]) I0109 10:00:43.917250 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-ht4culj6): vm(k8s-agentpool-27319582-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-ht4culj6/providers/microsoft.compute/disks/pvc-94e7935f-a640-446c-8b98-cfbc3188bd4b:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-94e7935f-a640-446c-8b98-cfbc3188bd4b false 1})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I0109 10:00:45.172670 1 utils.go:77] GRPC call: /csi.v1.Controller/ControllerUnpublishVolume I0109 10:00:45.172698 1 utils.go:78] GRPC request: {"node_id":"k8s-agentpool-27319582-vmss000001","volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-e8cabb73-ab02-4a5f-b1e7-f3a709038f7a"} I0109 10:00:45.172826 1 controllerserver.go:471] Trying to detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-e8cabb73-ab02-4a5f-b1e7-f3a709038f7a from node k8s-agentpool-27319582-vmss000001 I0109 10:00:54.035168 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-ht4culj6, k8s-agentpool-27319582-vmss, k8s-agentpool-27319582-vmss000001) successfully I0109 10:00:54.035210 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool-27319582-vmss, kubetest-ht4culj6, k8s-agentpool-27319582-vmss000001) for cacheKey(kubetest-ht4culj6/k8s-agentpool-27319582-vmss) updated successfully I0109 10:00:54.035248 1 controllerserver.go:413] Attach operation successful: volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-94e7935f-a640-446c-8b98-cfbc3188bd4b attached to node k8s-agentpool-27319582-vmss000001. ... skipping 83 lines ... I0109 10:02:00.561752 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1193 I0109 10:02:00.562118 1 controllerserver.go:383] GetDiskLun returned: cannot find Lun for disk pvc-858820b3-b0cb-417e-838b-39749c4f4cb1. Initiating attaching volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-858820b3-b0cb-417e-838b-39749c4f4cb1 to node k8s-agentpool-27319582-vmss000001 (vmState Succeeded). I0109 10:02:00.562155 1 controllerserver.go:408] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-858820b3-b0cb-417e-838b-39749c4f4cb1 to node k8s-agentpool-27319582-vmss000001 I0109 10:02:00.571481 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1193 I0109 10:02:00.571778 1 controllerserver.go:383] GetDiskLun returned: cannot find Lun for disk pvc-798a7a25-1155-4986-a546-10d3d1e3321c. Initiating attaching volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-798a7a25-1155-4986-a546-10d3d1e3321c to node k8s-agentpool-27319582-vmss000001 (vmState Succeeded). I0109 10:02:00.571821 1 controllerserver.go:408] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-798a7a25-1155-4986-a546-10d3d1e3321c to node k8s-agentpool-27319582-vmss000001 I0109 10:02:00.716594 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-ht4culj6): vm(k8s-agentpool-27319582-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-ht4culj6/providers/microsoft.compute/disks/pvc-3b6be1c2-7307-4d93-943c-bebf53cb2c61:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-3b6be1c2-7307-4d93-943c-bebf53cb2c61 false 0})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I0109 10:02:10.860934 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-ht4culj6, k8s-agentpool-27319582-vmss, k8s-agentpool-27319582-vmss000001) successfully I0109 10:02:10.860959 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool-27319582-vmss, kubetest-ht4culj6, k8s-agentpool-27319582-vmss000001) for cacheKey(kubetest-ht4culj6/k8s-agentpool-27319582-vmss) updated successfully I0109 10:02:10.860974 1 controllerserver.go:413] Attach operation successful: volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-3b6be1c2-7307-4d93-943c-bebf53cb2c61 attached to node k8s-agentpool-27319582-vmss000001. I0109 10:02:10.860983 1 controllerserver.go:433] attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-3b6be1c2-7307-4d93-943c-bebf53cb2c61 to node k8s-agentpool-27319582-vmss000001 successfully I0109 10:02:10.861009 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=10.317975019 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-ht4culj6" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-3b6be1c2-7307-4d93-943c-bebf53cb2c61" node="k8s-agentpool-27319582-vmss000001" result_code="succeeded" I0109 10:02:10.861019 1 utils.go:84] GRPC response: {"publish_context":{"LUN":"0"}} I0109 10:02:10.861102 1 azure_controller_common.go:255] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-858820b3-b0cb-417e-838b-39749c4f4cb1 lun 2 to node k8s-agentpool-27319582-vmss000001, diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-ht4culj6/providers/microsoft.compute/disks/pvc-798a7a25-1155-4986-a546-10d3d1e3321c:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-798a7a25-1155-4986-a546-10d3d1e3321c false 1}) /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-ht4culj6/providers/microsoft.compute/disks/pvc-858820b3-b0cb-417e-838b-39749c4f4cb1:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-858820b3-b0cb-417e-838b-39749c4f4cb1 false 2})] I0109 10:02:10.861127 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-ht4culj6): vm(k8s-agentpool-27319582-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-ht4culj6/providers/microsoft.compute/disks/pvc-798a7a25-1155-4986-a546-10d3d1e3321c:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-798a7a25-1155-4986-a546-10d3d1e3321c false 1}) /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-ht4culj6/providers/microsoft.compute/disks/pvc-858820b3-b0cb-417e-838b-39749c4f4cb1:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-858820b3-b0cb-417e-838b-39749c4f4cb1 false 2})]) I0109 10:02:11.053983 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-ht4culj6): vm(k8s-agentpool-27319582-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-ht4culj6/providers/microsoft.compute/disks/pvc-798a7a25-1155-4986-a546-10d3d1e3321c:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-798a7a25-1155-4986-a546-10d3d1e3321c false 1}) /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-ht4culj6/providers/microsoft.compute/disks/pvc-858820b3-b0cb-417e-838b-39749c4f4cb1:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-858820b3-b0cb-417e-838b-39749c4f4cb1 false 2})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I0109 10:02:26.206594 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-ht4culj6, k8s-agentpool-27319582-vmss, k8s-agentpool-27319582-vmss000001) successfully I0109 10:02:26.206632 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool-27319582-vmss, kubetest-ht4culj6, k8s-agentpool-27319582-vmss000001) for cacheKey(kubetest-ht4culj6/k8s-agentpool-27319582-vmss) updated successfully I0109 10:02:26.206663 1 controllerserver.go:413] Attach operation successful: volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-858820b3-b0cb-417e-838b-39749c4f4cb1 attached to node k8s-agentpool-27319582-vmss000001. I0109 10:02:26.206678 1 controllerserver.go:433] attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-858820b3-b0cb-417e-838b-39749c4f4cb1 to node k8s-agentpool-27319582-vmss000001 successfully I0109 10:02:26.206746 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=25.644598287 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-ht4culj6" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-858820b3-b0cb-417e-838b-39749c4f4cb1" node="k8s-agentpool-27319582-vmss000001" result_code="succeeded" I0109 10:02:26.206779 1 azure_controller_common.go:255] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-798a7a25-1155-4986-a546-10d3d1e3321c lun 1 to node k8s-agentpool-27319582-vmss000001, diskMap: map[] ... skipping 103 lines ... I0109 10:03:53.820314 1 controllerserver.go:408] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-3eaf3c2c-7eec-4fd4-a6a6-f6ec380eb180 to node k8s-agentpool-27319582-vmss000001 I0109 10:03:53.820361 1 azure_controller_common.go:255] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-3eaf3c2c-7eec-4fd4-a6a6-f6ec380eb180 lun 0 to node k8s-agentpool-27319582-vmss000001, diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-ht4culj6/providers/microsoft.compute/disks/pvc-3eaf3c2c-7eec-4fd4-a6a6-f6ec380eb180:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-3eaf3c2c-7eec-4fd4-a6a6-f6ec380eb180 false 0})] I0109 10:03:53.820397 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-ht4culj6): vm(k8s-agentpool-27319582-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-ht4culj6/providers/microsoft.compute/disks/pvc-3eaf3c2c-7eec-4fd4-a6a6-f6ec380eb180:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-3eaf3c2c-7eec-4fd4-a6a6-f6ec380eb180 false 0})]) I0109 10:03:53.838886 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1193 I0109 10:03:53.839160 1 controllerserver.go:383] GetDiskLun returned: cannot find Lun for disk pvc-a4701b24-96dd-48a8-9fcd-eb3d369c66ec. Initiating attaching volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-a4701b24-96dd-48a8-9fcd-eb3d369c66ec to node k8s-agentpool-27319582-vmss000001 (vmState Succeeded). I0109 10:03:53.839190 1 controllerserver.go:408] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-a4701b24-96dd-48a8-9fcd-eb3d369c66ec to node k8s-agentpool-27319582-vmss000001 I0109 10:03:54.726186 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-ht4culj6): vm(k8s-agentpool-27319582-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-ht4culj6/providers/microsoft.compute/disks/pvc-3eaf3c2c-7eec-4fd4-a6a6-f6ec380eb180:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-3eaf3c2c-7eec-4fd4-a6a6-f6ec380eb180 false 0})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I0109 10:04:04.898269 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-ht4culj6, k8s-agentpool-27319582-vmss, k8s-agentpool-27319582-vmss000001) successfully I0109 10:04:04.898312 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool-27319582-vmss, kubetest-ht4culj6, k8s-agentpool-27319582-vmss000001) for cacheKey(kubetest-ht4culj6/k8s-agentpool-27319582-vmss) updated successfully I0109 10:04:04.898341 1 controllerserver.go:413] Attach operation successful: volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-3eaf3c2c-7eec-4fd4-a6a6-f6ec380eb180 attached to node k8s-agentpool-27319582-vmss000001. I0109 10:04:04.898356 1 controllerserver.go:433] attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-3eaf3c2c-7eec-4fd4-a6a6-f6ec380eb180 to node k8s-agentpool-27319582-vmss000001 successfully I0109 10:04:04.898470 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=11.078168074 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-ht4culj6" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-3eaf3c2c-7eec-4fd4-a6a6-f6ec380eb180" node="k8s-agentpool-27319582-vmss000001" result_code="succeeded" I0109 10:04:04.898507 1 utils.go:84] GRPC response: {"publish_context":{"LUN":"0"}} I0109 10:04:04.898642 1 azure_controller_common.go:255] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-a4701b24-96dd-48a8-9fcd-eb3d369c66ec lun 1 to node k8s-agentpool-27319582-vmss000001, diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-ht4culj6/providers/microsoft.compute/disks/pvc-a4701b24-96dd-48a8-9fcd-eb3d369c66ec:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-a4701b24-96dd-48a8-9fcd-eb3d369c66ec false 1})] I0109 10:04:04.898767 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-ht4culj6): vm(k8s-agentpool-27319582-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-ht4culj6/providers/microsoft.compute/disks/pvc-a4701b24-96dd-48a8-9fcd-eb3d369c66ec:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-a4701b24-96dd-48a8-9fcd-eb3d369c66ec false 1})]) I0109 10:04:05.047608 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-ht4culj6): vm(k8s-agentpool-27319582-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-ht4culj6/providers/microsoft.compute/disks/pvc-a4701b24-96dd-48a8-9fcd-eb3d369c66ec:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-a4701b24-96dd-48a8-9fcd-eb3d369c66ec false 1})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I0109 10:04:15.303671 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-ht4culj6, k8s-agentpool-27319582-vmss, k8s-agentpool-27319582-vmss000001) successfully I0109 10:04:15.303724 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool-27319582-vmss, kubetest-ht4culj6, k8s-agentpool-27319582-vmss000001) for cacheKey(kubetest-ht4culj6/k8s-agentpool-27319582-vmss) updated successfully I0109 10:04:15.303748 1 controllerserver.go:413] Attach operation successful: volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-a4701b24-96dd-48a8-9fcd-eb3d369c66ec attached to node k8s-agentpool-27319582-vmss000001. I0109 10:04:15.303762 1 controllerserver.go:433] attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-a4701b24-96dd-48a8-9fcd-eb3d369c66ec to node k8s-agentpool-27319582-vmss000001 successfully I0109 10:04:15.303923 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=21.464633225 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-ht4culj6" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-a4701b24-96dd-48a8-9fcd-eb3d369c66ec" node="k8s-agentpool-27319582-vmss000001" result_code="succeeded" I0109 10:04:15.303966 1 utils.go:84] GRPC response: {"publish_context":{"LUN":"1"}} ... skipping 53 lines ... I0109 10:05:56.751488 1 utils.go:78] GRPC request: {"node_id":"k8s-agentpool-27319582-vmss000001","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-3d70faa1-7b52-4b74-8e59-a7686f96f787","csi.storage.k8s.io/pvc/name":"pvc-wbrs2","csi.storage.k8s.io/pvc/namespace":"azuredisk-8582","requestedsizegib":"10","skuName":"StandardSSD_ZRS","storage.kubernetes.io/csiProvisionerIdentity":"1673256990696-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-3d70faa1-7b52-4b74-8e59-a7686f96f787"} I0109 10:05:56.794411 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1193 I0109 10:05:56.794816 1 controllerserver.go:383] GetDiskLun returned: cannot find Lun for disk pvc-3d70faa1-7b52-4b74-8e59-a7686f96f787. Initiating attaching volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-3d70faa1-7b52-4b74-8e59-a7686f96f787 to node k8s-agentpool-27319582-vmss000001 (vmState Succeeded). I0109 10:05:56.794852 1 controllerserver.go:408] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-3d70faa1-7b52-4b74-8e59-a7686f96f787 to node k8s-agentpool-27319582-vmss000001 I0109 10:05:56.794893 1 azure_controller_common.go:255] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-3d70faa1-7b52-4b74-8e59-a7686f96f787 lun 0 to node k8s-agentpool-27319582-vmss000001, diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-ht4culj6/providers/microsoft.compute/disks/pvc-3d70faa1-7b52-4b74-8e59-a7686f96f787:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-3d70faa1-7b52-4b74-8e59-a7686f96f787 false 0})] I0109 10:05:56.794935 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-ht4culj6): vm(k8s-agentpool-27319582-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-ht4culj6/providers/microsoft.compute/disks/pvc-3d70faa1-7b52-4b74-8e59-a7686f96f787:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-3d70faa1-7b52-4b74-8e59-a7686f96f787 false 0})]) I0109 10:05:56.974961 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-ht4culj6): vm(k8s-agentpool-27319582-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-ht4culj6/providers/microsoft.compute/disks/pvc-3d70faa1-7b52-4b74-8e59-a7686f96f787:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-3d70faa1-7b52-4b74-8e59-a7686f96f787 false 0})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I0109 10:06:07.108662 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-ht4culj6, k8s-agentpool-27319582-vmss, k8s-agentpool-27319582-vmss000001) successfully I0109 10:06:07.108704 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool-27319582-vmss, kubetest-ht4culj6, k8s-agentpool-27319582-vmss000001) for cacheKey(kubetest-ht4culj6/k8s-agentpool-27319582-vmss) updated successfully I0109 10:06:07.108726 1 controllerserver.go:413] Attach operation successful: volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-3d70faa1-7b52-4b74-8e59-a7686f96f787 attached to node k8s-agentpool-27319582-vmss000001. I0109 10:06:07.108740 1 controllerserver.go:433] attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-3d70faa1-7b52-4b74-8e59-a7686f96f787 to node k8s-agentpool-27319582-vmss000001 successfully I0109 10:06:07.108786 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=10.313971513 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-ht4culj6" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-3d70faa1-7b52-4b74-8e59-a7686f96f787" node="k8s-agentpool-27319582-vmss000001" result_code="succeeded" I0109 10:06:07.108806 1 utils.go:84] GRPC response: {"publish_context":{"LUN":"0"}} ... skipping 37 lines ... I0109 10:06:40.958462 1 azure_controller_common.go:255] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-c0837e08-a9ae-4f8a-9789-b8a7d26d0930 lun 0 to node k8s-agentpool-27319582-vmss000001, diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-ht4culj6/providers/microsoft.compute/disks/pvc-c0837e08-a9ae-4f8a-9789-b8a7d26d0930:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-c0837e08-a9ae-4f8a-9789-b8a7d26d0930 false 0})] I0109 10:06:40.958515 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-ht4culj6): vm(k8s-agentpool-27319582-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-ht4culj6/providers/microsoft.compute/disks/pvc-c0837e08-a9ae-4f8a-9789-b8a7d26d0930:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-c0837e08-a9ae-4f8a-9789-b8a7d26d0930 false 0})]) I0109 10:06:40.958461 1 utils.go:84] GRPC response: {} I0109 10:06:40.974308 1 utils.go:77] GRPC call: /csi.v1.Controller/ControllerUnpublishVolume I0109 10:06:40.974330 1 utils.go:78] GRPC request: {"node_id":"k8s-agentpool-27319582-vmss000001","volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-3d70faa1-7b52-4b74-8e59-a7686f96f787"} I0109 10:06:40.974731 1 controllerserver.go:471] Trying to detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-3d70faa1-7b52-4b74-8e59-a7686f96f787 from node k8s-agentpool-27319582-vmss000001 I0109 10:06:41.091049 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-ht4culj6): vm(k8s-agentpool-27319582-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-ht4culj6/providers/microsoft.compute/disks/pvc-c0837e08-a9ae-4f8a-9789-b8a7d26d0930:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-c0837e08-a9ae-4f8a-9789-b8a7d26d0930 false 0})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I0109 10:06:51.181771 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-ht4culj6, k8s-agentpool-27319582-vmss, k8s-agentpool-27319582-vmss000001) successfully I0109 10:06:51.181804 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool-27319582-vmss, kubetest-ht4culj6, k8s-agentpool-27319582-vmss000001) for cacheKey(kubetest-ht4culj6/k8s-agentpool-27319582-vmss) updated successfully I0109 10:06:51.181832 1 controllerserver.go:413] Attach operation successful: volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-c0837e08-a9ae-4f8a-9789-b8a7d26d0930 attached to node k8s-agentpool-27319582-vmss000001. I0109 10:06:51.181846 1 controllerserver.go:433] attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-c0837e08-a9ae-4f8a-9789-b8a7d26d0930 to node k8s-agentpool-27319582-vmss000001 successfully I0109 10:06:51.181888 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=13.646615927 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-ht4culj6" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-c0837e08-a9ae-4f8a-9789-b8a7d26d0930" node="k8s-agentpool-27319582-vmss000001" result_code="succeeded" I0109 10:06:51.181919 1 utils.go:84] GRPC response: {"publish_context":{"LUN":"0"}} ... skipping 65 lines ... I0109 10:09:08.885566 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1193 I0109 10:09:08.929332 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 24988 I0109 10:09:08.932665 1 controllerserver.go:383] GetDiskLun returned: cannot find Lun for disk pvc-09e21ed1-c551-4d2b-9631-96e625236c9b. Initiating attaching volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-09e21ed1-c551-4d2b-9631-96e625236c9b to node k8s-agentpool-27319582-vmss000001 (vmState Succeeded). I0109 10:09:08.932723 1 controllerserver.go:408] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-09e21ed1-c551-4d2b-9631-96e625236c9b to node k8s-agentpool-27319582-vmss000001 I0109 10:09:08.932768 1 azure_controller_common.go:255] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-09e21ed1-c551-4d2b-9631-96e625236c9b lun 0 to node k8s-agentpool-27319582-vmss000001, diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-ht4culj6/providers/microsoft.compute/disks/pvc-09e21ed1-c551-4d2b-9631-96e625236c9b:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-09e21ed1-c551-4d2b-9631-96e625236c9b false 0})] I0109 10:09:08.932809 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-ht4culj6): vm(k8s-agentpool-27319582-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-ht4culj6/providers/microsoft.compute/disks/pvc-09e21ed1-c551-4d2b-9631-96e625236c9b:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-09e21ed1-c551-4d2b-9631-96e625236c9b false 0})]) I0109 10:09:09.099081 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-ht4culj6): vm(k8s-agentpool-27319582-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-ht4culj6/providers/microsoft.compute/disks/pvc-09e21ed1-c551-4d2b-9631-96e625236c9b:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-09e21ed1-c551-4d2b-9631-96e625236c9b false 0})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I0109 10:09:19.184898 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-ht4culj6, k8s-agentpool-27319582-vmss, k8s-agentpool-27319582-vmss000001) successfully I0109 10:09:19.184968 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool-27319582-vmss, kubetest-ht4culj6, k8s-agentpool-27319582-vmss000001) for cacheKey(kubetest-ht4culj6/k8s-agentpool-27319582-vmss) updated successfully I0109 10:09:19.184989 1 controllerserver.go:413] Attach operation successful: volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-09e21ed1-c551-4d2b-9631-96e625236c9b attached to node k8s-agentpool-27319582-vmss000001. I0109 10:09:19.185003 1 controllerserver.go:433] attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-09e21ed1-c551-4d2b-9631-96e625236c9b to node k8s-agentpool-27319582-vmss000001 successfully I0109 10:09:19.185045 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=10.29900473 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-ht4culj6" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-09e21ed1-c551-4d2b-9631-96e625236c9b" node="k8s-agentpool-27319582-vmss000001" result_code="succeeded" I0109 10:09:19.185066 1 utils.go:84] GRPC response: {"publish_context":{"LUN":"0"}} ... skipping 31 lines ... I0109 10:09:51.285404 1 azure_vmss_cache.go:327] refresh the cache of NonVmssUniformNodesCache in rg map[kubetest-ht4culj6:{}] I0109 10:09:51.323222 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 12 I0109 10:09:51.323334 1 controllerserver.go:383] GetDiskLun returned: cannot find Lun for disk pvc-ce5cef3f-c034-41a3-b499-1759c90dab07. Initiating attaching volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-ce5cef3f-c034-41a3-b499-1759c90dab07 to node k8s-agentpool-27319582-vmss000000 (vmState Succeeded). I0109 10:09:51.323385 1 controllerserver.go:408] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-ce5cef3f-c034-41a3-b499-1759c90dab07 to node k8s-agentpool-27319582-vmss000000 I0109 10:09:51.323428 1 azure_controller_common.go:255] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-ce5cef3f-c034-41a3-b499-1759c90dab07 lun 0 to node k8s-agentpool-27319582-vmss000000, diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-ht4culj6/providers/microsoft.compute/disks/pvc-ce5cef3f-c034-41a3-b499-1759c90dab07:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-ce5cef3f-c034-41a3-b499-1759c90dab07 false 0})] I0109 10:09:51.323483 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-ht4culj6): vm(k8s-agentpool-27319582-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-ht4culj6/providers/microsoft.compute/disks/pvc-ce5cef3f-c034-41a3-b499-1759c90dab07:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-ce5cef3f-c034-41a3-b499-1759c90dab07 false 0})]) I0109 10:09:51.498436 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-ht4culj6): vm(k8s-agentpool-27319582-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-ht4culj6/providers/microsoft.compute/disks/pvc-ce5cef3f-c034-41a3-b499-1759c90dab07:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-ce5cef3f-c034-41a3-b499-1759c90dab07 false 0})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I0109 10:10:01.661539 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-ht4culj6, k8s-agentpool-27319582-vmss, k8s-agentpool-27319582-vmss000000) successfully I0109 10:10:01.661580 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool-27319582-vmss, kubetest-ht4culj6, k8s-agentpool-27319582-vmss000000) for cacheKey(kubetest-ht4culj6/k8s-agentpool-27319582-vmss) updated successfully I0109 10:10:01.661605 1 controllerserver.go:413] Attach operation successful: volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-ce5cef3f-c034-41a3-b499-1759c90dab07 attached to node k8s-agentpool-27319582-vmss000000. I0109 10:10:01.661623 1 controllerserver.go:433] attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-ce5cef3f-c034-41a3-b499-1759c90dab07 to node k8s-agentpool-27319582-vmss000000 successfully I0109 10:10:01.661675 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=10.376252628 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-ht4culj6" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-ce5cef3f-c034-41a3-b499-1759c90dab07" node="k8s-agentpool-27319582-vmss000000" result_code="succeeded" I0109 10:10:01.661696 1 utils.go:84] GRPC response: {"publish_context":{"LUN":"0"}} ... skipping 99 lines ... I0109 10:12:24.949993 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1222 I0109 10:12:24.950255 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1234 I0109 10:12:24.950552 1 controllerserver.go:383] GetDiskLun returned: cannot find Lun for disk pvc-b99ea9cf-7509-4d81-817c-8b483874e303. Initiating attaching volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-b99ea9cf-7509-4d81-817c-8b483874e303 to node k8s-agentpool-27319582-vmss000001 (vmState Succeeded). I0109 10:12:24.950736 1 controllerserver.go:408] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-b99ea9cf-7509-4d81-817c-8b483874e303 to node k8s-agentpool-27319582-vmss000001 I0109 10:12:24.950571 1 controllerserver.go:383] GetDiskLun returned: cannot find Lun for disk pvc-c877d96b-e13b-40eb-84be-955fe301f295. Initiating attaching volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-c877d96b-e13b-40eb-84be-955fe301f295 to node k8s-agentpool-27319582-vmss000001 (vmState Succeeded). I0109 10:12:24.950785 1 controllerserver.go:408] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-c877d96b-e13b-40eb-84be-955fe301f295 to node k8s-agentpool-27319582-vmss000001 I0109 10:12:25.140387 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-ht4culj6): vm(k8s-agentpool-27319582-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-ht4culj6/providers/microsoft.compute/disks/pvc-8937d8ec-cd5e-4dbb-9f78-9ab518360bb1:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-8937d8ec-cd5e-4dbb-9f78-9ab518360bb1 false 0})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I0109 10:12:35.272149 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-ht4culj6, k8s-agentpool-27319582-vmss, k8s-agentpool-27319582-vmss000001) successfully I0109 10:12:35.272189 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool-27319582-vmss, kubetest-ht4culj6, k8s-agentpool-27319582-vmss000001) for cacheKey(kubetest-ht4culj6/k8s-agentpool-27319582-vmss) updated successfully I0109 10:12:35.272315 1 controllerserver.go:413] Attach operation successful: volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-8937d8ec-cd5e-4dbb-9f78-9ab518360bb1 attached to node k8s-agentpool-27319582-vmss000001. I0109 10:12:35.272473 1 controllerserver.go:433] attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-8937d8ec-cd5e-4dbb-9f78-9ab518360bb1 to node k8s-agentpool-27319582-vmss000001 successfully I0109 10:12:35.272570 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=10.325662393 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-ht4culj6" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-8937d8ec-cd5e-4dbb-9f78-9ab518360bb1" node="k8s-agentpool-27319582-vmss000001" result_code="succeeded" I0109 10:12:35.272653 1 utils.go:84] GRPC response: {"publish_context":{"LUN":"0"}} ... skipping 4 lines ... I0109 10:12:35.327202 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1447 I0109 10:12:35.327647 1 azure_controller_common.go:516] azureDisk - find disk: lun 0 name pvc-8937d8ec-cd5e-4dbb-9f78-9ab518360bb1 uri /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-8937d8ec-cd5e-4dbb-9f78-9ab518360bb1 I0109 10:12:35.327668 1 controllerserver.go:383] GetDiskLun returned: <nil>. Initiating attaching volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-8937d8ec-cd5e-4dbb-9f78-9ab518360bb1 to node k8s-agentpool-27319582-vmss000001 (vmState Succeeded). I0109 10:12:35.327707 1 controllerserver.go:398] Attach operation is successful. volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-8937d8ec-cd5e-4dbb-9f78-9ab518360bb1 is already attached to node k8s-agentpool-27319582-vmss000001 at lun 0. I0109 10:12:35.327755 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=0.000146901 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-ht4culj6" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-8937d8ec-cd5e-4dbb-9f78-9ab518360bb1" node="k8s-agentpool-27319582-vmss000001" result_code="succeeded" I0109 10:12:35.327771 1 utils.go:84] GRPC response: {"publish_context":{"LUN":"0"}} I0109 10:12:35.466467 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-ht4culj6): vm(k8s-agentpool-27319582-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-ht4culj6/providers/microsoft.compute/disks/pvc-b99ea9cf-7509-4d81-817c-8b483874e303:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-b99ea9cf-7509-4d81-817c-8b483874e303 false 1}) /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-ht4culj6/providers/microsoft.compute/disks/pvc-c877d96b-e13b-40eb-84be-955fe301f295:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-c877d96b-e13b-40eb-84be-955fe301f295 false 2})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I0109 10:12:45.603661 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-ht4culj6, k8s-agentpool-27319582-vmss, k8s-agentpool-27319582-vmss000001) successfully I0109 10:12:45.603700 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool-27319582-vmss, kubetest-ht4culj6, k8s-agentpool-27319582-vmss000001) for cacheKey(kubetest-ht4culj6/k8s-agentpool-27319582-vmss) updated successfully I0109 10:12:45.603743 1 controllerserver.go:413] Attach operation successful: volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-b99ea9cf-7509-4d81-817c-8b483874e303 attached to node k8s-agentpool-27319582-vmss000001. I0109 10:12:45.603761 1 controllerserver.go:433] attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-b99ea9cf-7509-4d81-817c-8b483874e303 to node k8s-agentpool-27319582-vmss000001 successfully I0109 10:12:45.603831 1 azure_controller_common.go:255] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-c877d96b-e13b-40eb-84be-955fe301f295 lun 2 to node k8s-agentpool-27319582-vmss000001, diskMap: map[] I0109 10:12:45.603859 1 controllerserver.go:413] Attach operation successful: volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-c877d96b-e13b-40eb-84be-955fe301f295 attached to node k8s-agentpool-27319582-vmss000001. ... skipping 95 lines ... I0109 10:14:38.288941 1 utils.go:78] GRPC request: {"node_id":"k8s-agentpool-27319582-vmss000001","volume_capability":{"AccessType":{"Mount":{"mount_flags":["barrier=1","acl"]}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-b07f6f35-0735-44e2-bf51-2c90a120131e","csi.storage.k8s.io/pvc/name":"pvc-azuredisk-volume-tester-8d5rq-0","csi.storage.k8s.io/pvc/namespace":"azuredisk-1387","requestedsizegib":"10","skuName":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1673256990696-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-b07f6f35-0735-44e2-bf51-2c90a120131e"} I0109 10:14:38.313194 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1248 I0109 10:14:38.313540 1 controllerserver.go:383] GetDiskLun returned: cannot find Lun for disk pvc-b07f6f35-0735-44e2-bf51-2c90a120131e. Initiating attaching volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-b07f6f35-0735-44e2-bf51-2c90a120131e to node k8s-agentpool-27319582-vmss000001 (vmState Succeeded). I0109 10:14:38.313575 1 controllerserver.go:408] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-b07f6f35-0735-44e2-bf51-2c90a120131e to node k8s-agentpool-27319582-vmss000001 I0109 10:14:38.313710 1 azure_controller_common.go:255] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-b07f6f35-0735-44e2-bf51-2c90a120131e lun 0 to node k8s-agentpool-27319582-vmss000001, diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-ht4culj6/providers/microsoft.compute/disks/pvc-b07f6f35-0735-44e2-bf51-2c90a120131e:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-b07f6f35-0735-44e2-bf51-2c90a120131e false 0})] I0109 10:14:38.313862 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-ht4culj6): vm(k8s-agentpool-27319582-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-ht4culj6/providers/microsoft.compute/disks/pvc-b07f6f35-0735-44e2-bf51-2c90a120131e:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-b07f6f35-0735-44e2-bf51-2c90a120131e false 0})]) I0109 10:14:38.445922 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-ht4culj6): vm(k8s-agentpool-27319582-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-ht4culj6/providers/microsoft.compute/disks/pvc-b07f6f35-0735-44e2-bf51-2c90a120131e:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-b07f6f35-0735-44e2-bf51-2c90a120131e false 0})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I0109 10:14:48.537169 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-ht4culj6, k8s-agentpool-27319582-vmss, k8s-agentpool-27319582-vmss000001) successfully I0109 10:14:48.537227 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool-27319582-vmss, kubetest-ht4culj6, k8s-agentpool-27319582-vmss000001) for cacheKey(kubetest-ht4culj6/k8s-agentpool-27319582-vmss) updated successfully I0109 10:14:48.537246 1 controllerserver.go:413] Attach operation successful: volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-b07f6f35-0735-44e2-bf51-2c90a120131e attached to node k8s-agentpool-27319582-vmss000001. I0109 10:14:48.537260 1 controllerserver.go:433] attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-b07f6f35-0735-44e2-bf51-2c90a120131e to node k8s-agentpool-27319582-vmss000001 successfully I0109 10:14:48.537300 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=10.223752589 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-ht4culj6" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-b07f6f35-0735-44e2-bf51-2c90a120131e" node="k8s-agentpool-27319582-vmss000001" result_code="succeeded" I0109 10:14:48.537322 1 utils.go:84] GRPC response: {"publish_context":{"LUN":"0"}} ... skipping 35 lines ... I0109 10:17:30.912743 1 utils.go:78] GRPC request: {"node_id":"k8s-agentpool-27319582-vmss000001","volume_capability":{"AccessType":{"Mount":{"mount_flags":["barrier=1","acl"]}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-b07f6f35-0735-44e2-bf51-2c90a120131e","csi.storage.k8s.io/pvc/name":"pvc-azuredisk-volume-tester-8d5rq-0","csi.storage.k8s.io/pvc/namespace":"azuredisk-1387","requestedsizegib":"10","skuName":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1673256990696-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-b07f6f35-0735-44e2-bf51-2c90a120131e"} I0109 10:17:30.962316 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1248 I0109 10:17:30.962704 1 controllerserver.go:383] GetDiskLun returned: cannot find Lun for disk pvc-b07f6f35-0735-44e2-bf51-2c90a120131e. Initiating attaching volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-b07f6f35-0735-44e2-bf51-2c90a120131e to node k8s-agentpool-27319582-vmss000001 (vmState Succeeded). I0109 10:17:30.962743 1 controllerserver.go:408] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-b07f6f35-0735-44e2-bf51-2c90a120131e to node k8s-agentpool-27319582-vmss000001 I0109 10:17:30.962780 1 azure_controller_common.go:255] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-b07f6f35-0735-44e2-bf51-2c90a120131e lun 0 to node k8s-agentpool-27319582-vmss000001, diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-ht4culj6/providers/microsoft.compute/disks/pvc-b07f6f35-0735-44e2-bf51-2c90a120131e:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-b07f6f35-0735-44e2-bf51-2c90a120131e false 0})] I0109 10:17:30.962858 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-ht4culj6): vm(k8s-agentpool-27319582-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-ht4culj6/providers/microsoft.compute/disks/pvc-b07f6f35-0735-44e2-bf51-2c90a120131e:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-b07f6f35-0735-44e2-bf51-2c90a120131e false 0})]) I0109 10:17:31.188779 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-ht4culj6): vm(k8s-agentpool-27319582-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-ht4culj6/providers/microsoft.compute/disks/pvc-b07f6f35-0735-44e2-bf51-2c90a120131e:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-b07f6f35-0735-44e2-bf51-2c90a120131e false 0})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I0109 10:17:41.336042 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-ht4culj6, k8s-agentpool-27319582-vmss, k8s-agentpool-27319582-vmss000001) successfully I0109 10:17:41.336084 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool-27319582-vmss, kubetest-ht4culj6, k8s-agentpool-27319582-vmss000001) for cacheKey(kubetest-ht4culj6/k8s-agentpool-27319582-vmss) updated successfully I0109 10:17:41.336107 1 controllerserver.go:413] Attach operation successful: volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-b07f6f35-0735-44e2-bf51-2c90a120131e attached to node k8s-agentpool-27319582-vmss000001. I0109 10:17:41.336143 1 controllerserver.go:433] attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-b07f6f35-0735-44e2-bf51-2c90a120131e to node k8s-agentpool-27319582-vmss000001 successfully I0109 10:17:41.336202 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=10.373500094 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-ht4culj6" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-b07f6f35-0735-44e2-bf51-2c90a120131e" node="k8s-agentpool-27319582-vmss000001" result_code="succeeded" I0109 10:17:41.336218 1 utils.go:84] GRPC response: {"publish_context":{"LUN":"0"}} ... skipping 19 lines ... I0109 10:18:07.209994 1 utils.go:78] GRPC request: {"node_id":"k8s-agentpool-27319582-vmss000001","volume_capability":{"AccessType":{"Mount":{"mount_flags":["barrier=1","acl"]}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-45378fd8-5914-4ae8-8ed0-db101629b51e","csi.storage.k8s.io/pvc/name":"pvc-vxk9r","csi.storage.k8s.io/pvc/namespace":"azuredisk-4801","requestedsizegib":"10","skuName":"StandardSSD_ZRS","storage.kubernetes.io/csiProvisionerIdentity":"1673256990696-8081-disk.csi.azure.com","tags":"disk=test"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-45378fd8-5914-4ae8-8ed0-db101629b51e"} I0109 10:18:07.235913 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1214 I0109 10:18:07.236286 1 controllerserver.go:383] GetDiskLun returned: cannot find Lun for disk pvc-45378fd8-5914-4ae8-8ed0-db101629b51e. Initiating attaching volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-45378fd8-5914-4ae8-8ed0-db101629b51e to node k8s-agentpool-27319582-vmss000001 (vmState Succeeded). I0109 10:18:07.236323 1 controllerserver.go:408] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-45378fd8-5914-4ae8-8ed0-db101629b51e to node k8s-agentpool-27319582-vmss000001 I0109 10:18:07.236357 1 azure_controller_common.go:255] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-45378fd8-5914-4ae8-8ed0-db101629b51e lun 1 to node k8s-agentpool-27319582-vmss000001, diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-ht4culj6/providers/microsoft.compute/disks/pvc-45378fd8-5914-4ae8-8ed0-db101629b51e:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-45378fd8-5914-4ae8-8ed0-db101629b51e false 1})] I0109 10:18:07.236415 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-ht4culj6): vm(k8s-agentpool-27319582-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-ht4culj6/providers/microsoft.compute/disks/pvc-45378fd8-5914-4ae8-8ed0-db101629b51e:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-45378fd8-5914-4ae8-8ed0-db101629b51e false 1})]) I0109 10:18:07.410700 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-ht4culj6): vm(k8s-agentpool-27319582-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-ht4culj6/providers/microsoft.compute/disks/pvc-45378fd8-5914-4ae8-8ed0-db101629b51e:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-45378fd8-5914-4ae8-8ed0-db101629b51e false 1})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I0109 10:18:17.503796 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-ht4culj6, k8s-agentpool-27319582-vmss, k8s-agentpool-27319582-vmss000001) successfully I0109 10:18:17.503833 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool-27319582-vmss, kubetest-ht4culj6, k8s-agentpool-27319582-vmss000001) for cacheKey(kubetest-ht4culj6/k8s-agentpool-27319582-vmss) updated successfully I0109 10:18:17.503852 1 controllerserver.go:413] Attach operation successful: volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-45378fd8-5914-4ae8-8ed0-db101629b51e attached to node k8s-agentpool-27319582-vmss000001. I0109 10:18:17.503865 1 controllerserver.go:433] attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-45378fd8-5914-4ae8-8ed0-db101629b51e to node k8s-agentpool-27319582-vmss000001 successfully I0109 10:18:17.503988 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=10.267616018 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-ht4culj6" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-45378fd8-5914-4ae8-8ed0-db101629b51e" node="k8s-agentpool-27319582-vmss000001" result_code="succeeded" I0109 10:18:17.504007 1 utils.go:84] GRPC response: {"publish_context":{"LUN":"1"}} ... skipping 16 lines ... I0109 10:18:51.728848 1 azure_controller_common.go:398] Trying to detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-b07f6f35-0735-44e2-bf51-2c90a120131e from node k8s-agentpool-27319582-vmss000001, diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-ht4culj6/providers/microsoft.compute/disks/pvc-b07f6f35-0735-44e2-bf51-2c90a120131e:pvc-b07f6f35-0735-44e2-bf51-2c90a120131e] E0109 10:18:51.728999 1 azure_controller_vmss.go:202] detach azure disk on node(k8s-agentpool-27319582-vmss000001): disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-ht4culj6/providers/microsoft.compute/disks/pvc-b07f6f35-0735-44e2-bf51-2c90a120131e:pvc-b07f6f35-0735-44e2-bf51-2c90a120131e]) not found I0109 10:18:51.729089 1 azure_controller_vmss.go:239] azureDisk - update(kubetest-ht4culj6): vm(k8s-agentpool-27319582-vmss000001) - detach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-ht4culj6/providers/microsoft.compute/disks/pvc-b07f6f35-0735-44e2-bf51-2c90a120131e:pvc-b07f6f35-0735-44e2-bf51-2c90a120131e]) I0109 10:18:54.358778 1 utils.go:77] GRPC call: /csi.v1.Controller/DeleteVolume I0109 10:18:54.358808 1 utils.go:78] GRPC request: {"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-b07f6f35-0735-44e2-bf51-2c90a120131e"} I0109 10:18:54.358907 1 controllerserver.go:317] deleting azure disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-b07f6f35-0735-44e2-bf51-2c90a120131e) I0109 10:18:54.358928 1 controllerserver.go:319] delete azure disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-b07f6f35-0735-44e2-bf51-2c90a120131e) returned with failed to delete disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-b07f6f35-0735-44e2-bf51-2c90a120131e) since it's in attaching or detaching state I0109 10:18:54.359015 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=6.49e-05 request="azuredisk_csi_driver_controller_delete_volume" resource_group="kubetest-ht4culj6" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-b07f6f35-0735-44e2-bf51-2c90a120131e" result_code="failed_csi_driver_controller_delete_volume" E0109 10:18:54.359031 1 utils.go:82] GRPC error: failed to delete disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-b07f6f35-0735-44e2-bf51-2c90a120131e) since it's in attaching or detaching state I0109 10:18:57.024960 1 azure_controller_vmss.go:252] azureDisk - update(kubetest-ht4culj6): vm(k8s-agentpool-27319582-vmss000001) - detach disk(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-ht4culj6/providers/microsoft.compute/disks/pvc-b07f6f35-0735-44e2-bf51-2c90a120131e:pvc-b07f6f35-0735-44e2-bf51-2c90a120131e]) returned with <nil> I0109 10:18:57.025036 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-ht4culj6, k8s-agentpool-27319582-vmss, k8s-agentpool-27319582-vmss000001) successfully I0109 10:18:57.025054 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool-27319582-vmss, kubetest-ht4culj6, k8s-agentpool-27319582-vmss000001) for cacheKey(kubetest-ht4culj6/k8s-agentpool-27319582-vmss) updated successfully I0109 10:18:57.025065 1 azure_controller_common.go:422] azureDisk - detach disk(pvc-b07f6f35-0735-44e2-bf51-2c90a120131e, /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-b07f6f35-0735-44e2-bf51-2c90a120131e) succeeded I0109 10:18:57.025092 1 controllerserver.go:480] detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-b07f6f35-0735-44e2-bf51-2c90a120131e from node k8s-agentpool-27319582-vmss000001 successfully I0109 10:18:57.025129 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=5.296631853 request="azuredisk_csi_driver_controller_unpublish_volume" resource_group="kubetest-ht4culj6" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-b07f6f35-0735-44e2-bf51-2c90a120131e" node="k8s-agentpool-27319582-vmss000001" result_code="succeeded" ... skipping 42 lines ... I0109 10:19:42.400974 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1192 I0109 10:19:42.462174 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 24988 I0109 10:19:42.465138 1 controllerserver.go:383] GetDiskLun returned: cannot find Lun for disk pvc-3d880953-0d08-4e0b-8ad6-57c30fb55adf. Initiating attaching volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-3d880953-0d08-4e0b-8ad6-57c30fb55adf to node k8s-agentpool-27319582-vmss000001 (vmState Succeeded). I0109 10:19:42.465175 1 controllerserver.go:408] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-3d880953-0d08-4e0b-8ad6-57c30fb55adf to node k8s-agentpool-27319582-vmss000001 I0109 10:19:42.465255 1 azure_controller_common.go:255] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-3d880953-0d08-4e0b-8ad6-57c30fb55adf lun 0 to node k8s-agentpool-27319582-vmss000001, diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-ht4culj6/providers/microsoft.compute/disks/pvc-3d880953-0d08-4e0b-8ad6-57c30fb55adf:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-3d880953-0d08-4e0b-8ad6-57c30fb55adf false 0})] I0109 10:19:42.465324 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-ht4culj6): vm(k8s-agentpool-27319582-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-ht4culj6/providers/microsoft.compute/disks/pvc-3d880953-0d08-4e0b-8ad6-57c30fb55adf:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-3d880953-0d08-4e0b-8ad6-57c30fb55adf false 0})]) I0109 10:19:42.609600 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-ht4culj6): vm(k8s-agentpool-27319582-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-ht4culj6/providers/microsoft.compute/disks/pvc-3d880953-0d08-4e0b-8ad6-57c30fb55adf:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-3d880953-0d08-4e0b-8ad6-57c30fb55adf false 0})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I0109 10:19:52.704294 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-ht4culj6, k8s-agentpool-27319582-vmss, k8s-agentpool-27319582-vmss000001) successfully I0109 10:19:52.704393 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool-27319582-vmss, kubetest-ht4culj6, k8s-agentpool-27319582-vmss000001) for cacheKey(kubetest-ht4culj6/k8s-agentpool-27319582-vmss) updated successfully I0109 10:19:52.704450 1 controllerserver.go:413] Attach operation successful: volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-3d880953-0d08-4e0b-8ad6-57c30fb55adf attached to node k8s-agentpool-27319582-vmss000001. I0109 10:19:52.704469 1 controllerserver.go:433] attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-3d880953-0d08-4e0b-8ad6-57c30fb55adf to node k8s-agentpool-27319582-vmss000001 successfully I0109 10:19:52.704549 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=10.303256882 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-ht4culj6" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-3d880953-0d08-4e0b-8ad6-57c30fb55adf" node="k8s-agentpool-27319582-vmss000001" result_code="succeeded" I0109 10:19:52.704579 1 utils.go:84] GRPC response: {"publish_context":{"LUN":"0"}} ... skipping 31 lines ... I0109 10:21:19.683788 1 utils.go:78] GRPC request: {"node_id":"k8s-agentpool-27319582-vmss000001","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-6c3fd710-67dd-4238-b510-735a9f7fe1a4","csi.storage.k8s.io/pvc/name":"pvc-azuredisk-volume-tester-q8kk9-0","csi.storage.k8s.io/pvc/namespace":"azuredisk-1166","requestedsizegib":"10","skuName":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1673256990696-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-6c3fd710-67dd-4238-b510-735a9f7fe1a4"} I0109 10:21:19.707962 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1248 I0109 10:21:19.708612 1 controllerserver.go:383] GetDiskLun returned: cannot find Lun for disk pvc-6c3fd710-67dd-4238-b510-735a9f7fe1a4. Initiating attaching volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-6c3fd710-67dd-4238-b510-735a9f7fe1a4 to node k8s-agentpool-27319582-vmss000001 (vmState Succeeded). I0109 10:21:19.708705 1 controllerserver.go:408] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-6c3fd710-67dd-4238-b510-735a9f7fe1a4 to node k8s-agentpool-27319582-vmss000001 I0109 10:21:19.708914 1 azure_controller_common.go:255] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-6c3fd710-67dd-4238-b510-735a9f7fe1a4 lun 0 to node k8s-agentpool-27319582-vmss000001, diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-ht4culj6/providers/microsoft.compute/disks/pvc-6c3fd710-67dd-4238-b510-735a9f7fe1a4:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-6c3fd710-67dd-4238-b510-735a9f7fe1a4 false 0})] I0109 10:21:19.708971 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-ht4culj6): vm(k8s-agentpool-27319582-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-ht4culj6/providers/microsoft.compute/disks/pvc-6c3fd710-67dd-4238-b510-735a9f7fe1a4:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-6c3fd710-67dd-4238-b510-735a9f7fe1a4 false 0})]) I0109 10:21:19.880271 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-ht4culj6): vm(k8s-agentpool-27319582-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-ht4culj6/providers/microsoft.compute/disks/pvc-6c3fd710-67dd-4238-b510-735a9f7fe1a4:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-6c3fd710-67dd-4238-b510-735a9f7fe1a4 false 0})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I0109 10:21:30.019218 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-ht4culj6, k8s-agentpool-27319582-vmss, k8s-agentpool-27319582-vmss000001) successfully I0109 10:21:30.019255 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool-27319582-vmss, kubetest-ht4culj6, k8s-agentpool-27319582-vmss000001) for cacheKey(kubetest-ht4culj6/k8s-agentpool-27319582-vmss) updated successfully I0109 10:21:30.019278 1 controllerserver.go:413] Attach operation successful: volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-6c3fd710-67dd-4238-b510-735a9f7fe1a4 attached to node k8s-agentpool-27319582-vmss000001. I0109 10:21:30.019292 1 controllerserver.go:433] attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-6c3fd710-67dd-4238-b510-735a9f7fe1a4 to node k8s-agentpool-27319582-vmss000001 successfully I0109 10:21:30.019337 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=10.310760336 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-ht4culj6" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-6c3fd710-67dd-4238-b510-735a9f7fe1a4" node="k8s-agentpool-27319582-vmss000001" result_code="succeeded" I0109 10:21:30.019356 1 utils.go:84] GRPC response: {"publish_context":{"LUN":"0"}} ... skipping 19 lines ... I0109 10:22:49.524358 1 utils.go:78] GRPC request: {"node_id":"k8s-agentpool-27319582-vmss000001","volume_capability":{"AccessType":{"Mount":{"mount_flags":["barrier=1","acl"]}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-4439dad2-4d5d-49b4-b310-7d0ba8493bf4","csi.storage.k8s.io/pvc/name":"pvc-x6pq7","csi.storage.k8s.io/pvc/namespace":"azuredisk-783","requestedsizegib":"10","skuName":"StandardSSD_ZRS","storage.kubernetes.io/csiProvisionerIdentity":"1673256990696-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-4439dad2-4d5d-49b4-b310-7d0ba8493bf4"} I0109 10:22:49.557915 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1192 I0109 10:22:49.558497 1 controllerserver.go:383] GetDiskLun returned: cannot find Lun for disk pvc-4439dad2-4d5d-49b4-b310-7d0ba8493bf4. Initiating attaching volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-4439dad2-4d5d-49b4-b310-7d0ba8493bf4 to node k8s-agentpool-27319582-vmss000001 (vmState Succeeded). I0109 10:22:49.558530 1 controllerserver.go:408] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-4439dad2-4d5d-49b4-b310-7d0ba8493bf4 to node k8s-agentpool-27319582-vmss000001 I0109 10:22:49.558592 1 azure_controller_common.go:255] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-4439dad2-4d5d-49b4-b310-7d0ba8493bf4 lun 1 to node k8s-agentpool-27319582-vmss000001, diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-ht4culj6/providers/microsoft.compute/disks/pvc-4439dad2-4d5d-49b4-b310-7d0ba8493bf4:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-4439dad2-4d5d-49b4-b310-7d0ba8493bf4 false 1})] I0109 10:22:49.558668 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-ht4culj6): vm(k8s-agentpool-27319582-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-ht4culj6/providers/microsoft.compute/disks/pvc-4439dad2-4d5d-49b4-b310-7d0ba8493bf4:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-4439dad2-4d5d-49b4-b310-7d0ba8493bf4 false 1})]) I0109 10:22:49.697532 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-ht4culj6): vm(k8s-agentpool-27319582-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-ht4culj6/providers/microsoft.compute/disks/pvc-4439dad2-4d5d-49b4-b310-7d0ba8493bf4:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-4439dad2-4d5d-49b4-b310-7d0ba8493bf4 false 1})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I0109 10:23:04.835293 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-ht4culj6, k8s-agentpool-27319582-vmss, k8s-agentpool-27319582-vmss000001) successfully I0109 10:23:04.835333 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool-27319582-vmss, kubetest-ht4culj6, k8s-agentpool-27319582-vmss000001) for cacheKey(kubetest-ht4culj6/k8s-agentpool-27319582-vmss) updated successfully I0109 10:23:04.835354 1 controllerserver.go:413] Attach operation successful: volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-4439dad2-4d5d-49b4-b310-7d0ba8493bf4 attached to node k8s-agentpool-27319582-vmss000001. I0109 10:23:04.835369 1 controllerserver.go:433] attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-4439dad2-4d5d-49b4-b310-7d0ba8493bf4 to node k8s-agentpool-27319582-vmss000001 successfully I0109 10:23:04.835410 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=15.276916469 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-ht4culj6" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-4439dad2-4d5d-49b4-b310-7d0ba8493bf4" node="k8s-agentpool-27319582-vmss000001" result_code="succeeded" I0109 10:23:04.835432 1 utils.go:84] GRPC response: {"publish_context":{"LUN":"1"}} ... skipping 24 lines ... I0109 10:23:31.901187 1 azure_controller_common.go:398] Trying to detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-6c3fd710-67dd-4238-b510-735a9f7fe1a4 from node k8s-agentpool-27319582-vmss000001, diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-ht4culj6/providers/microsoft.compute/disks/pvc-6c3fd710-67dd-4238-b510-735a9f7fe1a4:pvc-6c3fd710-67dd-4238-b510-735a9f7fe1a4] E0109 10:23:31.901226 1 azure_controller_vmss.go:202] detach azure disk on node(k8s-agentpool-27319582-vmss000001): disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-ht4culj6/providers/microsoft.compute/disks/pvc-6c3fd710-67dd-4238-b510-735a9f7fe1a4:pvc-6c3fd710-67dd-4238-b510-735a9f7fe1a4]) not found I0109 10:23:31.901243 1 azure_controller_vmss.go:239] azureDisk - update(kubetest-ht4culj6): vm(k8s-agentpool-27319582-vmss000001) - detach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-ht4culj6/providers/microsoft.compute/disks/pvc-6c3fd710-67dd-4238-b510-735a9f7fe1a4:pvc-6c3fd710-67dd-4238-b510-735a9f7fe1a4]) I0109 10:23:34.362957 1 utils.go:77] GRPC call: /csi.v1.Controller/DeleteVolume I0109 10:23:34.362983 1 utils.go:78] GRPC request: {"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-6c3fd710-67dd-4238-b510-735a9f7fe1a4"} I0109 10:23:34.363092 1 controllerserver.go:317] deleting azure disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-6c3fd710-67dd-4238-b510-735a9f7fe1a4) I0109 10:23:34.363110 1 controllerserver.go:319] delete azure disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-6c3fd710-67dd-4238-b510-735a9f7fe1a4) returned with failed to delete disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-6c3fd710-67dd-4238-b510-735a9f7fe1a4) since it's in attaching or detaching state I0109 10:23:34.363176 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=4.16e-05 request="azuredisk_csi_driver_controller_delete_volume" resource_group="kubetest-ht4culj6" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-6c3fd710-67dd-4238-b510-735a9f7fe1a4" result_code="failed_csi_driver_controller_delete_volume" E0109 10:23:34.363198 1 utils.go:82] GRPC error: failed to delete disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-6c3fd710-67dd-4238-b510-735a9f7fe1a4) since it's in attaching or detaching state I0109 10:23:37.112053 1 azure_controller_vmss.go:252] azureDisk - update(kubetest-ht4culj6): vm(k8s-agentpool-27319582-vmss000001) - detach disk(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-ht4culj6/providers/microsoft.compute/disks/pvc-6c3fd710-67dd-4238-b510-735a9f7fe1a4:pvc-6c3fd710-67dd-4238-b510-735a9f7fe1a4]) returned with <nil> I0109 10:23:37.112115 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-ht4culj6, k8s-agentpool-27319582-vmss, k8s-agentpool-27319582-vmss000001) successfully I0109 10:23:37.112135 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool-27319582-vmss, kubetest-ht4culj6, k8s-agentpool-27319582-vmss000001) for cacheKey(kubetest-ht4culj6/k8s-agentpool-27319582-vmss) updated successfully I0109 10:23:37.112146 1 azure_controller_common.go:422] azureDisk - detach disk(pvc-6c3fd710-67dd-4238-b510-735a9f7fe1a4, /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-6c3fd710-67dd-4238-b510-735a9f7fe1a4) succeeded I0109 10:23:37.112291 1 controllerserver.go:480] detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-6c3fd710-67dd-4238-b510-735a9f7fe1a4 from node k8s-agentpool-27319582-vmss000001 successfully I0109 10:23:37.112346 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=5.211345029 request="azuredisk_csi_driver_controller_unpublish_volume" resource_group="kubetest-ht4culj6" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-6c3fd710-67dd-4238-b510-735a9f7fe1a4" node="k8s-agentpool-27319582-vmss000001" result_code="succeeded" ... skipping 43 lines ... I0109 10:25:08.505199 1 azure_vmss_cache.go:327] refresh the cache of NonVmssUniformNodesCache in rg map[kubetest-ht4culj6:{}] I0109 10:25:08.530726 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 12 I0109 10:25:08.530981 1 controllerserver.go:383] GetDiskLun returned: cannot find Lun for disk pvc-e3755284-aef2-411d-981a-46ec15b0750a. Initiating attaching volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-e3755284-aef2-411d-981a-46ec15b0750a to node k8s-agentpool-27319582-vmss000001 (vmState Succeeded). I0109 10:25:08.531014 1 controllerserver.go:408] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-e3755284-aef2-411d-981a-46ec15b0750a to node k8s-agentpool-27319582-vmss000001 I0109 10:25:08.531146 1 azure_controller_common.go:255] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-e3755284-aef2-411d-981a-46ec15b0750a lun 0 to node k8s-agentpool-27319582-vmss000001, diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-ht4culj6/providers/microsoft.compute/disks/pvc-e3755284-aef2-411d-981a-46ec15b0750a:%!s(*provider.AttachDiskOptions=&{None pvc-e3755284-aef2-411d-981a-46ec15b0750a false 0})] I0109 10:25:08.531240 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-ht4culj6): vm(k8s-agentpool-27319582-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-ht4culj6/providers/microsoft.compute/disks/pvc-e3755284-aef2-411d-981a-46ec15b0750a:%!s(*provider.AttachDiskOptions=&{None pvc-e3755284-aef2-411d-981a-46ec15b0750a false 0})]) I0109 10:25:08.667406 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-ht4culj6): vm(k8s-agentpool-27319582-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-ht4culj6/providers/microsoft.compute/disks/pvc-e3755284-aef2-411d-981a-46ec15b0750a:%!s(*provider.AttachDiskOptions=&{None pvc-e3755284-aef2-411d-981a-46ec15b0750a false 0})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I0109 10:25:12.563080 1 utils.go:77] GRPC call: /csi.v1.Controller/ControllerPublishVolume I0109 10:25:12.563335 1 utils.go:78] GRPC request: {"node_id":"k8s-agentpool-27319582-vmss000000","volume_capability":{"AccessType":{"Block":{}},"access_mode":{"mode":5}},"volume_context":{"cachingmode":"None","csi.storage.k8s.io/pv/name":"pvc-e3755284-aef2-411d-981a-46ec15b0750a","csi.storage.k8s.io/pvc/name":"pvc-kw5nr","csi.storage.k8s.io/pvc/namespace":"azuredisk-7920","maxshares":"2","requestedsizegib":"10","skuname":"StandardSSD_ZRS","storage.kubernetes.io/csiProvisionerIdentity":"1673256990696-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-e3755284-aef2-411d-981a-46ec15b0750a"} I0109 10:25:12.591397 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1692 I0109 10:25:12.592129 1 controllerserver.go:383] GetDiskLun returned: cannot find Lun for disk pvc-e3755284-aef2-411d-981a-46ec15b0750a. Initiating attaching volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-e3755284-aef2-411d-981a-46ec15b0750a to node k8s-agentpool-27319582-vmss000000 (vmState Succeeded). I0109 10:25:12.592189 1 controllerserver.go:408] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-e3755284-aef2-411d-981a-46ec15b0750a to node k8s-agentpool-27319582-vmss000000 I0109 10:25:12.592426 1 azure_controller_common.go:255] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-e3755284-aef2-411d-981a-46ec15b0750a lun 0 to node k8s-agentpool-27319582-vmss000000, diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-ht4culj6/providers/microsoft.compute/disks/pvc-e3755284-aef2-411d-981a-46ec15b0750a:%!s(*provider.AttachDiskOptions=&{None pvc-e3755284-aef2-411d-981a-46ec15b0750a false 0})] I0109 10:25:12.592499 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-ht4culj6): vm(k8s-agentpool-27319582-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-ht4culj6/providers/microsoft.compute/disks/pvc-e3755284-aef2-411d-981a-46ec15b0750a:%!s(*provider.AttachDiskOptions=&{None pvc-e3755284-aef2-411d-981a-46ec15b0750a false 0})]) I0109 10:25:12.760888 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-ht4culj6): vm(k8s-agentpool-27319582-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-ht4culj6/providers/microsoft.compute/disks/pvc-e3755284-aef2-411d-981a-46ec15b0750a:%!s(*provider.AttachDiskOptions=&{None pvc-e3755284-aef2-411d-981a-46ec15b0750a false 0})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I0109 10:25:22.886205 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-ht4culj6, k8s-agentpool-27319582-vmss, k8s-agentpool-27319582-vmss000000) successfully I0109 10:25:22.886242 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool-27319582-vmss, kubetest-ht4culj6, k8s-agentpool-27319582-vmss000000) for cacheKey(kubetest-ht4culj6/k8s-agentpool-27319582-vmss) updated successfully I0109 10:25:22.886263 1 controllerserver.go:413] Attach operation successful: volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-e3755284-aef2-411d-981a-46ec15b0750a attached to node k8s-agentpool-27319582-vmss000000. I0109 10:25:22.886276 1 controllerserver.go:433] attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-e3755284-aef2-411d-981a-46ec15b0750a to node k8s-agentpool-27319582-vmss000000 successfully I0109 10:25:22.886326 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=10.29420988 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-ht4culj6" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-e3755284-aef2-411d-981a-46ec15b0750a" node="k8s-agentpool-27319582-vmss000000" result_code="succeeded" I0109 10:25:22.886347 1 utils.go:84] GRPC response: {"publish_context":{"LUN":"0"}} ... skipping 71 lines ... I0109 10:27:01.631941 1 utils.go:78] GRPC request: {"node_id":"k8s-agentpool-27319582-vmss000001","volume_capability":{"AccessType":{"Mount":{"mount_flags":["barrier=1","acl"]}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-716c3479-7cb5-4c96-9754-314148bceb52","csi.storage.k8s.io/pvc/name":"pvc-9zcz5","csi.storage.k8s.io/pvc/namespace":"azuredisk-1092","device-setting/device/queue_depth":"17","device-setting/queue/max_sectors_kb":"211","device-setting/queue/nr_requests":"44","device-setting/queue/read_ahead_kb":"256","device-setting/queue/rotational":"0","device-setting/queue/scheduler":"none","device-setting/queue/wbt_lat_usec":"0","perfProfile":"advanced","requestedsizegib":"10","skuname":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1673256990696-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-716c3479-7cb5-4c96-9754-314148bceb52"} I0109 10:27:01.650602 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1222 I0109 10:27:01.651091 1 controllerserver.go:383] GetDiskLun returned: cannot find Lun for disk pvc-716c3479-7cb5-4c96-9754-314148bceb52. Initiating attaching volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-716c3479-7cb5-4c96-9754-314148bceb52 to node k8s-agentpool-27319582-vmss000001 (vmState Succeeded). I0109 10:27:01.651464 1 controllerserver.go:408] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-716c3479-7cb5-4c96-9754-314148bceb52 to node k8s-agentpool-27319582-vmss000001 I0109 10:27:01.651529 1 azure_controller_common.go:255] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-716c3479-7cb5-4c96-9754-314148bceb52 lun 0 to node k8s-agentpool-27319582-vmss000001, diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-ht4culj6/providers/microsoft.compute/disks/pvc-716c3479-7cb5-4c96-9754-314148bceb52:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-716c3479-7cb5-4c96-9754-314148bceb52 false 0})] I0109 10:27:01.651573 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-ht4culj6): vm(k8s-agentpool-27319582-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-ht4culj6/providers/microsoft.compute/disks/pvc-716c3479-7cb5-4c96-9754-314148bceb52:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-716c3479-7cb5-4c96-9754-314148bceb52 false 0})]) I0109 10:27:01.783308 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-ht4culj6): vm(k8s-agentpool-27319582-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-ht4culj6/providers/microsoft.compute/disks/pvc-716c3479-7cb5-4c96-9754-314148bceb52:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-716c3479-7cb5-4c96-9754-314148bceb52 false 0})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I0109 10:27:11.857386 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-ht4culj6, k8s-agentpool-27319582-vmss, k8s-agentpool-27319582-vmss000001) successfully I0109 10:27:11.857424 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool-27319582-vmss, kubetest-ht4culj6, k8s-agentpool-27319582-vmss000001) for cacheKey(kubetest-ht4culj6/k8s-agentpool-27319582-vmss) updated successfully I0109 10:27:11.857445 1 controllerserver.go:413] Attach operation successful: volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-716c3479-7cb5-4c96-9754-314148bceb52 attached to node k8s-agentpool-27319582-vmss000001. I0109 10:27:11.857460 1 controllerserver.go:433] attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-716c3479-7cb5-4c96-9754-314148bceb52 to node k8s-agentpool-27319582-vmss000001 successfully I0109 10:27:11.857504 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=10.206414267 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-ht4culj6" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-716c3479-7cb5-4c96-9754-314148bceb52" node="k8s-agentpool-27319582-vmss000001" result_code="succeeded" I0109 10:27:11.857524 1 utils.go:84] GRPC response: {"publish_context":{"LUN":"0"}} ... skipping 31 lines ... I0109 10:28:09.121495 1 utils.go:78] GRPC request: {"node_id":"k8s-agentpool-27319582-vmss000001","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-fd6f69df-c4bc-4232-8e38-36036789acda","csi.storage.k8s.io/pvc/name":"pvc-azuredisk","csi.storage.k8s.io/pvc/namespace":"default","requestedsizegib":"10","skuName":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1673256990696-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-fd6f69df-c4bc-4232-8e38-36036789acda"} I0109 10:28:09.147320 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1219 I0109 10:28:09.147712 1 controllerserver.go:383] GetDiskLun returned: cannot find Lun for disk pvc-fd6f69df-c4bc-4232-8e38-36036789acda. Initiating attaching volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-fd6f69df-c4bc-4232-8e38-36036789acda to node k8s-agentpool-27319582-vmss000001 (vmState Succeeded). I0109 10:28:09.147760 1 controllerserver.go:408] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-fd6f69df-c4bc-4232-8e38-36036789acda to node k8s-agentpool-27319582-vmss000001 I0109 10:28:09.147821 1 azure_controller_common.go:255] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-fd6f69df-c4bc-4232-8e38-36036789acda lun 0 to node k8s-agentpool-27319582-vmss000001, diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-ht4culj6/providers/microsoft.compute/disks/pvc-fd6f69df-c4bc-4232-8e38-36036789acda:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-fd6f69df-c4bc-4232-8e38-36036789acda false 0})] I0109 10:28:09.147882 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-ht4culj6): vm(k8s-agentpool-27319582-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-ht4culj6/providers/microsoft.compute/disks/pvc-fd6f69df-c4bc-4232-8e38-36036789acda:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-fd6f69df-c4bc-4232-8e38-36036789acda false 0})]) I0109 10:28:09.277853 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-ht4culj6): vm(k8s-agentpool-27319582-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-ht4culj6/providers/microsoft.compute/disks/pvc-fd6f69df-c4bc-4232-8e38-36036789acda:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-fd6f69df-c4bc-4232-8e38-36036789acda false 0})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I0109 10:28:19.351113 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-ht4culj6, k8s-agentpool-27319582-vmss, k8s-agentpool-27319582-vmss000001) successfully I0109 10:28:19.351150 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool-27319582-vmss, kubetest-ht4culj6, k8s-agentpool-27319582-vmss000001) for cacheKey(kubetest-ht4culj6/k8s-agentpool-27319582-vmss) updated successfully I0109 10:28:19.351165 1 controllerserver.go:413] Attach operation successful: volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-fd6f69df-c4bc-4232-8e38-36036789acda attached to node k8s-agentpool-27319582-vmss000001. I0109 10:28:19.351173 1 controllerserver.go:433] attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-fd6f69df-c4bc-4232-8e38-36036789acda to node k8s-agentpool-27319582-vmss000001 successfully I0109 10:28:19.351201 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=10.203492606 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-ht4culj6" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-fd6f69df-c4bc-4232-8e38-36036789acda" node="k8s-agentpool-27319582-vmss000001" result_code="succeeded" I0109 10:28:19.351214 1 utils.go:84] GRPC response: {"publish_context":{"LUN":"0"}} ... skipping 10 lines ... I0109 10:28:33.312605 1 utils.go:78] GRPC request: {"node_id":"k8s-agentpool-27319582-vmss000000","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-20814651-2834-40ff-ba59-ce0a059a1ba9","csi.storage.k8s.io/pvc/name":"persistent-storage-statefulset-azuredisk-0","csi.storage.k8s.io/pvc/namespace":"default","requestedsizegib":"10","skuName":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1673256990696-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-20814651-2834-40ff-ba59-ce0a059a1ba9"} I0109 10:28:33.332552 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1248 I0109 10:28:33.332896 1 controllerserver.go:383] GetDiskLun returned: cannot find Lun for disk pvc-20814651-2834-40ff-ba59-ce0a059a1ba9. Initiating attaching volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-20814651-2834-40ff-ba59-ce0a059a1ba9 to node k8s-agentpool-27319582-vmss000000 (vmState Succeeded). I0109 10:28:33.332925 1 controllerserver.go:408] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-20814651-2834-40ff-ba59-ce0a059a1ba9 to node k8s-agentpool-27319582-vmss000000 I0109 10:28:33.332957 1 azure_controller_common.go:255] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-20814651-2834-40ff-ba59-ce0a059a1ba9 lun 0 to node k8s-agentpool-27319582-vmss000000, diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-ht4culj6/providers/microsoft.compute/disks/pvc-20814651-2834-40ff-ba59-ce0a059a1ba9:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-20814651-2834-40ff-ba59-ce0a059a1ba9 false 0})] I0109 10:28:33.333032 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-ht4culj6): vm(k8s-agentpool-27319582-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-ht4culj6/providers/microsoft.compute/disks/pvc-20814651-2834-40ff-ba59-ce0a059a1ba9:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-20814651-2834-40ff-ba59-ce0a059a1ba9 false 0})]) I0109 10:28:33.453013 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-ht4culj6): vm(k8s-agentpool-27319582-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-ht4culj6/providers/microsoft.compute/disks/pvc-20814651-2834-40ff-ba59-ce0a059a1ba9:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-20814651-2834-40ff-ba59-ce0a059a1ba9 false 0})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I0109 10:28:43.524014 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-ht4culj6, k8s-agentpool-27319582-vmss, k8s-agentpool-27319582-vmss000000) successfully I0109 10:28:43.524067 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool-27319582-vmss, kubetest-ht4culj6, k8s-agentpool-27319582-vmss000000) for cacheKey(kubetest-ht4culj6/k8s-agentpool-27319582-vmss) updated successfully I0109 10:28:43.524087 1 controllerserver.go:413] Attach operation successful: volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-20814651-2834-40ff-ba59-ce0a059a1ba9 attached to node k8s-agentpool-27319582-vmss000000. I0109 10:28:43.524100 1 controllerserver.go:433] attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-20814651-2834-40ff-ba59-ce0a059a1ba9 to node k8s-agentpool-27319582-vmss000000 successfully I0109 10:28:43.524157 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=10.191235783 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-ht4culj6" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-20814651-2834-40ff-ba59-ce0a059a1ba9" node="k8s-agentpool-27319582-vmss000000" result_code="succeeded" I0109 10:28:43.524176 1 utils.go:84] GRPC response: {"publish_context":{"LUN":"0"}} ... skipping 18 lines ... I0109 10:28:59.719159 1 utils.go:78] GRPC request: {"node_id":"k8s-agentpool-27319582-vmss000001","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-0d0d820e-4755-43ad-8b2e-094c70655e04","csi.storage.k8s.io/pvc/name":"persistent-storage-statefulset-azuredisk-nonroot-0","csi.storage.k8s.io/pvc/namespace":"default","requestedsizegib":"10","skuName":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1673256990696-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-0d0d820e-4755-43ad-8b2e-094c70655e04"} I0109 10:28:59.770840 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1256 I0109 10:28:59.771239 1 controllerserver.go:383] GetDiskLun returned: cannot find Lun for disk pvc-0d0d820e-4755-43ad-8b2e-094c70655e04. Initiating attaching volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-0d0d820e-4755-43ad-8b2e-094c70655e04 to node k8s-agentpool-27319582-vmss000001 (vmState Succeeded). I0109 10:28:59.771275 1 controllerserver.go:408] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-0d0d820e-4755-43ad-8b2e-094c70655e04 to node k8s-agentpool-27319582-vmss000001 I0109 10:28:59.771346 1 azure_controller_common.go:255] Trying to attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-0d0d820e-4755-43ad-8b2e-094c70655e04 lun 1 to node k8s-agentpool-27319582-vmss000001, diskMap: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-ht4culj6/providers/microsoft.compute/disks/pvc-0d0d820e-4755-43ad-8b2e-094c70655e04:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-0d0d820e-4755-43ad-8b2e-094c70655e04 false 1})] I0109 10:28:59.771436 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-ht4culj6): vm(k8s-agentpool-27319582-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-ht4culj6/providers/microsoft.compute/disks/pvc-0d0d820e-4755-43ad-8b2e-094c70655e04:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-0d0d820e-4755-43ad-8b2e-094c70655e04 false 1})]) I0109 10:28:59.945664 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-ht4culj6): vm(k8s-agentpool-27319582-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourcegroups/kubetest-ht4culj6/providers/microsoft.compute/disks/pvc-0d0d820e-4755-43ad-8b2e-094c70655e04:%!s(*provider.AttachDiskOptions=&{ReadOnly pvc-0d0d820e-4755-43ad-8b2e-094c70655e04 false 1})], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I0109 10:29:10.035359 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-ht4culj6, k8s-agentpool-27319582-vmss, k8s-agentpool-27319582-vmss000001) successfully I0109 10:29:10.035415 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool-27319582-vmss, kubetest-ht4culj6, k8s-agentpool-27319582-vmss000001) for cacheKey(kubetest-ht4culj6/k8s-agentpool-27319582-vmss) updated successfully I0109 10:29:10.035438 1 controllerserver.go:413] Attach operation successful: volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-0d0d820e-4755-43ad-8b2e-094c70655e04 attached to node k8s-agentpool-27319582-vmss000001. I0109 10:29:10.035456 1 controllerserver.go:433] attach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-0d0d820e-4755-43ad-8b2e-094c70655e04 to node k8s-agentpool-27319582-vmss000001 successfully I0109 10:29:10.035556 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=10.2643319 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-ht4culj6" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-0d0d820e-4755-43ad-8b2e-094c70655e04" node="k8s-agentpool-27319582-vmss000001" result_code="succeeded" I0109 10:29:10.035778 1 utils.go:84] GRPC response: {"publish_context":{"LUN":"1"}} ... skipping 12 lines ... Platform: linux/amd64 Topology Key: topology.disk.csi.azure.com/zone Streaming logs below: I0109 09:36:30.983228 1 azuredisk.go:175] driver userAgent: disk.csi.azure.com/v1.27.0-24f0924abd1b960083c095e7d894758dec8252c3 e2e-test I0109 09:36:30.985725 1 azure_disk_utils.go:162] reading cloud config from secret kube-system/azure-cloud-provider I0109 09:36:31.017590 1 azure_disk_utils.go:169] InitializeCloudFromSecret: failed to get cloud config from secret kube-system/azure-cloud-provider: failed to get secret kube-system/azure-cloud-provider: secrets "azure-cloud-provider" not found I0109 09:36:31.017613 1 azure_disk_utils.go:174] could not read cloud config from secret kube-system/azure-cloud-provider I0109 09:36:31.017621 1 azure_disk_utils.go:184] use default AZURE_CREDENTIAL_FILE env var: /etc/kubernetes/azure.json I0109 09:36:31.017653 1 azure_disk_utils.go:192] read cloud config from file: /etc/kubernetes/azure.json successfully I0109 09:36:31.018460 1 azure_auth.go:245] Using AzurePublicCloud environment I0109 09:36:31.018517 1 azure_auth.go:130] azure: using client_id+client_secret to retrieve access token I0109 09:36:31.018549 1 azure.go:777] Azure cloudprovider using try backoff: retries=6, exponent=1.500000, duration=5, jitter=1.000000 ... skipping 25 lines ... I0109 09:36:31.018920 1 azure_blobclient.go:67] Azure BlobClient using API version: 2021-09-01 I0109 09:36:31.018940 1 azure_vmasclient.go:70] Azure AvailabilitySetsClient (read ops) using rate limit config: QPS=6, bucket=20 I0109 09:36:31.018948 1 azure_vmasclient.go:73] Azure AvailabilitySetsClient (write ops) using rate limit config: QPS=100, bucket=1000 I0109 09:36:31.019024 1 azure.go:1008] attach/detach disk operation rate limit QPS: 6.000000, Bucket: 10 I0109 09:36:31.019048 1 azuredisk.go:192] disable UseInstanceMetadata for controller I0109 09:36:31.019058 1 azuredisk.go:204] cloud: AzurePublicCloud, location: westus2, rg: kubetest-ht4culj6, VMType: vmss, PrimaryScaleSetName: k8s-agentpool-27319582-vmss, PrimaryAvailabilitySetName: , DisableAvailabilitySetNodes: false I0109 09:36:31.021828 1 mount_linux.go:287] 'umount /tmp/kubelet-detect-safe-umount3895154225' failed with: exit status 32, output: umount: /tmp/kubelet-detect-safe-umount3895154225: must be superuser to unmount. I0109 09:36:31.021858 1 mount_linux.go:289] Detected umount with unsafe 'not mounted' behavior I0109 09:36:31.021948 1 driver.go:81] Enabling controller service capability: CREATE_DELETE_VOLUME I0109 09:36:31.021960 1 driver.go:81] Enabling controller service capability: PUBLISH_UNPUBLISH_VOLUME I0109 09:36:31.021967 1 driver.go:81] Enabling controller service capability: CREATE_DELETE_SNAPSHOT I0109 09:36:31.021973 1 driver.go:81] Enabling controller service capability: CLONE_VOLUME I0109 09:36:31.021978 1 driver.go:81] Enabling controller service capability: EXPAND_VOLUME ... skipping 62 lines ... Platform: linux/amd64 Topology Key: topology.disk.csi.azure.com/zone Streaming logs below: I0109 09:36:21.577535 1 azuredisk.go:175] driver userAgent: disk.csi.azure.com/v1.27.0-24f0924abd1b960083c095e7d894758dec8252c3 e2e-test I0109 09:36:21.577955 1 azure_disk_utils.go:162] reading cloud config from secret kube-system/azure-cloud-provider I0109 09:36:21.606297 1 azure_disk_utils.go:169] InitializeCloudFromSecret: failed to get cloud config from secret kube-system/azure-cloud-provider: failed to get secret kube-system/azure-cloud-provider: secrets "azure-cloud-provider" not found I0109 09:36:21.606320 1 azure_disk_utils.go:174] could not read cloud config from secret kube-system/azure-cloud-provider I0109 09:36:21.606329 1 azure_disk_utils.go:184] use default AZURE_CREDENTIAL_FILE env var: /etc/kubernetes/azure.json I0109 09:36:21.606395 1 azure_disk_utils.go:192] read cloud config from file: /etc/kubernetes/azure.json successfully I0109 09:36:21.607452 1 azure_auth.go:245] Using AzurePublicCloud environment I0109 09:36:21.607525 1 azure_auth.go:130] azure: using client_id+client_secret to retrieve access token I0109 09:36:21.607559 1 azure.go:777] Azure cloudprovider using try backoff: retries=6, exponent=1.500000, duration=5, jitter=1.000000 ... skipping 69 lines ... Platform: linux/amd64 Topology Key: topology.disk.csi.azure.com/zone Streaming logs below: I0109 09:36:25.804392 1 azuredisk.go:175] driver userAgent: disk.csi.azure.com/v1.27.0-24f0924abd1b960083c095e7d894758dec8252c3 e2e-test I0109 09:36:25.804963 1 azure_disk_utils.go:162] reading cloud config from secret kube-system/azure-cloud-provider I0109 09:36:25.839982 1 azure_disk_utils.go:169] InitializeCloudFromSecret: failed to get cloud config from secret kube-system/azure-cloud-provider: failed to get secret kube-system/azure-cloud-provider: secrets "azure-cloud-provider" not found I0109 09:36:25.840015 1 azure_disk_utils.go:174] could not read cloud config from secret kube-system/azure-cloud-provider I0109 09:36:25.840025 1 azure_disk_utils.go:184] use default AZURE_CREDENTIAL_FILE env var: /etc/kubernetes/azure.json I0109 09:36:25.840053 1 azure_disk_utils.go:192] read cloud config from file: /etc/kubernetes/azure.json successfully I0109 09:36:25.841431 1 azure_auth.go:245] Using AzurePublicCloud environment I0109 09:36:25.841503 1 azure_auth.go:130] azure: using client_id+client_secret to retrieve access token I0109 09:36:25.841539 1 azure.go:777] Azure cloudprovider using try backoff: retries=6, exponent=1.500000, duration=5, jitter=1.000000 ... skipping 148 lines ... I0109 09:41:19.589461 1 mount_linux.go:567] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0]) I0109 09:41:19.604795 1 mount_linux.go:570] Output: "" I0109 09:41:19.604819 1 mount_linux.go:529] Disk "/dev/disk/azure/scsi1/lun0" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/disk/azure/scsi1/lun0] I0109 09:41:20.070428 1 mount_linux.go:539] Disk successfully formatted (mkfs): ext4 - /dev/disk/azure/scsi1/lun0 /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-f62b55d7-4648-4389-b72a-610edf2d30c0/globalmount I0109 09:41:20.070478 1 mount_linux.go:557] Attempting to mount disk /dev/disk/azure/scsi1/lun0 in ext4 format at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-f62b55d7-4648-4389-b72a-610edf2d30c0/globalmount I0109 09:41:20.070533 1 mount_linux.go:220] Mounting cmd (mount) with arguments (-t ext4 -o invalid,mount,options,defaults /dev/disk/azure/scsi1/lun0 /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-f62b55d7-4648-4389-b72a-610edf2d30c0/globalmount) E0109 09:41:20.089622 1 mount_linux.go:232] Mount failed: exit status 32 Mounting command: mount Mounting arguments: -t ext4 -o invalid,mount,options,defaults /dev/disk/azure/scsi1/lun0 /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-f62b55d7-4648-4389-b72a-610edf2d30c0/globalmount Output: mount: /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-f62b55d7-4648-4389-b72a-610edf2d30c0/globalmount: wrong fs type, bad option, bad superblock on /dev/sdc, missing codepage or helper program, or other error. E0109 09:41:20.089693 1 utils.go:82] GRPC error: rpc error: code = Internal desc = could not format /dev/disk/azure/scsi1/lun0(lun: 0), and mount it at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-f62b55d7-4648-4389-b72a-610edf2d30c0/globalmount, failed with mount failed: exit status 32 Mounting command: mount Mounting arguments: -t ext4 -o invalid,mount,options,defaults /dev/disk/azure/scsi1/lun0 /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-f62b55d7-4648-4389-b72a-610edf2d30c0/globalmount Output: mount: /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-f62b55d7-4648-4389-b72a-610edf2d30c0/globalmount: wrong fs type, bad option, bad superblock on /dev/sdc, missing codepage or helper program, or other error. I0109 09:41:20.633027 1 utils.go:77] GRPC call: /csi.v1.Node/NodeStageVolume I0109 09:41:20.633054 1 utils.go:78] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-f62b55d7-4648-4389-b72a-610edf2d30c0/globalmount","volume_capability":{"AccessType":{"Mount":{"mount_flags":["invalid","mount","options"]}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-f62b55d7-4648-4389-b72a-610edf2d30c0","csi.storage.k8s.io/pvc/name":"pvc-s6b42","csi.storage.k8s.io/pvc/namespace":"azuredisk-5466","requestedsizegib":"10","skuName":"StandardSSD_ZRS","storage.kubernetes.io/csiProvisionerIdentity":"1673256990696-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-f62b55d7-4648-4389-b72a-610edf2d30c0"} I0109 09:41:22.500017 1 azure_common_linux.go:185] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/ I0109 09:41:22.500060 1 nodeserver.go:116] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType StandardSSD_ZRS I0109 09:41:22.500413 1 nodeserver.go:157] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-f62b55d7-4648-4389-b72a-610edf2d30c0/globalmount with mount options([invalid mount options]) I0109 09:41:22.500438 1 mount_linux.go:567] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0]) I0109 09:41:22.516496 1 mount_linux.go:570] Output: "DEVNAME=/dev/disk/azure/scsi1/lun0\nTYPE=ext4\n" I0109 09:41:22.516522 1 mount_linux.go:453] Checking for issues with fsck on disk: /dev/disk/azure/scsi1/lun0 I0109 09:41:22.532616 1 mount_linux.go:557] Attempting to mount disk /dev/disk/azure/scsi1/lun0 in ext4 format at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-f62b55d7-4648-4389-b72a-610edf2d30c0/globalmount I0109 09:41:22.532646 1 mount_linux.go:220] Mounting cmd (mount) with arguments (-t ext4 -o invalid,mount,options,defaults /dev/disk/azure/scsi1/lun0 /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-f62b55d7-4648-4389-b72a-610edf2d30c0/globalmount) E0109 09:41:22.550174 1 mount_linux.go:232] Mount failed: exit status 32 Mounting command: mount Mounting arguments: -t ext4 -o invalid,mount,options,defaults /dev/disk/azure/scsi1/lun0 /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-f62b55d7-4648-4389-b72a-610edf2d30c0/globalmount Output: mount: /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-f62b55d7-4648-4389-b72a-610edf2d30c0/globalmount: wrong fs type, bad option, bad superblock on /dev/sdc, missing codepage or helper program, or other error. E0109 09:41:22.550215 1 utils.go:82] GRPC error: rpc error: code = Internal desc = could not format /dev/disk/azure/scsi1/lun0(lun: 0), and mount it at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-f62b55d7-4648-4389-b72a-610edf2d30c0/globalmount, failed with mount failed: exit status 32 Mounting command: mount Mounting arguments: -t ext4 -o invalid,mount,options,defaults /dev/disk/azure/scsi1/lun0 /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-f62b55d7-4648-4389-b72a-610edf2d30c0/globalmount Output: mount: /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-f62b55d7-4648-4389-b72a-610edf2d30c0/globalmount: wrong fs type, bad option, bad superblock on /dev/sdc, missing codepage or helper program, or other error. I0109 09:41:23.570091 1 utils.go:77] GRPC call: /csi.v1.Node/NodeStageVolume I0109 09:41:23.570114 1 utils.go:78] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-f62b55d7-4648-4389-b72a-610edf2d30c0/globalmount","volume_capability":{"AccessType":{"Mount":{"mount_flags":["invalid","mount","options"]}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-f62b55d7-4648-4389-b72a-610edf2d30c0","csi.storage.k8s.io/pvc/name":"pvc-s6b42","csi.storage.k8s.io/pvc/namespace":"azuredisk-5466","requestedsizegib":"10","skuName":"StandardSSD_ZRS","storage.kubernetes.io/csiProvisionerIdentity":"1673256990696-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-f62b55d7-4648-4389-b72a-610edf2d30c0"} I0109 09:41:25.402872 1 azure_common_linux.go:185] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/ I0109 09:41:25.402907 1 nodeserver.go:116] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType StandardSSD_ZRS I0109 09:41:25.403230 1 nodeserver.go:157] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-f62b55d7-4648-4389-b72a-610edf2d30c0/globalmount with mount options([invalid mount options]) I0109 09:41:25.403245 1 mount_linux.go:567] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0]) I0109 09:41:25.416700 1 mount_linux.go:570] Output: "DEVNAME=/dev/disk/azure/scsi1/lun0\nTYPE=ext4\n" I0109 09:41:25.416727 1 mount_linux.go:453] Checking for issues with fsck on disk: /dev/disk/azure/scsi1/lun0 I0109 09:41:25.433547 1 mount_linux.go:557] Attempting to mount disk /dev/disk/azure/scsi1/lun0 in ext4 format at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-f62b55d7-4648-4389-b72a-610edf2d30c0/globalmount I0109 09:41:25.433583 1 mount_linux.go:220] Mounting cmd (mount) with arguments (-t ext4 -o invalid,mount,options,defaults /dev/disk/azure/scsi1/lun0 /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-f62b55d7-4648-4389-b72a-610edf2d30c0/globalmount) E0109 09:41:25.452699 1 mount_linux.go:232] Mount failed: exit status 32 Mounting command: mount Mounting arguments: -t ext4 -o invalid,mount,options,defaults /dev/disk/azure/scsi1/lun0 /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-f62b55d7-4648-4389-b72a-610edf2d30c0/globalmount Output: mount: /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-f62b55d7-4648-4389-b72a-610edf2d30c0/globalmount: wrong fs type, bad option, bad superblock on /dev/sdc, missing codepage or helper program, or other error. E0109 09:41:25.452737 1 utils.go:82] GRPC error: rpc error: code = Internal desc = could not format /dev/disk/azure/scsi1/lun0(lun: 0), and mount it at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-f62b55d7-4648-4389-b72a-610edf2d30c0/globalmount, failed with mount failed: exit status 32 Mounting command: mount Mounting arguments: -t ext4 -o invalid,mount,options,defaults /dev/disk/azure/scsi1/lun0 /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-f62b55d7-4648-4389-b72a-610edf2d30c0/globalmount Output: mount: /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-f62b55d7-4648-4389-b72a-610edf2d30c0/globalmount: wrong fs type, bad option, bad superblock on /dev/sdc, missing codepage or helper program, or other error. I0109 09:41:27.544172 1 utils.go:77] GRPC call: /csi.v1.Node/NodeStageVolume I0109 09:41:27.544200 1 utils.go:78] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-f62b55d7-4648-4389-b72a-610edf2d30c0/globalmount","volume_capability":{"AccessType":{"Mount":{"mount_flags":["invalid","mount","options"]}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-f62b55d7-4648-4389-b72a-610edf2d30c0","csi.storage.k8s.io/pvc/name":"pvc-s6b42","csi.storage.k8s.io/pvc/namespace":"azuredisk-5466","requestedsizegib":"10","skuName":"StandardSSD_ZRS","storage.kubernetes.io/csiProvisionerIdentity":"1673256990696-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-f62b55d7-4648-4389-b72a-610edf2d30c0"} I0109 09:41:29.381098 1 azure_common_linux.go:185] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/ I0109 09:41:29.381146 1 nodeserver.go:116] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType StandardSSD_ZRS I0109 09:41:29.381474 1 nodeserver.go:157] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-f62b55d7-4648-4389-b72a-610edf2d30c0/globalmount with mount options([invalid mount options]) I0109 09:41:29.381495 1 mount_linux.go:567] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0]) I0109 09:41:29.392271 1 mount_linux.go:570] Output: "DEVNAME=/dev/disk/azure/scsi1/lun0\nTYPE=ext4\n" I0109 09:41:29.392292 1 mount_linux.go:453] Checking for issues with fsck on disk: /dev/disk/azure/scsi1/lun0 I0109 09:41:29.407335 1 mount_linux.go:557] Attempting to mount disk /dev/disk/azure/scsi1/lun0 in ext4 format at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-f62b55d7-4648-4389-b72a-610edf2d30c0/globalmount I0109 09:41:29.407419 1 mount_linux.go:220] Mounting cmd (mount) with arguments (-t ext4 -o invalid,mount,options,defaults /dev/disk/azure/scsi1/lun0 /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-f62b55d7-4648-4389-b72a-610edf2d30c0/globalmount) E0109 09:41:29.428638 1 mount_linux.go:232] Mount failed: exit status 32 Mounting command: mount Mounting arguments: -t ext4 -o invalid,mount,options,defaults /dev/disk/azure/scsi1/lun0 /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-f62b55d7-4648-4389-b72a-610edf2d30c0/globalmount Output: mount: /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-f62b55d7-4648-4389-b72a-610edf2d30c0/globalmount: wrong fs type, bad option, bad superblock on /dev/sdc, missing codepage or helper program, or other error. E0109 09:41:29.428739 1 utils.go:82] GRPC error: rpc error: code = Internal desc = could not format /dev/disk/azure/scsi1/lun0(lun: 0), and mount it at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-f62b55d7-4648-4389-b72a-610edf2d30c0/globalmount, failed with mount failed: exit status 32 Mounting command: mount Mounting arguments: -t ext4 -o invalid,mount,options,defaults /dev/disk/azure/scsi1/lun0 /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-f62b55d7-4648-4389-b72a-610edf2d30c0/globalmount Output: mount: /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-f62b55d7-4648-4389-b72a-610edf2d30c0/globalmount: wrong fs type, bad option, bad superblock on /dev/sdc, missing codepage or helper program, or other error. I0109 09:42:27.126830 1 utils.go:77] GRPC call: /csi.v1.Node/NodeStageVolume I0109 09:42:27.126854 1 utils.go:78] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-508b6639-f8a0-4ce2-9ab0-38324ebb14e2","volume_capability":{"AccessType":{"Block":{}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-508b6639-f8a0-4ce2-9ab0-38324ebb14e2","csi.storage.k8s.io/pvc/name":"pvc-8db69","csi.storage.k8s.io/pvc/namespace":"azuredisk-2790","requestedsizegib":"10","skuName":"StandardSSD_ZRS","storage.kubernetes.io/csiProvisionerIdentity":"1673256990696-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-508b6639-f8a0-4ce2-9ab0-38324ebb14e2"} I0109 09:42:28.960212 1 azure_common_linux.go:185] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/ I0109 09:42:28.960267 1 nodeserver.go:116] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType StandardSSD_ZRS I0109 09:42:28.960295 1 utils.go:84] GRPC response: {} I0109 09:42:28.986131 1 utils.go:77] GRPC call: /csi.v1.Node/NodePublishVolume ... skipping 16 lines ... I0109 09:42:34.956023 1 utils.go:84] GRPC response: {} I0109 09:42:34.977611 1 utils.go:77] GRPC call: /csi.v1.Node/NodeUnstageVolume I0109 09:42:34.977629 1 utils.go:78] GRPC request: {"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-508b6639-f8a0-4ce2-9ab0-38324ebb14e2","volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-508b6639-f8a0-4ce2-9ab0-38324ebb14e2"} I0109 09:42:34.977683 1 nodeserver.go:201] NodeUnstageVolume: unmounting /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-508b6639-f8a0-4ce2-9ab0-38324ebb14e2 I0109 09:42:34.977699 1 mount_helper_common.go:93] unmounting "/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-508b6639-f8a0-4ce2-9ab0-38324ebb14e2" (corruptedMount: false, mounterCanSkipMountPointChecks: true) I0109 09:42:34.977710 1 mount_linux.go:362] Unmounting /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-508b6639-f8a0-4ce2-9ab0-38324ebb14e2 I0109 09:42:34.979894 1 mount_linux.go:375] ignoring 'not mounted' error for /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-508b6639-f8a0-4ce2-9ab0-38324ebb14e2 I0109 09:42:34.979904 1 mount_helper_common.go:150] Warning: deleting path "/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-508b6639-f8a0-4ce2-9ab0-38324ebb14e2" I0109 09:42:34.979988 1 nodeserver.go:206] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-508b6639-f8a0-4ce2-9ab0-38324ebb14e2 successfully I0109 09:42:34.980008 1 utils.go:84] GRPC response: {} I0109 09:43:36.665346 1 utils.go:77] GRPC call: /csi.v1.Node/NodeStageVolume I0109 09:43:36.665372 1 utils.go:78] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-261268db-1c68-4f25-a4c1-ecd4067cd0fe/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-261268db-1c68-4f25-a4c1-ecd4067cd0fe","csi.storage.k8s.io/pvc/name":"pvc-wpj5b","csi.storage.k8s.io/pvc/namespace":"azuredisk-5356","requestedsizegib":"10","resourceGroup":"azuredisk-csi-driver-test-0ab45119-9002-11ed-9173-829d2d0c1a9d","skuName":"StandardSSD_ZRS","storage.kubernetes.io/csiProvisionerIdentity":"1673256990696-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/azuredisk-csi-driver-test-0ab45119-9002-11ed-9173-829d2d0c1a9d/providers/Microsoft.Compute/disks/pvc-261268db-1c68-4f25-a4c1-ecd4067cd0fe"} I0109 09:43:38.503704 1 azure_common_linux.go:185] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/ ... skipping 648 lines ... I0109 10:04:32.179907 1 utils.go:84] GRPC response: {} I0109 10:04:32.220906 1 utils.go:77] GRPC call: /csi.v1.Node/NodeUnstageVolume I0109 10:04:32.220941 1 utils.go:78] GRPC request: {"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-a4701b24-96dd-48a8-9fcd-eb3d369c66ec","volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-a4701b24-96dd-48a8-9fcd-eb3d369c66ec"} I0109 10:04:32.221006 1 nodeserver.go:201] NodeUnstageVolume: unmounting /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-a4701b24-96dd-48a8-9fcd-eb3d369c66ec I0109 10:04:32.221037 1 mount_helper_common.go:93] unmounting "/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-a4701b24-96dd-48a8-9fcd-eb3d369c66ec" (corruptedMount: false, mounterCanSkipMountPointChecks: true) I0109 10:04:32.221061 1 mount_linux.go:362] Unmounting /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-a4701b24-96dd-48a8-9fcd-eb3d369c66ec I0109 10:04:32.224568 1 mount_linux.go:375] ignoring 'not mounted' error for /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-a4701b24-96dd-48a8-9fcd-eb3d369c66ec I0109 10:04:32.224606 1 mount_helper_common.go:150] Warning: deleting path "/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-a4701b24-96dd-48a8-9fcd-eb3d369c66ec" I0109 10:04:32.224710 1 nodeserver.go:206] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-a4701b24-96dd-48a8-9fcd-eb3d369c66ec successfully I0109 10:04:32.224732 1 utils.go:84] GRPC response: {} I0109 10:06:12.644711 1 utils.go:77] GRPC call: /csi.v1.Node/NodeStageVolume I0109 10:06:12.644737 1 utils.go:78] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-3d70faa1-7b52-4b74-8e59-a7686f96f787/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-3d70faa1-7b52-4b74-8e59-a7686f96f787","csi.storage.k8s.io/pvc/name":"pvc-wbrs2","csi.storage.k8s.io/pvc/namespace":"azuredisk-8582","requestedsizegib":"10","skuName":"StandardSSD_ZRS","storage.kubernetes.io/csiProvisionerIdentity":"1673256990696-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-3d70faa1-7b52-4b74-8e59-a7686f96f787"} I0109 10:06:14.470422 1 azure_common_linux.go:185] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/ ... skipping 594 lines ... I0109 10:26:07.258893 1 utils.go:84] GRPC response: {} I0109 10:26:07.295804 1 utils.go:77] GRPC call: /csi.v1.Node/NodeUnstageVolume I0109 10:26:07.295828 1 utils.go:78] GRPC request: {"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-e3755284-aef2-411d-981a-46ec15b0750a","volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-e3755284-aef2-411d-981a-46ec15b0750a"} I0109 10:26:07.295898 1 nodeserver.go:201] NodeUnstageVolume: unmounting /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-e3755284-aef2-411d-981a-46ec15b0750a I0109 10:26:07.295936 1 mount_helper_common.go:93] unmounting "/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-e3755284-aef2-411d-981a-46ec15b0750a" (corruptedMount: false, mounterCanSkipMountPointChecks: true) I0109 10:26:07.295966 1 mount_linux.go:362] Unmounting /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-e3755284-aef2-411d-981a-46ec15b0750a I0109 10:26:07.297722 1 mount_linux.go:375] ignoring 'not mounted' error for /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-e3755284-aef2-411d-981a-46ec15b0750a I0109 10:26:07.297737 1 mount_helper_common.go:150] Warning: deleting path "/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-e3755284-aef2-411d-981a-46ec15b0750a" I0109 10:26:07.297812 1 nodeserver.go:206] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-e3755284-aef2-411d-981a-46ec15b0750a successfully I0109 10:26:07.297825 1 utils.go:84] GRPC response: {} I0109 10:27:17.478244 1 utils.go:77] GRPC call: /csi.v1.Node/NodeStageVolume I0109 10:27:17.478278 1 utils.go:78] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-716c3479-7cb5-4c96-9754-314148bceb52/globalmount","volume_capability":{"AccessType":{"Mount":{"mount_flags":["barrier=1","acl"]}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-716c3479-7cb5-4c96-9754-314148bceb52","csi.storage.k8s.io/pvc/name":"pvc-9zcz5","csi.storage.k8s.io/pvc/namespace":"azuredisk-1092","device-setting/device/queue_depth":"17","device-setting/queue/max_sectors_kb":"211","device-setting/queue/nr_requests":"44","device-setting/queue/read_ahead_kb":"256","device-setting/queue/rotational":"0","device-setting/queue/scheduler":"none","device-setting/queue/wbt_lat_usec":"0","perfProfile":"advanced","requestedsizegib":"10","skuname":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1673256990696-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-716c3479-7cb5-4c96-9754-314148bceb52"} I0109 10:27:19.372565 1 azure_common_linux.go:185] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/ ... skipping 100 lines ... Platform: linux/amd64 Topology Key: topology.disk.csi.azure.com/zone Streaming logs below: I0109 09:36:28.265700 1 azuredisk.go:175] driver userAgent: disk.csi.azure.com/v1.27.0-24f0924abd1b960083c095e7d894758dec8252c3 e2e-test I0109 09:36:28.266222 1 azure_disk_utils.go:162] reading cloud config from secret kube-system/azure-cloud-provider I0109 09:36:28.315995 1 azure_disk_utils.go:169] InitializeCloudFromSecret: failed to get cloud config from secret kube-system/azure-cloud-provider: failed to get secret kube-system/azure-cloud-provider: secrets "azure-cloud-provider" not found I0109 09:36:28.316022 1 azure_disk_utils.go:174] could not read cloud config from secret kube-system/azure-cloud-provider I0109 09:36:28.316030 1 azure_disk_utils.go:184] use default AZURE_CREDENTIAL_FILE env var: /etc/kubernetes/azure.json I0109 09:36:28.316058 1 azure_disk_utils.go:192] read cloud config from file: /etc/kubernetes/azure.json successfully I0109 09:36:28.316781 1 azure_auth.go:245] Using AzurePublicCloud environment I0109 09:36:28.316830 1 azure_auth.go:130] azure: using client_id+client_secret to retrieve access token I0109 09:36:28.316865 1 azure.go:777] Azure cloudprovider using try backoff: retries=6, exponent=1.500000, duration=5, jitter=1.000000 ... skipping 203 lines ... I0109 10:26:06.744013 1 utils.go:84] GRPC response: {} I0109 10:26:06.790217 1 utils.go:77] GRPC call: /csi.v1.Node/NodeUnstageVolume I0109 10:26:06.790239 1 utils.go:78] GRPC request: {"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-e3755284-aef2-411d-981a-46ec15b0750a","volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-e3755284-aef2-411d-981a-46ec15b0750a"} I0109 10:26:06.790310 1 nodeserver.go:201] NodeUnstageVolume: unmounting /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-e3755284-aef2-411d-981a-46ec15b0750a I0109 10:26:06.790341 1 mount_helper_common.go:93] unmounting "/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-e3755284-aef2-411d-981a-46ec15b0750a" (corruptedMount: false, mounterCanSkipMountPointChecks: true) I0109 10:26:06.790357 1 mount_linux.go:362] Unmounting /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-e3755284-aef2-411d-981a-46ec15b0750a I0109 10:26:06.792368 1 mount_linux.go:375] ignoring 'not mounted' error for /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-e3755284-aef2-411d-981a-46ec15b0750a I0109 10:26:06.792389 1 mount_helper_common.go:150] Warning: deleting path "/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-e3755284-aef2-411d-981a-46ec15b0750a" I0109 10:26:06.792468 1 nodeserver.go:206] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-e3755284-aef2-411d-981a-46ec15b0750a successfully I0109 10:26:06.792486 1 utils.go:84] GRPC response: {} I0109 10:28:49.194970 1 utils.go:77] GRPC call: /csi.v1.Node/NodeStageVolume I0109 10:28:49.194991 1 utils.go:78] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-20814651-2834-40ff-ba59-ce0a059a1ba9/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-20814651-2834-40ff-ba59-ce0a059a1ba9","csi.storage.k8s.io/pvc/name":"persistent-storage-statefulset-azuredisk-0","csi.storage.k8s.io/pvc/namespace":"default","requestedsizegib":"10","skuName":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1673256990696-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ht4culj6/providers/Microsoft.Compute/disks/pvc-20814651-2834-40ff-ba59-ce0a059a1ba9"} I0109 10:28:50.855441 1 azure_common_linux.go:185] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/ ... skipping 594 lines ... cloudprovider_azure_op_duration_seconds_bucket{request="azuredisk_csi_driver_controller_unpublish_volume",resource_group="kubetest-ht4culj6",source="disk.csi.azure.com",subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e",le="300"} 50 cloudprovider_azure_op_duration_seconds_bucket{request="azuredisk_csi_driver_controller_unpublish_volume",resource_group="kubetest-ht4culj6",source="disk.csi.azure.com",subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e",le="600"} 50 cloudprovider_azure_op_duration_seconds_bucket{request="azuredisk_csi_driver_controller_unpublish_volume",resource_group="kubetest-ht4culj6",source="disk.csi.azure.com",subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e",le="1200"} 50 cloudprovider_azure_op_duration_seconds_bucket{request="azuredisk_csi_driver_controller_unpublish_volume",resource_group="kubetest-ht4culj6",source="disk.csi.azure.com",subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e",le="+Inf"} 50 cloudprovider_azure_op_duration_seconds_sum{request="azuredisk_csi_driver_controller_unpublish_volume",resource_group="kubetest-ht4culj6",source="disk.csi.azure.com",subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e"} 899.9215685079996 cloudprovider_azure_op_duration_seconds_count{request="azuredisk_csi_driver_controller_unpublish_volume",resource_group="kubetest-ht4culj6",source="disk.csi.azure.com",subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e"} 50 # HELP cloudprovider_azure_op_failure_count [ALPHA] Number of failed Azure service operations # TYPE cloudprovider_azure_op_failure_count counter cloudprovider_azure_op_failure_count{request="azuredisk_csi_driver_controller_delete_volume",resource_group="kubetest-ht4culj6",source="disk.csi.azure.com",subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e"} 2 # HELP disabled_metric_total [ALPHA] The count of disabled metrics. # TYPE disabled_metric_total counter disabled_metric_total 0 # HELP go_cgo_go_to_c_calls_calls_total Count of calls made from Go to C by the current process. ... skipping 67 lines ... # HELP go_gc_heap_objects_objects Number of objects, live or unswept, occupying heap memory. # TYPE go_gc_heap_objects_objects gauge go_gc_heap_objects_objects 52818 # HELP go_gc_heap_tiny_allocs_objects_total Count of small allocations that are packed together into blocks. These allocations are counted separately from other allocations because each individual allocation is not tracked by the runtime, only their block. Each block is already accounted for in allocs-by-size and frees-by-size. # TYPE go_gc_heap_tiny_allocs_objects_total counter go_gc_heap_tiny_allocs_objects_total 47739 # HELP go_gc_limiter_last_enabled_gc_cycle GC cycle the last time the GC CPU limiter was enabled. This metric is useful for diagnosing the root cause of an out-of-memory error, because the limiter trades memory for CPU time when the GC's CPU time gets too high. This is most likely to occur with use of SetMemoryLimit. The first GC cycle is cycle 1, so a value of 0 indicates that it was never enabled. # TYPE go_gc_limiter_last_enabled_gc_cycle gauge go_gc_limiter_last_enabled_gc_cycle 0 # HELP go_gc_pauses_seconds Distribution individual GC-related stop-the-world pause latencies. # TYPE go_gc_pauses_seconds histogram go_gc_pauses_seconds_bucket{le="9.999999999999999e-10"} 0 go_gc_pauses_seconds_bucket{le="9.999999999999999e-09"} 0 ... skipping 259 lines ... # HELP go_gc_heap_objects_objects Number of objects, live or unswept, occupying heap memory. # TYPE go_gc_heap_objects_objects gauge go_gc_heap_objects_objects 35988 # HELP go_gc_heap_tiny_allocs_objects_total Count of small allocations that are packed together into blocks. These allocations are counted separately from other allocations because each individual allocation is not tracked by the runtime, only their block. Each block is already accounted for in allocs-by-size and frees-by-size. # TYPE go_gc_heap_tiny_allocs_objects_total counter go_gc_heap_tiny_allocs_objects_total 4715 # HELP go_gc_limiter_last_enabled_gc_cycle GC cycle the last time the GC CPU limiter was enabled. This metric is useful for diagnosing the root cause of an out-of-memory error, because the limiter trades memory for CPU time when the GC's CPU time gets too high. This is most likely to occur with use of SetMemoryLimit. The first GC cycle is cycle 1, so a value of 0 indicates that it was never enabled. # TYPE go_gc_limiter_last_enabled_gc_cycle gauge go_gc_limiter_last_enabled_gc_cycle 0 # HELP go_gc_pauses_seconds Distribution individual GC-related stop-the-world pause latencies. # TYPE go_gc_pauses_seconds histogram go_gc_pauses_seconds_bucket{le="9.999999999999999e-10"} 0 go_gc_pauses_seconds_bucket{le="9.999999999999999e-09"} 0 ... skipping 272 lines ... [AfterSuite] [38;5;243m/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/suite_test.go:165[0m [38;5;243m------------------------------[0m [38;5;9m[1mSummarizing 1 Failure:[0m [38;5;9m[FAIL][0m [0mDynamic Provisioning [38;5;243m[multi-az] [0m[38;5;9m[1m[It] should create a pod, write to its pv, take a volume snapshot, overwrite data in original pv, create another pod from the snapshot, and read unaltered original data from original pv[disk.csi.azure.com][0m[0m [38;5;243m/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/testsuites/testsuites.go:823[0m [38;5;9m[1mRan 26 of 66 Specs in 3827.372 seconds[0m [38;5;9m[1mFAIL![0m -- [38;5;10m[1m25 Passed[0m | [38;5;9m[1m1 Failed[0m | [38;5;11m[1m0 Pending[0m | [38;5;14m[1m40 Skipped[0m [38;5;228mYou're using deprecated Ginkgo functionality:[0m [38;5;228m=============================================[0m [38;5;11mSupport for custom reporters has been removed in V2. Please read the documentation linked to below for Ginkgo's new behavior and for a migration path:[0m [1mLearn more at:[0m [38;5;14m[4mhttps://onsi.github.io/ginkgo/MIGRATING_TO_V2#removed-custom-reporters[0m [38;5;243mTo silence deprecations that can be silenced set the following environment variable:[0m [38;5;243mACK_GINKGO_DEPRECATIONS=2.4.0[0m --- FAIL: TestE2E (3827.37s) FAIL FAIL sigs.k8s.io/azuredisk-csi-driver/test/e2e 3827.437s FAIL make: *** [Makefile:261: e2e-test] Error 1 2023/01/09 10:30:13 process.go:155: Step 'make e2e-test' finished in 1h5m26.177530524s 2023/01/09 10:30:13 aksengine_helpers.go:425: downloading /root/tmp2587121816/log-dump.sh from https://raw.githubusercontent.com/kubernetes-sigs/cloud-provider-azure/master/hack/log-dump/log-dump.sh 2023/01/09 10:30:13 util.go:70: curl https://raw.githubusercontent.com/kubernetes-sigs/cloud-provider-azure/master/hack/log-dump/log-dump.sh 2023/01/09 10:30:14 process.go:153: Running: chmod +x /root/tmp2587121816/log-dump.sh 2023/01/09 10:30:14 process.go:155: Step 'chmod +x /root/tmp2587121816/log-dump.sh' finished in 2.826003ms 2023/01/09 10:30:14 aksengine_helpers.go:425: downloading /root/tmp2587121816/log-dump-daemonset.yaml from https://raw.githubusercontent.com/kubernetes-sigs/cloud-provider-azure/master/hack/log-dump/log-dump-daemonset.yaml ... skipping 63 lines ... ssh key file /root/.ssh/id_rsa does not exist. Exiting. 2023/01/09 10:30:48 process.go:155: Step 'bash -c /root/tmp2587121816/win-ci-logs-collector.sh kubetest-ht4culj6.westus2.cloudapp.azure.com /root/tmp2587121816 /root/.ssh/id_rsa' finished in 3.580889ms 2023/01/09 10:30:48 aksengine.go:1141: Deleting resource group: kubetest-ht4culj6. 2023/01/09 10:36:53 process.go:96: Saved XML output to /logs/artifacts/junit_runner.xml. 2023/01/09 10:36:53 process.go:153: Running: bash -c . hack/lib/version.sh && KUBE_ROOT=. kube::version::get_version_vars && echo "${KUBE_GIT_VERSION-}" 2023/01/09 10:36:53 process.go:155: Step 'bash -c . hack/lib/version.sh && KUBE_ROOT=. kube::version::get_version_vars && echo "${KUBE_GIT_VERSION-}"' finished in 267.937804ms 2023/01/09 10:36:53 main.go:328: Something went wrong: encountered 1 errors: [error during make e2e-test: exit status 2] + EXIT_VALUE=1 + set +o xtrace Cleaning up after docker in docker. ================================================================================ Cleaning up after docker 0f62db44de36 ... skipping 4 lines ...