kubernetes-sigs/azuredisk-csi-driver PR #1712 pull-azuredisk-csi-driver-external-e2e-single-az-mainv2 #1620832793518936064
Recent runs || View in Spyglass
Recent runs || View in Spyglass
| PR | edreed: [V2] build: disable provenance attestation to work around docker issue |
| Result | FAILURE |
| Tests | 1 failed / 13 succeeded |
| Started | |
| Elapsed | 17m32s |
| Revision | 8d725b098507951e2d5d31ed1a10dafcd0ea89e7 |
| Refs |
1712 |
| job-version | v1.27.0-alpha.1.182+b1667918bc8dc4 |
| kubetest-version | v20230127-9396ca613c |
| revision | v1.27.0-alpha.1.182+b1667918bc8dc4 |
error during make e2e-test: exit status 2
from junit_runner.xml
Filter through log files | View test history on testgrid
kubetest Check APIReachability
kubetest Deferred TearDown
kubetest DumpClusterLogs
kubetest GetDeployer
kubetest IsUp
kubetest Prepare
kubetest TearDown
kubetest TearDown Previous
kubetest Timeout
kubetest Up
kubetest kubectl version
kubetest list nodes
kubetest test setup
... skipping 1080 lines ... type: string type: object oneOf: - required: ["persistentVolumeClaimName"] - required: ["volumeSnapshotContentName"] volumeSnapshotClassName: description: 'VolumeSnapshotClassName is the name of the VolumeSnapshotClass requested by the VolumeSnapshot. VolumeSnapshotClassName may be left nil to indicate that the default SnapshotClass should be used. A given cluster may have multiple default Volume SnapshotClasses: one default per CSI Driver. If a VolumeSnapshot does not specify a SnapshotClass, VolumeSnapshotSource will be checked to figure out what the associated CSI Driver is, and the default VolumeSnapshotClass associated with that CSI Driver will be used. If more than one VolumeSnapshotClass exist for a given CSI Driver and more than one have been marked as default, CreateSnapshot will fail and generate an event. Empty string is not allowed for this field.' type: string required: - source type: object status: description: status represents the current information of a snapshot. Consumers must verify binding between VolumeSnapshot and VolumeSnapshotContent objects is successful (by validating that both VolumeSnapshot and VolumeSnapshotContent point at each other) before using this object. ... skipping 2 lines ... description: 'boundVolumeSnapshotContentName is the name of the VolumeSnapshotContent object to which this VolumeSnapshot object intends to bind to. If not specified, it indicates that the VolumeSnapshot object has not been successfully bound to a VolumeSnapshotContent object yet. NOTE: To avoid possible security issues, consumers must verify binding between VolumeSnapshot and VolumeSnapshotContent objects is successful (by validating that both VolumeSnapshot and VolumeSnapshotContent point at each other) before using this object.' type: string creationTime: description: creationTime is the timestamp when the point-in-time snapshot is taken by the underlying storage system. In dynamic snapshot creation case, this field will be filled in by the snapshot controller with the "creation_time" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "creation_time" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. If not specified, it may indicate that the creation time of the snapshot is unknown. format: date-time type: string error: description: error is the last observed error during snapshot creation, if any. This field could be helpful to upper level controllers(i.e., application controller) to decide whether they should continue on waiting for the snapshot to be created based on the type of error reported. The snapshot controller will keep retrying when an error occurs during the snapshot creation. Upon success, this error field will be cleared. properties: message: description: 'message is a string detailing the encountered error during snapshot creation if specified. NOTE: message may be logged, and it should not contain sensitive information.' type: string time: description: time is the timestamp when the error was encountered. format: date-time type: string type: object readyToUse: description: readyToUse indicates if the snapshot is ready to be used to restore a volume. In dynamic snapshot creation case, this field will be filled in by the snapshot controller with the "ready_to_use" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "ready_to_use" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it, otherwise, this field will be set to "True". If not specified, it means the readiness of a snapshot is unknown. type: boolean restoreSize: type: string description: restoreSize represents the minimum size of volume required to create a volume from this snapshot. In dynamic snapshot creation case, this field will be filled in by the snapshot controller with the "size_bytes" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "size_bytes" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. When restoring a volume from this snapshot, the size of the volume MUST NOT be smaller than the restoreSize if it is specified, otherwise the restoration will fail. If not specified, it indicates that the size is unknown. pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ x-kubernetes-int-or-string: true type: object required: - spec type: object ... skipping 60 lines ... type: string volumeSnapshotContentName: description: volumeSnapshotContentName specifies the name of a pre-existing VolumeSnapshotContent object representing an existing volume snapshot. This field should be set if the snapshot already exists and only needs a representation in Kubernetes. This field is immutable. type: string type: object volumeSnapshotClassName: description: 'VolumeSnapshotClassName is the name of the VolumeSnapshotClass requested by the VolumeSnapshot. VolumeSnapshotClassName may be left nil to indicate that the default SnapshotClass should be used. A given cluster may have multiple default Volume SnapshotClasses: one default per CSI Driver. If a VolumeSnapshot does not specify a SnapshotClass, VolumeSnapshotSource will be checked to figure out what the associated CSI Driver is, and the default VolumeSnapshotClass associated with that CSI Driver will be used. If more than one VolumeSnapshotClass exist for a given CSI Driver and more than one have been marked as default, CreateSnapshot will fail and generate an event. Empty string is not allowed for this field.' type: string required: - source type: object status: description: status represents the current information of a snapshot. Consumers must verify binding between VolumeSnapshot and VolumeSnapshotContent objects is successful (by validating that both VolumeSnapshot and VolumeSnapshotContent point at each other) before using this object. ... skipping 2 lines ... description: 'boundVolumeSnapshotContentName is the name of the VolumeSnapshotContent object to which this VolumeSnapshot object intends to bind to. If not specified, it indicates that the VolumeSnapshot object has not been successfully bound to a VolumeSnapshotContent object yet. NOTE: To avoid possible security issues, consumers must verify binding between VolumeSnapshot and VolumeSnapshotContent objects is successful (by validating that both VolumeSnapshot and VolumeSnapshotContent point at each other) before using this object.' type: string creationTime: description: creationTime is the timestamp when the point-in-time snapshot is taken by the underlying storage system. In dynamic snapshot creation case, this field will be filled in by the snapshot controller with the "creation_time" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "creation_time" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. If not specified, it may indicate that the creation time of the snapshot is unknown. format: date-time type: string error: description: error is the last observed error during snapshot creation, if any. This field could be helpful to upper level controllers(i.e., application controller) to decide whether they should continue on waiting for the snapshot to be created based on the type of error reported. The snapshot controller will keep retrying when an error occurs during the snapshot creation. Upon success, this error field will be cleared. properties: message: description: 'message is a string detailing the encountered error during snapshot creation if specified. NOTE: message may be logged, and it should not contain sensitive information.' type: string time: description: time is the timestamp when the error was encountered. format: date-time type: string type: object readyToUse: description: readyToUse indicates if the snapshot is ready to be used to restore a volume. In dynamic snapshot creation case, this field will be filled in by the snapshot controller with the "ready_to_use" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "ready_to_use" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it, otherwise, this field will be set to "True". If not specified, it means the readiness of a snapshot is unknown. type: boolean restoreSize: type: string description: restoreSize represents the minimum size of volume required to create a volume from this snapshot. In dynamic snapshot creation case, this field will be filled in by the snapshot controller with the "size_bytes" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "size_bytes" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. When restoring a volume from this snapshot, the size of the volume MUST NOT be smaller than the restoreSize if it is specified, otherwise the restoration will fail. If not specified, it indicates that the size is unknown. pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ x-kubernetes-int-or-string: true type: object required: - spec type: object ... skipping 254 lines ... description: status represents the current information of a snapshot. properties: creationTime: description: creationTime is the timestamp when the point-in-time snapshot is taken by the underlying storage system. In dynamic snapshot creation case, this field will be filled in by the CSI snapshotter sidecar with the "creation_time" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "creation_time" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. If not specified, it indicates the creation time is unknown. The format of this field is a Unix nanoseconds time encoded as an int64. On Unix, the command `date +%s%N` returns the current time in nanoseconds since 1970-01-01 00:00:00 UTC. format: int64 type: integer error: description: error is the last observed error during snapshot creation, if any. Upon success after retry, this error field will be cleared. properties: message: description: 'message is a string detailing the encountered error during snapshot creation if specified. NOTE: message may be logged, and it should not contain sensitive information.' type: string time: description: time is the timestamp when the error was encountered. format: date-time type: string type: object readyToUse: description: readyToUse indicates if a snapshot is ready to be used to restore a volume. In dynamic snapshot creation case, this field will be filled in by the CSI snapshotter sidecar with the "ready_to_use" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "ready_to_use" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it, otherwise, this field will be set to "True". If not specified, it means the readiness of a snapshot is unknown. type: boolean restoreSize: description: restoreSize represents the complete size of the snapshot in bytes. In dynamic snapshot creation case, this field will be filled in by the CSI snapshotter sidecar with the "size_bytes" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "size_bytes" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. When restoring a volume from this snapshot, the size of the volume MUST NOT be smaller than the restoreSize if it is specified, otherwise the restoration will fail. If not specified, it indicates that the size is unknown. format: int64 minimum: 0 type: integer snapshotHandle: description: snapshotHandle is the CSI "snapshot_id" of a snapshot on the underlying storage system. If not specified, it indicates that dynamic snapshot creation has either failed or it is still in progress. type: string type: object required: - spec type: object served: true ... skipping 108 lines ... description: status represents the current information of a snapshot. properties: creationTime: description: creationTime is the timestamp when the point-in-time snapshot is taken by the underlying storage system. In dynamic snapshot creation case, this field will be filled in by the CSI snapshotter sidecar with the "creation_time" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "creation_time" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. If not specified, it indicates the creation time is unknown. The format of this field is a Unix nanoseconds time encoded as an int64. On Unix, the command `date +%s%N` returns the current time in nanoseconds since 1970-01-01 00:00:00 UTC. format: int64 type: integer error: description: error is the last observed error during snapshot creation, if any. Upon success after retry, this error field will be cleared. properties: message: description: 'message is a string detailing the encountered error during snapshot creation if specified. NOTE: message may be logged, and it should not contain sensitive information.' type: string time: description: time is the timestamp when the error was encountered. format: date-time type: string type: object readyToUse: description: readyToUse indicates if a snapshot is ready to be used to restore a volume. In dynamic snapshot creation case, this field will be filled in by the CSI snapshotter sidecar with the "ready_to_use" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "ready_to_use" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it, otherwise, this field will be set to "True". If not specified, it means the readiness of a snapshot is unknown. type: boolean restoreSize: description: restoreSize represents the complete size of the snapshot in bytes. In dynamic snapshot creation case, this field will be filled in by the CSI snapshotter sidecar with the "size_bytes" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "size_bytes" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. When restoring a volume from this snapshot, the size of the volume MUST NOT be smaller than the restoreSize if it is specified, otherwise the restoration will fail. If not specified, it indicates that the size is unknown. format: int64 minimum: 0 type: integer snapshotHandle: description: snapshotHandle is the CSI "snapshot_id" of a snapshot on the underlying storage system. If not specified, it indicates that dynamic snapshot creation has either failed or it is still in progress. type: string type: object required: - spec type: object served: true ... skipping 359 lines ... - volumeName - volume_context - volume_id type: object status: description: status represents the current state of AzVolumeAttachment. includes error, state, and attachment status Required properties: detail: description: Status summarizes the current attachment state of the volume attachment Nil Status indicates that the volume has not yet been attached to the node properties: ... skipping 7 lines ... role: description: The current attachment role. type: string required: - role type: object error: description: Error occurred during attach/detach of volume properties: code: type: string message: type: string parameters: ... skipping 90 lines ... - volumeName - volume_context - volume_id type: object status: description: status represents the current state of AzVolumeAttachment. includes error, state, and attachment status properties: annotation: additionalProperties: type: string description: Annotations contains additional resource information to guide driver actions ... skipping 13 lines ... role: description: The current attachment role. type: string required: - role type: object error: description: Error occurred during attach/detach of volume properties: code: type: string message: type: string parameters: ... skipping 169 lines ... - maxMountReplicaCount - volumeCapability - volumeName type: object status: description: status represents the current state of AzVolume. includes error, state, and volume status properties: detail: description: Current status detail of the AzVolume Nil detail indicates that the volume has not been created properties: accessible_topology: ... skipping 28 lines ... type: string required: - capacity_bytes - node_expansion_required - volume_id type: object error: description: Error occurred during creation/deletion of volume properties: code: type: string message: type: string parameters: ... skipping 154 lines ... - maxMountReplicaCount - volumeCapability - volumeName type: object status: description: status represents the current state of AzVolume. includes error, state, and volume status properties: annotation: additionalProperties: type: string description: Annotations contains additional resource information to guide driver actions ... skipping 34 lines ... type: string required: - capacity_bytes - node_expansion_required - volume_id type: object error: description: Error occurred during creation/deletion of volume properties: code: type: string message: type: string parameters: ... skipping 1069 lines ... image: "mcr.microsoft.com/oss/kubernetes-csi/csi-resizer:v1.5.0" args: - "-csi-address=$(ADDRESS)" - "-v=2" - "-leader-election" - "--leader-election-namespace=kube-system" - '-handle-volume-inuse-error=false' - '-feature-gates=RecoverVolumeExpansionFailure=true' - "-timeout=240s" env: - name: ADDRESS value: /csi/csi.sock volumeMounts: ... skipping 380 lines ... I0201 17:21:38.281689 1 reflector.go:257] Listing and watching *v1.Node from k8s.io/client-go/informers/factory.go:150 I0201 17:21:38.281237 1 reflector.go:221] Starting reflector *v1.CustomResourceDefinition (30s) from k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132 I0201 17:21:38.281836 1 reflector.go:257] Listing and watching *v1.CustomResourceDefinition from k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132 I0201 17:21:38.381325 1 shared_informer.go:303] caches populated I0201 17:21:38.381421 1 azuredisk_v2.go:225] driver userAgent: test.csi.azure.com/latest-v2-8ab762a4766ca4352f3bbe3e0c05b326019e12ba e2e-test I0201 17:21:38.381438 1 azure_disk_utils.go:564] reading cloud config from secret kube-system/azure-cloud-provider I0201 17:21:38.384448 1 azure_disk_utils.go:571] InitializeCloudFromSecret: failed to get cloud config from secret kube-system/azure-cloud-provider: failed to get secret kube-system/azure-cloud-provider: secrets "azure-cloud-provider" not found I0201 17:21:38.384469 1 azure_disk_utils.go:576] could not read cloud config from secret kube-system/azure-cloud-provider I0201 17:21:38.384477 1 azure_disk_utils.go:586] use default AZURE_CREDENTIAL_FILE env var: /etc/kubernetes/azure.json I0201 17:21:38.384507 1 azure_disk_utils.go:594] read cloud config from file: /etc/kubernetes/azure.json successfully I0201 17:21:38.385199 1 azure_auth.go:253] Using AzurePublicCloud environment I0201 17:21:38.385215 1 azure_auth.go:104] azure: using managed identity extension to retrieve access token I0201 17:21:38.385222 1 azure_auth.go:110] azure: using User Assigned MSI ID to retrieve access token ... skipping 284 lines ... I0201 17:21:38.626292 1 shared_state.go:489] "msg"="Storing pod coredns-7cf78b68d8-zwm85 and claim [] to podToClaimsMap map." "disk.csi.azure.com/request-id"="e2c88e0d-a254-11ed-abc7-6045bd9323b2" I0201 17:21:38.626321 1 pod.go:91] "msg"="Creating replicas for pod kube-system/coredns-7cf78b68d8-zwm85." "disk.csi.azure.com/request-id"="e2c88e0d-a254-11ed-abc7-6045bd9323b2" "disk.csi/azure.com/pod-name"="kube-system/coredns-7cf78b68d8-zwm85" I0201 17:21:38.626357 1 shared_state.go:313] "msg"="Getting requested volumes for pod (kube-system/coredns-7cf78b68d8-zwm85)." "disk.csi.azure.com/request-id"="e2c88e0d-a254-11ed-abc7-6045bd9323b2" "disk.csi/azure.com/pod-name"="kube-system/coredns-7cf78b68d8-zwm85" I0201 17:21:38.626382 1 pod.go:99] "msg"="Pod kube-system/coredns-7cf78b68d8-zwm85 has 0 volumes. Volumes: []" "disk.csi.azure.com/request-id"="e2c88e0d-a254-11ed-abc7-6045bd9323b2" "disk.csi/azure.com/pod-name"="kube-system/coredns-7cf78b68d8-zwm85" I0201 17:21:38.626422 1 pod.go:89] "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="e2c88e0d-a254-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcilePod).createReplicas" "disk.csi/azure.com/pod-name"="kube-system/coredns-7cf78b68d8-zwm85" "latency"=92902 I0201 17:21:38.626442 1 shared_state.go:419] "msg"="Adding pod csi-test-controller-5c87f9dff6-n6xg5 to shared map with keyName kube-system/csi-test-controller-5c87f9dff6-n6xg5." "disk.csi.azure.com/request-id"="e2c88e0d-a254-11ed-abc7-6045bd9323b2" I0201 17:21:38.627187 1 shared_state.go:425] "msg"="Pod spec of pod csi-test-controller-5c87f9dff6-n6xg5 is: {Volumes:[{Name:socket-dir VolumeSource:{HostPath:nil EmptyDir:&EmptyDirVolumeSource{Medium:,SizeLimit:<nil>,} GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:azure-cred VolumeSource:{HostPath:&HostPathVolumeSource{Path:/etc/kubernetes/,Type:*DirectoryOrCreate,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:csi-test-controller-config VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:&ConfigMapVolumeSource{LocalObjectReference:LocalObjectReference{Name:csi-test-controller-config,},Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,} VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:kube-api-access-mvtk7 VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:&ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,} PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:csi-provisioner Image:mcr.microsoft.com/oss/kubernetes-csi/csi-provisioner:v3.2.0 Command:[] Args:[--feature-gates=Topology=true --csi-address=$(ADDRESS) --v=2 --timeout=30s --leader-election --leader-election-namespace=kube-system --worker-threads=100 --extra-create-metadata=true --strict-topology=true --kube-api-qps=50 --kube-api-burst=100] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:ADDRESS Value:/csi/csi.sock ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:524288000 scale:0} d:{Dec:<nil>} s:500Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}] Claims:[]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-mvtk7 ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:Always SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:csi-attacher Image:mcr.microsoft.com/oss/kubernetes-csi/csi-attacher:v3.5.0 Command:[] Args:[-v=2 -csi-address=$(ADDRESS) -timeout=600s -leader-election --leader-election-namespace=kube-system -worker-threads=500 -kube-api-qps=50 -kube-api-burst=100] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:ADDRESS Value:/csi/csi.sock ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:524288000 scale:0} d:{Dec:<nil>} s:500Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}] Claims:[]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-mvtk7 ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:Always SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:csi-snapshotter Image:mcr.microsoft.com/oss/kubernetes-csi/csi-snapshotter:v5.0.1 Command:[] Args:[-csi-address=$(ADDRESS) -leader-election --leader-election-namespace=kube-system -v=2] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:ADDRESS Value:/csi/csi.sock ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:104857600 scale:0} d:{Dec:<nil>} s:100Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}] Claims:[]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-mvtk7 ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:Always SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:csi-resizer Image:mcr.microsoft.com/oss/kubernetes-csi/csi-resizer:v1.5.0 Command:[] Args:[-csi-address=$(ADDRESS) -v=2 -leader-election --leader-election-namespace=kube-system -handle-volume-inuse-error=false -feature-gates=RecoverVolumeExpansionFailure=true -timeout=240s] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:ADDRESS Value:/csi/csi.sock ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:524288000 scale:0} d:{Dec:<nil>} s:500Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}] Claims:[]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-mvtk7 ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:Always SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:liveness-probe Image:mcr.microsoft.com/oss/kubernetes-csi/livenessprobe:v2.7.0 Command:[] Args:[--csi-address=/csi/csi.sock --probe-timeout=3s --health-port=29602 --v=2] WorkingDir: Ports:[] EnvFrom:[] Env:[] Resources:{Limits:map[memory:{i:{value:104857600 scale:0} d:{Dec:<nil>} s:100Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}] Claims:[]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-mvtk7 ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:Always SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:azuredisk Image:k8sprow.azurecr.io/azuredisk-csi:latest-v2-8ab762a4766ca4352f3bbe3e0c05b326019e12ba Command:[] Args:[--v=5 --config=/etc/csi-test-controller/config.yaml] WorkingDir: Ports:[{Name:healthz HostPort:29602 ContainerPort:29602 Protocol:TCP HostIP:} {Name:metrics HostPort:29604 ContainerPort:29604 Protocol:TCP HostIP:}] EnvFrom:[] Env:[{Name:AZURE_CREDENTIAL_FILE Value: ValueFrom:&EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:&ConfigMapKeySelector{LocalObjectReference:LocalObjectReference{Name:azure-cred-file,},Key:path,Optional:*true,},SecretKeyRef:nil,}} {Name:CSI_ENDPOINT Value:unix:///csi/csi.sock ValueFrom:nil} {Name:AZURE_GO_SDK_LOG_LEVEL Value: ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:524288000 scale:0} d:{Dec:<nil>} s:500Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}] Claims:[]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:azure-cred ReadOnly:false MountPath:/etc/kubernetes/ SubPath: MountPropagation:<nil> SubPathExpr:} {Name:csi-test-controller-config ReadOnly:false MountPath:/etc/csi-test-controller SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-mvtk7 ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/healthz,Port:{1 0 healthz},Host:,Scheme:HTTP,HTTPHeaders:[]HTTPHeader{},},TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:30,TimeoutSeconds:10,PeriodSeconds:30,SuccessThreshold:1,FailureThreshold:5,TerminationGracePeriodSeconds:nil,} ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:Always SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy:Always TerminationGracePeriodSeconds:0xc000ea6b10 ActiveDeadlineSeconds:<nil> DNSPolicy:ClusterFirst NodeSelector:map[kubernetes.io/os:linux] ServiceAccountName:csi-azuredisk-controller-sa DeprecatedServiceAccount:csi-azuredisk-controller-sa AutomountServiceAccountToken:<nil> NodeName:k8s-agentpool1-64138072-vmss000000 HostNetwork:true HostPID:false HostIPC:false ShareProcessNamespace:<nil> SecurityContext:&PodSecurityContext{SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,SupplementalGroups:[],FSGroup:nil,RunAsGroup:nil,Sysctls:[]Sysctl{},WindowsOptions:nil,FSGroupChangePolicy:nil,SeccompProfile:nil,} ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName:default-scheduler Tolerations:[{Key:node-role.kubernetes.io/master Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>} {Key:node-role.kubernetes.io/controlplane Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>} {Key:node-role.kubernetes.io/control-plane Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>} {Key:node.kubernetes.io/not-ready Operator:Exists Value: Effect:NoExecute TolerationSeconds:0xc000ea6b18} {Key:node.kubernetes.io/unreachable Operator:Exists Value: Effect:NoExecute TolerationSeconds:0xc000ea6b20}] HostAliases:[] PriorityClassName:system-cluster-critical Priority:0xc000ea6b28 DNSConfig:nil ReadinessGates:[] RuntimeClassName:<nil> EnableServiceLinks:0xc000ea6b2c PreemptionPolicy:0xc000917300 Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN:<nil> OS:nil HostUsers:<nil> SchedulingGates:[] ResourceClaims:[]}. With volumes: [{Name:socket-dir VolumeSource:{HostPath:nil EmptyDir:&EmptyDirVolumeSource{Medium:,SizeLimit:<nil>,} GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:azure-cred VolumeSource:{HostPath:&HostPathVolumeSource{Path:/etc/kubernetes/,Type:*DirectoryOrCreate,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:csi-test-controller-config VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:&ConfigMapVolumeSource{LocalObjectReference:LocalObjectReference{Name:csi-test-controller-config,},Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,} VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:kube-api-access-mvtk7 VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:&ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,} PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}]" "disk.csi.azure.com/request-id"="e2c88e0d-a254-11ed-abc7-6045bd9323b2" I0201 17:21:38.627365 1 shared_state.go:463] "msg"="Pod csi-test-controller-5c87f9dff6-n6xg5: Skipping Volume {socket-dir {nil &EmptyDirVolumeSource{Medium:,SizeLimit:<nil>,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}}. No persistent volume exists." "disk.csi.azure.com/request-id"="e2c88e0d-a254-11ed-abc7-6045bd9323b2" I0201 17:21:38.627428 1 shared_state.go:463] "msg"="Pod csi-test-controller-5c87f9dff6-n6xg5: Skipping Volume {azure-cred {&HostPathVolumeSource{Path:/etc/kubernetes/,Type:*DirectoryOrCreate,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}}. No persistent volume exists." "disk.csi.azure.com/request-id"="e2c88e0d-a254-11ed-abc7-6045bd9323b2" I0201 17:21:38.627475 1 shared_state.go:463] "msg"="Pod csi-test-controller-5c87f9dff6-n6xg5: Skipping Volume {csi-test-controller-config {nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil &ConfigMapVolumeSource{LocalObjectReference:LocalObjectReference{Name:csi-test-controller-config,},Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,} nil nil nil nil nil nil nil nil nil nil}}. No persistent volume exists." "disk.csi.azure.com/request-id"="e2c88e0d-a254-11ed-abc7-6045bd9323b2" I0201 17:21:38.627545 1 shared_state.go:463] "msg"="Pod csi-test-controller-5c87f9dff6-n6xg5: Skipping Volume {kube-api-access-mvtk7 {nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil &ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,} nil nil nil nil nil}}. No persistent volume exists." "disk.csi.azure.com/request-id"="e2c88e0d-a254-11ed-abc7-6045bd9323b2" I0201 17:21:38.627565 1 shared_state.go:489] "msg"="Storing pod csi-test-controller-5c87f9dff6-n6xg5 and claim [] to podToClaimsMap map." "disk.csi.azure.com/request-id"="e2c88e0d-a254-11ed-abc7-6045bd9323b2" I0201 17:21:38.627819 1 pod.go:91] "msg"="Creating replicas for pod kube-system/csi-test-controller-5c87f9dff6-n6xg5." "disk.csi.azure.com/request-id"="e2c88e0d-a254-11ed-abc7-6045bd9323b2" "disk.csi/azure.com/pod-name"="kube-system/csi-test-controller-5c87f9dff6-n6xg5" ... skipping 226 lines ... I0201 17:21:38.813159 1 shared_state.go:463] "msg"="Pod kube-proxy-lxwn4: Skipping Volume {kube-api-access-7p2p5 {nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil &ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,} nil nil nil nil nil}}. No persistent volume exists." I0201 17:21:38.813173 1 shared_state.go:489] "msg"="Storing pod kube-proxy-lxwn4 and claim [] to podToClaimsMap map." I0201 17:21:38.813194 1 pod.go:91] "msg"="Creating replicas for pod kube-system/kube-proxy-lxwn4." "disk.csi.azure.com/request-id"="e30611f0-a254-11ed-abc7-6045bd9323b2" "disk.csi/azure.com/pod-name"="kube-system/kube-proxy-lxwn4" I0201 17:21:38.813208 1 shared_state.go:313] "msg"="Getting requested volumes for pod (kube-system/kube-proxy-lxwn4)." "disk.csi.azure.com/request-id"="e30611f0-a254-11ed-abc7-6045bd9323b2" "disk.csi/azure.com/pod-name"="kube-system/kube-proxy-lxwn4" I0201 17:21:38.813224 1 pod.go:99] "msg"="Pod kube-system/kube-proxy-lxwn4 has 0 volumes. Volumes: []" "disk.csi.azure.com/request-id"="e30611f0-a254-11ed-abc7-6045bd9323b2" "disk.csi/azure.com/pod-name"="kube-system/kube-proxy-lxwn4" I0201 17:21:38.813245 1 pod.go:89] "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="e30611f0-a254-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcilePod).createReplicas" "disk.csi/azure.com/pod-name"="kube-system/kube-proxy-lxwn4" "latency"=48201 I0201 17:21:38.830029 1 node.go:90] AzDiskControllerManager "msg"="Node is now available. Will requeue failed replica creation requests." "controller"="node" "disk.csi.azure.com/node-name"="k8s-master-64138072-0" "namespace"="azure-disk-csi" "partition"="csi-azuredisk-controller" I0201 17:21:38.830097 1 shared_state.go:1203] "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="e308a5ac-a254-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*SharedState).tryCreateFailedReplicas" "latency"=22301 I0201 17:21:38.830107 1 node.go:90] AzDiskControllerManager "msg"="Node is now available. Will requeue failed replica creation requests." "controller"="node" "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000001" "namespace"="azure-disk-csi" "partition"="csi-azuredisk-controller" I0201 17:21:38.830172 1 node.go:90] AzDiskControllerManager "msg"="Node is now available. Will requeue failed replica creation requests." "controller"="node" "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000002" "namespace"="azure-disk-csi" "partition"="csi-azuredisk-controller" I0201 17:21:38.830178 1 shared_state.go:1203] "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="e308a8b5-a254-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*SharedState).tryCreateFailedReplicas" "latency"=28501 I0201 17:21:38.830208 1 shared_state.go:1203] "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="e308aa2f-a254-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*SharedState).tryCreateFailedReplicas" "latency"=5700 I0201 17:21:38.830259 1 node.go:90] AzDiskControllerManager "msg"="Node is now available. Will requeue failed replica creation requests." "controller"="node" "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000000" "namespace"="azure-disk-csi" "partition"="csi-azuredisk-controller" I0201 17:21:38.830310 1 shared_state.go:1203] "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="e308adf3-a254-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*SharedState).tryCreateFailedReplicas" "latency"=8200 I0201 17:21:38.912045 1 shared_state.go:419] "msg"="Adding pod csi-snapshot-controller-6b69b99bbc-s2kwb to shared map with keyName kube-system/csi-snapshot-controller-6b69b99bbc-s2kwb." I0201 17:21:38.912255 1 shared_state.go:425] "msg"="Pod spec of pod csi-snapshot-controller-6b69b99bbc-s2kwb is: {Volumes:[{Name:kube-api-access-7vrhx VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:&ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,} PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:csi-snapshot-controller Image:mcr.microsoft.com/oss/kubernetes-csi/snapshot-controller:v5.0.1 Command:[] Args:[--v=2 --leader-election=true --leader-election-namespace=kube-system] WorkingDir: Ports:[] EnvFrom:[] Env:[] Resources:{Limits:map[memory:{i:{value:104857600 scale:0} d:{Dec:<nil>} s:100Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}] Claims:[]} VolumeMounts:[{Name:kube-api-access-7vrhx ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:Always SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy:Always TerminationGracePeriodSeconds:0xc001080110 ActiveDeadlineSeconds:<nil> DNSPolicy:ClusterFirst NodeSelector:map[kubernetes.io/os:linux] ServiceAccountName:csi-snapshot-controller-sa DeprecatedServiceAccount:csi-snapshot-controller-sa AutomountServiceAccountToken:<nil> NodeName:k8s-agentpool1-64138072-vmss000001 HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace:<nil> SecurityContext:&PodSecurityContext{SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,SupplementalGroups:[],FSGroup:nil,RunAsGroup:nil,Sysctls:[]Sysctl{},WindowsOptions:nil,FSGroupChangePolicy:nil,SeccompProfile:nil,} ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName:default-scheduler Tolerations:[{Key:node-role.kubernetes.io/master Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>} {Key:node-role.kubernetes.io/controlplane Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>} {Key:node-role.kubernetes.io/control-plane Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>} {Key:node.kubernetes.io/not-ready Operator:Exists Value: Effect:NoExecute TolerationSeconds:0xc001080118} {Key:node.kubernetes.io/unreachable Operator:Exists Value: Effect:NoExecute TolerationSeconds:0xc001080120}] HostAliases:[] PriorityClassName:system-cluster-critical Priority:0xc001080128 DNSConfig:nil ReadinessGates:[] RuntimeClassName:<nil> EnableServiceLinks:0xc00108012c PreemptionPolicy:0xc0005f6680 Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN:<nil> OS:nil HostUsers:<nil> SchedulingGates:[] ResourceClaims:[]}. With volumes: [{Name:kube-api-access-7vrhx VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:&ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,} PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}]" I0201 17:21:38.912306 1 shared_state.go:463] "msg"="Pod csi-snapshot-controller-6b69b99bbc-s2kwb: Skipping Volume {kube-api-access-7vrhx {nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil &ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,} nil nil nil nil nil}}. No persistent volume exists." I0201 17:21:38.912321 1 shared_state.go:489] "msg"="Storing pod csi-snapshot-controller-6b69b99bbc-s2kwb and claim [] to podToClaimsMap map." I0201 17:21:38.912349 1 pod.go:91] "msg"="Creating replicas for pod kube-system/csi-snapshot-controller-6b69b99bbc-s2kwb." "disk.csi.azure.com/request-id"="e3153334-a254-11ed-abc7-6045bd9323b2" "disk.csi/azure.com/pod-name"="kube-system/csi-snapshot-controller-6b69b99bbc-s2kwb" ... skipping 6 lines ... I0201 17:21:39.013209 1 shared_state.go:489] "msg"="Storing pod cloud-node-manager-kwvgd and claim [] to podToClaimsMap map." I0201 17:21:39.013244 1 pod.go:91] "msg"="Creating replicas for pod kube-system/cloud-node-manager-kwvgd." "disk.csi.azure.com/request-id"="e3249826-a254-11ed-abc7-6045bd9323b2" "disk.csi/azure.com/pod-name"="kube-system/cloud-node-manager-kwvgd" I0201 17:21:39.013269 1 shared_state.go:313] "msg"="Getting requested volumes for pod (kube-system/cloud-node-manager-kwvgd)." "disk.csi.azure.com/request-id"="e3249826-a254-11ed-abc7-6045bd9323b2" "disk.csi/azure.com/pod-name"="kube-system/cloud-node-manager-kwvgd" I0201 17:21:39.013290 1 pod.go:99] "msg"="Pod kube-system/cloud-node-manager-kwvgd has 0 volumes. Volumes: []" "disk.csi.azure.com/request-id"="e3249826-a254-11ed-abc7-6045bd9323b2" "disk.csi/azure.com/pod-name"="kube-system/cloud-node-manager-kwvgd" I0201 17:21:39.013324 1 pod.go:89] "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="e3249826-a254-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcilePod).createReplicas" "disk.csi/azure.com/pod-name"="kube-system/cloud-node-manager-kwvgd" "latency"=77501 I0201 17:21:39.053351 1 shared_state.go:419] "msg"="Adding pod csi-test-controller-5c87f9dff6-n6xg5 to shared map with keyName kube-system/csi-test-controller-5c87f9dff6-n6xg5." I0201 17:21:39.053826 1 shared_state.go:425] "msg"="Pod spec of pod csi-test-controller-5c87f9dff6-n6xg5 is: {Volumes:[{Name:socket-dir VolumeSource:{HostPath:nil EmptyDir:&EmptyDirVolumeSource{Medium:,SizeLimit:<nil>,} GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:azure-cred VolumeSource:{HostPath:&HostPathVolumeSource{Path:/etc/kubernetes/,Type:*DirectoryOrCreate,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:csi-test-controller-config VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:&ConfigMapVolumeSource{LocalObjectReference:LocalObjectReference{Name:csi-test-controller-config,},Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,} VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:kube-api-access-mvtk7 VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:&ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,} PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:csi-provisioner Image:mcr.microsoft.com/oss/kubernetes-csi/csi-provisioner:v3.2.0 Command:[] Args:[--feature-gates=Topology=true --csi-address=$(ADDRESS) --v=2 --timeout=30s --leader-election --leader-election-namespace=kube-system --worker-threads=100 --extra-create-metadata=true --strict-topology=true --kube-api-qps=50 --kube-api-burst=100] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:ADDRESS Value:/csi/csi.sock ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:524288000 scale:0} d:{Dec:<nil>} s:500Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}] Claims:[]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-mvtk7 ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:Always SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:csi-attacher Image:mcr.microsoft.com/oss/kubernetes-csi/csi-attacher:v3.5.0 Command:[] Args:[-v=2 -csi-address=$(ADDRESS) -timeout=600s -leader-election --leader-election-namespace=kube-system -worker-threads=500 -kube-api-qps=50 -kube-api-burst=100] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:ADDRESS Value:/csi/csi.sock ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:524288000 scale:0} d:{Dec:<nil>} s:500Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}] Claims:[]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-mvtk7 ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:Always SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:csi-snapshotter Image:mcr.microsoft.com/oss/kubernetes-csi/csi-snapshotter:v5.0.1 Command:[] Args:[-csi-address=$(ADDRESS) -leader-election --leader-election-namespace=kube-system -v=2] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:ADDRESS Value:/csi/csi.sock ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:104857600 scale:0} d:{Dec:<nil>} s:100Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}] Claims:[]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-mvtk7 ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:Always SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:csi-resizer Image:mcr.microsoft.com/oss/kubernetes-csi/csi-resizer:v1.5.0 Command:[] Args:[-csi-address=$(ADDRESS) -v=2 -leader-election --leader-election-namespace=kube-system -handle-volume-inuse-error=false -feature-gates=RecoverVolumeExpansionFailure=true -timeout=240s] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:ADDRESS Value:/csi/csi.sock ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:524288000 scale:0} d:{Dec:<nil>} s:500Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}] Claims:[]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-mvtk7 ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:Always SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:liveness-probe Image:mcr.microsoft.com/oss/kubernetes-csi/livenessprobe:v2.7.0 Command:[] Args:[--csi-address=/csi/csi.sock --probe-timeout=3s --health-port=29602 --v=2] WorkingDir: Ports:[] EnvFrom:[] Env:[] Resources:{Limits:map[memory:{i:{value:104857600 scale:0} d:{Dec:<nil>} s:100Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}] Claims:[]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-mvtk7 ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:Always SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:azuredisk Image:k8sprow.azurecr.io/azuredisk-csi:latest-v2-8ab762a4766ca4352f3bbe3e0c05b326019e12ba Command:[] Args:[--v=5 --config=/etc/csi-test-controller/config.yaml] WorkingDir: Ports:[{Name:healthz HostPort:29602 ContainerPort:29602 Protocol:TCP HostIP:} {Name:metrics HostPort:29604 ContainerPort:29604 Protocol:TCP HostIP:}] EnvFrom:[] Env:[{Name:AZURE_CREDENTIAL_FILE Value: ValueFrom:&EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:&ConfigMapKeySelector{LocalObjectReference:LocalObjectReference{Name:azure-cred-file,},Key:path,Optional:*true,},SecretKeyRef:nil,}} {Name:CSI_ENDPOINT Value:unix:///csi/csi.sock ValueFrom:nil} {Name:AZURE_GO_SDK_LOG_LEVEL Value: ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:524288000 scale:0} d:{Dec:<nil>} s:500Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}] Claims:[]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:azure-cred ReadOnly:false MountPath:/etc/kubernetes/ SubPath: MountPropagation:<nil> SubPathExpr:} {Name:csi-test-controller-config ReadOnly:false MountPath:/etc/csi-test-controller SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-mvtk7 ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/healthz,Port:{1 0 healthz},Host:,Scheme:HTTP,HTTPHeaders:[]HTTPHeader{},},TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:30,TimeoutSeconds:10,PeriodSeconds:30,SuccessThreshold:1,FailureThreshold:5,TerminationGracePeriodSeconds:nil,} ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:Always SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy:Always TerminationGracePeriodSeconds:0xc0010811f8 ActiveDeadlineSeconds:<nil> DNSPolicy:ClusterFirst NodeSelector:map[kubernetes.io/os:linux] ServiceAccountName:csi-azuredisk-controller-sa DeprecatedServiceAccount:csi-azuredisk-controller-sa AutomountServiceAccountToken:<nil> NodeName:k8s-agentpool1-64138072-vmss000000 HostNetwork:true HostPID:false HostIPC:false ShareProcessNamespace:<nil> SecurityContext:&PodSecurityContext{SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,SupplementalGroups:[],FSGroup:nil,RunAsGroup:nil,Sysctls:[]Sysctl{},WindowsOptions:nil,FSGroupChangePolicy:nil,SeccompProfile:nil,} ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName:default-scheduler Tolerations:[{Key:node-role.kubernetes.io/master Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>} {Key:node-role.kubernetes.io/controlplane Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>} {Key:node-role.kubernetes.io/control-plane Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>} {Key:node.kubernetes.io/not-ready Operator:Exists Value: Effect:NoExecute TolerationSeconds:0xc001081200} {Key:node.kubernetes.io/unreachable Operator:Exists Value: Effect:NoExecute TolerationSeconds:0xc001081208}] HostAliases:[] PriorityClassName:system-cluster-critical Priority:0xc001081210 DNSConfig:nil ReadinessGates:[] RuntimeClassName:<nil> EnableServiceLinks:0xc001081214 PreemptionPolicy:0xc0005f74c0 Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN:<nil> OS:nil HostUsers:<nil> SchedulingGates:[] ResourceClaims:[]}. With volumes: [{Name:socket-dir VolumeSource:{HostPath:nil EmptyDir:&EmptyDirVolumeSource{Medium:,SizeLimit:<nil>,} GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:azure-cred VolumeSource:{HostPath:&HostPathVolumeSource{Path:/etc/kubernetes/,Type:*DirectoryOrCreate,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:csi-test-controller-config VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:&ConfigMapVolumeSource{LocalObjectReference:LocalObjectReference{Name:csi-test-controller-config,},Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,} VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:kube-api-access-mvtk7 VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:&ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,} PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}]" I0201 17:21:39.053877 1 shared_state.go:463] "msg"="Pod csi-test-controller-5c87f9dff6-n6xg5: Skipping Volume {socket-dir {nil &EmptyDirVolumeSource{Medium:,SizeLimit:<nil>,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}}. No persistent volume exists." I0201 17:21:39.053896 1 shared_state.go:463] "msg"="Pod csi-test-controller-5c87f9dff6-n6xg5: Skipping Volume {azure-cred {&HostPathVolumeSource{Path:/etc/kubernetes/,Type:*DirectoryOrCreate,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}}. No persistent volume exists." I0201 17:21:39.053925 1 shared_state.go:463] "msg"="Pod csi-test-controller-5c87f9dff6-n6xg5: Skipping Volume {csi-test-controller-config {nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil &ConfigMapVolumeSource{LocalObjectReference:LocalObjectReference{Name:csi-test-controller-config,},Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,} nil nil nil nil nil nil nil nil nil nil}}. No persistent volume exists." I0201 17:21:39.053963 1 shared_state.go:463] "msg"="Pod csi-test-controller-5c87f9dff6-n6xg5: Skipping Volume {kube-api-access-mvtk7 {nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil &ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,} nil nil nil nil nil}}. No persistent volume exists." I0201 17:21:39.053978 1 shared_state.go:489] "msg"="Storing pod csi-test-controller-5c87f9dff6-n6xg5 and claim [] to podToClaimsMap map." I0201 17:21:39.054007 1 pod.go:91] "msg"="Creating replicas for pod kube-system/csi-test-controller-5c87f9dff6-n6xg5." "disk.csi.azure.com/request-id"="e32ad065-a254-11ed-abc7-6045bd9323b2" "disk.csi/azure.com/pod-name"="kube-system/csi-test-controller-5c87f9dff6-n6xg5" ... skipping 146 lines ... I0201 17:21:40.212931 1 shared_state.go:489] "msg"="Storing pod azure-ip-masq-agent-7nfkx and claim [] to podToClaimsMap map." I0201 17:21:40.212957 1 pod.go:91] "msg"="Creating replicas for pod kube-system/azure-ip-masq-agent-7nfkx." "disk.csi.azure.com/request-id"="e3dba807-a254-11ed-abc7-6045bd9323b2" "disk.csi/azure.com/pod-name"="kube-system/azure-ip-masq-agent-7nfkx" I0201 17:21:40.212975 1 shared_state.go:313] "msg"="Getting requested volumes for pod (kube-system/azure-ip-masq-agent-7nfkx)." "disk.csi.azure.com/request-id"="e3dba807-a254-11ed-abc7-6045bd9323b2" "disk.csi/azure.com/pod-name"="kube-system/azure-ip-masq-agent-7nfkx" I0201 17:21:40.212993 1 pod.go:99] "msg"="Pod kube-system/azure-ip-masq-agent-7nfkx has 0 volumes. Volumes: []" "disk.csi.azure.com/request-id"="e3dba807-a254-11ed-abc7-6045bd9323b2" "disk.csi/azure.com/pod-name"="kube-system/azure-ip-masq-agent-7nfkx" I0201 17:21:40.213022 1 pod.go:89] "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="e3dba807-a254-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcilePod).createReplicas" "disk.csi/azure.com/pod-name"="kube-system/azure-ip-masq-agent-7nfkx" "latency"=67001 I0201 17:21:40.311876 1 shared_state.go:419] "msg"="Adding pod csi-test-controller-5c87f9dff6-n6xg5 to shared map with keyName kube-system/csi-test-controller-5c87f9dff6-n6xg5." I0201 17:21:40.312323 1 shared_state.go:425] "msg"="Pod spec of pod csi-test-controller-5c87f9dff6-n6xg5 is: {Volumes:[{Name:socket-dir VolumeSource:{HostPath:nil EmptyDir:&EmptyDirVolumeSource{Medium:,SizeLimit:<nil>,} GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:azure-cred VolumeSource:{HostPath:&HostPathVolumeSource{Path:/etc/kubernetes/,Type:*DirectoryOrCreate,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:csi-test-controller-config VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:&ConfigMapVolumeSource{LocalObjectReference:LocalObjectReference{Name:csi-test-controller-config,},Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,} VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:kube-api-access-mvtk7 VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:&ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,} PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:csi-provisioner Image:mcr.microsoft.com/oss/kubernetes-csi/csi-provisioner:v3.2.0 Command:[] Args:[--feature-gates=Topology=true --csi-address=$(ADDRESS) --v=2 --timeout=30s --leader-election --leader-election-namespace=kube-system --worker-threads=100 --extra-create-metadata=true --strict-topology=true --kube-api-qps=50 --kube-api-burst=100] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:ADDRESS Value:/csi/csi.sock ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:524288000 scale:0} d:{Dec:<nil>} s:500Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}] Claims:[]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-mvtk7 ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:Always SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:csi-attacher Image:mcr.microsoft.com/oss/kubernetes-csi/csi-attacher:v3.5.0 Command:[] Args:[-v=2 -csi-address=$(ADDRESS) -timeout=600s -leader-election --leader-election-namespace=kube-system -worker-threads=500 -kube-api-qps=50 -kube-api-burst=100] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:ADDRESS Value:/csi/csi.sock ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:524288000 scale:0} d:{Dec:<nil>} s:500Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}] Claims:[]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-mvtk7 ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:Always SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:csi-snapshotter Image:mcr.microsoft.com/oss/kubernetes-csi/csi-snapshotter:v5.0.1 Command:[] Args:[-csi-address=$(ADDRESS) -leader-election --leader-election-namespace=kube-system -v=2] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:ADDRESS Value:/csi/csi.sock ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:104857600 scale:0} d:{Dec:<nil>} s:100Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}] Claims:[]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-mvtk7 ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:Always SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:csi-resizer Image:mcr.microsoft.com/oss/kubernetes-csi/csi-resizer:v1.5.0 Command:[] Args:[-csi-address=$(ADDRESS) -v=2 -leader-election --leader-election-namespace=kube-system -handle-volume-inuse-error=false -feature-gates=RecoverVolumeExpansionFailure=true -timeout=240s] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:ADDRESS Value:/csi/csi.sock ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:524288000 scale:0} d:{Dec:<nil>} s:500Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}] Claims:[]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-mvtk7 ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:Always SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:liveness-probe Image:mcr.microsoft.com/oss/kubernetes-csi/livenessprobe:v2.7.0 Command:[] Args:[--csi-address=/csi/csi.sock --probe-timeout=3s --health-port=29602 --v=2] WorkingDir: Ports:[] EnvFrom:[] Env:[] Resources:{Limits:map[memory:{i:{value:104857600 scale:0} d:{Dec:<nil>} s:100Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}] Claims:[]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-mvtk7 ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:Always SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:azuredisk Image:k8sprow.azurecr.io/azuredisk-csi:latest-v2-8ab762a4766ca4352f3bbe3e0c05b326019e12ba Command:[] Args:[--v=5 --config=/etc/csi-test-controller/config.yaml] WorkingDir: Ports:[{Name:healthz HostPort:29602 ContainerPort:29602 Protocol:TCP HostIP:} {Name:metrics HostPort:29604 ContainerPort:29604 Protocol:TCP HostIP:}] EnvFrom:[] Env:[{Name:AZURE_CREDENTIAL_FILE Value: ValueFrom:&EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:&ConfigMapKeySelector{LocalObjectReference:LocalObjectReference{Name:azure-cred-file,},Key:path,Optional:*true,},SecretKeyRef:nil,}} {Name:CSI_ENDPOINT Value:unix:///csi/csi.sock ValueFrom:nil} {Name:AZURE_GO_SDK_LOG_LEVEL Value: ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:524288000 scale:0} d:{Dec:<nil>} s:500Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}] Claims:[]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:azure-cred ReadOnly:false MountPath:/etc/kubernetes/ SubPath: MountPropagation:<nil> SubPathExpr:} {Name:csi-test-controller-config ReadOnly:false MountPath:/etc/csi-test-controller SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-mvtk7 ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/healthz,Port:{1 0 healthz},Host:,Scheme:HTTP,HTTPHeaders:[]HTTPHeader{},},TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:30,TimeoutSeconds:10,PeriodSeconds:30,SuccessThreshold:1,FailureThreshold:5,TerminationGracePeriodSeconds:nil,} ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:Always SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy:Always TerminationGracePeriodSeconds:0xc001265e58 ActiveDeadlineSeconds:<nil> DNSPolicy:ClusterFirst NodeSelector:map[kubernetes.io/os:linux] ServiceAccountName:csi-azuredisk-controller-sa DeprecatedServiceAccount:csi-azuredisk-controller-sa AutomountServiceAccountToken:<nil> NodeName:k8s-agentpool1-64138072-vmss000000 HostNetwork:true HostPID:false HostIPC:false ShareProcessNamespace:<nil> SecurityContext:&PodSecurityContext{SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,SupplementalGroups:[],FSGroup:nil,RunAsGroup:nil,Sysctls:[]Sysctl{},WindowsOptions:nil,FSGroupChangePolicy:nil,SeccompProfile:nil,} ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName:default-scheduler Tolerations:[{Key:node-role.kubernetes.io/master Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>} {Key:node-role.kubernetes.io/controlplane Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>} {Key:node-role.kubernetes.io/control-plane Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>} {Key:node.kubernetes.io/not-ready Operator:Exists Value: Effect:NoExecute TolerationSeconds:0xc001265e60} {Key:node.kubernetes.io/unreachable Operator:Exists Value: Effect:NoExecute TolerationSeconds:0xc001265e68}] HostAliases:[] PriorityClassName:system-cluster-critical Priority:0xc001265e70 DNSConfig:nil ReadinessGates:[] RuntimeClassName:<nil> EnableServiceLinks:0xc001265e74 PreemptionPolicy:0xc00091ba10 Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN:<nil> OS:nil HostUsers:<nil> SchedulingGates:[] ResourceClaims:[]}. With volumes: [{Name:socket-dir VolumeSource:{HostPath:nil EmptyDir:&EmptyDirVolumeSource{Medium:,SizeLimit:<nil>,} GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:azure-cred VolumeSource:{HostPath:&HostPathVolumeSource{Path:/etc/kubernetes/,Type:*DirectoryOrCreate,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:csi-test-controller-config VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:&ConfigMapVolumeSource{LocalObjectReference:LocalObjectReference{Name:csi-test-controller-config,},Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,} VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:kube-api-access-mvtk7 VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:&ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,} PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}]" I0201 17:21:40.312377 1 shared_state.go:463] "msg"="Pod csi-test-controller-5c87f9dff6-n6xg5: Skipping Volume {socket-dir {nil &EmptyDirVolumeSource{Medium:,SizeLimit:<nil>,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}}. No persistent volume exists." I0201 17:21:40.312398 1 shared_state.go:463] "msg"="Pod csi-test-controller-5c87f9dff6-n6xg5: Skipping Volume {azure-cred {&HostPathVolumeSource{Path:/etc/kubernetes/,Type:*DirectoryOrCreate,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}}. No persistent volume exists." I0201 17:21:40.312426 1 shared_state.go:463] "msg"="Pod csi-test-controller-5c87f9dff6-n6xg5: Skipping Volume {csi-test-controller-config {nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil &ConfigMapVolumeSource{LocalObjectReference:LocalObjectReference{Name:csi-test-controller-config,},Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,} nil nil nil nil nil nil nil nil nil nil}}. No persistent volume exists." I0201 17:21:40.312464 1 shared_state.go:463] "msg"="Pod csi-test-controller-5c87f9dff6-n6xg5: Skipping Volume {kube-api-access-mvtk7 {nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil &ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,} nil nil nil nil nil}}. No persistent volume exists." I0201 17:21:40.312477 1 shared_state.go:489] "msg"="Storing pod csi-test-controller-5c87f9dff6-n6xg5 and claim [] to podToClaimsMap map." I0201 17:21:40.312506 1 pod.go:91] "msg"="Creating replicas for pod kube-system/csi-test-controller-5c87f9dff6-n6xg5." "disk.csi.azure.com/request-id"="e3ead886-a254-11ed-abc7-6045bd9323b2" "disk.csi/azure.com/pod-name"="kube-system/csi-test-controller-5c87f9dff6-n6xg5" ... skipping 123 lines ... I0201 17:21:46.049222 1 shared_state.go:489] "msg"="Storing pod deployment-azuredisk-69468c8c97-2h9fp and claim [] to podToClaimsMap map." I0201 17:21:46.071181 1 utils.go:78] GRPC call: /csi.v1.Controller/CreateVolume I0201 17:21:46.071204 1 utils.go:79] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"requisite":[{"segments":{"topology.test.csi.azure.com/zone":""}}]},"capacity_range":{"required_bytes":10737418240},"name":"pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05","parameters":{"csi.storage.k8s.io/pv/name":"pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05","csi.storage.k8s.io/pvc/name":"pvc-azuredisk","csi.storage.k8s.io/pvc/namespace":"default","skuName":"StandardSSD_LRS"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]} I0201 17:21:46.071499 1 crdprovisioner.go:308] "msg"="Creating AzVolume CRI" "csi.storage.k8s.io/pv/name"="pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05" "disk.csi.azure.com/request-id"="e7599845-a254-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/volume-name"="pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05" I0201 17:21:46.086380 1 crdprovisioner.go:316] "msg"="Successfully created AzVolume CRI" "csi.storage.k8s.io/pv/name"="pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05" "disk.csi.azure.com/request-id"="e7599845-a254-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/volume-name"="pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05" I0201 17:21:46.086406 1 conditionwatcher.go:99] Adding a condition function for azvolume (pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05) I0201 17:21:46.090602 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0201 17:21:46.091839 1 azure_disk_utils.go:1103] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate" "csi.storage.k8s.io/pv/name"="pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05" "disk.csi.azure.com/request-id"="e7599845-a254-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05" "latency"=10589813 "number_of_net_retry"=0 "number_of_retry"=0 I0201 17:21:46.091864 1 azvolume.go:158] "msg"="Creating Volume..." "csi.storage.k8s.io/pv/name"="pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05" "disk.csi.azure.com/request-id"="e7599845-a254-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/volume-name"="pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05" I0201 17:21:46.179973 1 util.go:124] Send.sendRequest got response with ContentLength 253, StatusCode 404 and responseBody length 253 I0201 17:21:46.180002 1 azure_diskclient.go:139] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-96oxu0dv/providers/Microsoft.Compute/disks/pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05' under resource group 'kubetest-96oxu0dv' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}} I0201 17:21:46.180113 1 cloudprovisioner.go:246] begin to create disk(pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05) account type(StandardSSD_LRS) rg(kubetest-96oxu0dv) location() size(10) selectedAvailabilityZone() maxShares(0) I0201 17:21:46.213580 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 3502 I0201 17:21:46.213776 1 azure_managedDiskController.go:94] azureDisk - creating new managed Name:pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05 StorageAccountType:StandardSSD_LRS Size:10 I0201 17:21:46.472101 1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller I0201 17:21:48.479079 1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller I0201 17:21:48.551195 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1193 I0201 17:21:48.551380 1 azure_managedDiskController.go:280] azureDisk - created new MD Name:pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05 StorageAccountType:StandardSSD_LRS Size:10 I0201 17:21:48.551433 1 cloudprovisioner.go:320] "msg"="create disk(pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05) account type(StandardSSD_LRS) rg(kubetest-96oxu0dv) location() size(10) tags(map[kubernetes.io-created-for-pv-name:pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05 kubernetes.io-created-for-pvc-name:pvc-azuredisk kubernetes.io-created-for-pvc-namespace:default]) successfully" "csi.storage.k8s.io/pv/name"="pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05" "disk.csi.azure.com/request-id"="e7599845-a254-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/volume-name"="pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05" I0201 17:21:48.551470 1 cloudprovisioner.go:142] "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05" "disk.csi.azure.com/request-id"="e7599845-a254-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05" "latency"=2459538391 I0201 17:21:48.562610 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0201 17:21:48.562661 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).CreateVolume" "csi.storage.k8s.io/pv/name"="pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05" "disk.csi.azure.com/request-id"="e7599845-a254-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05" "latency"=2476189026 I0201 17:21:48.562686 1 crdprovisioner.go:228] "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05" "disk.csi.azure.com/request-id"="e7599845-a254-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05" "latency"=2491214928 I0201 17:21:48.562736 1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=2.491262229 request="azuredisk_csi_driver_controller_create_volume" resource_group="kubetest-96oxu0dv" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="test.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-96oxu0dv/providers/Microsoft.Compute/disks/pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05" result_code="succeeded" I0201 17:21:48.562755 1 utils.go:85] GRPC response: {"volume":{"accessible_topology":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"capacity_bytes":10737418240,"content_source":{"Type":{"Volume":{}}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05","csi.storage.k8s.io/pvc/name":"pvc-azuredisk","csi.storage.k8s.io/pvc/namespace":"default","requestedsizegib":"10","skuName":"StandardSSD_LRS"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-96oxu0dv/providers/Microsoft.Compute/disks/pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05"}} I0201 17:21:48.563844 1 azure_disk_utils.go:1103] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "csi.storage.k8s.io/pv/name"="pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05" "disk.csi.azure.com/request-id"="e7599845-a254-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05" "latency"=12318648 "number_of_net_retry"=0 "number_of_retry"=0 I0201 17:21:48.563892 1 azvolume.go:165] "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05" "disk.csi.azure.com/request-id"="e7599845-a254-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "disk.csi.azure.com/volume-name"="pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05" "latency"=2471984841 I0201 17:21:48.563921 1 workflow.go:155] "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05" "disk.csi.azure.com/request-id"="e7599845-a254-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate" "disk.csi.azure.com/volume-name"="pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05" "latency"=2482683756 I0201 17:21:49.140358 1 utils.go:78] GRPC call: /csi.v1.Controller/ControllerPublishVolume I0201 17:21:49.140379 1 utils.go:79] GRPC request: {"node_id":"k8s-agentpool1-64138072-vmss000001","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05","csi.storage.k8s.io/pvc/name":"pvc-azuredisk","csi.storage.k8s.io/pvc/namespace":"default","requestedsizegib":"10","skuName":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1675272099190-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-96oxu0dv/providers/Microsoft.Compute/disks/pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05"} I0201 17:21:49.140677 1 azure_disk_utils.go:1068] "msg"="Label selector is: disk.csi.azure.com/node-name=k8s-agentpool1-64138072-vmss000001,disk.csi.azure.com/volume-name!=pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05." "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000001" "disk.csi.azure.com/request-id"="e92de632-a254-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05" I0201 17:21:49.145575 1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05-k8s-agentpool1-64138072-vmss000001-attachment) I0201 17:21:49.150670 1 azure_disk_utils.go:1103] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000001" "disk.csi.azure.com/request-id"="e92de632-a254-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05" "latency"=5055909 "number_of_net_retry"=0 "number_of_retry"=0 I0201 17:21:49.150715 1 attach_detach.go:192] "msg"="Attaching volume" "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000001" "disk.csi.azure.com/request-id"="e92de632-a254-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05" I0201 17:21:49.151157 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0201 17:21:49.192485 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1193 I0201 17:21:49.230666 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 4987 I0201 17:21:49.340261 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 29043 I0201 17:21:49.342304 1 cloudprovisioner.go:405] "msg"="GetDiskLun returned: -1. Initiating attaching volume \"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-96oxu0dv/providers/Microsoft.Compute/disks/pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05\" to node \"k8s-agentpool1-64138072-vmss000001\"." "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000001" "disk.csi.azure.com/request-id"="e92de632-a254-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05" I0201 17:21:49.342343 1 cloudprovisioner.go:424] "msg"="Trying to attach volume \"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-96oxu0dv/providers/Microsoft.Compute/disks/pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05\" to node \"k8s-agentpool1-64138072-vmss000001\"." "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000001" "disk.csi.azure.com/request-id"="e92de632-a254-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05" I0201 17:21:50.342535 1 batch.go:224] "cloud-provider-azure: Delayed processing of batch due to start delay" type="batch" operation="attach_disk" key="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e|kubetest-96oxu0dv|k8s-agentpool1-64138072-vmss000001" delay="1s" I0201 17:21:50.342655 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-96oxu0dv): vm(k8s-agentpool1-64138072-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-96oxu0dv/providers/Microsoft.Compute/disks/pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05:AttachDiskOptions{diskName: "pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05", lun: 0}]) I0201 17:21:50.351544 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0201 17:21:50.351621 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000001" "disk.csi.azure.com/request-id"="e92de632-a254-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05" "latency"=1205969648 I0201 17:21:50.351665 1 crdprovisioner.go:727] "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000001" "disk.csi.azure.com/request-id"="e92de632-a254-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLun" "disk.csi.azure.com/volume-name"="pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05" "latency"=1206107351 I0201 17:21:50.351695 1 crdprovisioner.go:486] "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000001" "disk.csi.azure.com/request-id"="e92de632-a254-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).PublishVolume" "disk.csi.azure.com/volume-name"="pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05" "latency"=1211139860 I0201 17:21:50.351775 1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=1.211233661 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-96oxu0dv" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="test.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-96oxu0dv/providers/Microsoft.Compute/disks/pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05" node="k8s-agentpool1-64138072-vmss000001" result_code="succeeded" I0201 17:21:50.351791 1 utils.go:85] GRPC response: {"publish_context":{"LUN":"0"}} I0201 17:21:50.351909 1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05-k8s-agentpool1-64138072-vmss000001-attachment) I0201 17:21:50.353007 1 azure_disk_utils.go:1103] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3.4" "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000001" "disk.csi.azure.com/request-id"="e92de632-a254-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05" "latency"=10134218 "number_of_net_retry"=0 "number_of_retry"=0 I0201 17:21:50.486120 1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller I0201 17:21:51.069908 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-96oxu0dv): vm(k8s-agentpool1-64138072-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-96oxu0dv/providers/Microsoft.Compute/disks/pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05:AttachDiskOptions{diskName: "pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05", lun: 0}], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I0201 17:21:51.069962 1 azure_controller_common.go:344] azuredisk - trying to attach disks to node k8s-agentpool1-64138072-vmss000001: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-96oxu0dv/providers/Microsoft.Compute/disks/pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05:AttachDiskOptions{diskName: "pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05", lun: 0}] I0201 17:21:52.496278 1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller I0201 17:21:54.503742 1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller I0201 17:21:56.510083 1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller I0201 17:21:58.517609 1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller I0201 17:22:00.525782 1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller I0201 17:22:01.163392 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-96oxu0dv, k8s-agentpool1-64138072-vmss, k8s-agentpool1-64138072-vmss000001) successfully I0201 17:22:01.163439 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool1-64138072-vmss, kubetest-96oxu0dv, k8s-agentpool1-64138072-vmss000001) for cacheKey(kubetest-96oxu0dv/k8s-agentpool1-64138072-vmss) updated successfully I0201 17:22:01.163525 1 cloudprovisioner.go:448] "msg"="attach operation successful: volume \"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-96oxu0dv/providers/Microsoft.Compute/disks/pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05\" attached to node \"k8s-agentpool1-64138072-vmss000001\"." "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000001" "disk.csi.azure.com/request-id"="e92de632-a254-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05" I0201 17:22:01.163578 1 cloudprovisioner.go:440] "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000001" "disk.csi.azure.com/request-id"="e92de632-a254-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).PublishVolume.func3" "disk.csi.azure.com/volume-name"="pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05" "latency"=11821162346 I0201 17:22:01.163614 1 workflow.go:155] "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000001" "disk.csi.azure.com/request-id"="e92de632-a254-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).PublishVolume" "disk.csi.azure.com/volume-name"="pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05" "latency"=12012816569 I0201 17:22:01.163751 1 azure_disk_utils.go:1165] "msg"="Skip update. No update needed." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).updateVolumeAttachmentWithResult" "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000001" "disk.csi.azure.com/request-id"="e92de632-a254-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05" I0201 17:22:01.163805 1 azure_disk_utils.go:1103] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).updateVolumeAttachmentWithResult" "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000001" "disk.csi.azure.com/request-id"="e92de632-a254-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05" "latency"=133003 "number_of_net_retry"=0 "number_of_retry"=0 I0201 17:22:01.178649 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0201 17:22:01.178678 1 azure_disk_utils.go:1103] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3.4" "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000001" "disk.csi.azure.com/request-id"="e92de632-a254-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05" "latency"=14818018 "number_of_net_retry"=0 "number_of_retry"=0 I0201 17:22:01.178700 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000001" "disk.csi.azure.com/request-id"="e9e6bac6-a254-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05" "latency"=10826723648 I0201 17:22:01.178716 1 attach_detach.go:198] "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000001" "disk.csi.azure.com/request-id"="e92de632-a254-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3" "disk.csi.azure.com/volume-name"="pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05" "latency"=12027953895 I0201 17:22:01.178721 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).ControllerPublishVolume.func2" "disk.csi.azure.com/request-id"="e9e6bac6-a254-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=10826860652 I0201 17:22:01.178746 1 workflow.go:155] "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000001" "disk.csi.azure.com/request-id"="e92de632-a254-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach" "disk.csi.azure.com/volume-name"="pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05" "latency"=12033165107 I0201 17:22:01.178762 1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=12.038225015 request="azuredisk_csi_driver_attach_volume_latency" resource_group="kubetest-96oxu0dv" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="test.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-96oxu0dv/providers/Microsoft.Compute/disks/pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05" node="k8s-agentpool1-64138072-vmss000001" result_code="succeeded" ... skipping 23 lines ... I0201 17:22:07.066826 1 shared_state.go:489] "msg"="Storing pod statefulset-azuredisk-0 and claim [] to podToClaimsMap map." I0201 17:22:07.081833 1 utils.go:78] GRPC call: /csi.v1.Controller/CreateVolume I0201 17:22:07.081857 1 utils.go:79] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"requisite":[{"segments":{"topology.test.csi.azure.com/zone":""}}]},"capacity_range":{"required_bytes":10737418240},"name":"pvc-553191ed-6cf7-4436-a08d-abb08cda4724","parameters":{"csi.storage.k8s.io/pv/name":"pvc-553191ed-6cf7-4436-a08d-abb08cda4724","csi.storage.k8s.io/pvc/name":"persistent-storage-statefulset-azuredisk-0","csi.storage.k8s.io/pvc/namespace":"default","skuName":"StandardSSD_LRS"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]} I0201 17:22:07.082094 1 crdprovisioner.go:308] "msg"="Creating AzVolume CRI" "csi.storage.k8s.io/pv/name"="pvc-553191ed-6cf7-4436-a08d-abb08cda4724" "disk.csi.azure.com/request-id"="f3df8e7b-a254-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/volume-name"="pvc-553191ed-6cf7-4436-a08d-abb08cda4724" I0201 17:22:07.091897 1 crdprovisioner.go:316] "msg"="Successfully created AzVolume CRI" "csi.storage.k8s.io/pv/name"="pvc-553191ed-6cf7-4436-a08d-abb08cda4724" "disk.csi.azure.com/request-id"="f3df8e7b-a254-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/volume-name"="pvc-553191ed-6cf7-4436-a08d-abb08cda4724" I0201 17:22:07.091921 1 conditionwatcher.go:99] Adding a condition function for azvolume (pvc-553191ed-6cf7-4436-a08d-abb08cda4724) I0201 17:22:07.092290 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0201 17:22:07.097705 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0201 17:22:07.098552 1 azure_disk_utils.go:1103] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate" "csi.storage.k8s.io/pv/name"="pvc-553191ed-6cf7-4436-a08d-abb08cda4724" "disk.csi.azure.com/request-id"="f3df8e7b-a254-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-553191ed-6cf7-4436-a08d-abb08cda4724" "latency"=6074030 "number_of_net_retry"=0 "number_of_retry"=0 I0201 17:22:07.098580 1 azvolume.go:158] "msg"="Creating Volume..." "csi.storage.k8s.io/pv/name"="pvc-553191ed-6cf7-4436-a08d-abb08cda4724" "disk.csi.azure.com/request-id"="f3df8e7b-a254-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/volume-name"="pvc-553191ed-6cf7-4436-a08d-abb08cda4724" I0201 17:22:07.114198 1 util.go:124] Send.sendRequest got response with ContentLength 253, StatusCode 404 and responseBody length 253 I0201 17:22:07.114223 1 azure_diskclient.go:139] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-96oxu0dv/providers/Microsoft.Compute/disks/pvc-553191ed-6cf7-4436-a08d-abb08cda4724, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-553191ed-6cf7-4436-a08d-abb08cda4724' under resource group 'kubetest-96oxu0dv' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}} I0201 17:22:07.114269 1 cloudprovisioner.go:246] begin to create disk(pvc-553191ed-6cf7-4436-a08d-abb08cda4724) account type(StandardSSD_LRS) rg(kubetest-96oxu0dv) location() size(10) selectedAvailabilityZone() maxShares(0) I0201 17:22:07.147826 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 5068 I0201 17:22:07.147985 1 azure_managedDiskController.go:94] azureDisk - creating new managed Name:pvc-553191ed-6cf7-4436-a08d-abb08cda4724 StorageAccountType:StandardSSD_LRS Size:10 I0201 17:22:08.332758 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0201 17:22:08.332809 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0201 17:22:08.332831 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0201 17:22:08.332865 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0201 17:22:08.337396 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0201 17:22:08.560388 1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller I0201 17:22:09.411400 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1222 I0201 17:22:09.411640 1 azure_managedDiskController.go:280] azureDisk - created new MD Name:pvc-553191ed-6cf7-4436-a08d-abb08cda4724 StorageAccountType:StandardSSD_LRS Size:10 I0201 17:22:09.411701 1 cloudprovisioner.go:320] "msg"="create disk(pvc-553191ed-6cf7-4436-a08d-abb08cda4724) account type(StandardSSD_LRS) rg(kubetest-96oxu0dv) location() size(10) tags(map[kubernetes.io-created-for-pv-name:pvc-553191ed-6cf7-4436-a08d-abb08cda4724 kubernetes.io-created-for-pvc-name:persistent-storage-statefulset-azuredisk-0 kubernetes.io-created-for-pvc-namespace:default]) successfully" "csi.storage.k8s.io/pv/name"="pvc-553191ed-6cf7-4436-a08d-abb08cda4724" "disk.csi.azure.com/request-id"="f3df8e7b-a254-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/volume-name"="pvc-553191ed-6cf7-4436-a08d-abb08cda4724" I0201 17:22:09.411751 1 cloudprovisioner.go:142] "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-553191ed-6cf7-4436-a08d-abb08cda4724" "disk.csi.azure.com/request-id"="f3df8e7b-a254-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-553191ed-6cf7-4436-a08d-abb08cda4724" "latency"=2313089744 I0201 17:22:09.421873 1 azure_disk_utils.go:1103] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "csi.storage.k8s.io/pv/name"="pvc-553191ed-6cf7-4436-a08d-abb08cda4724" "disk.csi.azure.com/request-id"="f3df8e7b-a254-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-553191ed-6cf7-4436-a08d-abb08cda4724" "latency"=10063814 "number_of_net_retry"=0 "number_of_retry"=0 I0201 17:22:09.421909 1 azvolume.go:165] "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-553191ed-6cf7-4436-a08d-abb08cda4724" "disk.csi.azure.com/request-id"="f3df8e7b-a254-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "disk.csi.azure.com/volume-name"="pvc-553191ed-6cf7-4436-a08d-abb08cda4724" "latency"=2323288861 I0201 17:22:09.421928 1 workflow.go:155] "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-553191ed-6cf7-4436-a08d-abb08cda4724" "disk.csi.azure.com/request-id"="f3df8e7b-a254-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate" "disk.csi.azure.com/volume-name"="pvc-553191ed-6cf7-4436-a08d-abb08cda4724" "latency"=2329467694 I0201 17:22:09.422776 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0201 17:22:09.422831 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).CreateVolume" "csi.storage.k8s.io/pv/name"="pvc-553191ed-6cf7-4436-a08d-abb08cda4724" "disk.csi.azure.com/request-id"="f3df8e7b-a254-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-553191ed-6cf7-4436-a08d-abb08cda4724" "latency"=2330847223 I0201 17:22:09.422860 1 crdprovisioner.go:228] "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-553191ed-6cf7-4436-a08d-abb08cda4724" "disk.csi.azure.com/request-id"="f3df8e7b-a254-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-553191ed-6cf7-4436-a08d-abb08cda4724" "latency"=2340798536 I0201 17:22:09.422895 1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=2.340850036 request="azuredisk_csi_driver_controller_create_volume" resource_group="kubetest-96oxu0dv" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="test.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-96oxu0dv/providers/Microsoft.Compute/disks/pvc-553191ed-6cf7-4436-a08d-abb08cda4724" result_code="succeeded" I0201 17:22:09.422915 1 utils.go:85] GRPC response: {"volume":{"accessible_topology":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"capacity_bytes":10737418240,"content_source":{"Type":{"Volume":{}}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-553191ed-6cf7-4436-a08d-abb08cda4724","csi.storage.k8s.io/pvc/name":"persistent-storage-statefulset-azuredisk-0","csi.storage.k8s.io/pvc/namespace":"default","requestedsizegib":"10","skuName":"StandardSSD_LRS"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-96oxu0dv/providers/Microsoft.Compute/disks/pvc-553191ed-6cf7-4436-a08d-abb08cda4724"}} I0201 17:22:10.159012 1 utils.go:78] GRPC call: /csi.v1.Controller/ControllerPublishVolume I0201 17:22:10.159031 1 utils.go:79] GRPC request: {"node_id":"k8s-agentpool1-64138072-vmss000002","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-553191ed-6cf7-4436-a08d-abb08cda4724","csi.storage.k8s.io/pvc/name":"persistent-storage-statefulset-azuredisk-0","csi.storage.k8s.io/pvc/namespace":"default","requestedsizegib":"10","skuName":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1675272099190-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-96oxu0dv/providers/Microsoft.Compute/disks/pvc-553191ed-6cf7-4436-a08d-abb08cda4724"} I0201 17:22:10.159277 1 azure_disk_utils.go:1068] "msg"="Label selector is: disk.csi.azure.com/node-name=k8s-agentpool1-64138072-vmss000002,disk.csi.azure.com/volume-name!=pvc-553191ed-6cf7-4436-a08d-abb08cda4724." "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000002" "disk.csi.azure.com/request-id"="f5b51644-a254-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-553191ed-6cf7-4436-a08d-abb08cda4724" I0201 17:22:10.163850 1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-553191ed-6cf7-4436-a08d-abb08cda4724-k8s-agentpool1-64138072-vmss000002-attachment) I0201 17:22:10.164221 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0201 17:22:10.169813 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0201 17:22:10.170084 1 azure_disk_utils.go:1103] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000002" "disk.csi.azure.com/request-id"="f5b51644-a254-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-553191ed-6cf7-4436-a08d-abb08cda4724" "latency"=5561219 "number_of_net_retry"=0 "number_of_retry"=0 I0201 17:22:10.170109 1 attach_detach.go:192] "msg"="Attaching volume" "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000002" "disk.csi.azure.com/request-id"="f5b51644-a254-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-553191ed-6cf7-4436-a08d-abb08cda4724" I0201 17:22:10.192680 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1222 I0201 17:22:10.192912 1 cloudprovisioner.go:405] "msg"="GetDiskLun returned: -1. Initiating attaching volume \"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-96oxu0dv/providers/Microsoft.Compute/disks/pvc-553191ed-6cf7-4436-a08d-abb08cda4724\" to node \"k8s-agentpool1-64138072-vmss000002\"." "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000002" "disk.csi.azure.com/request-id"="f5b51644-a254-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-553191ed-6cf7-4436-a08d-abb08cda4724" I0201 17:22:10.192957 1 cloudprovisioner.go:424] "msg"="Trying to attach volume \"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-96oxu0dv/providers/Microsoft.Compute/disks/pvc-553191ed-6cf7-4436-a08d-abb08cda4724\" to node \"k8s-agentpool1-64138072-vmss000002\"." "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000002" "disk.csi.azure.com/request-id"="f5b51644-a254-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-553191ed-6cf7-4436-a08d-abb08cda4724" I0201 17:22:10.567782 1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller I0201 17:22:11.193716 1 batch.go:224] "cloud-provider-azure: Delayed processing of batch due to start delay" type="batch" operation="attach_disk" key="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e|kubetest-96oxu0dv|k8s-agentpool1-64138072-vmss000002" delay="1s" I0201 17:22:11.193847 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-96oxu0dv): vm(k8s-agentpool1-64138072-vmss000002) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-96oxu0dv/providers/Microsoft.Compute/disks/pvc-553191ed-6cf7-4436-a08d-abb08cda4724:AttachDiskOptions{diskName: "pvc-553191ed-6cf7-4436-a08d-abb08cda4724", lun: 0}]) I0201 17:22:11.202742 1 azure_disk_utils.go:1103] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3.4" "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000002" "disk.csi.azure.com/request-id"="f5b51644-a254-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-553191ed-6cf7-4436-a08d-abb08cda4724" "latency"=8781987 "number_of_net_retry"=0 "number_of_retry"=0 I0201 17:22:11.203243 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0201 17:22:11.203307 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000002" "disk.csi.azure.com/request-id"="f5b51644-a254-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-553191ed-6cf7-4436-a08d-abb08cda4724" "latency"=1039342572 I0201 17:22:11.203349 1 crdprovisioner.go:727] "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000002" "disk.csi.azure.com/request-id"="f5b51644-a254-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLun" "disk.csi.azure.com/volume-name"="pvc-553191ed-6cf7-4436-a08d-abb08cda4724" "latency"=1039505675 I0201 17:22:11.203389 1 crdprovisioner.go:486] "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000002" "disk.csi.azure.com/request-id"="f5b51644-a254-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).PublishVolume" "disk.csi.azure.com/volume-name"="pvc-553191ed-6cf7-4436-a08d-abb08cda4724" "latency"=1044205276 I0201 17:22:11.203429 1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=1.044282578 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-96oxu0dv" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="test.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-96oxu0dv/providers/Microsoft.Compute/disks/pvc-553191ed-6cf7-4436-a08d-abb08cda4724" node="k8s-agentpool1-64138072-vmss000002" result_code="succeeded" I0201 17:22:11.203447 1 utils.go:85] GRPC response: {"publish_context":{"LUN":"0"}} I0201 17:22:11.203559 1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-553191ed-6cf7-4436-a08d-abb08cda4724-k8s-agentpool1-64138072-vmss000002-attachment) I0201 17:22:11.359103 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-96oxu0dv): vm(k8s-agentpool1-64138072-vmss000002) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-96oxu0dv/providers/Microsoft.Compute/disks/pvc-553191ed-6cf7-4436-a08d-abb08cda4724:AttachDiskOptions{diskName: "pvc-553191ed-6cf7-4436-a08d-abb08cda4724", lun: 0}], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I0201 17:22:11.359144 1 azure_controller_common.go:344] azuredisk - trying to attach disks to node k8s-agentpool1-64138072-vmss000002: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-96oxu0dv/providers/Microsoft.Compute/disks/pvc-553191ed-6cf7-4436-a08d-abb08cda4724:AttachDiskOptions{diskName: "pvc-553191ed-6cf7-4436-a08d-abb08cda4724", lun: 0}] I0201 17:22:12.575185 1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller I0201 17:22:14.583248 1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller I0201 17:22:16.591556 1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller I0201 17:22:18.601433 1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller I0201 17:22:20.611116 1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller ... skipping 4 lines ... I0201 17:22:21.479710 1 workflow.go:155] "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000002" "disk.csi.azure.com/request-id"="f5b51644-a254-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).PublishVolume" "disk.csi.azure.com/volume-name"="pvc-553191ed-6cf7-4436-a08d-abb08cda4724" "latency"=11309544460 I0201 17:22:21.479802 1 azure_disk_utils.go:1165] "msg"="Skip update. No update needed." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).updateVolumeAttachmentWithResult" "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000002" "disk.csi.azure.com/request-id"="f5b51644-a254-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-553191ed-6cf7-4436-a08d-abb08cda4724" I0201 17:22:21.479859 1 azure_disk_utils.go:1103] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).updateVolumeAttachmentWithResult" "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000002" "disk.csi.azure.com/request-id"="f5b51644-a254-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-553191ed-6cf7-4436-a08d-abb08cda4724" "latency"=138003 "number_of_net_retry"=0 "number_of_retry"=0 I0201 17:22:21.493915 1 azure_disk_utils.go:1103] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3.4" "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000002" "disk.csi.azure.com/request-id"="f5b51644-a254-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-553191ed-6cf7-4436-a08d-abb08cda4724" "latency"=14005085 "number_of_net_retry"=0 "number_of_retry"=0 I0201 17:22:21.493967 1 attach_detach.go:198] "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000002" "disk.csi.azure.com/request-id"="f5b51644-a254-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3" "disk.csi.azure.com/volume-name"="pvc-553191ed-6cf7-4436-a08d-abb08cda4724" "latency"=11323808451 I0201 17:22:21.493993 1 workflow.go:155] "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000002" "disk.csi.azure.com/request-id"="f5b51644-a254-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach" "disk.csi.azure.com/volume-name"="pvc-553191ed-6cf7-4436-a08d-abb08cda4724" "latency"=11329506472 I0201 17:22:21.494169 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0201 17:22:21.494222 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000002" "disk.csi.azure.com/request-id"="f65470de-a254-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-553191ed-6cf7-4436-a08d-abb08cda4724" "latency"=10290597509 I0201 17:22:21.494251 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).ControllerPublishVolume.func2" "disk.csi.azure.com/request-id"="f65470de-a254-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=10290724912 I0201 17:22:21.494288 1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=11.335132992 request="azuredisk_csi_driver_attach_volume_latency" resource_group="kubetest-96oxu0dv" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="test.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-96oxu0dv/providers/Microsoft.Compute/disks/pvc-553191ed-6cf7-4436-a08d-abb08cda4724" node="k8s-agentpool1-64138072-vmss000002" result_code="succeeded" I0201 17:22:21.494320 1 controllerserver_v2.go:264] "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="f65470de-a254-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).ControllerPublishVolume.func2" "latency"=10290802713 I0201 17:22:22.617895 1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller I0201 17:22:24.625627 1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller ... skipping 21 lines ... I0201 17:22:30.571765 1 shared_state.go:489] "msg"="Storing pod statefulset-azuredisk-nonroot-0 and claim [] to podToClaimsMap map." I0201 17:22:30.582638 1 utils.go:78] GRPC call: /csi.v1.Controller/CreateVolume I0201 17:22:30.582688 1 utils.go:79] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"requisite":[{"segments":{"topology.test.csi.azure.com/zone":""}}]},"capacity_range":{"required_bytes":10737418240},"name":"pvc-3251af1b-608b-4fbd-8f12-5159afe840de","parameters":{"csi.storage.k8s.io/pv/name":"pvc-3251af1b-608b-4fbd-8f12-5159afe840de","csi.storage.k8s.io/pvc/name":"persistent-storage-statefulset-azuredisk-nonroot-0","csi.storage.k8s.io/pvc/namespace":"default","skuName":"StandardSSD_LRS"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]} I0201 17:22:30.582892 1 crdprovisioner.go:308] "msg"="Creating AzVolume CRI" "csi.storage.k8s.io/pv/name"="pvc-3251af1b-608b-4fbd-8f12-5159afe840de" "disk.csi.azure.com/request-id"="01e17ea1-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/volume-name"="pvc-3251af1b-608b-4fbd-8f12-5159afe840de" I0201 17:22:30.591859 1 crdprovisioner.go:316] "msg"="Successfully created AzVolume CRI" "csi.storage.k8s.io/pv/name"="pvc-3251af1b-608b-4fbd-8f12-5159afe840de" "disk.csi.azure.com/request-id"="01e17ea1-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/volume-name"="pvc-3251af1b-608b-4fbd-8f12-5159afe840de" I0201 17:22:30.591882 1 conditionwatcher.go:99] Adding a condition function for azvolume (pvc-3251af1b-608b-4fbd-8f12-5159afe840de) I0201 17:22:30.594025 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0201 17:22:30.605425 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0201 17:22:30.605871 1 azure_disk_utils.go:1103] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate" "csi.storage.k8s.io/pv/name"="pvc-3251af1b-608b-4fbd-8f12-5159afe840de" "disk.csi.azure.com/request-id"="01e17ea1-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-3251af1b-608b-4fbd-8f12-5159afe840de" "latency"=11712646 "number_of_net_retry"=0 "number_of_retry"=0 I0201 17:22:30.605896 1 azvolume.go:158] "msg"="Creating Volume..." "csi.storage.k8s.io/pv/name"="pvc-3251af1b-608b-4fbd-8f12-5159afe840de" "disk.csi.azure.com/request-id"="01e17ea1-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/volume-name"="pvc-3251af1b-608b-4fbd-8f12-5159afe840de" I0201 17:22:30.620735 1 util.go:124] Send.sendRequest got response with ContentLength 253, StatusCode 404 and responseBody length 253 I0201 17:22:30.620765 1 azure_diskclient.go:139] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-96oxu0dv/providers/Microsoft.Compute/disks/pvc-3251af1b-608b-4fbd-8f12-5159afe840de, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-3251af1b-608b-4fbd-8f12-5159afe840de' under resource group 'kubetest-96oxu0dv' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}} I0201 17:22:30.620817 1 cloudprovisioner.go:246] begin to create disk(pvc-3251af1b-608b-4fbd-8f12-5159afe840de) account type(StandardSSD_LRS) rg(kubetest-96oxu0dv) location() size(10) selectedAvailabilityZone() maxShares(0) I0201 17:22:30.650796 1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller I0201 17:22:30.671931 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 6663 I0201 17:22:30.672171 1 azure_managedDiskController.go:94] azureDisk - creating new managed Name:pvc-3251af1b-608b-4fbd-8f12-5159afe840de StorageAccountType:StandardSSD_LRS Size:10 I0201 17:22:32.659150 1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller I0201 17:22:33.171672 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1230 I0201 17:22:33.171825 1 azure_managedDiskController.go:280] azureDisk - created new MD Name:pvc-3251af1b-608b-4fbd-8f12-5159afe840de StorageAccountType:StandardSSD_LRS Size:10 I0201 17:22:33.171871 1 cloudprovisioner.go:320] "msg"="create disk(pvc-3251af1b-608b-4fbd-8f12-5159afe840de) account type(StandardSSD_LRS) rg(kubetest-96oxu0dv) location() size(10) tags(map[kubernetes.io-created-for-pv-name:pvc-3251af1b-608b-4fbd-8f12-5159afe840de kubernetes.io-created-for-pvc-name:persistent-storage-statefulset-azuredisk-nonroot-0 kubernetes.io-created-for-pvc-namespace:default]) successfully" "csi.storage.k8s.io/pv/name"="pvc-3251af1b-608b-4fbd-8f12-5159afe840de" "disk.csi.azure.com/request-id"="01e17ea1-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/volume-name"="pvc-3251af1b-608b-4fbd-8f12-5159afe840de" I0201 17:22:33.171904 1 cloudprovisioner.go:142] "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-3251af1b-608b-4fbd-8f12-5159afe840de" "disk.csi.azure.com/request-id"="01e17ea1-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-3251af1b-608b-4fbd-8f12-5159afe840de" "latency"=2565952412 I0201 17:22:33.184959 1 azure_disk_utils.go:1103] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "csi.storage.k8s.io/pv/name"="pvc-3251af1b-608b-4fbd-8f12-5159afe840de" "disk.csi.azure.com/request-id"="01e17ea1-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-3251af1b-608b-4fbd-8f12-5159afe840de" "latency"=12994873 "number_of_net_retry"=0 "number_of_retry"=0 I0201 17:22:33.184999 1 azvolume.go:165] "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-3251af1b-608b-4fbd-8f12-5159afe840de" "disk.csi.azure.com/request-id"="01e17ea1-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "disk.csi.azure.com/volume-name"="pvc-3251af1b-608b-4fbd-8f12-5159afe840de" "latency"=2579070188 I0201 17:22:33.185017 1 workflow.go:155] "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-3251af1b-608b-4fbd-8f12-5159afe840de" "disk.csi.azure.com/request-id"="01e17ea1-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate" "disk.csi.azure.com/volume-name"="pvc-3251af1b-608b-4fbd-8f12-5159afe840de" "latency"=2590882137 I0201 17:22:33.185221 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0201 17:22:33.185291 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).CreateVolume" "csi.storage.k8s.io/pv/name"="pvc-3251af1b-608b-4fbd-8f12-5159afe840de" "disk.csi.azure.com/request-id"="01e17ea1-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-3251af1b-608b-4fbd-8f12-5159afe840de" "latency"=2593343988 I0201 17:22:33.185334 1 crdprovisioner.go:228] "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-3251af1b-608b-4fbd-8f12-5159afe840de" "disk.csi.azure.com/request-id"="01e17ea1-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-3251af1b-608b-4fbd-8f12-5159afe840de" "latency"=2602456981 I0201 17:22:33.185370 1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=2.602508181 request="azuredisk_csi_driver_controller_create_volume" resource_group="kubetest-96oxu0dv" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="test.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-96oxu0dv/providers/Microsoft.Compute/disks/pvc-3251af1b-608b-4fbd-8f12-5159afe840de" result_code="succeeded" I0201 17:22:33.185384 1 utils.go:85] GRPC response: {"volume":{"accessible_topology":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"capacity_bytes":10737418240,"content_source":{"Type":{"Volume":{}}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-3251af1b-608b-4fbd-8f12-5159afe840de","csi.storage.k8s.io/pvc/name":"persistent-storage-statefulset-azuredisk-nonroot-0","csi.storage.k8s.io/pvc/namespace":"default","requestedsizegib":"10","skuName":"StandardSSD_LRS"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-96oxu0dv/providers/Microsoft.Compute/disks/pvc-3251af1b-608b-4fbd-8f12-5159afe840de"}} I0201 17:22:33.614683 1 utils.go:78] GRPC call: /csi.v1.Controller/ControllerPublishVolume I0201 17:22:33.614703 1 utils.go:79] GRPC request: {"node_id":"k8s-agentpool1-64138072-vmss000000","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-3251af1b-608b-4fbd-8f12-5159afe840de","csi.storage.k8s.io/pvc/name":"persistent-storage-statefulset-azuredisk-nonroot-0","csi.storage.k8s.io/pvc/namespace":"default","requestedsizegib":"10","skuName":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1675272099190-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-96oxu0dv/providers/Microsoft.Compute/disks/pvc-3251af1b-608b-4fbd-8f12-5159afe840de"} I0201 17:22:33.614928 1 azure_disk_utils.go:1068] "msg"="Label selector is: disk.csi.azure.com/node-name=k8s-agentpool1-64138072-vmss000000,disk.csi.azure.com/volume-name!=pvc-3251af1b-608b-4fbd-8f12-5159afe840de." "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000000" "disk.csi.azure.com/request-id"="03b022ac-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-3251af1b-608b-4fbd-8f12-5159afe840de" I0201 17:22:33.622207 1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-3251af1b-608b-4fbd-8f12-5159afe840de-k8s-agentpool1-64138072-vmss000000-attachment) I0201 17:22:33.626939 1 azure_disk_utils.go:1103] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000000" "disk.csi.azure.com/request-id"="03b022ac-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-3251af1b-608b-4fbd-8f12-5159afe840de" "latency"=4787301 "number_of_net_retry"=0 "number_of_retry"=0 I0201 17:22:33.626967 1 attach_detach.go:192] "msg"="Attaching volume" "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000000" "disk.csi.azure.com/request-id"="03b022ac-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-3251af1b-608b-4fbd-8f12-5159afe840de" I0201 17:22:33.627770 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0201 17:22:33.680988 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1230 I0201 17:22:33.681190 1 cloudprovisioner.go:405] "msg"="GetDiskLun returned: -1. Initiating attaching volume \"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-96oxu0dv/providers/Microsoft.Compute/disks/pvc-3251af1b-608b-4fbd-8f12-5159afe840de\" to node \"k8s-agentpool1-64138072-vmss000000\"." "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000000" "disk.csi.azure.com/request-id"="03b022ac-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-3251af1b-608b-4fbd-8f12-5159afe840de" I0201 17:22:33.681223 1 cloudprovisioner.go:424] "msg"="Trying to attach volume \"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-96oxu0dv/providers/Microsoft.Compute/disks/pvc-3251af1b-608b-4fbd-8f12-5159afe840de\" to node \"k8s-agentpool1-64138072-vmss000000\"." "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000000" "disk.csi.azure.com/request-id"="03b022ac-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-3251af1b-608b-4fbd-8f12-5159afe840de" I0201 17:22:34.667232 1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller I0201 17:22:34.682384 1 batch.go:224] "cloud-provider-azure: Delayed processing of batch due to start delay" type="batch" operation="attach_disk" key="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e|kubetest-96oxu0dv|k8s-agentpool1-64138072-vmss000000" delay="1s" I0201 17:22:34.682475 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-96oxu0dv): vm(k8s-agentpool1-64138072-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-96oxu0dv/providers/Microsoft.Compute/disks/pvc-3251af1b-608b-4fbd-8f12-5159afe840de:AttachDiskOptions{diskName: "pvc-3251af1b-608b-4fbd-8f12-5159afe840de", lun: 0}]) I0201 17:22:34.690734 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0201 17:22:34.690812 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000000" "disk.csi.azure.com/request-id"="03b022ac-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-3251af1b-608b-4fbd-8f12-5159afe840de" "latency"=1068525292 I0201 17:22:34.690854 1 crdprovisioner.go:727] "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000000" "disk.csi.azure.com/request-id"="03b022ac-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLun" "disk.csi.azure.com/volume-name"="pvc-3251af1b-608b-4fbd-8f12-5159afe840de" "latency"=1068661695 I0201 17:22:34.690884 1 crdprovisioner.go:486] "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000000" "disk.csi.azure.com/request-id"="03b022ac-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).PublishVolume" "disk.csi.azure.com/volume-name"="pvc-3251af1b-608b-4fbd-8f12-5159afe840de" "latency"=1076049351 I0201 17:22:34.690953 1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=1.076125752 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-96oxu0dv" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="test.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-96oxu0dv/providers/Microsoft.Compute/disks/pvc-3251af1b-608b-4fbd-8f12-5159afe840de" node="k8s-agentpool1-64138072-vmss000000" result_code="succeeded" I0201 17:22:34.690977 1 utils.go:85] GRPC response: {"publish_context":{"LUN":"0"}} I0201 17:22:34.691145 1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-3251af1b-608b-4fbd-8f12-5159afe840de-k8s-agentpool1-64138072-vmss000000-attachment) ... skipping 3 lines ... I0201 17:22:34.695373 1 azure_disk_utils.go:1068] "msg"="Label selector is: disk.csi.azure.com/node-name=k8s-agentpool1-64138072-vmss000000,disk.csi.azure.com/volume-name!=pvc-3251af1b-608b-4fbd-8f12-5159afe840de." "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000000" "disk.csi.azure.com/request-id"="0454ff81-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-3251af1b-608b-4fbd-8f12-5159afe840de" I0201 17:22:34.695463 1 crdprovisioner.go:727] "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000000" "disk.csi.azure.com/request-id"="0454ff81-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLun" "disk.csi.azure.com/volume-name"="pvc-3251af1b-608b-4fbd-8f12-5159afe840de" "latency"=21600 I0201 17:22:34.695488 1 crdprovisioner.go:486] "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000000" "disk.csi.azure.com/request-id"="0454ff81-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).PublishVolume" "disk.csi.azure.com/volume-name"="pvc-3251af1b-608b-4fbd-8f12-5159afe840de" "latency"=213705 I0201 17:22:34.695529 1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=0.000276205 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-96oxu0dv" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="test.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-96oxu0dv/providers/Microsoft.Compute/disks/pvc-3251af1b-608b-4fbd-8f12-5159afe840de" node="k8s-agentpool1-64138072-vmss000000" result_code="succeeded" I0201 17:22:34.695543 1 utils.go:85] GRPC response: {"publish_context":{"LUN":"0"}} I0201 17:22:34.695633 1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-3251af1b-608b-4fbd-8f12-5159afe840de-k8s-agentpool1-64138072-vmss000000-attachment) I0201 17:22:34.845365 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-96oxu0dv): vm(k8s-agentpool1-64138072-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-96oxu0dv/providers/Microsoft.Compute/disks/pvc-3251af1b-608b-4fbd-8f12-5159afe840de:AttachDiskOptions{diskName: "pvc-3251af1b-608b-4fbd-8f12-5159afe840de", lun: 0}], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I0201 17:22:34.845403 1 azure_controller_common.go:344] azuredisk - trying to attach disks to node k8s-agentpool1-64138072-vmss000000: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-96oxu0dv/providers/Microsoft.Compute/disks/pvc-3251af1b-608b-4fbd-8f12-5159afe840de:AttachDiskOptions{diskName: "pvc-3251af1b-608b-4fbd-8f12-5159afe840de", lun: 0}] I0201 17:22:36.674745 1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller I0201 17:22:38.333319 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0201 17:22:38.333342 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0201 17:22:38.333328 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0201 17:22:38.333434 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0201 17:22:38.333446 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0201 17:22:38.337783 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0201 17:22:38.683373 1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller I0201 17:22:40.690189 1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller I0201 17:22:42.700601 1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller I0201 17:22:44.708558 1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller I0201 17:22:44.983553 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-96oxu0dv, k8s-agentpool1-64138072-vmss, k8s-agentpool1-64138072-vmss000000) successfully I0201 17:22:44.983581 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool1-64138072-vmss, kubetest-96oxu0dv, k8s-agentpool1-64138072-vmss000000) for cacheKey(kubetest-96oxu0dv/k8s-agentpool1-64138072-vmss) updated successfully I0201 17:22:44.983639 1 cloudprovisioner.go:448] "msg"="attach operation successful: volume \"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-96oxu0dv/providers/Microsoft.Compute/disks/pvc-3251af1b-608b-4fbd-8f12-5159afe840de\" attached to node \"k8s-agentpool1-64138072-vmss000000\"." "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000000" "disk.csi.azure.com/request-id"="03b022ac-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-3251af1b-608b-4fbd-8f12-5159afe840de" I0201 17:22:44.983688 1 cloudprovisioner.go:440] "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000000" "disk.csi.azure.com/request-id"="03b022ac-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).PublishVolume.func3" "disk.csi.azure.com/volume-name"="pvc-3251af1b-608b-4fbd-8f12-5159afe840de" "latency"=11302400785 I0201 17:22:44.983723 1 workflow.go:155] "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000000" "disk.csi.azure.com/request-id"="03b022ac-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).PublishVolume" "disk.csi.azure.com/volume-name"="pvc-3251af1b-608b-4fbd-8f12-5159afe840de" "latency"=11356695728 I0201 17:22:44.983879 1 azure_disk_utils.go:1165] "msg"="Skip update. No update needed." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).updateVolumeAttachmentWithResult" "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000000" "disk.csi.azure.com/request-id"="03b022ac-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-3251af1b-608b-4fbd-8f12-5159afe840de" I0201 17:22:44.983931 1 azure_disk_utils.go:1103] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).updateVolumeAttachmentWithResult" "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000000" "disk.csi.azure.com/request-id"="03b022ac-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-3251af1b-608b-4fbd-8f12-5159afe840de" "latency"=168003 "number_of_net_retry"=0 "number_of_retry"=0 I0201 17:22:44.999562 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0201 17:22:44.999597 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0201 17:22:44.999618 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000000" "disk.csi.azure.com/request-id"="04550d1b-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-3251af1b-608b-4fbd-8f12-5159afe840de" "latency"=10303921744 I0201 17:22:44.999654 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).ControllerPublishVolume.func2" "disk.csi.azure.com/request-id"="04550d1b-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=10304021446 I0201 17:22:44.999709 1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=10.304446755 request="azuredisk_csi_driver_attach_volume_latency" resource_group="kubetest-96oxu0dv" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="test.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-96oxu0dv/providers/Microsoft.Compute/disks/pvc-3251af1b-608b-4fbd-8f12-5159afe840de" node="k8s-agentpool1-64138072-vmss000000" result_code="succeeded" I0201 17:22:44.999742 1 controllerserver_v2.go:264] "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="04550d1b-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).ControllerPublishVolume.func2" "latency"=10304119648 I0201 17:22:44.999660 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000000" "disk.csi.azure.com/request-id"="04545b7d-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-3251af1b-608b-4fbd-8f12-5159afe840de" "latency"=10308433539 I0201 17:22:44.999773 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).ControllerPublishVolume.func2" "disk.csi.azure.com/request-id"="04545b7d-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=10308685145 ... skipping 30 lines ... I0201 17:22:54.806514 1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller I0201 17:22:54.836151 1 utils.go:78] GRPC call: /csi.v1.Controller/CreateVolume I0201 17:22:54.836175 1 utils.go:79] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"requisite":[{"segments":{"topology.test.csi.azure.com/zone":""}}]},"capacity_range":{"required_bytes":10737418240},"name":"pvc-5557970e-c6fd-4e7f-9153-ca2f088888ba","parameters":{"csi.storage.k8s.io/pv/name":"pvc-5557970e-c6fd-4e7f-9153-ca2f088888ba","csi.storage.k8s.io/pvc/name":"nginx-azuredisk-ephemeral-azuredisk01","csi.storage.k8s.io/pvc/namespace":"default","skuName":"StandardSSD_LRS"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]} I0201 17:22:54.836403 1 crdprovisioner.go:308] "msg"="Creating AzVolume CRI" "csi.storage.k8s.io/pv/name"="pvc-5557970e-c6fd-4e7f-9153-ca2f088888ba" "disk.csi.azure.com/request-id"="10564961-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/volume-name"="pvc-5557970e-c6fd-4e7f-9153-ca2f088888ba" I0201 17:22:54.853374 1 crdprovisioner.go:316] "msg"="Successfully created AzVolume CRI" "csi.storage.k8s.io/pv/name"="pvc-5557970e-c6fd-4e7f-9153-ca2f088888ba" "disk.csi.azure.com/request-id"="10564961-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/volume-name"="pvc-5557970e-c6fd-4e7f-9153-ca2f088888ba" I0201 17:22:54.853394 1 conditionwatcher.go:99] Adding a condition function for azvolume (pvc-5557970e-c6fd-4e7f-9153-ca2f088888ba) I0201 17:22:54.858677 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0201 17:22:54.859520 1 azure_disk_utils.go:1103] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate" "csi.storage.k8s.io/pv/name"="pvc-5557970e-c6fd-4e7f-9153-ca2f088888ba" "disk.csi.azure.com/request-id"="10564961-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-5557970e-c6fd-4e7f-9153-ca2f088888ba" "latency"=7222374 "number_of_net_retry"=0 "number_of_retry"=0 I0201 17:22:54.859556 1 azvolume.go:158] "msg"="Creating Volume..." "csi.storage.k8s.io/pv/name"="pvc-5557970e-c6fd-4e7f-9153-ca2f088888ba" "disk.csi.azure.com/request-id"="10564961-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/volume-name"="pvc-5557970e-c6fd-4e7f-9153-ca2f088888ba" I0201 17:22:54.875043 1 util.go:124] Send.sendRequest got response with ContentLength 253, StatusCode 404 and responseBody length 253 I0201 17:22:54.875067 1 azure_diskclient.go:139] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-96oxu0dv/providers/Microsoft.Compute/disks/pvc-5557970e-c6fd-4e7f-9153-ca2f088888ba, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-5557970e-c6fd-4e7f-9153-ca2f088888ba' under resource group 'kubetest-96oxu0dv' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}} I0201 17:22:54.875111 1 cloudprovisioner.go:246] begin to create disk(pvc-5557970e-c6fd-4e7f-9153-ca2f088888ba) account type(StandardSSD_LRS) rg(kubetest-96oxu0dv) location() size(10) selectedAvailabilityZone() maxShares(0) I0201 17:22:54.939104 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 8266 I0201 17:22:54.939360 1 azure_managedDiskController.go:94] azureDisk - creating new managed Name:pvc-5557970e-c6fd-4e7f-9153-ca2f088888ba StorageAccountType:StandardSSD_LRS Size:10 I0201 17:22:56.813268 1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller I0201 17:22:57.208981 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1217 I0201 17:22:57.209162 1 azure_managedDiskController.go:280] azureDisk - created new MD Name:pvc-5557970e-c6fd-4e7f-9153-ca2f088888ba StorageAccountType:StandardSSD_LRS Size:10 I0201 17:22:57.209230 1 cloudprovisioner.go:320] "msg"="create disk(pvc-5557970e-c6fd-4e7f-9153-ca2f088888ba) account type(StandardSSD_LRS) rg(kubetest-96oxu0dv) location() size(10) tags(map[kubernetes.io-created-for-pv-name:pvc-5557970e-c6fd-4e7f-9153-ca2f088888ba kubernetes.io-created-for-pvc-name:nginx-azuredisk-ephemeral-azuredisk01 kubernetes.io-created-for-pvc-namespace:default]) successfully" "csi.storage.k8s.io/pv/name"="pvc-5557970e-c6fd-4e7f-9153-ca2f088888ba" "disk.csi.azure.com/request-id"="10564961-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/volume-name"="pvc-5557970e-c6fd-4e7f-9153-ca2f088888ba" I0201 17:22:57.209266 1 cloudprovisioner.go:142] "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-5557970e-c6fd-4e7f-9153-ca2f088888ba" "disk.csi.azure.com/request-id"="10564961-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-5557970e-c6fd-4e7f-9153-ca2f088888ba" "latency"=2349615416 I0201 17:22:57.219293 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0201 17:22:57.219346 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).CreateVolume" "csi.storage.k8s.io/pv/name"="pvc-5557970e-c6fd-4e7f-9153-ca2f088888ba" "disk.csi.azure.com/request-id"="10564961-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-5557970e-c6fd-4e7f-9153-ca2f088888ba" "latency"=2365902210 I0201 17:22:57.219376 1 crdprovisioner.go:228] "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-5557970e-c6fd-4e7f-9153-ca2f088888ba" "disk.csi.azure.com/request-id"="10564961-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-5557970e-c6fd-4e7f-9153-ca2f088888ba" "latency"=2382995024 I0201 17:22:57.219422 1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=2.383046725 request="azuredisk_csi_driver_controller_create_volume" resource_group="kubetest-96oxu0dv" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="test.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-96oxu0dv/providers/Microsoft.Compute/disks/pvc-5557970e-c6fd-4e7f-9153-ca2f088888ba" result_code="succeeded" I0201 17:22:57.219444 1 utils.go:85] GRPC response: {"volume":{"accessible_topology":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"capacity_bytes":10737418240,"content_source":{"Type":{"Volume":{}}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-5557970e-c6fd-4e7f-9153-ca2f088888ba","csi.storage.k8s.io/pvc/name":"nginx-azuredisk-ephemeral-azuredisk01","csi.storage.k8s.io/pvc/namespace":"default","requestedsizegib":"10","skuName":"StandardSSD_LRS"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-96oxu0dv/providers/Microsoft.Compute/disks/pvc-5557970e-c6fd-4e7f-9153-ca2f088888ba"}} I0201 17:22:57.220886 1 azure_disk_utils.go:1103] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "csi.storage.k8s.io/pv/name"="pvc-5557970e-c6fd-4e7f-9153-ca2f088888ba" "disk.csi.azure.com/request-id"="10564961-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-5557970e-c6fd-4e7f-9153-ca2f088888ba" "latency"=11570380 "number_of_net_retry"=0 "number_of_retry"=0 I0201 17:22:57.220916 1 azvolume.go:165] "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-5557970e-c6fd-4e7f-9153-ca2f088888ba" "disk.csi.azure.com/request-id"="10564961-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "disk.csi.azure.com/volume-name"="pvc-5557970e-c6fd-4e7f-9153-ca2f088888ba" "latency"=2361318899 I0201 17:22:57.220933 1 workflow.go:155] "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-5557970e-c6fd-4e7f-9153-ca2f088888ba" "disk.csi.azure.com/request-id"="10564961-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate" "disk.csi.azure.com/volume-name"="pvc-5557970e-c6fd-4e7f-9153-ca2f088888ba" "latency"=2368672678 I0201 17:22:57.878116 1 utils.go:78] GRPC call: /csi.v1.Controller/ControllerPublishVolume I0201 17:22:57.878137 1 utils.go:79] GRPC request: {"node_id":"k8s-agentpool1-64138072-vmss000002","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-5557970e-c6fd-4e7f-9153-ca2f088888ba","csi.storage.k8s.io/pvc/name":"nginx-azuredisk-ephemeral-azuredisk01","csi.storage.k8s.io/pvc/namespace":"default","requestedsizegib":"10","skuName":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1675272099190-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-96oxu0dv/providers/Microsoft.Compute/disks/pvc-5557970e-c6fd-4e7f-9153-ca2f088888ba"} I0201 17:22:57.878409 1 azure_disk_utils.go:1068] "msg"="Label selector is: disk.csi.azure.com/node-name=k8s-agentpool1-64138072-vmss000002,disk.csi.azure.com/volume-name!=pvc-5557970e-c6fd-4e7f-9153-ca2f088888ba." "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000002" "disk.csi.azure.com/request-id"="122672a6-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-5557970e-c6fd-4e7f-9153-ca2f088888ba" I0201 17:22:57.890788 1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-5557970e-c6fd-4e7f-9153-ca2f088888ba-k8s-agentpool1-64138072-vmss000002-attachment) I0201 17:22:57.898549 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0201 17:22:57.899432 1 azure_disk_utils.go:1103] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000002" "disk.csi.azure.com/request-id"="122672a6-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-5557970e-c6fd-4e7f-9153-ca2f088888ba" "latency"=9570398 "number_of_net_retry"=0 "number_of_retry"=0 I0201 17:22:57.899459 1 attach_detach.go:192] "msg"="Attaching volume" "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000002" "disk.csi.azure.com/request-id"="122672a6-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-5557970e-c6fd-4e7f-9153-ca2f088888ba" I0201 17:22:57.941700 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1217 I0201 17:22:57.941972 1 cloudprovisioner.go:405] "msg"="GetDiskLun returned: -1. Initiating attaching volume \"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-96oxu0dv/providers/Microsoft.Compute/disks/pvc-5557970e-c6fd-4e7f-9153-ca2f088888ba\" to node \"k8s-agentpool1-64138072-vmss000002\"." "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000002" "disk.csi.azure.com/request-id"="122672a6-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-5557970e-c6fd-4e7f-9153-ca2f088888ba" I0201 17:22:57.942010 1 cloudprovisioner.go:424] "msg"="Trying to attach volume \"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-96oxu0dv/providers/Microsoft.Compute/disks/pvc-5557970e-c6fd-4e7f-9153-ca2f088888ba\" to node \"k8s-agentpool1-64138072-vmss000002\"." "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000002" "disk.csi.azure.com/request-id"="122672a6-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-5557970e-c6fd-4e7f-9153-ca2f088888ba" I0201 17:22:58.822168 1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller I0201 17:22:58.942766 1 batch.go:224] "cloud-provider-azure: Delayed processing of batch due to start delay" type="batch" operation="attach_disk" key="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e|kubetest-96oxu0dv|k8s-agentpool1-64138072-vmss000002" delay="1s" I0201 17:22:58.942958 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-96oxu0dv): vm(k8s-agentpool1-64138072-vmss000002) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-96oxu0dv/providers/Microsoft.Compute/disks/pvc-5557970e-c6fd-4e7f-9153-ca2f088888ba:AttachDiskOptions{diskName: "pvc-5557970e-c6fd-4e7f-9153-ca2f088888ba", lun: 1}]) I0201 17:22:58.952192 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0201 17:22:58.952255 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000002" "disk.csi.azure.com/request-id"="122672a6-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-5557970e-c6fd-4e7f-9153-ca2f088888ba" "latency"=1061398553 I0201 17:22:58.952282 1 crdprovisioner.go:727] "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000002" "disk.csi.azure.com/request-id"="122672a6-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLun" "disk.csi.azure.com/volume-name"="pvc-5557970e-c6fd-4e7f-9153-ca2f088888ba" "latency"=1061517955 I0201 17:22:58.952310 1 crdprovisioner.go:486] "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000002" "disk.csi.azure.com/request-id"="122672a6-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).PublishVolume" "disk.csi.azure.com/volume-name"="pvc-5557970e-c6fd-4e7f-9153-ca2f088888ba" "latency"=1074006014 I0201 17:22:58.952350 1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=1.074069216 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-96oxu0dv" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="test.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-96oxu0dv/providers/Microsoft.Compute/disks/pvc-5557970e-c6fd-4e7f-9153-ca2f088888ba" node="k8s-agentpool1-64138072-vmss000002" result_code="succeeded" I0201 17:22:58.952361 1 utils.go:85] GRPC response: {"publish_context":{"LUN":"1"}} I0201 17:22:58.952528 1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-5557970e-c6fd-4e7f-9153-ca2f088888ba-k8s-agentpool1-64138072-vmss000002-attachment) ... skipping 3 lines ... I0201 17:22:58.956576 1 azure_disk_utils.go:1068] "msg"="Label selector is: disk.csi.azure.com/node-name=k8s-agentpool1-64138072-vmss000002,disk.csi.azure.com/volume-name!=pvc-5557970e-c6fd-4e7f-9153-ca2f088888ba." "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000002" "disk.csi.azure.com/request-id"="12caf71c-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-5557970e-c6fd-4e7f-9153-ca2f088888ba" I0201 17:22:58.956668 1 crdprovisioner.go:727] "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000002" "disk.csi.azure.com/request-id"="12caf71c-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLun" "disk.csi.azure.com/volume-name"="pvc-5557970e-c6fd-4e7f-9153-ca2f088888ba" "latency"=20000 I0201 17:22:58.956695 1 crdprovisioner.go:486] "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000002" "disk.csi.azure.com/request-id"="12caf71c-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).PublishVolume" "disk.csi.azure.com/volume-name"="pvc-5557970e-c6fd-4e7f-9153-ca2f088888ba" "latency"=211905 I0201 17:22:58.956732 1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=0.000269805 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-96oxu0dv" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="test.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-96oxu0dv/providers/Microsoft.Compute/disks/pvc-5557970e-c6fd-4e7f-9153-ca2f088888ba" node="k8s-agentpool1-64138072-vmss000002" result_code="succeeded" I0201 17:22:58.956744 1 utils.go:85] GRPC response: {"publish_context":{"LUN":"1"}} I0201 17:22:58.956822 1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-5557970e-c6fd-4e7f-9153-ca2f088888ba-k8s-agentpool1-64138072-vmss000002-attachment) I0201 17:22:59.164261 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-96oxu0dv): vm(k8s-agentpool1-64138072-vmss000002) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-96oxu0dv/providers/Microsoft.Compute/disks/pvc-5557970e-c6fd-4e7f-9153-ca2f088888ba:AttachDiskOptions{diskName: "pvc-5557970e-c6fd-4e7f-9153-ca2f088888ba", lun: 1}], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I0201 17:22:59.164301 1 azure_controller_common.go:344] azuredisk - trying to attach disks to node k8s-agentpool1-64138072-vmss000002: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-96oxu0dv/providers/Microsoft.Compute/disks/pvc-5557970e-c6fd-4e7f-9153-ca2f088888ba:AttachDiskOptions{diskName: "pvc-5557970e-c6fd-4e7f-9153-ca2f088888ba", lun: 1}] I0201 17:23:00.830519 1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller I0201 17:23:02.838897 1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller I0201 17:23:04.844889 1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller I0201 17:23:06.852934 1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller I0201 17:23:08.334501 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0201 17:23:08.334583 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0201 17:23:08.334643 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0201 17:23:08.334669 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0201 17:23:08.334679 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0201 17:23:08.337891 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0201 17:23:08.859159 1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller I0201 17:23:10.866836 1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller I0201 17:23:12.874265 1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller I0201 17:23:14.280966 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-96oxu0dv, k8s-agentpool1-64138072-vmss, k8s-agentpool1-64138072-vmss000002) successfully I0201 17:23:14.280999 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool1-64138072-vmss, kubetest-96oxu0dv, k8s-agentpool1-64138072-vmss000002) for cacheKey(kubetest-96oxu0dv/k8s-agentpool1-64138072-vmss) updated successfully I0201 17:23:14.281059 1 cloudprovisioner.go:448] "msg"="attach operation successful: volume \"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-96oxu0dv/providers/Microsoft.Compute/disks/pvc-5557970e-c6fd-4e7f-9153-ca2f088888ba\" attached to node \"k8s-agentpool1-64138072-vmss000002\"." "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000002" "disk.csi.azure.com/request-id"="122672a6-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-5557970e-c6fd-4e7f-9153-ca2f088888ba" I0201 17:23:14.281098 1 cloudprovisioner.go:440] "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000002" "disk.csi.azure.com/request-id"="122672a6-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).PublishVolume.func3" "disk.csi.azure.com/volume-name"="pvc-5557970e-c6fd-4e7f-9153-ca2f088888ba" "latency"=16339024513 I0201 17:23:14.281128 1 workflow.go:155] "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000002" "disk.csi.azure.com/request-id"="122672a6-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).PublishVolume" "disk.csi.azure.com/volume-name"="pvc-5557970e-c6fd-4e7f-9153-ca2f088888ba" "latency"=16381595897 I0201 17:23:14.281272 1 azure_disk_utils.go:1165] "msg"="Skip update. No update needed." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).updateVolumeAttachmentWithResult" "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000002" "disk.csi.azure.com/request-id"="122672a6-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-5557970e-c6fd-4e7f-9153-ca2f088888ba" I0201 17:23:14.281335 1 azure_disk_utils.go:1103] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).updateVolumeAttachmentWithResult" "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000002" "disk.csi.azure.com/request-id"="122672a6-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-5557970e-c6fd-4e7f-9153-ca2f088888ba" "latency"=122503 "number_of_net_retry"=0 "number_of_retry"=0 I0201 17:23:14.304179 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0201 17:23:14.304241 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000002" "disk.csi.azure.com/request-id"="12cb03be-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-5557970e-c6fd-4e7f-9153-ca2f088888ba" "latency"=15347350010 I0201 17:23:14.304252 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0201 17:23:14.304268 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).ControllerPublishVolume.func2" "disk.csi.azure.com/request-id"="12cb03be-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=15347455212 I0201 17:23:14.304331 1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=15.34785332 request="azuredisk_csi_driver_attach_volume_latency" resource_group="kubetest-96oxu0dv" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="test.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-96oxu0dv/providers/Microsoft.Compute/disks/pvc-5557970e-c6fd-4e7f-9153-ca2f088888ba" node="k8s-agentpool1-64138072-vmss000002" result_code="succeeded" I0201 17:23:14.304347 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000002" "disk.csi.azure.com/request-id"="12ca5aa7-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-5557970e-c6fd-4e7f-9153-ca2f088888ba" "latency"=15351755902 I0201 17:23:14.304380 1 controllerserver_v2.go:264] "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="12cb03be-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).ControllerPublishVolume.func2" "latency"=15347556114 I0201 17:23:14.304385 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).ControllerPublishVolume.func2" "disk.csi.azure.com/request-id"="12ca5aa7-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=15351883205 I0201 17:23:14.304421 1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=16.426152125 request="azuredisk_csi_driver_attach_volume_latency" resource_group="kubetest-96oxu0dv" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="test.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-96oxu0dv/providers/Microsoft.Compute/disks/pvc-5557970e-c6fd-4e7f-9153-ca2f088888ba" node="k8s-agentpool1-64138072-vmss000002" result_code="succeeded" ... skipping 30 lines ... I0201 17:23:19.945934 1 shared_state.go:489] "msg"="Storing pod daemonset-azuredisk-ephemeral-c5kr6 and claim [] to podToClaimsMap map." I0201 17:23:19.991903 1 utils.go:78] GRPC call: /csi.v1.Controller/CreateVolume I0201 17:23:19.991928 1 utils.go:79] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"requisite":[{"segments":{"topology.test.csi.azure.com/zone":""}}]},"capacity_range":{"required_bytes":10737418240},"name":"pvc-72c7ff8f-6519-4968-b40e-0a925c2f0891","parameters":{"csi.storage.k8s.io/pv/name":"pvc-72c7ff8f-6519-4968-b40e-0a925c2f0891","csi.storage.k8s.io/pvc/name":"daemonset-azuredisk-ephemeral-c5kr6-azuredisk","csi.storage.k8s.io/pvc/namespace":"default","skuName":"StandardSSD_LRS"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]} I0201 17:23:19.992168 1 crdprovisioner.go:308] "msg"="Creating AzVolume CRI" "csi.storage.k8s.io/pv/name"="pvc-72c7ff8f-6519-4968-b40e-0a925c2f0891" "disk.csi.azure.com/request-id"="1f54c00b-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/volume-name"="pvc-72c7ff8f-6519-4968-b40e-0a925c2f0891" I0201 17:23:20.001680 1 crdprovisioner.go:316] "msg"="Successfully created AzVolume CRI" "csi.storage.k8s.io/pv/name"="pvc-72c7ff8f-6519-4968-b40e-0a925c2f0891" "disk.csi.azure.com/request-id"="1f54c00b-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/volume-name"="pvc-72c7ff8f-6519-4968-b40e-0a925c2f0891" I0201 17:23:20.001701 1 conditionwatcher.go:99] Adding a condition function for azvolume (pvc-72c7ff8f-6519-4968-b40e-0a925c2f0891) I0201 17:23:20.012789 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0201 17:23:20.013187 1 azure_disk_utils.go:1103] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate" "csi.storage.k8s.io/pv/name"="pvc-72c7ff8f-6519-4968-b40e-0a925c2f0891" "disk.csi.azure.com/request-id"="1f54c00b-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-72c7ff8f-6519-4968-b40e-0a925c2f0891" "latency"=11677042 "number_of_net_retry"=0 "number_of_retry"=0 I0201 17:23:20.013211 1 azvolume.go:158] "msg"="Creating Volume..." "csi.storage.k8s.io/pv/name"="pvc-72c7ff8f-6519-4968-b40e-0a925c2f0891" "disk.csi.azure.com/request-id"="1f54c00b-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/volume-name"="pvc-72c7ff8f-6519-4968-b40e-0a925c2f0891" I0201 17:23:20.027889 1 util.go:124] Send.sendRequest got response with ContentLength 253, StatusCode 404 and responseBody length 253 I0201 17:23:20.027935 1 azure_diskclient.go:139] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-96oxu0dv/providers/Microsoft.Compute/disks/pvc-72c7ff8f-6519-4968-b40e-0a925c2f0891, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-72c7ff8f-6519-4968-b40e-0a925c2f0891' under resource group 'kubetest-96oxu0dv' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}} I0201 17:23:20.028029 1 cloudprovisioner.go:246] begin to create disk(pvc-72c7ff8f-6519-4968-b40e-0a925c2f0891) account type(StandardSSD_LRS) rg(kubetest-96oxu0dv) location() size(10) selectedAvailabilityZone() maxShares(0) I0201 17:23:20.081211 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 9856 I0201 17:23:20.081502 1 azure_managedDiskController.go:94] azureDisk - creating new managed Name:pvc-72c7ff8f-6519-4968-b40e-0a925c2f0891 StorageAccountType:StandardSSD_LRS Size:10 I0201 17:23:20.906986 1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller I0201 17:23:21.856981 1 utils.go:78] GRPC call: /csi.v1.Controller/CreateVolume I0201 17:23:21.857011 1 utils.go:79] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"requisite":[{"segments":{"topology.test.csi.azure.com/zone":""}}]},"capacity_range":{"required_bytes":10737418240},"name":"pvc-2802f37a-394b-486e-88b0-bb86c39febf9","parameters":{"csi.storage.k8s.io/pv/name":"pvc-2802f37a-394b-486e-88b0-bb86c39febf9","csi.storage.k8s.io/pvc/name":"daemonset-azuredisk-ephemeral-j9t99-azuredisk","csi.storage.k8s.io/pvc/namespace":"default","skuName":"StandardSSD_LRS"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]} ... skipping 2 lines ... I0201 17:23:21.857247 1 utils.go:79] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"requisite":[{"segments":{"topology.test.csi.azure.com/zone":""}}]},"capacity_range":{"required_bytes":10737418240},"name":"pvc-5f7b8fd9-c2af-41d7-9247-71e90b7a7305","parameters":{"csi.storage.k8s.io/pv/name":"pvc-5f7b8fd9-c2af-41d7-9247-71e90b7a7305","csi.storage.k8s.io/pvc/name":"daemonset-azuredisk-ephemeral-j6lzv-azuredisk","csi.storage.k8s.io/pvc/namespace":"default","skuName":"StandardSSD_LRS"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]} I0201 17:23:21.857418 1 crdprovisioner.go:308] "msg"="Creating AzVolume CRI" "csi.storage.k8s.io/pv/name"="pvc-5f7b8fd9-c2af-41d7-9247-71e90b7a7305" "disk.csi.azure.com/request-id"="20715e33-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/volume-name"="pvc-5f7b8fd9-c2af-41d7-9247-71e90b7a7305" I0201 17:23:21.867301 1 crdprovisioner.go:316] "msg"="Successfully created AzVolume CRI" "csi.storage.k8s.io/pv/name"="pvc-2802f37a-394b-486e-88b0-bb86c39febf9" "disk.csi.azure.com/request-id"="20715739-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/volume-name"="pvc-2802f37a-394b-486e-88b0-bb86c39febf9" I0201 17:23:21.867318 1 conditionwatcher.go:99] Adding a condition function for azvolume (pvc-2802f37a-394b-486e-88b0-bb86c39febf9) I0201 17:23:21.867594 1 crdprovisioner.go:316] "msg"="Successfully created AzVolume CRI" "csi.storage.k8s.io/pv/name"="pvc-5f7b8fd9-c2af-41d7-9247-71e90b7a7305" "disk.csi.azure.com/request-id"="20715e33-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/volume-name"="pvc-5f7b8fd9-c2af-41d7-9247-71e90b7a7305" I0201 17:23:21.867607 1 conditionwatcher.go:99] Adding a condition function for azvolume (pvc-5f7b8fd9-c2af-41d7-9247-71e90b7a7305) I0201 17:23:21.868765 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0201 17:23:21.880319 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0201 17:23:21.880752 1 azure_disk_utils.go:1103] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate" "csi.storage.k8s.io/pv/name"="pvc-5f7b8fd9-c2af-41d7-9247-71e90b7a7305" "disk.csi.azure.com/request-id"="20715e33-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-5f7b8fd9-c2af-41d7-9247-71e90b7a7305" "latency"=9308393 "number_of_net_retry"=0 "number_of_retry"=0 I0201 17:23:21.880777 1 azvolume.go:158] "msg"="Creating Volume..." "csi.storage.k8s.io/pv/name"="pvc-5f7b8fd9-c2af-41d7-9247-71e90b7a7305" "disk.csi.azure.com/request-id"="20715e33-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/volume-name"="pvc-5f7b8fd9-c2af-41d7-9247-71e90b7a7305" I0201 17:23:21.883465 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0201 17:23:21.883571 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0201 17:23:21.885675 1 azure_disk_utils.go:1103] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate" "csi.storage.k8s.io/pv/name"="pvc-2802f37a-394b-486e-88b0-bb86c39febf9" "disk.csi.azure.com/request-id"="20715739-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-2802f37a-394b-486e-88b0-bb86c39febf9" "latency"=16735548 "number_of_net_retry"=0 "number_of_retry"=0 I0201 17:23:21.885705 1 azvolume.go:158] "msg"="Creating Volume..." "csi.storage.k8s.io/pv/name"="pvc-2802f37a-394b-486e-88b0-bb86c39febf9" "disk.csi.azure.com/request-id"="20715739-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/volume-name"="pvc-2802f37a-394b-486e-88b0-bb86c39febf9" I0201 17:23:21.915207 1 util.go:124] Send.sendRequest got response with ContentLength 253, StatusCode 404 and responseBody length 253 I0201 17:23:21.915229 1 azure_diskclient.go:139] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-96oxu0dv/providers/Microsoft.Compute/disks/pvc-5f7b8fd9-c2af-41d7-9247-71e90b7a7305, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-5f7b8fd9-c2af-41d7-9247-71e90b7a7305' under resource group 'kubetest-96oxu0dv' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}} I0201 17:23:21.915273 1 cloudprovisioner.go:246] begin to create disk(pvc-5f7b8fd9-c2af-41d7-9247-71e90b7a7305) account type(StandardSSD_LRS) rg(kubetest-96oxu0dv) location() size(10) selectedAvailabilityZone() maxShares(0) I0201 17:23:21.930510 1 util.go:124] Send.sendRequest got response with ContentLength 253, StatusCode 404 and responseBody length 253 I0201 17:23:21.930535 1 azure_diskclient.go:139] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-96oxu0dv/providers/Microsoft.Compute/disks/pvc-2802f37a-394b-486e-88b0-bb86c39febf9, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-2802f37a-394b-486e-88b0-bb86c39febf9' under resource group 'kubetest-96oxu0dv' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}} I0201 17:23:21.930583 1 cloudprovisioner.go:246] begin to create disk(pvc-2802f37a-394b-486e-88b0-bb86c39febf9) account type(StandardSSD_LRS) rg(kubetest-96oxu0dv) location() size(10) selectedAvailabilityZone() maxShares(0) I0201 17:23:21.983427 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 11220 I0201 17:23:21.983710 1 azure_managedDiskController.go:94] azureDisk - creating new managed Name:pvc-5f7b8fd9-c2af-41d7-9247-71e90b7a7305 StorageAccountType:StandardSSD_LRS Size:10 I0201 17:23:21.990055 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 11220 I0201 17:23:21.990302 1 azure_managedDiskController.go:94] azureDisk - creating new managed Name:pvc-2802f37a-394b-486e-88b0-bb86c39febf9 StorageAccountType:StandardSSD_LRS Size:10 I0201 17:23:22.382183 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1225 I0201 17:23:22.382340 1 azure_managedDiskController.go:280] azureDisk - created new MD Name:pvc-72c7ff8f-6519-4968-b40e-0a925c2f0891 StorageAccountType:StandardSSD_LRS Size:10 I0201 17:23:22.382388 1 cloudprovisioner.go:320] "msg"="create disk(pvc-72c7ff8f-6519-4968-b40e-0a925c2f0891) account type(StandardSSD_LRS) rg(kubetest-96oxu0dv) location() size(10) tags(map[kubernetes.io-created-for-pv-name:pvc-72c7ff8f-6519-4968-b40e-0a925c2f0891 kubernetes.io-created-for-pvc-name:daemonset-azuredisk-ephemeral-c5kr6-azuredisk kubernetes.io-created-for-pvc-namespace:default]) successfully" "csi.storage.k8s.io/pv/name"="pvc-72c7ff8f-6519-4968-b40e-0a925c2f0891" "disk.csi.azure.com/request-id"="1f54c00b-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/volume-name"="pvc-72c7ff8f-6519-4968-b40e-0a925c2f0891" I0201 17:23:22.382417 1 cloudprovisioner.go:142] "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-72c7ff8f-6519-4968-b40e-0a925c2f0891" "disk.csi.azure.com/request-id"="1f54c00b-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-72c7ff8f-6519-4968-b40e-0a925c2f0891" "latency"=2369147957 I0201 17:23:22.391846 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0201 17:23:22.391895 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).CreateVolume" "csi.storage.k8s.io/pv/name"="pvc-72c7ff8f-6519-4968-b40e-0a925c2f0891" "disk.csi.azure.com/request-id"="1f54c00b-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-72c7ff8f-6519-4968-b40e-0a925c2f0891" "latency"=2390142592 I0201 17:23:22.391915 1 crdprovisioner.go:228] "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-72c7ff8f-6519-4968-b40e-0a925c2f0891" "disk.csi.azure.com/request-id"="1f54c00b-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-72c7ff8f-6519-4968-b40e-0a925c2f0891" "latency"=2399789792 I0201 17:23:22.391945 1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=2.399831693 request="azuredisk_csi_driver_controller_create_volume" resource_group="kubetest-96oxu0dv" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="test.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-96oxu0dv/providers/Microsoft.Compute/disks/pvc-72c7ff8f-6519-4968-b40e-0a925c2f0891" result_code="succeeded" I0201 17:23:22.391965 1 utils.go:85] GRPC response: {"volume":{"accessible_topology":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"capacity_bytes":10737418240,"content_source":{"Type":{"Volume":{}}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-72c7ff8f-6519-4968-b40e-0a925c2f0891","csi.storage.k8s.io/pvc/name":"daemonset-azuredisk-ephemeral-c5kr6-azuredisk","csi.storage.k8s.io/pvc/namespace":"default","requestedsizegib":"10","skuName":"StandardSSD_LRS"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-96oxu0dv/providers/Microsoft.Compute/disks/pvc-72c7ff8f-6519-4968-b40e-0a925c2f0891"}} I0201 17:23:22.394129 1 azure_disk_utils.go:1103] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "csi.storage.k8s.io/pv/name"="pvc-72c7ff8f-6519-4968-b40e-0a925c2f0891" "disk.csi.azure.com/request-id"="1f54c00b-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-72c7ff8f-6519-4968-b40e-0a925c2f0891" "latency"=11663542 "number_of_net_retry"=0 "number_of_retry"=0 I0201 17:23:22.394167 1 azvolume.go:165] "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-72c7ff8f-6519-4968-b40e-0a925c2f0891" "disk.csi.azure.com/request-id"="1f54c00b-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "disk.csi.azure.com/volume-name"="pvc-72c7ff8f-6519-4968-b40e-0a925c2f0891" "latency"=2380922001 ... skipping 2 lines ... I0201 17:23:23.024208 1 utils.go:78] GRPC call: /csi.v1.Controller/ControllerPublishVolume I0201 17:23:23.024231 1 utils.go:79] GRPC request: {"node_id":"k8s-agentpool1-64138072-vmss000002","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-72c7ff8f-6519-4968-b40e-0a925c2f0891","csi.storage.k8s.io/pvc/name":"daemonset-azuredisk-ephemeral-c5kr6-azuredisk","csi.storage.k8s.io/pvc/namespace":"default","requestedsizegib":"10","skuName":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1675272099190-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-96oxu0dv/providers/Microsoft.Compute/disks/pvc-72c7ff8f-6519-4968-b40e-0a925c2f0891"} I0201 17:23:23.024726 1 azure_disk_utils.go:1068] "msg"="Label selector is: disk.csi.azure.com/node-name=k8s-agentpool1-64138072-vmss000002,disk.csi.azure.com/volume-name!=pvc-72c7ff8f-6519-4968-b40e-0a925c2f0891." "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000002" "disk.csi.azure.com/request-id"="21236f6c-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-72c7ff8f-6519-4968-b40e-0a925c2f0891" I0201 17:23:23.030297 1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-72c7ff8f-6519-4968-b40e-0a925c2f0891-k8s-agentpool1-64138072-vmss000002-attachment) I0201 17:23:23.034899 1 azure_disk_utils.go:1103] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000002" "disk.csi.azure.com/request-id"="21236f6c-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-72c7ff8f-6519-4968-b40e-0a925c2f0891" "latency"=5404412 "number_of_net_retry"=0 "number_of_retry"=0 I0201 17:23:23.034931 1 attach_detach.go:192] "msg"="Attaching volume" "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000002" "disk.csi.azure.com/request-id"="21236f6c-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-72c7ff8f-6519-4968-b40e-0a925c2f0891" I0201 17:23:23.035709 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0201 17:23:23.057081 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1225 I0201 17:23:23.057334 1 cloudprovisioner.go:405] "msg"="GetDiskLun returned: -1. Initiating attaching volume \"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-96oxu0dv/providers/Microsoft.Compute/disks/pvc-72c7ff8f-6519-4968-b40e-0a925c2f0891\" to node \"k8s-agentpool1-64138072-vmss000002\"." "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000002" "disk.csi.azure.com/request-id"="21236f6c-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-72c7ff8f-6519-4968-b40e-0a925c2f0891" I0201 17:23:23.057368 1 cloudprovisioner.go:424] "msg"="Trying to attach volume \"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-96oxu0dv/providers/Microsoft.Compute/disks/pvc-72c7ff8f-6519-4968-b40e-0a925c2f0891\" to node \"k8s-agentpool1-64138072-vmss000002\"." "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000002" "disk.csi.azure.com/request-id"="21236f6c-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-72c7ff8f-6519-4968-b40e-0a925c2f0891" I0201 17:23:24.057815 1 batch.go:224] "cloud-provider-azure: Delayed processing of batch due to start delay" type="batch" operation="attach_disk" key="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e|kubetest-96oxu0dv|k8s-agentpool1-64138072-vmss000002" delay="1s" I0201 17:23:24.057936 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-96oxu0dv): vm(k8s-agentpool1-64138072-vmss000002) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-96oxu0dv/providers/Microsoft.Compute/disks/pvc-72c7ff8f-6519-4968-b40e-0a925c2f0891:AttachDiskOptions{diskName: "pvc-72c7ff8f-6519-4968-b40e-0a925c2f0891", lun: 2}]) I0201 17:23:24.066450 1 azure_disk_utils.go:1103] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3.4" "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000002" "disk.csi.azure.com/request-id"="21236f6c-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-72c7ff8f-6519-4968-b40e-0a925c2f0891" "latency"=8345473 "number_of_net_retry"=0 "number_of_retry"=0 I0201 17:23:24.066841 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0201 17:23:24.066908 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000002" "disk.csi.azure.com/request-id"="21236f6c-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-72c7ff8f-6519-4968-b40e-0a925c2f0891" "latency"=1036544107 I0201 17:23:24.066942 1 crdprovisioner.go:727] "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000002" "disk.csi.azure.com/request-id"="21236f6c-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLun" "disk.csi.azure.com/volume-name"="pvc-72c7ff8f-6519-4968-b40e-0a925c2f0891" "latency"=1036702510 I0201 17:23:24.067015 1 crdprovisioner.go:486] "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000002" "disk.csi.azure.com/request-id"="21236f6c-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).PublishVolume" "disk.csi.azure.com/volume-name"="pvc-72c7ff8f-6519-4968-b40e-0a925c2f0891" "latency"=1042585832 I0201 17:23:24.067064 1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=1.042691035 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-96oxu0dv" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="test.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-96oxu0dv/providers/Microsoft.Compute/disks/pvc-72c7ff8f-6519-4968-b40e-0a925c2f0891" node="k8s-agentpool1-64138072-vmss000002" result_code="succeeded" I0201 17:23:24.067080 1 utils.go:85] GRPC response: {"publish_context":{"LUN":"2"}} I0201 17:23:24.067165 1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-72c7ff8f-6519-4968-b40e-0a925c2f0891-k8s-agentpool1-64138072-vmss000002-attachment) I0201 17:23:24.219930 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-96oxu0dv): vm(k8s-agentpool1-64138072-vmss000002) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-96oxu0dv/providers/Microsoft.Compute/disks/pvc-72c7ff8f-6519-4968-b40e-0a925c2f0891:AttachDiskOptions{diskName: "pvc-72c7ff8f-6519-4968-b40e-0a925c2f0891", lun: 2}], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I0201 17:23:24.219976 1 azure_controller_common.go:344] azuredisk - trying to attach disks to node k8s-agentpool1-64138072-vmss000002: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-96oxu0dv/providers/Microsoft.Compute/disks/pvc-72c7ff8f-6519-4968-b40e-0a925c2f0891:AttachDiskOptions{diskName: "pvc-72c7ff8f-6519-4968-b40e-0a925c2f0891", lun: 2}] I0201 17:23:24.374071 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1225 I0201 17:23:24.374265 1 azure_managedDiskController.go:280] azureDisk - created new MD Name:pvc-2802f37a-394b-486e-88b0-bb86c39febf9 StorageAccountType:StandardSSD_LRS Size:10 I0201 17:23:24.374332 1 cloudprovisioner.go:320] "msg"="create disk(pvc-2802f37a-394b-486e-88b0-bb86c39febf9) account type(StandardSSD_LRS) rg(kubetest-96oxu0dv) location() size(10) tags(map[kubernetes.io-created-for-pv-name:pvc-2802f37a-394b-486e-88b0-bb86c39febf9 kubernetes.io-created-for-pvc-name:daemonset-azuredisk-ephemeral-j9t99-azuredisk kubernetes.io-created-for-pvc-namespace:default]) successfully" "csi.storage.k8s.io/pv/name"="pvc-2802f37a-394b-486e-88b0-bb86c39febf9" "disk.csi.azure.com/request-id"="20715739-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/volume-name"="pvc-2802f37a-394b-486e-88b0-bb86c39febf9" I0201 17:23:24.374368 1 cloudprovisioner.go:142] "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-2802f37a-394b-486e-88b0-bb86c39febf9" "disk.csi.azure.com/request-id"="20715739-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-2802f37a-394b-486e-88b0-bb86c39febf9" "latency"=2488566835 I0201 17:23:24.383238 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0201 17:23:24.383300 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).CreateVolume" "csi.storage.k8s.io/pv/name"="pvc-2802f37a-394b-486e-88b0-bb86c39febf9" "disk.csi.azure.com/request-id"="20715739-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-2802f37a-394b-486e-88b0-bb86c39febf9" "latency"=2515924002 I0201 17:23:24.383337 1 crdprovisioner.go:228] "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-2802f37a-394b-486e-88b0-bb86c39febf9" "disk.csi.azure.com/request-id"="20715739-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-2802f37a-394b-486e-88b0-bb86c39febf9" "latency"=2526108713 I0201 17:23:24.383377 1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=2.526158715 request="azuredisk_csi_driver_controller_create_volume" resource_group="kubetest-96oxu0dv" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="test.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-96oxu0dv/providers/Microsoft.Compute/disks/pvc-2802f37a-394b-486e-88b0-bb86c39febf9" result_code="succeeded" I0201 17:23:24.383458 1 azure_disk_utils.go:1103] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "csi.storage.k8s.io/pv/name"="pvc-2802f37a-394b-486e-88b0-bb86c39febf9" "disk.csi.azure.com/request-id"="20715739-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-2802f37a-394b-486e-88b0-bb86c39febf9" "latency"=9028687 "number_of_net_retry"=0 "number_of_retry"=0 I0201 17:23:24.383511 1 azvolume.go:165] "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-2802f37a-394b-486e-88b0-bb86c39febf9" "disk.csi.azure.com/request-id"="20715739-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "disk.csi.azure.com/volume-name"="pvc-2802f37a-394b-486e-88b0-bb86c39febf9" "latency"=2497753925 I0201 17:23:24.383543 1 workflow.go:155] "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-2802f37a-394b-486e-88b0-bb86c39febf9" "disk.csi.azure.com/request-id"="20715739-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate" "disk.csi.azure.com/volume-name"="pvc-2802f37a-394b-486e-88b0-bb86c39febf9" "latency"=2514621175 ... skipping 2 lines ... I0201 17:23:24.384851 1 azure_managedDiskController.go:280] azureDisk - created new MD Name:pvc-5f7b8fd9-c2af-41d7-9247-71e90b7a7305 StorageAccountType:StandardSSD_LRS Size:10 I0201 17:23:24.384893 1 cloudprovisioner.go:320] "msg"="create disk(pvc-5f7b8fd9-c2af-41d7-9247-71e90b7a7305) account type(StandardSSD_LRS) rg(kubetest-96oxu0dv) location() size(10) tags(map[kubernetes.io-created-for-pv-name:pvc-5f7b8fd9-c2af-41d7-9247-71e90b7a7305 kubernetes.io-created-for-pvc-name:daemonset-azuredisk-ephemeral-j6lzv-azuredisk kubernetes.io-created-for-pvc-namespace:default]) successfully" "csi.storage.k8s.io/pv/name"="pvc-5f7b8fd9-c2af-41d7-9247-71e90b7a7305" "disk.csi.azure.com/request-id"="20715e33-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/volume-name"="pvc-5f7b8fd9-c2af-41d7-9247-71e90b7a7305" I0201 17:23:24.384920 1 cloudprovisioner.go:142] "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-5f7b8fd9-c2af-41d7-9247-71e90b7a7305" "disk.csi.azure.com/request-id"="20715e33-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-5f7b8fd9-c2af-41d7-9247-71e90b7a7305" "latency"=2504078456 I0201 17:23:24.395708 1 azure_disk_utils.go:1103] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "csi.storage.k8s.io/pv/name"="pvc-5f7b8fd9-c2af-41d7-9247-71e90b7a7305" "disk.csi.azure.com/request-id"="20715e33-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-5f7b8fd9-c2af-41d7-9247-71e90b7a7305" "latency"=10744023 "number_of_net_retry"=0 "number_of_retry"=0 I0201 17:23:24.395746 1 azvolume.go:165] "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-5f7b8fd9-c2af-41d7-9247-71e90b7a7305" "disk.csi.azure.com/request-id"="20715e33-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "disk.csi.azure.com/volume-name"="pvc-5f7b8fd9-c2af-41d7-9247-71e90b7a7305" "latency"=2514926282 I0201 17:23:24.395768 1 workflow.go:155] "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-5f7b8fd9-c2af-41d7-9247-71e90b7a7305" "disk.csi.azure.com/request-id"="20715e33-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate" "disk.csi.azure.com/volume-name"="pvc-5f7b8fd9-c2af-41d7-9247-71e90b7a7305" "latency"=2524338377 I0201 17:23:24.400139 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0201 17:23:24.400176 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).CreateVolume" "csi.storage.k8s.io/pv/name"="pvc-5f7b8fd9-c2af-41d7-9247-71e90b7a7305" "disk.csi.azure.com/request-id"="20715e33-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-5f7b8fd9-c2af-41d7-9247-71e90b7a7305" "latency"=2532534046 I0201 17:23:24.400200 1 crdprovisioner.go:228] "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-5f7b8fd9-c2af-41d7-9247-71e90b7a7305" "disk.csi.azure.com/request-id"="20715e33-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-5f7b8fd9-c2af-41d7-9247-71e90b7a7305" "latency"=2542797760 I0201 17:23:24.400226 1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=2.5428310610000002 request="azuredisk_csi_driver_controller_create_volume" resource_group="kubetest-96oxu0dv" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="test.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-96oxu0dv/providers/Microsoft.Compute/disks/pvc-5f7b8fd9-c2af-41d7-9247-71e90b7a7305" result_code="succeeded" I0201 17:23:24.400237 1 utils.go:85] GRPC response: {"volume":{"accessible_topology":[{"segments":{"topology.test.csi.azure.com/zone":""}}],"capacity_bytes":10737418240,"content_source":{"Type":{"Volume":{}}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-5f7b8fd9-c2af-41d7-9247-71e90b7a7305","csi.storage.k8s.io/pvc/name":"daemonset-azuredisk-ephemeral-j6lzv-azuredisk","csi.storage.k8s.io/pvc/namespace":"default","requestedsizegib":"10","skuName":"StandardSSD_LRS"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-96oxu0dv/providers/Microsoft.Compute/disks/pvc-5f7b8fd9-c2af-41d7-9247-71e90b7a7305"}} I0201 17:23:24.921547 1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller I0201 17:23:24.944026 1 utils.go:78] GRPC call: /csi.v1.Controller/ControllerPublishVolume ... skipping 5 lines ... I0201 17:23:24.958419 1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-5f7b8fd9-c2af-41d7-9247-71e90b7a7305-k8s-agentpool1-64138072-vmss000001-attachment) I0201 17:23:24.971176 1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-2802f37a-394b-486e-88b0-bb86c39febf9-k8s-agentpool1-64138072-vmss000000-attachment) I0201 17:23:24.973879 1 azure_disk_utils.go:1103] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000001" "disk.csi.azure.com/request-id"="22486023-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-5f7b8fd9-c2af-41d7-9247-71e90b7a7305" "latency"=16423990 "number_of_net_retry"=0 "number_of_retry"=0 I0201 17:23:24.973908 1 attach_detach.go:192] "msg"="Attaching volume" "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000001" "disk.csi.azure.com/request-id"="22486023-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-5f7b8fd9-c2af-41d7-9247-71e90b7a7305" I0201 17:23:24.974010 1 azure_disk_utils.go:1103] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000000" "disk.csi.azure.com/request-id"="2249a4d1-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-2802f37a-394b-486e-88b0-bb86c39febf9" "latency"=11788509 "number_of_net_retry"=0 "number_of_retry"=0 I0201 17:23:24.974039 1 attach_detach.go:192] "msg"="Attaching volume" "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000000" "disk.csi.azure.com/request-id"="2249a4d1-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-2802f37a-394b-486e-88b0-bb86c39febf9" I0201 17:23:24.977220 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0201 17:23:24.977519 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0201 17:23:24.994767 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1225 I0201 17:23:24.994961 1 cloudprovisioner.go:405] "msg"="GetDiskLun returned: -1. Initiating attaching volume \"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-96oxu0dv/providers/Microsoft.Compute/disks/pvc-2802f37a-394b-486e-88b0-bb86c39febf9\" to node \"k8s-agentpool1-64138072-vmss000000\"." "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000000" "disk.csi.azure.com/request-id"="2249a4d1-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-2802f37a-394b-486e-88b0-bb86c39febf9" I0201 17:23:24.994997 1 cloudprovisioner.go:424] "msg"="Trying to attach volume \"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-96oxu0dv/providers/Microsoft.Compute/disks/pvc-2802f37a-394b-486e-88b0-bb86c39febf9\" to node \"k8s-agentpool1-64138072-vmss000000\"." "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000000" "disk.csi.azure.com/request-id"="2249a4d1-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-2802f37a-394b-486e-88b0-bb86c39febf9" I0201 17:23:24.995738 1 util.go:124] Send.sendRequest got response with ContentLength -1, StatusCode 200 and responseBody length 1225 I0201 17:23:24.995925 1 cloudprovisioner.go:405] "msg"="GetDiskLun returned: -1. Initiating attaching volume \"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-96oxu0dv/providers/Microsoft.Compute/disks/pvc-5f7b8fd9-c2af-41d7-9247-71e90b7a7305\" to node \"k8s-agentpool1-64138072-vmss000001\"." "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000001" "disk.csi.azure.com/request-id"="22486023-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-5f7b8fd9-c2af-41d7-9247-71e90b7a7305" I0201 17:23:24.995954 1 cloudprovisioner.go:424] "msg"="Trying to attach volume \"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-96oxu0dv/providers/Microsoft.Compute/disks/pvc-5f7b8fd9-c2af-41d7-9247-71e90b7a7305\" to node \"k8s-agentpool1-64138072-vmss000001\"." "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000001" "disk.csi.azure.com/request-id"="22486023-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-5f7b8fd9-c2af-41d7-9247-71e90b7a7305" I0201 17:23:25.995375 1 batch.go:224] "cloud-provider-azure: Delayed processing of batch due to start delay" type="batch" operation="attach_disk" key="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e|kubetest-96oxu0dv|k8s-agentpool1-64138072-vmss000000" delay="1s" I0201 17:23:25.995476 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-96oxu0dv): vm(k8s-agentpool1-64138072-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-96oxu0dv/providers/Microsoft.Compute/disks/pvc-2802f37a-394b-486e-88b0-bb86c39febf9:AttachDiskOptions{diskName: "pvc-2802f37a-394b-486e-88b0-bb86c39febf9", lun: 1}]) I0201 17:23:25.996514 1 batch.go:224] "cloud-provider-azure: Delayed processing of batch due to start delay" type="batch" operation="attach_disk" key="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e|kubetest-96oxu0dv|k8s-agentpool1-64138072-vmss000001" delay="1s" I0201 17:23:25.996588 1 azure_controller_vmss.go:110] azureDisk - update(kubetest-96oxu0dv): vm(k8s-agentpool1-64138072-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-96oxu0dv/providers/Microsoft.Compute/disks/pvc-5f7b8fd9-c2af-41d7-9247-71e90b7a7305:AttachDiskOptions{diskName: "pvc-5f7b8fd9-c2af-41d7-9247-71e90b7a7305", lun: 1}]) I0201 17:23:26.004961 1 azure_disk_utils.go:1103] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3.4" "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000001" "disk.csi.azure.com/request-id"="22486023-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-5f7b8fd9-c2af-41d7-9247-71e90b7a7305" "latency"=8217545 "number_of_net_retry"=0 "number_of_retry"=0 I0201 17:23:26.005504 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0201 17:23:26.005570 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000001" "disk.csi.azure.com/request-id"="22486023-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-5f7b8fd9-c2af-41d7-9247-71e90b7a7305" "latency"=1047085740 I0201 17:23:26.005613 1 crdprovisioner.go:727] "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000001" "disk.csi.azure.com/request-id"="22486023-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLun" "disk.csi.azure.com/volume-name"="pvc-5f7b8fd9-c2af-41d7-9247-71e90b7a7305" "latency"=1047205342 I0201 17:23:26.005635 1 crdprovisioner.go:486] "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000001" "disk.csi.azure.com/request-id"="22486023-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).PublishVolume" "disk.csi.azure.com/volume-name"="pvc-5f7b8fd9-c2af-41d7-9247-71e90b7a7305" "latency"=1061437395 I0201 17:23:26.005685 1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=1.061511497 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-96oxu0dv" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="test.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-96oxu0dv/providers/Microsoft.Compute/disks/pvc-5f7b8fd9-c2af-41d7-9247-71e90b7a7305" node="k8s-agentpool1-64138072-vmss000001" result_code="succeeded" I0201 17:23:26.005754 1 utils.go:85] GRPC response: {"publish_context":{"LUN":"1"}} I0201 17:23:26.005841 1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-5f7b8fd9-c2af-41d7-9247-71e90b7a7305-k8s-agentpool1-64138072-vmss000001-attachment) I0201 17:23:26.013227 1 azure_disk_utils.go:1103] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3.4" "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000000" "disk.csi.azure.com/request-id"="2249a4d1-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-2802f37a-394b-486e-88b0-bb86c39febf9" "latency"=17653513 "number_of_net_retry"=0 "number_of_retry"=0 I0201 17:23:26.014743 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0201 17:23:26.014803 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000000" "disk.csi.azure.com/request-id"="2249a4d1-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-2802f37a-394b-486e-88b0-bb86c39febf9" "latency"=1043562478 I0201 17:23:26.014831 1 crdprovisioner.go:727] "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000000" "disk.csi.azure.com/request-id"="2249a4d1-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLun" "disk.csi.azure.com/volume-name"="pvc-2802f37a-394b-486e-88b0-bb86c39febf9" "latency"=1043665080 I0201 17:23:26.014883 1 crdprovisioner.go:486] "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000000" "disk.csi.azure.com/request-id"="2249a4d1-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).PublishVolume" "disk.csi.azure.com/volume-name"="pvc-2802f37a-394b-486e-88b0-bb86c39febf9" "latency"=1062370312 I0201 17:23:26.014928 1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=1.062436812 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-96oxu0dv" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="test.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-96oxu0dv/providers/Microsoft.Compute/disks/pvc-2802f37a-394b-486e-88b0-bb86c39febf9" node="k8s-agentpool1-64138072-vmss000000" result_code="succeeded" I0201 17:23:26.014949 1 utils.go:85] GRPC response: {"publish_context":{"LUN":"1"}} I0201 17:23:26.015119 1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-2802f37a-394b-486e-88b0-bb86c39febf9-k8s-agentpool1-64138072-vmss000000-attachment) I0201 17:23:26.170963 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-96oxu0dv): vm(k8s-agentpool1-64138072-vmss000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-96oxu0dv/providers/Microsoft.Compute/disks/pvc-5f7b8fd9-c2af-41d7-9247-71e90b7a7305:AttachDiskOptions{diskName: "pvc-5f7b8fd9-c2af-41d7-9247-71e90b7a7305", lun: 1}], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I0201 17:23:26.171012 1 azure_controller_common.go:344] azuredisk - trying to attach disks to node k8s-agentpool1-64138072-vmss000001: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-96oxu0dv/providers/Microsoft.Compute/disks/pvc-5f7b8fd9-c2af-41d7-9247-71e90b7a7305:AttachDiskOptions{diskName: "pvc-5f7b8fd9-c2af-41d7-9247-71e90b7a7305", lun: 1}] I0201 17:23:26.757187 1 azure_controller_vmss.go:122] azureDisk - update(kubetest-96oxu0dv): vm(k8s-agentpool1-64138072-vmss000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-96oxu0dv/providers/Microsoft.Compute/disks/pvc-2802f37a-394b-486e-88b0-bb86c39febf9:AttachDiskOptions{diskName: "pvc-2802f37a-394b-486e-88b0-bb86c39febf9", lun: 1}], %!s(*retry.Error=<nil>)) returned with %!v(MISSING) I0201 17:23:26.757230 1 azure_controller_common.go:344] azuredisk - trying to attach disks to node k8s-agentpool1-64138072-vmss000000: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-96oxu0dv/providers/Microsoft.Compute/disks/pvc-2802f37a-394b-486e-88b0-bb86c39febf9:AttachDiskOptions{diskName: "pvc-2802f37a-394b-486e-88b0-bb86c39febf9", lun: 1}] I0201 17:23:26.929341 1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller I0201 17:23:28.937475 1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller I0201 17:23:30.945282 1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller I0201 17:23:32.953187 1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller I0201 17:23:34.960532 1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller I0201 17:23:36.967166 1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller I0201 17:23:38.334753 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0201 17:23:38.334814 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0201 17:23:38.334838 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0201 17:23:38.334927 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0201 17:23:38.335020 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0201 17:23:38.335056 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0201 17:23:38.338359 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0201 17:23:38.975759 1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller I0201 17:23:39.333416 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-96oxu0dv, k8s-agentpool1-64138072-vmss, k8s-agentpool1-64138072-vmss000002) successfully I0201 17:23:39.333446 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool1-64138072-vmss, kubetest-96oxu0dv, k8s-agentpool1-64138072-vmss000002) for cacheKey(kubetest-96oxu0dv/k8s-agentpool1-64138072-vmss) updated successfully I0201 17:23:39.333515 1 cloudprovisioner.go:448] "msg"="attach operation successful: volume \"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-96oxu0dv/providers/Microsoft.Compute/disks/pvc-72c7ff8f-6519-4968-b40e-0a925c2f0891\" attached to node \"k8s-agentpool1-64138072-vmss000002\"." "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000002" "disk.csi.azure.com/request-id"="21236f6c-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-72c7ff8f-6519-4968-b40e-0a925c2f0891" I0201 17:23:39.333571 1 cloudprovisioner.go:440] "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000002" "disk.csi.azure.com/request-id"="21236f6c-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).PublishVolume.func3" "disk.csi.azure.com/volume-name"="pvc-72c7ff8f-6519-4968-b40e-0a925c2f0891" "latency"=16275913280 I0201 17:23:39.333609 1 workflow.go:155] "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000002" "disk.csi.azure.com/request-id"="21236f6c-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).PublishVolume" "disk.csi.azure.com/volume-name"="pvc-72c7ff8f-6519-4968-b40e-0a925c2f0891" "latency"=16298618051 I0201 17:23:39.333663 1 azure_disk_utils.go:1165] "msg"="Skip update. No update needed." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).updateVolumeAttachmentWithResult" "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000002" "disk.csi.azure.com/request-id"="21236f6c-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-72c7ff8f-6519-4968-b40e-0a925c2f0891" I0201 17:23:39.333703 1 azure_disk_utils.go:1103] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).updateVolumeAttachmentWithResult" "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000002" "disk.csi.azure.com/request-id"="21236f6c-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-72c7ff8f-6519-4968-b40e-0a925c2f0891" "latency"=105802 "number_of_net_retry"=0 "number_of_retry"=0 I0201 17:23:39.343691 1 azure_disk_utils.go:1103] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3.4" "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000002" "disk.csi.azure.com/request-id"="21236f6c-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-72c7ff8f-6519-4968-b40e-0a925c2f0891" "latency"=9945506 "number_of_net_retry"=0 "number_of_retry"=0 I0201 17:23:39.343754 1 attach_detach.go:198] "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000002" "disk.csi.azure.com/request-id"="21236f6c-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3" "disk.csi.azure.com/volume-name"="pvc-72c7ff8f-6519-4968-b40e-0a925c2f0891" "latency"=16308759962 I0201 17:23:39.343779 1 workflow.go:155] "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000002" "disk.csi.azure.com/request-id"="21236f6c-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach" "disk.csi.azure.com/volume-name"="pvc-72c7ff8f-6519-4968-b40e-0a925c2f0891" "latency"=16314317277 I0201 17:23:39.344200 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0201 17:23:39.344252 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000002" "disk.csi.azure.com/request-id"="21c28bce-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-72c7ff8f-6519-4968-b40e-0a925c2f0891" "latency"=15277018855 I0201 17:23:39.344283 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).ControllerPublishVolume.func2" "disk.csi.azure.com/request-id"="21c28bce-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=15277136457 I0201 17:23:39.344329 1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=16.319955494 request="azuredisk_csi_driver_attach_volume_latency" resource_group="kubetest-96oxu0dv" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="test.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-96oxu0dv/providers/Microsoft.Compute/disks/pvc-72c7ff8f-6519-4968-b40e-0a925c2f0891" node="k8s-agentpool1-64138072-vmss000002" result_code="succeeded" I0201 17:23:39.344357 1 controllerserver_v2.go:264] "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="21c28bce-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).ControllerPublishVolume.func2" "latency"=15277224059 I0201 17:23:40.983325 1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller I0201 17:23:41.359553 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-96oxu0dv, k8s-agentpool1-64138072-vmss, k8s-agentpool1-64138072-vmss000001) successfully ... skipping 3 lines ... I0201 17:23:41.359755 1 workflow.go:155] "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000001" "disk.csi.azure.com/request-id"="22486023-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).PublishVolume" "disk.csi.azure.com/volume-name"="pvc-5f7b8fd9-c2af-41d7-9247-71e90b7a7305" "latency"=16385787366 I0201 17:23:41.359906 1 azure_disk_utils.go:1165] "msg"="Skip update. No update needed." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).updateVolumeAttachmentWithResult" "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000001" "disk.csi.azure.com/request-id"="22486023-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-5f7b8fd9-c2af-41d7-9247-71e90b7a7305" I0201 17:23:41.359961 1 azure_disk_utils.go:1103] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).updateVolumeAttachmentWithResult" "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000001" "disk.csi.azure.com/request-id"="22486023-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-5f7b8fd9-c2af-41d7-9247-71e90b7a7305" "latency"=126903 "number_of_net_retry"=0 "number_of_retry"=0 I0201 17:23:41.369106 1 azure_disk_utils.go:1103] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3.4" "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000001" "disk.csi.azure.com/request-id"="22486023-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-5f7b8fd9-c2af-41d7-9247-71e90b7a7305" "latency"=9094187 "number_of_net_retry"=0 "number_of_retry"=0 I0201 17:23:41.369145 1 attach_detach.go:198] "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000001" "disk.csi.azure.com/request-id"="22486023-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3" "disk.csi.azure.com/volume-name"="pvc-5f7b8fd9-c2af-41d7-9247-71e90b7a7305" "latency"=16395196359 I0201 17:23:41.369164 1 workflow.go:155] "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000001" "disk.csi.azure.com/request-id"="22486023-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach" "disk.csi.azure.com/volume-name"="pvc-5f7b8fd9-c2af-41d7-9247-71e90b7a7305" "latency"=16411748753 I0201 17:23:41.371023 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0201 17:23:41.371069 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000001" "disk.csi.azure.com/request-id"="22ea5c96-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-5f7b8fd9-c2af-41d7-9247-71e90b7a7305" "latency"=15365156626 I0201 17:23:41.371094 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).ControllerPublishVolume.func2" "disk.csi.azure.com/request-id"="22ea5c96-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=15365295129 I0201 17:23:41.371128 1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=16.426954128 request="azuredisk_csi_driver_attach_volume_latency" resource_group="kubetest-96oxu0dv" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="test.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-96oxu0dv/providers/Microsoft.Compute/disks/pvc-5f7b8fd9-c2af-41d7-9247-71e90b7a7305" node="k8s-agentpool1-64138072-vmss000001" result_code="succeeded" I0201 17:23:41.371152 1 controllerserver_v2.go:264] "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="22ea5c96-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).ControllerPublishVolume.func2" "latency"=15365367430 I0201 17:23:41.911142 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-96oxu0dv, k8s-agentpool1-64138072-vmss, k8s-agentpool1-64138072-vmss000000) successfully I0201 17:23:41.911174 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool1-64138072-vmss, kubetest-96oxu0dv, k8s-agentpool1-64138072-vmss000000) for cacheKey(kubetest-96oxu0dv/k8s-agentpool1-64138072-vmss) updated successfully I0201 17:23:41.911269 1 cloudprovisioner.go:448] "msg"="attach operation successful: volume \"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-96oxu0dv/providers/Microsoft.Compute/disks/pvc-2802f37a-394b-486e-88b0-bb86c39febf9\" attached to node \"k8s-agentpool1-64138072-vmss000000\"." "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000000" "disk.csi.azure.com/request-id"="2249a4d1-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-2802f37a-394b-486e-88b0-bb86c39febf9" I0201 17:23:41.911327 1 cloudprovisioner.go:440] "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000000" "disk.csi.azure.com/request-id"="2249a4d1-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).PublishVolume.func3" "disk.csi.azure.com/volume-name"="pvc-2802f37a-394b-486e-88b0-bb86c39febf9" "latency"=16916259769 I0201 17:23:41.911363 1 workflow.go:155] "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000000" "disk.csi.azure.com/request-id"="2249a4d1-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).PublishVolume" "disk.csi.azure.com/volume-name"="pvc-2802f37a-394b-486e-88b0-bb86c39febf9" "latency"=16937268941 I0201 17:23:41.911444 1 azure_disk_utils.go:1165] "msg"="Skip update. No update needed." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).updateVolumeAttachmentWithResult" "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000000" "disk.csi.azure.com/request-id"="2249a4d1-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-2802f37a-394b-486e-88b0-bb86c39febf9" I0201 17:23:41.911497 1 azure_disk_utils.go:1103] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).updateVolumeAttachmentWithResult" "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000000" "disk.csi.azure.com/request-id"="2249a4d1-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-2802f37a-394b-486e-88b0-bb86c39febf9" "latency"=131402 "number_of_net_retry"=0 "number_of_retry"=0 I0201 17:23:41.919100 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0201 17:23:41.919163 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000000" "disk.csi.azure.com/request-id"="22ebc75d-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-2802f37a-394b-486e-88b0-bb86c39febf9" "latency"=15903963567 I0201 17:23:41.919195 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).ControllerPublishVolume.func2" "disk.csi.azure.com/request-id"="22ebc75d-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=15904104470 I0201 17:23:41.919230 1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=16.966734086 request="azuredisk_csi_driver_attach_volume_latency" resource_group="kubetest-96oxu0dv" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="test.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-96oxu0dv/providers/Microsoft.Compute/disks/pvc-2802f37a-394b-486e-88b0-bb86c39febf9" node="k8s-agentpool1-64138072-vmss000000" result_code="succeeded" I0201 17:23:41.919260 1 controllerserver_v2.go:264] "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="22ebc75d-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).ControllerPublishVolume.func2" "latency"=15904195072 I0201 17:23:41.920531 1 azure_disk_utils.go:1103] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3.4" "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000000" "disk.csi.azure.com/request-id"="2249a4d1-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-2802f37a-394b-486e-88b0-bb86c39febf9" "latency"=8990085 "number_of_net_retry"=0 "number_of_retry"=0 I0201 17:23:41.920566 1 attach_detach.go:198] "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000000" "disk.csi.azure.com/request-id"="2249a4d1-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3" "disk.csi.azure.com/volume-name"="pvc-2802f37a-394b-486e-88b0-bb86c39febf9" "latency"=16946495932 ... skipping 60 lines ... I0201 17:24:25.058444 1 utils.go:78] GRPC call: /csi.v1.Controller/ControllerUnpublishVolume I0201 17:24:25.058467 1 utils.go:79] GRPC request: {"node_id":"k8s-agentpool1-64138072-vmss000001","volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-96oxu0dv/providers/Microsoft.Compute/disks/pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05"} I0201 17:24:25.058646 1 crdprovisioner.go:927] "msg"="Requesting AzVolumeAttachment (pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05-k8s-agentpool1-64138072-vmss000001-attachment) detachment" "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000001" "disk.csi.azure.com/request-id"="e92de632-a254-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05" I0201 17:24:25.064619 1 replica.go:160] "msg"="Garbage collection of AzVolumeAttachments for AzVolume (pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05) scheduled in 5m0s." "disk.csi.azure.com/request-id"="461e058b-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/volume-name"="pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05" I0201 17:24:25.064786 1 azure_disk_utils.go:1103] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).detachVolume" "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000001" "disk.csi.azure.com/request-id"="e92de632-a254-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05" "latency"=6082522 "number_of_net_retry"=0 "number_of_retry"=0 I0201 17:24:25.064849 1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05-k8s-agentpool1-64138072-vmss000001-attachment) I0201 17:24:25.081302 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0201 17:24:25.083467 1 azure_disk_utils.go:1103] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach" "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000001" "disk.csi.azure.com/request-id"="e92de632-a254-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05" "latency"=18786778 "number_of_net_retry"=0 "number_of_retry"=0 I0201 17:24:25.083495 1 attach_detach.go:340] "msg"="Detaching volume" "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000001" "disk.csi.azure.com/request-id"="e92de632-a254-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05" I0201 17:24:25.083577 1 cloudprovisioner.go:481] "msg"="Trying to detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-96oxu0dv/providers/Microsoft.Compute/disks/pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05 from node k8s-agentpool1-64138072-vmss000001" "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000001" "disk.csi.azure.com/request-id"="e92de632-a254-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05" I0201 17:24:25.162136 1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller I0201 17:24:26.084579 1 batch.go:224] "cloud-provider-azure: Delayed processing of batch due to start delay" type="batch" operation="detach_disk" key="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e|kubetest-96oxu0dv|k8s-agentpool1-64138072-vmss000001" delay="1s" I0201 17:24:26.084632 1 azure_controller_common.go:461] azuredisk - trying to detach disks from node k8s-agentpool1-64138072-vmss000001: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-96oxu0dv/providers/Microsoft.Compute/disks/pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05:pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05] ... skipping 5 lines ... I0201 17:24:33.192644 1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller I0201 17:24:35.199836 1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller I0201 17:24:37.206669 1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller I0201 17:24:38.335948 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0201 17:24:38.335994 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0201 17:24:38.336018 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0201 17:24:38.336053 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0201 17:24:38.339402 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0201 17:24:39.214054 1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller I0201 17:24:41.222331 1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller I0201 17:24:41.423399 1 azure_controller_vmss.go:252] azureDisk - update(kubetest-96oxu0dv): vm(k8s-agentpool1-64138072-vmss000001) - detach disk(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-96oxu0dv/providers/Microsoft.Compute/disks/pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05:pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05]) returned with <nil> I0201 17:24:41.423449 1 azure_vmss_cache.go:275] DeleteCacheForNode(kubetest-96oxu0dv, k8s-agentpool1-64138072-vmss, k8s-agentpool1-64138072-vmss000001) successfully I0201 17:24:41.423461 1 azure_vmss_cache.go:313] updateCache(k8s-agentpool1-64138072-vmss, kubetest-96oxu0dv, k8s-agentpool1-64138072-vmss000001) for cacheKey(kubetest-96oxu0dv/k8s-agentpool1-64138072-vmss) updated successfully I0201 17:24:41.423468 1 azure_controller_common.go:473] azuredisk - successfully detached disks from node k8s-agentpool1-64138072-vmss000001: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-96oxu0dv/providers/Microsoft.Compute/disks/pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05:pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05] I0201 17:24:41.423503 1 azure_controller_common.go:434] azureDisk - detach disk(pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05, /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-96oxu0dv/providers/Microsoft.Compute/disks/pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05) succeeded I0201 17:24:41.423536 1 cloudprovisioner.go:491] "msg"="detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-96oxu0dv/providers/Microsoft.Compute/disks/pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05 from node k8s-agentpool1-64138072-vmss000001 successfully" "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000001" "disk.csi.azure.com/request-id"="e92de632-a254-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05" I0201 17:24:41.423565 1 cloudprovisioner.go:471] "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000001" "disk.csi.azure.com/request-id"="e92de632-a254-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).UnpublishVolume" "disk.csi.azure.com/volume-name"="pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05" "latency"=16340010882 I0201 17:24:41.431811 1 attach_detach.go:346] "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000001" "disk.csi.azure.com/request-id"="e92de632-a254-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach.func3" "disk.csi.azure.com/volume-name"="pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05" "latency"=16348259950 I0201 17:24:41.431848 1 workflow.go:155] "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000001" "disk.csi.azure.com/request-id"="e92de632-a254-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach" "disk.csi.azure.com/volume-name"="pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05" "latency"=16367193031 I0201 17:24:41.432674 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0201 17:24:41.443472 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0201 17:24:41.443519 1 azure_disk_utils.go:1103] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).removeFinalizer" "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000001" "disk.csi.azure.com/request-id"="e92de632-a254-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05" "latency"=10502714 "number_of_net_retry"=0 "number_of_retry"=0 I0201 17:24:41.443529 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForDetach" "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000001" "disk.csi.azure.com/request-id"="e92de632-a254-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05" "latency"=16378619964 I0201 17:24:41.443553 1 attach_detach.go:379] "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000001" "disk.csi.azure.com/request-id"="e92de632-a254-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).removeFinalizer" "disk.csi.azure.com/volume-name"="pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05" "latency"=10564216 I0201 17:24:41.443568 1 crdprovisioner.go:938] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).detachVolume" "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000001" "disk.csi.azure.com/request-id"="e92de632-a254-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForDetach" "disk.csi.azure.com/volume-name"="pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05" "latency"=16378727167 I0201 17:24:41.443597 1 crdprovisioner.go:836] "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000001" "disk.csi.azure.com/request-id"="461d1bf9-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).UnpublishVolume" "disk.csi.azure.com/volume-name"="pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05" "latency"=16384967293 I0201 17:24:41.443638 1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=16.385037094 request="azuredisk_csi_driver_controller_unpublish_volume" resource_group="kubetest-96oxu0dv" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="test.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-96oxu0dv/providers/Microsoft.Compute/disks/pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05" node="k8s-agentpool1-64138072-vmss000001" result_code="succeeded" ... skipping 10 lines ... I0201 17:24:49.249535 1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller I0201 17:24:50.025188 1 utils.go:78] GRPC call: /csi.v1.Controller/DeleteVolume I0201 17:24:50.025212 1 utils.go:79] GRPC request: {"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-96oxu0dv/providers/Microsoft.Compute/disks/pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05"} I0201 17:24:50.025293 1 controllerserver_v2.go:201] deleting disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-96oxu0dv/providers/Microsoft.Compute/disks/pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05) I0201 17:24:50.042843 1 azure_disk_utils.go:1103] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).DeleteVolume" "csi.storage.k8s.io/pv/name"="pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05" "disk.csi.azure.com/request-id"="54febc21-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05" "latency"=17427150 "number_of_net_retry"=0 "number_of_retry"=0 I0201 17:24:50.042866 1 conditionwatcher.go:99] Adding a condition function for azvolume (pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05) I0201 17:24:50.051969 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0201 17:24:50.066194 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0201 17:24:50.066331 1 azure_disk_utils.go:1103] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerDelete" "csi.storage.k8s.io/pv/name"="pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05" "disk.csi.azure.com/request-id"="e7599845-a254-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05" "latency"=14464191 "number_of_net_retry"=0 "number_of_retry"=0 I0201 17:24:50.066361 1 azvolume.go:246] "msg"="Deleting Volume..." "csi.storage.k8s.io/pv/name"="pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05" "disk.csi.azure.com/request-id"="e7599845-a254-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/volume-name"="pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05" I0201 17:24:50.066444 1 shared_state.go:1037] "msg"="AzVolumeAttachment clean up requested by azvolume-controller for AzVolume (pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05)" "csi.storage.k8s.io/pv/name"="pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05" "disk.csi.azure.com/request-id"="e7599845-a254-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/volume-name"="pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05" I0201 17:24:50.066475 1 azure_disk_utils.go:1039] "msg"="Getting AzVolumeAttachment list for volume (pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05)" "csi.storage.k8s.io/pv/name"="pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05" "disk.csi.azure.com/request-id"="e7599845-a254-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/volume-name"="pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05" I0201 17:24:50.066518 1 azure_disk_utils.go:1068] "msg"="Label selector is: disk.csi.azure.com/volume-name=pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05." "csi.storage.k8s.io/pv/name"="pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05" "disk.csi.azure.com/request-id"="e7599845-a254-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/volume-name"="pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05" I0201 17:24:50.066576 1 shared_state.go:1035] "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05" "disk.csi.azure.com/request-id"="e7599845-a254-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*SharedState).cleanUpAzVolumeAttachmentByVolume" "disk.csi.azure.com/volume-name"="pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05" "latency"=130802 ... skipping 9 lines ... I0201 17:24:51.818532 1 shared_state.go:198] "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="56104f5f-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*SharedState).addToOperationQueue" "disk.csi.azure.com/volume-name"="pvc-5557970e-c6fd-4e7f-9153-ca2f088888ba" "latency"=230605 I0201 17:24:51.818557 1 workflow.go:155] "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="56104f5f-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcilePV).triggerRelease" "disk.csi.azure.com/volume-name"="pvc-5557970e-c6fd-4e7f-9153-ca2f088888ba" "latency"=266905 I0201 17:24:53.267728 1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller I0201 17:24:55.276585 1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller I0201 17:24:55.378715 1 azure_managedDiskController.go:317] azureDisk - deleted a managed disk: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-96oxu0dv/providers/Microsoft.Compute/disks/pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05 I0201 17:24:55.378777 1 cloudprovisioner.go:337] "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05" "disk.csi.azure.com/request-id"="e7599845-a254-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).DeleteVolume" "disk.csi.azure.com/volume-name"="pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05" "latency"=5312139361 I0201 17:24:55.391236 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0201 17:24:55.391287 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).DeleteVolume" "csi.storage.k8s.io/pv/name"="pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05" "disk.csi.azure.com/request-id"="54febc21-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05" "latency"=5339762618 I0201 17:24:55.391315 1 crdprovisioner.go:375] "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05" "disk.csi.azure.com/request-id"="54febc21-a255-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).DeleteVolume" "disk.csi.azure.com/volume-name"="pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05" "latency"=5365929145 I0201 17:24:55.391323 1 controllerserver_v2.go:203] delete disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-96oxu0dv/providers/Microsoft.Compute/disks/pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05) returned with <nil> I0201 17:24:55.391354 1 azure_metrics.go:117] "Observed Request Latency" latency_seconds=5.366040546 request="azuredisk_csi_driver_controller_delete_volume" resource_group="kubetest-96oxu0dv" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="test.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-96oxu0dv/providers/Microsoft.Compute/disks/pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05" result_code="succeeded" I0201 17:24:55.391364 1 utils.go:85] GRPC response: {} I0201 17:24:55.395328 1 azure_disk_utils.go:1103] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerDelete.func4" "csi.storage.k8s.io/pv/name"="pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05" "disk.csi.azure.com/request-id"="e7599845-a254-11ed-abc7-6045bd9323b2" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05" "latency"=16500035 "number_of_net_retry"=0 "number_of_retry"=0 ... skipping 41 lines ... I0201 17:21:20.799397 1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumes?allowWatchBookmarks=true&resourceVersion=1324&timeout=6m26s&timeoutSeconds=386&watch=true 200 OK in 4 milliseconds I0201 17:21:20.799679 1 reflector.go:338] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: watch of *v1beta2.AzVolume closed with: too old resource version: 1324 (1325) I0201 17:21:20.874733 1 shared_informer.go:303] caches populated I0201 17:21:20.874766 1 azuredisk_v2.go:225] driver userAgent: test.csi.azure.com/latest-v2-8ab762a4766ca4352f3bbe3e0c05b326019e12ba e2e-test I0201 17:21:20.874779 1 azure_disk_utils.go:564] reading cloud config from secret kube-system/azure-cloud-provider I0201 17:21:20.876688 1 round_trippers.go:553] GET https://10.0.0.1:443/api/v1/namespaces/kube-system/secrets/azure-cloud-provider 404 Not Found in 1 milliseconds I0201 17:21:20.876835 1 azure_disk_utils.go:571] InitializeCloudFromSecret: failed to get cloud config from secret kube-system/azure-cloud-provider: failed to get secret kube-system/azure-cloud-provider: secrets "azure-cloud-provider" not found I0201 17:21:20.876846 1 azure_disk_utils.go:576] could not read cloud config from secret kube-system/azure-cloud-provider I0201 17:21:20.876855 1 azure_disk_utils.go:586] use default AZURE_CREDENTIAL_FILE env var: /etc/kubernetes/azure.json I0201 17:21:20.876888 1 azure_disk_utils.go:594] read cloud config from file: /etc/kubernetes/azure.json successfully I0201 17:21:20.877701 1 azure_auth.go:253] Using AzurePublicCloud environment I0201 17:21:20.877726 1 azure_auth.go:104] azure: using managed identity extension to retrieve access token I0201 17:21:20.877732 1 azure_auth.go:110] azure: using User Assigned MSI ID to retrieve access token ... skipping 53 lines ... I0201 17:21:21.001856 1 round_trippers.go:553] POST https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes 201 Created in 2 milliseconds I0201 17:21:21.006426 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-master-64138072-0/status 200 OK in 4 milliseconds I0201 17:21:21.006869 1 azuredisk_v2.go:554] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-master-64138072-0" "disk.csi.azure.com/request-id"="d867c084-a254-11ed-8b93-6045bd8960eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).registerAzDriverNode" "latency"=8411675 I0201 17:21:21.007190 1 server.go:117] Listening for connections on address: &net.UnixAddr{Name:"//csi/csi.sock", Net:"unix"} I0201 17:21:21.007324 1 azuredisk_v2.go:578] "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-02-01T17:21:21Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-master-64138072-0" "disk.csi.azure.com/request-id"="d8690f24-a254-11ed-8b93-6045bd8960eb" I0201 17:21:21.009961 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-master-64138072-0/status 409 Conflict in 2 milliseconds E0201 17:21:21.010207 1 azuredisk_v2.go:613] "msg"="Failed to update AzDriverNode status after creation" "error"="Operation cannot be fulfilled on azdrivernodes.disk.csi.azure.com \"k8s-master-64138072-0\": the object has been modified; please apply your changes to the latest version and try again" "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-master-64138072-0" "disk.csi.azure.com/request-id"="d8690f24-a254-11ed-8b93-6045bd8960eb" E0201 17:21:21.010257 1 azuredisk_v2.go:565] "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [Operation cannot be fulfilled on azdrivernodes.disk.csi.azure.com \"k8s-master-64138072-0\": the object has been modified; please apply your changes to the latest version and try again]" "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-master-64138072-0" "disk.csi.azure.com/request-id"="d8690f24-a254-11ed-8b93-6045bd8960eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=3244829 I0201 17:21:21.010285 1 azuredisk_v2.go:528] Starting heartbeat loop with initial delay 15.312s and frequency 30s I0201 17:21:21.359773 1 utils.go:78] GRPC call: /csi.v1.Identity/GetPluginInfo I0201 17:21:21.359798 1 utils.go:79] GRPC request: {} I0201 17:21:21.363908 1 utils.go:85] GRPC response: {"name":"test.csi.azure.com","vendor_version":"latest-v2-8ab762a4766ca4352f3bbe3e0c05b326019e12ba"} I0201 17:21:21.677100 1 utils.go:78] GRPC call: /csi.v1.Identity/GetPluginInfo I0201 17:21:21.677226 1 utils.go:79] GRPC request: {} ... skipping 116 lines ... I0201 17:21:20.797152 1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumes?limit=500&resourceVersion=0 200 OK in 245 milliseconds I0201 17:21:20.800566 1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumes?allowWatchBookmarks=true&resourceVersion=1325&timeout=9m54s&timeoutSeconds=594&watch=true 200 OK in 2 milliseconds I0201 17:21:20.851834 1 shared_informer.go:303] caches populated I0201 17:21:20.851885 1 azuredisk_v2.go:225] driver userAgent: test.csi.azure.com/latest-v2-8ab762a4766ca4352f3bbe3e0c05b326019e12ba e2e-test I0201 17:21:20.851896 1 azure_disk_utils.go:564] reading cloud config from secret kube-system/azure-cloud-provider I0201 17:21:20.855809 1 round_trippers.go:553] GET https://10.0.0.1:443/api/v1/namespaces/kube-system/secrets/azure-cloud-provider 404 Not Found in 3 milliseconds I0201 17:21:20.856007 1 azure_disk_utils.go:571] InitializeCloudFromSecret: failed to get cloud config from secret kube-system/azure-cloud-provider: failed to get secret kube-system/azure-cloud-provider: secrets "azure-cloud-provider" not found I0201 17:21:20.856023 1 azure_disk_utils.go:576] could not read cloud config from secret kube-system/azure-cloud-provider I0201 17:21:20.856029 1 azure_disk_utils.go:586] use default AZURE_CREDENTIAL_FILE env var: /etc/kubernetes/azure.json I0201 17:21:20.856056 1 azure_disk_utils.go:594] read cloud config from file: /etc/kubernetes/azure.json successfully I0201 17:21:20.856695 1 azure_auth.go:253] Using AzurePublicCloud environment I0201 17:21:20.856714 1 azure_auth.go:104] azure: using managed identity extension to retrieve access token I0201 17:21:20.856719 1 azure_auth.go:110] azure: using User Assigned MSI ID to retrieve access token ... skipping 84 lines ... I0201 17:21:52.354464 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities I0201 17:21:52.354481 1 utils.go:79] GRPC request: {} I0201 17:21:52.354513 1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]} I0201 17:21:52.355512 1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume I0201 17:21:52.355528 1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05","csi.storage.k8s.io/pvc/name":"pvc-azuredisk","csi.storage.k8s.io/pvc/namespace":"default","requestedsizegib":"10","skuName":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1675272099190-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-96oxu0dv/providers/Microsoft.Compute/disks/pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05"} I0201 17:21:52.355766 1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05-k8s-agentpool1-64138072-vmss000001-attachment) I0201 17:22:01.181154 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0201 17:22:01.181234 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000001" "disk.csi.azure.com/request-id"="eb187fd7-a254-11ed-aac2-6045bd932703" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05" "latency"=8825400063 I0201 17:22:01.181265 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="eb187fd7-a254-11ed-aac2-6045bd932703" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=8825511966 I0201 17:22:02.850273 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/ I0201 17:22:02.850329 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType StandardSSD_LRS I0201 17:22:02.850365 1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05/globalmount with mount options([]) I0201 17:22:02.850381 1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0]) ... skipping 85 lines ... I0201 17:23:36.047473 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities I0201 17:23:36.047494 1 utils.go:79] GRPC request: {} I0201 17:23:36.047529 1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]} I0201 17:23:36.048255 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetVolumeStats I0201 17:23:36.048267 1 utils.go:79] GRPC request: {"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-96oxu0dv/providers/Microsoft.Compute/disks/pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05","volume_path":"/var/lib/kubelet/pods/dbc71cf9-1cc4-445a-a2ad-607a6710ed34/volumes/kubernetes.io~csi/pvc-f9b4293f-d46e-4754-9605-2fd02bf23d05/mount"} I0201 17:23:36.048346 1 utils.go:85] GRPC response: {"usage":[{"available":10447216640,"total":10464022528,"unit":1,"used":28672},{"available":655348,"total":655360,"unit":2,"used":12}]} I0201 17:23:41.372320 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0201 17:23:41.372396 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000001" "disk.csi.azure.com/request-id"="27c8a422-a255-11ed-aac2-6045bd932703" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-5f7b8fd9-c2af-41d7-9247-71e90b7a7305" "latency"=7198909629 I0201 17:23:41.372421 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="27c8a422-a255-11ed-aac2-6045bd932703" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=7199016733 I0201 17:23:41.970349 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdd under /dev/disk/azure/scsi1/ I0201 17:23:41.970401 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType StandardSSD_LRS I0201 17:23:41.970435 1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun1 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-5f7b8fd9-c2af-41d7-9247-71e90b7a7305/globalmount with mount options([]) I0201 17:23:41.970453 1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun1" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun1]) ... skipping 115 lines ... I0201 17:21:25.809164 1 round_trippers.go:553] GET https://10.0.0.1:443/api/v1/nodes?allowWatchBookmarks=true&resourceVersion=1351&timeout=9m7s&timeoutSeconds=547&watch=true 200 OK in 0 milliseconds I0201 17:21:25.810569 1 round_trippers.go:553] GET https://10.0.0.1:443/apis/apiextensions.k8s.io/v1/customresourcedefinitions?allowWatchBookmarks=true&resourceVersion=1130&timeout=6m28s&timeoutSeconds=388&watch=true 200 OK in 0 milliseconds I0201 17:21:25.857776 1 shared_informer.go:303] caches populated I0201 17:21:25.857813 1 azuredisk_v2.go:225] driver userAgent: test.csi.azure.com/latest-v2-8ab762a4766ca4352f3bbe3e0c05b326019e12ba e2e-test I0201 17:21:25.857823 1 azure_disk_utils.go:564] reading cloud config from secret kube-system/azure-cloud-provider I0201 17:21:25.861065 1 round_trippers.go:553] GET https://10.0.0.1:443/api/v1/namespaces/kube-system/secrets/azure-cloud-provider 404 Not Found in 3 milliseconds I0201 17:21:25.861350 1 azure_disk_utils.go:571] InitializeCloudFromSecret: failed to get cloud config from secret kube-system/azure-cloud-provider: failed to get secret kube-system/azure-cloud-provider: secrets "azure-cloud-provider" not found I0201 17:21:25.861368 1 azure_disk_utils.go:576] could not read cloud config from secret kube-system/azure-cloud-provider I0201 17:21:25.861374 1 azure_disk_utils.go:586] use default AZURE_CREDENTIAL_FILE env var: /etc/kubernetes/azure.json I0201 17:21:25.861402 1 azure_disk_utils.go:594] read cloud config from file: /etc/kubernetes/azure.json successfully I0201 17:21:25.862059 1 azure_auth.go:253] Using AzurePublicCloud environment I0201 17:21:25.862078 1 azure_auth.go:104] azure: using managed identity extension to retrieve access token I0201 17:21:25.862097 1 azure_auth.go:110] azure: using User Assigned MSI ID to retrieve access token ... skipping 101 lines ... I0201 17:22:35.901038 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I0201 17:22:35.901059 1 utils.go:79] GRPC request: {} I0201 17:22:35.901098 1 utils.go:85] GRPC response: {"ready":{"value":true}} I0201 17:22:37.972598 1 azuredisk_v2.go:578] "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-02-01T17:22:37Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000000" "disk.csi.azure.com/request-id"="06490f16-a255-11ed-b238-6045bd9323b2" I0201 17:22:37.982214 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-64138072-vmss000000/status 200 OK in 9 milliseconds I0201 17:22:37.982458 1 azuredisk_v2.go:565] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000000" "disk.csi.azure.com/request-id"="06490f16-a255-11ed-b238-6045bd9323b2" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=9976815 I0201 17:22:44.999514 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0201 17:22:44.999642 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000000" "disk.csi.azure.com/request-id"="04d35ef6-a255-11ed-b238-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-3251af1b-608b-4fbd-8f12-5159afe840de" "latency"=9475958717 I0201 17:22:44.999684 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="04d35ef6-a255-11ed-b238-6045bd9323b2" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9476214422 I0201 17:22:46.764597 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/ I0201 17:22:46.764640 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType StandardSSD_LRS I0201 17:22:46.764668 1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-3251af1b-608b-4fbd-8f12-5159afe840de/globalmount with mount options([]) I0201 17:22:46.764682 1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0]) ... skipping 62 lines ... I0201 17:23:35.901250 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I0201 17:23:35.901275 1 utils.go:79] GRPC request: {} I0201 17:23:35.901327 1 utils.go:85] GRPC response: {"ready":{"value":true}} I0201 17:23:37.972522 1 azuredisk_v2.go:578] "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-02-01T17:23:37Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000000" "disk.csi.azure.com/request-id"="2a0c552b-a255-11ed-b238-6045bd9323b2" I0201 17:23:37.981367 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-64138072-vmss000000/status 200 OK in 8 milliseconds I0201 17:23:37.981771 1 azuredisk_v2.go:565] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000000" "disk.csi.azure.com/request-id"="2a0c552b-a255-11ed-b238-6045bd9323b2" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=9257811 I0201 17:23:41.919232 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0201 17:23:41.919301 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000000" "disk.csi.azure.com/request-id"="236dbee6-a255-11ed-b238-6045bd9323b2" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-2802f37a-394b-486e-88b0-bb86c39febf9" "latency"=15052344188 I0201 17:23:41.919338 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="236dbee6-a255-11ed-b238-6045bd9323b2" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=15052504091 I0201 17:23:42.542884 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdd under /dev/disk/azure/scsi1/ I0201 17:23:42.542934 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType StandardSSD_LRS I0201 17:23:42.542963 1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun1 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-2802f37a-394b-486e-88b0-bb86c39febf9/globalmount with mount options([]) I0201 17:23:42.542971 1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun1" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun1]) ... skipping 121 lines ... I0201 17:21:20.983185 1 round_trippers.go:553] GET https://10.0.0.1:443/api/v1/nodes?allowWatchBookmarks=true&resourceVersion=963&timeout=8m59s&timeoutSeconds=539&watch=true 200 OK in 0 milliseconds I0201 17:21:20.983622 1 round_trippers.go:553] GET https://10.0.0.1:443/apis/apiextensions.k8s.io/v1/customresourcedefinitions?allowWatchBookmarks=true&resourceVersion=1130&timeout=6m11s&timeoutSeconds=371&watch=true 200 OK in 0 milliseconds I0201 17:21:21.028550 1 shared_informer.go:303] caches populated I0201 17:21:21.028627 1 azuredisk_v2.go:225] driver userAgent: test.csi.azure.com/latest-v2-8ab762a4766ca4352f3bbe3e0c05b326019e12ba e2e-test I0201 17:21:21.028641 1 azure_disk_utils.go:564] reading cloud config from secret kube-system/azure-cloud-provider I0201 17:21:21.030892 1 round_trippers.go:553] GET https://10.0.0.1:443/api/v1/namespaces/kube-system/secrets/azure-cloud-provider 404 Not Found in 2 milliseconds I0201 17:21:21.031230 1 azure_disk_utils.go:571] InitializeCloudFromSecret: failed to get cloud config from secret kube-system/azure-cloud-provider: failed to get secret kube-system/azure-cloud-provider: secrets "azure-cloud-provider" not found I0201 17:21:21.031247 1 azure_disk_utils.go:576] could not read cloud config from secret kube-system/azure-cloud-provider I0201 17:21:21.031253 1 azure_disk_utils.go:586] use default AZURE_CREDENTIAL_FILE env var: /etc/kubernetes/azure.json I0201 17:21:21.031295 1 azure_disk_utils.go:594] read cloud config from file: /etc/kubernetes/azure.json successfully I0201 17:21:21.032094 1 azure_auth.go:253] Using AzurePublicCloud environment I0201 17:21:21.032115 1 azure_auth.go:104] azure: using managed identity extension to retrieve access token I0201 17:21:21.032122 1 azure_auth.go:110] azure: using User Assigned MSI ID to retrieve access token ... skipping 91 lines ... I0201 17:22:13.433769 1 utils.go:79] GRPC request: {} I0201 17:22:13.433806 1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]} I0201 17:22:13.434925 1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume I0201 17:22:13.434941 1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-553191ed-6cf7-4436-a08d-abb08cda4724/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-553191ed-6cf7-4436-a08d-abb08cda4724","csi.storage.k8s.io/pvc/name":"persistent-storage-statefulset-azuredisk-0","csi.storage.k8s.io/pvc/namespace":"default","requestedsizegib":"10","skuName":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1675272099190-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-96oxu0dv/providers/Microsoft.Compute/disks/pvc-553191ed-6cf7-4436-a08d-abb08cda4724"} I0201 17:22:13.435272 1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-553191ed-6cf7-4436-a08d-abb08cda4724-k8s-agentpool1-64138072-vmss000002-attachment) I0201 17:22:20.973775 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0201 17:22:20.973919 1 conditionwatcher.go:173] condition result: succeeded: false, error: <nil> I0201 17:22:20.974856 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0201 17:22:20.980031 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0201 17:22:20.984342 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:132: forcing resync I0201 17:22:21.056370 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:132: forcing resync I0201 17:22:21.496651 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0201 17:22:21.496746 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000002" "disk.csi.azure.com/request-id"="f7a8f9d2-a254-11ed-bff7-6045bd932154" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-553191ed-6cf7-4436-a08d-abb08cda4724" "latency"=8061398992 I0201 17:22:21.496773 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="f7a8f9d2-a254-11ed-bff7-6045bd932154" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=8061526794 I0201 17:22:23.158819 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/ I0201 17:22:23.158877 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType StandardSSD_LRS I0201 17:22:23.158911 1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-553191ed-6cf7-4436-a08d-abb08cda4724/globalmount with mount options([]) I0201 17:22:23.158922 1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0]) ... skipping 57 lines ... I0201 17:23:07.631886 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetVolumeStats I0201 17:23:07.631909 1 utils.go:79] GRPC request: {"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-96oxu0dv/providers/Microsoft.Compute/disks/pvc-553191ed-6cf7-4436-a08d-abb08cda4724","volume_path":"/var/lib/kubelet/pods/1d152f1d-4eaf-47b6-8c4b-70381147e6e1/volumes/kubernetes.io~csi/pvc-553191ed-6cf7-4436-a08d-abb08cda4724/mount"} I0201 17:23:07.632064 1 utils.go:85] GRPC response: {"usage":[{"available":10447216640,"total":10464022528,"unit":1,"used":28672},{"available":655348,"total":655360,"unit":2,"used":12}]} I0201 17:23:10.199326 1 azuredisk_v2.go:578] "msg"="Updating heartbeat" "LastHeartbeatTime"="2023-02-01T17:23:10Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000002" "disk.csi.azure.com/request-id"="197e7985-a255-11ed-bff7-6045bd932154" I0201 17:23:10.206724 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-64138072-vmss000002/status 200 OK in 7 milliseconds I0201 17:23:10.207092 1 azuredisk_v2.go:565] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000002" "disk.csi.azure.com/request-id"="197e7985-a255-11ed-bff7-6045bd932154" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=7795829 I0201 17:23:14.305320 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0201 17:23:14.305407 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000002" "disk.csi.azure.com/request-id"="15ee2214-a255-11ed-bff7-6045bd932154" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-5557970e-c6fd-4e7f-9153-ca2f088888ba" "latency"=10085186253 I0201 17:23:14.305436 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="15ee2214-a255-11ed-bff7-6045bd932154" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=10085312657 I0201 17:23:14.879137 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdd under /dev/disk/azure/scsi1/ I0201 17:23:14.879199 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType StandardSSD_LRS I0201 17:23:14.879242 1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun1 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-5557970e-c6fd-4e7f-9153-ca2f088888ba/globalmount with mount options([]) I0201 17:23:14.879254 1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun1" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun1]) ... skipping 42 lines ... I0201 17:23:24.979848 1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume I0201 17:23:24.979864 1 utils.go:79] GRPC request: {"publish_context":{"LUN":"2"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-72c7ff8f-6519-4968-b40e-0a925c2f0891/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-72c7ff8f-6519-4968-b40e-0a925c2f0891","csi.storage.k8s.io/pvc/name":"daemonset-azuredisk-ephemeral-c5kr6-azuredisk","csi.storage.k8s.io/pvc/namespace":"default","requestedsizegib":"10","skuName":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1675272099190-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-96oxu0dv/providers/Microsoft.Compute/disks/pvc-72c7ff8f-6519-4968-b40e-0a925c2f0891"} I0201 17:23:24.980091 1 conditionwatcher.go:99] Adding a condition function for azvolumeattachments (pvc-72c7ff8f-6519-4968-b40e-0a925c2f0891-k8s-agentpool1-64138072-vmss000002-attachment) I0201 17:23:36.827869 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I0201 17:23:36.827902 1 utils.go:79] GRPC request: {} I0201 17:23:36.827962 1 utils.go:85] GRPC response: {"ready":{"value":true}} I0201 17:23:39.346389 1 conditionwatcher.go:173] condition result: succeeded: true, error: <nil> I0201 17:23:39.346503 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-64138072-vmss000002" "disk.csi.azure.com/request-id"="224dd920-a255-11ed-bff7-6045bd932154" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-72c7ff8f-6519-4968-b40e-0a925c2f0891" "latency"=14366338502 I0201 17:23:39.346541 1 crdprovisioner.go:743] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="224dd920-a255-11ed-bff7-6045bd932154" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=14366473205 I0201 17:23:39.935387 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun2 by sde under /dev/disk/azure/scsi1/ I0201 17:23:39.935453 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun2. perfProfile none accountType StandardSSD_LRS I0201 17:23:39.935493 1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun2 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-72c7ff8f-6519-4968-b40e-0a925c2f0891/globalmount with mount options([]) I0201 17:23:39.935508 1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun2" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun2]) ... skipping 453 lines ... # HELP go_gc_heap_objects_objects Number of objects, live or unswept, occupying heap memory. # TYPE go_gc_heap_objects_objects gauge go_gc_heap_objects_objects 109291 # HELP go_gc_heap_tiny_allocs_objects_total Count of small allocations that are packed together into blocks. These allocations are counted separately from other allocations because each individual allocation is not tracked by the runtime, only their block. Each block is already accounted for in allocs-by-size and frees-by-size. # TYPE go_gc_heap_tiny_allocs_objects_total counter go_gc_heap_tiny_allocs_objects_total 50415 # HELP go_gc_limiter_last_enabled_gc_cycle GC cycle the last time the GC CPU limiter was enabled. This metric is useful for diagnosing the root cause of an out-of-memory error, because the limiter trades memory for CPU time when the GC's CPU time gets too high. This is most likely to occur with use of SetMemoryLimit. The first GC cycle is cycle 1, so a value of 0 indicates that it was never enabled. # TYPE go_gc_limiter_last_enabled_gc_cycle gauge go_gc_limiter_last_enabled_gc_cycle 0 # HELP go_gc_pauses_seconds Distribution individual GC-related stop-the-world pause latencies. # TYPE go_gc_pauses_seconds histogram go_gc_pauses_seconds_bucket{le="9.999999999999999e-10"} 0 go_gc_pauses_seconds_bucket{le="9.999999999999999e-09"} 0 ... skipping 175 lines ... # HELP process_virtual_memory_max_bytes Maximum amount of virtual memory available in bytes. # TYPE process_virtual_memory_max_bytes gauge process_virtual_memory_max_bytes 1.8446744073709552e+19 # HELP registered_metric_total [ALPHA] The count of registered metrics broken by stability level and deprecation version. # TYPE registered_metric_total counter registered_metric_total{deprecated_version="",stability_level="ALPHA"} 16 make: *** [Makefile:344: e2e-test] Error 127 2023/02/01 17:25:11 process.go:155: Step 'make e2e-test' finished in 4m55.012460508s 2023/02/01 17:25:11 aksengine_helpers.go:425: downloading /root/tmp982705966/log-dump.sh from https://raw.githubusercontent.com/kubernetes-sigs/cloud-provider-azure/master/hack/log-dump/log-dump.sh 2023/02/01 17:25:11 util.go:70: curl https://raw.githubusercontent.com/kubernetes-sigs/cloud-provider-azure/master/hack/log-dump/log-dump.sh 2023/02/01 17:25:11 process.go:153: Running: chmod +x /root/tmp982705966/log-dump.sh 2023/02/01 17:25:11 process.go:155: Step 'chmod +x /root/tmp982705966/log-dump.sh' finished in 1.170217ms 2023/02/01 17:25:11 aksengine_helpers.go:425: downloading /root/tmp982705966/log-dump-daemonset.yaml from https://raw.githubusercontent.com/kubernetes-sigs/cloud-provider-azure/master/hack/log-dump/log-dump-daemonset.yaml ... skipping 76 lines ... ssh key file /root/.ssh/id_rsa does not exist. Exiting. 2023/02/01 17:26:08 process.go:155: Step 'bash -c /root/tmp982705966/win-ci-logs-collector.sh kubetest-96oxu0dv.westeurope.cloudapp.azure.com /root/tmp982705966 /root/.ssh/id_rsa' finished in 3.653059ms 2023/02/01 17:26:08 aksengine.go:1141: Deleting resource group: kubetest-96oxu0dv. 2023/02/01 17:33:19 process.go:96: Saved XML output to /logs/artifacts/junit_runner.xml. 2023/02/01 17:33:19 process.go:153: Running: bash -c . hack/lib/version.sh && KUBE_ROOT=. kube::version::get_version_vars && echo "${KUBE_GIT_VERSION-}" 2023/02/01 17:33:20 process.go:155: Step 'bash -c . hack/lib/version.sh && KUBE_ROOT=. kube::version::get_version_vars && echo "${KUBE_GIT_VERSION-}"' finished in 246.565914ms 2023/02/01 17:33:20 main.go:328: Something went wrong: encountered 1 errors: [error during make e2e-test: exit status 2] + EXIT_VALUE=1 + set +o xtrace Cleaning up after docker in docker. ================================================================================ Cleaning up after docker Stopping Docker: dockerProgram process in pidfile '/var/run/docker-ssd.pid', 1 process(es), refused to die. ... skipping 3 lines ...