kubernetes-sigs/azuredisk-csi-driver PR #1361 pull-azuredisk-csi-driver-e2e-migration-mainv2 #1536774126474104832
Recent runs || View in Spyglass
Recent runs || View in Spyglass
| PR | sunpa93: [V2] feat: update volume attachment with publish context upon attachment completion |
| Result | FAILURE |
| Tests | 0 failed / 0 succeeded |
| Started | |
| Elapsed | 3h15m |
| Revision | 9b9063dcff266eed86fe430464db72c1fe3a3758 |
| Refs |
1361 |
... skipping 85 lines ... 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 11156 100 11156 0 0 187k 0 --:--:-- --:--:-- --:--:-- 191k Downloading https://get.helm.sh/helm-v3.9.0-linux-amd64.tar.gz Verifying checksum... Done. Preparing to install helm into /usr/local/bin helm installed into /usr/local/bin/helm docker pull k8sprow.azurecr.io/azuredisk-csi:latest-v2-df2db40004aa147caa3762ffb782a447d13cb39d || make container-all push-manifest Error response from daemon: manifest for k8sprow.azurecr.io/azuredisk-csi:latest-v2-df2db40004aa147caa3762ffb782a447d13cb39d not found: manifest unknown: manifest tagged by "latest-v2-df2db40004aa147caa3762ffb782a447d13cb39d" is not found make[2]: Entering directory '/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver' CGO_ENABLED=0 GOOS=windows go build -a -ldflags "-X sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.driverVersion=latest-v2-df2db40004aa147caa3762ffb782a447d13cb39d -X sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.gitCommit=df2db40004aa147caa3762ffb782a447d13cb39d -X sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.buildDate=2022-06-14T18:24:02Z -extldflags "-static"" -tags azurediskv2 -mod vendor -o _output/amd64/azurediskpluginv2.exe ./pkg/azurediskplugin docker buildx rm container-builder || true error: no builder "container-builder" found docker buildx create --use --name=container-builder container-builder # enable qemu for arm64 build # https://github.com/docker/buildx/issues/464#issuecomment-741507760 docker run --privileged --rm tonistiigi/binfmt --uninstall qemu-aarch64 Unable to find image 'tonistiigi/binfmt:latest' locally ... skipping 700 lines ... } } ] } make[2]: Leaving directory '/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver' docker pull k8sprow.azurecr.io/azdiskschedulerextender-csi:latest-v2-df2db40004aa147caa3762ffb782a447d13cb39d || make azdiskschedulerextender-all push-manifest-azdiskschedulerextender Error response from daemon: manifest for k8sprow.azurecr.io/azdiskschedulerextender-csi:latest-v2-df2db40004aa147caa3762ffb782a447d13cb39d not found: manifest unknown: manifest tagged by "latest-v2-df2db40004aa147caa3762ffb782a447d13cb39d" is not found make[2]: Entering directory '/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver' docker buildx rm container-builder || true docker buildx create --use --name=container-builder container-builder # enable qemu for arm64 build # https://github.com/docker/buildx/issues/464#issuecomment-741507760 ... skipping 855 lines ... type: string type: object oneOf: - required: ["persistentVolumeClaimName"] - required: ["volumeSnapshotContentName"] volumeSnapshotClassName: description: 'VolumeSnapshotClassName is the name of the VolumeSnapshotClass requested by the VolumeSnapshot. VolumeSnapshotClassName may be left nil to indicate that the default SnapshotClass should be used. A given cluster may have multiple default Volume SnapshotClasses: one default per CSI Driver. If a VolumeSnapshot does not specify a SnapshotClass, VolumeSnapshotSource will be checked to figure out what the associated CSI Driver is, and the default VolumeSnapshotClass associated with that CSI Driver will be used. If more than one VolumeSnapshotClass exist for a given CSI Driver and more than one have been marked as default, CreateSnapshot will fail and generate an event. Empty string is not allowed for this field.' type: string required: - source type: object status: description: status represents the current information of a snapshot. Consumers must verify binding between VolumeSnapshot and VolumeSnapshotContent objects is successful (by validating that both VolumeSnapshot and VolumeSnapshotContent point at each other) before using this object. ... skipping 2 lines ... description: 'boundVolumeSnapshotContentName is the name of the VolumeSnapshotContent object to which this VolumeSnapshot object intends to bind to. If not specified, it indicates that the VolumeSnapshot object has not been successfully bound to a VolumeSnapshotContent object yet. NOTE: To avoid possible security issues, consumers must verify binding between VolumeSnapshot and VolumeSnapshotContent objects is successful (by validating that both VolumeSnapshot and VolumeSnapshotContent point at each other) before using this object.' type: string creationTime: description: creationTime is the timestamp when the point-in-time snapshot is taken by the underlying storage system. In dynamic snapshot creation case, this field will be filled in by the snapshot controller with the "creation_time" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "creation_time" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. If not specified, it may indicate that the creation time of the snapshot is unknown. format: date-time type: string error: description: error is the last observed error during snapshot creation, if any. This field could be helpful to upper level controllers(i.e., application controller) to decide whether they should continue on waiting for the snapshot to be created based on the type of error reported. The snapshot controller will keep retrying when an error occurrs during the snapshot creation. Upon success, this error field will be cleared. properties: message: description: 'message is a string detailing the encountered error during snapshot creation if specified. NOTE: message may be logged, and it should not contain sensitive information.' type: string time: description: time is the timestamp when the error was encountered. format: date-time type: string type: object readyToUse: description: readyToUse indicates if the snapshot is ready to be used to restore a volume. In dynamic snapshot creation case, this field will be filled in by the snapshot controller with the "ready_to_use" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "ready_to_use" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it, otherwise, this field will be set to "True". If not specified, it means the readiness of a snapshot is unknown. type: boolean restoreSize: type: string description: restoreSize represents the minimum size of volume required to create a volume from this snapshot. In dynamic snapshot creation case, this field will be filled in by the snapshot controller with the "size_bytes" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "size_bytes" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. When restoring a volume from this snapshot, the size of the volume MUST NOT be smaller than the restoreSize if it is specified, otherwise the restoration will fail. If not specified, it indicates that the size is unknown. pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ x-kubernetes-int-or-string: true type: object required: - spec type: object ... skipping 60 lines ... type: string volumeSnapshotContentName: description: volumeSnapshotContentName specifies the name of a pre-existing VolumeSnapshotContent object representing an existing volume snapshot. This field should be set if the snapshot already exists and only needs a representation in Kubernetes. This field is immutable. type: string type: object volumeSnapshotClassName: description: 'VolumeSnapshotClassName is the name of the VolumeSnapshotClass requested by the VolumeSnapshot. VolumeSnapshotClassName may be left nil to indicate that the default SnapshotClass should be used. A given cluster may have multiple default Volume SnapshotClasses: one default per CSI Driver. If a VolumeSnapshot does not specify a SnapshotClass, VolumeSnapshotSource will be checked to figure out what the associated CSI Driver is, and the default VolumeSnapshotClass associated with that CSI Driver will be used. If more than one VolumeSnapshotClass exist for a given CSI Driver and more than one have been marked as default, CreateSnapshot will fail and generate an event. Empty string is not allowed for this field.' type: string required: - source type: object status: description: status represents the current information of a snapshot. Consumers must verify binding between VolumeSnapshot and VolumeSnapshotContent objects is successful (by validating that both VolumeSnapshot and VolumeSnapshotContent point at each other) before using this object. ... skipping 2 lines ... description: 'boundVolumeSnapshotContentName is the name of the VolumeSnapshotContent object to which this VolumeSnapshot object intends to bind to. If not specified, it indicates that the VolumeSnapshot object has not been successfully bound to a VolumeSnapshotContent object yet. NOTE: To avoid possible security issues, consumers must verify binding between VolumeSnapshot and VolumeSnapshotContent objects is successful (by validating that both VolumeSnapshot and VolumeSnapshotContent point at each other) before using this object.' type: string creationTime: description: creationTime is the timestamp when the point-in-time snapshot is taken by the underlying storage system. In dynamic snapshot creation case, this field will be filled in by the snapshot controller with the "creation_time" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "creation_time" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. If not specified, it may indicate that the creation time of the snapshot is unknown. format: date-time type: string error: description: error is the last observed error during snapshot creation, if any. This field could be helpful to upper level controllers(i.e., application controller) to decide whether they should continue on waiting for the snapshot to be created based on the type of error reported. The snapshot controller will keep retrying when an error occurrs during the snapshot creation. Upon success, this error field will be cleared. properties: message: description: 'message is a string detailing the encountered error during snapshot creation if specified. NOTE: message may be logged, and it should not contain sensitive information.' type: string time: description: time is the timestamp when the error was encountered. format: date-time type: string type: object readyToUse: description: readyToUse indicates if the snapshot is ready to be used to restore a volume. In dynamic snapshot creation case, this field will be filled in by the snapshot controller with the "ready_to_use" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "ready_to_use" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it, otherwise, this field will be set to "True". If not specified, it means the readiness of a snapshot is unknown. type: boolean restoreSize: type: string description: restoreSize represents the minimum size of volume required to create a volume from this snapshot. In dynamic snapshot creation case, this field will be filled in by the snapshot controller with the "size_bytes" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "size_bytes" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. When restoring a volume from this snapshot, the size of the volume MUST NOT be smaller than the restoreSize if it is specified, otherwise the restoration will fail. If not specified, it indicates that the size is unknown. pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ x-kubernetes-int-or-string: true type: object required: - spec type: object ... skipping 254 lines ... description: status represents the current information of a snapshot. properties: creationTime: description: creationTime is the timestamp when the point-in-time snapshot is taken by the underlying storage system. In dynamic snapshot creation case, this field will be filled in by the CSI snapshotter sidecar with the "creation_time" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "creation_time" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. If not specified, it indicates the creation time is unknown. The format of this field is a Unix nanoseconds time encoded as an int64. On Unix, the command `date +%s%N` returns the current time in nanoseconds since 1970-01-01 00:00:00 UTC. format: int64 type: integer error: description: error is the last observed error during snapshot creation, if any. Upon success after retry, this error field will be cleared. properties: message: description: 'message is a string detailing the encountered error during snapshot creation if specified. NOTE: message may be logged, and it should not contain sensitive information.' type: string time: description: time is the timestamp when the error was encountered. format: date-time type: string type: object readyToUse: description: readyToUse indicates if a snapshot is ready to be used to restore a volume. In dynamic snapshot creation case, this field will be filled in by the CSI snapshotter sidecar with the "ready_to_use" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "ready_to_use" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it, otherwise, this field will be set to "True". If not specified, it means the readiness of a snapshot is unknown. type: boolean restoreSize: description: restoreSize represents the complete size of the snapshot in bytes. In dynamic snapshot creation case, this field will be filled in by the CSI snapshotter sidecar with the "size_bytes" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "size_bytes" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. When restoring a volume from this snapshot, the size of the volume MUST NOT be smaller than the restoreSize if it is specified, otherwise the restoration will fail. If not specified, it indicates that the size is unknown. format: int64 minimum: 0 type: integer snapshotHandle: description: snapshotHandle is the CSI "snapshot_id" of a snapshot on the underlying storage system. If not specified, it indicates that dynamic snapshot creation has either failed or it is still in progress. type: string type: object required: - spec type: object served: true ... skipping 108 lines ... description: status represents the current information of a snapshot. properties: creationTime: description: creationTime is the timestamp when the point-in-time snapshot is taken by the underlying storage system. In dynamic snapshot creation case, this field will be filled in by the CSI snapshotter sidecar with the "creation_time" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "creation_time" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. If not specified, it indicates the creation time is unknown. The format of this field is a Unix nanoseconds time encoded as an int64. On Unix, the command `date +%s%N` returns the current time in nanoseconds since 1970-01-01 00:00:00 UTC. format: int64 type: integer error: description: error is the last observed error during snapshot creation, if any. Upon success after retry, this error field will be cleared. properties: message: description: 'message is a string detailing the encountered error during snapshot creation if specified. NOTE: message may be logged, and it should not contain sensitive information.' type: string time: description: time is the timestamp when the error was encountered. format: date-time type: string type: object readyToUse: description: readyToUse indicates if a snapshot is ready to be used to restore a volume. In dynamic snapshot creation case, this field will be filled in by the CSI snapshotter sidecar with the "ready_to_use" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "ready_to_use" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it, otherwise, this field will be set to "True". If not specified, it means the readiness of a snapshot is unknown. type: boolean restoreSize: description: restoreSize represents the complete size of the snapshot in bytes. In dynamic snapshot creation case, this field will be filled in by the CSI snapshotter sidecar with the "size_bytes" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "size_bytes" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. When restoring a volume from this snapshot, the size of the volume MUST NOT be smaller than the restoreSize if it is specified, otherwise the restoration will fail. If not specified, it indicates that the size is unknown. format: int64 minimum: 0 type: integer snapshotHandle: description: snapshotHandle is the CSI "snapshot_id" of a snapshot on the underlying storage system. If not specified, it indicates that dynamic snapshot creation has either failed or it is still in progress. type: string type: object required: - spec type: object served: true ... skipping 244 lines ... - volumeName - volume_context - volume_id type: object status: description: status represents the current state of AzVolumeAttachment. includes error, state, and attachment status properties: annotation: additionalProperties: type: string description: Annotations contains additional resource information to guide driver actions ... skipping 13 lines ... role: description: The current attachment role. type: string required: - role type: object error: description: Error occurred during attach/detach of volume properties: code: type: string message: type: string parameters: ... skipping 173 lines ... - maxMountReplicaCount - volumeCapability - volumeName type: object status: description: status represents the current state of AzVolume. includes error, state, and volume status properties: annotation: additionalProperties: type: string description: Annotations contains additional resource information to guide driver actions ... skipping 34 lines ... type: string required: - capacity_bytes - node_expansion_required - volume_id type: object error: description: Error occurred during creation/deletion of volume properties: code: type: string message: type: string parameters: ... skipping 1061 lines ... image: "mcr.microsoft.com/oss/kubernetes-csi/csi-resizer:v1.4.0" args: - "-csi-address=$(ADDRESS)" - "-v=2" - "-leader-election" - "--leader-election-namespace=kube-system" - '-handle-volume-inuse-error=false' - '-feature-gates=RecoverVolumeExpansionFailure=true' - "-timeout=240s" env: - name: ADDRESS value: /csi/csi.sock volumeMounts: ... skipping 284 lines ... [1mSTEP[0m: setting up the StorageClass [1mSTEP[0m: creating a StorageClass [1mSTEP[0m: setting up the PVC and PV [1mSTEP[0m: creating a PVC [1mSTEP[0m: setting up the pod [1mSTEP[0m: deploying the pod [1mSTEP[0m: checking that the pod's command exits with no error Jun 14 18:32:16.793: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-wvrdt" in namespace "azuredisk-4241" to be "Succeeded or Failed" Jun 14 18:32:16.846: INFO: Pod "azuredisk-volume-tester-wvrdt": Phase="Pending", Reason="", readiness=false. Elapsed: 53.110132ms Jun 14 18:32:18.900: INFO: Pod "azuredisk-volume-tester-wvrdt": Phase="Pending", Reason="", readiness=false. Elapsed: 2.107147063s Jun 14 18:32:20.955: INFO: Pod "azuredisk-volume-tester-wvrdt": Phase="Pending", Reason="", readiness=false. Elapsed: 4.161717354s Jun 14 18:32:23.011: INFO: Pod "azuredisk-volume-tester-wvrdt": Phase="Pending", Reason="", readiness=false. Elapsed: 6.218198104s Jun 14 18:32:25.065: INFO: Pod "azuredisk-volume-tester-wvrdt": Phase="Pending", Reason="", readiness=false. Elapsed: 8.271989903s Jun 14 18:32:27.119: INFO: Pod "azuredisk-volume-tester-wvrdt": Phase="Pending", Reason="", readiness=false. Elapsed: 10.326149464s ... skipping 427 lines ... Jun 14 18:47:06.801: INFO: Pod "azuredisk-volume-tester-wvrdt": Phase="Pending", Reason="", readiness=false. Elapsed: 14m50.008259849s Jun 14 18:47:08.857: INFO: Pod "azuredisk-volume-tester-wvrdt": Phase="Pending", Reason="", readiness=false. Elapsed: 14m52.063975904s Jun 14 18:47:10.912: INFO: Pod "azuredisk-volume-tester-wvrdt": Phase="Pending", Reason="", readiness=false. Elapsed: 14m54.11883811s Jun 14 18:47:12.966: INFO: Pod "azuredisk-volume-tester-wvrdt": Phase="Pending", Reason="", readiness=false. Elapsed: 14m56.173355143s Jun 14 18:47:15.021: INFO: Pod "azuredisk-volume-tester-wvrdt": Phase="Pending", Reason="", readiness=false. Elapsed: 14m58.227677719s Jun 14 18:47:17.022: INFO: deleting Pod "azuredisk-4241"/"azuredisk-volume-tester-wvrdt" Jun 14 18:47:17.107: INFO: Error getting logs for pod azuredisk-volume-tester-wvrdt: the server rejected our request for an unknown reason (get pods azuredisk-volume-tester-wvrdt) [1mSTEP[0m: Deleting pod azuredisk-volume-tester-wvrdt in namespace azuredisk-4241 [1mSTEP[0m: validating provisioned PV [1mSTEP[0m: checking the PV Jun 14 18:47:17.272: INFO: deleting PVC "azuredisk-4241"/"pvc-vlgrv" Jun 14 18:47:17.272: INFO: Deleting PersistentVolumeClaim "pvc-vlgrv" [1mSTEP[0m: waiting for claim's PV "pvc-e6291a08-817d-41da-ad5a-8f6c567374d6" to be deleted ... skipping 124 lines ... Jun 14 18:57:19.161: INFO: At 2022-06-14 18:32:16 +0000 UTC - event for pvc-vlgrv: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "disk.csi.azure.com" or manually created by system administrator Jun 14 18:57:19.161: INFO: At 2022-06-14 18:32:16 +0000 UTC - event for pvc-vlgrv: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding Jun 14 18:57:19.161: INFO: At 2022-06-14 18:32:16 +0000 UTC - event for pvc-vlgrv: {disk.csi.azure.com_k8s-master-20224640-0_7f146b18-d5c4-4b0e-86dd-3b66836d87bc } Provisioning: External provisioner is provisioning volume for claim "azuredisk-4241/pvc-vlgrv" Jun 14 18:57:19.161: INFO: At 2022-06-14 18:32:19 +0000 UTC - event for pvc-vlgrv: {disk.csi.azure.com_k8s-master-20224640-0_7f146b18-d5c4-4b0e-86dd-3b66836d87bc } ProvisioningSucceeded: Successfully provisioned volume pvc-e6291a08-817d-41da-ad5a-8f6c567374d6 Jun 14 18:57:19.161: INFO: At 2022-06-14 18:32:21 +0000 UTC - event for azuredisk-volume-tester-wvrdt: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-e6291a08-817d-41da-ad5a-8f6c567374d6" Jun 14 18:57:19.161: INFO: At 2022-06-14 18:34:22 +0000 UTC - event for azuredisk-volume-tester-wvrdt: {kubelet k8s-agentpool1-20224640-0} FailedMount: Unable to attach or mount volumes: unmounted volumes=[test-volume-1], unattached volumes=[test-volume-1 kube-api-access-ns2lg]: timed out waiting for the condition Jun 14 18:57:19.161: INFO: At 2022-06-14 18:34:27 +0000 UTC - event for azuredisk-volume-tester-wvrdt: {kubelet k8s-agentpool1-20224640-0} FailedMount: MountVolume.MountDevice failed for volume "pvc-e6291a08-817d-41da-ad5a-8f6c567374d6" : rpc error: code = DeadlineExceeded desc = context deadline exceeded Jun 14 18:57:19.161: INFO: At 2022-06-14 18:38:53 +0000 UTC - event for azuredisk-volume-tester-wvrdt: {kubelet k8s-agentpool1-20224640-0} FailedMount: Unable to attach or mount volumes: unmounted volumes=[test-volume-1], unattached volumes=[kube-api-access-ns2lg test-volume-1]: timed out waiting for the condition Jun 14 18:57:19.215: INFO: POD NODE PHASE GRACE CONDITIONS Jun 14 18:57:19.215: INFO: Jun 14 18:57:19.322: INFO: Logging node info for node k8s-agentpool1-20224640-0 Jun 14 18:57:19.376: INFO: Node Info: &Node{ObjectMeta:{k8s-agentpool1-20224640-0 d40122eb-7d63-4c24-87d0-a4294fda80c2 7051 0 2022-06-14 18:19:50 +0000 UTC <nil> <nil> map[agentpool:agentpool1 beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_DS2_v2 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:0 kubernetes.azure.com/cluster:kubetest-w0zpdk2w kubernetes.azure.com/role:agent kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-agentpool1-20224640-0 kubernetes.io/os:linux kubernetes.io/role:agent node-role.kubernetes.io/agent: node.kubernetes.io/instance-type:Standard_DS2_v2 storageprofile:managed storagetier:Premium_LRS topology.disk.csi.azure.com/zone: topology.kubernetes.io/region:westus2 topology.kubernetes.io/zone:0] map[csi.volume.kubernetes.io/nodeid:{"disk.csi.azure.com":"k8s-agentpool1-20224640-0"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2022-06-14 18:19:50 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:agentpool":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.azure.com/cluster":{},"f:kubernetes.azure.com/role":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:storageprofile":{},"f:storagetier":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kubectl-label Update v1 2022-06-14 18:19:58 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/agent":{}}}} } {kube-controller-manager Update v1 2022-06-14 18:20:04 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"10.244.0.0/24\"":{}}}} } {kubelet Update v1 2022-06-14 18:49:38 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.disk.csi.azure.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status} {kube-controller-manager Update v1 2022-06-14 18:55:39 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status}]},Spec:NodeSpec{PodCIDR:10.244.0.0/24,DoNotUseExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-w0zpdk2w/providers/Microsoft.Compute/virtualMachines/k8s-agentpool1-20224640-0,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[10.244.0.0/24],},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-azure-disk: {{8 0} {<nil>} 8 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{31036686336 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7287771136 0} {<nil>} 7116964Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-azure-disk: {{8 0} {<nil>} 8 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{27933017657 0} {<nil>} 27933017657 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{6501339136 0} {<nil>} 6348964Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-06-14 18:20:25 +0000 UTC,LastTransitionTime:2022-06-14 18:20:25 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-06-14 18:54:44 +0000 UTC,LastTransitionTime:2022-06-14 18:19:49 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-06-14 18:54:44 +0000 UTC,LastTransitionTime:2022-06-14 18:19:49 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-06-14 18:54:44 +0000 UTC,LastTransitionTime:2022-06-14 18:19:49 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-06-14 18:54:44 +0000 UTC,LastTransitionTime:2022-06-14 18:20:04 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:Hostname,Address:k8s-agentpool1-20224640-0,},NodeAddress{Type:InternalIP,Address:10.240.0.4,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:fcfd567ea6a54e9fb7ca2e3949562363,SystemUUID:c20271e3-7d7e-f940-9c6d-928cc89ffc2d,BootID:2a1d1325-bc1b-47c9-9540-2fd7da933292,KernelVersion:5.4.0-1074-azure,OSImage:Ubuntu 18.04.6 LTS,ContainerRuntimeVersion:docker://20.10.11+azure-3,KubeletVersion:v1.23.7,KubeProxyVersion:v1.23.7,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/azuredisk-csi@sha256:423eb6cf602c064c8b2deefead5ceadd6324ed41b3d995dab5d0f6f0f4d4710f mcr.microsoft.com/oss/kubernetes-csi/azuredisk-csi:v1.10.0],SizeBytes:245959792,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/azurefile-csi@sha256:9e2ecabcf9dd9943e6600eb9fb460f45b4dc61af7cabe95d115082a029db2aaf mcr.microsoft.com/oss/kubernetes-csi/azurefile-csi:v1.9.0],SizeBytes:230470852,},ContainerImage{Names:[mcr.microsoft.com/containernetworking/azure-npm@sha256:106f669f48e5e80c4ec0afb49858ead72cf4b901cd8664e7bf81f8d789e56e12 mcr.microsoft.com/containernetworking/azure-npm:v1.2.2_hotfix],SizeBytes:175230380,},ContainerImage{Names:[k8sprow.azurecr.io/azuredisk-csi@sha256:528fc89c187c74420dabdef86e06f05893e1510fce69111a2a05280dd64be2a9 k8sprow.azurecr.io/azuredisk-csi:latest-v2-df2db40004aa147caa3762ffb782a447d13cb39d],SizeBytes:136461498,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/secrets-store/driver@sha256:c0d040a1c4fbfceb65663e31c09ea40f4f78e356437610cbc3fbb4bb409bd6f1 mcr.microsoft.com/oss/kubernetes-csi/secrets-store/driver:v0.0.19],SizeBytes:123229697,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-proxy@sha256:0c890e918d42c158fab68867bade2ab1db04c3408452db9611a264ba3e43e6b3 mcr.microsoft.com/oss/kubernetes/kube-proxy:v1.23.7],SizeBytes:112296063,},ContainerImage{Names:[mcr.microsoft.com/oss/azure/secrets-store/provider-azure@sha256:6f67f3d0c7cdde5702f8ce7f101b6519daa0237f0c34fecb7c058b6af8c22ad1 mcr.microsoft.com/oss/azure/secrets-store/provider-azure:0.0.12],SizeBytes:101061355,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/autoscaler/cluster-autoscaler@sha256:6f0c680d375c62e74351f8ff3ed6ddb9b72ca759e0645c329b95f64264654a6d mcr.microsoft.com/oss/kubernetes/autoscaler/cluster-autoscaler:v1.22.1],SizeBytes:99962810,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-addon-manager@sha256:32e2836018c96e73533bd4642fe438e465b81dcbfa8b7b61935a6f4d0246c7ae mcr.microsoft.com/oss/kubernetes/kube-addon-manager:v9.1.3],SizeBytes:86832059,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-addon-manager@sha256:92c2c5aad9012ee32d2a43a74966cc0adc6ccb1705ad15abb10485ecf406d88b mcr.microsoft.com/oss/kubernetes/kube-addon-manager:v9.1.5],SizeBytes:84094027,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/metrics-server@sha256:1ef9d57ce41ffcc328b92494c3bfafe401e0b9a1694a295301a1385337d52815 mcr.microsoft.com/oss/kubernetes/metrics-server:v0.5.2],SizeBytes:64327621,},ContainerImage{Names:[mcr.microsoft.com/oss/nvidia/k8s-device-plugin@sha256:0f5b52bf28239234e831697d96db63ac03cde70fe68058f964504ab7564ee810 mcr.microsoft.com/oss/nvidia/k8s-device-plugin:1.0.0-beta6],SizeBytes:64160241,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-provisioner@sha256:e9ddadc44ba87a4a27f67e54760a14f9986885b534b3dff170a14eae1e35d213 mcr.microsoft.com/oss/kubernetes-csi/csi-provisioner:v3.0.0],SizeBytes:56881280,},ContainerImage{Names:[k8sprow.azurecr.io/azdiskschedulerextender-csi@sha256:3df91ffec98f04827c70772d08849e6874d1e0ee2432173ef523c099553547dc k8sprow.azurecr.io/azdiskschedulerextender-csi:latest-v2-df2db40004aa147caa3762ffb782a447d13cb39d],SizeBytes:56760067,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-resizer@sha256:c5bb71ceaac60b1a4b58739fa07b709f6248c452ff6272a384d2f7648895a750 mcr.microsoft.com/oss/kubernetes-csi/csi-resizer:v1.3.0],SizeBytes:54313772,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-snapshotter@sha256:61849a026511cf332c87d73d0a7aed803b510c3ede197ec755389686d490de72 mcr.microsoft.com/oss/kubernetes-csi/csi-snapshotter:v4.2.1],SizeBytes:54210936,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-attacher@sha256:6b41e7153ebdfdc1501aa65184624bc15fd33a52d93f88ec3a758d0f8c9b8c10 mcr.microsoft.com/oss/kubernetes-csi/csi-attacher:v3.3.0],SizeBytes:53842561,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/snapshot-controller@sha256:8c3fc3c2667004ad6bbdf723bb64c5da66a5cb8b11d4ee59b67179b686223b13 mcr.microsoft.com/oss/kubernetes-csi/snapshot-controller:v5.0.1],SizeBytes:52732401,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/snapshot-controller@sha256:be5a8dc1f990828f653c77bd0a0f1bbd13197c3019f6e1d99d590389bac36705 mcr.microsoft.com/oss/kubernetes-csi/snapshot-controller:v4.2.1],SizeBytes:51575245,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager@sha256:0ad67f9919522a07318034641ae09bf2079b417e9944c65914410594ce645468 mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager:v1.1.4],SizeBytes:51478397,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager@sha256:31ec4f7daccd3e7a8504e12657d7830651ecacbe4a487daca1b1b7695a64b070 mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager:v1.23.1],SizeBytes:51249021,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager@sha256:011712ed90fb8efcf27928b0a47ed04b98baebb31cb1b2d8ab676977ec18eedc mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager:v1.1.4],SizeBytes:50868093,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager@sha256:0f9a8fbaed65192ed7dd795be4f9c1dc48ebdef0a241fb62d456f4bed40d9875 mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager:v1.23.1],SizeBytes:50679677,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-scheduler@sha256:c35f45e8dfaa30363e589296415a057f9d32813920fc3820d6557491657f199a mcr.microsoft.com/oss/kubernetes/kube-scheduler:v1.21.2],SizeBytes:50607491,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/ip-masq-agent@sha256:1244155f2ed3f33ff154cc343b8ad285f3391d95afd7d4b1c6dcc420bc0ba3cf mcr.microsoft.com/oss/kubernetes/ip-masq-agent:v2.5.0],SizeBytes:50146762,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager@sha256:7c907ff70b90a0bdf8fae63bd744018469dd9839cde1fd0515b93e0bbd14b34e mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager:v1.0.8],SizeBytes:48963453,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager@sha256:3987d7a8c6922ce1952ee19c5cb6ea75aac7b7c1b07aa79277ad038c69fb7a31 mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager:v1.0.8],SizeBytes:48349053,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/coredns@sha256:f873bf7f0928461efe10697fa76cf0ad7a1ae3041c5b57b50dd3d0b72d273f8c mcr.microsoft.com/oss/kubernetes/coredns:1.8.6],SizeBytes:46804601,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager@sha256:8073113a20882642a980b338635cdc5945e5673a18aef192090e6fde2b89a75c mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager:v0.6.0],SizeBytes:45909032,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager@sha256:6a32329628bdea3c6d75e98aad6155b65d2e2b98ca616eb33f9ac562912804c6 mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager:v0.6.0],SizeBytes:45229096,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager@sha256:ef6c4ba564b4d11d270f7d1563c50fbeb30ccc3b94146e5059228c49f95875f5 mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager:v0.7.11],SizeBytes:44916605,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager@sha256:dbcc384758ba5ca6d249596d471292ed3785e31cdb854d48b84d70794b669b4c mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager:v0.7.11],SizeBytes:43679613,},ContainerImage{Names:[mcr.microsoft.com/oss/etcd-io/etcd@sha256:cf587862e3f1b6fa4d9a2565520a34f164bdf72c50f37af8c3c668160593246e mcr.microsoft.com/oss/etcd-io/etcd:v3.3.25],SizeBytes:41832119,},ContainerImage{Names:[mcr.microsoft.com/k8s/aad-pod-identity/mic@sha256:bd9465be94966b9a917e1e3904fa5e63dd91772ccadf304e18ffd8e4ad8ccedd mcr.microsoft.com/k8s/aad-pod-identity/mic:1.6.1],SizeBytes:41374894,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/autoscaler/cluster-proportional-autoscaler@sha256:c849d75d61943ce7f51b4c049f1a79d19a08253966c8f49c4cfb6414cc33db8b mcr.microsoft.com/oss/kubernetes/autoscaler/cluster-proportional-autoscaler:1.8.5],SizeBytes:40661903,},ContainerImage{Names:[mcr.microsoft.com/k8s/aad-pod-identity/nmi@sha256:02128fefcdb7593ac53fc342e2c53a0fc6fabd813036bf60457bf43cc2940116 mcr.microsoft.com/k8s/aad-pod-identity/nmi:1.6.1],SizeBytes:38007982,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:921f301c44dda06a325164accf22e78ecc570b5c7d9d6ee4c66bd8cbb2b60b9a mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.16],SizeBytes:26970670,},ContainerImage{Names:[mcr.microsoft.com/k8s/kms/keyvault@sha256:1a27e175f8c125209e32d2957b5509fe20757bd8cb309ff9da598799b56326fb mcr.microsoft.com/k8s/kms/keyvault:v0.0.10],SizeBytes:23077387,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-node-driver-registrar@sha256:dbec3a8166686b09b242176ab5b99e993da4126438bbce68147c3fd654f35662 mcr.microsoft.com/oss/kubernetes-csi/csi-node-driver-registrar:v2.4.0],SizeBytes:19547289,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/livenessprobe@sha256:c96a6255c42766f6b8bb1a7cda02b0060ab1b20b2e2dafcc64ec09e7646745a6 mcr.microsoft.com/oss/kubernetes-csi/livenessprobe:v2.5.0],SizeBytes:17573341,},ContainerImage{Names:[mcr.microsoft.com/oss/busybox/busybox@sha256:582a641242b49809af3a1a522f9aae8c3f047d1c6ca1dd9d8cdabd349e45b1a9 mcr.microsoft.com/oss/busybox/busybox:1.33.1],SizeBytes:1235829,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8],SizeBytes:1159025,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/pause@sha256:e3b8c20681593c21b344ad801fbb8abaf564427ee3a57a9fcfa3b455f917ce46 mcr.microsoft.com/oss/kubernetes/pause:3.4.1],SizeBytes:682696,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} ... skipping 91 lines ... [90m/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning_test.go:40[0m [single-az] [90m/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning_test.go:44[0m [91m[1mshould create a volume on demand with mount options [kubernetes.io/azure-disk] [disk.csi.azure.com] [Windows] [csi-azuredisk-scheduler-extender] [It][0m [90m/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning_test.go:102[0m [91mUnexpected error: <*errors.errorString | 0xc000688950>: { s: "Gave up after waiting 15m0s for pod \"azuredisk-volume-tester-wvrdt\" to be \"Succeeded or Failed\"", } Gave up after waiting 15m0s for pod "azuredisk-volume-tester-wvrdt" to be "Succeeded or Failed" occurred[0m /home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/resources/pod.go:89 [90m------------------------------[0m [0mDynamic Provisioning[0m [90m[single-az][0m [1mshould create a pod with volume mount subpath [disk.csi.azure.com] [Windows][0m ... skipping 97 lines ... [1mSTEP[0m: setting up the PVC and PV [1mSTEP[0m: creating a PVC [1mSTEP[0m: setting up the pod [1mSTEP[0m: deploying the pod [1mSTEP[0m: checking that the pod has 'FailedMount' event Jun 14 18:59:35.571: INFO: deleting Pod "azuredisk-2935"/"azuredisk-volume-tester-mh6kp" Jun 14 18:59:35.660: INFO: Error getting logs for pod azuredisk-volume-tester-mh6kp: the server rejected our request for an unknown reason (get pods azuredisk-volume-tester-mh6kp) [1mSTEP[0m: Deleting pod azuredisk-volume-tester-mh6kp in namespace azuredisk-2935 [1mSTEP[0m: validating provisioned PV [1mSTEP[0m: checking the PV Jun 14 18:59:35.823: INFO: deleting PVC "azuredisk-2935"/"pvc-gcq7z" Jun 14 18:59:35.824: INFO: Deleting PersistentVolumeClaim "pvc-gcq7z" [1mSTEP[0m: waiting for claim's PV "pvc-45e32b8f-048e-47db-9efd-1af2daa64036" to be deleted ... skipping 124 lines ... Jun 14 19:09:37.907: INFO: At 2022-06-14 18:57:25 +0000 UTC - event for pvc-gcq7z: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding Jun 14 19:09:37.907: INFO: At 2022-06-14 18:57:25 +0000 UTC - event for pvc-gcq7z: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "disk.csi.azure.com" or manually created by system administrator Jun 14 19:09:37.907: INFO: At 2022-06-14 18:57:25 +0000 UTC - event for pvc-gcq7z: {disk.csi.azure.com_k8s-master-20224640-0_7f146b18-d5c4-4b0e-86dd-3b66836d87bc } Provisioning: External provisioner is provisioning volume for claim "azuredisk-2935/pvc-gcq7z" Jun 14 19:09:37.908: INFO: At 2022-06-14 18:57:28 +0000 UTC - event for pvc-gcq7z: {disk.csi.azure.com_k8s-master-20224640-0_7f146b18-d5c4-4b0e-86dd-3b66836d87bc } ProvisioningSucceeded: Successfully provisioned volume pvc-45e32b8f-048e-47db-9efd-1af2daa64036 Jun 14 19:09:37.908: INFO: At 2022-06-14 18:57:30 +0000 UTC - event for azuredisk-volume-tester-mh6kp: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-45e32b8f-048e-47db-9efd-1af2daa64036" Jun 14 19:09:37.908: INFO: At 2022-06-14 18:59:31 +0000 UTC - event for azuredisk-volume-tester-mh6kp: {kubelet k8s-agentpool1-20224640-1} FailedMount: Unable to attach or mount volumes: unmounted volumes=[test-volume-1], unattached volumes=[test-volume-1 kube-api-access-wssnt]: timed out waiting for the condition Jun 14 19:09:37.908: INFO: At 2022-06-14 18:59:34 +0000 UTC - event for azuredisk-volume-tester-mh6kp: {kubelet k8s-agentpool1-20224640-1} FailedMount: MountVolume.MountDevice failed for volume "pvc-45e32b8f-048e-47db-9efd-1af2daa64036" : rpc error: code = DeadlineExceeded desc = context deadline exceeded Jun 14 19:09:37.963: INFO: POD NODE PHASE GRACE CONDITIONS Jun 14 19:09:37.963: INFO: Jun 14 19:09:38.072: INFO: Logging node info for node k8s-agentpool1-20224640-0 Jun 14 19:09:38.129: INFO: Node Info: &Node{ObjectMeta:{k8s-agentpool1-20224640-0 d40122eb-7d63-4c24-87d0-a4294fda80c2 9737 0 2022-06-14 18:19:50 +0000 UTC <nil> <nil> map[agentpool:agentpool1 beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_DS2_v2 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:0 kubernetes.azure.com/cluster:kubetest-w0zpdk2w kubernetes.azure.com/role:agent kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-agentpool1-20224640-0 kubernetes.io/os:linux kubernetes.io/role:agent node-role.kubernetes.io/agent: node.kubernetes.io/instance-type:Standard_DS2_v2 storageprofile:managed storagetier:Premium_LRS topology.disk.csi.azure.com/zone: topology.kubernetes.io/region:westus2 topology.kubernetes.io/zone:0] map[csi.volume.kubernetes.io/nodeid:{"disk.csi.azure.com":"k8s-agentpool1-20224640-0"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2022-06-14 18:19:50 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:agentpool":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.azure.com/cluster":{},"f:kubernetes.azure.com/role":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:storageprofile":{},"f:storagetier":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kubectl-label Update v1 2022-06-14 18:19:58 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/agent":{}}}} } {kube-controller-manager Update v1 2022-06-14 18:20:04 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"10.244.0.0/24\"":{}}}} } {kubelet Update v1 2022-06-14 18:49:38 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.disk.csi.azure.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status} {kube-controller-manager Update v1 2022-06-14 19:07:51 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:volumesAttached":{}}} status}]},Spec:NodeSpec{PodCIDR:10.244.0.0/24,DoNotUseExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-w0zpdk2w/providers/Microsoft.Compute/virtualMachines/k8s-agentpool1-20224640-0,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[10.244.0.0/24],},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-azure-disk: {{8 0} {<nil>} 8 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{31036686336 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7287771136 0} {<nil>} 7116964Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-azure-disk: {{8 0} {<nil>} 8 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{27933017657 0} {<nil>} 27933017657 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{6501339136 0} {<nil>} 6348964Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-06-14 18:20:25 +0000 UTC,LastTransitionTime:2022-06-14 18:20:25 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-06-14 19:04:57 +0000 UTC,LastTransitionTime:2022-06-14 18:19:49 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-06-14 19:04:57 +0000 UTC,LastTransitionTime:2022-06-14 18:19:49 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-06-14 19:04:57 +0000 UTC,LastTransitionTime:2022-06-14 18:19:49 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-06-14 19:04:57 +0000 UTC,LastTransitionTime:2022-06-14 18:20:04 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:Hostname,Address:k8s-agentpool1-20224640-0,},NodeAddress{Type:InternalIP,Address:10.240.0.4,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:fcfd567ea6a54e9fb7ca2e3949562363,SystemUUID:c20271e3-7d7e-f940-9c6d-928cc89ffc2d,BootID:2a1d1325-bc1b-47c9-9540-2fd7da933292,KernelVersion:5.4.0-1074-azure,OSImage:Ubuntu 18.04.6 LTS,ContainerRuntimeVersion:docker://20.10.11+azure-3,KubeletVersion:v1.23.7,KubeProxyVersion:v1.23.7,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/azuredisk-csi@sha256:423eb6cf602c064c8b2deefead5ceadd6324ed41b3d995dab5d0f6f0f4d4710f mcr.microsoft.com/oss/kubernetes-csi/azuredisk-csi:v1.10.0],SizeBytes:245959792,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/azurefile-csi@sha256:9e2ecabcf9dd9943e6600eb9fb460f45b4dc61af7cabe95d115082a029db2aaf mcr.microsoft.com/oss/kubernetes-csi/azurefile-csi:v1.9.0],SizeBytes:230470852,},ContainerImage{Names:[mcr.microsoft.com/containernetworking/azure-npm@sha256:106f669f48e5e80c4ec0afb49858ead72cf4b901cd8664e7bf81f8d789e56e12 mcr.microsoft.com/containernetworking/azure-npm:v1.2.2_hotfix],SizeBytes:175230380,},ContainerImage{Names:[k8sprow.azurecr.io/azuredisk-csi@sha256:528fc89c187c74420dabdef86e06f05893e1510fce69111a2a05280dd64be2a9 k8sprow.azurecr.io/azuredisk-csi:latest-v2-df2db40004aa147caa3762ffb782a447d13cb39d],SizeBytes:136461498,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/secrets-store/driver@sha256:c0d040a1c4fbfceb65663e31c09ea40f4f78e356437610cbc3fbb4bb409bd6f1 mcr.microsoft.com/oss/kubernetes-csi/secrets-store/driver:v0.0.19],SizeBytes:123229697,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-proxy@sha256:0c890e918d42c158fab68867bade2ab1db04c3408452db9611a264ba3e43e6b3 mcr.microsoft.com/oss/kubernetes/kube-proxy:v1.23.7],SizeBytes:112296063,},ContainerImage{Names:[mcr.microsoft.com/oss/azure/secrets-store/provider-azure@sha256:6f67f3d0c7cdde5702f8ce7f101b6519daa0237f0c34fecb7c058b6af8c22ad1 mcr.microsoft.com/oss/azure/secrets-store/provider-azure:0.0.12],SizeBytes:101061355,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/autoscaler/cluster-autoscaler@sha256:6f0c680d375c62e74351f8ff3ed6ddb9b72ca759e0645c329b95f64264654a6d mcr.microsoft.com/oss/kubernetes/autoscaler/cluster-autoscaler:v1.22.1],SizeBytes:99962810,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-addon-manager@sha256:32e2836018c96e73533bd4642fe438e465b81dcbfa8b7b61935a6f4d0246c7ae mcr.microsoft.com/oss/kubernetes/kube-addon-manager:v9.1.3],SizeBytes:86832059,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-addon-manager@sha256:92c2c5aad9012ee32d2a43a74966cc0adc6ccb1705ad15abb10485ecf406d88b mcr.microsoft.com/oss/kubernetes/kube-addon-manager:v9.1.5],SizeBytes:84094027,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/metrics-server@sha256:1ef9d57ce41ffcc328b92494c3bfafe401e0b9a1694a295301a1385337d52815 mcr.microsoft.com/oss/kubernetes/metrics-server:v0.5.2],SizeBytes:64327621,},ContainerImage{Names:[mcr.microsoft.com/oss/nvidia/k8s-device-plugin@sha256:0f5b52bf28239234e831697d96db63ac03cde70fe68058f964504ab7564ee810 mcr.microsoft.com/oss/nvidia/k8s-device-plugin:1.0.0-beta6],SizeBytes:64160241,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-provisioner@sha256:e9ddadc44ba87a4a27f67e54760a14f9986885b534b3dff170a14eae1e35d213 mcr.microsoft.com/oss/kubernetes-csi/csi-provisioner:v3.0.0],SizeBytes:56881280,},ContainerImage{Names:[k8sprow.azurecr.io/azdiskschedulerextender-csi@sha256:3df91ffec98f04827c70772d08849e6874d1e0ee2432173ef523c099553547dc k8sprow.azurecr.io/azdiskschedulerextender-csi:latest-v2-df2db40004aa147caa3762ffb782a447d13cb39d],SizeBytes:56760067,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-resizer@sha256:c5bb71ceaac60b1a4b58739fa07b709f6248c452ff6272a384d2f7648895a750 mcr.microsoft.com/oss/kubernetes-csi/csi-resizer:v1.3.0],SizeBytes:54313772,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-snapshotter@sha256:61849a026511cf332c87d73d0a7aed803b510c3ede197ec755389686d490de72 mcr.microsoft.com/oss/kubernetes-csi/csi-snapshotter:v4.2.1],SizeBytes:54210936,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-attacher@sha256:6b41e7153ebdfdc1501aa65184624bc15fd33a52d93f88ec3a758d0f8c9b8c10 mcr.microsoft.com/oss/kubernetes-csi/csi-attacher:v3.3.0],SizeBytes:53842561,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/snapshot-controller@sha256:8c3fc3c2667004ad6bbdf723bb64c5da66a5cb8b11d4ee59b67179b686223b13 mcr.microsoft.com/oss/kubernetes-csi/snapshot-controller:v5.0.1],SizeBytes:52732401,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/snapshot-controller@sha256:be5a8dc1f990828f653c77bd0a0f1bbd13197c3019f6e1d99d590389bac36705 mcr.microsoft.com/oss/kubernetes-csi/snapshot-controller:v4.2.1],SizeBytes:51575245,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager@sha256:0ad67f9919522a07318034641ae09bf2079b417e9944c65914410594ce645468 mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager:v1.1.4],SizeBytes:51478397,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager@sha256:31ec4f7daccd3e7a8504e12657d7830651ecacbe4a487daca1b1b7695a64b070 mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager:v1.23.1],SizeBytes:51249021,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager@sha256:011712ed90fb8efcf27928b0a47ed04b98baebb31cb1b2d8ab676977ec18eedc mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager:v1.1.4],SizeBytes:50868093,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager@sha256:0f9a8fbaed65192ed7dd795be4f9c1dc48ebdef0a241fb62d456f4bed40d9875 mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager:v1.23.1],SizeBytes:50679677,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-scheduler@sha256:c35f45e8dfaa30363e589296415a057f9d32813920fc3820d6557491657f199a mcr.microsoft.com/oss/kubernetes/kube-scheduler:v1.21.2],SizeBytes:50607491,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/ip-masq-agent@sha256:1244155f2ed3f33ff154cc343b8ad285f3391d95afd7d4b1c6dcc420bc0ba3cf mcr.microsoft.com/oss/kubernetes/ip-masq-agent:v2.5.0],SizeBytes:50146762,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager@sha256:7c907ff70b90a0bdf8fae63bd744018469dd9839cde1fd0515b93e0bbd14b34e mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager:v1.0.8],SizeBytes:48963453,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager@sha256:3987d7a8c6922ce1952ee19c5cb6ea75aac7b7c1b07aa79277ad038c69fb7a31 mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager:v1.0.8],SizeBytes:48349053,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/coredns@sha256:f873bf7f0928461efe10697fa76cf0ad7a1ae3041c5b57b50dd3d0b72d273f8c mcr.microsoft.com/oss/kubernetes/coredns:1.8.6],SizeBytes:46804601,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager@sha256:8073113a20882642a980b338635cdc5945e5673a18aef192090e6fde2b89a75c mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager:v0.6.0],SizeBytes:45909032,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager@sha256:6a32329628bdea3c6d75e98aad6155b65d2e2b98ca616eb33f9ac562912804c6 mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager:v0.6.0],SizeBytes:45229096,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager@sha256:ef6c4ba564b4d11d270f7d1563c50fbeb30ccc3b94146e5059228c49f95875f5 mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager:v0.7.11],SizeBytes:44916605,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager@sha256:dbcc384758ba5ca6d249596d471292ed3785e31cdb854d48b84d70794b669b4c mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager:v0.7.11],SizeBytes:43679613,},ContainerImage{Names:[mcr.microsoft.com/oss/etcd-io/etcd@sha256:cf587862e3f1b6fa4d9a2565520a34f164bdf72c50f37af8c3c668160593246e mcr.microsoft.com/oss/etcd-io/etcd:v3.3.25],SizeBytes:41832119,},ContainerImage{Names:[mcr.microsoft.com/k8s/aad-pod-identity/mic@sha256:bd9465be94966b9a917e1e3904fa5e63dd91772ccadf304e18ffd8e4ad8ccedd mcr.microsoft.com/k8s/aad-pod-identity/mic:1.6.1],SizeBytes:41374894,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/autoscaler/cluster-proportional-autoscaler@sha256:c849d75d61943ce7f51b4c049f1a79d19a08253966c8f49c4cfb6414cc33db8b mcr.microsoft.com/oss/kubernetes/autoscaler/cluster-proportional-autoscaler:1.8.5],SizeBytes:40661903,},ContainerImage{Names:[mcr.microsoft.com/k8s/aad-pod-identity/nmi@sha256:02128fefcdb7593ac53fc342e2c53a0fc6fabd813036bf60457bf43cc2940116 mcr.microsoft.com/k8s/aad-pod-identity/nmi:1.6.1],SizeBytes:38007982,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:921f301c44dda06a325164accf22e78ecc570b5c7d9d6ee4c66bd8cbb2b60b9a mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.16],SizeBytes:26970670,},ContainerImage{Names:[mcr.microsoft.com/k8s/kms/keyvault@sha256:1a27e175f8c125209e32d2957b5509fe20757bd8cb309ff9da598799b56326fb mcr.microsoft.com/k8s/kms/keyvault:v0.0.10],SizeBytes:23077387,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-node-driver-registrar@sha256:dbec3a8166686b09b242176ab5b99e993da4126438bbce68147c3fd654f35662 mcr.microsoft.com/oss/kubernetes-csi/csi-node-driver-registrar:v2.4.0],SizeBytes:19547289,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/livenessprobe@sha256:c96a6255c42766f6b8bb1a7cda02b0060ab1b20b2e2dafcc64ec09e7646745a6 mcr.microsoft.com/oss/kubernetes-csi/livenessprobe:v2.5.0],SizeBytes:17573341,},ContainerImage{Names:[mcr.microsoft.com/oss/busybox/busybox@sha256:582a641242b49809af3a1a522f9aae8c3f047d1c6ca1dd9d8cdabd349e45b1a9 mcr.microsoft.com/oss/busybox/busybox:1.33.1],SizeBytes:1235829,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8],SizeBytes:1159025,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/pause@sha256:e3b8c20681593c21b344ad801fbb8abaf564427ee3a57a9fcfa3b455f917ce46 mcr.microsoft.com/oss/kubernetes/pause:3.4.1],SizeBytes:682696,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/disk.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-w0zpdk2w/providers/Microsoft.Compute/disks/pvc-e6291a08-817d-41da-ad5a-8f6c567374d6,DevicePath:,},},Config:nil,},} Jun 14 19:09:38.131: INFO: ... skipping 90 lines ... [90m/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning_test.go:40[0m [single-az] [90m/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning_test.go:44[0m [91m[1mshould receive FailedMount event with invalid mount options [kubernetes.io/azure-disk] [disk.csi.azure.com] [csi-azuredisk-scheduler-extender] [It][0m [90m/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning_test.go:269[0m [91mUnexpected error: <*errors.errorString | 0xc0009fc0f0>: { s: "PersistentVolume pvc-45e32b8f-048e-47db-9efd-1af2daa64036 still exists within 10m0s", } PersistentVolume pvc-45e32b8f-048e-47db-9efd-1af2daa64036 still exists within 10m0s occurred[0m ... skipping 15 lines ... [1mSTEP[0m: setting up the StorageClass [1mSTEP[0m: creating a StorageClass [1mSTEP[0m: setting up the PVC and PV [1mSTEP[0m: creating a PVC [1mSTEP[0m: setting up the pod [1mSTEP[0m: deploying the pod [1mSTEP[0m: checking that the pod's command exits with no error Jun 14 19:09:40.924: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-7mqk5" in namespace "azuredisk-306" to be "Succeeded or Failed" Jun 14 19:09:40.978: INFO: Pod "azuredisk-volume-tester-7mqk5": Phase="Pending", Reason="", readiness=false. Elapsed: 54.433332ms Jun 14 19:09:43.034: INFO: Pod "azuredisk-volume-tester-7mqk5": Phase="Pending", Reason="", readiness=false. Elapsed: 2.110808642s Jun 14 19:09:45.092: INFO: Pod "azuredisk-volume-tester-7mqk5": Phase="Pending", Reason="", readiness=false. Elapsed: 4.167858183s Jun 14 19:09:47.149: INFO: Pod "azuredisk-volume-tester-7mqk5": Phase="Pending", Reason="", readiness=false. Elapsed: 6.225710189s Jun 14 19:09:49.206: INFO: Pod "azuredisk-volume-tester-7mqk5": Phase="Pending", Reason="", readiness=false. Elapsed: 8.282624618s Jun 14 19:09:51.266: INFO: Pod "azuredisk-volume-tester-7mqk5": Phase="Pending", Reason="", readiness=false. Elapsed: 10.342177917s ... skipping 427 lines ... Jun 14 19:24:31.452: INFO: Pod "azuredisk-volume-tester-7mqk5": Phase="Pending", Reason="", readiness=false. Elapsed: 14m50.528645019s Jun 14 19:24:33.508: INFO: Pod "azuredisk-volume-tester-7mqk5": Phase="Pending", Reason="", readiness=false. Elapsed: 14m52.5838714s Jun 14 19:24:35.564: INFO: Pod "azuredisk-volume-tester-7mqk5": Phase="Pending", Reason="", readiness=false. Elapsed: 14m54.640311922s Jun 14 19:24:37.620: INFO: Pod "azuredisk-volume-tester-7mqk5": Phase="Pending", Reason="", readiness=false. Elapsed: 14m56.696590251s Jun 14 19:24:39.676: INFO: Pod "azuredisk-volume-tester-7mqk5": Phase="Pending", Reason="", readiness=false. Elapsed: 14m58.752803425s Jun 14 19:24:41.678: INFO: deleting Pod "azuredisk-306"/"azuredisk-volume-tester-7mqk5" Jun 14 19:24:41.766: INFO: Error getting logs for pod azuredisk-volume-tester-7mqk5: the server rejected our request for an unknown reason (get pods azuredisk-volume-tester-7mqk5) [1mSTEP[0m: Deleting pod azuredisk-volume-tester-7mqk5 in namespace azuredisk-306 [1mSTEP[0m: validating provisioned PV [1mSTEP[0m: checking the PV Jun 14 19:24:41.932: INFO: deleting PVC "azuredisk-306"/"pvc-bszlc" Jun 14 19:24:41.932: INFO: Deleting PersistentVolumeClaim "pvc-bszlc" [1mSTEP[0m: waiting for claim's PV "pvc-b914cba7-48ba-457d-809b-8ae01ce1e4cb" to be deleted ... skipping 124 lines ... Jun 14 19:34:43.816: INFO: At 2022-06-14 19:09:40 +0000 UTC - event for pvc-bszlc: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding Jun 14 19:34:43.816: INFO: At 2022-06-14 19:09:40 +0000 UTC - event for pvc-bszlc: {disk.csi.azure.com_k8s-master-20224640-0_7f146b18-d5c4-4b0e-86dd-3b66836d87bc } Provisioning: External provisioner is provisioning volume for claim "azuredisk-306/pvc-bszlc" Jun 14 19:34:43.816: INFO: At 2022-06-14 19:09:40 +0000 UTC - event for pvc-bszlc: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "disk.csi.azure.com" or manually created by system administrator Jun 14 19:34:43.816: INFO: At 2022-06-14 19:09:43 +0000 UTC - event for pvc-bszlc: {disk.csi.azure.com_k8s-master-20224640-0_7f146b18-d5c4-4b0e-86dd-3b66836d87bc } ProvisioningSucceeded: Successfully provisioned volume pvc-b914cba7-48ba-457d-809b-8ae01ce1e4cb Jun 14 19:34:43.816: INFO: At 2022-06-14 19:09:45 +0000 UTC - event for azuredisk-volume-tester-7mqk5: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-b914cba7-48ba-457d-809b-8ae01ce1e4cb" Jun 14 19:34:43.816: INFO: At 2022-06-14 19:11:46 +0000 UTC - event for azuredisk-volume-tester-7mqk5: {kubelet k8s-agentpool1-20224640-0} FailedMount: Unable to attach or mount volumes: unmounted volumes=[test-volume-1], unattached volumes=[kube-api-access-fxt7c test-volume-1]: timed out waiting for the condition Jun 14 19:34:43.816: INFO: At 2022-06-14 19:11:53 +0000 UTC - event for azuredisk-volume-tester-7mqk5: {kubelet k8s-agentpool1-20224640-0} FailedMapVolume: MapVolume.SetUpDevice failed for volume "pvc-b914cba7-48ba-457d-809b-8ae01ce1e4cb" : rpc error: code = DeadlineExceeded desc = context deadline exceeded Jun 14 19:34:43.876: INFO: POD NODE PHASE GRACE CONDITIONS Jun 14 19:34:43.876: INFO: Jun 14 19:34:43.982: INFO: Logging node info for node k8s-agentpool1-20224640-0 Jun 14 19:34:44.036: INFO: Node Info: &Node{ObjectMeta:{k8s-agentpool1-20224640-0 d40122eb-7d63-4c24-87d0-a4294fda80c2 15483 0 2022-06-14 18:19:50 +0000 UTC <nil> <nil> map[agentpool:agentpool1 beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_DS2_v2 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:0 kubernetes.azure.com/cluster:kubetest-w0zpdk2w kubernetes.azure.com/role:agent kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-agentpool1-20224640-0 kubernetes.io/os:linux kubernetes.io/role:agent node-role.kubernetes.io/agent: node.kubernetes.io/instance-type:Standard_DS2_v2 storageprofile:managed storagetier:Premium_LRS topology.disk.csi.azure.com/zone: topology.kubernetes.io/region:westus2 topology.kubernetes.io/zone:0] map[csi.volume.kubernetes.io/nodeid:{"disk.csi.azure.com":"k8s-agentpool1-20224640-0"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2022-06-14 18:19:50 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:agentpool":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.azure.com/cluster":{},"f:kubernetes.azure.com/role":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:storageprofile":{},"f:storagetier":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kubectl-label Update v1 2022-06-14 18:19:58 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/agent":{}}}} } {kube-controller-manager Update v1 2022-06-14 18:20:04 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"10.244.0.0/24\"":{}}}} } {kubelet Update v1 2022-06-14 19:27:03 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.disk.csi.azure.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status} {kube-controller-manager Update v1 2022-06-14 19:34:25 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:volumesAttached":{}}} status}]},Spec:NodeSpec{PodCIDR:10.244.0.0/24,DoNotUseExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-w0zpdk2w/providers/Microsoft.Compute/virtualMachines/k8s-agentpool1-20224640-0,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[10.244.0.0/24],},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-azure-disk: {{8 0} {<nil>} 8 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{31036686336 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7287771136 0} {<nil>} 7116964Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-azure-disk: {{8 0} {<nil>} 8 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{27933017657 0} {<nil>} 27933017657 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{6501339136 0} {<nil>} 6348964Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-06-14 18:20:25 +0000 UTC,LastTransitionTime:2022-06-14 18:20:25 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-06-14 19:32:10 +0000 UTC,LastTransitionTime:2022-06-14 18:19:49 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-06-14 19:32:10 +0000 UTC,LastTransitionTime:2022-06-14 18:19:49 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-06-14 19:32:10 +0000 UTC,LastTransitionTime:2022-06-14 18:19:49 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-06-14 19:32:10 +0000 UTC,LastTransitionTime:2022-06-14 18:20:04 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:Hostname,Address:k8s-agentpool1-20224640-0,},NodeAddress{Type:InternalIP,Address:10.240.0.4,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:fcfd567ea6a54e9fb7ca2e3949562363,SystemUUID:c20271e3-7d7e-f940-9c6d-928cc89ffc2d,BootID:2a1d1325-bc1b-47c9-9540-2fd7da933292,KernelVersion:5.4.0-1074-azure,OSImage:Ubuntu 18.04.6 LTS,ContainerRuntimeVersion:docker://20.10.11+azure-3,KubeletVersion:v1.23.7,KubeProxyVersion:v1.23.7,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/azuredisk-csi@sha256:423eb6cf602c064c8b2deefead5ceadd6324ed41b3d995dab5d0f6f0f4d4710f mcr.microsoft.com/oss/kubernetes-csi/azuredisk-csi:v1.10.0],SizeBytes:245959792,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/azurefile-csi@sha256:9e2ecabcf9dd9943e6600eb9fb460f45b4dc61af7cabe95d115082a029db2aaf mcr.microsoft.com/oss/kubernetes-csi/azurefile-csi:v1.9.0],SizeBytes:230470852,},ContainerImage{Names:[mcr.microsoft.com/containernetworking/azure-npm@sha256:106f669f48e5e80c4ec0afb49858ead72cf4b901cd8664e7bf81f8d789e56e12 mcr.microsoft.com/containernetworking/azure-npm:v1.2.2_hotfix],SizeBytes:175230380,},ContainerImage{Names:[k8sprow.azurecr.io/azuredisk-csi@sha256:528fc89c187c74420dabdef86e06f05893e1510fce69111a2a05280dd64be2a9 k8sprow.azurecr.io/azuredisk-csi:latest-v2-df2db40004aa147caa3762ffb782a447d13cb39d],SizeBytes:136461498,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/secrets-store/driver@sha256:c0d040a1c4fbfceb65663e31c09ea40f4f78e356437610cbc3fbb4bb409bd6f1 mcr.microsoft.com/oss/kubernetes-csi/secrets-store/driver:v0.0.19],SizeBytes:123229697,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-proxy@sha256:0c890e918d42c158fab68867bade2ab1db04c3408452db9611a264ba3e43e6b3 mcr.microsoft.com/oss/kubernetes/kube-proxy:v1.23.7],SizeBytes:112296063,},ContainerImage{Names:[mcr.microsoft.com/oss/azure/secrets-store/provider-azure@sha256:6f67f3d0c7cdde5702f8ce7f101b6519daa0237f0c34fecb7c058b6af8c22ad1 mcr.microsoft.com/oss/azure/secrets-store/provider-azure:0.0.12],SizeBytes:101061355,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/autoscaler/cluster-autoscaler@sha256:6f0c680d375c62e74351f8ff3ed6ddb9b72ca759e0645c329b95f64264654a6d mcr.microsoft.com/oss/kubernetes/autoscaler/cluster-autoscaler:v1.22.1],SizeBytes:99962810,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-addon-manager@sha256:32e2836018c96e73533bd4642fe438e465b81dcbfa8b7b61935a6f4d0246c7ae mcr.microsoft.com/oss/kubernetes/kube-addon-manager:v9.1.3],SizeBytes:86832059,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-addon-manager@sha256:92c2c5aad9012ee32d2a43a74966cc0adc6ccb1705ad15abb10485ecf406d88b mcr.microsoft.com/oss/kubernetes/kube-addon-manager:v9.1.5],SizeBytes:84094027,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/metrics-server@sha256:1ef9d57ce41ffcc328b92494c3bfafe401e0b9a1694a295301a1385337d52815 mcr.microsoft.com/oss/kubernetes/metrics-server:v0.5.2],SizeBytes:64327621,},ContainerImage{Names:[mcr.microsoft.com/oss/nvidia/k8s-device-plugin@sha256:0f5b52bf28239234e831697d96db63ac03cde70fe68058f964504ab7564ee810 mcr.microsoft.com/oss/nvidia/k8s-device-plugin:1.0.0-beta6],SizeBytes:64160241,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-provisioner@sha256:e9ddadc44ba87a4a27f67e54760a14f9986885b534b3dff170a14eae1e35d213 mcr.microsoft.com/oss/kubernetes-csi/csi-provisioner:v3.0.0],SizeBytes:56881280,},ContainerImage{Names:[k8sprow.azurecr.io/azdiskschedulerextender-csi@sha256:3df91ffec98f04827c70772d08849e6874d1e0ee2432173ef523c099553547dc k8sprow.azurecr.io/azdiskschedulerextender-csi:latest-v2-df2db40004aa147caa3762ffb782a447d13cb39d],SizeBytes:56760067,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-resizer@sha256:c5bb71ceaac60b1a4b58739fa07b709f6248c452ff6272a384d2f7648895a750 mcr.microsoft.com/oss/kubernetes-csi/csi-resizer:v1.3.0],SizeBytes:54313772,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-snapshotter@sha256:61849a026511cf332c87d73d0a7aed803b510c3ede197ec755389686d490de72 mcr.microsoft.com/oss/kubernetes-csi/csi-snapshotter:v4.2.1],SizeBytes:54210936,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-attacher@sha256:6b41e7153ebdfdc1501aa65184624bc15fd33a52d93f88ec3a758d0f8c9b8c10 mcr.microsoft.com/oss/kubernetes-csi/csi-attacher:v3.3.0],SizeBytes:53842561,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/snapshot-controller@sha256:8c3fc3c2667004ad6bbdf723bb64c5da66a5cb8b11d4ee59b67179b686223b13 mcr.microsoft.com/oss/kubernetes-csi/snapshot-controller:v5.0.1],SizeBytes:52732401,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/snapshot-controller@sha256:be5a8dc1f990828f653c77bd0a0f1bbd13197c3019f6e1d99d590389bac36705 mcr.microsoft.com/oss/kubernetes-csi/snapshot-controller:v4.2.1],SizeBytes:51575245,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager@sha256:0ad67f9919522a07318034641ae09bf2079b417e9944c65914410594ce645468 mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager:v1.1.4],SizeBytes:51478397,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager@sha256:31ec4f7daccd3e7a8504e12657d7830651ecacbe4a487daca1b1b7695a64b070 mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager:v1.23.1],SizeBytes:51249021,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager@sha256:011712ed90fb8efcf27928b0a47ed04b98baebb31cb1b2d8ab676977ec18eedc mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager:v1.1.4],SizeBytes:50868093,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager@sha256:0f9a8fbaed65192ed7dd795be4f9c1dc48ebdef0a241fb62d456f4bed40d9875 mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager:v1.23.1],SizeBytes:50679677,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-scheduler@sha256:c35f45e8dfaa30363e589296415a057f9d32813920fc3820d6557491657f199a mcr.microsoft.com/oss/kubernetes/kube-scheduler:v1.21.2],SizeBytes:50607491,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/ip-masq-agent@sha256:1244155f2ed3f33ff154cc343b8ad285f3391d95afd7d4b1c6dcc420bc0ba3cf mcr.microsoft.com/oss/kubernetes/ip-masq-agent:v2.5.0],SizeBytes:50146762,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager@sha256:7c907ff70b90a0bdf8fae63bd744018469dd9839cde1fd0515b93e0bbd14b34e mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager:v1.0.8],SizeBytes:48963453,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager@sha256:3987d7a8c6922ce1952ee19c5cb6ea75aac7b7c1b07aa79277ad038c69fb7a31 mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager:v1.0.8],SizeBytes:48349053,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/coredns@sha256:f873bf7f0928461efe10697fa76cf0ad7a1ae3041c5b57b50dd3d0b72d273f8c mcr.microsoft.com/oss/kubernetes/coredns:1.8.6],SizeBytes:46804601,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager@sha256:8073113a20882642a980b338635cdc5945e5673a18aef192090e6fde2b89a75c mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager:v0.6.0],SizeBytes:45909032,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager@sha256:6a32329628bdea3c6d75e98aad6155b65d2e2b98ca616eb33f9ac562912804c6 mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager:v0.6.0],SizeBytes:45229096,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager@sha256:ef6c4ba564b4d11d270f7d1563c50fbeb30ccc3b94146e5059228c49f95875f5 mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager:v0.7.11],SizeBytes:44916605,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager@sha256:dbcc384758ba5ca6d249596d471292ed3785e31cdb854d48b84d70794b669b4c mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager:v0.7.11],SizeBytes:43679613,},ContainerImage{Names:[mcr.microsoft.com/oss/etcd-io/etcd@sha256:cf587862e3f1b6fa4d9a2565520a34f164bdf72c50f37af8c3c668160593246e mcr.microsoft.com/oss/etcd-io/etcd:v3.3.25],SizeBytes:41832119,},ContainerImage{Names:[mcr.microsoft.com/k8s/aad-pod-identity/mic@sha256:bd9465be94966b9a917e1e3904fa5e63dd91772ccadf304e18ffd8e4ad8ccedd mcr.microsoft.com/k8s/aad-pod-identity/mic:1.6.1],SizeBytes:41374894,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/autoscaler/cluster-proportional-autoscaler@sha256:c849d75d61943ce7f51b4c049f1a79d19a08253966c8f49c4cfb6414cc33db8b mcr.microsoft.com/oss/kubernetes/autoscaler/cluster-proportional-autoscaler:1.8.5],SizeBytes:40661903,},ContainerImage{Names:[mcr.microsoft.com/k8s/aad-pod-identity/nmi@sha256:02128fefcdb7593ac53fc342e2c53a0fc6fabd813036bf60457bf43cc2940116 mcr.microsoft.com/k8s/aad-pod-identity/nmi:1.6.1],SizeBytes:38007982,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:921f301c44dda06a325164accf22e78ecc570b5c7d9d6ee4c66bd8cbb2b60b9a mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.16],SizeBytes:26970670,},ContainerImage{Names:[mcr.microsoft.com/k8s/kms/keyvault@sha256:1a27e175f8c125209e32d2957b5509fe20757bd8cb309ff9da598799b56326fb mcr.microsoft.com/k8s/kms/keyvault:v0.0.10],SizeBytes:23077387,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-node-driver-registrar@sha256:dbec3a8166686b09b242176ab5b99e993da4126438bbce68147c3fd654f35662 mcr.microsoft.com/oss/kubernetes-csi/csi-node-driver-registrar:v2.4.0],SizeBytes:19547289,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/livenessprobe@sha256:c96a6255c42766f6b8bb1a7cda02b0060ab1b20b2e2dafcc64ec09e7646745a6 mcr.microsoft.com/oss/kubernetes-csi/livenessprobe:v2.5.0],SizeBytes:17573341,},ContainerImage{Names:[mcr.microsoft.com/oss/busybox/busybox@sha256:582a641242b49809af3a1a522f9aae8c3f047d1c6ca1dd9d8cdabd349e45b1a9 mcr.microsoft.com/oss/busybox/busybox:1.33.1],SizeBytes:1235829,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8],SizeBytes:1159025,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/pause@sha256:e3b8c20681593c21b344ad801fbb8abaf564427ee3a57a9fcfa3b455f917ce46 mcr.microsoft.com/oss/kubernetes/pause:3.4.1],SizeBytes:682696,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/disk.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-w0zpdk2w/providers/Microsoft.Compute/disks/pvc-e6291a08-817d-41da-ad5a-8f6c567374d6,DevicePath:,},},Config:nil,},} Jun 14 19:34:44.036: INFO: ... skipping 90 lines ... [90m/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning_test.go:40[0m [single-az] [90m/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning_test.go:44[0m [91m[1mshould create a raw block volume on demand [kubernetes.io/azure-disk] [disk.csi.azure.com] [csi-azuredisk-scheduler-extender] [It][0m [90m/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning_test.go:306[0m [91mUnexpected error: <*errors.errorString | 0xc000aa46d0>: { s: "Gave up after waiting 15m0s for pod \"azuredisk-volume-tester-7mqk5\" to be \"Succeeded or Failed\"", } Gave up after waiting 15m0s for pod "azuredisk-volume-tester-7mqk5" to be "Succeeded or Failed" occurred[0m /home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/resources/pod.go:89 [90m------------------------------[0m [0mDynamic Provisioning[0m [90m[single-az][0m [1mshould create a volume on demand and mount it as readOnly in a pod [kubernetes.io/azure-disk] [disk.csi.azure.com] [Windows] [csi-azuredisk-scheduler-extender][0m ... skipping 11 lines ... [1mSTEP[0m: setting up the StorageClass [1mSTEP[0m: creating a StorageClass [1mSTEP[0m: setting up the PVC and PV [1mSTEP[0m: creating a PVC [1mSTEP[0m: setting up the pod [1mSTEP[0m: deploying the pod [1mSTEP[0m: checking that the pod's command exits with an error Jun 14 19:34:46.333: INFO: Waiting up to 10m0s for pod "azuredisk-volume-tester-xd2b9" in namespace "azuredisk-5743" to be "Error status code" Jun 14 19:34:46.387: INFO: Pod "azuredisk-volume-tester-xd2b9": Phase="Pending", Reason="", readiness=false. Elapsed: 53.792777ms Jun 14 19:34:48.441: INFO: Pod "azuredisk-volume-tester-xd2b9": Phase="Pending", Reason="", readiness=false. Elapsed: 2.108022907s Jun 14 19:34:50.495: INFO: Pod "azuredisk-volume-tester-xd2b9": Phase="Pending", Reason="", readiness=false. Elapsed: 4.162157019s Jun 14 19:34:52.549: INFO: Pod "azuredisk-volume-tester-xd2b9": Phase="Pending", Reason="", readiness=false. Elapsed: 6.216642263s Jun 14 19:34:54.604: INFO: Pod "azuredisk-volume-tester-xd2b9": Phase="Pending", Reason="", readiness=false. Elapsed: 8.271495126s Jun 14 19:34:56.659: INFO: Pod "azuredisk-volume-tester-xd2b9": Phase="Pending", Reason="", readiness=false. Elapsed: 10.326213977s ... skipping 282 lines ... Jun 14 19:44:38.069: INFO: Pod "azuredisk-volume-tester-xd2b9": Phase="Pending", Reason="", readiness=false. Elapsed: 9m51.736499467s Jun 14 19:44:40.122: INFO: Pod "azuredisk-volume-tester-xd2b9": Phase="Pending", Reason="", readiness=false. Elapsed: 9m53.789685118s Jun 14 19:44:42.176: INFO: Pod "azuredisk-volume-tester-xd2b9": Phase="Pending", Reason="", readiness=false. Elapsed: 9m55.842709172s Jun 14 19:44:44.229: INFO: Pod "azuredisk-volume-tester-xd2b9": Phase="Pending", Reason="", readiness=false. Elapsed: 9m57.896175109s Jun 14 19:44:46.283: INFO: Pod "azuredisk-volume-tester-xd2b9": Phase="Pending", Reason="", readiness=false. Elapsed: 9m59.949969936s Jun 14 19:44:48.284: INFO: deleting Pod "azuredisk-5743"/"azuredisk-volume-tester-xd2b9" Jun 14 19:44:48.364: INFO: Error getting logs for pod azuredisk-volume-tester-xd2b9: the server rejected our request for an unknown reason (get pods azuredisk-volume-tester-xd2b9) [1mSTEP[0m: Deleting pod azuredisk-volume-tester-xd2b9 in namespace azuredisk-5743 [1mSTEP[0m: validating provisioned PV [1mSTEP[0m: checking the PV Jun 14 19:44:48.530: INFO: deleting PVC "azuredisk-5743"/"pvc-ppcbc" Jun 14 19:44:48.530: INFO: Deleting PersistentVolumeClaim "pvc-ppcbc" [1mSTEP[0m: waiting for claim's PV "pvc-6454c46a-2b27-4b6f-92d5-b3ee4b0a3811" to be deleted ... skipping 123 lines ... Jun 14 19:54:50.350: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for azuredisk-volume-tester-xd2b9: { } Scheduled: Successfully assigned azuredisk-5743/azuredisk-volume-tester-xd2b9 to k8s-agentpool1-20224640-1 Jun 14 19:54:50.350: INFO: At 2022-06-14 19:34:46 +0000 UTC - event for pvc-ppcbc: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "disk.csi.azure.com" or manually created by system administrator Jun 14 19:54:50.350: INFO: At 2022-06-14 19:34:46 +0000 UTC - event for pvc-ppcbc: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding Jun 14 19:54:50.350: INFO: At 2022-06-14 19:34:46 +0000 UTC - event for pvc-ppcbc: {disk.csi.azure.com_k8s-master-20224640-0_7f146b18-d5c4-4b0e-86dd-3b66836d87bc } Provisioning: External provisioner is provisioning volume for claim "azuredisk-5743/pvc-ppcbc" Jun 14 19:54:50.350: INFO: At 2022-06-14 19:34:49 +0000 UTC - event for pvc-ppcbc: {disk.csi.azure.com_k8s-master-20224640-0_7f146b18-d5c4-4b0e-86dd-3b66836d87bc } ProvisioningSucceeded: Successfully provisioned volume pvc-6454c46a-2b27-4b6f-92d5-b3ee4b0a3811 Jun 14 19:54:50.350: INFO: At 2022-06-14 19:34:52 +0000 UTC - event for azuredisk-volume-tester-xd2b9: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-6454c46a-2b27-4b6f-92d5-b3ee4b0a3811" Jun 14 19:54:50.350: INFO: At 2022-06-14 19:36:52 +0000 UTC - event for azuredisk-volume-tester-xd2b9: {kubelet k8s-agentpool1-20224640-1} FailedMount: MountVolume.MountDevice failed for volume "pvc-6454c46a-2b27-4b6f-92d5-b3ee4b0a3811" : rpc error: code = DeadlineExceeded desc = context deadline exceeded Jun 14 19:54:50.350: INFO: At 2022-06-14 19:36:53 +0000 UTC - event for azuredisk-volume-tester-xd2b9: {kubelet k8s-agentpool1-20224640-1} FailedMount: Unable to attach or mount volumes: unmounted volumes=[test-volume-1], unattached volumes=[test-volume-1 kube-api-access-rf7nm]: timed out waiting for the condition Jun 14 19:54:50.350: INFO: At 2022-06-14 19:39:07 +0000 UTC - event for azuredisk-volume-tester-xd2b9: {kubelet k8s-agentpool1-20224640-1} FailedMount: Unable to attach or mount volumes: unmounted volumes=[test-volume-1], unattached volumes=[kube-api-access-rf7nm test-volume-1]: timed out waiting for the condition Jun 14 19:54:50.405: INFO: POD NODE PHASE GRACE CONDITIONS Jun 14 19:54:50.405: INFO: Jun 14 19:54:50.514: INFO: Logging node info for node k8s-agentpool1-20224640-0 ... skipping 92 lines ... [90m/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning_test.go:40[0m [single-az] [90m/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning_test.go:44[0m [91m[1mshould create a volume on demand and mount it as readOnly in a pod [kubernetes.io/azure-disk] [disk.csi.azure.com] [Windows] [csi-azuredisk-scheduler-extender] [It][0m [90m/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning_test.go:338[0m [91mUnexpected error: <*errors.errorString | 0xc000aa59f0>: { s: "Gave up after waiting 10m0s for pod \"azuredisk-volume-tester-xd2b9\" to be \"Error status code\"", } Gave up after waiting 10m0s for pod "azuredisk-volume-tester-xd2b9" to be "Error status code" occurred[0m /home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/resources/pod.go:114 [90m------------------------------[0m [0mDynamic Provisioning[0m [90m[single-az][0m [1mshould create multiple PV objects, bind to PVCs and attach all to different pods on the same node [kubernetes.io/azure-disk] [disk.csi.azure.com] [Windows] [csi-azuredisk-scheduler-extender][0m ... skipping 13 lines ... [1mSTEP[0m: setting up the PVC and PV [1mSTEP[0m: creating a PVC [1mSTEP[0m: setting up the pod [1mSTEP[0m: deploying the pod [1mSTEP[0m: checking that the pod is running Jun 14 20:09:53.412: INFO: deleting Pod "azuredisk-1188"/"azuredisk-volume-tester-f6whr" Jun 14 20:09:53.501: INFO: Error getting logs for pod azuredisk-volume-tester-f6whr: the server rejected our request for an unknown reason (get pods azuredisk-volume-tester-f6whr) [1mSTEP[0m: Deleting pod azuredisk-volume-tester-f6whr in namespace azuredisk-1188 [1mSTEP[0m: validating provisioned PV [1mSTEP[0m: checking the PV Jun 14 20:09:53.663: INFO: deleting PVC "azuredisk-1188"/"pvc-99hns" Jun 14 20:09:53.663: INFO: Deleting PersistentVolumeClaim "pvc-99hns" [1mSTEP[0m: waiting for claim's PV "pvc-28956818-78c7-4087-b9b3-a6f87aabc914" to be deleted ... skipping 123 lines ... Jun 14 20:19:55.588: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for azuredisk-volume-tester-f6whr: { } Scheduled: Successfully assigned azuredisk-1188/azuredisk-volume-tester-f6whr to k8s-agentpool1-20224640-0 Jun 14 20:19:55.588: INFO: At 2022-06-14 19:54:53 +0000 UTC - event for pvc-99hns: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "disk.csi.azure.com" or manually created by system administrator Jun 14 20:19:55.588: INFO: At 2022-06-14 19:54:53 +0000 UTC - event for pvc-99hns: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding Jun 14 20:19:55.588: INFO: At 2022-06-14 19:54:53 +0000 UTC - event for pvc-99hns: {disk.csi.azure.com_k8s-master-20224640-0_7f146b18-d5c4-4b0e-86dd-3b66836d87bc } Provisioning: External provisioner is provisioning volume for claim "azuredisk-1188/pvc-99hns" Jun 14 20:19:55.588: INFO: At 2022-06-14 19:54:55 +0000 UTC - event for pvc-99hns: {disk.csi.azure.com_k8s-master-20224640-0_7f146b18-d5c4-4b0e-86dd-3b66836d87bc } ProvisioningSucceeded: Successfully provisioned volume pvc-28956818-78c7-4087-b9b3-a6f87aabc914 Jun 14 20:19:55.588: INFO: At 2022-06-14 19:54:57 +0000 UTC - event for azuredisk-volume-tester-f6whr: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-28956818-78c7-4087-b9b3-a6f87aabc914" Jun 14 20:19:55.588: INFO: At 2022-06-14 19:56:58 +0000 UTC - event for azuredisk-volume-tester-f6whr: {kubelet k8s-agentpool1-20224640-0} FailedMount: MountVolume.MountDevice failed for volume "pvc-28956818-78c7-4087-b9b3-a6f87aabc914" : rpc error: code = DeadlineExceeded desc = context deadline exceeded Jun 14 20:19:55.588: INFO: At 2022-06-14 19:56:59 +0000 UTC - event for azuredisk-volume-tester-f6whr: {kubelet k8s-agentpool1-20224640-0} FailedMount: Unable to attach or mount volumes: unmounted volumes=[test-volume-1], unattached volumes=[test-volume-1 kube-api-access-tp8dl]: timed out waiting for the condition Jun 14 20:19:55.588: INFO: At 2022-06-14 19:59:14 +0000 UTC - event for azuredisk-volume-tester-f6whr: {kubelet k8s-agentpool1-20224640-0} FailedMount: Unable to attach or mount volumes: unmounted volumes=[test-volume-1], unattached volumes=[kube-api-access-tp8dl test-volume-1]: timed out waiting for the condition Jun 14 20:19:55.642: INFO: POD NODE PHASE GRACE CONDITIONS Jun 14 20:19:55.642: INFO: Jun 14 20:19:55.748: INFO: Logging node info for node k8s-agentpool1-20224640-0 ... skipping 92 lines ... [90m/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning_test.go:40[0m [single-az] [90m/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning_test.go:44[0m [91m[1mshould create multiple PV objects, bind to PVCs and attach all to different pods on the same node [kubernetes.io/azure-disk] [disk.csi.azure.com] [Windows] [csi-azuredisk-scheduler-extender] [It][0m [90m/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning_test.go:372[0m [91mUnexpected error: <*errors.errorString | 0xc0002ab250>: { s: "timed out waiting for the condition", } timed out waiting for the condition occurred[0m ... skipping 305 lines ... Jun 14 20:35:00.406: INFO: At 2022-06-14 20:19:58 +0000 UTC - event for azuredisk-volume-tester-plfv7-c575df97f: {replicaset-controller } SuccessfulCreate: Created pod: azuredisk-volume-tester-plfv7-c575df97f-xlqhx Jun 14 20:35:00.406: INFO: At 2022-06-14 20:19:58 +0000 UTC - event for pvc-f62dk: {disk.csi.azure.com_k8s-master-20224640-0_7f146b18-d5c4-4b0e-86dd-3b66836d87bc } Provisioning: External provisioner is provisioning volume for claim "azuredisk-6303/pvc-f62dk" Jun 14 20:35:00.406: INFO: At 2022-06-14 20:19:58 +0000 UTC - event for pvc-f62dk: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "disk.csi.azure.com" or manually created by system administrator Jun 14 20:35:00.406: INFO: At 2022-06-14 20:20:00 +0000 UTC - event for pvc-f62dk: {disk.csi.azure.com_k8s-master-20224640-0_7f146b18-d5c4-4b0e-86dd-3b66836d87bc } ProvisioningSucceeded: Successfully provisioned volume pvc-06f78b88-898a-485f-a227-353dc87b1b7d Jun 14 20:35:00.406: INFO: At 2022-06-14 20:20:02 +0000 UTC - event for azuredisk-volume-tester-plfv7-c575df97f-xlqhx: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-06f78b88-898a-485f-a227-353dc87b1b7d" Jun 14 20:35:00.406: INFO: At 2022-06-14 20:22:04 +0000 UTC - event for azuredisk-volume-tester-plfv7-c575df97f-xlqhx: {kubelet k8s-agentpool1-20224640-1} FailedMount: Unable to attach or mount volumes: unmounted volumes=[test-volume-1], unattached volumes=[test-volume-1 kube-api-access-hw4g7]: timed out waiting for the condition Jun 14 20:35:00.406: INFO: At 2022-06-14 20:22:11 +0000 UTC - event for azuredisk-volume-tester-plfv7-c575df97f-xlqhx: {kubelet k8s-agentpool1-20224640-1} FailedMount: MountVolume.MountDevice failed for volume "pvc-06f78b88-898a-485f-a227-353dc87b1b7d" : rpc error: code = DeadlineExceeded desc = context deadline exceeded Jun 14 20:35:00.460: INFO: POD NODE PHASE GRACE CONDITIONS Jun 14 20:35:00.460: INFO: Jun 14 20:35:00.569: INFO: Logging node info for node k8s-agentpool1-20224640-0 Jun 14 20:35:00.624: INFO: Node Info: &Node{ObjectMeta:{k8s-agentpool1-20224640-0 d40122eb-7d63-4c24-87d0-a4294fda80c2 28705 0 2022-06-14 18:19:50 +0000 UTC <nil> <nil> map[agentpool:agentpool1 beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_DS2_v2 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:0 kubernetes.azure.com/cluster:kubetest-w0zpdk2w kubernetes.azure.com/role:agent kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-agentpool1-20224640-0 kubernetes.io/os:linux kubernetes.io/role:agent node-role.kubernetes.io/agent: node.kubernetes.io/instance-type:Standard_DS2_v2 storageprofile:managed storagetier:Premium_LRS topology.disk.csi.azure.com/zone: topology.kubernetes.io/region:westus2 topology.kubernetes.io/zone:0] map[csi.volume.kubernetes.io/nodeid:{"disk.csi.azure.com":"k8s-agentpool1-20224640-0"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2022-06-14 18:19:50 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:agentpool":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.azure.com/cluster":{},"f:kubernetes.azure.com/role":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:storageprofile":{},"f:storagetier":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kubectl-label Update v1 2022-06-14 18:19:58 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/agent":{}}}} } {kube-controller-manager Update v1 2022-06-14 18:20:04 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"10.244.0.0/24\"":{}}}} } {kube-controller-manager Update v1 2022-06-14 19:36:27 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:volumesAttached":{}}} status} {kubelet Update v1 2022-06-14 20:12:10 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.disk.csi.azure.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:10.244.0.0/24,DoNotUseExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-w0zpdk2w/providers/Microsoft.Compute/virtualMachines/k8s-agentpool1-20224640-0,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[10.244.0.0/24],},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-azure-disk: {{8 0} {<nil>} 8 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{31036686336 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7287771136 0} {<nil>} 7116964Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-azure-disk: {{8 0} {<nil>} 8 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{27933017657 0} {<nil>} 27933017657 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{6501339136 0} {<nil>} 6348964Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-06-14 18:20:25 +0000 UTC,LastTransitionTime:2022-06-14 18:20:25 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-06-14 20:32:37 +0000 UTC,LastTransitionTime:2022-06-14 18:19:49 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-06-14 20:32:37 +0000 UTC,LastTransitionTime:2022-06-14 18:19:49 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-06-14 20:32:37 +0000 UTC,LastTransitionTime:2022-06-14 18:19:49 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-06-14 20:32:37 +0000 UTC,LastTransitionTime:2022-06-14 18:20:04 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:Hostname,Address:k8s-agentpool1-20224640-0,},NodeAddress{Type:InternalIP,Address:10.240.0.4,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:fcfd567ea6a54e9fb7ca2e3949562363,SystemUUID:c20271e3-7d7e-f940-9c6d-928cc89ffc2d,BootID:2a1d1325-bc1b-47c9-9540-2fd7da933292,KernelVersion:5.4.0-1074-azure,OSImage:Ubuntu 18.04.6 LTS,ContainerRuntimeVersion:docker://20.10.11+azure-3,KubeletVersion:v1.23.7,KubeProxyVersion:v1.23.7,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/azuredisk-csi@sha256:423eb6cf602c064c8b2deefead5ceadd6324ed41b3d995dab5d0f6f0f4d4710f mcr.microsoft.com/oss/kubernetes-csi/azuredisk-csi:v1.10.0],SizeBytes:245959792,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/azurefile-csi@sha256:9e2ecabcf9dd9943e6600eb9fb460f45b4dc61af7cabe95d115082a029db2aaf mcr.microsoft.com/oss/kubernetes-csi/azurefile-csi:v1.9.0],SizeBytes:230470852,},ContainerImage{Names:[mcr.microsoft.com/containernetworking/azure-npm@sha256:106f669f48e5e80c4ec0afb49858ead72cf4b901cd8664e7bf81f8d789e56e12 mcr.microsoft.com/containernetworking/azure-npm:v1.2.2_hotfix],SizeBytes:175230380,},ContainerImage{Names:[k8sprow.azurecr.io/azuredisk-csi@sha256:528fc89c187c74420dabdef86e06f05893e1510fce69111a2a05280dd64be2a9 k8sprow.azurecr.io/azuredisk-csi:latest-v2-df2db40004aa147caa3762ffb782a447d13cb39d],SizeBytes:136461498,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/secrets-store/driver@sha256:c0d040a1c4fbfceb65663e31c09ea40f4f78e356437610cbc3fbb4bb409bd6f1 mcr.microsoft.com/oss/kubernetes-csi/secrets-store/driver:v0.0.19],SizeBytes:123229697,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-proxy@sha256:0c890e918d42c158fab68867bade2ab1db04c3408452db9611a264ba3e43e6b3 mcr.microsoft.com/oss/kubernetes/kube-proxy:v1.23.7],SizeBytes:112296063,},ContainerImage{Names:[mcr.microsoft.com/oss/azure/secrets-store/provider-azure@sha256:6f67f3d0c7cdde5702f8ce7f101b6519daa0237f0c34fecb7c058b6af8c22ad1 mcr.microsoft.com/oss/azure/secrets-store/provider-azure:0.0.12],SizeBytes:101061355,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/autoscaler/cluster-autoscaler@sha256:6f0c680d375c62e74351f8ff3ed6ddb9b72ca759e0645c329b95f64264654a6d mcr.microsoft.com/oss/kubernetes/autoscaler/cluster-autoscaler:v1.22.1],SizeBytes:99962810,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-addon-manager@sha256:32e2836018c96e73533bd4642fe438e465b81dcbfa8b7b61935a6f4d0246c7ae mcr.microsoft.com/oss/kubernetes/kube-addon-manager:v9.1.3],SizeBytes:86832059,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-addon-manager@sha256:92c2c5aad9012ee32d2a43a74966cc0adc6ccb1705ad15abb10485ecf406d88b mcr.microsoft.com/oss/kubernetes/kube-addon-manager:v9.1.5],SizeBytes:84094027,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/metrics-server@sha256:1ef9d57ce41ffcc328b92494c3bfafe401e0b9a1694a295301a1385337d52815 mcr.microsoft.com/oss/kubernetes/metrics-server:v0.5.2],SizeBytes:64327621,},ContainerImage{Names:[mcr.microsoft.com/oss/nvidia/k8s-device-plugin@sha256:0f5b52bf28239234e831697d96db63ac03cde70fe68058f964504ab7564ee810 mcr.microsoft.com/oss/nvidia/k8s-device-plugin:1.0.0-beta6],SizeBytes:64160241,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-provisioner@sha256:e9ddadc44ba87a4a27f67e54760a14f9986885b534b3dff170a14eae1e35d213 mcr.microsoft.com/oss/kubernetes-csi/csi-provisioner:v3.0.0],SizeBytes:56881280,},ContainerImage{Names:[k8sprow.azurecr.io/azdiskschedulerextender-csi@sha256:3df91ffec98f04827c70772d08849e6874d1e0ee2432173ef523c099553547dc k8sprow.azurecr.io/azdiskschedulerextender-csi:latest-v2-df2db40004aa147caa3762ffb782a447d13cb39d],SizeBytes:56760067,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-resizer@sha256:c5bb71ceaac60b1a4b58739fa07b709f6248c452ff6272a384d2f7648895a750 mcr.microsoft.com/oss/kubernetes-csi/csi-resizer:v1.3.0],SizeBytes:54313772,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-snapshotter@sha256:61849a026511cf332c87d73d0a7aed803b510c3ede197ec755389686d490de72 mcr.microsoft.com/oss/kubernetes-csi/csi-snapshotter:v4.2.1],SizeBytes:54210936,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-attacher@sha256:6b41e7153ebdfdc1501aa65184624bc15fd33a52d93f88ec3a758d0f8c9b8c10 mcr.microsoft.com/oss/kubernetes-csi/csi-attacher:v3.3.0],SizeBytes:53842561,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/snapshot-controller@sha256:8c3fc3c2667004ad6bbdf723bb64c5da66a5cb8b11d4ee59b67179b686223b13 mcr.microsoft.com/oss/kubernetes-csi/snapshot-controller:v5.0.1],SizeBytes:52732401,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/snapshot-controller@sha256:be5a8dc1f990828f653c77bd0a0f1bbd13197c3019f6e1d99d590389bac36705 mcr.microsoft.com/oss/kubernetes-csi/snapshot-controller:v4.2.1],SizeBytes:51575245,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager@sha256:0ad67f9919522a07318034641ae09bf2079b417e9944c65914410594ce645468 mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager:v1.1.4],SizeBytes:51478397,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager@sha256:31ec4f7daccd3e7a8504e12657d7830651ecacbe4a487daca1b1b7695a64b070 mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager:v1.23.1],SizeBytes:51249021,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager@sha256:011712ed90fb8efcf27928b0a47ed04b98baebb31cb1b2d8ab676977ec18eedc mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager:v1.1.4],SizeBytes:50868093,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager@sha256:0f9a8fbaed65192ed7dd795be4f9c1dc48ebdef0a241fb62d456f4bed40d9875 mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager:v1.23.1],SizeBytes:50679677,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-scheduler@sha256:c35f45e8dfaa30363e589296415a057f9d32813920fc3820d6557491657f199a mcr.microsoft.com/oss/kubernetes/kube-scheduler:v1.21.2],SizeBytes:50607491,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/ip-masq-agent@sha256:1244155f2ed3f33ff154cc343b8ad285f3391d95afd7d4b1c6dcc420bc0ba3cf mcr.microsoft.com/oss/kubernetes/ip-masq-agent:v2.5.0],SizeBytes:50146762,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager@sha256:7c907ff70b90a0bdf8fae63bd744018469dd9839cde1fd0515b93e0bbd14b34e mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager:v1.0.8],SizeBytes:48963453,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager@sha256:3987d7a8c6922ce1952ee19c5cb6ea75aac7b7c1b07aa79277ad038c69fb7a31 mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager:v1.0.8],SizeBytes:48349053,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/coredns@sha256:f873bf7f0928461efe10697fa76cf0ad7a1ae3041c5b57b50dd3d0b72d273f8c mcr.microsoft.com/oss/kubernetes/coredns:1.8.6],SizeBytes:46804601,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager@sha256:8073113a20882642a980b338635cdc5945e5673a18aef192090e6fde2b89a75c mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager:v0.6.0],SizeBytes:45909032,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager@sha256:6a32329628bdea3c6d75e98aad6155b65d2e2b98ca616eb33f9ac562912804c6 mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager:v0.6.0],SizeBytes:45229096,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager@sha256:ef6c4ba564b4d11d270f7d1563c50fbeb30ccc3b94146e5059228c49f95875f5 mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager:v0.7.11],SizeBytes:44916605,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager@sha256:dbcc384758ba5ca6d249596d471292ed3785e31cdb854d48b84d70794b669b4c mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager:v0.7.11],SizeBytes:43679613,},ContainerImage{Names:[mcr.microsoft.com/oss/etcd-io/etcd@sha256:cf587862e3f1b6fa4d9a2565520a34f164bdf72c50f37af8c3c668160593246e mcr.microsoft.com/oss/etcd-io/etcd:v3.3.25],SizeBytes:41832119,},ContainerImage{Names:[mcr.microsoft.com/k8s/aad-pod-identity/mic@sha256:bd9465be94966b9a917e1e3904fa5e63dd91772ccadf304e18ffd8e4ad8ccedd mcr.microsoft.com/k8s/aad-pod-identity/mic:1.6.1],SizeBytes:41374894,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/autoscaler/cluster-proportional-autoscaler@sha256:c849d75d61943ce7f51b4c049f1a79d19a08253966c8f49c4cfb6414cc33db8b mcr.microsoft.com/oss/kubernetes/autoscaler/cluster-proportional-autoscaler:1.8.5],SizeBytes:40661903,},ContainerImage{Names:[mcr.microsoft.com/k8s/aad-pod-identity/nmi@sha256:02128fefcdb7593ac53fc342e2c53a0fc6fabd813036bf60457bf43cc2940116 mcr.microsoft.com/k8s/aad-pod-identity/nmi:1.6.1],SizeBytes:38007982,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:921f301c44dda06a325164accf22e78ecc570b5c7d9d6ee4c66bd8cbb2b60b9a mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.16],SizeBytes:26970670,},ContainerImage{Names:[mcr.microsoft.com/k8s/kms/keyvault@sha256:1a27e175f8c125209e32d2957b5509fe20757bd8cb309ff9da598799b56326fb mcr.microsoft.com/k8s/kms/keyvault:v0.0.10],SizeBytes:23077387,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-node-driver-registrar@sha256:dbec3a8166686b09b242176ab5b99e993da4126438bbce68147c3fd654f35662 mcr.microsoft.com/oss/kubernetes-csi/csi-node-driver-registrar:v2.4.0],SizeBytes:19547289,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/livenessprobe@sha256:c96a6255c42766f6b8bb1a7cda02b0060ab1b20b2e2dafcc64ec09e7646745a6 mcr.microsoft.com/oss/kubernetes-csi/livenessprobe:v2.5.0],SizeBytes:17573341,},ContainerImage{Names:[mcr.microsoft.com/oss/busybox/busybox@sha256:582a641242b49809af3a1a522f9aae8c3f047d1c6ca1dd9d8cdabd349e45b1a9 mcr.microsoft.com/oss/busybox/busybox:1.33.1],SizeBytes:1235829,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8],SizeBytes:1159025,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/pause@sha256:e3b8c20681593c21b344ad801fbb8abaf564427ee3a57a9fcfa3b455f917ce46 mcr.microsoft.com/oss/kubernetes/pause:3.4.1],SizeBytes:682696,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/disk.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-w0zpdk2w/providers/Microsoft.Compute/disks/pvc-e6291a08-817d-41da-ad5a-8f6c567374d6,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/disk.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-w0zpdk2w/providers/Microsoft.Compute/disks/pvc-b914cba7-48ba-457d-809b-8ae01ce1e4cb,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/disk.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-w0zpdk2w/providers/Microsoft.Compute/disks/pvc-28956818-78c7-4087-b9b3-a6f87aabc914,DevicePath:,},},Config:nil,},} Jun 14 20:35:00.624: INFO: ... skipping 90 lines ... [90m/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning_test.go:40[0m [single-az] [90m/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning_test.go:44[0m [91m[1mshould create a deployment object, write and read to it, delete the pod and write and read to it again [kubernetes.io/azure-disk] [disk.csi.azure.com] [Windows] [csi-azuredisk-scheduler-extender] [It][0m [90m/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning_test.go:436[0m [91mUnexpected error: <*errors.errorString | 0xc0009fcd00>: { s: "error waiting for deployment \"azuredisk-volume-tester-plfv7\" status to match expectation: deployment status: v1.DeploymentStatus{ObservedGeneration:1, Replicas:1, UpdatedReplicas:1, ReadyReplicas:0, AvailableReplicas:0, UnavailableReplicas:1, Conditions:[]v1.DeploymentCondition{v1.DeploymentCondition{Type:\"Available\", Status:\"False\", LastUpdateTime:time.Date(2022, time.June, 14, 20, 19, 58, 0, time.Local), LastTransitionTime:time.Date(2022, time.June, 14, 20, 19, 58, 0, time.Local), Reason:\"MinimumReplicasUnavailable\", Message:\"Deployment does not have minimum availability.\"}, v1.DeploymentCondition{Type:\"Progressing\", Status:\"True\", LastUpdateTime:time.Date(2022, time.June, 14, 20, 19, 58, 0, time.Local), LastTransitionTime:time.Date(2022, time.June, 14, 20, 19, 58, 0, time.Local), Reason:\"ReplicaSetUpdated\", Message:\"ReplicaSet \\\"azuredisk-volume-tester-plfv7-c575df97f\\\" is progressing.\"}}, CollisionCount:(*int32)(nil)}", } error waiting for deployment "azuredisk-volume-tester-plfv7" status to match expectation: deployment status: v1.DeploymentStatus{ObservedGeneration:1, Replicas:1, UpdatedReplicas:1, ReadyReplicas:0, AvailableReplicas:0, UnavailableReplicas:1, Conditions:[]v1.DeploymentCondition{v1.DeploymentCondition{Type:"Available", Status:"False", LastUpdateTime:time.Date(2022, time.June, 14, 20, 19, 58, 0, time.Local), LastTransitionTime:time.Date(2022, time.June, 14, 20, 19, 58, 0, time.Local), Reason:"MinimumReplicasUnavailable", Message:"Deployment does not have minimum availability."}, v1.DeploymentCondition{Type:"Progressing", Status:"True", LastUpdateTime:time.Date(2022, time.June, 14, 20, 19, 58, 0, time.Local), LastTransitionTime:time.Date(2022, time.June, 14, 20, 19, 58, 0, time.Local), Reason:"ReplicaSetUpdated", Message:"ReplicaSet \"azuredisk-volume-tester-plfv7-c575df97f\" is progressing."}}, CollisionCount:(*int32)(nil)} occurred[0m /home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/resources/deployment.go:126 [90m------------------------------[0m [0mDynamic Provisioning[0m [90m[single-az][0m [1mshould delete PV with reclaimPolicy "Delete" [kubernetes.io/azure-disk] [disk.csi.azure.com] [Windows][0m ... skipping 150 lines ... [1mSTEP[0m: setting up the StorageClass [1mSTEP[0m: creating a StorageClass [1mSTEP[0m: setting up the PVC and PV [1mSTEP[0m: creating a PVC [1mSTEP[0m: setting up the pod [1mSTEP[0m: deploying the pod [1mSTEP[0m: checking that the pod's command exits with no error Jun 14 20:35:22.148: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-tpdqt" in namespace "azuredisk-2265" to be "Succeeded or Failed" Jun 14 20:35:22.206: INFO: Pod "azuredisk-volume-tester-tpdqt": Phase="Pending", Reason="", readiness=false. Elapsed: 57.748741ms Jun 14 20:35:24.264: INFO: Pod "azuredisk-volume-tester-tpdqt": Phase="Pending", Reason="", readiness=false. Elapsed: 2.115330728s Jun 14 20:35:26.320: INFO: Pod "azuredisk-volume-tester-tpdqt": Phase="Pending", Reason="", readiness=false. Elapsed: 4.171298264s Jun 14 20:35:28.375: INFO: Pod "azuredisk-volume-tester-tpdqt": Phase="Pending", Reason="", readiness=false. Elapsed: 6.226566348s Jun 14 20:35:30.431: INFO: Pod "azuredisk-volume-tester-tpdqt": Phase="Pending", Reason="", readiness=false. Elapsed: 8.282605746s Jun 14 20:35:32.491: INFO: Pod "azuredisk-volume-tester-tpdqt": Phase="Pending", Reason="", readiness=false. Elapsed: 10.342995466s ... skipping 427 lines ... Jun 14 20:50:12.491: INFO: Pod "azuredisk-volume-tester-tpdqt": Phase="Pending", Reason="", readiness=false. Elapsed: 14m50.342779772s Jun 14 20:50:14.546: INFO: Pod "azuredisk-volume-tester-tpdqt": Phase="Pending", Reason="", readiness=false. Elapsed: 14m52.397160187s Jun 14 20:50:16.600: INFO: Pod "azuredisk-volume-tester-tpdqt": Phase="Pending", Reason="", readiness=false. Elapsed: 14m54.451428577s Jun 14 20:50:18.655: INFO: Pod "azuredisk-volume-tester-tpdqt": Phase="Pending", Reason="", readiness=false. Elapsed: 14m56.506436129s Jun 14 20:50:20.711: INFO: Pod "azuredisk-volume-tester-tpdqt": Phase="Pending", Reason="", readiness=false. Elapsed: 14m58.562276718s Jun 14 20:50:22.712: INFO: deleting Pod "azuredisk-2265"/"azuredisk-volume-tester-tpdqt" Jun 14 20:50:22.792: INFO: Error getting logs for pod azuredisk-volume-tester-tpdqt: the server rejected our request for an unknown reason (get pods azuredisk-volume-tester-tpdqt) [1mSTEP[0m: Deleting pod azuredisk-volume-tester-tpdqt in namespace azuredisk-2265 [1mSTEP[0m: validating provisioned PV [1mSTEP[0m: checking the PV Jun 14 20:50:22.955: INFO: deleting PVC "azuredisk-2265"/"pvc-qv8ls" Jun 14 20:50:22.955: INFO: Deleting PersistentVolumeClaim "pvc-qv8ls" [1mSTEP[0m: waiting for claim's PV "pvc-46307ead-9ebe-420c-8343-39942020e2d3" to be deleted ... skipping 325 lines ... Jun 14 21:16:31.140: INFO: PersistentVolume pvc-eaf7b142-4cf7-487b-8d39-47b1a7796a37 found and phase=Released (6m4.153578253s) Jun 14 21:16:36.196: INFO: PersistentVolume pvc-eaf7b142-4cf7-487b-8d39-47b1a7796a37 found and phase=Released (6m9.210472457s) Jun 14 21:16:41.253: INFO: PersistentVolume pvc-eaf7b142-4cf7-487b-8d39-47b1a7796a37 found and phase=Released (6m14.266933473s) Jun 14 21:16:46.312: INFO: PersistentVolume pvc-eaf7b142-4cf7-487b-8d39-47b1a7796a37 found and phase=Released (6m19.326085179s) Jun 14 21:16:51.369: INFO: PersistentVolume pvc-eaf7b142-4cf7-487b-8d39-47b1a7796a37 found and phase=Released (6m24.38288457s) Jun 14 21:16:56.429: INFO: PersistentVolume pvc-eaf7b142-4cf7-487b-8d39-47b1a7796a37 found and phase=Released (6m29.442862779s) {"component":"entrypoint","file":"k8s.io/test-infra/prow/entrypoint/run.go:165","func":"k8s.io/test-infra/prow/entrypoint.Options.ExecuteProcess","level":"error","msg":"Process did not finish before 3h0m0s timeout","severity":"error","time":"2022-06-14T21:16:57Z"} ++ early_exit_handler ++ '[' -n 177 ']' ++ kill -TERM 177 ++ cleanup_dind ++ [[ true == \t\r\u\e ]] ++ echo 'Cleaning up after docker' ... skipping 71 lines ... Jun 14 21:20:28.957: INFO: At 2022-06-14 20:35:24 +0000 UTC - event for pvc-qv8ls: {disk.csi.azure.com_k8s-master-20224640-0_7f146b18-d5c4-4b0e-86dd-3b66836d87bc } ProvisioningSucceeded: Successfully provisioned volume pvc-46307ead-9ebe-420c-8343-39942020e2d3 Jun 14 21:20:28.958: INFO: At 2022-06-14 20:35:24 +0000 UTC - event for pvc-zn9ft: {disk.csi.azure.com_k8s-master-20224640-0_7f146b18-d5c4-4b0e-86dd-3b66836d87bc } ProvisioningSucceeded: Successfully provisioned volume pvc-a6919324-dfd5-458c-b905-552bf919db25 Jun 14 21:20:28.958: INFO: At 2022-06-14 20:35:26 +0000 UTC - event for azuredisk-volume-tester-tpdqt: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-46307ead-9ebe-420c-8343-39942020e2d3" Jun 14 21:20:28.958: INFO: At 2022-06-14 20:35:26 +0000 UTC - event for azuredisk-volume-tester-tpdqt: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-a6919324-dfd5-458c-b905-552bf919db25" Jun 14 21:20:28.958: INFO: At 2022-06-14 20:35:26 +0000 UTC - event for azuredisk-volume-tester-tpdqt: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-eaf7b142-4cf7-487b-8d39-47b1a7796a37" Jun 14 21:20:28.958: INFO: At 2022-06-14 20:37:28 +0000 UTC - event for azuredisk-volume-tester-tpdqt: {kubelet k8s-agentpool1-20224640-0} FailedMount: Unable to attach or mount volumes: unmounted volumes=[test-volume-2 test-volume-3 test-volume-1], unattached volumes=[test-volume-2 test-volume-3 kube-api-access-xgl99 test-volume-1]: timed out waiting for the condition Jun 14 21:20:28.958: INFO: At 2022-06-14 20:37:30 +0000 UTC - event for azuredisk-volume-tester-tpdqt: {kubelet k8s-agentpool1-20224640-0} FailedMount: MountVolume.MountDevice failed for volume "pvc-46307ead-9ebe-420c-8343-39942020e2d3" : rpc error: code = DeadlineExceeded desc = context deadline exceeded Jun 14 21:20:28.958: INFO: At 2022-06-14 20:37:30 +0000 UTC - event for azuredisk-volume-tester-tpdqt: {kubelet k8s-agentpool1-20224640-0} FailedMount: MountVolume.MountDevice failed for volume "pvc-eaf7b142-4cf7-487b-8d39-47b1a7796a37" : rpc error: code = DeadlineExceeded desc = context deadline exceeded Jun 14 21:20:28.958: INFO: At 2022-06-14 20:37:30 +0000 UTC - event for azuredisk-volume-tester-tpdqt: {kubelet k8s-agentpool1-20224640-0} FailedMount: MountVolume.MountDevice failed for volume "pvc-a6919324-dfd5-458c-b905-552bf919db25" : rpc error: code = DeadlineExceeded desc = context deadline exceeded Jun 14 21:20:28.958: INFO: At 2022-06-14 20:39:45 +0000 UTC - event for azuredisk-volume-tester-tpdqt: {kubelet k8s-agentpool1-20224640-0} FailedMount: Unable to attach or mount volumes: unmounted volumes=[test-volume-1 test-volume-2 test-volume-3], unattached volumes=[test-volume-1 test-volume-2 test-volume-3 kube-api-access-xgl99]: timed out waiting for the condition Jun 14 21:20:28.958: INFO: At 2022-06-14 20:44:19 +0000 UTC - event for azuredisk-volume-tester-tpdqt: {kubelet k8s-agentpool1-20224640-0} FailedMount: Unable to attach or mount volumes: unmounted volumes=[test-volume-3 test-volume-1 test-volume-2], unattached volumes=[test-volume-3 kube-api-access-xgl99 test-volume-1 test-volume-2]: timed out waiting for the condition Jun 14 21:20:28.958: INFO: At 2022-06-14 20:47:46 +0000 UTC - event for azuredisk-volume-tester-tpdqt: {kubelet k8s-agentpool1-20224640-0} FailedMount: MountVolume.MountDevice failed for volume "pvc-a6919324-dfd5-458c-b905-552bf919db25" : rpc error: code = Internal desc = failed to wait for volume (/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-w0zpdk2w/providers/Microsoft.Compute/disks/pvc-a6919324-dfd5-458c-b905-552bf919db25) to be attached to node (k8s-agentpool1-20224640-0): context deadline exceeded Jun 14 21:20:28.958: INFO: At 2022-06-14 20:48:48 +0000 UTC - event for azuredisk-volume-tester-tpdqt: {kubelet k8s-agentpool1-20224640-0} FailedMount: Unable to attach or mount volumes: unmounted volumes=[test-volume-1 test-volume-2 test-volume-3], unattached volumes=[kube-api-access-xgl99 test-volume-1 test-volume-2 test-volume-3]: timed out waiting for the condition Jun 14 21:20:29.012: INFO: POD NODE PHASE GRACE CONDITIONS Jun 14 21:20:29.013: INFO: Jun 14 21:20:29.120: INFO: Logging node info for node k8s-agentpool1-20224640-0 Jun 14 21:20:29.175: INFO: Node Info: &Node{ObjectMeta:{k8s-agentpool1-20224640-0 d40122eb-7d63-4c24-87d0-a4294fda80c2 38861 0 2022-06-14 18:19:50 +0000 UTC <nil> <nil> map[agentpool:agentpool1 beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_DS2_v2 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:0 kubernetes.azure.com/cluster:kubetest-w0zpdk2w kubernetes.azure.com/role:agent kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-agentpool1-20224640-0 kubernetes.io/os:linux kubernetes.io/role:agent node-role.kubernetes.io/agent: node.kubernetes.io/instance-type:Standard_DS2_v2 storageprofile:managed storagetier:Premium_LRS topology.disk.csi.azure.com/zone: topology.kubernetes.io/region:westus2 topology.kubernetes.io/zone:0] map[csi.volume.kubernetes.io/nodeid:{"disk.csi.azure.com":"k8s-agentpool1-20224640-0"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2022-06-14 18:19:50 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:agentpool":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.azure.com/cluster":{},"f:kubernetes.azure.com/role":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:storageprofile":{},"f:storagetier":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kubectl-label Update v1 2022-06-14 18:19:58 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/agent":{}}}} } {kube-controller-manager Update v1 2022-06-14 18:20:04 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"10.244.0.0/24\"":{}}}} } {kube-controller-manager Update v1 2022-06-14 19:36:27 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:volumesAttached":{}}} status} {kubelet Update v1 2022-06-14 20:52:42 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.disk.csi.azure.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:10.244.0.0/24,DoNotUseExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-w0zpdk2w/providers/Microsoft.Compute/virtualMachines/k8s-agentpool1-20224640-0,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[10.244.0.0/24],},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-azure-disk: {{8 0} {<nil>} 8 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{31036686336 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7287771136 0} {<nil>} 7116964Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-azure-disk: {{8 0} {<nil>} 8 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{27933017657 0} {<nil>} 27933017657 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{6501339136 0} {<nil>} 6348964Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-06-14 18:20:25 +0000 UTC,LastTransitionTime:2022-06-14 18:20:25 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-06-14 21:18:15 +0000 UTC,LastTransitionTime:2022-06-14 18:19:49 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-06-14 21:18:15 +0000 UTC,LastTransitionTime:2022-06-14 18:19:49 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-06-14 21:18:15 +0000 UTC,LastTransitionTime:2022-06-14 18:19:49 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-06-14 21:18:15 +0000 UTC,LastTransitionTime:2022-06-14 18:20:04 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:Hostname,Address:k8s-agentpool1-20224640-0,},NodeAddress{Type:InternalIP,Address:10.240.0.4,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:fcfd567ea6a54e9fb7ca2e3949562363,SystemUUID:c20271e3-7d7e-f940-9c6d-928cc89ffc2d,BootID:2a1d1325-bc1b-47c9-9540-2fd7da933292,KernelVersion:5.4.0-1074-azure,OSImage:Ubuntu 18.04.6 LTS,ContainerRuntimeVersion:docker://20.10.11+azure-3,KubeletVersion:v1.23.7,KubeProxyVersion:v1.23.7,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/azuredisk-csi@sha256:423eb6cf602c064c8b2deefead5ceadd6324ed41b3d995dab5d0f6f0f4d4710f mcr.microsoft.com/oss/kubernetes-csi/azuredisk-csi:v1.10.0],SizeBytes:245959792,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/azurefile-csi@sha256:9e2ecabcf9dd9943e6600eb9fb460f45b4dc61af7cabe95d115082a029db2aaf mcr.microsoft.com/oss/kubernetes-csi/azurefile-csi:v1.9.0],SizeBytes:230470852,},ContainerImage{Names:[mcr.microsoft.com/containernetworking/azure-npm@sha256:106f669f48e5e80c4ec0afb49858ead72cf4b901cd8664e7bf81f8d789e56e12 mcr.microsoft.com/containernetworking/azure-npm:v1.2.2_hotfix],SizeBytes:175230380,},ContainerImage{Names:[k8sprow.azurecr.io/azuredisk-csi@sha256:528fc89c187c74420dabdef86e06f05893e1510fce69111a2a05280dd64be2a9 k8sprow.azurecr.io/azuredisk-csi:latest-v2-df2db40004aa147caa3762ffb782a447d13cb39d],SizeBytes:136461498,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/secrets-store/driver@sha256:c0d040a1c4fbfceb65663e31c09ea40f4f78e356437610cbc3fbb4bb409bd6f1 mcr.microsoft.com/oss/kubernetes-csi/secrets-store/driver:v0.0.19],SizeBytes:123229697,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-proxy@sha256:0c890e918d42c158fab68867bade2ab1db04c3408452db9611a264ba3e43e6b3 mcr.microsoft.com/oss/kubernetes/kube-proxy:v1.23.7],SizeBytes:112296063,},ContainerImage{Names:[mcr.microsoft.com/oss/azure/secrets-store/provider-azure@sha256:6f67f3d0c7cdde5702f8ce7f101b6519daa0237f0c34fecb7c058b6af8c22ad1 mcr.microsoft.com/oss/azure/secrets-store/provider-azure:0.0.12],SizeBytes:101061355,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/autoscaler/cluster-autoscaler@sha256:6f0c680d375c62e74351f8ff3ed6ddb9b72ca759e0645c329b95f64264654a6d mcr.microsoft.com/oss/kubernetes/autoscaler/cluster-autoscaler:v1.22.1],SizeBytes:99962810,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-addon-manager@sha256:32e2836018c96e73533bd4642fe438e465b81dcbfa8b7b61935a6f4d0246c7ae mcr.microsoft.com/oss/kubernetes/kube-addon-manager:v9.1.3],SizeBytes:86832059,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-addon-manager@sha256:92c2c5aad9012ee32d2a43a74966cc0adc6ccb1705ad15abb10485ecf406d88b mcr.microsoft.com/oss/kubernetes/kube-addon-manager:v9.1.5],SizeBytes:84094027,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/metrics-server@sha256:1ef9d57ce41ffcc328b92494c3bfafe401e0b9a1694a295301a1385337d52815 mcr.microsoft.com/oss/kubernetes/metrics-server:v0.5.2],SizeBytes:64327621,},ContainerImage{Names:[mcr.microsoft.com/oss/nvidia/k8s-device-plugin@sha256:0f5b52bf28239234e831697d96db63ac03cde70fe68058f964504ab7564ee810 mcr.microsoft.com/oss/nvidia/k8s-device-plugin:1.0.0-beta6],SizeBytes:64160241,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-provisioner@sha256:e9ddadc44ba87a4a27f67e54760a14f9986885b534b3dff170a14eae1e35d213 mcr.microsoft.com/oss/kubernetes-csi/csi-provisioner:v3.0.0],SizeBytes:56881280,},ContainerImage{Names:[k8sprow.azurecr.io/azdiskschedulerextender-csi@sha256:3df91ffec98f04827c70772d08849e6874d1e0ee2432173ef523c099553547dc k8sprow.azurecr.io/azdiskschedulerextender-csi:latest-v2-df2db40004aa147caa3762ffb782a447d13cb39d],SizeBytes:56760067,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-resizer@sha256:c5bb71ceaac60b1a4b58739fa07b709f6248c452ff6272a384d2f7648895a750 mcr.microsoft.com/oss/kubernetes-csi/csi-resizer:v1.3.0],SizeBytes:54313772,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-snapshotter@sha256:61849a026511cf332c87d73d0a7aed803b510c3ede197ec755389686d490de72 mcr.microsoft.com/oss/kubernetes-csi/csi-snapshotter:v4.2.1],SizeBytes:54210936,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-attacher@sha256:6b41e7153ebdfdc1501aa65184624bc15fd33a52d93f88ec3a758d0f8c9b8c10 mcr.microsoft.com/oss/kubernetes-csi/csi-attacher:v3.3.0],SizeBytes:53842561,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/snapshot-controller@sha256:8c3fc3c2667004ad6bbdf723bb64c5da66a5cb8b11d4ee59b67179b686223b13 mcr.microsoft.com/oss/kubernetes-csi/snapshot-controller:v5.0.1],SizeBytes:52732401,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/snapshot-controller@sha256:be5a8dc1f990828f653c77bd0a0f1bbd13197c3019f6e1d99d590389bac36705 mcr.microsoft.com/oss/kubernetes-csi/snapshot-controller:v4.2.1],SizeBytes:51575245,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager@sha256:0ad67f9919522a07318034641ae09bf2079b417e9944c65914410594ce645468 mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager:v1.1.4],SizeBytes:51478397,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager@sha256:31ec4f7daccd3e7a8504e12657d7830651ecacbe4a487daca1b1b7695a64b070 mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager:v1.23.1],SizeBytes:51249021,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager@sha256:011712ed90fb8efcf27928b0a47ed04b98baebb31cb1b2d8ab676977ec18eedc mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager:v1.1.4],SizeBytes:50868093,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager@sha256:0f9a8fbaed65192ed7dd795be4f9c1dc48ebdef0a241fb62d456f4bed40d9875 mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager:v1.23.1],SizeBytes:50679677,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-scheduler@sha256:c35f45e8dfaa30363e589296415a057f9d32813920fc3820d6557491657f199a mcr.microsoft.com/oss/kubernetes/kube-scheduler:v1.21.2],SizeBytes:50607491,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/ip-masq-agent@sha256:1244155f2ed3f33ff154cc343b8ad285f3391d95afd7d4b1c6dcc420bc0ba3cf mcr.microsoft.com/oss/kubernetes/ip-masq-agent:v2.5.0],SizeBytes:50146762,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager@sha256:7c907ff70b90a0bdf8fae63bd744018469dd9839cde1fd0515b93e0bbd14b34e mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager:v1.0.8],SizeBytes:48963453,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager@sha256:3987d7a8c6922ce1952ee19c5cb6ea75aac7b7c1b07aa79277ad038c69fb7a31 mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager:v1.0.8],SizeBytes:48349053,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/coredns@sha256:f873bf7f0928461efe10697fa76cf0ad7a1ae3041c5b57b50dd3d0b72d273f8c mcr.microsoft.com/oss/kubernetes/coredns:1.8.6],SizeBytes:46804601,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager@sha256:8073113a20882642a980b338635cdc5945e5673a18aef192090e6fde2b89a75c mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager:v0.6.0],SizeBytes:45909032,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager@sha256:6a32329628bdea3c6d75e98aad6155b65d2e2b98ca616eb33f9ac562912804c6 mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager:v0.6.0],SizeBytes:45229096,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager@sha256:ef6c4ba564b4d11d270f7d1563c50fbeb30ccc3b94146e5059228c49f95875f5 mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager:v0.7.11],SizeBytes:44916605,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager@sha256:dbcc384758ba5ca6d249596d471292ed3785e31cdb854d48b84d70794b669b4c mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager:v0.7.11],SizeBytes:43679613,},ContainerImage{Names:[mcr.microsoft.com/oss/etcd-io/etcd@sha256:cf587862e3f1b6fa4d9a2565520a34f164bdf72c50f37af8c3c668160593246e mcr.microsoft.com/oss/etcd-io/etcd:v3.3.25],SizeBytes:41832119,},ContainerImage{Names:[mcr.microsoft.com/k8s/aad-pod-identity/mic@sha256:bd9465be94966b9a917e1e3904fa5e63dd91772ccadf304e18ffd8e4ad8ccedd mcr.microsoft.com/k8s/aad-pod-identity/mic:1.6.1],SizeBytes:41374894,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/autoscaler/cluster-proportional-autoscaler@sha256:c849d75d61943ce7f51b4c049f1a79d19a08253966c8f49c4cfb6414cc33db8b mcr.microsoft.com/oss/kubernetes/autoscaler/cluster-proportional-autoscaler:1.8.5],SizeBytes:40661903,},ContainerImage{Names:[mcr.microsoft.com/k8s/aad-pod-identity/nmi@sha256:02128fefcdb7593ac53fc342e2c53a0fc6fabd813036bf60457bf43cc2940116 mcr.microsoft.com/k8s/aad-pod-identity/nmi:1.6.1],SizeBytes:38007982,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:921f301c44dda06a325164accf22e78ecc570b5c7d9d6ee4c66bd8cbb2b60b9a mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.16],SizeBytes:26970670,},ContainerImage{Names:[mcr.microsoft.com/k8s/kms/keyvault@sha256:1a27e175f8c125209e32d2957b5509fe20757bd8cb309ff9da598799b56326fb mcr.microsoft.com/k8s/kms/keyvault:v0.0.10],SizeBytes:23077387,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-node-driver-registrar@sha256:dbec3a8166686b09b242176ab5b99e993da4126438bbce68147c3fd654f35662 mcr.microsoft.com/oss/kubernetes-csi/csi-node-driver-registrar:v2.4.0],SizeBytes:19547289,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/livenessprobe@sha256:c96a6255c42766f6b8bb1a7cda02b0060ab1b20b2e2dafcc64ec09e7646745a6 mcr.microsoft.com/oss/kubernetes-csi/livenessprobe:v2.5.0],SizeBytes:17573341,},ContainerImage{Names:[mcr.microsoft.com/oss/busybox/busybox@sha256:582a641242b49809af3a1a522f9aae8c3f047d1c6ca1dd9d8cdabd349e45b1a9 mcr.microsoft.com/oss/busybox/busybox:1.33.1],SizeBytes:1235829,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8],SizeBytes:1159025,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/pause@sha256:e3b8c20681593c21b344ad801fbb8abaf564427ee3a57a9fcfa3b455f917ce46 mcr.microsoft.com/oss/kubernetes/pause:3.4.1],SizeBytes:682696,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/disk.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-w0zpdk2w/providers/Microsoft.Compute/disks/pvc-eaf7b142-4cf7-487b-8d39-47b1a7796a37,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/disk.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-w0zpdk2w/providers/Microsoft.Compute/disks/pvc-a6919324-dfd5-458c-b905-552bf919db25,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/disk.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-w0zpdk2w/providers/Microsoft.Compute/disks/pvc-46307ead-9ebe-420c-8343-39942020e2d3,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/disk.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-w0zpdk2w/providers/Microsoft.Compute/disks/pvc-e6291a08-817d-41da-ad5a-8f6c567374d6,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/disk.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-w0zpdk2w/providers/Microsoft.Compute/disks/pvc-b914cba7-48ba-457d-809b-8ae01ce1e4cb,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/disk.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-w0zpdk2w/providers/Microsoft.Compute/disks/pvc-28956818-78c7-4087-b9b3-a6f87aabc914,DevicePath:,},},Config:nil,},} ... skipping 91 lines ... [90m/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning_test.go:40[0m [single-az] [90m/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning_test.go:44[0m [91m[1mshould create multiple PV objects, bind to PVCs and attach all to a single pod [kubernetes.io/azure-disk] [disk.csi.azure.com] [Windows] [csi-azuredisk-scheduler-extender] [It][0m [90m/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning_test.go:589[0m [91mUnexpected error: <*errors.errorString | 0xc00061c3e0>: { s: "Gave up after waiting 15m0s for pod \"azuredisk-volume-tester-tpdqt\" to be \"Succeeded or Failed\"", } Gave up after waiting 15m0s for pod "azuredisk-volume-tester-tpdqt" to be "Succeeded or Failed" occurred[0m /home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/resources/pod.go:89 [90m------------------------------[0m [0mDynamic Provisioning[0m [90m[single-az][0m [1mshould create a raw block volume and a filesystem volume on demand and bind to the same pod [kubernetes.io/azure-disk] [disk.csi.azure.com] [csi-azuredisk-scheduler-extender][0m ... skipping 16 lines ... [1mSTEP[0m: setting up the StorageClass [1mSTEP[0m: creating a StorageClass [1mSTEP[0m: setting up the PVC and PV [1mSTEP[0m: creating a PVC [1mSTEP[0m: setting up the pod [1mSTEP[0m: deploying the pod [1mSTEP[0m: checking that the pod's command exits with no error Jun 14 21:20:31.601: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-kh7mc" in namespace "azuredisk-2447" to be "Succeeded or Failed" Jun 14 21:20:31.656: INFO: Pod "azuredisk-volume-tester-kh7mc": Phase="Pending", Reason="", readiness=false. Elapsed: 54.480404ms Jun 14 21:20:33.711: INFO: Pod "azuredisk-volume-tester-kh7mc": Phase="Pending", Reason="", readiness=false. Elapsed: 2.109748634s Jun 14 21:20:35.767: INFO: Pod "azuredisk-volume-tester-kh7mc": Phase="Pending", Reason="", readiness=false. Elapsed: 4.165501865s Jun 14 21:20:37.822: INFO: Pod "azuredisk-volume-tester-kh7mc": Phase="Pending", Reason="", readiness=false. Elapsed: 6.220890112s Jun 14 21:20:39.877: INFO: Pod "azuredisk-volume-tester-kh7mc": Phase="Pending", Reason="", readiness=false. Elapsed: 8.276116824s Jun 14 21:20:41.934: INFO: Pod "azuredisk-volume-tester-kh7mc": Phase="Pending", Reason="", readiness=false. Elapsed: 10.332926184s ... skipping 322 lines ... Jun 14 21:31:46.159: INFO: Pod "azuredisk-volume-tester-kh7mc": Phase="Pending", Reason="", readiness=false. Elapsed: 11m14.557733144s Jun 14 21:31:48.215: INFO: Pod "azuredisk-volume-tester-kh7mc": Phase="Pending", Reason="", readiness=false. Elapsed: 11m16.61352825s Jun 14 21:31:50.270: INFO: Pod "azuredisk-volume-tester-kh7mc": Phase="Pending", Reason="", readiness=false. Elapsed: 11m18.668936102s Jun 14 21:31:52.333: INFO: Pod "azuredisk-volume-tester-kh7mc": Phase="Pending", Reason="", readiness=false. Elapsed: 11m20.732133055s Jun 14 21:31:54.392: INFO: Pod "azuredisk-volume-tester-kh7mc": Phase="Pending", Reason="", readiness=false. Elapsed: 11m22.790418561s Jun 14 21:31:56.448: INFO: Pod "azuredisk-volume-tester-kh7mc": Phase="Pending", Reason="", readiness=false. Elapsed: 11m24.846344695s {"component":"entrypoint","file":"k8s.io/test-infra/prow/entrypoint/run.go:255","func":"k8s.io/test-infra/prow/entrypoint.gracefullyTerminate","level":"error","msg":"Process did not exit before 15m0s grace period","severity":"error","time":"2022-06-14T21:31:57Z"} {"component":"entrypoint","error":"os: process already finished","file":"k8s.io/test-infra/prow/entrypoint/run.go:257","func":"k8s.io/test-infra/prow/entrypoint.gracefullyTerminate","level":"error","msg":"Could not kill process after grace period","severity":"error","time":"2022-06-14T21:31:57Z"}