kubernetes-sigs/azuredisk-csi-driver PR #1361 pull-azuredisk-csi-driver-e2e-migration-mainv2 #1537963419754303488
Recent runs || View in Spyglass
Recent runs || View in Spyglass
| PR | sunpa93: [V2] feat: update volume attachment with publish context upon attachment completion |
| Result | FAILURE |
| Tests | 0 failed / 0 succeeded |
| Started | |
| Elapsed | 3h15m |
| Revision | 8cd628015d6d61769e30d7e7d0957243476a5bcb |
| Refs |
1361 |
... skipping 94 lines ... 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 11156 100 11156 0 0 213k 0 --:--:-- --:--:-- --:--:-- 213k Downloading https://get.helm.sh/helm-v3.9.0-linux-amd64.tar.gz Verifying checksum... Done. Preparing to install helm into /usr/local/bin helm installed into /usr/local/bin/helm docker pull k8sprow.azurecr.io/azuredisk-csi:latest-v2-444ca6fc5bce97ea7c52380569d0659c5e105ce7 || make container-all push-manifest Error response from daemon: manifest for k8sprow.azurecr.io/azuredisk-csi:latest-v2-444ca6fc5bce97ea7c52380569d0659c5e105ce7 not found: manifest unknown: manifest tagged by "latest-v2-444ca6fc5bce97ea7c52380569d0659c5e105ce7" is not found make[2]: Entering directory '/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver' CGO_ENABLED=0 GOOS=windows go build -a -ldflags "-X sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.driverVersion=latest-v2-444ca6fc5bce97ea7c52380569d0659c5e105ce7 -X sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.gitCommit=444ca6fc5bce97ea7c52380569d0659c5e105ce7 -X sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.buildDate=2022-06-18T01:08:51Z -extldflags "-static"" -tags azurediskv2 -mod vendor -o _output/amd64/azurediskpluginv2.exe ./pkg/azurediskplugin docker buildx rm container-builder || true error: no builder "container-builder" found docker buildx create --use --name=container-builder container-builder # enable qemu for arm64 build # https://github.com/docker/buildx/issues/464#issuecomment-741507760 docker run --privileged --rm tonistiigi/binfmt --uninstall qemu-aarch64 Unable to find image 'tonistiigi/binfmt:latest' locally ... skipping 724 lines ... } } ] } make[2]: Leaving directory '/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver' docker pull k8sprow.azurecr.io/azdiskschedulerextender-csi:latest-v2-444ca6fc5bce97ea7c52380569d0659c5e105ce7 || make azdiskschedulerextender-all push-manifest-azdiskschedulerextender Error response from daemon: manifest for k8sprow.azurecr.io/azdiskschedulerextender-csi:latest-v2-444ca6fc5bce97ea7c52380569d0659c5e105ce7 not found: manifest unknown: manifest tagged by "latest-v2-444ca6fc5bce97ea7c52380569d0659c5e105ce7" is not found make[2]: Entering directory '/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver' docker buildx rm container-builder || true docker buildx create --use --name=container-builder container-builder # enable qemu for arm64 build # https://github.com/docker/buildx/issues/464#issuecomment-741507760 ... skipping 860 lines ... type: string type: object oneOf: - required: ["persistentVolumeClaimName"] - required: ["volumeSnapshotContentName"] volumeSnapshotClassName: description: 'VolumeSnapshotClassName is the name of the VolumeSnapshotClass requested by the VolumeSnapshot. VolumeSnapshotClassName may be left nil to indicate that the default SnapshotClass should be used. A given cluster may have multiple default Volume SnapshotClasses: one default per CSI Driver. If a VolumeSnapshot does not specify a SnapshotClass, VolumeSnapshotSource will be checked to figure out what the associated CSI Driver is, and the default VolumeSnapshotClass associated with that CSI Driver will be used. If more than one VolumeSnapshotClass exist for a given CSI Driver and more than one have been marked as default, CreateSnapshot will fail and generate an event. Empty string is not allowed for this field.' type: string required: - source type: object status: description: status represents the current information of a snapshot. Consumers must verify binding between VolumeSnapshot and VolumeSnapshotContent objects is successful (by validating that both VolumeSnapshot and VolumeSnapshotContent point at each other) before using this object. ... skipping 2 lines ... description: 'boundVolumeSnapshotContentName is the name of the VolumeSnapshotContent object to which this VolumeSnapshot object intends to bind to. If not specified, it indicates that the VolumeSnapshot object has not been successfully bound to a VolumeSnapshotContent object yet. NOTE: To avoid possible security issues, consumers must verify binding between VolumeSnapshot and VolumeSnapshotContent objects is successful (by validating that both VolumeSnapshot and VolumeSnapshotContent point at each other) before using this object.' type: string creationTime: description: creationTime is the timestamp when the point-in-time snapshot is taken by the underlying storage system. In dynamic snapshot creation case, this field will be filled in by the snapshot controller with the "creation_time" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "creation_time" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. If not specified, it may indicate that the creation time of the snapshot is unknown. format: date-time type: string error: description: error is the last observed error during snapshot creation, if any. This field could be helpful to upper level controllers(i.e., application controller) to decide whether they should continue on waiting for the snapshot to be created based on the type of error reported. The snapshot controller will keep retrying when an error occurrs during the snapshot creation. Upon success, this error field will be cleared. properties: message: description: 'message is a string detailing the encountered error during snapshot creation if specified. NOTE: message may be logged, and it should not contain sensitive information.' type: string time: description: time is the timestamp when the error was encountered. format: date-time type: string type: object readyToUse: description: readyToUse indicates if the snapshot is ready to be used to restore a volume. In dynamic snapshot creation case, this field will be filled in by the snapshot controller with the "ready_to_use" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "ready_to_use" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it, otherwise, this field will be set to "True". If not specified, it means the readiness of a snapshot is unknown. type: boolean restoreSize: type: string description: restoreSize represents the minimum size of volume required to create a volume from this snapshot. In dynamic snapshot creation case, this field will be filled in by the snapshot controller with the "size_bytes" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "size_bytes" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. When restoring a volume from this snapshot, the size of the volume MUST NOT be smaller than the restoreSize if it is specified, otherwise the restoration will fail. If not specified, it indicates that the size is unknown. pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ x-kubernetes-int-or-string: true type: object required: - spec type: object ... skipping 60 lines ... type: string volumeSnapshotContentName: description: volumeSnapshotContentName specifies the name of a pre-existing VolumeSnapshotContent object representing an existing volume snapshot. This field should be set if the snapshot already exists and only needs a representation in Kubernetes. This field is immutable. type: string type: object volumeSnapshotClassName: description: 'VolumeSnapshotClassName is the name of the VolumeSnapshotClass requested by the VolumeSnapshot. VolumeSnapshotClassName may be left nil to indicate that the default SnapshotClass should be used. A given cluster may have multiple default Volume SnapshotClasses: one default per CSI Driver. If a VolumeSnapshot does not specify a SnapshotClass, VolumeSnapshotSource will be checked to figure out what the associated CSI Driver is, and the default VolumeSnapshotClass associated with that CSI Driver will be used. If more than one VolumeSnapshotClass exist for a given CSI Driver and more than one have been marked as default, CreateSnapshot will fail and generate an event. Empty string is not allowed for this field.' type: string required: - source type: object status: description: status represents the current information of a snapshot. Consumers must verify binding between VolumeSnapshot and VolumeSnapshotContent objects is successful (by validating that both VolumeSnapshot and VolumeSnapshotContent point at each other) before using this object. ... skipping 2 lines ... description: 'boundVolumeSnapshotContentName is the name of the VolumeSnapshotContent object to which this VolumeSnapshot object intends to bind to. If not specified, it indicates that the VolumeSnapshot object has not been successfully bound to a VolumeSnapshotContent object yet. NOTE: To avoid possible security issues, consumers must verify binding between VolumeSnapshot and VolumeSnapshotContent objects is successful (by validating that both VolumeSnapshot and VolumeSnapshotContent point at each other) before using this object.' type: string creationTime: description: creationTime is the timestamp when the point-in-time snapshot is taken by the underlying storage system. In dynamic snapshot creation case, this field will be filled in by the snapshot controller with the "creation_time" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "creation_time" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. If not specified, it may indicate that the creation time of the snapshot is unknown. format: date-time type: string error: description: error is the last observed error during snapshot creation, if any. This field could be helpful to upper level controllers(i.e., application controller) to decide whether they should continue on waiting for the snapshot to be created based on the type of error reported. The snapshot controller will keep retrying when an error occurrs during the snapshot creation. Upon success, this error field will be cleared. properties: message: description: 'message is a string detailing the encountered error during snapshot creation if specified. NOTE: message may be logged, and it should not contain sensitive information.' type: string time: description: time is the timestamp when the error was encountered. format: date-time type: string type: object readyToUse: description: readyToUse indicates if the snapshot is ready to be used to restore a volume. In dynamic snapshot creation case, this field will be filled in by the snapshot controller with the "ready_to_use" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "ready_to_use" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it, otherwise, this field will be set to "True". If not specified, it means the readiness of a snapshot is unknown. type: boolean restoreSize: type: string description: restoreSize represents the minimum size of volume required to create a volume from this snapshot. In dynamic snapshot creation case, this field will be filled in by the snapshot controller with the "size_bytes" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "size_bytes" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. When restoring a volume from this snapshot, the size of the volume MUST NOT be smaller than the restoreSize if it is specified, otherwise the restoration will fail. If not specified, it indicates that the size is unknown. pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ x-kubernetes-int-or-string: true type: object required: - spec type: object ... skipping 254 lines ... description: status represents the current information of a snapshot. properties: creationTime: description: creationTime is the timestamp when the point-in-time snapshot is taken by the underlying storage system. In dynamic snapshot creation case, this field will be filled in by the CSI snapshotter sidecar with the "creation_time" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "creation_time" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. If not specified, it indicates the creation time is unknown. The format of this field is a Unix nanoseconds time encoded as an int64. On Unix, the command `date +%s%N` returns the current time in nanoseconds since 1970-01-01 00:00:00 UTC. format: int64 type: integer error: description: error is the last observed error during snapshot creation, if any. Upon success after retry, this error field will be cleared. properties: message: description: 'message is a string detailing the encountered error during snapshot creation if specified. NOTE: message may be logged, and it should not contain sensitive information.' type: string time: description: time is the timestamp when the error was encountered. format: date-time type: string type: object readyToUse: description: readyToUse indicates if a snapshot is ready to be used to restore a volume. In dynamic snapshot creation case, this field will be filled in by the CSI snapshotter sidecar with the "ready_to_use" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "ready_to_use" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it, otherwise, this field will be set to "True". If not specified, it means the readiness of a snapshot is unknown. type: boolean restoreSize: description: restoreSize represents the complete size of the snapshot in bytes. In dynamic snapshot creation case, this field will be filled in by the CSI snapshotter sidecar with the "size_bytes" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "size_bytes" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. When restoring a volume from this snapshot, the size of the volume MUST NOT be smaller than the restoreSize if it is specified, otherwise the restoration will fail. If not specified, it indicates that the size is unknown. format: int64 minimum: 0 type: integer snapshotHandle: description: snapshotHandle is the CSI "snapshot_id" of a snapshot on the underlying storage system. If not specified, it indicates that dynamic snapshot creation has either failed or it is still in progress. type: string type: object required: - spec type: object served: true ... skipping 108 lines ... description: status represents the current information of a snapshot. properties: creationTime: description: creationTime is the timestamp when the point-in-time snapshot is taken by the underlying storage system. In dynamic snapshot creation case, this field will be filled in by the CSI snapshotter sidecar with the "creation_time" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "creation_time" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. If not specified, it indicates the creation time is unknown. The format of this field is a Unix nanoseconds time encoded as an int64. On Unix, the command `date +%s%N` returns the current time in nanoseconds since 1970-01-01 00:00:00 UTC. format: int64 type: integer error: description: error is the last observed error during snapshot creation, if any. Upon success after retry, this error field will be cleared. properties: message: description: 'message is a string detailing the encountered error during snapshot creation if specified. NOTE: message may be logged, and it should not contain sensitive information.' type: string time: description: time is the timestamp when the error was encountered. format: date-time type: string type: object readyToUse: description: readyToUse indicates if a snapshot is ready to be used to restore a volume. In dynamic snapshot creation case, this field will be filled in by the CSI snapshotter sidecar with the "ready_to_use" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "ready_to_use" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it, otherwise, this field will be set to "True". If not specified, it means the readiness of a snapshot is unknown. type: boolean restoreSize: description: restoreSize represents the complete size of the snapshot in bytes. In dynamic snapshot creation case, this field will be filled in by the CSI snapshotter sidecar with the "size_bytes" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "size_bytes" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. When restoring a volume from this snapshot, the size of the volume MUST NOT be smaller than the restoreSize if it is specified, otherwise the restoration will fail. If not specified, it indicates that the size is unknown. format: int64 minimum: 0 type: integer snapshotHandle: description: snapshotHandle is the CSI "snapshot_id" of a snapshot on the underlying storage system. If not specified, it indicates that dynamic snapshot creation has either failed or it is still in progress. type: string type: object required: - spec type: object served: true ... skipping 244 lines ... - volumeName - volume_context - volume_id type: object status: description: status represents the current state of AzVolumeAttachment. includes error, state, and attachment status properties: annotation: additionalProperties: type: string description: Annotations contains additional resource information to guide driver actions ... skipping 13 lines ... role: description: The current attachment role. type: string required: - role type: object error: description: Error occurred during attach/detach of volume properties: code: type: string message: type: string parameters: ... skipping 173 lines ... - maxMountReplicaCount - volumeCapability - volumeName type: object status: description: status represents the current state of AzVolume. includes error, state, and volume status properties: annotation: additionalProperties: type: string description: Annotations contains additional resource information to guide driver actions ... skipping 34 lines ... type: string required: - capacity_bytes - node_expansion_required - volume_id type: object error: description: Error occurred during creation/deletion of volume properties: code: type: string message: type: string parameters: ... skipping 1061 lines ... image: "mcr.microsoft.com/oss/kubernetes-csi/csi-resizer:v1.4.0" args: - "-csi-address=$(ADDRESS)" - "-v=2" - "-leader-election" - "--leader-election-namespace=kube-system" - '-handle-volume-inuse-error=false' - '-feature-gates=RecoverVolumeExpansionFailure=true' - "-timeout=240s" env: - name: ADDRESS value: /csi/csi.sock volumeMounts: ... skipping 421 lines ... [1mSTEP[0m: Waiting for a default service account to be provisioned in namespace [1mSTEP[0m: Waiting for kube-root-ca.crt to be provisioned in namespace I0618 01:17:18.893771 15039 azuredisk_driver.go:52] Using azure disk driver: kubernetes.io/azure-disk [1mSTEP[0m: Successfully provisioned AzureDisk volume: "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-u6qgg0s3/providers/Microsoft.Compute/disks/pre-provisioned-inline-volume" [1mSTEP[0m: deploying the pod [1mSTEP[0m: checking that the pod's command exits with no error Jun 18 01:17:23.038: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-6gztt" in namespace "azuredisk-8819" to be "Succeeded or Failed" Jun 18 01:17:23.105: INFO: Pod "azuredisk-volume-tester-6gztt": Phase="Pending", Reason="", readiness=false. Elapsed: 66.806504ms Jun 18 01:17:25.141: INFO: Pod "azuredisk-volume-tester-6gztt": Phase="Pending", Reason="", readiness=false. Elapsed: 2.102520606s Jun 18 01:17:27.176: INFO: Pod "azuredisk-volume-tester-6gztt": Phase="Pending", Reason="", readiness=false. Elapsed: 4.137533835s Jun 18 01:17:29.211: INFO: Pod "azuredisk-volume-tester-6gztt": Phase="Pending", Reason="", readiness=false. Elapsed: 6.172541373s Jun 18 01:17:31.245: INFO: Pod "azuredisk-volume-tester-6gztt": Phase="Pending", Reason="", readiness=false. Elapsed: 8.207247842s Jun 18 01:17:33.280: INFO: Pod "azuredisk-volume-tester-6gztt": Phase="Pending", Reason="", readiness=false. Elapsed: 10.241778931s ... skipping 432 lines ... Jun 18 01:32:14.544: INFO: Pod "azuredisk-volume-tester-6gztt": Phase="Pending", Reason="", readiness=false. Elapsed: 14m51.505570593s Jun 18 01:32:16.578: INFO: Pod "azuredisk-volume-tester-6gztt": Phase="Pending", Reason="", readiness=false. Elapsed: 14m53.539786086s Jun 18 01:32:18.615: INFO: Pod "azuredisk-volume-tester-6gztt": Phase="Pending", Reason="", readiness=false. Elapsed: 14m55.576670126s Jun 18 01:32:20.654: INFO: Pod "azuredisk-volume-tester-6gztt": Phase="Pending", Reason="", readiness=false. Elapsed: 14m57.616290797s Jun 18 01:32:22.690: INFO: Pod "azuredisk-volume-tester-6gztt": Phase="Pending", Reason="", readiness=false. Elapsed: 14m59.651741615s Jun 18 01:32:24.690: INFO: deleting Pod "azuredisk-8819"/"azuredisk-volume-tester-6gztt" Jun 18 01:32:24.752: INFO: Error getting logs for pod azuredisk-volume-tester-6gztt: the server rejected our request for an unknown reason (get pods azuredisk-volume-tester-6gztt) [1mSTEP[0m: Deleting pod azuredisk-volume-tester-6gztt in namespace azuredisk-8819 [1mSTEP[0m: Collecting events from namespace "azuredisk-8819". [1mSTEP[0m: Found 4 events. Jun 18 01:32:24.820: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for azuredisk-volume-tester-6gztt: { } Scheduled: Successfully assigned azuredisk-8819/azuredisk-volume-tester-6gztt to k8s-agentpool1-42835052-0 Jun 18 01:32:24.820: INFO: At 2022-06-18 01:17:24 +0000 UTC - event for azuredisk-volume-tester-6gztt: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-u6qgg0s3/providers/Microsoft.Compute/disks/pre-provisioned-inline-volume" Jun 18 01:32:24.820: INFO: At 2022-06-18 01:19:26 +0000 UTC - event for azuredisk-volume-tester-6gztt: {kubelet k8s-agentpool1-42835052-0} FailedMount: Unable to attach or mount volumes: unmounted volumes=[test-volume-1], unattached volumes=[test-volume-1 kube-api-access-t7qk7]: timed out waiting for the condition Jun 18 01:32:24.820: INFO: At 2022-06-18 01:19:32 +0000 UTC - event for azuredisk-volume-tester-6gztt: {kubelet k8s-agentpool1-42835052-0} FailedMount: MountVolume.MountDevice failed for volume "/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-u6qgg0s3/providers/Microsoft.Compute/disks/pre-provisioned-inline-volume" : rpc error: code = DeadlineExceeded desc = context deadline exceeded Jun 18 01:32:24.854: INFO: POD NODE PHASE GRACE CONDITIONS Jun 18 01:32:24.854: INFO: azuredisk-volume-tester-6gztt k8s-agentpool1-42835052-0 Pending 30s [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2022-06-18 01:17:23 +0000 UTC } {Ready False 0001-01-01 00:00:00 +0000 UTC 2022-06-18 01:17:23 +0000 UTC ContainersNotReady containers with unready status: [volume-tester]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2022-06-18 01:17:23 +0000 UTC ContainersNotReady containers with unready status: [volume-tester]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2022-06-18 01:17:23 +0000 UTC }] Jun 18 01:32:24.854: INFO: Jun 18 01:32:24.950: INFO: Logging node info for node k8s-agentpool1-42835052-0 Jun 18 01:32:24.985: INFO: Node Info: &Node{ObjectMeta:{k8s-agentpool1-42835052-0 b3450a63-c373-4a96-b548-c74917f39933 4287 0 2022-06-18 01:05:58 +0000 UTC <nil> <nil> map[agentpool:agentpool1 beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_DS2_v2 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:canadacentral failure-domain.beta.kubernetes.io/zone:0 kubernetes.azure.com/cluster:kubetest-u6qgg0s3 kubernetes.azure.com/role:agent kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-agentpool1-42835052-0 kubernetes.io/os:linux kubernetes.io/role:agent node-role.kubernetes.io/agent: node.kubernetes.io/instance-type:Standard_DS2_v2 storageprofile:managed storagetier:Premium_LRS topology.disk.csi.azure.com/zone: topology.kubernetes.io/region:canadacentral topology.kubernetes.io/zone:0] map[csi.volume.kubernetes.io/nodeid:{"disk.csi.azure.com":"k8s-agentpool1-42835052-0"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2022-06-18 01:05:58 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:agentpool":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.azure.com/cluster":{},"f:kubernetes.azure.com/role":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:storageprofile":{},"f:storagetier":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kube-controller-manager Update v1 2022-06-18 01:06:07 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"10.244.2.0/24\"":{}}}} } {kubectl-label Update v1 2022-06-18 01:06:53 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/agent":{}}}} } {kube-controller-manager Update v1 2022-06-18 01:17:24 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:volumesAttached":{}}} status} {kubelet Update v1 2022-06-18 01:17:32 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.disk.csi.azure.com/zone":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:10.244.2.0/24,DoNotUseExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-u6qgg0s3/providers/Microsoft.Compute/virtualMachines/k8s-agentpool1-42835052-0,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[10.244.2.0/24],},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-azure-disk: {{8 0} {<nil>} 8 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{31036686336 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7287771136 0} {<nil>} 7116964Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-azure-disk: {{8 0} {<nil>} 8 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{27933017657 0} {<nil>} 27933017657 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{6501339136 0} {<nil>} 6348964Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-06-18 01:06:26 +0000 UTC,LastTransitionTime:2022-06-18 01:06:26 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-06-18 01:27:46 +0000 UTC,LastTransitionTime:2022-06-18 01:05:58 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-06-18 01:27:46 +0000 UTC,LastTransitionTime:2022-06-18 01:05:58 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-06-18 01:27:46 +0000 UTC,LastTransitionTime:2022-06-18 01:05:58 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-06-18 01:27:46 +0000 UTC,LastTransitionTime:2022-06-18 01:06:06 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:Hostname,Address:k8s-agentpool1-42835052-0,},NodeAddress{Type:InternalIP,Address:10.240.0.5,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:fd2eb91f0eeb436286bc238b58e4dc04,SystemUUID:c3ccb155-829c-6a40-8fdc-f652683a9f79,BootID:0082daec-2a09-4d8b-b449-5299bd7dc135,KernelVersion:5.4.0-1074-azure,OSImage:Ubuntu 18.04.6 LTS,ContainerRuntimeVersion:docker://20.10.11+azure-3,KubeletVersion:v1.23.7,KubeProxyVersion:v1.23.7,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/azuredisk-csi@sha256:423eb6cf602c064c8b2deefead5ceadd6324ed41b3d995dab5d0f6f0f4d4710f mcr.microsoft.com/oss/kubernetes-csi/azuredisk-csi:v1.10.0],SizeBytes:245959792,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/azurefile-csi@sha256:9e2ecabcf9dd9943e6600eb9fb460f45b4dc61af7cabe95d115082a029db2aaf mcr.microsoft.com/oss/kubernetes-csi/azurefile-csi:v1.9.0],SizeBytes:230470852,},ContainerImage{Names:[mcr.microsoft.com/containernetworking/azure-npm@sha256:106f669f48e5e80c4ec0afb49858ead72cf4b901cd8664e7bf81f8d789e56e12 mcr.microsoft.com/containernetworking/azure-npm:v1.2.2_hotfix],SizeBytes:175230380,},ContainerImage{Names:[k8sprow.azurecr.io/azuredisk-csi@sha256:7a2395a48c394c3f4eae282f9d9873df45a18e8a409ccddf03b7583367b0a7da k8sprow.azurecr.io/azuredisk-csi:latest-v2-444ca6fc5bce97ea7c52380569d0659c5e105ce7],SizeBytes:136453655,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/secrets-store/driver@sha256:c0d040a1c4fbfceb65663e31c09ea40f4f78e356437610cbc3fbb4bb409bd6f1 mcr.microsoft.com/oss/kubernetes-csi/secrets-store/driver:v0.0.19],SizeBytes:123229697,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-proxy@sha256:0c890e918d42c158fab68867bade2ab1db04c3408452db9611a264ba3e43e6b3 mcr.microsoft.com/oss/kubernetes/kube-proxy:v1.23.7],SizeBytes:112296063,},ContainerImage{Names:[mcr.microsoft.com/oss/azure/secrets-store/provider-azure@sha256:6f67f3d0c7cdde5702f8ce7f101b6519daa0237f0c34fecb7c058b6af8c22ad1 mcr.microsoft.com/oss/azure/secrets-store/provider-azure:0.0.12],SizeBytes:101061355,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/autoscaler/cluster-autoscaler@sha256:6f0c680d375c62e74351f8ff3ed6ddb9b72ca759e0645c329b95f64264654a6d mcr.microsoft.com/oss/kubernetes/autoscaler/cluster-autoscaler:v1.22.1],SizeBytes:99962810,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-addon-manager@sha256:32e2836018c96e73533bd4642fe438e465b81dcbfa8b7b61935a6f4d0246c7ae mcr.microsoft.com/oss/kubernetes/kube-addon-manager:v9.1.3],SizeBytes:86832059,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-addon-manager@sha256:92c2c5aad9012ee32d2a43a74966cc0adc6ccb1705ad15abb10485ecf406d88b mcr.microsoft.com/oss/kubernetes/kube-addon-manager:v9.1.5],SizeBytes:84094027,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/metrics-server@sha256:1ef9d57ce41ffcc328b92494c3bfafe401e0b9a1694a295301a1385337d52815 mcr.microsoft.com/oss/kubernetes/metrics-server:v0.5.2],SizeBytes:64327621,},ContainerImage{Names:[mcr.microsoft.com/oss/nvidia/k8s-device-plugin@sha256:0f5b52bf28239234e831697d96db63ac03cde70fe68058f964504ab7564ee810 mcr.microsoft.com/oss/nvidia/k8s-device-plugin:1.0.0-beta6],SizeBytes:64160241,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-provisioner@sha256:542c9258a1441e4927883ea73425e21f9a14043a649686bf6b51700f34c64f8e mcr.microsoft.com/oss/kubernetes-csi/csi-provisioner:v3.1.0],SizeBytes:57858438,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-provisioner@sha256:e9ddadc44ba87a4a27f67e54760a14f9986885b534b3dff170a14eae1e35d213 mcr.microsoft.com/oss/kubernetes-csi/csi-provisioner:v3.0.0],SizeBytes:56881280,},ContainerImage{Names:[k8sprow.azurecr.io/azdiskschedulerextender-csi@sha256:295239e73ff3a728379df39f3985db8347e1547ecb2658620f29f3fdc1058d34 k8sprow.azurecr.io/azdiskschedulerextender-csi:latest-v2-444ca6fc5bce97ea7c52380569d0659c5e105ce7],SizeBytes:56760067,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-resizer@sha256:ab6c2e18a4d943f2bf8688e1779622277a1d14c355c9d4bfc0d761d86c5108b3 mcr.microsoft.com/oss/kubernetes-csi/csi-resizer:v1.4.0],SizeBytes:55577322,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-snapshotter@sha256:a889e925e15f9423f7842f1b769f64cbcf6a20b6956122836fc835cf22d9073f mcr.microsoft.com/oss/kubernetes-csi/csi-snapshotter:v5.0.1],SizeBytes:55314459,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-attacher@sha256:19fbca01394f7ed80151731180cfde0a3367f038beae2a97cda7928034010057 mcr.microsoft.com/oss/kubernetes-csi/csi-attacher:v3.4.0],SizeBytes:54954134,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-resizer@sha256:c5bb71ceaac60b1a4b58739fa07b709f6248c452ff6272a384d2f7648895a750 mcr.microsoft.com/oss/kubernetes-csi/csi-resizer:v1.3.0],SizeBytes:54313772,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-snapshotter@sha256:61849a026511cf332c87d73d0a7aed803b510c3ede197ec755389686d490de72 mcr.microsoft.com/oss/kubernetes-csi/csi-snapshotter:v4.2.1],SizeBytes:54210936,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-attacher@sha256:6b41e7153ebdfdc1501aa65184624bc15fd33a52d93f88ec3a758d0f8c9b8c10 mcr.microsoft.com/oss/kubernetes-csi/csi-attacher:v3.3.0],SizeBytes:53842561,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/snapshot-controller@sha256:8c3fc3c2667004ad6bbdf723bb64c5da66a5cb8b11d4ee59b67179b686223b13 mcr.microsoft.com/oss/kubernetes-csi/snapshot-controller:v5.0.1],SizeBytes:52732401,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/snapshot-controller@sha256:be5a8dc1f990828f653c77bd0a0f1bbd13197c3019f6e1d99d590389bac36705 mcr.microsoft.com/oss/kubernetes-csi/snapshot-controller:v4.2.1],SizeBytes:51575245,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager@sha256:0ad67f9919522a07318034641ae09bf2079b417e9944c65914410594ce645468 mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager:v1.1.4],SizeBytes:51478397,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager@sha256:31ec4f7daccd3e7a8504e12657d7830651ecacbe4a487daca1b1b7695a64b070 mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager:v1.23.1],SizeBytes:51249021,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager@sha256:011712ed90fb8efcf27928b0a47ed04b98baebb31cb1b2d8ab676977ec18eedc mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager:v1.1.4],SizeBytes:50868093,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager@sha256:0f9a8fbaed65192ed7dd795be4f9c1dc48ebdef0a241fb62d456f4bed40d9875 mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager:v1.23.1],SizeBytes:50679677,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-scheduler@sha256:c35f45e8dfaa30363e589296415a057f9d32813920fc3820d6557491657f199a mcr.microsoft.com/oss/kubernetes/kube-scheduler:v1.21.2],SizeBytes:50607491,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/ip-masq-agent@sha256:1244155f2ed3f33ff154cc343b8ad285f3391d95afd7d4b1c6dcc420bc0ba3cf mcr.microsoft.com/oss/kubernetes/ip-masq-agent:v2.5.0],SizeBytes:50146762,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager@sha256:7c907ff70b90a0bdf8fae63bd744018469dd9839cde1fd0515b93e0bbd14b34e mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager:v1.0.8],SizeBytes:48963453,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager@sha256:3987d7a8c6922ce1952ee19c5cb6ea75aac7b7c1b07aa79277ad038c69fb7a31 mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager:v1.0.8],SizeBytes:48349053,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/coredns@sha256:f873bf7f0928461efe10697fa76cf0ad7a1ae3041c5b57b50dd3d0b72d273f8c mcr.microsoft.com/oss/kubernetes/coredns:1.8.6],SizeBytes:46804601,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager@sha256:8073113a20882642a980b338635cdc5945e5673a18aef192090e6fde2b89a75c mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager:v0.6.0],SizeBytes:45909032,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager@sha256:6a32329628bdea3c6d75e98aad6155b65d2e2b98ca616eb33f9ac562912804c6 mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager:v0.6.0],SizeBytes:45229096,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager@sha256:ef6c4ba564b4d11d270f7d1563c50fbeb30ccc3b94146e5059228c49f95875f5 mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager:v0.7.11],SizeBytes:44916605,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager@sha256:dbcc384758ba5ca6d249596d471292ed3785e31cdb854d48b84d70794b669b4c mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager:v0.7.11],SizeBytes:43679613,},ContainerImage{Names:[mcr.microsoft.com/oss/etcd-io/etcd@sha256:cf587862e3f1b6fa4d9a2565520a34f164bdf72c50f37af8c3c668160593246e mcr.microsoft.com/oss/etcd-io/etcd:v3.3.25],SizeBytes:41832119,},ContainerImage{Names:[mcr.microsoft.com/k8s/aad-pod-identity/mic@sha256:bd9465be94966b9a917e1e3904fa5e63dd91772ccadf304e18ffd8e4ad8ccedd mcr.microsoft.com/k8s/aad-pod-identity/mic:1.6.1],SizeBytes:41374894,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/autoscaler/cluster-proportional-autoscaler@sha256:c849d75d61943ce7f51b4c049f1a79d19a08253966c8f49c4cfb6414cc33db8b mcr.microsoft.com/oss/kubernetes/autoscaler/cluster-proportional-autoscaler:1.8.5],SizeBytes:40661903,},ContainerImage{Names:[mcr.microsoft.com/k8s/aad-pod-identity/nmi@sha256:02128fefcdb7593ac53fc342e2c53a0fc6fabd813036bf60457bf43cc2940116 mcr.microsoft.com/k8s/aad-pod-identity/nmi:1.6.1],SizeBytes:38007982,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:921f301c44dda06a325164accf22e78ecc570b5c7d9d6ee4c66bd8cbb2b60b9a mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.16],SizeBytes:26970670,},ContainerImage{Names:[mcr.microsoft.com/k8s/kms/keyvault@sha256:1a27e175f8c125209e32d2957b5509fe20757bd8cb309ff9da598799b56326fb mcr.microsoft.com/k8s/kms/keyvault:v0.0.10],SizeBytes:23077387,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-node-driver-registrar@sha256:dbec3a8166686b09b242176ab5b99e993da4126438bbce68147c3fd654f35662 mcr.microsoft.com/oss/kubernetes-csi/csi-node-driver-registrar:v2.4.0],SizeBytes:19547289,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/livenessprobe@sha256:c96a6255c42766f6b8bb1a7cda02b0060ab1b20b2e2dafcc64ec09e7646745a6 mcr.microsoft.com/oss/kubernetes-csi/livenessprobe:v2.5.0],SizeBytes:17573341,},ContainerImage{Names:[mcr.microsoft.com/oss/busybox/busybox@sha256:582a641242b49809af3a1a522f9aae8c3f047d1c6ca1dd9d8cdabd349e45b1a9 mcr.microsoft.com/oss/busybox/busybox:1.33.1],SizeBytes:1235829,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8],SizeBytes:1159025,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/pause@sha256:e3b8c20681593c21b344ad801fbb8abaf564427ee3a57a9fcfa3b455f917ce46 mcr.microsoft.com/oss/kubernetes/pause:3.4.1],SizeBytes:682696,},},VolumesInUse:[kubernetes.io/csi/disk.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-u6qgg0s3/providers/Microsoft.Compute/disks/pre-provisioned-inline-volume],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/disk.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-u6qgg0s3/providers/Microsoft.Compute/disks/pre-provisioned-inline-volume,DevicePath:,},},Config:nil,},} ... skipping 93 lines ... [90m/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/pre_provisioning_test.go:49[0m [single-az] [90m/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/pre_provisioning_test.go:109[0m [91m[1mshould create an inline volume by in-tree driver [kubernetes.io/azure-disk] [It][0m [90m/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/pre_provisioning_test.go:368[0m [91mUnexpected error: <*errors.errorString | 0xc00061a930>: { s: "Gave up after waiting 15m0s for pod \"azuredisk-volume-tester-6gztt\" to be \"Succeeded or Failed\"", } Gave up after waiting 15m0s for pod "azuredisk-volume-tester-6gztt" to be "Succeeded or Failed" occurred[0m /home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/resources/pod.go:89 [90m------------------------------[0m [0mDynamic Provisioning[0m [90m[single-az][0m [1mshould create a volume on demand with mount options [kubernetes.io/azure-disk] [disk.csi.azure.com] [Windows] [csi-azuredisk-scheduler-extender][0m ... skipping 11 lines ... [1mSTEP[0m: setting up the StorageClass [1mSTEP[0m: creating a StorageClass [1mSTEP[0m: setting up the PVC and PV [1mSTEP[0m: creating a PVC [1mSTEP[0m: setting up the pod [1mSTEP[0m: deploying the pod [1mSTEP[0m: checking that the pod's command exits with no error Jun 18 01:32:26.559: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-gq6mh" in namespace "azuredisk-3213" to be "Succeeded or Failed" Jun 18 01:32:26.593: INFO: Pod "azuredisk-volume-tester-gq6mh": Phase="Pending", Reason="", readiness=false. Elapsed: 33.917744ms Jun 18 01:32:28.627: INFO: Pod "azuredisk-volume-tester-gq6mh": Phase="Pending", Reason="", readiness=false. Elapsed: 2.068160207s Jun 18 01:32:30.662: INFO: Pod "azuredisk-volume-tester-gq6mh": Phase="Pending", Reason="", readiness=false. Elapsed: 4.103387675s Jun 18 01:32:32.698: INFO: Pod "azuredisk-volume-tester-gq6mh": Phase="Pending", Reason="", readiness=false. Elapsed: 6.138654658s Jun 18 01:32:34.732: INFO: Pod "azuredisk-volume-tester-gq6mh": Phase="Pending", Reason="", readiness=false. Elapsed: 8.173190196s Jun 18 01:32:36.768: INFO: Pod "azuredisk-volume-tester-gq6mh": Phase="Pending", Reason="", readiness=false. Elapsed: 10.208830019s ... skipping 432 lines ... Jun 18 01:47:17.975: INFO: Pod "azuredisk-volume-tester-gq6mh": Phase="Pending", Reason="", readiness=false. Elapsed: 14m51.416159073s Jun 18 01:47:20.010: INFO: Pod "azuredisk-volume-tester-gq6mh": Phase="Pending", Reason="", readiness=false. Elapsed: 14m53.451434797s Jun 18 01:47:22.045: INFO: Pod "azuredisk-volume-tester-gq6mh": Phase="Pending", Reason="", readiness=false. Elapsed: 14m55.486529917s Jun 18 01:47:24.081: INFO: Pod "azuredisk-volume-tester-gq6mh": Phase="Pending", Reason="", readiness=false. Elapsed: 14m57.521890479s Jun 18 01:47:26.115: INFO: Pod "azuredisk-volume-tester-gq6mh": Phase="Pending", Reason="", readiness=false. Elapsed: 14m59.556202858s Jun 18 01:47:28.117: INFO: deleting Pod "azuredisk-3213"/"azuredisk-volume-tester-gq6mh" Jun 18 01:47:28.202: INFO: Error getting logs for pod azuredisk-volume-tester-gq6mh: the server rejected our request for an unknown reason (get pods azuredisk-volume-tester-gq6mh) [1mSTEP[0m: Deleting pod azuredisk-volume-tester-gq6mh in namespace azuredisk-3213 [1mSTEP[0m: validating provisioned PV [1mSTEP[0m: checking the PV Jun 18 01:47:28.309: INFO: deleting PVC "azuredisk-3213"/"pvc-qhj7l" Jun 18 01:47:28.309: INFO: Deleting PersistentVolumeClaim "pvc-qhj7l" [1mSTEP[0m: waiting for claim's PV "pvc-717567da-6cbc-4d33-9178-e7387d5ba600" to be deleted ... skipping 125 lines ... Jun 18 01:57:32.753: INFO: At 2022-06-18 01:32:26 +0000 UTC - event for pvc-qhj7l: {disk.csi.azure.com_k8s-agentpool1-42835052-0_53a95263-aec0-4b87-a465-aa36e34998da } Provisioning: External provisioner is provisioning volume for claim "azuredisk-3213/pvc-qhj7l" Jun 18 01:57:32.753: INFO: At 2022-06-18 01:32:26 +0000 UTC - event for pvc-qhj7l: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding Jun 18 01:57:32.753: INFO: At 2022-06-18 01:32:26 +0000 UTC - event for pvc-qhj7l: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "disk.csi.azure.com" or manually created by system administrator Jun 18 01:57:32.753: INFO: At 2022-06-18 01:32:29 +0000 UTC - event for pvc-qhj7l: {disk.csi.azure.com_k8s-agentpool1-42835052-0_53a95263-aec0-4b87-a465-aa36e34998da } ProvisioningSucceeded: Successfully provisioned volume pvc-717567da-6cbc-4d33-9178-e7387d5ba600 Jun 18 01:57:32.753: INFO: At 2022-06-18 01:32:31 +0000 UTC - event for azuredisk-volume-tester-gq6mh: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-717567da-6cbc-4d33-9178-e7387d5ba600" Jun 18 01:57:32.753: INFO: At 2022-06-18 01:34:32 +0000 UTC - event for azuredisk-volume-tester-gq6mh: {kubelet k8s-agentpool1-42835052-1} FailedMount: Unable to attach or mount volumes: unmounted volumes=[test-volume-1], unattached volumes=[test-volume-1 kube-api-access-4tdhr]: timed out waiting for the condition Jun 18 01:57:32.753: INFO: At 2022-06-18 01:34:37 +0000 UTC - event for azuredisk-volume-tester-gq6mh: {kubelet k8s-agentpool1-42835052-1} FailedMount: MountVolume.MountDevice failed for volume "pvc-717567da-6cbc-4d33-9178-e7387d5ba600" : rpc error: code = DeadlineExceeded desc = context deadline exceeded Jun 18 01:57:32.753: INFO: At 2022-06-18 01:45:54 +0000 UTC - event for azuredisk-volume-tester-gq6mh: {kubelet k8s-agentpool1-42835052-1} FailedMount: Unable to attach or mount volumes: unmounted volumes=[test-volume-1], unattached volumes=[kube-api-access-4tdhr test-volume-1]: timed out waiting for the condition Jun 18 01:57:32.786: INFO: POD NODE PHASE GRACE CONDITIONS Jun 18 01:57:32.786: INFO: Jun 18 01:57:32.852: INFO: Logging node info for node k8s-agentpool1-42835052-0 Jun 18 01:57:32.885: INFO: Node Info: &Node{ObjectMeta:{k8s-agentpool1-42835052-0 b3450a63-c373-4a96-b548-c74917f39933 10560 0 2022-06-18 01:05:58 +0000 UTC <nil> <nil> map[agentpool:agentpool1 beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_DS2_v2 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:canadacentral failure-domain.beta.kubernetes.io/zone:0 kubernetes.azure.com/cluster:kubetest-u6qgg0s3 kubernetes.azure.com/role:agent kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-agentpool1-42835052-0 kubernetes.io/os:linux kubernetes.io/role:agent node-role.kubernetes.io/agent: node.kubernetes.io/instance-type:Standard_DS2_v2 storageprofile:managed storagetier:Premium_LRS topology.disk.csi.azure.com/zone: topology.kubernetes.io/region:canadacentral topology.kubernetes.io/zone:0] map[csi.volume.kubernetes.io/nodeid:{"disk.csi.azure.com":"k8s-agentpool1-42835052-0"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2022-06-18 01:05:58 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:agentpool":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.azure.com/cluster":{},"f:kubernetes.azure.com/role":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:storageprofile":{},"f:storagetier":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kube-controller-manager Update v1 2022-06-18 01:06:07 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"10.244.2.0/24\"":{}}}} } {kubectl-label Update v1 2022-06-18 01:06:53 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/agent":{}}}} } {kubelet Update v1 2022-06-18 01:34:45 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.disk.csi.azure.com/zone":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status} {kube-controller-manager Update v1 2022-06-18 01:57:04 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:volumesAttached":{}}} status}]},Spec:NodeSpec{PodCIDR:10.244.2.0/24,DoNotUseExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-u6qgg0s3/providers/Microsoft.Compute/virtualMachines/k8s-agentpool1-42835052-0,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[10.244.2.0/24],},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-azure-disk: {{8 0} {<nil>} 8 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{31036686336 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7287771136 0} {<nil>} 7116964Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-azure-disk: {{8 0} {<nil>} 8 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{27933017657 0} {<nil>} 27933017657 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{6501339136 0} {<nil>} 6348964Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-06-18 01:06:26 +0000 UTC,LastTransitionTime:2022-06-18 01:06:26 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-06-18 01:55:12 +0000 UTC,LastTransitionTime:2022-06-18 01:05:58 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-06-18 01:55:12 +0000 UTC,LastTransitionTime:2022-06-18 01:05:58 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-06-18 01:55:12 +0000 UTC,LastTransitionTime:2022-06-18 01:05:58 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-06-18 01:55:12 +0000 UTC,LastTransitionTime:2022-06-18 01:06:06 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:Hostname,Address:k8s-agentpool1-42835052-0,},NodeAddress{Type:InternalIP,Address:10.240.0.5,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:fd2eb91f0eeb436286bc238b58e4dc04,SystemUUID:c3ccb155-829c-6a40-8fdc-f652683a9f79,BootID:0082daec-2a09-4d8b-b449-5299bd7dc135,KernelVersion:5.4.0-1074-azure,OSImage:Ubuntu 18.04.6 LTS,ContainerRuntimeVersion:docker://20.10.11+azure-3,KubeletVersion:v1.23.7,KubeProxyVersion:v1.23.7,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/azuredisk-csi@sha256:423eb6cf602c064c8b2deefead5ceadd6324ed41b3d995dab5d0f6f0f4d4710f mcr.microsoft.com/oss/kubernetes-csi/azuredisk-csi:v1.10.0],SizeBytes:245959792,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/azurefile-csi@sha256:9e2ecabcf9dd9943e6600eb9fb460f45b4dc61af7cabe95d115082a029db2aaf mcr.microsoft.com/oss/kubernetes-csi/azurefile-csi:v1.9.0],SizeBytes:230470852,},ContainerImage{Names:[mcr.microsoft.com/containernetworking/azure-npm@sha256:106f669f48e5e80c4ec0afb49858ead72cf4b901cd8664e7bf81f8d789e56e12 mcr.microsoft.com/containernetworking/azure-npm:v1.2.2_hotfix],SizeBytes:175230380,},ContainerImage{Names:[k8sprow.azurecr.io/azuredisk-csi@sha256:7a2395a48c394c3f4eae282f9d9873df45a18e8a409ccddf03b7583367b0a7da k8sprow.azurecr.io/azuredisk-csi:latest-v2-444ca6fc5bce97ea7c52380569d0659c5e105ce7],SizeBytes:136453655,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/secrets-store/driver@sha256:c0d040a1c4fbfceb65663e31c09ea40f4f78e356437610cbc3fbb4bb409bd6f1 mcr.microsoft.com/oss/kubernetes-csi/secrets-store/driver:v0.0.19],SizeBytes:123229697,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-proxy@sha256:0c890e918d42c158fab68867bade2ab1db04c3408452db9611a264ba3e43e6b3 mcr.microsoft.com/oss/kubernetes/kube-proxy:v1.23.7],SizeBytes:112296063,},ContainerImage{Names:[mcr.microsoft.com/oss/azure/secrets-store/provider-azure@sha256:6f67f3d0c7cdde5702f8ce7f101b6519daa0237f0c34fecb7c058b6af8c22ad1 mcr.microsoft.com/oss/azure/secrets-store/provider-azure:0.0.12],SizeBytes:101061355,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/autoscaler/cluster-autoscaler@sha256:6f0c680d375c62e74351f8ff3ed6ddb9b72ca759e0645c329b95f64264654a6d mcr.microsoft.com/oss/kubernetes/autoscaler/cluster-autoscaler:v1.22.1],SizeBytes:99962810,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-addon-manager@sha256:32e2836018c96e73533bd4642fe438e465b81dcbfa8b7b61935a6f4d0246c7ae mcr.microsoft.com/oss/kubernetes/kube-addon-manager:v9.1.3],SizeBytes:86832059,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-addon-manager@sha256:92c2c5aad9012ee32d2a43a74966cc0adc6ccb1705ad15abb10485ecf406d88b mcr.microsoft.com/oss/kubernetes/kube-addon-manager:v9.1.5],SizeBytes:84094027,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/metrics-server@sha256:1ef9d57ce41ffcc328b92494c3bfafe401e0b9a1694a295301a1385337d52815 mcr.microsoft.com/oss/kubernetes/metrics-server:v0.5.2],SizeBytes:64327621,},ContainerImage{Names:[mcr.microsoft.com/oss/nvidia/k8s-device-plugin@sha256:0f5b52bf28239234e831697d96db63ac03cde70fe68058f964504ab7564ee810 mcr.microsoft.com/oss/nvidia/k8s-device-plugin:1.0.0-beta6],SizeBytes:64160241,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-provisioner@sha256:542c9258a1441e4927883ea73425e21f9a14043a649686bf6b51700f34c64f8e mcr.microsoft.com/oss/kubernetes-csi/csi-provisioner:v3.1.0],SizeBytes:57858438,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-provisioner@sha256:e9ddadc44ba87a4a27f67e54760a14f9986885b534b3dff170a14eae1e35d213 mcr.microsoft.com/oss/kubernetes-csi/csi-provisioner:v3.0.0],SizeBytes:56881280,},ContainerImage{Names:[k8sprow.azurecr.io/azdiskschedulerextender-csi@sha256:295239e73ff3a728379df39f3985db8347e1547ecb2658620f29f3fdc1058d34 k8sprow.azurecr.io/azdiskschedulerextender-csi:latest-v2-444ca6fc5bce97ea7c52380569d0659c5e105ce7],SizeBytes:56760067,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-resizer@sha256:ab6c2e18a4d943f2bf8688e1779622277a1d14c355c9d4bfc0d761d86c5108b3 mcr.microsoft.com/oss/kubernetes-csi/csi-resizer:v1.4.0],SizeBytes:55577322,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-snapshotter@sha256:a889e925e15f9423f7842f1b769f64cbcf6a20b6956122836fc835cf22d9073f mcr.microsoft.com/oss/kubernetes-csi/csi-snapshotter:v5.0.1],SizeBytes:55314459,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-attacher@sha256:19fbca01394f7ed80151731180cfde0a3367f038beae2a97cda7928034010057 mcr.microsoft.com/oss/kubernetes-csi/csi-attacher:v3.4.0],SizeBytes:54954134,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-resizer@sha256:c5bb71ceaac60b1a4b58739fa07b709f6248c452ff6272a384d2f7648895a750 mcr.microsoft.com/oss/kubernetes-csi/csi-resizer:v1.3.0],SizeBytes:54313772,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-snapshotter@sha256:61849a026511cf332c87d73d0a7aed803b510c3ede197ec755389686d490de72 mcr.microsoft.com/oss/kubernetes-csi/csi-snapshotter:v4.2.1],SizeBytes:54210936,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-attacher@sha256:6b41e7153ebdfdc1501aa65184624bc15fd33a52d93f88ec3a758d0f8c9b8c10 mcr.microsoft.com/oss/kubernetes-csi/csi-attacher:v3.3.0],SizeBytes:53842561,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/snapshot-controller@sha256:8c3fc3c2667004ad6bbdf723bb64c5da66a5cb8b11d4ee59b67179b686223b13 mcr.microsoft.com/oss/kubernetes-csi/snapshot-controller:v5.0.1],SizeBytes:52732401,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/snapshot-controller@sha256:be5a8dc1f990828f653c77bd0a0f1bbd13197c3019f6e1d99d590389bac36705 mcr.microsoft.com/oss/kubernetes-csi/snapshot-controller:v4.2.1],SizeBytes:51575245,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager@sha256:0ad67f9919522a07318034641ae09bf2079b417e9944c65914410594ce645468 mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager:v1.1.4],SizeBytes:51478397,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager@sha256:31ec4f7daccd3e7a8504e12657d7830651ecacbe4a487daca1b1b7695a64b070 mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager:v1.23.1],SizeBytes:51249021,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager@sha256:011712ed90fb8efcf27928b0a47ed04b98baebb31cb1b2d8ab676977ec18eedc mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager:v1.1.4],SizeBytes:50868093,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager@sha256:0f9a8fbaed65192ed7dd795be4f9c1dc48ebdef0a241fb62d456f4bed40d9875 mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager:v1.23.1],SizeBytes:50679677,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-scheduler@sha256:c35f45e8dfaa30363e589296415a057f9d32813920fc3820d6557491657f199a mcr.microsoft.com/oss/kubernetes/kube-scheduler:v1.21.2],SizeBytes:50607491,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/ip-masq-agent@sha256:1244155f2ed3f33ff154cc343b8ad285f3391d95afd7d4b1c6dcc420bc0ba3cf mcr.microsoft.com/oss/kubernetes/ip-masq-agent:v2.5.0],SizeBytes:50146762,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager@sha256:7c907ff70b90a0bdf8fae63bd744018469dd9839cde1fd0515b93e0bbd14b34e mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager:v1.0.8],SizeBytes:48963453,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager@sha256:3987d7a8c6922ce1952ee19c5cb6ea75aac7b7c1b07aa79277ad038c69fb7a31 mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager:v1.0.8],SizeBytes:48349053,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/coredns@sha256:f873bf7f0928461efe10697fa76cf0ad7a1ae3041c5b57b50dd3d0b72d273f8c mcr.microsoft.com/oss/kubernetes/coredns:1.8.6],SizeBytes:46804601,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager@sha256:8073113a20882642a980b338635cdc5945e5673a18aef192090e6fde2b89a75c mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager:v0.6.0],SizeBytes:45909032,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager@sha256:6a32329628bdea3c6d75e98aad6155b65d2e2b98ca616eb33f9ac562912804c6 mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager:v0.6.0],SizeBytes:45229096,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager@sha256:ef6c4ba564b4d11d270f7d1563c50fbeb30ccc3b94146e5059228c49f95875f5 mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager:v0.7.11],SizeBytes:44916605,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager@sha256:dbcc384758ba5ca6d249596d471292ed3785e31cdb854d48b84d70794b669b4c mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager:v0.7.11],SizeBytes:43679613,},ContainerImage{Names:[mcr.microsoft.com/oss/etcd-io/etcd@sha256:cf587862e3f1b6fa4d9a2565520a34f164bdf72c50f37af8c3c668160593246e mcr.microsoft.com/oss/etcd-io/etcd:v3.3.25],SizeBytes:41832119,},ContainerImage{Names:[mcr.microsoft.com/k8s/aad-pod-identity/mic@sha256:bd9465be94966b9a917e1e3904fa5e63dd91772ccadf304e18ffd8e4ad8ccedd mcr.microsoft.com/k8s/aad-pod-identity/mic:1.6.1],SizeBytes:41374894,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/autoscaler/cluster-proportional-autoscaler@sha256:c849d75d61943ce7f51b4c049f1a79d19a08253966c8f49c4cfb6414cc33db8b mcr.microsoft.com/oss/kubernetes/autoscaler/cluster-proportional-autoscaler:1.8.5],SizeBytes:40661903,},ContainerImage{Names:[mcr.microsoft.com/k8s/aad-pod-identity/nmi@sha256:02128fefcdb7593ac53fc342e2c53a0fc6fabd813036bf60457bf43cc2940116 mcr.microsoft.com/k8s/aad-pod-identity/nmi:1.6.1],SizeBytes:38007982,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:921f301c44dda06a325164accf22e78ecc570b5c7d9d6ee4c66bd8cbb2b60b9a mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.16],SizeBytes:26970670,},ContainerImage{Names:[mcr.microsoft.com/k8s/kms/keyvault@sha256:1a27e175f8c125209e32d2957b5509fe20757bd8cb309ff9da598799b56326fb mcr.microsoft.com/k8s/kms/keyvault:v0.0.10],SizeBytes:23077387,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-node-driver-registrar@sha256:dbec3a8166686b09b242176ab5b99e993da4126438bbce68147c3fd654f35662 mcr.microsoft.com/oss/kubernetes-csi/csi-node-driver-registrar:v2.4.0],SizeBytes:19547289,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/livenessprobe@sha256:c96a6255c42766f6b8bb1a7cda02b0060ab1b20b2e2dafcc64ec09e7646745a6 mcr.microsoft.com/oss/kubernetes-csi/livenessprobe:v2.5.0],SizeBytes:17573341,},ContainerImage{Names:[mcr.microsoft.com/oss/busybox/busybox@sha256:582a641242b49809af3a1a522f9aae8c3f047d1c6ca1dd9d8cdabd349e45b1a9 mcr.microsoft.com/oss/busybox/busybox:1.33.1],SizeBytes:1235829,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8],SizeBytes:1159025,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/pause@sha256:e3b8c20681593c21b344ad801fbb8abaf564427ee3a57a9fcfa3b455f917ce46 mcr.microsoft.com/oss/kubernetes/pause:3.4.1],SizeBytes:682696,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/disk.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-u6qgg0s3/providers/Microsoft.Compute/disks/pre-provisioned-inline-volume,DevicePath:,},},Config:nil,},} ... skipping 91 lines ... [90m/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning_test.go:40[0m [single-az] [90m/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning_test.go:44[0m [91m[1mshould create a volume on demand with mount options [kubernetes.io/azure-disk] [disk.csi.azure.com] [Windows] [csi-azuredisk-scheduler-extender] [It][0m [90m/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning_test.go:102[0m [91mUnexpected error: <*errors.errorString | 0xc00098b130>: { s: "Gave up after waiting 15m0s for pod \"azuredisk-volume-tester-gq6mh\" to be \"Succeeded or Failed\"", } Gave up after waiting 15m0s for pod "azuredisk-volume-tester-gq6mh" to be "Succeeded or Failed" occurred[0m /home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/resources/pod.go:89 [90m------------------------------[0m [0mDynamic Provisioning[0m [90m[single-az][0m [1mshould create a pod with volume mount subpath [disk.csi.azure.com] [Windows][0m ... skipping 97 lines ... [1mSTEP[0m: setting up the PVC and PV [1mSTEP[0m: creating a PVC [1mSTEP[0m: setting up the pod [1mSTEP[0m: deploying the pod [1mSTEP[0m: checking that the pod has 'FailedMount' event Jun 18 01:59:46.699: INFO: deleting Pod "azuredisk-8260"/"azuredisk-volume-tester-xxjpj" Jun 18 01:59:46.758: INFO: Error getting logs for pod azuredisk-volume-tester-xxjpj: the server rejected our request for an unknown reason (get pods azuredisk-volume-tester-xxjpj) [1mSTEP[0m: Deleting pod azuredisk-volume-tester-xxjpj in namespace azuredisk-8260 [1mSTEP[0m: validating provisioned PV [1mSTEP[0m: checking the PV Jun 18 01:59:46.860: INFO: deleting PVC "azuredisk-8260"/"pvc-t95lf" Jun 18 01:59:46.860: INFO: Deleting PersistentVolumeClaim "pvc-t95lf" [1mSTEP[0m: waiting for claim's PV "pvc-76934f53-9b56-4add-b7c0-aca144f3145d" to be deleted ... skipping 125 lines ... Jun 18 02:09:51.297: INFO: At 2022-06-18 01:57:36 +0000 UTC - event for pvc-t95lf: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding Jun 18 02:09:51.297: INFO: At 2022-06-18 01:57:36 +0000 UTC - event for pvc-t95lf: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "disk.csi.azure.com" or manually created by system administrator Jun 18 02:09:51.297: INFO: At 2022-06-18 01:57:36 +0000 UTC - event for pvc-t95lf: {disk.csi.azure.com_k8s-agentpool1-42835052-0_53a95263-aec0-4b87-a465-aa36e34998da } Provisioning: External provisioner is provisioning volume for claim "azuredisk-8260/pvc-t95lf" Jun 18 02:09:51.297: INFO: At 2022-06-18 01:57:39 +0000 UTC - event for pvc-t95lf: {disk.csi.azure.com_k8s-agentpool1-42835052-0_53a95263-aec0-4b87-a465-aa36e34998da } ProvisioningSucceeded: Successfully provisioned volume pvc-76934f53-9b56-4add-b7c0-aca144f3145d Jun 18 02:09:51.297: INFO: At 2022-06-18 01:57:41 +0000 UTC - event for azuredisk-volume-tester-xxjpj: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-76934f53-9b56-4add-b7c0-aca144f3145d" Jun 18 02:09:51.297: INFO: At 2022-06-18 01:59:42 +0000 UTC - event for azuredisk-volume-tester-xxjpj: {kubelet k8s-agentpool1-42835052-0} FailedMount: Unable to attach or mount volumes: unmounted volumes=[test-volume-1], unattached volumes=[test-volume-1 kube-api-access-kckt5]: timed out waiting for the condition Jun 18 02:09:51.297: INFO: At 2022-06-18 01:59:45 +0000 UTC - event for azuredisk-volume-tester-xxjpj: {kubelet k8s-agentpool1-42835052-0} FailedMount: MountVolume.MountDevice failed for volume "pvc-76934f53-9b56-4add-b7c0-aca144f3145d" : rpc error: code = DeadlineExceeded desc = context deadline exceeded Jun 18 02:09:51.330: INFO: POD NODE PHASE GRACE CONDITIONS Jun 18 02:09:51.330: INFO: Jun 18 02:09:51.401: INFO: Logging node info for node k8s-agentpool1-42835052-0 Jun 18 02:09:51.435: INFO: Node Info: &Node{ObjectMeta:{k8s-agentpool1-42835052-0 b3450a63-c373-4a96-b548-c74917f39933 13282 0 2022-06-18 01:05:58 +0000 UTC <nil> <nil> map[agentpool:agentpool1 beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_DS2_v2 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:canadacentral failure-domain.beta.kubernetes.io/zone:0 kubernetes.azure.com/cluster:kubetest-u6qgg0s3 kubernetes.azure.com/role:agent kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-agentpool1-42835052-0 kubernetes.io/os:linux kubernetes.io/role:agent node-role.kubernetes.io/agent: node.kubernetes.io/instance-type:Standard_DS2_v2 storageprofile:managed storagetier:Premium_LRS topology.disk.csi.azure.com/zone: topology.kubernetes.io/region:canadacentral topology.kubernetes.io/zone:0] map[csi.volume.kubernetes.io/nodeid:{"disk.csi.azure.com":"k8s-agentpool1-42835052-0"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2022-06-18 01:05:58 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:agentpool":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.azure.com/cluster":{},"f:kubernetes.azure.com/role":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:storageprofile":{},"f:storagetier":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kube-controller-manager Update v1 2022-06-18 01:06:07 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"10.244.2.0/24\"":{}}}} } {kubectl-label Update v1 2022-06-18 01:06:53 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/agent":{}}}} } {kubelet Update v1 2022-06-18 02:01:49 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.disk.csi.azure.com/zone":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status} {kube-controller-manager Update v1 2022-06-18 02:09:20 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:volumesAttached":{}}} status}]},Spec:NodeSpec{PodCIDR:10.244.2.0/24,DoNotUseExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-u6qgg0s3/providers/Microsoft.Compute/virtualMachines/k8s-agentpool1-42835052-0,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[10.244.2.0/24],},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-azure-disk: {{8 0} {<nil>} 8 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{31036686336 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7287771136 0} {<nil>} 7116964Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-azure-disk: {{8 0} {<nil>} 8 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{27933017657 0} {<nil>} 27933017657 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{6501339136 0} {<nil>} 6348964Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-06-18 01:06:26 +0000 UTC,LastTransitionTime:2022-06-18 01:06:26 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-06-18 02:06:56 +0000 UTC,LastTransitionTime:2022-06-18 01:05:58 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-06-18 02:06:56 +0000 UTC,LastTransitionTime:2022-06-18 01:05:58 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-06-18 02:06:56 +0000 UTC,LastTransitionTime:2022-06-18 01:05:58 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-06-18 02:06:56 +0000 UTC,LastTransitionTime:2022-06-18 01:06:06 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:Hostname,Address:k8s-agentpool1-42835052-0,},NodeAddress{Type:InternalIP,Address:10.240.0.5,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:fd2eb91f0eeb436286bc238b58e4dc04,SystemUUID:c3ccb155-829c-6a40-8fdc-f652683a9f79,BootID:0082daec-2a09-4d8b-b449-5299bd7dc135,KernelVersion:5.4.0-1074-azure,OSImage:Ubuntu 18.04.6 LTS,ContainerRuntimeVersion:docker://20.10.11+azure-3,KubeletVersion:v1.23.7,KubeProxyVersion:v1.23.7,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/azuredisk-csi@sha256:423eb6cf602c064c8b2deefead5ceadd6324ed41b3d995dab5d0f6f0f4d4710f mcr.microsoft.com/oss/kubernetes-csi/azuredisk-csi:v1.10.0],SizeBytes:245959792,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/azurefile-csi@sha256:9e2ecabcf9dd9943e6600eb9fb460f45b4dc61af7cabe95d115082a029db2aaf mcr.microsoft.com/oss/kubernetes-csi/azurefile-csi:v1.9.0],SizeBytes:230470852,},ContainerImage{Names:[mcr.microsoft.com/containernetworking/azure-npm@sha256:106f669f48e5e80c4ec0afb49858ead72cf4b901cd8664e7bf81f8d789e56e12 mcr.microsoft.com/containernetworking/azure-npm:v1.2.2_hotfix],SizeBytes:175230380,},ContainerImage{Names:[k8sprow.azurecr.io/azuredisk-csi@sha256:7a2395a48c394c3f4eae282f9d9873df45a18e8a409ccddf03b7583367b0a7da k8sprow.azurecr.io/azuredisk-csi:latest-v2-444ca6fc5bce97ea7c52380569d0659c5e105ce7],SizeBytes:136453655,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/secrets-store/driver@sha256:c0d040a1c4fbfceb65663e31c09ea40f4f78e356437610cbc3fbb4bb409bd6f1 mcr.microsoft.com/oss/kubernetes-csi/secrets-store/driver:v0.0.19],SizeBytes:123229697,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-proxy@sha256:0c890e918d42c158fab68867bade2ab1db04c3408452db9611a264ba3e43e6b3 mcr.microsoft.com/oss/kubernetes/kube-proxy:v1.23.7],SizeBytes:112296063,},ContainerImage{Names:[mcr.microsoft.com/oss/azure/secrets-store/provider-azure@sha256:6f67f3d0c7cdde5702f8ce7f101b6519daa0237f0c34fecb7c058b6af8c22ad1 mcr.microsoft.com/oss/azure/secrets-store/provider-azure:0.0.12],SizeBytes:101061355,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/autoscaler/cluster-autoscaler@sha256:6f0c680d375c62e74351f8ff3ed6ddb9b72ca759e0645c329b95f64264654a6d mcr.microsoft.com/oss/kubernetes/autoscaler/cluster-autoscaler:v1.22.1],SizeBytes:99962810,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-addon-manager@sha256:32e2836018c96e73533bd4642fe438e465b81dcbfa8b7b61935a6f4d0246c7ae mcr.microsoft.com/oss/kubernetes/kube-addon-manager:v9.1.3],SizeBytes:86832059,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-addon-manager@sha256:92c2c5aad9012ee32d2a43a74966cc0adc6ccb1705ad15abb10485ecf406d88b mcr.microsoft.com/oss/kubernetes/kube-addon-manager:v9.1.5],SizeBytes:84094027,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/metrics-server@sha256:1ef9d57ce41ffcc328b92494c3bfafe401e0b9a1694a295301a1385337d52815 mcr.microsoft.com/oss/kubernetes/metrics-server:v0.5.2],SizeBytes:64327621,},ContainerImage{Names:[mcr.microsoft.com/oss/nvidia/k8s-device-plugin@sha256:0f5b52bf28239234e831697d96db63ac03cde70fe68058f964504ab7564ee810 mcr.microsoft.com/oss/nvidia/k8s-device-plugin:1.0.0-beta6],SizeBytes:64160241,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-provisioner@sha256:542c9258a1441e4927883ea73425e21f9a14043a649686bf6b51700f34c64f8e mcr.microsoft.com/oss/kubernetes-csi/csi-provisioner:v3.1.0],SizeBytes:57858438,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-provisioner@sha256:e9ddadc44ba87a4a27f67e54760a14f9986885b534b3dff170a14eae1e35d213 mcr.microsoft.com/oss/kubernetes-csi/csi-provisioner:v3.0.0],SizeBytes:56881280,},ContainerImage{Names:[k8sprow.azurecr.io/azdiskschedulerextender-csi@sha256:295239e73ff3a728379df39f3985db8347e1547ecb2658620f29f3fdc1058d34 k8sprow.azurecr.io/azdiskschedulerextender-csi:latest-v2-444ca6fc5bce97ea7c52380569d0659c5e105ce7],SizeBytes:56760067,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-resizer@sha256:ab6c2e18a4d943f2bf8688e1779622277a1d14c355c9d4bfc0d761d86c5108b3 mcr.microsoft.com/oss/kubernetes-csi/csi-resizer:v1.4.0],SizeBytes:55577322,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-snapshotter@sha256:a889e925e15f9423f7842f1b769f64cbcf6a20b6956122836fc835cf22d9073f mcr.microsoft.com/oss/kubernetes-csi/csi-snapshotter:v5.0.1],SizeBytes:55314459,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-attacher@sha256:19fbca01394f7ed80151731180cfde0a3367f038beae2a97cda7928034010057 mcr.microsoft.com/oss/kubernetes-csi/csi-attacher:v3.4.0],SizeBytes:54954134,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-resizer@sha256:c5bb71ceaac60b1a4b58739fa07b709f6248c452ff6272a384d2f7648895a750 mcr.microsoft.com/oss/kubernetes-csi/csi-resizer:v1.3.0],SizeBytes:54313772,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-snapshotter@sha256:61849a026511cf332c87d73d0a7aed803b510c3ede197ec755389686d490de72 mcr.microsoft.com/oss/kubernetes-csi/csi-snapshotter:v4.2.1],SizeBytes:54210936,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-attacher@sha256:6b41e7153ebdfdc1501aa65184624bc15fd33a52d93f88ec3a758d0f8c9b8c10 mcr.microsoft.com/oss/kubernetes-csi/csi-attacher:v3.3.0],SizeBytes:53842561,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/snapshot-controller@sha256:8c3fc3c2667004ad6bbdf723bb64c5da66a5cb8b11d4ee59b67179b686223b13 mcr.microsoft.com/oss/kubernetes-csi/snapshot-controller:v5.0.1],SizeBytes:52732401,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/snapshot-controller@sha256:be5a8dc1f990828f653c77bd0a0f1bbd13197c3019f6e1d99d590389bac36705 mcr.microsoft.com/oss/kubernetes-csi/snapshot-controller:v4.2.1],SizeBytes:51575245,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager@sha256:0ad67f9919522a07318034641ae09bf2079b417e9944c65914410594ce645468 mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager:v1.1.4],SizeBytes:51478397,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager@sha256:31ec4f7daccd3e7a8504e12657d7830651ecacbe4a487daca1b1b7695a64b070 mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager:v1.23.1],SizeBytes:51249021,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager@sha256:011712ed90fb8efcf27928b0a47ed04b98baebb31cb1b2d8ab676977ec18eedc mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager:v1.1.4],SizeBytes:50868093,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager@sha256:0f9a8fbaed65192ed7dd795be4f9c1dc48ebdef0a241fb62d456f4bed40d9875 mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager:v1.23.1],SizeBytes:50679677,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-scheduler@sha256:c35f45e8dfaa30363e589296415a057f9d32813920fc3820d6557491657f199a mcr.microsoft.com/oss/kubernetes/kube-scheduler:v1.21.2],SizeBytes:50607491,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/ip-masq-agent@sha256:1244155f2ed3f33ff154cc343b8ad285f3391d95afd7d4b1c6dcc420bc0ba3cf mcr.microsoft.com/oss/kubernetes/ip-masq-agent:v2.5.0],SizeBytes:50146762,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager@sha256:7c907ff70b90a0bdf8fae63bd744018469dd9839cde1fd0515b93e0bbd14b34e mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager:v1.0.8],SizeBytes:48963453,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager@sha256:3987d7a8c6922ce1952ee19c5cb6ea75aac7b7c1b07aa79277ad038c69fb7a31 mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager:v1.0.8],SizeBytes:48349053,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/coredns@sha256:f873bf7f0928461efe10697fa76cf0ad7a1ae3041c5b57b50dd3d0b72d273f8c mcr.microsoft.com/oss/kubernetes/coredns:1.8.6],SizeBytes:46804601,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager@sha256:8073113a20882642a980b338635cdc5945e5673a18aef192090e6fde2b89a75c mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager:v0.6.0],SizeBytes:45909032,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager@sha256:6a32329628bdea3c6d75e98aad6155b65d2e2b98ca616eb33f9ac562912804c6 mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager:v0.6.0],SizeBytes:45229096,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager@sha256:ef6c4ba564b4d11d270f7d1563c50fbeb30ccc3b94146e5059228c49f95875f5 mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager:v0.7.11],SizeBytes:44916605,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager@sha256:dbcc384758ba5ca6d249596d471292ed3785e31cdb854d48b84d70794b669b4c mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager:v0.7.11],SizeBytes:43679613,},ContainerImage{Names:[mcr.microsoft.com/oss/etcd-io/etcd@sha256:cf587862e3f1b6fa4d9a2565520a34f164bdf72c50f37af8c3c668160593246e mcr.microsoft.com/oss/etcd-io/etcd:v3.3.25],SizeBytes:41832119,},ContainerImage{Names:[mcr.microsoft.com/k8s/aad-pod-identity/mic@sha256:bd9465be94966b9a917e1e3904fa5e63dd91772ccadf304e18ffd8e4ad8ccedd mcr.microsoft.com/k8s/aad-pod-identity/mic:1.6.1],SizeBytes:41374894,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/autoscaler/cluster-proportional-autoscaler@sha256:c849d75d61943ce7f51b4c049f1a79d19a08253966c8f49c4cfb6414cc33db8b mcr.microsoft.com/oss/kubernetes/autoscaler/cluster-proportional-autoscaler:1.8.5],SizeBytes:40661903,},ContainerImage{Names:[mcr.microsoft.com/k8s/aad-pod-identity/nmi@sha256:02128fefcdb7593ac53fc342e2c53a0fc6fabd813036bf60457bf43cc2940116 mcr.microsoft.com/k8s/aad-pod-identity/nmi:1.6.1],SizeBytes:38007982,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:921f301c44dda06a325164accf22e78ecc570b5c7d9d6ee4c66bd8cbb2b60b9a mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.16],SizeBytes:26970670,},ContainerImage{Names:[mcr.microsoft.com/k8s/kms/keyvault@sha256:1a27e175f8c125209e32d2957b5509fe20757bd8cb309ff9da598799b56326fb mcr.microsoft.com/k8s/kms/keyvault:v0.0.10],SizeBytes:23077387,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-node-driver-registrar@sha256:dbec3a8166686b09b242176ab5b99e993da4126438bbce68147c3fd654f35662 mcr.microsoft.com/oss/kubernetes-csi/csi-node-driver-registrar:v2.4.0],SizeBytes:19547289,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/livenessprobe@sha256:c96a6255c42766f6b8bb1a7cda02b0060ab1b20b2e2dafcc64ec09e7646745a6 mcr.microsoft.com/oss/kubernetes-csi/livenessprobe:v2.5.0],SizeBytes:17573341,},ContainerImage{Names:[mcr.microsoft.com/oss/busybox/busybox@sha256:582a641242b49809af3a1a522f9aae8c3f047d1c6ca1dd9d8cdabd349e45b1a9 mcr.microsoft.com/oss/busybox/busybox:1.33.1],SizeBytes:1235829,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8],SizeBytes:1159025,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/pause@sha256:e3b8c20681593c21b344ad801fbb8abaf564427ee3a57a9fcfa3b455f917ce46 mcr.microsoft.com/oss/kubernetes/pause:3.4.1],SizeBytes:682696,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/disk.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-u6qgg0s3/providers/Microsoft.Compute/disks/pre-provisioned-inline-volume,DevicePath:,},},Config:nil,},} Jun 18 02:09:51.435: INFO: ... skipping 90 lines ... [90m/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning_test.go:40[0m [single-az] [90m/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning_test.go:44[0m [91m[1mshould receive FailedMount event with invalid mount options [kubernetes.io/azure-disk] [disk.csi.azure.com] [csi-azuredisk-scheduler-extender] [It][0m [90m/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning_test.go:269[0m [91mUnexpected error: <*errors.errorString | 0xc000ab8240>: { s: "PersistentVolume pvc-76934f53-9b56-4add-b7c0-aca144f3145d still exists within 10m0s", } PersistentVolume pvc-76934f53-9b56-4add-b7c0-aca144f3145d still exists within 10m0s occurred[0m ... skipping 15 lines ... [1mSTEP[0m: setting up the StorageClass [1mSTEP[0m: creating a StorageClass [1mSTEP[0m: setting up the PVC and PV [1mSTEP[0m: creating a PVC [1mSTEP[0m: setting up the pod [1mSTEP[0m: deploying the pod [1mSTEP[0m: checking that the pod's command exits with no error Jun 18 02:09:53.045: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-ntt8j" in namespace "azuredisk-290" to be "Succeeded or Failed" Jun 18 02:09:53.083: INFO: Pod "azuredisk-volume-tester-ntt8j": Phase="Pending", Reason="", readiness=false. Elapsed: 37.706548ms Jun 18 02:09:55.118: INFO: Pod "azuredisk-volume-tester-ntt8j": Phase="Pending", Reason="", readiness=false. Elapsed: 2.07290721s Jun 18 02:09:57.153: INFO: Pod "azuredisk-volume-tester-ntt8j": Phase="Pending", Reason="", readiness=false. Elapsed: 4.107748791s Jun 18 02:09:59.187: INFO: Pod "azuredisk-volume-tester-ntt8j": Phase="Pending", Reason="", readiness=false. Elapsed: 6.141947633s Jun 18 02:10:01.222: INFO: Pod "azuredisk-volume-tester-ntt8j": Phase="Pending", Reason="", readiness=false. Elapsed: 8.176934608s Jun 18 02:10:03.257: INFO: Pod "azuredisk-volume-tester-ntt8j": Phase="Pending", Reason="", readiness=false. Elapsed: 10.211603246s ... skipping 432 lines ... Jun 18 02:24:44.540: INFO: Pod "azuredisk-volume-tester-ntt8j": Phase="Pending", Reason="", readiness=false. Elapsed: 14m51.494146239s Jun 18 02:24:46.574: INFO: Pod "azuredisk-volume-tester-ntt8j": Phase="Pending", Reason="", readiness=false. Elapsed: 14m53.528813252s Jun 18 02:24:48.616: INFO: Pod "azuredisk-volume-tester-ntt8j": Phase="Pending", Reason="", readiness=false. Elapsed: 14m55.570852518s Jun 18 02:24:50.650: INFO: Pod "azuredisk-volume-tester-ntt8j": Phase="Pending", Reason="", readiness=false. Elapsed: 14m57.604675187s Jun 18 02:24:52.684: INFO: Pod "azuredisk-volume-tester-ntt8j": Phase="Pending", Reason="", readiness=false. Elapsed: 14m59.638784087s Jun 18 02:24:54.685: INFO: deleting Pod "azuredisk-290"/"azuredisk-volume-tester-ntt8j" Jun 18 02:24:54.749: INFO: Error getting logs for pod azuredisk-volume-tester-ntt8j: the server rejected our request for an unknown reason (get pods azuredisk-volume-tester-ntt8j) [1mSTEP[0m: Deleting pod azuredisk-volume-tester-ntt8j in namespace azuredisk-290 [1mSTEP[0m: validating provisioned PV [1mSTEP[0m: checking the PV Jun 18 02:24:54.851: INFO: deleting PVC "azuredisk-290"/"pvc-znh2k" Jun 18 02:24:54.851: INFO: Deleting PersistentVolumeClaim "pvc-znh2k" [1mSTEP[0m: waiting for claim's PV "pvc-06e3f965-ea1d-453a-b87d-70ca47229942" to be deleted ... skipping 125 lines ... Jun 18 02:34:59.319: INFO: At 2022-06-18 02:09:52 +0000 UTC - event for pvc-znh2k: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding Jun 18 02:34:59.319: INFO: At 2022-06-18 02:09:53 +0000 UTC - event for pvc-znh2k: {disk.csi.azure.com_k8s-agentpool1-42835052-0_53a95263-aec0-4b87-a465-aa36e34998da } Provisioning: External provisioner is provisioning volume for claim "azuredisk-290/pvc-znh2k" Jun 18 02:34:59.319: INFO: At 2022-06-18 02:09:53 +0000 UTC - event for pvc-znh2k: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "disk.csi.azure.com" or manually created by system administrator Jun 18 02:34:59.319: INFO: At 2022-06-18 02:09:55 +0000 UTC - event for pvc-znh2k: {disk.csi.azure.com_k8s-agentpool1-42835052-0_53a95263-aec0-4b87-a465-aa36e34998da } ProvisioningSucceeded: Successfully provisioned volume pvc-06e3f965-ea1d-453a-b87d-70ca47229942 Jun 18 02:34:59.319: INFO: At 2022-06-18 02:09:57 +0000 UTC - event for azuredisk-volume-tester-ntt8j: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-06e3f965-ea1d-453a-b87d-70ca47229942" Jun 18 02:34:59.319: INFO: At 2022-06-18 02:11:59 +0000 UTC - event for azuredisk-volume-tester-ntt8j: {kubelet k8s-agentpool1-42835052-1} FailedMount: Unable to attach or mount volumes: unmounted volumes=[test-volume-1], unattached volumes=[kube-api-access-rd2xs test-volume-1]: timed out waiting for the condition Jun 18 02:34:59.319: INFO: At 2022-06-18 02:12:02 +0000 UTC - event for azuredisk-volume-tester-ntt8j: {kubelet k8s-agentpool1-42835052-1} FailedMapVolume: MapVolume.SetUpDevice failed for volume "pvc-06e3f965-ea1d-453a-b87d-70ca47229942" : rpc error: code = DeadlineExceeded desc = context deadline exceeded Jun 18 02:34:59.319: INFO: At 2022-06-18 02:18:06 +0000 UTC - event for azuredisk-volume-tester-ntt8j: {kubelet k8s-agentpool1-42835052-1} FailedMapVolume: MapVolume.SetUpDevice failed for volume "pvc-06e3f965-ea1d-453a-b87d-70ca47229942" : rpc error: code = Internal desc = failed to wait for volume (/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-u6qgg0s3/providers/Microsoft.Compute/disks/pvc-06e3f965-ea1d-453a-b87d-70ca47229942) to be attached to node (k8s-agentpool1-42835052-1): context deadline exceeded Jun 18 02:34:59.352: INFO: POD NODE PHASE GRACE CONDITIONS Jun 18 02:34:59.352: INFO: Jun 18 02:34:59.418: INFO: Logging node info for node k8s-agentpool1-42835052-0 Jun 18 02:34:59.452: INFO: Node Info: &Node{ObjectMeta:{k8s-agentpool1-42835052-0 b3450a63-c373-4a96-b548-c74917f39933 18725 0 2022-06-18 01:05:58 +0000 UTC <nil> <nil> map[agentpool:agentpool1 beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_DS2_v2 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:canadacentral failure-domain.beta.kubernetes.io/zone:0 kubernetes.azure.com/cluster:kubetest-u6qgg0s3 kubernetes.azure.com/role:agent kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-agentpool1-42835052-0 kubernetes.io/os:linux kubernetes.io/role:agent node-role.kubernetes.io/agent: node.kubernetes.io/instance-type:Standard_DS2_v2 storageprofile:managed storagetier:Premium_LRS topology.disk.csi.azure.com/zone: topology.kubernetes.io/region:canadacentral topology.kubernetes.io/zone:0] map[csi.volume.kubernetes.io/nodeid:{"disk.csi.azure.com":"k8s-agentpool1-42835052-0"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2022-06-18 01:05:58 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:agentpool":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.azure.com/cluster":{},"f:kubernetes.azure.com/role":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:storageprofile":{},"f:storagetier":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kube-controller-manager Update v1 2022-06-18 01:06:07 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"10.244.2.0/24\"":{}}}} } {kubectl-label Update v1 2022-06-18 01:06:53 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/agent":{}}}} } {kubelet Update v1 2022-06-18 02:01:49 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.disk.csi.azure.com/zone":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status} {kube-controller-manager Update v1 2022-06-18 02:11:22 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:volumesAttached":{}}} status}]},Spec:NodeSpec{PodCIDR:10.244.2.0/24,DoNotUseExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-u6qgg0s3/providers/Microsoft.Compute/virtualMachines/k8s-agentpool1-42835052-0,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[10.244.2.0/24],},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-azure-disk: {{8 0} {<nil>} 8 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{31036686336 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7287771136 0} {<nil>} 7116964Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-azure-disk: {{8 0} {<nil>} 8 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{27933017657 0} {<nil>} 27933017657 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{6501339136 0} {<nil>} 6348964Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-06-18 01:06:26 +0000 UTC,LastTransitionTime:2022-06-18 01:06:26 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-06-18 02:32:30 +0000 UTC,LastTransitionTime:2022-06-18 01:05:58 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-06-18 02:32:30 +0000 UTC,LastTransitionTime:2022-06-18 01:05:58 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-06-18 02:32:30 +0000 UTC,LastTransitionTime:2022-06-18 01:05:58 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-06-18 02:32:30 +0000 UTC,LastTransitionTime:2022-06-18 01:06:06 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:Hostname,Address:k8s-agentpool1-42835052-0,},NodeAddress{Type:InternalIP,Address:10.240.0.5,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:fd2eb91f0eeb436286bc238b58e4dc04,SystemUUID:c3ccb155-829c-6a40-8fdc-f652683a9f79,BootID:0082daec-2a09-4d8b-b449-5299bd7dc135,KernelVersion:5.4.0-1074-azure,OSImage:Ubuntu 18.04.6 LTS,ContainerRuntimeVersion:docker://20.10.11+azure-3,KubeletVersion:v1.23.7,KubeProxyVersion:v1.23.7,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/azuredisk-csi@sha256:423eb6cf602c064c8b2deefead5ceadd6324ed41b3d995dab5d0f6f0f4d4710f mcr.microsoft.com/oss/kubernetes-csi/azuredisk-csi:v1.10.0],SizeBytes:245959792,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/azurefile-csi@sha256:9e2ecabcf9dd9943e6600eb9fb460f45b4dc61af7cabe95d115082a029db2aaf mcr.microsoft.com/oss/kubernetes-csi/azurefile-csi:v1.9.0],SizeBytes:230470852,},ContainerImage{Names:[mcr.microsoft.com/containernetworking/azure-npm@sha256:106f669f48e5e80c4ec0afb49858ead72cf4b901cd8664e7bf81f8d789e56e12 mcr.microsoft.com/containernetworking/azure-npm:v1.2.2_hotfix],SizeBytes:175230380,},ContainerImage{Names:[k8sprow.azurecr.io/azuredisk-csi@sha256:7a2395a48c394c3f4eae282f9d9873df45a18e8a409ccddf03b7583367b0a7da k8sprow.azurecr.io/azuredisk-csi:latest-v2-444ca6fc5bce97ea7c52380569d0659c5e105ce7],SizeBytes:136453655,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/secrets-store/driver@sha256:c0d040a1c4fbfceb65663e31c09ea40f4f78e356437610cbc3fbb4bb409bd6f1 mcr.microsoft.com/oss/kubernetes-csi/secrets-store/driver:v0.0.19],SizeBytes:123229697,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-proxy@sha256:0c890e918d42c158fab68867bade2ab1db04c3408452db9611a264ba3e43e6b3 mcr.microsoft.com/oss/kubernetes/kube-proxy:v1.23.7],SizeBytes:112296063,},ContainerImage{Names:[mcr.microsoft.com/oss/azure/secrets-store/provider-azure@sha256:6f67f3d0c7cdde5702f8ce7f101b6519daa0237f0c34fecb7c058b6af8c22ad1 mcr.microsoft.com/oss/azure/secrets-store/provider-azure:0.0.12],SizeBytes:101061355,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/autoscaler/cluster-autoscaler@sha256:6f0c680d375c62e74351f8ff3ed6ddb9b72ca759e0645c329b95f64264654a6d mcr.microsoft.com/oss/kubernetes/autoscaler/cluster-autoscaler:v1.22.1],SizeBytes:99962810,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-addon-manager@sha256:32e2836018c96e73533bd4642fe438e465b81dcbfa8b7b61935a6f4d0246c7ae mcr.microsoft.com/oss/kubernetes/kube-addon-manager:v9.1.3],SizeBytes:86832059,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-addon-manager@sha256:92c2c5aad9012ee32d2a43a74966cc0adc6ccb1705ad15abb10485ecf406d88b mcr.microsoft.com/oss/kubernetes/kube-addon-manager:v9.1.5],SizeBytes:84094027,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/metrics-server@sha256:1ef9d57ce41ffcc328b92494c3bfafe401e0b9a1694a295301a1385337d52815 mcr.microsoft.com/oss/kubernetes/metrics-server:v0.5.2],SizeBytes:64327621,},ContainerImage{Names:[mcr.microsoft.com/oss/nvidia/k8s-device-plugin@sha256:0f5b52bf28239234e831697d96db63ac03cde70fe68058f964504ab7564ee810 mcr.microsoft.com/oss/nvidia/k8s-device-plugin:1.0.0-beta6],SizeBytes:64160241,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-provisioner@sha256:542c9258a1441e4927883ea73425e21f9a14043a649686bf6b51700f34c64f8e mcr.microsoft.com/oss/kubernetes-csi/csi-provisioner:v3.1.0],SizeBytes:57858438,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-provisioner@sha256:e9ddadc44ba87a4a27f67e54760a14f9986885b534b3dff170a14eae1e35d213 mcr.microsoft.com/oss/kubernetes-csi/csi-provisioner:v3.0.0],SizeBytes:56881280,},ContainerImage{Names:[k8sprow.azurecr.io/azdiskschedulerextender-csi@sha256:295239e73ff3a728379df39f3985db8347e1547ecb2658620f29f3fdc1058d34 k8sprow.azurecr.io/azdiskschedulerextender-csi:latest-v2-444ca6fc5bce97ea7c52380569d0659c5e105ce7],SizeBytes:56760067,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-resizer@sha256:ab6c2e18a4d943f2bf8688e1779622277a1d14c355c9d4bfc0d761d86c5108b3 mcr.microsoft.com/oss/kubernetes-csi/csi-resizer:v1.4.0],SizeBytes:55577322,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-snapshotter@sha256:a889e925e15f9423f7842f1b769f64cbcf6a20b6956122836fc835cf22d9073f mcr.microsoft.com/oss/kubernetes-csi/csi-snapshotter:v5.0.1],SizeBytes:55314459,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-attacher@sha256:19fbca01394f7ed80151731180cfde0a3367f038beae2a97cda7928034010057 mcr.microsoft.com/oss/kubernetes-csi/csi-attacher:v3.4.0],SizeBytes:54954134,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-resizer@sha256:c5bb71ceaac60b1a4b58739fa07b709f6248c452ff6272a384d2f7648895a750 mcr.microsoft.com/oss/kubernetes-csi/csi-resizer:v1.3.0],SizeBytes:54313772,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-snapshotter@sha256:61849a026511cf332c87d73d0a7aed803b510c3ede197ec755389686d490de72 mcr.microsoft.com/oss/kubernetes-csi/csi-snapshotter:v4.2.1],SizeBytes:54210936,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-attacher@sha256:6b41e7153ebdfdc1501aa65184624bc15fd33a52d93f88ec3a758d0f8c9b8c10 mcr.microsoft.com/oss/kubernetes-csi/csi-attacher:v3.3.0],SizeBytes:53842561,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/snapshot-controller@sha256:8c3fc3c2667004ad6bbdf723bb64c5da66a5cb8b11d4ee59b67179b686223b13 mcr.microsoft.com/oss/kubernetes-csi/snapshot-controller:v5.0.1],SizeBytes:52732401,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/snapshot-controller@sha256:be5a8dc1f990828f653c77bd0a0f1bbd13197c3019f6e1d99d590389bac36705 mcr.microsoft.com/oss/kubernetes-csi/snapshot-controller:v4.2.1],SizeBytes:51575245,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager@sha256:0ad67f9919522a07318034641ae09bf2079b417e9944c65914410594ce645468 mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager:v1.1.4],SizeBytes:51478397,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager@sha256:31ec4f7daccd3e7a8504e12657d7830651ecacbe4a487daca1b1b7695a64b070 mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager:v1.23.1],SizeBytes:51249021,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager@sha256:011712ed90fb8efcf27928b0a47ed04b98baebb31cb1b2d8ab676977ec18eedc mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager:v1.1.4],SizeBytes:50868093,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager@sha256:0f9a8fbaed65192ed7dd795be4f9c1dc48ebdef0a241fb62d456f4bed40d9875 mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager:v1.23.1],SizeBytes:50679677,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-scheduler@sha256:c35f45e8dfaa30363e589296415a057f9d32813920fc3820d6557491657f199a mcr.microsoft.com/oss/kubernetes/kube-scheduler:v1.21.2],SizeBytes:50607491,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/ip-masq-agent@sha256:1244155f2ed3f33ff154cc343b8ad285f3391d95afd7d4b1c6dcc420bc0ba3cf mcr.microsoft.com/oss/kubernetes/ip-masq-agent:v2.5.0],SizeBytes:50146762,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager@sha256:7c907ff70b90a0bdf8fae63bd744018469dd9839cde1fd0515b93e0bbd14b34e mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager:v1.0.8],SizeBytes:48963453,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager@sha256:3987d7a8c6922ce1952ee19c5cb6ea75aac7b7c1b07aa79277ad038c69fb7a31 mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager:v1.0.8],SizeBytes:48349053,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/coredns@sha256:f873bf7f0928461efe10697fa76cf0ad7a1ae3041c5b57b50dd3d0b72d273f8c mcr.microsoft.com/oss/kubernetes/coredns:1.8.6],SizeBytes:46804601,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager@sha256:8073113a20882642a980b338635cdc5945e5673a18aef192090e6fde2b89a75c mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager:v0.6.0],SizeBytes:45909032,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager@sha256:6a32329628bdea3c6d75e98aad6155b65d2e2b98ca616eb33f9ac562912804c6 mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager:v0.6.0],SizeBytes:45229096,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager@sha256:ef6c4ba564b4d11d270f7d1563c50fbeb30ccc3b94146e5059228c49f95875f5 mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager:v0.7.11],SizeBytes:44916605,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager@sha256:dbcc384758ba5ca6d249596d471292ed3785e31cdb854d48b84d70794b669b4c mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager:v0.7.11],SizeBytes:43679613,},ContainerImage{Names:[mcr.microsoft.com/oss/etcd-io/etcd@sha256:cf587862e3f1b6fa4d9a2565520a34f164bdf72c50f37af8c3c668160593246e mcr.microsoft.com/oss/etcd-io/etcd:v3.3.25],SizeBytes:41832119,},ContainerImage{Names:[mcr.microsoft.com/k8s/aad-pod-identity/mic@sha256:bd9465be94966b9a917e1e3904fa5e63dd91772ccadf304e18ffd8e4ad8ccedd mcr.microsoft.com/k8s/aad-pod-identity/mic:1.6.1],SizeBytes:41374894,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/autoscaler/cluster-proportional-autoscaler@sha256:c849d75d61943ce7f51b4c049f1a79d19a08253966c8f49c4cfb6414cc33db8b mcr.microsoft.com/oss/kubernetes/autoscaler/cluster-proportional-autoscaler:1.8.5],SizeBytes:40661903,},ContainerImage{Names:[mcr.microsoft.com/k8s/aad-pod-identity/nmi@sha256:02128fefcdb7593ac53fc342e2c53a0fc6fabd813036bf60457bf43cc2940116 mcr.microsoft.com/k8s/aad-pod-identity/nmi:1.6.1],SizeBytes:38007982,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:921f301c44dda06a325164accf22e78ecc570b5c7d9d6ee4c66bd8cbb2b60b9a mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.16],SizeBytes:26970670,},ContainerImage{Names:[mcr.microsoft.com/k8s/kms/keyvault@sha256:1a27e175f8c125209e32d2957b5509fe20757bd8cb309ff9da598799b56326fb mcr.microsoft.com/k8s/kms/keyvault:v0.0.10],SizeBytes:23077387,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-node-driver-registrar@sha256:dbec3a8166686b09b242176ab5b99e993da4126438bbce68147c3fd654f35662 mcr.microsoft.com/oss/kubernetes-csi/csi-node-driver-registrar:v2.4.0],SizeBytes:19547289,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/livenessprobe@sha256:c96a6255c42766f6b8bb1a7cda02b0060ab1b20b2e2dafcc64ec09e7646745a6 mcr.microsoft.com/oss/kubernetes-csi/livenessprobe:v2.5.0],SizeBytes:17573341,},ContainerImage{Names:[mcr.microsoft.com/oss/busybox/busybox@sha256:582a641242b49809af3a1a522f9aae8c3f047d1c6ca1dd9d8cdabd349e45b1a9 mcr.microsoft.com/oss/busybox/busybox:1.33.1],SizeBytes:1235829,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8],SizeBytes:1159025,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/pause@sha256:e3b8c20681593c21b344ad801fbb8abaf564427ee3a57a9fcfa3b455f917ce46 mcr.microsoft.com/oss/kubernetes/pause:3.4.1],SizeBytes:682696,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/disk.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-u6qgg0s3/providers/Microsoft.Compute/disks/pre-provisioned-inline-volume,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/disk.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-u6qgg0s3/providers/Microsoft.Compute/disks/pvc-76934f53-9b56-4add-b7c0-aca144f3145d,DevicePath:,},},Config:nil,},} Jun 18 02:34:59.453: INFO: ... skipping 90 lines ... [90m/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning_test.go:40[0m [single-az] [90m/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning_test.go:44[0m [91m[1mshould create a raw block volume on demand [kubernetes.io/azure-disk] [disk.csi.azure.com] [csi-azuredisk-scheduler-extender] [It][0m [90m/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning_test.go:306[0m [91mUnexpected error: <*errors.errorString | 0xc0011364e0>: { s: "Gave up after waiting 15m0s for pod \"azuredisk-volume-tester-ntt8j\" to be \"Succeeded or Failed\"", } Gave up after waiting 15m0s for pod "azuredisk-volume-tester-ntt8j" to be "Succeeded or Failed" occurred[0m /home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/resources/pod.go:89 [90m------------------------------[0m [0mDynamic Provisioning[0m [90m[single-az][0m [1mshould create a volume on demand and mount it as readOnly in a pod [kubernetes.io/azure-disk] [disk.csi.azure.com] [Windows] [csi-azuredisk-scheduler-extender][0m ... skipping 11 lines ... [1mSTEP[0m: setting up the StorageClass [1mSTEP[0m: creating a StorageClass [1mSTEP[0m: setting up the PVC and PV [1mSTEP[0m: creating a PVC [1mSTEP[0m: setting up the pod [1mSTEP[0m: deploying the pod [1mSTEP[0m: checking that the pod's command exits with an error Jun 18 02:35:01.068: INFO: Waiting up to 10m0s for pod "azuredisk-volume-tester-rjk2v" in namespace "azuredisk-8203" to be "Error status code" Jun 18 02:35:01.101: INFO: Pod "azuredisk-volume-tester-rjk2v": Phase="Pending", Reason="", readiness=false. Elapsed: 33.112951ms Jun 18 02:35:03.137: INFO: Pod "azuredisk-volume-tester-rjk2v": Phase="Pending", Reason="", readiness=false. Elapsed: 2.069111351s Jun 18 02:35:05.172: INFO: Pod "azuredisk-volume-tester-rjk2v": Phase="Pending", Reason="", readiness=false. Elapsed: 4.103975848s Jun 18 02:35:07.207: INFO: Pod "azuredisk-volume-tester-rjk2v": Phase="Pending", Reason="", readiness=false. Elapsed: 6.138255165s Jun 18 02:35:09.242: INFO: Pod "azuredisk-volume-tester-rjk2v": Phase="Pending", Reason="", readiness=false. Elapsed: 8.173522974s Jun 18 02:35:11.277: INFO: Pod "azuredisk-volume-tester-rjk2v": Phase="Pending", Reason="", readiness=false. Elapsed: 10.20907324s ... skipping 284 lines ... Jun 18 02:44:51.268: INFO: Pod "azuredisk-volume-tester-rjk2v": Phase="Pending", Reason="", readiness=false. Elapsed: 9m50.199850762s Jun 18 02:44:53.304: INFO: Pod "azuredisk-volume-tester-rjk2v": Phase="Pending", Reason="", readiness=false. Elapsed: 9m52.235516892s Jun 18 02:44:55.339: INFO: Pod "azuredisk-volume-tester-rjk2v": Phase="Pending", Reason="", readiness=false. Elapsed: 9m54.271094036s Jun 18 02:44:57.374: INFO: Pod "azuredisk-volume-tester-rjk2v": Phase="Pending", Reason="", readiness=false. Elapsed: 9m56.30547551s Jun 18 02:44:59.408: INFO: Pod "azuredisk-volume-tester-rjk2v": Phase="Pending", Reason="", readiness=false. Elapsed: 9m58.339933937s Jun 18 02:45:01.412: INFO: deleting Pod "azuredisk-8203"/"azuredisk-volume-tester-rjk2v" Jun 18 02:45:01.476: INFO: Error getting logs for pod azuredisk-volume-tester-rjk2v: the server rejected our request for an unknown reason (get pods azuredisk-volume-tester-rjk2v) [1mSTEP[0m: Deleting pod azuredisk-volume-tester-rjk2v in namespace azuredisk-8203 [1mSTEP[0m: validating provisioned PV [1mSTEP[0m: checking the PV Jun 18 02:45:01.579: INFO: deleting PVC "azuredisk-8203"/"pvc-fk4fq" Jun 18 02:45:01.579: INFO: Deleting PersistentVolumeClaim "pvc-fk4fq" [1mSTEP[0m: waiting for claim's PV "pvc-209e49e4-921f-4216-8dcf-c99d5ad92944" to be deleted ... skipping 125 lines ... Jun 18 02:55:06.028: INFO: At 2022-06-18 02:35:01 +0000 UTC - event for pvc-fk4fq: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding Jun 18 02:55:06.028: INFO: At 2022-06-18 02:35:01 +0000 UTC - event for pvc-fk4fq: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "disk.csi.azure.com" or manually created by system administrator Jun 18 02:55:06.028: INFO: At 2022-06-18 02:35:01 +0000 UTC - event for pvc-fk4fq: {disk.csi.azure.com_k8s-agentpool1-42835052-0_53a95263-aec0-4b87-a465-aa36e34998da } Provisioning: External provisioner is provisioning volume for claim "azuredisk-8203/pvc-fk4fq" Jun 18 02:55:06.028: INFO: At 2022-06-18 02:35:03 +0000 UTC - event for pvc-fk4fq: {disk.csi.azure.com_k8s-agentpool1-42835052-0_53a95263-aec0-4b87-a465-aa36e34998da } ProvisioningSucceeded: Successfully provisioned volume pvc-209e49e4-921f-4216-8dcf-c99d5ad92944 Jun 18 02:55:06.028: INFO: At 2022-06-18 02:35:05 +0000 UTC - event for azuredisk-volume-tester-rjk2v: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-209e49e4-921f-4216-8dcf-c99d5ad92944" Jun 18 02:55:06.028: INFO: At 2022-06-18 02:37:07 +0000 UTC - event for azuredisk-volume-tester-rjk2v: {kubelet k8s-agentpool1-42835052-0} FailedMount: Unable to attach or mount volumes: unmounted volumes=[test-volume-1], unattached volumes=[test-volume-1 kube-api-access-x4lng]: timed out waiting for the condition Jun 18 02:55:06.028: INFO: At 2022-06-18 02:37:14 +0000 UTC - event for azuredisk-volume-tester-rjk2v: {kubelet k8s-agentpool1-42835052-0} FailedMount: MountVolume.MountDevice failed for volume "pvc-209e49e4-921f-4216-8dcf-c99d5ad92944" : rpc error: code = DeadlineExceeded desc = context deadline exceeded Jun 18 02:55:06.061: INFO: POD NODE PHASE GRACE CONDITIONS Jun 18 02:55:06.061: INFO: Jun 18 02:55:06.127: INFO: Logging node info for node k8s-agentpool1-42835052-0 Jun 18 02:55:06.160: INFO: Node Info: &Node{ObjectMeta:{k8s-agentpool1-42835052-0 b3450a63-c373-4a96-b548-c74917f39933 23207 0 2022-06-18 01:05:58 +0000 UTC <nil> <nil> map[agentpool:agentpool1 beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_DS2_v2 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:canadacentral failure-domain.beta.kubernetes.io/zone:0 kubernetes.azure.com/cluster:kubetest-u6qgg0s3 kubernetes.azure.com/role:agent kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-agentpool1-42835052-0 kubernetes.io/os:linux kubernetes.io/role:agent node-role.kubernetes.io/agent: node.kubernetes.io/instance-type:Standard_DS2_v2 storageprofile:managed storagetier:Premium_LRS topology.disk.csi.azure.com/zone: topology.kubernetes.io/region:canadacentral topology.kubernetes.io/zone:0] map[csi.volume.kubernetes.io/nodeid:{"disk.csi.azure.com":"k8s-agentpool1-42835052-0"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2022-06-18 01:05:58 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:agentpool":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.azure.com/cluster":{},"f:kubernetes.azure.com/role":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:storageprofile":{},"f:storagetier":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kube-controller-manager Update v1 2022-06-18 01:06:07 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"10.244.2.0/24\"":{}}}} } {kubectl-label Update v1 2022-06-18 01:06:53 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/agent":{}}}} } {kube-controller-manager Update v1 2022-06-18 02:11:22 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:volumesAttached":{}}} status} {kubelet Update v1 2022-06-18 02:47:40 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.disk.csi.azure.com/zone":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:10.244.2.0/24,DoNotUseExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-u6qgg0s3/providers/Microsoft.Compute/virtualMachines/k8s-agentpool1-42835052-0,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[10.244.2.0/24],},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-azure-disk: {{8 0} {<nil>} 8 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{31036686336 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7287771136 0} {<nil>} 7116964Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-azure-disk: {{8 0} {<nil>} 8 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{27933017657 0} {<nil>} 27933017657 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{6501339136 0} {<nil>} 6348964Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-06-18 01:06:26 +0000 UTC,LastTransitionTime:2022-06-18 01:06:26 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-06-18 02:52:45 +0000 UTC,LastTransitionTime:2022-06-18 01:05:58 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-06-18 02:52:45 +0000 UTC,LastTransitionTime:2022-06-18 01:05:58 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-06-18 02:52:45 +0000 UTC,LastTransitionTime:2022-06-18 01:05:58 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-06-18 02:52:45 +0000 UTC,LastTransitionTime:2022-06-18 01:06:06 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:Hostname,Address:k8s-agentpool1-42835052-0,},NodeAddress{Type:InternalIP,Address:10.240.0.5,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:fd2eb91f0eeb436286bc238b58e4dc04,SystemUUID:c3ccb155-829c-6a40-8fdc-f652683a9f79,BootID:0082daec-2a09-4d8b-b449-5299bd7dc135,KernelVersion:5.4.0-1074-azure,OSImage:Ubuntu 18.04.6 LTS,ContainerRuntimeVersion:docker://20.10.11+azure-3,KubeletVersion:v1.23.7,KubeProxyVersion:v1.23.7,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/azuredisk-csi@sha256:423eb6cf602c064c8b2deefead5ceadd6324ed41b3d995dab5d0f6f0f4d4710f mcr.microsoft.com/oss/kubernetes-csi/azuredisk-csi:v1.10.0],SizeBytes:245959792,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/azurefile-csi@sha256:9e2ecabcf9dd9943e6600eb9fb460f45b4dc61af7cabe95d115082a029db2aaf mcr.microsoft.com/oss/kubernetes-csi/azurefile-csi:v1.9.0],SizeBytes:230470852,},ContainerImage{Names:[mcr.microsoft.com/containernetworking/azure-npm@sha256:106f669f48e5e80c4ec0afb49858ead72cf4b901cd8664e7bf81f8d789e56e12 mcr.microsoft.com/containernetworking/azure-npm:v1.2.2_hotfix],SizeBytes:175230380,},ContainerImage{Names:[k8sprow.azurecr.io/azuredisk-csi@sha256:7a2395a48c394c3f4eae282f9d9873df45a18e8a409ccddf03b7583367b0a7da k8sprow.azurecr.io/azuredisk-csi:latest-v2-444ca6fc5bce97ea7c52380569d0659c5e105ce7],SizeBytes:136453655,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/secrets-store/driver@sha256:c0d040a1c4fbfceb65663e31c09ea40f4f78e356437610cbc3fbb4bb409bd6f1 mcr.microsoft.com/oss/kubernetes-csi/secrets-store/driver:v0.0.19],SizeBytes:123229697,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-proxy@sha256:0c890e918d42c158fab68867bade2ab1db04c3408452db9611a264ba3e43e6b3 mcr.microsoft.com/oss/kubernetes/kube-proxy:v1.23.7],SizeBytes:112296063,},ContainerImage{Names:[mcr.microsoft.com/oss/azure/secrets-store/provider-azure@sha256:6f67f3d0c7cdde5702f8ce7f101b6519daa0237f0c34fecb7c058b6af8c22ad1 mcr.microsoft.com/oss/azure/secrets-store/provider-azure:0.0.12],SizeBytes:101061355,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/autoscaler/cluster-autoscaler@sha256:6f0c680d375c62e74351f8ff3ed6ddb9b72ca759e0645c329b95f64264654a6d mcr.microsoft.com/oss/kubernetes/autoscaler/cluster-autoscaler:v1.22.1],SizeBytes:99962810,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-addon-manager@sha256:32e2836018c96e73533bd4642fe438e465b81dcbfa8b7b61935a6f4d0246c7ae mcr.microsoft.com/oss/kubernetes/kube-addon-manager:v9.1.3],SizeBytes:86832059,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-addon-manager@sha256:92c2c5aad9012ee32d2a43a74966cc0adc6ccb1705ad15abb10485ecf406d88b mcr.microsoft.com/oss/kubernetes/kube-addon-manager:v9.1.5],SizeBytes:84094027,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/metrics-server@sha256:1ef9d57ce41ffcc328b92494c3bfafe401e0b9a1694a295301a1385337d52815 mcr.microsoft.com/oss/kubernetes/metrics-server:v0.5.2],SizeBytes:64327621,},ContainerImage{Names:[mcr.microsoft.com/oss/nvidia/k8s-device-plugin@sha256:0f5b52bf28239234e831697d96db63ac03cde70fe68058f964504ab7564ee810 mcr.microsoft.com/oss/nvidia/k8s-device-plugin:1.0.0-beta6],SizeBytes:64160241,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-provisioner@sha256:542c9258a1441e4927883ea73425e21f9a14043a649686bf6b51700f34c64f8e mcr.microsoft.com/oss/kubernetes-csi/csi-provisioner:v3.1.0],SizeBytes:57858438,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-provisioner@sha256:e9ddadc44ba87a4a27f67e54760a14f9986885b534b3dff170a14eae1e35d213 mcr.microsoft.com/oss/kubernetes-csi/csi-provisioner:v3.0.0],SizeBytes:56881280,},ContainerImage{Names:[k8sprow.azurecr.io/azdiskschedulerextender-csi@sha256:295239e73ff3a728379df39f3985db8347e1547ecb2658620f29f3fdc1058d34 k8sprow.azurecr.io/azdiskschedulerextender-csi:latest-v2-444ca6fc5bce97ea7c52380569d0659c5e105ce7],SizeBytes:56760067,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-resizer@sha256:ab6c2e18a4d943f2bf8688e1779622277a1d14c355c9d4bfc0d761d86c5108b3 mcr.microsoft.com/oss/kubernetes-csi/csi-resizer:v1.4.0],SizeBytes:55577322,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-snapshotter@sha256:a889e925e15f9423f7842f1b769f64cbcf6a20b6956122836fc835cf22d9073f mcr.microsoft.com/oss/kubernetes-csi/csi-snapshotter:v5.0.1],SizeBytes:55314459,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-attacher@sha256:19fbca01394f7ed80151731180cfde0a3367f038beae2a97cda7928034010057 mcr.microsoft.com/oss/kubernetes-csi/csi-attacher:v3.4.0],SizeBytes:54954134,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-resizer@sha256:c5bb71ceaac60b1a4b58739fa07b709f6248c452ff6272a384d2f7648895a750 mcr.microsoft.com/oss/kubernetes-csi/csi-resizer:v1.3.0],SizeBytes:54313772,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-snapshotter@sha256:61849a026511cf332c87d73d0a7aed803b510c3ede197ec755389686d490de72 mcr.microsoft.com/oss/kubernetes-csi/csi-snapshotter:v4.2.1],SizeBytes:54210936,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-attacher@sha256:6b41e7153ebdfdc1501aa65184624bc15fd33a52d93f88ec3a758d0f8c9b8c10 mcr.microsoft.com/oss/kubernetes-csi/csi-attacher:v3.3.0],SizeBytes:53842561,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/snapshot-controller@sha256:8c3fc3c2667004ad6bbdf723bb64c5da66a5cb8b11d4ee59b67179b686223b13 mcr.microsoft.com/oss/kubernetes-csi/snapshot-controller:v5.0.1],SizeBytes:52732401,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/snapshot-controller@sha256:be5a8dc1f990828f653c77bd0a0f1bbd13197c3019f6e1d99d590389bac36705 mcr.microsoft.com/oss/kubernetes-csi/snapshot-controller:v4.2.1],SizeBytes:51575245,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager@sha256:0ad67f9919522a07318034641ae09bf2079b417e9944c65914410594ce645468 mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager:v1.1.4],SizeBytes:51478397,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager@sha256:31ec4f7daccd3e7a8504e12657d7830651ecacbe4a487daca1b1b7695a64b070 mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager:v1.23.1],SizeBytes:51249021,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager@sha256:011712ed90fb8efcf27928b0a47ed04b98baebb31cb1b2d8ab676977ec18eedc mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager:v1.1.4],SizeBytes:50868093,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager@sha256:0f9a8fbaed65192ed7dd795be4f9c1dc48ebdef0a241fb62d456f4bed40d9875 mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager:v1.23.1],SizeBytes:50679677,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-scheduler@sha256:c35f45e8dfaa30363e589296415a057f9d32813920fc3820d6557491657f199a mcr.microsoft.com/oss/kubernetes/kube-scheduler:v1.21.2],SizeBytes:50607491,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/ip-masq-agent@sha256:1244155f2ed3f33ff154cc343b8ad285f3391d95afd7d4b1c6dcc420bc0ba3cf mcr.microsoft.com/oss/kubernetes/ip-masq-agent:v2.5.0],SizeBytes:50146762,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager@sha256:7c907ff70b90a0bdf8fae63bd744018469dd9839cde1fd0515b93e0bbd14b34e mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager:v1.0.8],SizeBytes:48963453,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager@sha256:3987d7a8c6922ce1952ee19c5cb6ea75aac7b7c1b07aa79277ad038c69fb7a31 mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager:v1.0.8],SizeBytes:48349053,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/coredns@sha256:f873bf7f0928461efe10697fa76cf0ad7a1ae3041c5b57b50dd3d0b72d273f8c mcr.microsoft.com/oss/kubernetes/coredns:1.8.6],SizeBytes:46804601,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager@sha256:8073113a20882642a980b338635cdc5945e5673a18aef192090e6fde2b89a75c mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager:v0.6.0],SizeBytes:45909032,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager@sha256:6a32329628bdea3c6d75e98aad6155b65d2e2b98ca616eb33f9ac562912804c6 mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager:v0.6.0],SizeBytes:45229096,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager@sha256:ef6c4ba564b4d11d270f7d1563c50fbeb30ccc3b94146e5059228c49f95875f5 mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager:v0.7.11],SizeBytes:44916605,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager@sha256:dbcc384758ba5ca6d249596d471292ed3785e31cdb854d48b84d70794b669b4c mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager:v0.7.11],SizeBytes:43679613,},ContainerImage{Names:[mcr.microsoft.com/oss/etcd-io/etcd@sha256:cf587862e3f1b6fa4d9a2565520a34f164bdf72c50f37af8c3c668160593246e mcr.microsoft.com/oss/etcd-io/etcd:v3.3.25],SizeBytes:41832119,},ContainerImage{Names:[mcr.microsoft.com/k8s/aad-pod-identity/mic@sha256:bd9465be94966b9a917e1e3904fa5e63dd91772ccadf304e18ffd8e4ad8ccedd mcr.microsoft.com/k8s/aad-pod-identity/mic:1.6.1],SizeBytes:41374894,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/autoscaler/cluster-proportional-autoscaler@sha256:c849d75d61943ce7f51b4c049f1a79d19a08253966c8f49c4cfb6414cc33db8b mcr.microsoft.com/oss/kubernetes/autoscaler/cluster-proportional-autoscaler:1.8.5],SizeBytes:40661903,},ContainerImage{Names:[mcr.microsoft.com/k8s/aad-pod-identity/nmi@sha256:02128fefcdb7593ac53fc342e2c53a0fc6fabd813036bf60457bf43cc2940116 mcr.microsoft.com/k8s/aad-pod-identity/nmi:1.6.1],SizeBytes:38007982,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:921f301c44dda06a325164accf22e78ecc570b5c7d9d6ee4c66bd8cbb2b60b9a mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.16],SizeBytes:26970670,},ContainerImage{Names:[mcr.microsoft.com/k8s/kms/keyvault@sha256:1a27e175f8c125209e32d2957b5509fe20757bd8cb309ff9da598799b56326fb mcr.microsoft.com/k8s/kms/keyvault:v0.0.10],SizeBytes:23077387,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-node-driver-registrar@sha256:dbec3a8166686b09b242176ab5b99e993da4126438bbce68147c3fd654f35662 mcr.microsoft.com/oss/kubernetes-csi/csi-node-driver-registrar:v2.4.0],SizeBytes:19547289,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/livenessprobe@sha256:c96a6255c42766f6b8bb1a7cda02b0060ab1b20b2e2dafcc64ec09e7646745a6 mcr.microsoft.com/oss/kubernetes-csi/livenessprobe:v2.5.0],SizeBytes:17573341,},ContainerImage{Names:[mcr.microsoft.com/oss/busybox/busybox@sha256:582a641242b49809af3a1a522f9aae8c3f047d1c6ca1dd9d8cdabd349e45b1a9 mcr.microsoft.com/oss/busybox/busybox:1.33.1],SizeBytes:1235829,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8],SizeBytes:1159025,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/pause@sha256:e3b8c20681593c21b344ad801fbb8abaf564427ee3a57a9fcfa3b455f917ce46 mcr.microsoft.com/oss/kubernetes/pause:3.4.1],SizeBytes:682696,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/disk.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-u6qgg0s3/providers/Microsoft.Compute/disks/pre-provisioned-inline-volume,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/disk.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-u6qgg0s3/providers/Microsoft.Compute/disks/pvc-76934f53-9b56-4add-b7c0-aca144f3145d,DevicePath:,},},Config:nil,},} Jun 18 02:55:06.161: INFO: ... skipping 90 lines ... [90m/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning_test.go:40[0m [single-az] [90m/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning_test.go:44[0m [91m[1mshould create a volume on demand and mount it as readOnly in a pod [kubernetes.io/azure-disk] [disk.csi.azure.com] [Windows] [csi-azuredisk-scheduler-extender] [It][0m [90m/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning_test.go:338[0m [91mUnexpected error: <*errors.errorString | 0xc000b4afc0>: { s: "Gave up after waiting 10m0s for pod \"azuredisk-volume-tester-rjk2v\" to be \"Error status code\"", } Gave up after waiting 10m0s for pod "azuredisk-volume-tester-rjk2v" to be "Error status code" occurred[0m /home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/resources/pod.go:114 [90m------------------------------[0m [0mDynamic Provisioning[0m [90m[single-az][0m [1mshould create multiple PV objects, bind to PVCs and attach all to different pods on the same node [kubernetes.io/azure-disk] [disk.csi.azure.com] [Windows] [csi-azuredisk-scheduler-extender][0m ... skipping 13 lines ... [1mSTEP[0m: setting up the PVC and PV [1mSTEP[0m: creating a PVC [1mSTEP[0m: setting up the pod [1mSTEP[0m: deploying the pod [1mSTEP[0m: checking that the pod is running Jun 18 03:10:07.838: INFO: deleting Pod "azuredisk-1615"/"azuredisk-volume-tester-lm97d" Jun 18 03:10:07.910: INFO: Error getting logs for pod azuredisk-volume-tester-lm97d: the server rejected our request for an unknown reason (get pods azuredisk-volume-tester-lm97d) [1mSTEP[0m: Deleting pod azuredisk-volume-tester-lm97d in namespace azuredisk-1615 [1mSTEP[0m: validating provisioned PV [1mSTEP[0m: checking the PV Jun 18 03:10:08.012: INFO: deleting PVC "azuredisk-1615"/"pvc-8bgns" Jun 18 03:10:08.012: INFO: Deleting PersistentVolumeClaim "pvc-8bgns" [1mSTEP[0m: waiting for claim's PV "pvc-7bb233f2-027d-4df5-9c6a-9f55f5fc77f5" to be deleted ... skipping 124 lines ... Jun 18 03:20:12.471: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for azuredisk-volume-tester-lm97d: { } Scheduled: Successfully assigned azuredisk-1615/azuredisk-volume-tester-lm97d to k8s-agentpool1-42835052-1 Jun 18 03:20:12.471: INFO: At 2022-06-18 02:55:07 +0000 UTC - event for pvc-8bgns: {disk.csi.azure.com_k8s-agentpool1-42835052-0_53a95263-aec0-4b87-a465-aa36e34998da } Provisioning: External provisioner is provisioning volume for claim "azuredisk-1615/pvc-8bgns" Jun 18 03:20:12.471: INFO: At 2022-06-18 02:55:07 +0000 UTC - event for pvc-8bgns: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding Jun 18 03:20:12.471: INFO: At 2022-06-18 02:55:07 +0000 UTC - event for pvc-8bgns: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "disk.csi.azure.com" or manually created by system administrator Jun 18 03:20:12.471: INFO: At 2022-06-18 02:55:10 +0000 UTC - event for pvc-8bgns: {disk.csi.azure.com_k8s-agentpool1-42835052-0_53a95263-aec0-4b87-a465-aa36e34998da } ProvisioningSucceeded: Successfully provisioned volume pvc-7bb233f2-027d-4df5-9c6a-9f55f5fc77f5 Jun 18 03:20:12.471: INFO: At 2022-06-18 02:55:12 +0000 UTC - event for azuredisk-volume-tester-lm97d: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-7bb233f2-027d-4df5-9c6a-9f55f5fc77f5" Jun 18 03:20:12.471: INFO: At 2022-06-18 02:57:12 +0000 UTC - event for azuredisk-volume-tester-lm97d: {kubelet k8s-agentpool1-42835052-1} FailedMount: MountVolume.MountDevice failed for volume "pvc-7bb233f2-027d-4df5-9c6a-9f55f5fc77f5" : rpc error: code = DeadlineExceeded desc = context deadline exceeded Jun 18 03:20:12.471: INFO: At 2022-06-18 02:57:13 +0000 UTC - event for azuredisk-volume-tester-lm97d: {kubelet k8s-agentpool1-42835052-1} FailedMount: Unable to attach or mount volumes: unmounted volumes=[test-volume-1], unattached volumes=[test-volume-1 kube-api-access-kjtzp]: timed out waiting for the condition Jun 18 03:20:12.471: INFO: At 2022-06-18 02:59:29 +0000 UTC - event for azuredisk-volume-tester-lm97d: {kubelet k8s-agentpool1-42835052-1} FailedMount: Unable to attach or mount volumes: unmounted volumes=[test-volume-1], unattached volumes=[kube-api-access-kjtzp test-volume-1]: timed out waiting for the condition Jun 18 03:20:12.504: INFO: POD NODE PHASE GRACE CONDITIONS Jun 18 03:20:12.504: INFO: Jun 18 03:20:12.568: INFO: Logging node info for node k8s-agentpool1-42835052-0 ... skipping 92 lines ... [90m/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning_test.go:40[0m [single-az] [90m/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning_test.go:44[0m [91m[1mshould create multiple PV objects, bind to PVCs and attach all to different pods on the same node [kubernetes.io/azure-disk] [disk.csi.azure.com] [Windows] [csi-azuredisk-scheduler-extender] [It][0m [90m/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning_test.go:372[0m [91mUnexpected error: <*errors.errorString | 0xc000323270>: { s: "timed out waiting for the condition", } timed out waiting for the condition occurred[0m ... skipping 306 lines ... Jun 18 03:35:18.875: INFO: At 2022-06-18 03:20:14 +0000 UTC - event for pvc-vcv45: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "disk.csi.azure.com" or manually created by system administrator Jun 18 03:35:18.875: INFO: At 2022-06-18 03:20:14 +0000 UTC - event for pvc-vcv45: {disk.csi.azure.com_k8s-agentpool1-42835052-0_53a95263-aec0-4b87-a465-aa36e34998da } Provisioning: External provisioner is provisioning volume for claim "azuredisk-5212/pvc-vcv45" Jun 18 03:35:18.875: INFO: At 2022-06-18 03:20:14 +0000 UTC - event for pvc-vcv45: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding Jun 18 03:35:18.875: INFO: At 2022-06-18 03:20:16 +0000 UTC - event for pvc-vcv45: {disk.csi.azure.com_k8s-agentpool1-42835052-0_53a95263-aec0-4b87-a465-aa36e34998da } ProvisioningSucceeded: Successfully provisioned volume pvc-bd4a7b21-2487-44c9-b0eb-44929cbd50f3 Jun 18 03:35:18.875: INFO: At 2022-06-18 03:20:19 +0000 UTC - event for azuredisk-volume-tester-7rdpz-69999bb84d-hvtjw: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-bd4a7b21-2487-44c9-b0eb-44929cbd50f3" Jun 18 03:35:18.875: INFO: At 2022-06-18 03:22:20 +0000 UTC - event for azuredisk-volume-tester-7rdpz-69999bb84d-hvtjw: {kubelet k8s-agentpool1-42835052-0} FailedMount: Unable to attach or mount volumes: unmounted volumes=[test-volume-1], unattached volumes=[test-volume-1 kube-api-access-r7b5t]: timed out waiting for the condition Jun 18 03:35:18.875: INFO: At 2022-06-18 03:22:20 +0000 UTC - event for azuredisk-volume-tester-7rdpz-69999bb84d-hvtjw: {kubelet k8s-agentpool1-42835052-0} FailedMount: MountVolume.MountDevice failed for volume "pvc-bd4a7b21-2487-44c9-b0eb-44929cbd50f3" : rpc error: code = DeadlineExceeded desc = context deadline exceeded Jun 18 03:35:18.909: INFO: POD NODE PHASE GRACE CONDITIONS Jun 18 03:35:18.909: INFO: Jun 18 03:35:18.977: INFO: Logging node info for node k8s-agentpool1-42835052-0 Jun 18 03:35:19.013: INFO: Node Info: &Node{ObjectMeta:{k8s-agentpool1-42835052-0 b3450a63-c373-4a96-b548-c74917f39933 32163 0 2022-06-18 01:05:58 +0000 UTC <nil> <nil> map[agentpool:agentpool1 beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_DS2_v2 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:canadacentral failure-domain.beta.kubernetes.io/zone:0 kubernetes.azure.com/cluster:kubetest-u6qgg0s3 kubernetes.azure.com/role:agent kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-agentpool1-42835052-0 kubernetes.io/os:linux kubernetes.io/role:agent node-role.kubernetes.io/agent: node.kubernetes.io/instance-type:Standard_DS2_v2 storageprofile:managed storagetier:Premium_LRS topology.disk.csi.azure.com/zone: topology.kubernetes.io/region:canadacentral topology.kubernetes.io/zone:0] map[csi.volume.kubernetes.io/nodeid:{"disk.csi.azure.com":"k8s-agentpool1-42835052-0"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2022-06-18 01:05:58 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:agentpool":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.azure.com/cluster":{},"f:kubernetes.azure.com/role":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:storageprofile":{},"f:storagetier":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kube-controller-manager Update v1 2022-06-18 01:06:07 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"10.244.2.0/24\"":{}}}} } {kubectl-label Update v1 2022-06-18 01:06:53 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/agent":{}}}} } {kube-controller-manager Update v1 2022-06-18 02:11:22 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:volumesAttached":{}}} status} {kubelet Update v1 2022-06-18 03:28:30 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.disk.csi.azure.com/zone":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:10.244.2.0/24,DoNotUseExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-u6qgg0s3/providers/Microsoft.Compute/virtualMachines/k8s-agentpool1-42835052-0,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[10.244.2.0/24],},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-azure-disk: {{8 0} {<nil>} 8 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{31036686336 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7287771136 0} {<nil>} 7116964Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-azure-disk: {{8 0} {<nil>} 8 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{27933017657 0} {<nil>} 27933017657 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{6501339136 0} {<nil>} 6348964Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-06-18 01:06:26 +0000 UTC,LastTransitionTime:2022-06-18 01:06:26 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-06-18 03:33:37 +0000 UTC,LastTransitionTime:2022-06-18 01:05:58 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-06-18 03:33:37 +0000 UTC,LastTransitionTime:2022-06-18 01:05:58 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-06-18 03:33:37 +0000 UTC,LastTransitionTime:2022-06-18 01:05:58 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-06-18 03:33:37 +0000 UTC,LastTransitionTime:2022-06-18 01:06:06 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:Hostname,Address:k8s-agentpool1-42835052-0,},NodeAddress{Type:InternalIP,Address:10.240.0.5,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:fd2eb91f0eeb436286bc238b58e4dc04,SystemUUID:c3ccb155-829c-6a40-8fdc-f652683a9f79,BootID:0082daec-2a09-4d8b-b449-5299bd7dc135,KernelVersion:5.4.0-1074-azure,OSImage:Ubuntu 18.04.6 LTS,ContainerRuntimeVersion:docker://20.10.11+azure-3,KubeletVersion:v1.23.7,KubeProxyVersion:v1.23.7,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/azuredisk-csi@sha256:423eb6cf602c064c8b2deefead5ceadd6324ed41b3d995dab5d0f6f0f4d4710f mcr.microsoft.com/oss/kubernetes-csi/azuredisk-csi:v1.10.0],SizeBytes:245959792,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/azurefile-csi@sha256:9e2ecabcf9dd9943e6600eb9fb460f45b4dc61af7cabe95d115082a029db2aaf mcr.microsoft.com/oss/kubernetes-csi/azurefile-csi:v1.9.0],SizeBytes:230470852,},ContainerImage{Names:[mcr.microsoft.com/containernetworking/azure-npm@sha256:106f669f48e5e80c4ec0afb49858ead72cf4b901cd8664e7bf81f8d789e56e12 mcr.microsoft.com/containernetworking/azure-npm:v1.2.2_hotfix],SizeBytes:175230380,},ContainerImage{Names:[k8sprow.azurecr.io/azuredisk-csi@sha256:7a2395a48c394c3f4eae282f9d9873df45a18e8a409ccddf03b7583367b0a7da k8sprow.azurecr.io/azuredisk-csi:latest-v2-444ca6fc5bce97ea7c52380569d0659c5e105ce7],SizeBytes:136453655,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/secrets-store/driver@sha256:c0d040a1c4fbfceb65663e31c09ea40f4f78e356437610cbc3fbb4bb409bd6f1 mcr.microsoft.com/oss/kubernetes-csi/secrets-store/driver:v0.0.19],SizeBytes:123229697,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-proxy@sha256:0c890e918d42c158fab68867bade2ab1db04c3408452db9611a264ba3e43e6b3 mcr.microsoft.com/oss/kubernetes/kube-proxy:v1.23.7],SizeBytes:112296063,},ContainerImage{Names:[mcr.microsoft.com/oss/azure/secrets-store/provider-azure@sha256:6f67f3d0c7cdde5702f8ce7f101b6519daa0237f0c34fecb7c058b6af8c22ad1 mcr.microsoft.com/oss/azure/secrets-store/provider-azure:0.0.12],SizeBytes:101061355,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/autoscaler/cluster-autoscaler@sha256:6f0c680d375c62e74351f8ff3ed6ddb9b72ca759e0645c329b95f64264654a6d mcr.microsoft.com/oss/kubernetes/autoscaler/cluster-autoscaler:v1.22.1],SizeBytes:99962810,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-addon-manager@sha256:32e2836018c96e73533bd4642fe438e465b81dcbfa8b7b61935a6f4d0246c7ae mcr.microsoft.com/oss/kubernetes/kube-addon-manager:v9.1.3],SizeBytes:86832059,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-addon-manager@sha256:92c2c5aad9012ee32d2a43a74966cc0adc6ccb1705ad15abb10485ecf406d88b mcr.microsoft.com/oss/kubernetes/kube-addon-manager:v9.1.5],SizeBytes:84094027,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/metrics-server@sha256:1ef9d57ce41ffcc328b92494c3bfafe401e0b9a1694a295301a1385337d52815 mcr.microsoft.com/oss/kubernetes/metrics-server:v0.5.2],SizeBytes:64327621,},ContainerImage{Names:[mcr.microsoft.com/oss/nvidia/k8s-device-plugin@sha256:0f5b52bf28239234e831697d96db63ac03cde70fe68058f964504ab7564ee810 mcr.microsoft.com/oss/nvidia/k8s-device-plugin:1.0.0-beta6],SizeBytes:64160241,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-provisioner@sha256:542c9258a1441e4927883ea73425e21f9a14043a649686bf6b51700f34c64f8e mcr.microsoft.com/oss/kubernetes-csi/csi-provisioner:v3.1.0],SizeBytes:57858438,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-provisioner@sha256:e9ddadc44ba87a4a27f67e54760a14f9986885b534b3dff170a14eae1e35d213 mcr.microsoft.com/oss/kubernetes-csi/csi-provisioner:v3.0.0],SizeBytes:56881280,},ContainerImage{Names:[k8sprow.azurecr.io/azdiskschedulerextender-csi@sha256:295239e73ff3a728379df39f3985db8347e1547ecb2658620f29f3fdc1058d34 k8sprow.azurecr.io/azdiskschedulerextender-csi:latest-v2-444ca6fc5bce97ea7c52380569d0659c5e105ce7],SizeBytes:56760067,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-resizer@sha256:ab6c2e18a4d943f2bf8688e1779622277a1d14c355c9d4bfc0d761d86c5108b3 mcr.microsoft.com/oss/kubernetes-csi/csi-resizer:v1.4.0],SizeBytes:55577322,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-snapshotter@sha256:a889e925e15f9423f7842f1b769f64cbcf6a20b6956122836fc835cf22d9073f mcr.microsoft.com/oss/kubernetes-csi/csi-snapshotter:v5.0.1],SizeBytes:55314459,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-attacher@sha256:19fbca01394f7ed80151731180cfde0a3367f038beae2a97cda7928034010057 mcr.microsoft.com/oss/kubernetes-csi/csi-attacher:v3.4.0],SizeBytes:54954134,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-resizer@sha256:c5bb71ceaac60b1a4b58739fa07b709f6248c452ff6272a384d2f7648895a750 mcr.microsoft.com/oss/kubernetes-csi/csi-resizer:v1.3.0],SizeBytes:54313772,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-snapshotter@sha256:61849a026511cf332c87d73d0a7aed803b510c3ede197ec755389686d490de72 mcr.microsoft.com/oss/kubernetes-csi/csi-snapshotter:v4.2.1],SizeBytes:54210936,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-attacher@sha256:6b41e7153ebdfdc1501aa65184624bc15fd33a52d93f88ec3a758d0f8c9b8c10 mcr.microsoft.com/oss/kubernetes-csi/csi-attacher:v3.3.0],SizeBytes:53842561,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/snapshot-controller@sha256:8c3fc3c2667004ad6bbdf723bb64c5da66a5cb8b11d4ee59b67179b686223b13 mcr.microsoft.com/oss/kubernetes-csi/snapshot-controller:v5.0.1],SizeBytes:52732401,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/snapshot-controller@sha256:be5a8dc1f990828f653c77bd0a0f1bbd13197c3019f6e1d99d590389bac36705 mcr.microsoft.com/oss/kubernetes-csi/snapshot-controller:v4.2.1],SizeBytes:51575245,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager@sha256:0ad67f9919522a07318034641ae09bf2079b417e9944c65914410594ce645468 mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager:v1.1.4],SizeBytes:51478397,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager@sha256:31ec4f7daccd3e7a8504e12657d7830651ecacbe4a487daca1b1b7695a64b070 mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager:v1.23.1],SizeBytes:51249021,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager@sha256:011712ed90fb8efcf27928b0a47ed04b98baebb31cb1b2d8ab676977ec18eedc mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager:v1.1.4],SizeBytes:50868093,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager@sha256:0f9a8fbaed65192ed7dd795be4f9c1dc48ebdef0a241fb62d456f4bed40d9875 mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager:v1.23.1],SizeBytes:50679677,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-scheduler@sha256:c35f45e8dfaa30363e589296415a057f9d32813920fc3820d6557491657f199a mcr.microsoft.com/oss/kubernetes/kube-scheduler:v1.21.2],SizeBytes:50607491,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/ip-masq-agent@sha256:1244155f2ed3f33ff154cc343b8ad285f3391d95afd7d4b1c6dcc420bc0ba3cf mcr.microsoft.com/oss/kubernetes/ip-masq-agent:v2.5.0],SizeBytes:50146762,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager@sha256:7c907ff70b90a0bdf8fae63bd744018469dd9839cde1fd0515b93e0bbd14b34e mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager:v1.0.8],SizeBytes:48963453,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager@sha256:3987d7a8c6922ce1952ee19c5cb6ea75aac7b7c1b07aa79277ad038c69fb7a31 mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager:v1.0.8],SizeBytes:48349053,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/coredns@sha256:f873bf7f0928461efe10697fa76cf0ad7a1ae3041c5b57b50dd3d0b72d273f8c mcr.microsoft.com/oss/kubernetes/coredns:1.8.6],SizeBytes:46804601,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager@sha256:8073113a20882642a980b338635cdc5945e5673a18aef192090e6fde2b89a75c mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager:v0.6.0],SizeBytes:45909032,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager@sha256:6a32329628bdea3c6d75e98aad6155b65d2e2b98ca616eb33f9ac562912804c6 mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager:v0.6.0],SizeBytes:45229096,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager@sha256:ef6c4ba564b4d11d270f7d1563c50fbeb30ccc3b94146e5059228c49f95875f5 mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager:v0.7.11],SizeBytes:44916605,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager@sha256:dbcc384758ba5ca6d249596d471292ed3785e31cdb854d48b84d70794b669b4c mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager:v0.7.11],SizeBytes:43679613,},ContainerImage{Names:[mcr.microsoft.com/oss/etcd-io/etcd@sha256:cf587862e3f1b6fa4d9a2565520a34f164bdf72c50f37af8c3c668160593246e mcr.microsoft.com/oss/etcd-io/etcd:v3.3.25],SizeBytes:41832119,},ContainerImage{Names:[mcr.microsoft.com/k8s/aad-pod-identity/mic@sha256:bd9465be94966b9a917e1e3904fa5e63dd91772ccadf304e18ffd8e4ad8ccedd mcr.microsoft.com/k8s/aad-pod-identity/mic:1.6.1],SizeBytes:41374894,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/autoscaler/cluster-proportional-autoscaler@sha256:c849d75d61943ce7f51b4c049f1a79d19a08253966c8f49c4cfb6414cc33db8b mcr.microsoft.com/oss/kubernetes/autoscaler/cluster-proportional-autoscaler:1.8.5],SizeBytes:40661903,},ContainerImage{Names:[mcr.microsoft.com/k8s/aad-pod-identity/nmi@sha256:02128fefcdb7593ac53fc342e2c53a0fc6fabd813036bf60457bf43cc2940116 mcr.microsoft.com/k8s/aad-pod-identity/nmi:1.6.1],SizeBytes:38007982,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:921f301c44dda06a325164accf22e78ecc570b5c7d9d6ee4c66bd8cbb2b60b9a mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.16],SizeBytes:26970670,},ContainerImage{Names:[mcr.microsoft.com/k8s/kms/keyvault@sha256:1a27e175f8c125209e32d2957b5509fe20757bd8cb309ff9da598799b56326fb mcr.microsoft.com/k8s/kms/keyvault:v0.0.10],SizeBytes:23077387,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-node-driver-registrar@sha256:dbec3a8166686b09b242176ab5b99e993da4126438bbce68147c3fd654f35662 mcr.microsoft.com/oss/kubernetes-csi/csi-node-driver-registrar:v2.4.0],SizeBytes:19547289,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/livenessprobe@sha256:c96a6255c42766f6b8bb1a7cda02b0060ab1b20b2e2dafcc64ec09e7646745a6 mcr.microsoft.com/oss/kubernetes-csi/livenessprobe:v2.5.0],SizeBytes:17573341,},ContainerImage{Names:[mcr.microsoft.com/oss/busybox/busybox@sha256:582a641242b49809af3a1a522f9aae8c3f047d1c6ca1dd9d8cdabd349e45b1a9 mcr.microsoft.com/oss/busybox/busybox:1.33.1],SizeBytes:1235829,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8],SizeBytes:1159025,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/pause@sha256:e3b8c20681593c21b344ad801fbb8abaf564427ee3a57a9fcfa3b455f917ce46 mcr.microsoft.com/oss/kubernetes/pause:3.4.1],SizeBytes:682696,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/disk.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-u6qgg0s3/providers/Microsoft.Compute/disks/pre-provisioned-inline-volume,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/disk.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-u6qgg0s3/providers/Microsoft.Compute/disks/pvc-76934f53-9b56-4add-b7c0-aca144f3145d,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/disk.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-u6qgg0s3/providers/Microsoft.Compute/disks/pvc-209e49e4-921f-4216-8dcf-c99d5ad92944,DevicePath:,},},Config:nil,},} Jun 18 03:35:19.014: INFO: ... skipping 90 lines ... [90m/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning_test.go:40[0m [single-az] [90m/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning_test.go:44[0m [91m[1mshould create a deployment object, write and read to it, delete the pod and write and read to it again [kubernetes.io/azure-disk] [disk.csi.azure.com] [Windows] [csi-azuredisk-scheduler-extender] [It][0m [90m/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning_test.go:436[0m [91mUnexpected error: <*errors.errorString | 0xc000e10740>: { s: "error waiting for deployment \"azuredisk-volume-tester-7rdpz\" status to match expectation: deployment status: v1.DeploymentStatus{ObservedGeneration:1, Replicas:1, UpdatedReplicas:1, ReadyReplicas:0, AvailableReplicas:0, UnavailableReplicas:1, Conditions:[]v1.DeploymentCondition{v1.DeploymentCondition{Type:\"Available\", Status:\"False\", LastUpdateTime:time.Date(2022, time.June, 18, 3, 20, 14, 0, time.Local), LastTransitionTime:time.Date(2022, time.June, 18, 3, 20, 14, 0, time.Local), Reason:\"MinimumReplicasUnavailable\", Message:\"Deployment does not have minimum availability.\"}, v1.DeploymentCondition{Type:\"Progressing\", Status:\"True\", LastUpdateTime:time.Date(2022, time.June, 18, 3, 20, 14, 0, time.Local), LastTransitionTime:time.Date(2022, time.June, 18, 3, 20, 14, 0, time.Local), Reason:\"ReplicaSetUpdated\", Message:\"ReplicaSet \\\"azuredisk-volume-tester-7rdpz-69999bb84d\\\" is progressing.\"}}, CollisionCount:(*int32)(nil)}", } error waiting for deployment "azuredisk-volume-tester-7rdpz" status to match expectation: deployment status: v1.DeploymentStatus{ObservedGeneration:1, Replicas:1, UpdatedReplicas:1, ReadyReplicas:0, AvailableReplicas:0, UnavailableReplicas:1, Conditions:[]v1.DeploymentCondition{v1.DeploymentCondition{Type:"Available", Status:"False", LastUpdateTime:time.Date(2022, time.June, 18, 3, 20, 14, 0, time.Local), LastTransitionTime:time.Date(2022, time.June, 18, 3, 20, 14, 0, time.Local), Reason:"MinimumReplicasUnavailable", Message:"Deployment does not have minimum availability."}, v1.DeploymentCondition{Type:"Progressing", Status:"True", LastUpdateTime:time.Date(2022, time.June, 18, 3, 20, 14, 0, time.Local), LastTransitionTime:time.Date(2022, time.June, 18, 3, 20, 14, 0, time.Local), Reason:"ReplicaSetUpdated", Message:"ReplicaSet \"azuredisk-volume-tester-7rdpz-69999bb84d\" is progressing."}}, CollisionCount:(*int32)(nil)} occurred[0m /home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/resources/deployment.go:126 [90m------------------------------[0m [0mDynamic Provisioning[0m [90m[single-az][0m [1mshould delete PV with reclaimPolicy "Delete" [kubernetes.io/azure-disk] [disk.csi.azure.com] [Windows][0m ... skipping 150 lines ... [1mSTEP[0m: setting up the StorageClass [1mSTEP[0m: creating a StorageClass [1mSTEP[0m: setting up the PVC and PV [1mSTEP[0m: creating a PVC [1mSTEP[0m: setting up the pod [1mSTEP[0m: deploying the pod [1mSTEP[0m: checking that the pod's command exits with no error Jun 18 03:35:39.220: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-9k6z8" in namespace "azuredisk-8419" to be "Succeeded or Failed" Jun 18 03:35:39.261: INFO: Pod "azuredisk-volume-tester-9k6z8": Phase="Pending", Reason="", readiness=false. Elapsed: 41.229262ms Jun 18 03:35:41.295: INFO: Pod "azuredisk-volume-tester-9k6z8": Phase="Pending", Reason="", readiness=false. Elapsed: 2.074887307s Jun 18 03:35:43.330: INFO: Pod "azuredisk-volume-tester-9k6z8": Phase="Pending", Reason="", readiness=false. Elapsed: 4.109778271s Jun 18 03:35:45.366: INFO: Pod "azuredisk-volume-tester-9k6z8": Phase="Pending", Reason="", readiness=false. Elapsed: 6.146109506s Jun 18 03:35:47.401: INFO: Pod "azuredisk-volume-tester-9k6z8": Phase="Pending", Reason="", readiness=false. Elapsed: 8.181444167s Jun 18 03:35:49.436: INFO: Pod "azuredisk-volume-tester-9k6z8": Phase="Pending", Reason="", readiness=false. Elapsed: 10.215904001s ... skipping 432 lines ... Jun 18 03:50:30.754: INFO: Pod "azuredisk-volume-tester-9k6z8": Phase="Pending", Reason="", readiness=false. Elapsed: 14m51.53392375s Jun 18 03:50:32.789: INFO: Pod "azuredisk-volume-tester-9k6z8": Phase="Pending", Reason="", readiness=false. Elapsed: 14m53.569402364s Jun 18 03:50:34.824: INFO: Pod "azuredisk-volume-tester-9k6z8": Phase="Pending", Reason="", readiness=false. Elapsed: 14m55.60401637s Jun 18 03:50:36.859: INFO: Pod "azuredisk-volume-tester-9k6z8": Phase="Pending", Reason="", readiness=false. Elapsed: 14m57.639169322s Jun 18 03:50:38.895: INFO: Pod "azuredisk-volume-tester-9k6z8": Phase="Pending", Reason="", readiness=false. Elapsed: 14m59.674948852s Jun 18 03:50:40.896: INFO: deleting Pod "azuredisk-8419"/"azuredisk-volume-tester-9k6z8" Jun 18 03:50:40.968: INFO: Error getting logs for pod azuredisk-volume-tester-9k6z8: the server rejected our request for an unknown reason (get pods azuredisk-volume-tester-9k6z8) [1mSTEP[0m: Deleting pod azuredisk-volume-tester-9k6z8 in namespace azuredisk-8419 [1mSTEP[0m: validating provisioned PV [1mSTEP[0m: checking the PV Jun 18 03:50:41.071: INFO: deleting PVC "azuredisk-8419"/"pvc-4x2s5" Jun 18 03:50:41.071: INFO: Deleting PersistentVolumeClaim "pvc-4x2s5" [1mSTEP[0m: waiting for claim's PV "pvc-58225466-bf6e-4b54-853c-0db68673ba76" to be deleted ... skipping 148 lines ... Jun 18 04:02:26.354: INFO: PersistentVolume pvc-2341cc0e-1f43-4722-9cb8-55203a3e1d93 found and phase=Released (1m40.773345843s) Jun 18 04:02:31.389: INFO: PersistentVolume pvc-2341cc0e-1f43-4722-9cb8-55203a3e1d93 found and phase=Released (1m45.808379684s) Jun 18 04:02:36.427: INFO: PersistentVolume pvc-2341cc0e-1f43-4722-9cb8-55203a3e1d93 found and phase=Released (1m50.846788632s) Jun 18 04:02:41.466: INFO: PersistentVolume pvc-2341cc0e-1f43-4722-9cb8-55203a3e1d93 found and phase=Released (1m55.88547056s) Jun 18 04:02:46.504: INFO: PersistentVolume pvc-2341cc0e-1f43-4722-9cb8-55203a3e1d93 found and phase=Released (2m0.923983963s) Jun 18 04:02:51.542: INFO: PersistentVolume pvc-2341cc0e-1f43-4722-9cb8-55203a3e1d93 found and phase=Released (2m5.961227968s) {"component":"entrypoint","file":"k8s.io/test-infra/prow/entrypoint/run.go:165","func":"k8s.io/test-infra/prow/entrypoint.Options.ExecuteProcess","level":"error","msg":"Process did not finish before 3h0m0s timeout","severity":"error","time":"2022-06-18T04:02:52Z"} ++ early_exit_handler ++ '[' -n 178 ']' ++ kill -TERM 178 ++ cleanup_dind ++ [[ true == \t\r\u\e ]] ++ echo 'Cleaning up after docker' ... skipping 195 lines ... Jun 18 04:17:23.023: INFO: PersistentVolume pvc-47a66dbf-d386-4474-a164-f7a465b06db4 found and phase=Released (6m32.892322255s) Jun 18 04:17:28.057: INFO: PersistentVolume pvc-47a66dbf-d386-4474-a164-f7a465b06db4 found and phase=Released (6m37.925561933s) Jun 18 04:17:33.091: INFO: PersistentVolume pvc-47a66dbf-d386-4474-a164-f7a465b06db4 found and phase=Released (6m42.959901237s) Jun 18 04:17:38.127: INFO: PersistentVolume pvc-47a66dbf-d386-4474-a164-f7a465b06db4 found and phase=Released (6m47.996253535s) Jun 18 04:17:43.161: INFO: PersistentVolume pvc-47a66dbf-d386-4474-a164-f7a465b06db4 found and phase=Released (6m53.030475343s) Jun 18 04:17:48.199: INFO: PersistentVolume pvc-47a66dbf-d386-4474-a164-f7a465b06db4 found and phase=Released (6m58.06775799s) {"component":"entrypoint","file":"k8s.io/test-infra/prow/entrypoint/run.go:255","func":"k8s.io/test-infra/prow/entrypoint.gracefullyTerminate","level":"error","msg":"Process did not exit before 15m0s grace period","severity":"error","time":"2022-06-18T04:17:52Z"} {"component":"entrypoint","error":"os: process already finished","file":"k8s.io/test-infra/prow/entrypoint/run.go:257","func":"k8s.io/test-infra/prow/entrypoint.gracefullyTerminate","level":"error","msg":"Could not kill process after grace period","severity":"error","time":"2022-06-18T04:17:52Z"}